mbox series

[21.11,v2,0/3] eal: add memory pre-allocation from existing files

Message ID 20210716110806.2566788-1-dkozlyuk@nvidia.com (mailing list archive)
Headers
Series eal: add memory pre-allocation from existing files |

Message

Dmitry Kozlyuk July 16, 2021, 11:08 a.m. UTC 
  Hugepage allocation from the system takes time, resulting in slow
startup or sporadic delays later. Most of the time spent in kernel
is zero-filling memory for security reasons, which may be irrelevant
in a controlled environment. The bottleneck is memory access speed,
so for speeduup the amount of memory cleared must be reduced.
We propose a new EAL option --mem-file FILE1,FILE2,... to quickly
allocate dirty pages from existing files and clean it as necessary.
A new malloc_perf_autotest is provided to estimate the impact.
More details are explained in relevant patches.

v2: fix CI failures

Dmitry Kozlyuk (2):
  eal/linux: make hugetlbfs analysis reusable
  app/test: add allocator performance autotest

Viacheslav Ovsiienko (1):
  eal: add memory pre-allocation from existing files

 app/test/meson.build                          |   2 +
 app/test/test_malloc_perf.c                   | 157 +++++++++
 doc/guides/linux_gsg/linux_eal_parameters.rst |  17 +
 lib/eal/common/eal_common_dynmem.c            |   6 +
 lib/eal/common/eal_common_options.c           |  23 ++
 lib/eal/common/eal_internal_cfg.h             |   4 +
 lib/eal/common/eal_memalloc.h                 |   8 +-
 lib/eal/common/eal_options.h                  |   2 +
 lib/eal/common/malloc_elem.c                  |   5 +
 lib/eal/common/malloc_heap.h                  |   8 +
 lib/eal/common/rte_malloc.c                   |  16 +-
 lib/eal/include/rte_memory.h                  |   4 +-
 lib/eal/linux/eal.c                           |  28 ++
 lib/eal/linux/eal_hugepage_info.c             | 158 ++++++---
 lib/eal/linux/eal_hugepage_info.h             |  39 +++
 lib/eal/linux/eal_memalloc.c                  | 328 +++++++++++++++++-
 16 files changed, 735 insertions(+), 70 deletions(-)
 create mode 100644 app/test/test_malloc_perf.c
 create mode 100644 lib/eal/linux/eal_hugepage_info.h

Comments

Dmitry Kozlyuk Aug. 9, 2021, 9:45 a.m. UTC | #1 
2021-07-16 14:08 (UTC+0300), Dmitry Kozlyuk:
> Hugepage allocation from the system takes time, resulting in slow
> startup or sporadic delays later. Most of the time spent in kernel
> is zero-filling memory for security reasons, which may be irrelevant
> in a controlled environment. The bottleneck is memory access speed,
> so for speeduup the amount of memory cleared must be reduced.
> We propose a new EAL option --mem-file FILE1,FILE2,... to quickly
> allocate dirty pages from existing files and clean it as necessary.
> A new malloc_perf_autotest is provided to estimate the impact.
> More details are explained in relevant patches.
> 
> v2: fix CI failures

Hi Anatoly, could you review please?
Dmitry Kozlyuk Aug. 30, 2021, 8:21 a.m. UTC | #2 
> -----Original Message-----
> From: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
> Sent: August 9, 2021, 12:45
> [...]
> Hi Anatoly, could you review please?

Hi Anatoly, did you have time to look?
Patch 1/3 may be useful on its own, because it makes adding and testing patches like [1] easier.

[1]: http://patchwork.dpdk.org/project/dpdk/patch/20210809112434.383123-1-john.levon@nutanix.com/