| Message ID | ed8bab1aec7b3bd9433435f32054df0e54911fff.1544701282.git.anatoly.burakov@intel.com (mailing list archive) |
|---|---|
| State | Accepted, archived |
| Delegated to: | Thomas Monjalon |
| Headers | show |
| Series | Allow using virtio-user without hugepages | expand |
| Context | Check | Description |
|---|---|---|
| ci/checkpatch | success | coding style OK |
| ci/Intel-compilation | success | Compilation OK |
On 13-Dec-18 11:43 AM, Anatoly Burakov wrote: > When running in no-huge mode, we anonymously allocate our memory. > While this works for regular NICs and vdev's, it's not suitable > for memory sharing scenarios such as virtio with vhost_user > backend. > > To fix this, allocate no-huge memory using memfd, and register > it with memalloc just like any other memseg fd. This will enable > using rte_memseg_get_fd() API with --no-huge EAL flag. > > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com> > Acked-by: Tiwei Bie <tiwei.bie@intel.com> > --- > > Notes: > v3: > - Clarify release notes to state that the changes apply to > virtio-user NICs rather than virtio in general > > v2: > - Detect memfd support at compile time > - Change memfd-related log level to debug > > doc/guides/rel_notes/release_19_02.rst | 5 +++ > lib/librte_eal/linuxapp/eal/eal_memory.c | 54 +++++++++++++++++++++++- > 2 files changed, 57 insertions(+), 2 deletions(-) > > diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst > index 960098582..f733ad139 100644 > --- a/doc/guides/rel_notes/release_19_02.rst > +++ b/doc/guides/rel_notes/release_19_02.rst > @@ -23,6 +23,11 @@ DPDK Release 19.02 > New Features > ------------ > > +* **Support for using VirtIO without hugepages** ^^ oops, forgot to fix the title... Should be virtio-user.
On 12/13/18 12:43 PM, Anatoly Burakov wrote: > When running in no-huge mode, we anonymously allocate our memory. > While this works for regular NICs and vdev's, it's not suitable > for memory sharing scenarios such as virtio with vhost_user > backend. > > To fix this, allocate no-huge memory using memfd, and register > it with memalloc just like any other memseg fd. This will enable > using rte_memseg_get_fd() API with --no-huge EAL flag. > > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com> > Acked-by: Tiwei Bie <tiwei.bie@intel.com> > --- > > Notes: > v3: > - Clarify release notes to state that the changes apply to > virtio-user NICs rather than virtio in general > > v2: > - Detect memfd support at compile time > - Change memfd-related log level to debug > > doc/guides/rel_notes/release_19_02.rst | 5 +++ > lib/librte_eal/linuxapp/eal/eal_memory.c | 54 +++++++++++++++++++++++- > 2 files changed, 57 insertions(+), 2 deletions(-) > > diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst > index 960098582..f733ad139 100644 > --- a/doc/guides/rel_notes/release_19_02.rst > +++ b/doc/guides/rel_notes/release_19_02.rst > @@ -23,6 +23,11 @@ DPDK Release 19.02 > New Features > ------------ > > +* **Support for using VirtIO without hugepages** > + With the title change you suggested: Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com> Thanks, Maxime
diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst index 960098582..f733ad139 100644 --- a/doc/guides/rel_notes/release_19_02.rst +++ b/doc/guides/rel_notes/release_19_02.rst @@ -23,6 +23,11 @@ DPDK Release 19.02 New Features ------------ +* **Support for using VirtIO without hugepages** + + The --no-huge mode was augmented to use memfd-backed memory (on systems that + support memfd), to allow using virtio-user-based NICs without hugepages. + .. This section should contain new features added in this release. Sample format: diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c index 32feb415d..7d922a965 100644 --- a/lib/librte_eal/linuxapp/eal/eal_memory.c +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c @@ -25,6 +25,10 @@ #include <sys/time.h> #include <signal.h> #include <setjmp.h> +#ifdef F_ADD_SEALS /* if file sealing is supported, so is memfd */ +#include <linux/memfd.h> +#define MEMFD_SUPPORTED +#endif #ifdef RTE_EAL_NUMA_AWARE_HUGEPAGES #include <numa.h> #include <numaif.h> @@ -1341,12 +1345,18 @@ eal_legacy_hugepage_init(void) /* hugetlbfs can be disabled */ if (internal_config.no_hugetlbfs) { struct rte_memseg_list *msl; + int n_segs, cur_seg, fd, flags; +#ifdef MEMFD_SUPPORTED + int memfd; +#endif uint64_t page_sz; - int n_segs, cur_seg; /* nohuge mode is legacy mode */ internal_config.legacy_mem = 1; + /* nohuge mode is single-file segments mode */ + internal_config.single_file_segments = 1; + /* create a memseg list */ msl = &mcfg->memsegs[0]; @@ -1359,8 +1369,38 @@ eal_legacy_hugepage_init(void) return -1; } + /* set up parameters for anonymous mmap */ + fd = -1; + flags = MAP_PRIVATE | MAP_ANONYMOUS; + +#ifdef MEMFD_SUPPORTED + /* create a memfd and store it in the segment fd table */ + memfd = memfd_create("nohuge", 0); + if (memfd < 0) { + RTE_LOG(DEBUG, EAL, "Cannot create memfd: %s\n", + strerror(errno)); + RTE_LOG(DEBUG, EAL, "Falling back to anonymous map\n"); + } else { + /* we got an fd - now resize it */ + if (ftruncate(memfd, internal_config.memory) < 0) { + RTE_LOG(ERR, EAL, "Cannot resize memfd: %s\n", + strerror(errno)); + RTE_LOG(ERR, EAL, "Falling back to anonymous map\n"); + close(memfd); + } else { + /* creating memfd-backed file was successful. + * we want changes to memfd to be visible to + * other processes (such as vhost backend), so + * map it as shared memory. + */ + RTE_LOG(DEBUG, EAL, "Using memfd for anonymous memory\n"); + fd = memfd; + flags = MAP_SHARED; + } + } +#endif addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + flags, fd, 0); if (addr == MAP_FAILED) { RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__, strerror(errno)); @@ -1371,6 +1411,16 @@ eal_legacy_hugepage_init(void) msl->socket_id = 0; msl->len = internal_config.memory; + /* we're in single-file segments mode, so only the segment list + * fd needs to be set up. + */ + if (fd != -1) { + if (eal_memalloc_set_seg_list_fd(0, fd) < 0) { + RTE_LOG(ERR, EAL, "Cannot set up segment list fd\n"); + /* not a serious error, proceed */ + } + } + /* populate memsegs. each memseg is one page long */ for (cur_seg = 0; cur_seg < n_segs; cur_seg++) { arr = &msl->memseg_arr;