Cover Detail
Show a cover letter.
GET /api/covers/45490/?format=api
{ "id": 45490, "url": "http://patches.dpdk.org/api/covers/45490/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/cover/cover.1538044725.git.anatoly.burakov@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<cover.1538044725.git.anatoly.burakov@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/cover.1538044725.git.anatoly.burakov@intel.com", "date": "2018-09-27T10:40:57", "name": "[v6,00/21] Support externally allocated memory in DPDK", "submitter": { "id": 4, "url": "http://patches.dpdk.org/api/people/4/?format=api", "name": "Burakov, Anatoly", "email": "anatoly.burakov@intel.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/cover/cover.1538044725.git.anatoly.burakov@intel.com/mbox/", "series": [ { "id": 1543, "url": "http://patches.dpdk.org/api/series/1543/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=1543", "date": "2018-09-27T10:40:57", "name": "Support externally allocated memory in DPDK", "version": 6, "mbox": "http://patches.dpdk.org/series/1543/mbox/" } ], "comments": "http://patches.dpdk.org/api/covers/45490/comments/", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 919D41B14F;\n\tThu, 27 Sep 2018 12:41:25 +0200 (CEST)", "from mga05.intel.com (mga05.intel.com [192.55.52.43])\n\tby dpdk.org (Postfix) with ESMTP id 410F31B136\n\tfor <dev@dpdk.org>; Thu, 27 Sep 2018 12:41:24 +0200 (CEST)", "from orsmga006.jf.intel.com ([10.7.209.51])\n\tby fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t27 Sep 2018 03:41:23 -0700", "from irvmail001.ir.intel.com ([163.33.26.43])\n\tby orsmga006.jf.intel.com with ESMTP; 27 Sep 2018 03:41:19 -0700", "from sivswdev01.ir.intel.com (sivswdev01.ir.intel.com\n\t[10.237.217.45])\n\tby irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id\n\tw8RAfIKF028345; Thu, 27 Sep 2018 11:41:18 +0100", "from sivswdev01.ir.intel.com (localhost [127.0.0.1])\n\tby sivswdev01.ir.intel.com with ESMTP id w8RAfI4Q022665;\n\tThu, 27 Sep 2018 11:41:18 +0100", "(from aburakov@localhost)\n\tby sivswdev01.ir.intel.com with LOCAL id w8RAfIN7022650;\n\tThu, 27 Sep 2018 11:41:18 +0100" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.54,310,1534834800\"; d=\"scan'208\";a=\"77745679\"", "From": "Anatoly Burakov <anatoly.burakov@intel.com>", "To": "dev@dpdk.org", "Cc": "laszlo.madarassy@ericsson.com, laszlo.vadkerti@ericsson.com,\n\tandras.kovacs@ericsson.com, winnie.tian@ericsson.com,\n\tdaniel.andrasi@ericsson.com, janos.kobor@ericsson.com,\n\tgeza.koblo@ericsson.com, srinath.mannam@broadcom.com,\n\tscott.branden@broadcom.com, ajit.khaparde@broadcom.com,\n\tkeith.wiles@intel.com, bruce.richardson@intel.com, thomas@monjalon.net,\n\tshreyansh.jain@nxp.com, shahafs@mellanox.com,\n\tarybchenko@solarflare.com, alejandro.lucero@netronome.com", "Date": "Thu, 27 Sep 2018 11:40:57 +0100", "Message-Id": "<cover.1538044725.git.anatoly.burakov@intel.com>", "X-Mailer": "git-send-email 1.7.0.7", "In-Reply-To": "<cover.1537960448.git.anatoly.burakov@intel.com>", "References": "<cover.1537960448.git.anatoly.burakov@intel.com>", "Subject": "[dpdk-dev] [PATCH v6 00/21] Support externally allocated memory in\n\tDPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "This is a proposal to enable using externally allocated memory\nin DPDK.\n\nIn a nutshell, here is what is being done here:\n\n- Index internal malloc heaps by NUMA node index, rather than NUMA\n node itself (external heaps will have ID's in order of creation)\n- Add identifier string to malloc heap, to uniquely identify it\n - Each new heap will receive a unique socket ID that will be used by\n allocator to decide from which heap (internal or external) to\n allocate requested amount of memory\n- Allow creating named heaps and add/remove memory to/from those heaps\n- Allocate memseg lists at runtime, to keep track of IOVA addresses\n of externally allocated memory\n - If IOVA addresses aren't provided, use RTE_BAD_IOVA\n- Allow malloc and memzones to allocate from external heaps\n- Allow other data structures to allocate from externall heaps\n\nThe responsibility to ensure memory is accessible before using it is\non the shoulders of the user - there is no checking done with regards\nto validity of the memory (nor could there be...).\n\nThe general approach is to create heap and add memory into it. For any\nother process wishing to use the same memory, said memory must first\nbe attached (otherwise some things will not work).\n\nA design decision was made to make multiprocess synchronization a\nmanual process. Due to underlying issues with attaching to fbarrays in\nsecondary processes, this design was deemed to be better because we\ndon't want to fail to create external heap in the primary because\nsomething in the secondary has failed when in fact we may not eve have\nwanted this memory to be accessible in the secondary in the first\nplace.\n\nUsing external memory in multiprocess is *hard*, because not only\nmemory space needs to be preallocated, but it also needs to be attached\nin each process to allow other processes to access the page table. The\nattach API call may or may not succeed, depending on memory layout, for\nreasons similar to other multiprocess failures. This is treated as a\n\"known issue\" for this release.\n\nv6 -> v5 changes:\n- Fixed documentation formatting as per Marko's comments\n\nv5 -> v4 changes:\n- All processes are now able to create and destroy malloc heaps\n- Memory is automatically mapped for DMA on adding it to heap\n- Mem event callbacks are triggered on adding/removing memory\n- Fixed compile issues on FreeBSD\n- Better documentation on API/ABI changes\n\nv4 -> v3 changes:\n- Dropped sample application in favor of new testpmd flag\n- Added new flag to testpmd, with four options of mempool allocation\n- Added new API to check if a socket ID belongs to an external heap\n- Adjusted malloc and mempool code to not make any assumptions about\n IOVA-contiguousness when dealing with externally allocated memory\n\nv3 -> v2 changes:\n- Rebase on top of latest master\n- Clarifications added to mempool code as per Andrew Rynchenko's\n comments\n\nv2 -> v1 changes:\n- Fixed NULL dereference on heap socket ID lookup\n- Fixed memseg offset calculation on adding memory to heap\n- Improved unit test to test for above bugfixes\n- Restricted heap creation to primary processes only\n- Added sample application\n- Added documentation\n\nRFC -> v1 changes:\n- Removed the \"named heaps\" API, allocate using fake socket ID instead\n- Added multiprocess support\n- Everything is now thread-safe\n- Numerous bugfixes and API improvements\n\nAnatoly Burakov (21):\n mem: add length to memseg list\n mem: allow memseg lists to be marked as external\n malloc: index heaps using heap ID rather than NUMA node\n mem: do not check for invalid socket ID\n flow_classify: do not check for invalid socket ID\n pipeline: do not check for invalid socket ID\n sched: do not check for invalid socket ID\n malloc: add name to malloc heaps\n malloc: add function to query socket ID of named heap\n malloc: add function to check if socket is external\n malloc: allow creating malloc heaps\n malloc: allow destroying heaps\n malloc: allow adding memory to named heaps\n malloc: allow removing memory from named heaps\n malloc: allow attaching to external memory chunks\n malloc: allow detaching from external memory\n malloc: enable event callbacks for external memory\n test: add unit tests for external memory support\n app/testpmd: add support for external memory\n doc: add external memory feature to the release notes\n doc: add external memory feature to programmer's guide\n\n app/test-pmd/config.c | 21 +-\n app/test-pmd/parameters.c | 23 +-\n app/test-pmd/testpmd.c | 305 ++++++++++++-\n app/test-pmd/testpmd.h | 13 +-\n config/common_base | 1 +\n config/rte_config.h | 1 +\n .../prog_guide/env_abstraction_layer.rst | 37 ++\n doc/guides/rel_notes/deprecation.rst | 15 -\n doc/guides/rel_notes/release_18_11.rst | 28 +-\n doc/guides/testpmd_app_ug/run_app.rst | 12 +\n drivers/bus/fslmc/fslmc_vfio.c | 14 +-\n drivers/bus/pci/linux/pci.c | 2 +-\n drivers/net/mlx4/mlx4_mr.c | 3 +\n drivers/net/mlx5/mlx5.c | 5 +-\n drivers/net/mlx5/mlx5_mr.c | 3 +\n drivers/net/virtio/virtio_user/vhost_kernel.c | 5 +-\n .../net/virtio/virtio_user/virtio_user_dev.c | 8 +\n lib/librte_eal/bsdapp/eal/Makefile | 2 +-\n lib/librte_eal/bsdapp/eal/eal.c | 3 +\n lib/librte_eal/bsdapp/eal/eal_memory.c | 9 +-\n lib/librte_eal/common/eal_common_memory.c | 8 +-\n lib/librte_eal/common/eal_common_memzone.c | 8 +-\n .../common/include/rte_eal_memconfig.h | 9 +-\n lib/librte_eal/common/include/rte_malloc.h | 192 ++++++++\n .../common/include/rte_malloc_heap.h | 3 +\n lib/librte_eal/common/include/rte_memory.h | 9 +\n lib/librte_eal/common/malloc_elem.c | 10 +-\n lib/librte_eal/common/malloc_heap.c | 316 +++++++++++--\n lib/librte_eal/common/malloc_heap.h | 17 +\n lib/librte_eal/common/rte_malloc.c | 429 +++++++++++++++++-\n lib/librte_eal/linuxapp/eal/Makefile | 2 +-\n lib/librte_eal/linuxapp/eal/eal.c | 10 +-\n lib/librte_eal/linuxapp/eal/eal_memalloc.c | 12 +-\n lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +-\n lib/librte_eal/linuxapp/eal/eal_vfio.c | 27 +-\n lib/librte_eal/meson.build | 2 +-\n lib/librte_eal/rte_eal_version.map | 8 +\n lib/librte_flow_classify/rte_flow_classify.c | 3 +-\n lib/librte_mempool/rte_mempool.c | 57 ++-\n lib/librte_pipeline/rte_pipeline.c | 3 +-\n lib/librte_sched/rte_sched.c | 2 +-\n test/test/Makefile | 1 +\n test/test/autotest_data.py | 14 +-\n test/test/meson.build | 1 +\n test/test/test_external_mem.c | 389 ++++++++++++++++\n test/test/test_malloc.c | 3 +\n test/test/test_memzone.c | 3 +\n 47 files changed, 1913 insertions(+), 139 deletions(-)\n create mode 100644 test/test/test_external_mem.c" }