Cover Comment List
List cover comments
GET /api/covers/44222/comments/?format=api
[ { "id": 85473, "web_url": "http://patches.dpdk.org/comment/85473/", "msgid": "<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "list_archive_url": "https://inbox.dpdk.org/dev/DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com", "date": "2018-09-13T07:44:15", "subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "submitter": { "id": 634, "url": "http://patches.dpdk.org/api/people/634/?format=api", "name": "Shahaf Shuler", "email": "shahafs@mellanox.com" }, "content": "Hi Anatoly,\n\nFirst thanks for the patchset, it is a great enhancement. \n\nSee question below. \n\nTuesday, September 4, 2018 4:12 PM, Anatoly Burakov:\n> Subject: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n> DPDK\n> \n> This is a proposal to enable using externally allocated memory in DPDK.\n> \n> In a nutshell, here is what is being done here:\n> \n> - Index internal malloc heaps by NUMA node index, rather than NUMA\n> node itself (external heaps will have ID's in order of creation)\n> - Add identifier string to malloc heap, to uniquely identify it\n> - Each new heap will receive a unique socket ID that will be used by\n> allocator to decide from which heap (internal or external) to\n> allocate requested amount of memory\n> - Allow creating named heaps and add/remove memory to/from those\n> heaps\n> - Allocate memseg lists at runtime, to keep track of IOVA addresses\n> of externally allocated memory\n> - If IOVA addresses aren't provided, use RTE_BAD_IOVA\n> - Allow malloc and memzones to allocate from external heaps\n> - Allow other data structures to allocate from externall heaps\n> \n> The responsibility to ensure memory is accessible before using it is on the\n> shoulders of the user - there is no checking done with regards to validity of\n> the memory (nor could there be...).\n\nThat makes sense. However who should be in-charge of mapping this memory for dma access?\nThe user or internally be the PMD when encounter the first packet or while traversing the existing mempools? \n\n> \n> The general approach is to create heap and add memory into it. For any other\n> process wishing to use the same memory, said memory must first be\n> attached (otherwise some things will not work).\n> \n> A design decision was made to make multiprocess synchronization a manual\n> process. Due to underlying issues with attaching to fbarrays in secondary\n> processes, this design was deemed to be better because we don't want to\n> fail to create external heap in the primary because something in the\n> secondary has failed when in fact we may not eve have wanted this memory\n> to be accessible in the secondary in the first place.\n> \n> Using external memory in multiprocess is *hard*, because not only memory\n> space needs to be preallocated, but it also needs to be attached in each\n> process to allow other processes to access the page table. The attach API call\n> may or may not succeed, depending on memory layout, for reasons similar to\n> other multiprocess failures. This is treated as a \"known issue\" for this release.\n> \n> RFC -> v1 changes:\n> - Removed the \"named heaps\" API, allocate using fake socket ID instead\n> - Added multiprocess support\n> - Everything is now thread-safe\n> - Numerous bugfixes and API improvements\n> \n> Anatoly Burakov (16):\n> mem: add length to memseg list\n> mem: allow memseg lists to be marked as external\n> malloc: index heaps using heap ID rather than NUMA node\n> mem: do not check for invalid socket ID\n> flow_classify: do not check for invalid socket ID\n> pipeline: do not check for invalid socket ID\n> sched: do not check for invalid socket ID\n> malloc: add name to malloc heaps\n> malloc: add function to query socket ID of named heap\n> malloc: allow creating malloc heaps\n> malloc: allow destroying heaps\n> malloc: allow adding memory to named heaps\n> malloc: allow removing memory from named heaps\n> malloc: allow attaching to external memory chunks\n> malloc: allow detaching from external memory\n> test: add unit tests for external memory support\n> \n> config/common_base | 1 +\n> config/rte_config.h | 1 +\n> drivers/bus/fslmc/fslmc_vfio.c | 7 +-\n> drivers/bus/pci/linux/pci.c | 2 +-\n> drivers/net/mlx4/mlx4_mr.c | 3 +\n> drivers/net/mlx5/mlx5.c | 5 +-\n> drivers/net/mlx5/mlx5_mr.c | 3 +\n> drivers/net/virtio/virtio_user/vhost_kernel.c | 5 +-\n> lib/librte_eal/bsdapp/eal/eal.c | 3 +\n> lib/librte_eal/bsdapp/eal/eal_memory.c | 9 +-\n> lib/librte_eal/common/eal_common_memory.c | 9 +-\n> lib/librte_eal/common/eal_common_memzone.c | 8 +-\n> .../common/include/rte_eal_memconfig.h | 6 +-\n> lib/librte_eal/common/include/rte_malloc.h | 181 +++++++++\n> .../common/include/rte_malloc_heap.h | 3 +\n> lib/librte_eal/common/include/rte_memory.h | 9 +\n> lib/librte_eal/common/malloc_heap.c | 287 +++++++++++--\n> lib/librte_eal/common/malloc_heap.h | 17 +\n> lib/librte_eal/common/rte_malloc.c | 383 ++++++++++++++++-\n> lib/librte_eal/linuxapp/eal/eal.c | 3 +\n> lib/librte_eal/linuxapp/eal/eal_memalloc.c | 12 +-\n> lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +-\n> lib/librte_eal/linuxapp/eal/eal_vfio.c | 17 +-\n> lib/librte_eal/rte_eal_version.map | 7 +\n> lib/librte_flow_classify/rte_flow_classify.c | 3 +-\n> lib/librte_mempool/rte_mempool.c | 31 +-\n> lib/librte_pipeline/rte_pipeline.c | 3 +-\n> lib/librte_sched/rte_sched.c | 2 +-\n> test/test/Makefile | 1 +\n> test/test/autotest_data.py | 14 +-\n> test/test/meson.build | 1 +\n> test/test/test_external_mem.c | 384 ++++++++++++++++++\n> test/test/test_malloc.c | 3 +\n> test/test/test_memzone.c | 3 +\n> 34 files changed, 1346 insertions(+), 84 deletions(-) create mode 100644\n> test/test/test_external_mem.c\n> \n> --\n> 2.17.1", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 1E1FA4CBB;\n\tThu, 13 Sep 2018 09:44:19 +0200 (CEST)", "from EUR03-AM5-obe.outbound.protection.outlook.com\n\t(mail-eopbgr30046.outbound.protection.outlook.com [40.107.3.46])\n\tby dpdk.org (Postfix) with ESMTP id 815434CB5\n\tfor <dev@dpdk.org>; Thu, 13 Sep 2018 09:44:17 +0200 (CEST)", "from DB7PR05MB4426.eurprd05.prod.outlook.com (52.134.109.15) by\n\tDB7PR05MB4939.eurprd05.prod.outlook.com (20.176.235.151) with\n\tMicrosoft SMTP Server (version=TLS1_2,\n\tcipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\t15.20.1122.18; Thu, 13 Sep 2018 07:44:15 +0000", "from DB7PR05MB4426.eurprd05.prod.outlook.com\n\t([fe80::c8e7:d9c1:5054:693b]) by\n\tDB7PR05MB4426.eurprd05.prod.outlook.com\n\t([fe80::c8e7:d9c1:5054:693b%6]) with mapi id 15.20.1122.020;\n\tThu, 13 Sep 2018 07:44:15 +0000" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;\n\ts=selector1;\n\th=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n\tbh=Djsf/CqAGDMeEkzO9OE9HmCt9zLw2hFS8SF3PJ9FdYA=;\n\tb=XwyAxvE5AEiDe29z1iNE2lW/BrA43poSFeZKRxcsR+Mv5rBJt8mpNoqZFwowlsshilUsuwkf9VGSBwY3T0QvAbOTrt8od6IZxXU/MXsaUpet+CgTzHs8TANs0xPasMn2ATBmC2MOAySer5rvvZYz8/lr+gShJEqG8WfRDIPmsvM=", "From": "Shahaf Shuler <shahafs@mellanox.com>", "To": "Anatoly Burakov <anatoly.burakov@intel.com>,\n\t\"dev@dpdk.org\" <dev@dpdk.org>", "CC": "\"laszlo.madarassy@ericsson.com\" <laszlo.madarassy@ericsson.com>,\n\t\"laszlo.vadkerti@ericsson.com\" <laszlo.vadkerti@ericsson.com>,\n\t\"andras.kovacs@ericsson.com\" <andras.kovacs@ericsson.com>,\n\t\"winnie.tian@ericsson.com\" <winnie.tian@ericsson.com>,\n\t\"daniel.andrasi@ericsson.com\" <daniel.andrasi@ericsson.com>,\n\t\"janos.kobor@ericsson.com\" <janos.kobor@ericsson.com>,\n\t\"srinath.mannam@broadcom.com\" <srinath.mannam@broadcom.com>,\n\t\"scott.branden@broadcom.com\" <scott.branden@broadcom.com>,\n\t\"ajit.khaparde@broadcom.com\" <ajit.khaparde@broadcom.com>,\n\t\"keith.wiles@intel.com\" <keith.wiles@intel.com>,\n\t\"bruce.richardson@intel.com\"\n\t<bruce.richardson@intel.com>, Thomas Monjalon <thomas@monjalon.net>", "Thread-Topic": "[dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "Thread-Index": "AQHURFDf2xhCgwt9L0+pUaIzcsQ80aTt4bgQ", "Date": "Thu, 13 Sep 2018 07:44:15 +0000", "Message-ID": "<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "References": "<cover.1536064999.git.anatoly.burakov@intel.com>", "In-Reply-To": "<cover.1536064999.git.anatoly.burakov@intel.com>", "Accept-Language": "en-US", "Content-Language": "en-US", "X-MS-Has-Attach": "", "X-MS-TNEF-Correlator": "", "authentication-results": "spf=none (sender IP is )\n\tsmtp.mailfrom=shahafs@mellanox.com; ", "x-originating-ip": "[193.47.165.251]", "x-ms-publictraffictype": "Email", "x-microsoft-exchange-diagnostics": "1; DB7PR05MB4939;\n\t6:+ViBy4Q+oEMaLuYdnCKYbHKsGEUksBE/Nal1LR1AO8rg2uWLudHSzfl19oC5DFTtCDcQsvgm1b+8phgbxvOXQb9cQAuN/8nf+jtWa6XR6q/Wv1NfEnJNqzNaNQdjs/p3DSSpqIPw1IbXshhIf1bMSpZT0cdMSd26YRw4fXcE5Vjo3gRoONW6V9H+GCpYneu88MTrhxqGBPDpwNkvnJ88rvDxpZGGfopx2jUx9KPwbB40OGkxEjkmB5S4QBSqYCGECYsG7ZHoKQNNr8XK6u1ONaXCONb0VFCooKPRwzFed2dfy7HsG5x4dqr/BSyMS4NV6pKGGFTKD47HNRGw5jmP8qTYcbDqxuznTgflsDwoC6ou9nUC6fyh0TI4p1tA/XkBnmoHh+j+yHznzkG6T+W/CqL/XIaFtIXUd7qBluQiVG16x63uHcH37d8ps5znA+Z1XmT9By1ztXacusxAfxGePQ==;\n\t5:i5vgojNRkabRah5Wjos8laOmCP8cQbldHRgH8gQHyrGyDJ7veAKldjd4DJKm/eQwA3EdZMFCe9zoZdeZjyuOY43674eNFflWEXuTD3oNkql4L+R6laHtMEgYU2ttY09AVunSZxI5/B5cTw97OKfA7LPmcOqT5FMs9lrYsl7Y6R0=;\n\t7:IGufxm61os+zD8o/VDESASeYVN/XO28I/iJPTbGvMR9h4yEr/beO+v0Chu/5+msH9SX9zUaqoAe89XBbjQkNA7iRdKuC/iw/4bhPLg1nAXsUzNa06ZevePlCAMvkPi7OwvLLPvOkLWtoVjeTD8H2op+MDf5/uaVM8TaOGmEj3eJIpmp6kXygt/lXpxz42/Dy3WAArKpC+rltLOi0UGHJ/sjOscCiECaDQmpsXRvgkyGyqIY53NnIz2bcS7joM1+E", "x-ms-exchange-antispam-srfa-diagnostics": "SOS;", "x-ms-office365-filtering-correlation-id": "449df17e-f2ea-4f1b-2e82-08d6194cb44a", "x-ms-office365-filtering-ht": "Tenant", "x-microsoft-antispam": "BCL:0; PCL:0;\n\tRULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020);\n\tSRVR:DB7PR05MB4939; ", "x-ms-traffictypediagnostic": "DB7PR05MB4939:", "x-ld-processed": "a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr", "x-microsoft-antispam-prvs": "<DB7PR05MB4939F2E910099A246A6B8298C31A0@DB7PR05MB4939.eurprd05.prod.outlook.com>", "x-exchange-antispam-report-test": "UriScan:;", "x-ms-exchange-senderadcheck": "1", "x-exchange-antispam-report-cfa-test": "BCL:0; PCL:0;\n\tRULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231311)(944501410)(52105095)(3002001)(10201501046)(93006095)(93001095)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699050);\n\tSRVR:DB7PR05MB4939; BCL:0; PCL:0; RULEID:; SRVR:DB7PR05MB4939; ", "x-forefront-prvs": "07943272E1", "x-forefront-antispam-report": "SFV:NSPM;\n\tSFS:(10009020)(396003)(366004)(39860400002)(376002)(346002)(136003)(51914003)(199004)(189003)(229853002)(97736004)(8676002)(2900100001)(68736007)(14454004)(561944003)(8936002)(33656002)(105586002)(6116002)(81166006)(81156014)(26005)(2906002)(106356001)(3846002)(6506007)(102836004)(9686003)(7696005)(76176011)(99286004)(7416002)(316002)(110136005)(55016002)(54906003)(66066001)(5250100002)(53936002)(5024004)(14444005)(256004)(4326008)(25786009)(186003)(5660300001)(6246003)(446003)(11346002)(476003)(486006)(86362001)(2501003)(7736002)(478600001)(74316002)(6436002)(305945005);\n\tDIR:OUT; SFP:1101; SCL:1; SRVR:DB7PR05MB4939;\n\tH:DB7PR05MB4426.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;\n\tPTR:InfoNoRecords; A:1; MX:1; ", "received-spf": "None (protection.outlook.com: mellanox.com does not designate\n\tpermitted sender hosts)", "x-microsoft-antispam-message-info": "S0GryIeeOtFbHGUmIJzotP+b9dqoznainv51bXurv5S5dDT+Hfo9di3JgtCuqumSyb8UL0+SBARCR+WKWMrEjJT4ipc8KypOXKEFQfMtLW+P9YXg2pTSRwPSZK2PqgQ0is/9Rm/7lCz7Ozbg41SBkru4iJu+7TWFvOobLQbzK3Wv0ObbI0Kejire8SRO/BWaYrcL7KhnZNHYDW+mXekIoN0VPwif/cs3J/ZTKKi3NHoKZiY1dz+SgAGa6TfQMzW2Uk0/t6552d5oqHQcAU+FuNg1WdjgkbE15uBGZyhLi6yKX721luPf7Mguo7qhl/oZUXXeV4U7ClJ9tO9i3BxuodhUed7iUAVx6CIe7VcQr04=", "spamdiagnosticoutput": "1:99", "spamdiagnosticmetadata": "NSPM", "Content-Type": "text/plain; charset=\"us-ascii\"", "Content-Transfer-Encoding": "quoted-printable", "MIME-Version": "1.0", "X-OriginatorOrg": "Mellanox.com", "X-MS-Exchange-CrossTenant-Network-Message-Id": "449df17e-f2ea-4f1b-2e82-08d6194cb44a", "X-MS-Exchange-CrossTenant-originalarrivaltime": "13 Sep 2018 07:44:15.1412\n\t(UTC)", "X-MS-Exchange-CrossTenant-fromentityheader": "Hosted", "X-MS-Exchange-CrossTenant-id": "a652971c-7d2e-4d9b-a6a4-d149256f461b", "X-MS-Exchange-Transport-CrossTenantHeadersStamped": "DB7PR05MB4939", "Subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 85616, "web_url": "http://patches.dpdk.org/comment/85616/", "msgid": "<9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com", "date": "2018-09-17T10:07:17", "subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "submitter": { "id": 4, "url": "http://patches.dpdk.org/api/people/4/?format=api", "name": "Anatoly Burakov", "email": "anatoly.burakov@intel.com" }, "content": "On 13-Sep-18 8:44 AM, Shahaf Shuler wrote:\n> Hi Anatoly,\n> \n> First thanks for the patchset, it is a great enhancement.\n> \n> See question below.\n> \n> Tuesday, September 4, 2018 4:12 PM, Anatoly Burakov:\n>> Subject: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n>> DPDK\n>>\n>> This is a proposal to enable using externally allocated memory in DPDK.\n>>\n>> In a nutshell, here is what is being done here:\n>>\n>> - Index internal malloc heaps by NUMA node index, rather than NUMA\n>> node itself (external heaps will have ID's in order of creation)\n>> - Add identifier string to malloc heap, to uniquely identify it\n>> - Each new heap will receive a unique socket ID that will be used by\n>> allocator to decide from which heap (internal or external) to\n>> allocate requested amount of memory\n>> - Allow creating named heaps and add/remove memory to/from those\n>> heaps\n>> - Allocate memseg lists at runtime, to keep track of IOVA addresses\n>> of externally allocated memory\n>> - If IOVA addresses aren't provided, use RTE_BAD_IOVA\n>> - Allow malloc and memzones to allocate from external heaps\n>> - Allow other data structures to allocate from externall heaps\n>>\n>> The responsibility to ensure memory is accessible before using it is on the\n>> shoulders of the user - there is no checking done with regards to validity of\n>> the memory (nor could there be...).\n> \n> That makes sense. However who should be in-charge of mapping this memory for dma access?\n> The user or internally be the PMD when encounter the first packet or while traversing the existing mempools?\n> \nHi Shahaf,\n\nThere are two ways this can be solved. The first way is to perform VFIO \nmapping automatically on adding/attaching memory. The second is to force \nuser to do it manually. For now, the latter is chosen because user knows \nbest if they intend to do DMA on that memory, but i'm open to suggestions.\n\nThere is an issue with some devices and buses (i.e. bus/fslmc) bypassing \nEAL VFIO infrastructure and performing their own VFIO/DMA mapping magic, \nbut solving that problem is outside the scope of this patchset. Those \ndevices/buses should fix themselves :)\n\nWhen not using VFIO, it's out of our hands anyway.", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 7E65725D9;\n\tMon, 17 Sep 2018 12:07:32 +0200 (CEST)", "from mga17.intel.com (mga17.intel.com [192.55.52.151])\n\tby dpdk.org (Postfix) with ESMTP id C496E19F5\n\tfor <dev@dpdk.org>; Mon, 17 Sep 2018 12:07:30 +0200 (CEST)", "from orsmga003.jf.intel.com ([10.7.209.27])\n\tby fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t17 Sep 2018 03:07:29 -0700", "from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.55])\n\t([10.237.220.55])\n\tby orsmga003.jf.intel.com with ESMTP; 17 Sep 2018 03:07:18 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.53,385,1531810800\"; d=\"scan'208\";a=\"84154100\"", "To": "Shahaf Shuler <shahafs@mellanox.com>, \"dev@dpdk.org\" <dev@dpdk.org>", "Cc": "\"laszlo.madarassy@ericsson.com\" <laszlo.madarassy@ericsson.com>,\n\t\"laszlo.vadkerti@ericsson.com\" <laszlo.vadkerti@ericsson.com>,\n\t\"andras.kovacs@ericsson.com\" <andras.kovacs@ericsson.com>,\n\t\"winnie.tian@ericsson.com\" <winnie.tian@ericsson.com>,\n\t\"daniel.andrasi@ericsson.com\" <daniel.andrasi@ericsson.com>,\n\t\"janos.kobor@ericsson.com\" <janos.kobor@ericsson.com>,\n\t\"srinath.mannam@broadcom.com\" <srinath.mannam@broadcom.com>,\n\t\"scott.branden@broadcom.com\" <scott.branden@broadcom.com>,\n\t\"ajit.khaparde@broadcom.com\" <ajit.khaparde@broadcom.com>,\n\t\"keith.wiles@intel.com\" <keith.wiles@intel.com>,\n\t\"bruce.richardson@intel.com\" <bruce.richardson@intel.com>,\n\tThomas Monjalon <thomas@monjalon.net>", "References": "<cover.1536064999.git.anatoly.burakov@intel.com>\n\t<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "From": "\"Burakov, Anatoly\" <anatoly.burakov@intel.com>", "Message-ID": "<9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com>", "Date": "Mon, 17 Sep 2018 11:07:17 +0100", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101\n\tThunderbird/52.9.1", "MIME-Version": "1.0", "In-Reply-To": "<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "Content-Type": "text/plain; charset=utf-8; format=flowed", "Content-Language": "en-US", "Content-Transfer-Encoding": "7bit", "Subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 85643, "web_url": "http://patches.dpdk.org/comment/85643/", "msgid": "<DB7PR05MB442699F68941ED3EA7DB9FD0C31E0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "list_archive_url": "https://inbox.dpdk.org/dev/DB7PR05MB442699F68941ED3EA7DB9FD0C31E0@DB7PR05MB4426.eurprd05.prod.outlook.com", "date": "2018-09-17T12:16:40", "subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "submitter": { "id": 634, "url": "http://patches.dpdk.org/api/people/634/?format=api", "name": "Shahaf Shuler", "email": "shahafs@mellanox.com" }, "content": "Monday, September 17, 2018 1:07 PM, Burakov, Anatoly:\n> Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory\n> in DPDK\n> \n> On 13-Sep-18 8:44 AM, Shahaf Shuler wrote:\n\n[...]\n\n> >> The responsibility to ensure memory is accessible before using it is\n> >> on the shoulders of the user - there is no checking done with regards\n> >> to validity of the memory (nor could there be...).\n> >\n> > That makes sense. However who should be in-charge of mapping this\n> memory for dma access?\n> > The user or internally be the PMD when encounter the first packet or while\n> traversing the existing mempools?\n> >\n> Hi Shahaf,\n> \n> There are two ways this can be solved. The first way is to perform VFIO\n> mapping automatically on adding/attaching memory. The second is to force\n> user to do it manually. For now, the latter is chosen because user knows best\n> if they intend to do DMA on that memory, but i'm open to suggestions.\n\nI agree with that approach, and will add not only if the mempool is for dma or not but also which ports will use this mempool (this can effect on the mapping). \nHowever I don't think this is generic enough to use only VFIO. As you said, there are some devices not using VFIO for mapping rather some proprietary driver utility. \nIMO DPDK should introduce generic and device agnostic APIs to the user. \n\nMy suggestion is instead of doing vfio_dma_map that or vfio_dma_unmap that have a generic dma_map(uint8_t port, address, len). Each driver will register with its own mapping callback (can be vfio_dma_map).\nIt can be outside of this series, just wondering the people opinion on such approach. \n\n> \n> There is an issue with some devices and buses (i.e. bus/fslmc) bypassing EAL\n> VFIO infrastructure and performing their own VFIO/DMA mapping magic, but\n> solving that problem is outside the scope of this patchset. Those\n> devices/buses should fix themselves :)\n> \n> When not using VFIO, it's out of our hands anyway.\n\nWhy? \nVFIO is not a must requirement for devices in DPDK. \n\n> \n> --\n> Thanks,\n> Anatoly", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 1FE795F2B;\n\tMon, 17 Sep 2018 14:16:43 +0200 (CEST)", "from EUR01-VE1-obe.outbound.protection.outlook.com\n\t(mail-ve1eur01on0043.outbound.protection.outlook.com [104.47.1.43])\n\tby dpdk.org (Postfix) with ESMTP id 11F065F29\n\tfor <dev@dpdk.org>; Mon, 17 Sep 2018 14:16:42 +0200 (CEST)", "from DB7PR05MB4426.eurprd05.prod.outlook.com (52.134.109.15) by\n\tDB7PR05MB5381.eurprd05.prod.outlook.com (20.177.122.210) with\n\tMicrosoft SMTP Server (version=TLS1_2,\n\tcipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\t15.20.1122.18; Mon, 17 Sep 2018 12:16:40 +0000", "from DB7PR05MB4426.eurprd05.prod.outlook.com\n\t([fe80::c8e7:d9c1:5054:693b]) by\n\tDB7PR05MB4426.eurprd05.prod.outlook.com\n\t([fe80::c8e7:d9c1:5054:693b%6]) with mapi id 15.20.1122.020;\n\tMon, 17 Sep 2018 12:16:40 +0000" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;\n\ts=selector1;\n\th=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n\tbh=syrxU84lwQfjdAU05Bxw1XYE0O+L0uRcNKWi3jx+ZQM=;\n\tb=Hxl0feMpwLzU9iriT2L8A+jNxLgkKP/Axj0keS7X5LMlXfe4Shce6/6etS0DrE6RJBEGU2XNQXf5GqttWvMm152YeijEoU9DdasGTXFqXCK3nufX6x1DmSkhuUz/LfwDCCpKOuqTqsGA+KP60CID3c+C9IpYaApWZ85pad8Pl80=", "From": "Shahaf Shuler <shahafs@mellanox.com>", "To": "\"Burakov, Anatoly\" <anatoly.burakov@intel.com>, \"dev@dpdk.org\"\n\t<dev@dpdk.org>", "CC": "\"laszlo.madarassy@ericsson.com\" <laszlo.madarassy@ericsson.com>,\n\t\"laszlo.vadkerti@ericsson.com\" <laszlo.vadkerti@ericsson.com>,\n\t\"andras.kovacs@ericsson.com\" <andras.kovacs@ericsson.com>,\n\t\"winnie.tian@ericsson.com\" <winnie.tian@ericsson.com>,\n\t\"daniel.andrasi@ericsson.com\" <daniel.andrasi@ericsson.com>,\n\t\"janos.kobor@ericsson.com\" <janos.kobor@ericsson.com>,\n\t\"srinath.mannam@broadcom.com\" <srinath.mannam@broadcom.com>,\n\t\"scott.branden@broadcom.com\" <scott.branden@broadcom.com>,\n\t\"ajit.khaparde@broadcom.com\" <ajit.khaparde@broadcom.com>,\n\t\"keith.wiles@intel.com\" <keith.wiles@intel.com>,\n\t\"bruce.richardson@intel.com\"\n\t<bruce.richardson@intel.com>, Thomas Monjalon <thomas@monjalon.net>", "Thread-Topic": "[dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "Thread-Index": "AQHURFDf2xhCgwt9L0+pUaIzcsQ80aTt4bgQgAZyE4CAACIrMA==", "Date": "Mon, 17 Sep 2018 12:16:40 +0000", "Message-ID": "<DB7PR05MB442699F68941ED3EA7DB9FD0C31E0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "References": "<cover.1536064999.git.anatoly.burakov@intel.com>\n\t<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>\n\t<9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com>", "In-Reply-To": "<9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com>", "Accept-Language": "en-US", "Content-Language": "en-US", "X-MS-Has-Attach": "", "X-MS-TNEF-Correlator": "", "authentication-results": "spf=none (sender IP is )\n\tsmtp.mailfrom=shahafs@mellanox.com; ", "x-originating-ip": "[31.154.10.105]", "x-ms-publictraffictype": "Email", "x-microsoft-exchange-diagnostics": "1; DB7PR05MB5381;\n\t6:RntbfK6RpCZlg9OuotSnMy/UTnwqdHj/m/59mRxKlIquBjD1fUHs1JA51JhacrIMTVTQ/Eab+UsiV93sstswg1k4Psq2HpA7XeIRHJA+WY6Wp0+dgNFDXUXN/ItlJBsUIUqWqMKdLcnodAII/k8Jv+eL+VTvD+dGbut7w6X2gT197YkWdDZnto+qmB8zVSyKOoKcIt5MBPL6FkEdQtryP3+XsWggpsD2lNden1yiaYvwkt4TN/9whj51l8LNkU4O/qOSez9RFmyNFJQm+sH4wg57oLcVgKRJKEd1WuRXr52POxlpBRkegTsAOX0DWIja1BrpYXI2tPUFYuFCHqTk1r9hJTSiexV1IodT5BcneCCCzKW0o8bRZDAmhp1m2sY82sGgWHfyRdQlxt/hHhWl0nz8dGq6vXHLUqOpq6zcoxIr8vj93Tm6GnqApQn3joOh0PY4BOCh0DKzIhSVYESWxA==;\n\t5:sd+iyms17p7MW0xjkq0SBsCbjWfAxmJuTC2cnsmTPaPJJ1WaEfpiS4CrsuIRWo1yTB79klikghRmy+CtcG26RF9JAWzvXBDEuAwVvFPb7CyGuZxjXz+BuI0CJeubvQnTMrjFpZ/3ehHCRbVAKIuB0gP/ZYmUyn3Wm4RU6XMYAEs=;\n\t7:QJpBaZBZaVbaVHbifrrcaU2S5YQjvuGkkdvsJ8yEyJsCcQqqqvywZVGO+gqiYxvSddLQOS4w42akhS414SsuEJ2eB5oA3IFV3uBmZQjSFoQGYIXdfmJCR7PAMNTHOIAjRJ6qmdjdKX7iOOq0i4yEPkV+vuYXAKyakdFP0xg6mWlC5Ml1QBrb44V+8jd8xIQRhFlZT1s7nIk9Tkr/TxPKopyoxuZHMn2BNp0pqljxca4mvnhi24pSWvcJelWQtPwv", "x-ms-exchange-antispam-srfa-diagnostics": "SOS;", "x-ms-office365-filtering-correlation-id": "5114a58f-528a-4fdb-3f13-08d61c976c50", "x-ms-office365-filtering-ht": "Tenant", "x-microsoft-antispam": "BCL:0; PCL:0;\n\tRULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020);\n\tSRVR:DB7PR05MB5381; ", "x-ms-traffictypediagnostic": "DB7PR05MB5381:", "x-ld-processed": "a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr", "x-microsoft-antispam-prvs": "<DB7PR05MB5381CA421C18665A9A341A3EC31E0@DB7PR05MB5381.eurprd05.prod.outlook.com>", "x-exchange-antispam-report-test": "UriScan:;", "x-ms-exchange-senderadcheck": "1", "x-exchange-antispam-report-cfa-test": "BCL:0; PCL:0;\n\tRULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3231355)(944501410)(52105095)(93006095)(93001095)(3002001)(10201501046)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(201708071742011)(7699050);\n\tSRVR:DB7PR05MB5381; BCL:0; PCL:0; RULEID:; SRVR:DB7PR05MB5381; ", "x-forefront-prvs": "0798146F16", "x-forefront-antispam-report": "SFV:NSPM;\n\tSFS:(10009020)(39860400002)(136003)(366004)(346002)(396003)(376002)(189003)(199004)(2900100001)(6436002)(5250100002)(3846002)(14444005)(256004)(5024004)(6116002)(68736007)(305945005)(4326008)(478600001)(33656002)(6246003)(2501003)(25786009)(7736002)(14454004)(86362001)(8936002)(81156014)(8676002)(110136005)(54906003)(11346002)(9686003)(486006)(53936002)(316002)(7416002)(53546011)(6506007)(97736004)(7696005)(99286004)(76176011)(229853002)(446003)(102836004)(106356001)(105586002)(81166006)(476003)(186003)(26005)(5660300001)(2906002)(55016002)(66066001)(74316002);\n\tDIR:OUT; SFP:1101; SCL:1; SRVR:DB7PR05MB5381;\n\tH:DB7PR05MB4426.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;\n\tPTR:InfoNoRecords; MX:1; A:1; ", "received-spf": "None (protection.outlook.com: mellanox.com does not designate\n\tpermitted sender hosts)", "x-microsoft-antispam-message-info": "PIhHAz7apIO444us1AKRBoGoGn1NQf58izQrbUwcLwrUNsbgs/svtDX4CUnQ7ksQZ4Xh3X9pO+O3sgTuzD0AGfWH1GhWuiw6TFCyEKCZn9/S9c/UK8WoDK/b694o0SuchaM8RR/AJlXd78RaivA/YrgjiWlgrG1NG5yCrjT74Gc+XSIHJyIipaEuuxqY3A7paj1CWdu0CoJg2OUrA0MZ71OfQvPpwPfnpnL+oLB4sVLHF7ADatvAYL4VnGH6DIjWaEIg7361yXHj0Ckz4rD2bZFfaFPVQxhGHUv6E+UqCbnStuUhWHmstzkn3G7HwIRBnryqkgtOU5VOjaH4++bqZkCDhD1SKLs1UK7DaRxduwA=", "spamdiagnosticoutput": "1:99", "spamdiagnosticmetadata": "NSPM", "Content-Type": "text/plain; charset=\"utf-8\"", "Content-Transfer-Encoding": "base64", "MIME-Version": "1.0", "X-OriginatorOrg": "Mellanox.com", "X-MS-Exchange-CrossTenant-Network-Message-Id": "5114a58f-528a-4fdb-3f13-08d61c976c50", "X-MS-Exchange-CrossTenant-originalarrivaltime": "17 Sep 2018 12:16:40.1891\n\t(UTC)", "X-MS-Exchange-CrossTenant-fromentityheader": "Hosted", "X-MS-Exchange-CrossTenant-id": "a652971c-7d2e-4d9b-a6a4-d149256f461b", "X-MS-Exchange-Transport-CrossTenantHeadersStamped": "DB7PR05MB5381", "Subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 85654, "web_url": "http://patches.dpdk.org/comment/85654/", "msgid": "<b6449567-d6ff-12cd-5a33-d18ecaf73a94@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/b6449567-d6ff-12cd-5a33-d18ecaf73a94@intel.com", "date": "2018-09-17T13:00:20", "subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "submitter": { "id": 4, "url": "http://patches.dpdk.org/api/people/4/?format=api", "name": "Anatoly Burakov", "email": "anatoly.burakov@intel.com" }, "content": "On 17-Sep-18 1:16 PM, Shahaf Shuler wrote:\n> Monday, September 17, 2018 1:07 PM, Burakov, Anatoly:\n>> Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory\n>> in DPDK\n>>\n>> On 13-Sep-18 8:44 AM, Shahaf Shuler wrote:\n> \n> [...]\n> \n>>>> The responsibility to ensure memory is accessible before using it is\n>>>> on the shoulders of the user - there is no checking done with regards\n>>>> to validity of the memory (nor could there be...).\n>>>\n>>> That makes sense. However who should be in-charge of mapping this\n>> memory for dma access?\n>>> The user or internally be the PMD when encounter the first packet or while\n>> traversing the existing mempools?\n>>>\n>> Hi Shahaf,\n>>\n>> There are two ways this can be solved. The first way is to perform VFIO\n>> mapping automatically on adding/attaching memory. The second is to force\n>> user to do it manually. For now, the latter is chosen because user knows best\n>> if they intend to do DMA on that memory, but i'm open to suggestions.\n> \n> I agree with that approach, and will add not only if the mempool is for dma or not but also which ports will use this mempool (this can effect on the mapping).\n\nThat is perhaps too hardware-specific - this should probably be handled \ninside the driver callbacks.\n\n> However I don't think this is generic enough to use only VFIO. As you said, there are some devices not using VFIO for mapping rather some proprietary driver utility.\n> IMO DPDK should introduce generic and device agnostic APIs to the user.\n> \n> My suggestion is instead of doing vfio_dma_map that or vfio_dma_unmap that have a generic dma_map(uint8_t port, address, len). Each driver will register with its own mapping callback (can be vfio_dma_map).\n> It can be outside of this series, just wondering the people opinion on such approach.\n\nI don't disagree. I don't like bus/net/etc drivers doing their own thing \nwith regards to mapping, and i would by far prefer generic way to set up \nDMA maps, to which VFIO will be a subscriber.\n\n> \n>>\n>> There is an issue with some devices and buses (i.e. bus/fslmc) bypassing EAL\n>> VFIO infrastructure and performing their own VFIO/DMA mapping magic, but\n>> solving that problem is outside the scope of this patchset. Those\n>> devices/buses should fix themselves :)\n>>\n>> When not using VFIO, it's out of our hands anyway.\n> \n> Why?\n> VFIO is not a must requirement for devices in DPDK.\n\nWhen i say \"out of our hands\", what i mean to say is, currently as far \nas EAL API is concerned, there is no DMA mapping outside of VFIO.\n\n> \n>>\n>> --\n>> Thanks,\n>> Anatoly", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id A45878E7F;\n\tMon, 17 Sep 2018 15:00:26 +0200 (CEST)", "from mga02.intel.com (mga02.intel.com [134.134.136.20])\n\tby dpdk.org (Postfix) with ESMTP id 46B9D8E73\n\tfor <dev@dpdk.org>; Mon, 17 Sep 2018 15:00:25 +0200 (CEST)", "from orsmga003.jf.intel.com ([10.7.209.27])\n\tby orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t17 Sep 2018 06:00:24 -0700", "from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.55])\n\t([10.237.220.55])\n\tby orsmga003.jf.intel.com with ESMTP; 17 Sep 2018 06:00:20 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.53,385,1531810800\"; d=\"scan'208\";a=\"84191457\"", "To": "Shahaf Shuler <shahafs@mellanox.com>, \"dev@dpdk.org\" <dev@dpdk.org>", "Cc": "\"laszlo.madarassy@ericsson.com\" <laszlo.madarassy@ericsson.com>,\n\t\"laszlo.vadkerti@ericsson.com\" <laszlo.vadkerti@ericsson.com>,\n\t\"andras.kovacs@ericsson.com\" <andras.kovacs@ericsson.com>,\n\t\"winnie.tian@ericsson.com\" <winnie.tian@ericsson.com>,\n\t\"daniel.andrasi@ericsson.com\" <daniel.andrasi@ericsson.com>,\n\t\"janos.kobor@ericsson.com\" <janos.kobor@ericsson.com>,\n\t\"srinath.mannam@broadcom.com\" <srinath.mannam@broadcom.com>,\n\t\"scott.branden@broadcom.com\" <scott.branden@broadcom.com>,\n\t\"ajit.khaparde@broadcom.com\" <ajit.khaparde@broadcom.com>,\n\t\"keith.wiles@intel.com\" <keith.wiles@intel.com>,\n\t\"bruce.richardson@intel.com\" <bruce.richardson@intel.com>,\n\tThomas Monjalon <thomas@monjalon.net>", "References": "<cover.1536064999.git.anatoly.burakov@intel.com>\n\t<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>\n\t<9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com>\n\t<DB7PR05MB442699F68941ED3EA7DB9FD0C31E0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "From": "\"Burakov, Anatoly\" <anatoly.burakov@intel.com>", "Message-ID": "<b6449567-d6ff-12cd-5a33-d18ecaf73a94@intel.com>", "Date": "Mon, 17 Sep 2018 14:00:20 +0100", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101\n\tThunderbird/52.9.1", "MIME-Version": "1.0", "In-Reply-To": "<DB7PR05MB442699F68941ED3EA7DB9FD0C31E0@DB7PR05MB4426.eurprd05.prod.outlook.com>", "Content-Type": "text/plain; charset=utf-8; format=flowed", "Content-Language": "en-US", "Content-Transfer-Encoding": "7bit", "Subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 85710, "web_url": "http://patches.dpdk.org/comment/85710/", "msgid": "<bf854303-5060-5939-aadd-458e467533bc@nxp.com>", "list_archive_url": "https://inbox.dpdk.org/dev/bf854303-5060-5939-aadd-458e467533bc@nxp.com", "date": "2018-09-18T12:29:56", "subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "submitter": { "id": 497, "url": "http://patches.dpdk.org/api/people/497/?format=api", "name": "Shreyansh Jain", "email": "shreyansh.jain@nxp.com" }, "content": "On Monday 17 September 2018 06:30 PM, Burakov, Anatoly wrote:\n> On 17-Sep-18 1:16 PM, Shahaf Shuler wrote:\n>> Monday, September 17, 2018 1:07 PM, Burakov, Anatoly:\n>>> Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated \n>>> memory\n>>> in DPDK\n>>>\n>>> On 13-Sep-18 8:44 AM, Shahaf Shuler wrote:\n>>\n>> [...]\n>>\n>>>>> The responsibility to ensure memory is accessible before using it is\n>>>>> on the shoulders of the user - there is no checking done with regards\n>>>>> to validity of the memory (nor could there be...).\n>>>>\n>>>> That makes sense. However who should be in-charge of mapping this\n>>> memory for dma access?\n>>>> The user or internally be the PMD when encounter the first packet or \n>>>> while\n>>> traversing the existing mempools?\n>>>>\n>>> Hi Shahaf,\n>>>\n>>> There are two ways this can be solved. The first way is to perform VFIO\n>>> mapping automatically on adding/attaching memory. The second is to force\n>>> user to do it manually. For now, the latter is chosen because user \n>>> knows best\n>>> if they intend to do DMA on that memory, but i'm open to suggestions.\n>>\n>> I agree with that approach, and will add not only if the mempool is \n>> for dma or not but also which ports will use this mempool (this can \n>> effect on the mapping).\n> \n> That is perhaps too hardware-specific - this should probably be handled \n> inside the driver callbacks.\n> \n>> However I don't think this is generic enough to use only VFIO. As you \n>> said, there are some devices not using VFIO for mapping rather some \n>> proprietary driver utility.\n>> IMO DPDK should introduce generic and device agnostic APIs to the user.\n>>\n>> My suggestion is instead of doing vfio_dma_map that or vfio_dma_unmap \n>> that have a generic dma_map(uint8_t port, address, len). Each driver \n>> will register with its own mapping callback (can be vfio_dma_map).\n>> It can be outside of this series, just wondering the people opinion on \n>> such approach.\n> \n> I don't disagree. I don't like bus/net/etc drivers doing their own thing \n> with regards to mapping, and i would by far prefer generic way to set up \n> DMA maps, to which VFIO will be a subscriber.\n> \n>>\n>>>\n>>> There is an issue with some devices and buses (i.e. bus/fslmc) \n>>> bypassing EAL\n>>> VFIO infrastructure and performing their own VFIO/DMA mapping magic, but\n>>> solving that problem is outside the scope of this patchset. Those\n>>> devices/buses should fix themselves :)\n\nDMA mapping is a very common principle and can be easily be a candidate \nfor lets-make-generic-movement, but, being close to hardware (or \nhardware specific), it does require the driver to have some flexibility \nin terms of its eventual implementation.\n\nI maintain one of those drivers (bus/fslmc) in DPDK which needs to have \nspecial VFIO layer - and from that experience, I can say that VFIO \nmapping does require some flexibility. SoC semantics are sometimes too \ncomplex to pin to general-universally-agreed-standard concept. (or, one \ncan easily call it a 'bug', while it is a 'feature' for others :D)\n\nIn fact, NXP has another driver (bus/dpaa) which doesn't even work with \nVFIO - loves to work directly with Phys_addr. And, it is not at a lower \npriority than one with VFIO.\n\nThus, I really don't think a strongly controlled VFIO mapping should be \nEAL's responsibility. Failure because of lack of mapping is a driver's \nproblem.\n\n>>>\n>>> When not using VFIO, it's out of our hands anyway.\n>>\n>> Why?\n>> VFIO is not a must requirement for devices in DPDK.\n> \n> When i say \"out of our hands\", what i mean to say is, currently as far \n> as EAL API is concerned, there is no DMA mapping outside of VFIO.\n> \n>>\n>>>\n>>> -- \n>>> Thanks,\n>>> Anatoly\n> \n>", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 7096A137C;\n\tTue, 18 Sep 2018 14:30:28 +0200 (CEST)", "from EUR03-DB5-obe.outbound.protection.outlook.com\n\t(mail-eopbgr40043.outbound.protection.outlook.com [40.107.4.43])\n\tby dpdk.org (Postfix) with ESMTP id 8BCBE1041\n\tfor <dev@dpdk.org>; Tue, 18 Sep 2018 14:30:26 +0200 (CEST)", "from [10.232.14.39] (14.142.187.166) by\n\tAM0PR04MB4674.eurprd04.prod.outlook.com (2603:10a6:208:75::16) with\n\tMicrosoft SMTP Server (version=TLS1_2,\n\tcipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n\t15.20.1143.18; Tue, 18 Sep 2018 12:30:19 +0000" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1;\n\th=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n\tbh=yKQGPoIehCByTD77NeSXXI0sT1Vs/F2TY2cwGsIpAzE=;\n\tb=S18qpyD0wAYBbdTWlY32uRBpwUNhlJexUV4yrEPC2VRKyY+RdY7dY4W0HrL5UEIz+Dv1MPai63+/14f7X2ihMMzeG1MYh0tZWindhoKsSYNanASW3Jj+YnhpO8hBclDiOD6DvhMD6U83BYOx51Wtdnb6eCfeVe32kk7CWScWDQ0=", "Authentication-Results": "spf=none (sender IP is )\n\tsmtp.mailfrom=shreyansh.jain@nxp.com; ", "To": "\"Burakov, Anatoly\" <anatoly.burakov@intel.com>,\n\tShahaf Shuler <shahafs@mellanox.com>", "Cc": "\"dev@dpdk.org\" <dev@dpdk.org>,\n\t\"laszlo.madarassy@ericsson.com\" <laszlo.madarassy@ericsson.com>,\n\t\"laszlo.vadkerti@ericsson.com\" <laszlo.vadkerti@ericsson.com>,\n\t\"andras.kovacs@ericsson.com\" <andras.kovacs@ericsson.com>,\n\t\"winnie.tian@ericsson.com\" <winnie.tian@ericsson.com>,\n\t\"daniel.andrasi@ericsson.com\" <daniel.andrasi@ericsson.com>,\n\t\"janos.kobor@ericsson.com\" <janos.kobor@ericsson.com>,\n\t\"srinath.mannam@broadcom.com\" <srinath.mannam@broadcom.com>,\n\t\"scott.branden@broadcom.com\" <scott.branden@broadcom.com>,\n\t\"ajit.khaparde@broadcom.com\" <ajit.khaparde@broadcom.com>,\n\t\"keith.wiles@intel.com\" <keith.wiles@intel.com>,\n\t\"bruce.richardson@intel.com\" <bruce.richardson@intel.com>,\n\tThomas Monjalon <thomas@monjalon.net>", "References": "<cover.1536064999.git.anatoly.burakov@intel.com>\n\t<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>\n\t<9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com>\n\t<DB7PR05MB442699F68941ED3EA7DB9FD0C31E0@DB7PR05MB4426.eurprd05.prod.outlook.com>\n\t<b6449567-d6ff-12cd-5a33-d18ecaf73a94@intel.com>", "From": "Shreyansh Jain <shreyansh.jain@nxp.com>", "Message-ID": "<bf854303-5060-5939-aadd-458e467533bc@nxp.com>", "Date": "Tue, 18 Sep 2018 17:59:56 +0530", "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.9.1", "MIME-Version": "1.0", "In-Reply-To": "<b6449567-d6ff-12cd-5a33-d18ecaf73a94@intel.com>", "Content-Type": "text/plain; charset=utf-8; format=flowed", "Content-Language": "en-US", "Content-Transfer-Encoding": "7bit", "X-Originating-IP": "[14.142.187.166]", "X-ClientProxiedBy": "SG2PR04CA0196.apcprd04.prod.outlook.com\n\t(2603:1096:4:14::34) To AM0PR04MB4674.eurprd04.prod.outlook.com\n\t(2603:10a6:208:75::16)", "X-MS-PublicTrafficType": "Email", "X-MS-Office365-Filtering-Correlation-Id": "c93c5c5e-d617-47fb-f730-08d61d62824c", "X-MS-Office365-Filtering-HT": "Tenant", "X-Microsoft-Antispam": "BCL:0; PCL:0;\n\tRULEID:(7020095)(4652040)(8989299)(4534165)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020);\n\tSRVR:AM0PR04MB4674; ", "X-Microsoft-Exchange-Diagnostics": [ "1; AM0PR04MB4674;\n\t3:dQUqTmjOMFzI1SiurAHK1fdiIG5I46hsJiASHa/c4oTvPAHvGPHQl2xqXLcvq3JB3Ky3OcRY67Hlpa2UHannnk5gYvbVACm4+oJxhNsSskHmd36b8G0QhWR+mxjE82DnWp/sGUT30bklqe8oP+tRDgXdYludMl2aOTVQnpXLj1vHqkkSYF/Lc1F5Y7sqpcyMRH15evsDiXpy4NcQ3XiGcM79Ae0XoqL2I6PGF21G3fy+sAxq8jMQneW3ypItYqoH;\n\t25:zHpq9NBhqvuPpYSwnYw+ub9RgoMl0Gfht3GNz7Z6ZcZXDEJTsdsMbVYYHJHxIqOG1PuUfn9pHMRca8Q72bXCEAJDwjMxdfRi5rXKWyOT0qVZ1Z3DxZuZnI8/Cqwti3z2H420wlfhT0phUeANap62XSDgyPfDKLZh0PRsbAA3gGAfc2Znrp6ztntF+v+nfmRgbv4JIUTsFFJCxdx+l/c0T3iWA1nZqv2OyfYibTS9WVQYkAlHsc77ZnXePOFq0QDKTpVp5aMhj87enpncK7aCZymh6O25UV2ynLpzhWeFxSDzE3LNtPRDRy2fuTBz7AI+6B17uBUfMYuvSeFFKgQ33A==;\n\t31:zTzNlXQu+rFi9mu5yP/iHKYLwbwMp8wQYMTlPSGkzPZTHdW0GzJthC7YDsmRLYaO7z4Jkox6Upfx9dHbkc5V+cwqNfxTcM9YGo94jgYU0wb5/xlwQmXgCJq7vTq4ETgLE0Wj04/VC+fbEc+DcGPcs6h2gbcbl+cz6B9dlYgLfSDSDD5875DNEXSqL2lyqEnyADkkxR1/kU4q7EXlWP2IclZZlMH9NPzK/uWzHtqDhz8=", "1; AM0PR04MB4674;\n\t20:KfBQMF+Py4Z5vh7x/hLlesALFKlvhoKlVRcnZjBv1wJmnwSXVM4Jo+QHqBZ5LoB+CmA/avF/hBOZ7Iw4KoRfGTcD1kZjB3s9XUIzbRZZEXUBQfJY3m2s/V3sksB9c0GqMK0g9vyRddeVLICFih2pjqOy/qfaL8Crtg24P5vjnjwDdLl+jO9OnmQEbYBrnoIOhrh/QpaQQctbSHTeOgrotIIZDVOmcVDLLjzATuCuRU2zZaEeMITePuBoBUUtpejTzhMp77WZLBRk1YaBFg0md3VGimsru36yknC+GvIgZ+h60bQZ7TpFI5G5PDgtNIpZecLN/QyWeRGZ5rR1lebsQ506u+fXoKhC4ZXrrqtS5GpPjU6WGd0qrx/hc/eGSU0YKrLKDP3hLdUhOL3xZ2krpMUuYGec5/OGjfCvRDftn+rGriFmj/6Qzox3VRdkhg9rkCtXe4UWqKGPZ/72lRFQqytK14d2BQ4yxfwHTxgfK9+FCSbCdII6dXlFdV0vmOWn;\n\t4:Qgkk2DkaoaJQczZiQ3DIZv26GHLTdtMG2Zc3bgqg7VLWYx7m0MAVotedMCfECQW9QBflmga8sqUE7Jwmsr5j6hZgYW3pFW9/29WavuXz9ftrLIrKOZrrcExs3bnFiYF720bJg+fDKm8qhk72nbzmcFniBuTILATYWyAiAEagp/6OW3+Y+WtqBhKHajJt8AjSv0xCXvgBjkB6A7sSQgERAlotuQM6oCfyFwtsrGb9nnWVS8u83WEL24phFSXAdjUnUwUGIfE5Hs6TILkq4tdSxNwxHm1gxBi00fvdRnuBGtAyWo333fN1H7dGXPj94Aj7", "=?utf-8?q?1=3BAM0PR04MB4674=3B23=3ADXm6?=\n\t=?utf-8?q?POXcsl41n8yRgRpxuzp4OdIpwfJggypR/AJ94bl14A45xPPiHH0ALfBS?=\n\t=?utf-8?q?uhs2LsSfPQiRSAkV6Vfk7GLOkplKZZrs1JaKsCZy843gsMN6yfXibKP8?=\n\t=?utf-8?q?HZ62K7GdhQIWNVtr1+u6SMbRq6fsPQJmIPGdvq3TimC+9nxd+4m9N066?=\n\t=?utf-8?q?kGbfBvcgcSoa5M86N45Y8/gVE+qP9KdajpVv3OAOCXx94h+9UkbOvsSO?=\n\t=?utf-8?q?bmBdcffeR+UcJdNDjQB7KwDUnqKsC7l2oDlJcAjlZkucLUeENeGt01Sm?=\n\t=?utf-8?q?hjDn7liUHpNBov13bXrK47rBcr5pVqMGgMJQQjtFPTwMWL79wGau4XjO?=\n\t=?utf-8?q?NWDro7myBOKzApHaVTWpLvBOyDirjrP+E15qKqs56uqRLkLrvcsuAA5y?=\n\t=?utf-8?q?g6kZdgHRnXQ1MlzDzOW0dlud4MQW5iAcXP1eOHpGEYD24fGqtfEcFbdl?=\n\t=?utf-8?q?GpmyiZo0k2oRp73reAy5ehPdC46NM/TK+HrEvgCsb7LIGjQkMFkK3g/3?=\n\t=?utf-8?q?cVm35vZxukMzAcYeb54voO5lPTr0h0LR0Qi3THP8hLMNgEa4mSXzhjgp?=\n\t=?utf-8?q?hkewrBcC++vqgH2MlybXkU0bQPGhgKUgSELzXd7knMZcESg6e7BFMQuk?=\n\t=?utf-8?q?pKelIdpgg3mXeK0B9uIlX6ODuuquk7DLjDGjkA0/qYajbhu1RC1OFUpI?=\n\t=?utf-8?q?MBJ7QGfQoj9HtWyUEYjdmCsaduZ/RgYXdQmktR5Ze4uN74RvawJo/PBG?=\n\t=?utf-8?q?Epw3roIpkx7DiFKAm2GsRYpG2Nhm3E5GLT+Q2Vz4YmhL/hV+DzSUAHpz?=\n\t=?utf-8?q?rNckL0K8GVRN0Vw4uJBW+SHcHV3FguAe3nCbjooLgMH4rmYYhn5fR6Eh?=\n\t=?utf-8?q?3sG22KDfcZSjXPzXhZGYP7qxsEk703vhGJGhMirZ4UxBv2++ISAZEFNe?=\n\t=?utf-8?q?x2BOWoAZZlWUF7pAdxpdQcG4KxDv9PsK9g+KgfqeNs5YdXn1HVLdjawt?=\n\t=?utf-8?q?VErqmGSoKCnisH5XDhHm6j/Hwt5/MAJ1Sb3TvQAPvVx9rNm5ajAKzJpK?=\n\t=?utf-8?q?eC88OpIAO9M8T+1V7VSXAzqvPGSi6RbTkpuq2tav/cKNjkpMdAA8xB8E?=\n\t=?utf-8?q?mOlXrR8lXyj8n/eL6D6TpHwhiTwmlkbTVE4cFXqd/9VMUr4KgOIfQCyG?=\n\t=?utf-8?q?+smeagxwKLX4iW1owOhrttNc8XxkxtTrPP2aH0Trm+1cGzoOJkTb1SGu?=\n\t=?utf-8?q?w4Kg/03u8i9ztN8OKBARRKzfm1J82Jq8UUyEOCUu298FypplGoxmRQFn?=\n\t=?utf-8?q?pytca0/1DJOLAGbF5QcPPEnmNrotBxAAS5suuJZdFTj0NoatXYXjZs1g?=\n\t=?utf-8?q?UASdfbZY2FOnA4196VhEAxs7B9oQC2y8CYU8/I4iKnkXjniu5JeK9BW0?=\n\t=?utf-8?q?/yNLVYKZ0leiJs5QJJ3aAHwvZr7Pk5hbAWb0tOW32PM/bDmMHz2n7BYE?=\n\t=?utf-8?q?LAz/yt5BZHNtvpBrog2NIwUtG1wdStU2Zav6YZTa0PY3vnyiEc4nxuhd?=\n\t=?utf-8?q?r4smqTnFsISebNXiUYH5xNznhOrERWc8jiSFHV/ln7r9nPg5aeeFPosc?=\n\t=?utf-8?q?SHbxD//lQ1VeNObZv9xkHddU1tjBMpYcHDdvKC+eu4MwOhymPXvWD1QG?=\n\t=?utf-8?q?lpLex7/RJbM+RjcmtRAXVAq5c3latxzMrQ71LhWHRgnMmhKQIA=3D=3D?=", "1; AM0PR04MB4674;\n\t6:XBoAghrN7UHxRm113iLOhOFwdc8oqa3xrZoNdcJ5Ku/ZBedD/dazlKeLixenJGPuiNzjAcbEeOG74A3sGhE0Rd6PaR8Oix6MOPsTsXWCNkqkdJ2uemkMbciGhez5M92dPrLfyY2qDccKm7bIf3JSjM5TlXXTaR9GWRTjszZqhh55Y6jC+5JJbtRh7pt5/l3ZKGOVqyWFZmcSET9VkF1vxWLb6yqcW62KcDlsTcNK2uG60SuFYGCKKEumOARWTyn2+H78gfb4ghLoXfr/A6kOpA0euSTH2SHBTKM7UuySW288dtB270ZLTDIKz/sPzzt4FxVkRvpmZ2XOUQrDxge3M7rxbLo//3znvCr8mlEGTTnXRTeJx/fQHF+gJxmV1z0xsoLpnmEmZ4sSrMWJRT8iOm0D0coyqaXHnVLgbV/SVTOb7LrYu+ciQ0pGg9Ab+GoTUVHLwEg4xUUwA3hTq5qJ8g==;\n\t5:OsBu9VwX3FFCkY8JiiSt3kY6Iil3Szusc7/7hmkej8zWC7u+YRe5gEBG41s22jZFPv2Wrac/KmdqfUQhZk7O2Ih8UntOCKf9zdUle6x36MJ/qA/6yN6ZmdWmaY1WMX8R23gyvIFl5P8e2WR2bttRWKgFNoEXzQLuDQ4n0uDhl7A=;\n\t7:Xvdke1y7YOZ7uebgcNtoTmOihHU+TWkyGD9g3GsUhm2+XBZAmv58Lg76vfivhNphW6C5UBVvPfovIC6uAWm8jqPfNMsMI1M/gbL8+l0iAvgiVFDmR/9vMAisa283znAuzTmHXqAyt3owWg3Mzpug3xWDiO1F+syD1xpQLsMq/jVPX8iVdZnNuobTNmF8VNQuBJ14azOjcgjZvD0AGLWXsT/D/VEEwT7Jb7o/eLaI1a/Cvmlh7sZaYfpHWUow4Yzl" ], "X-MS-TrafficTypeDiagnostic": "AM0PR04MB4674:", "X-Microsoft-Antispam-PRVS": "<AM0PR04MB4674E9D332B4031F948445A6901D0@AM0PR04MB4674.eurprd04.prod.outlook.com>", "X-Exchange-Antispam-Report-Test": "UriScan:(17755550239193);", "X-MS-Exchange-SenderADCheck": "1", "X-Exchange-Antispam-Report-CFA-Test": "BCL:0; PCL:0;\n\tRULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(3002001)(3231355)(944501410)(52105095)(93006095)(93001095)(6055026)(149027)(150027)(6041310)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699050);\n\tSRVR:AM0PR04MB4674; BCL:0; PCL:0; RULEID:; SRVR:AM0PR04MB4674; ", "X-Forefront-PRVS": "0799B1B2D7", "X-Forefront-Antispam-Report": "SFV:NSPM;\n\tSFS:(10009020)(6049001)(376002)(396003)(39860400002)(136003)(346002)(366004)(189003)(199004)(31696002)(76176011)(7736002)(8936002)(305945005)(8676002)(446003)(67846002)(44832011)(25786009)(81156014)(81166006)(86362001)(36756003)(478600001)(386003)(93886005)(55236004)(3846002)(6116002)(53546011)(5660300001)(50466002)(65806001)(65956001)(66066001)(64126003)(6666003)(7416002)(68736007)(2906002)(47776003)(65826007)(230700001)(316002)(16576012)(6486002)(229853002)(110136005)(54906003)(5009440100003)(956004)(52146003)(2616005)(2486003)(26005)(6246003)(14444005)(5024004)(58126008)(77096007)(52116002)(97736004)(4326008)(53936002)(186003)(16526019)(31686004)(106356001)(23676004)(11346002)(476003)(486006)(105586002)(110426005);\n\tDIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR04MB4674; H:[10.232.14.39]; FPR:;\n\tSPF:None; \n\tLANG:en; PTR:InfoNoRecords; MX:1; A:1; ", "Received-SPF": "None (protection.outlook.com: nxp.com does not designate\n\tpermitted sender hosts)", "X-Microsoft-Antispam-Message-Info": "OTQdg8mmFepscjBhIHaIUuPorhFpeF9oKS70y40BqvhKc8v9ZSMNp3WMN1jchm9G8QKH1YUDNfyqbg0d9OpNML44vTrDyMj820NR9DgAVZhNU3t5Q237GT2H/AzIvFPYjjVFl9KdijhWiV0Stf5cM+NFgYNSnxTgRlLEiO88RCGGPJWMxvZWowvL3QKNZv+9cJOjPLCfKjFeI5UjBWjHJJfXLAXpUv9wlFyy3vGrOfnYwgkIwt0e/3OYS678YdbySH7rrxrvUazektDOzvBsY7Ls38xaueeZ40jKpTsFc0UDllSAKFqAq5YCOHs+QpR4RGjCId0aIbvkz73l8Ywvy5mTm7uXNIxQnqjj0YticQo=", "SpamDiagnosticOutput": "1:99", "SpamDiagnosticMetadata": "NSPM", "X-OriginatorOrg": "nxp.com", "X-MS-Exchange-CrossTenant-OriginalArrivalTime": "18 Sep 2018 12:30:19.6188\n\t(UTC)", "X-MS-Exchange-CrossTenant-Network-Message-Id": "c93c5c5e-d617-47fb-f730-08d61d62824c", "X-MS-Exchange-CrossTenant-FromEntityHeader": "Hosted", "X-MS-Exchange-CrossTenant-Id": "686ea1d3-bc2b-4c6f-a92c-d99c5c301635", "X-MS-Exchange-Transport-CrossTenantHeadersStamped": "AM0PR04MB4674", "Subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 85735, "web_url": "http://patches.dpdk.org/comment/85735/", "msgid": "<cd2327e2-f48a-8765-1e83-72bb4b643e0f@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/cd2327e2-f48a-8765-1e83-72bb4b643e0f@intel.com", "date": "2018-09-18T15:15:17", "subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "submitter": { "id": 4, "url": "http://patches.dpdk.org/api/people/4/?format=api", "name": "Anatoly Burakov", "email": "anatoly.burakov@intel.com" }, "content": "On 18-Sep-18 1:29 PM, Shreyansh Jain wrote:\n> On Monday 17 September 2018 06:30 PM, Burakov, Anatoly wrote:\n>> On 17-Sep-18 1:16 PM, Shahaf Shuler wrote:\n>>> Monday, September 17, 2018 1:07 PM, Burakov, Anatoly:\n>>>> Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated \n>>>> memory\n>>>> in DPDK\n>>>>\n>>>> On 13-Sep-18 8:44 AM, Shahaf Shuler wrote:\n>>>\n>>> [...]\n>>>\n>>>>>> The responsibility to ensure memory is accessible before using it is\n>>>>>> on the shoulders of the user - there is no checking done with regards\n>>>>>> to validity of the memory (nor could there be...).\n>>>>>\n>>>>> That makes sense. However who should be in-charge of mapping this\n>>>> memory for dma access?\n>>>>> The user or internally be the PMD when encounter the first packet \n>>>>> or while\n>>>> traversing the existing mempools?\n>>>>>\n>>>> Hi Shahaf,\n>>>>\n>>>> There are two ways this can be solved. The first way is to perform VFIO\n>>>> mapping automatically on adding/attaching memory. The second is to \n>>>> force\n>>>> user to do it manually. For now, the latter is chosen because user \n>>>> knows best\n>>>> if they intend to do DMA on that memory, but i'm open to suggestions.\n>>>\n>>> I agree with that approach, and will add not only if the mempool is \n>>> for dma or not but also which ports will use this mempool (this can \n>>> effect on the mapping).\n>>\n>> That is perhaps too hardware-specific - this should probably be \n>> handled inside the driver callbacks.\n>>\n>>> However I don't think this is generic enough to use only VFIO. As you \n>>> said, there are some devices not using VFIO for mapping rather some \n>>> proprietary driver utility.\n>>> IMO DPDK should introduce generic and device agnostic APIs to the user.\n>>>\n>>> My suggestion is instead of doing vfio_dma_map that or vfio_dma_unmap \n>>> that have a generic dma_map(uint8_t port, address, len). Each driver \n>>> will register with its own mapping callback (can be vfio_dma_map).\n>>> It can be outside of this series, just wondering the people opinion \n>>> on such approach.\n>>\n>> I don't disagree. I don't like bus/net/etc drivers doing their own \n>> thing with regards to mapping, and i would by far prefer generic way \n>> to set up DMA maps, to which VFIO will be a subscriber.\n>>\n>>>\n>>>>\n>>>> There is an issue with some devices and buses (i.e. bus/fslmc) \n>>>> bypassing EAL\n>>>> VFIO infrastructure and performing their own VFIO/DMA mapping magic, \n>>>> but\n>>>> solving that problem is outside the scope of this patchset. Those\n>>>> devices/buses should fix themselves :)\n> \n> DMA mapping is a very common principle and can be easily be a candidate \n> for lets-make-generic-movement, but, being close to hardware (or \n> hardware specific), it does require the driver to have some flexibility \n> in terms of its eventual implementation.\n\nPerhaps i didn't word my response clearly enough. I didn't mean to say \n(or imply) that EAL must handle all DMA mappings itself. Rather, EAL \nshould provide a generic infrastructure of maintaining current mappings \netc., and provide a subscription mechanism for other users (e.g. \ndrivers) so that the details of implementation of exactly how to map \nthings for DMA is up to the drivers.\n\nIn other words, we agree :)\n\n> \n> I maintain one of those drivers (bus/fslmc) in DPDK which needs to have \n> special VFIO layer - and from that experience, I can say that VFIO \n> mapping does require some flexibility. SoC semantics are sometimes too \n> complex to pin to general-universally-agreed-standard concept. (or, one \n> can easily call it a 'bug', while it is a 'feature' for others :D)\n> \n> In fact, NXP has another driver (bus/dpaa) which doesn't even work with \n> VFIO - loves to work directly with Phys_addr. And, it is not at a lower \n> priority than one with VFIO.\n> \n> Thus, I really don't think a strongly controlled VFIO mapping should be \n> EAL's responsibility. Failure because of lack of mapping is a driver's \n> problem.\n> \n\nWhile EAL doesn't necessarily need to be involved with mapping things \nfor VFIO, i believe it does need to be the authority on what gets \nmapped. The user needs a way to make arbitrary memory available for DMA \n- this is where EAL comes in. VFIO itself can be factored out into a \nseparate subsystem (DMA drivers, anyone? :D ), but given that memory \ncometh and goeth (external memory included), and given that some things \ntend to be a bit complicated [*], EAL needs to know when something is \nsupposed to be mapped or unmapped, and when to notify subscribers that \nthey may have to refresh their DMA maps.\n\n[*] for example, VFIO can only do mappings whenever there are devices \nactually attached to a VFIO container, so we have to maintain all maps \nbetween hotplug events to ensure that memory set up for DMA doesn't \nsilently get unmapped on device detach and subsequent attach.", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id CBA4A56A3;\n\tTue, 18 Sep 2018 17:15:32 +0200 (CEST)", "from mga01.intel.com (mga01.intel.com [192.55.52.88])\n\tby dpdk.org (Postfix) with ESMTP id 5A48A559A\n\tfor <dev@dpdk.org>; Tue, 18 Sep 2018 17:15:31 +0200 (CEST)", "from fmsmga005.fm.intel.com ([10.253.24.32])\n\tby fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t18 Sep 2018 08:15:30 -0700", "from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.55])\n\t([10.237.220.55])\n\tby fmsmga005.fm.intel.com with ESMTP; 18 Sep 2018 08:15:19 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.53,390,1531810800\"; d=\"scan'208\";a=\"263654987\"", "To": "Shreyansh Jain <shreyansh.jain@nxp.com>,\n\tShahaf Shuler <shahafs@mellanox.com>", "Cc": "\"dev@dpdk.org\" <dev@dpdk.org>,\n\t\"laszlo.madarassy@ericsson.com\" <laszlo.madarassy@ericsson.com>,\n\t\"laszlo.vadkerti@ericsson.com\" <laszlo.vadkerti@ericsson.com>,\n\t\"andras.kovacs@ericsson.com\" <andras.kovacs@ericsson.com>,\n\t\"winnie.tian@ericsson.com\" <winnie.tian@ericsson.com>,\n\t\"daniel.andrasi@ericsson.com\" <daniel.andrasi@ericsson.com>,\n\t\"janos.kobor@ericsson.com\" <janos.kobor@ericsson.com>,\n\t\"srinath.mannam@broadcom.com\" <srinath.mannam@broadcom.com>,\n\t\"scott.branden@broadcom.com\" <scott.branden@broadcom.com>,\n\t\"ajit.khaparde@broadcom.com\" <ajit.khaparde@broadcom.com>,\n\t\"keith.wiles@intel.com\" <keith.wiles@intel.com>,\n\t\"bruce.richardson@intel.com\" <bruce.richardson@intel.com>,\n\tThomas Monjalon <thomas@monjalon.net>", "References": "<cover.1536064999.git.anatoly.burakov@intel.com>\n\t<DB7PR05MB4426F086D6F1ED069A8363BAC31A0@DB7PR05MB4426.eurprd05.prod.outlook.com>\n\t<9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com>\n\t<DB7PR05MB442699F68941ED3EA7DB9FD0C31E0@DB7PR05MB4426.eurprd05.prod.outlook.com>\n\t<b6449567-d6ff-12cd-5a33-d18ecaf73a94@intel.com>\n\t<bf854303-5060-5939-aadd-458e467533bc@nxp.com>", "From": "\"Burakov, Anatoly\" <anatoly.burakov@intel.com>", "Message-ID": "<cd2327e2-f48a-8765-1e83-72bb4b643e0f@intel.com>", "Date": "Tue, 18 Sep 2018 16:15:17 +0100", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101\n\tThunderbird/52.9.1", "MIME-Version": "1.0", "In-Reply-To": "<bf854303-5060-5939-aadd-458e467533bc@nxp.com>", "Content-Type": "text/plain; charset=utf-8; format=flowed", "Content-Language": "en-US", "Content-Transfer-Encoding": "7bit", "Subject": "Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in\n\tDPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null } ]