Cover Detail
Show a cover letter.
GET /api/covers/172/?format=api
{ "id": 172, "url": "http://patches.dpdk.org/api/covers/172/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/cover/20230922081912.7090-1-bruce.richardson@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20230922081912.7090-1-bruce.richardson@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20230922081912.7090-1-bruce.richardson@intel.com", "date": "2023-09-22T08:19:07", "name": "[RFC,0/5] Using shared mempools for zero-copy IO proxying", "submitter": { "id": 20, "url": "http://patches.dpdk.org/api/people/20/?format=api", "name": "Bruce Richardson", "email": "bruce.richardson@intel.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/cover/20230922081912.7090-1-bruce.richardson@intel.com/mbox/", "series": [ { "id": 29602, "url": "http://patches.dpdk.org/api/series/29602/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=29602", "date": "2023-09-22T08:19:07", "name": "Using shared mempools for zero-copy IO proxying", "version": 1, "mbox": "http://patches.dpdk.org/series/29602/mbox/" } ], "comments": "http://patches.dpdk.org/api/covers/172/comments/", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 6E669425CD;\n\tFri, 22 Sep 2023 10:19:43 +0200 (CEST)", "from mails.dpdk.org (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id DE56F402E2;\n\tFri, 22 Sep 2023 10:19:42 +0200 (CEST)", "from mgamail.intel.com (mgamail.intel.com [192.55.52.93])\n by mails.dpdk.org (Postfix) with ESMTP id 6C5214013F\n for <dev@dpdk.org>; Fri, 22 Sep 2023 10:19:41 +0200 (CEST)", "from fmsmga005.fm.intel.com ([10.253.24.32])\n by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 22 Sep 2023 01:19:21 -0700", "from silpixa00401385.ir.intel.com ([10.237.214.14])\n by fmsmga005.fm.intel.com with ESMTP; 22 Sep 2023 01:19:20 -0700" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple;\n d=intel.com; i=@intel.com; q=dns/txt; s=Intel;\n t=1695370781; x=1726906781;\n h=from:to:cc:subject:date:message-id:mime-version:\n content-transfer-encoding;\n bh=L6dmhe+FsTvT2gIp2SyMWKOyxF0hRN8qD3bdJ8wnvk8=;\n b=m5hSorR9SNqvJRiT/TS9XqQKMY809T1W7tKmwYJAYD16lm89iyjIx1z+\n 75EoFhitgi7dyks0pJf4Tw+IWof5ygVZeLi2bMCmW0TuNBaLpsYXIVXI/\n bAGVCulKH4gWxZwCY0cULXBn+Sc5dGmJG6qCQWr8Y/e/gLFc88Z+AY9dm\n t2QRjm8t3BoKAE0xWSuXbI/xTamZG4Wn7N2h1OtE3GP9a691DQUiCm7lw\n oiykIwYDNVO3V5EmSZyIAeGu72ugT7C7lZeJZDJjkXEDndkmUaSYZdbQR\n uofePTTf1uS/yB/Q1p/+WR6ysh3+WsI4cvsOFu6wrOd2jwkozPyvMEejR g==;", "X-IronPort-AV": [ "E=McAfee;i=\"6600,9927,10840\"; a=\"378063954\"", "E=Sophos;i=\"6.03,167,1694761200\"; d=\"scan'208\";a=\"378063954\"", "E=McAfee;i=\"6600,9927,10840\"; a=\"1078281108\"", "E=Sophos;i=\"6.03,167,1694761200\"; d=\"scan'208\";a=\"1078281108\"" ], "X-ExtLoop1": "1", "From": "Bruce Richardson <bruce.richardson@intel.com>", "To": "dev@dpdk.org", "Cc": "Bruce Richardson <bruce.richardson@intel.com>", "Subject": "[RFC PATCH 0/5] Using shared mempools for zero-copy IO proxying", "Date": "Fri, 22 Sep 2023 09:19:07 +0100", "Message-Id": "<20230922081912.7090-1-bruce.richardson@intel.com>", "X-Mailer": "git-send-email 2.39.2", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "Following my talk at the recent DPDK Summit [1], here is an RFC patchset\ncontaining the prototypes I created which led to the talk. This\npatchset is simply to demonstrate:\n\n* what is currently possible with DPDK in terms of zero-copy IPC\n* where the big gaps, and general problem areas are\n* what the performance is like doing zero-copy between processes\n* how we may look to have new deployment models for DPDK apps.\n\nThis cover letter is quite long, as it covers how to run the demo app\nand use the drivers included in this set. I felt it more accessible this\nway than putting it in rst files in the patches. This patchset depends\nupon patchsets [2] and [3]\n\n[1] https://dpdksummit2023.sched.com/event/1P9wU\n[2] http://patches.dpdk.org/project/dpdk/list/?series=29536\n[3] http://patches.dpdk.org/project/dpdk/list/?series=29538\n\nOverview\n--------\n\nThe patchset contains at a high level the following parts: a proxy\napplication which performs packet IO and steers traffic on a per-queue\nbasis to other applications which connect to it via unix sockets, and a\nset of drivers to be used by those applications so that they can\n(hopefully) receive packets from the proxy app without any changes to\ntheir own code. This all helps to demonstrate the feasibility of zero-\ncopy packet transfer between independent DPDK apps.\n\nThe drivers are:\n* a bus driver, which makes the connection to the proxy app via\n the unix socket. Thereafter it accepts the shared memory from the\n proxy and maps it into the running process for use for buffers and\n rings etc. It also handled communication with the proxy app on behalf\n of the other two drivers\n* a mempool driver, which simply manages a set of buffers on the basis\n of offsets within the shared memory area rather than using pointers.\n The big downside of its use is that it assumes all the objects stored\n in the mempool are mbufs. (As described in my talk, this is a big\n issue where I'm not sure we have a good solution available right now\n to resolve it)\n* an ethernet driver, which creates an rx and tx ring in shared memory\n for use in communicating with the proxy app. All buffers sent/received\n are converted to offsets within the shared memory area.\n\nThe proxy app itself implements all the other logic - mostly inside\ndatapath.c - to allow the connecting app to run. When an app connects to\nthe unix socket, the proxy app uses memfd to create a hugepage block to\nbe passed through to the \"guest\" app, and then sends/receives the\nmessages from the drivers until the app connection is up and running to\nhandle traffic. [Ideally, this IPC over unix socket mechanism should\nprobably be generalized into a library used by the app, but for now it's\njust built-in]. As stated above, the steering of traffic is done\nper-queue, that is, each app connects to a specific socket corresponding\nto a NIC queue. For demo purposes, the traffic to the queues is just\ndistributed using RSS, but obviously it would be possible to use e.g.\nrte_flow to do more interesting distribution in future.\n\nRunning the Apps\n----------------\n\nTo get things all working just do a DPDK build as normal. Then run the\nio-proxy app. It only takes a single parameter of the core number to\nuse. For example, on my system I run it on lcore 25:\n\n\t./build/app/dpdk-io-proxy 25\n\nThe sockets to be created and how they map to ports/queues is controlled\nvia commandline, but a startup script can be provided, which just needs\nto be in the current directory and name \"dpdk-io-proxy.cmds\". Patch 5 of\nthis set contains an example setup that I use. Therefore it's\nrecommended that you run the proxy app from a directory containing that\nfile. If so, the proxy app will use two ports and create two queues on\neach, mapping them to 4 unix socket files in /tmp. (Each socket is\ncreated in its own directory to simplify use with docker containers as\ndescribed below in next section).\n\nNo traffic is handled by the app until other end-user apps connect to\nit. Testpmd works as that second \"guest\" app without any changes to it.\nTo run multiple testpmd instances, each taking traffic from a unique RX\nqueue and forwarding it back, the following sequence of commands can be\nused [in this case, doing forwarding on cores 26 through 29, and using\nthe 4 unix sockets configured using the startup file referenced above].\n\n\t./build/app/dpdk-testpmd -l 24,26 --no-huge -m1 --no-shconf \\\n\t\t-a sock:/tmp/socket_0_0/sock -- --forward-mode=macswap\n\t./build/app/dpdk-testpmd -l 24,27 --no-huge -m1 --no-shconf \\\n\t\t-a sock:/tmp/socket_0_1/sock -- --forward-mode=macswap\n\t./build/app/dpdk-testpmd -l 24,28 --no-huge -m1 --no-shconf \\\n\t\t-a sock:/tmp/socket_1_0/sock -- --forward-mode=macswap\n\t./build/app/dpdk-testpmd -l 24,29 --no-huge -m1 --no-shconf \\\n\t\t-a sock:/tmp/socket_1_1/sock -- --forward-mode=macswap\n\nNOTE:\n* the \"--no-huge -m1\" is present to guarantee that no regular DPDK\n hugepage memory is being used by the app. It's all coming from the\n proxy app's memfd\n* the \"--no-shconf\" parameter is necessary just to avoid us needing to\n specify a unix file-prefix for each instance\n* the forwarding type to be used is optional, macswap is chosen just to\n have some work done inside testpmd to prove it can touch the packet\n payload, not just the mbuf header.\n\nUsing with docker containers\n----------------------------\n\nThe testpmd instances run above can also be run within a docker\ncontainer. Using a dockerfile like below we can run testpmd in a\ncontainer getting the packets in a zero-copy manner from the io-proxy\nrunning on the host.\n\n # syntax=docker/dockerfile:1-labs\n FROM alpine\n RUN apk add --update alpine-sdk \\\n py3-elftools meson ninja \\\n bsd-compat-headers \\\n linux-headers \\\n numactl-dev \\\n bash\n ADD . dpdk\n WORKDIR dpdk\n RUN rm -rf build\n RUN meson setup -Denable_drivers=*/shared_mem -Ddisable_libs=* \\\n -Denable_apps=test-pmd -Dtests=false build\n RUN ninja -v -C build\n ENTRYPOINT [\"/dpdk/build/app/dpdk-testpmd\"]\n\nTo access the proxy, all the container needs is access to the unix\nsocket on the filesystem. Since in the example startup script each\nsocket is placed in its own directory we can use \"--volume\" parameter to\ngive each instance it's own unique unix socket, and therefore proxied\nNIC RX/TX queue. To run four testpmd instances as above, just in\ncontainers the following commands can be used - assuming the dockerfile\nabove is built to an image called \"testpmd\".\n\n\tdocker run -it --volume=/tmp/socket_0_0:/run testpmd \\\n\t\t-l 24,26 --no-huge -a sock:/run/sock -- \\\n\t\t--no-mlockall --forward-mode=macswap\n\tdocker run -it --volume=/tmp/socket_0_1:/run testpmd \\\n\t\t-l 24,27 --no-huge -a sock:/run/sock -- \\\n\t\t--no-mlockall --forward-mode=macswap\n\tdocker run -it --volume=/tmp/socket_1_0:/run testpmd \\\n\t\t-l 24,28 --no-huge -a sock:/run/sock -- \\\n\t\t--no-mlockall --forward-mode=macswap\n\tdocker run -it --volume=/tmp/socket_1_1:/run testpmd \\\n\t\t-l 24,29 --no-huge -a sock:/run/sock -- \\\n\t\t--no-mlockall --forward-mode=macswap\n\nNOTE: since these docker testpmd instances don't access IO or allocate\nhugepages directly, they should be runable without extra privileges, so\nlong as they can connect to the unix socket.\n\nAdditional info\n---------------\n\n* Stats are available via app commandline\n* By default (#define in code), the proxy app only uses 2 queues per\n port, so you can't configure more than that via cmdline\n* Any ports used by the proxy script must support queue reconfiguration\n at runtime without stopping the port.\n* When a \"guest\" process connected to a socket terminates, all shared\n memory used by that process is detroyed and a new memfd created on\n reconnect.\n* The above setups using testpmd are the only ways in which this app and\n drivers have been tested. I would be hopeful that other apps would\n work too, but there are quite a few limitations (see my DPDK summit\n talk for some more details on those).\n\nCongratulations on reading this far! :-)\nAll comments/feedback on this welcome.\n\nBruce Richardson (5):\n bus: new driver to accept shared memory over unix socket\n mempool: driver for mempools of mbufs on shared memory\n net: new ethdev driver to communicate using shared mem\n app: add IO proxy app using shared memory interfaces\n app/io-proxy: add startup commands\n\n app/io-proxy/command_fns.c | 160 ++++++\n app/io-proxy/commands.list | 6 +\n app/io-proxy/datapath.c | 595 +++++++++++++++++++++\n app/io-proxy/datapath.h | 37 ++\n app/io-proxy/datapath_mp.c | 78 +++\n app/io-proxy/dpdk-io-proxy.cmds | 6 +\n app/io-proxy/main.c | 71 +++\n app/io-proxy/meson.build | 12 +\n app/meson.build | 1 +\n drivers/bus/meson.build | 1 +\n drivers/bus/shared_mem/meson.build | 11 +\n drivers/bus/shared_mem/shared_mem_bus.c | 323 +++++++++++\n drivers/bus/shared_mem/shared_mem_bus.h | 75 +++\n drivers/bus/shared_mem/version.map | 11 +\n drivers/mempool/meson.build | 1 +\n drivers/mempool/shared_mem/meson.build | 10 +\n drivers/mempool/shared_mem/shared_mem_mp.c | 94 ++++\n drivers/net/meson.build | 1 +\n drivers/net/shared_mem/meson.build | 11 +\n drivers/net/shared_mem/shared_mem_eth.c | 295 ++++++++++\n 20 files changed, 1799 insertions(+)\n create mode 100644 app/io-proxy/command_fns.c\n create mode 100644 app/io-proxy/commands.list\n create mode 100644 app/io-proxy/datapath.c\n create mode 100644 app/io-proxy/datapath.h\n create mode 100644 app/io-proxy/datapath_mp.c\n create mode 100644 app/io-proxy/dpdk-io-proxy.cmds\n create mode 100644 app/io-proxy/main.c\n create mode 100644 app/io-proxy/meson.build\n create mode 100644 drivers/bus/shared_mem/meson.build\n create mode 100644 drivers/bus/shared_mem/shared_mem_bus.c\n create mode 100644 drivers/bus/shared_mem/shared_mem_bus.h\n create mode 100644 drivers/bus/shared_mem/version.map\n create mode 100644 drivers/mempool/shared_mem/meson.build\n create mode 100644 drivers/mempool/shared_mem/shared_mem_mp.c\n create mode 100644 drivers/net/shared_mem/meson.build\n create mode 100644 drivers/net/shared_mem/shared_mem_eth.c\n\n--\n2.39.2" }