Cover Comment List
List cover comments
GET /api/covers/52142/comments/?format=api
[ { "id": 94633, "web_url": "http://patches.dpdk.org/comment/94633/", "msgid": "<CAD+H993RyPiikX8VD4iCjL0CBN3Ycsd=KeD9N+Brp5QmFoKf3w@mail.gmail.com>", "list_archive_url": "https://inbox.dpdk.org/dev/CAD+H993RyPiikX8VD4iCjL0CBN3Ycsd=KeD9N+Brp5QmFoKf3w@mail.gmail.com", "date": "2019-04-08T08:44:07", "subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "submitter": { "id": 270, "url": "http://patches.dpdk.org/api/people/270/?format=api", "name": "Alejandro Lucero", "email": "alejandro.lucero@netronome.com" }, "content": "On Wed, Apr 3, 2019 at 8:19 AM Tiwei Bie <tiwei.bie@intel.com> wrote:\n\n> Hi everyone,\n>\n> This is a draft implementation of the mdev (Mediated device [1])\n> bus support in DPDK. Mdev is a way to virtualize devices in Linux\n> kernel. Based on the device-api (mdev_type/device_api), there could\n> be different types of mdev devices (e.g. vfio-pci). In this RFC,\n> one mdev bus is introduced to scan the mdev devices in the system\n> and do the probe based on the device-api.\n>\n> Take the mdev devices whose device-api is \"vfio-pci\" as an example,\n> in this RFC, these devices will be probed by a mdev driver provided\n> by PCI bus, which will plug them to the PCI bus. And they will be\n> probed with the drivers registered on the PCI bus based on VendorID/\n> DeviceID/... then.\n>\n> +----------+\n> | mdev bus |\n> +----+-----+\n> |\n> +----------------+----+------+------+\n> | | | |\n> mdev_vfio_pci ......\n> (device-api: vfio-pci)\n>\n> There are also other ways to add mdev device support in DPDK (e.g.\n> let PCI bus scan /sys/bus/mdev/devices directly). Comments would be\n> appreciated!\n>\n>\nHi Tiwei,\n\nThanks for the patchset. I was close to send a patchset with the same mdev\nsupport, but I'm glad to see your patchset first because I think it is\ninteresting to see another view of how to implemented this.\n\nAfter going through your patch I was a bit confused about how the mdev\ndevice to mdev driver match was done. But then I realized the approach you\nare following is different to my implementation, likely due to having\ndifferent purposes. If I understand the idea behind, you want to have same\nPCI PMD drivers working with devices, PCI devices, created from mediated\ndevices. That is the reason there is just one mdev driver, the one for\nvfio-pci mediated devices type.\n\nMy approach was different and I though having specific PMD mdev support was\nnecessary, with the PMD requiring to register a mdev driver. I can see,\nafter reading your patch, it can be perfectly possible to have the same\nPMDs for \"pure\" PCI devices and PCI devices made from mediated devices, and\nif the PMD requires to do something different due to the mediated devices\nintrinsics, then explicitly supporting that per PMD. I got specific ioctl\ncalls between the PMD and the mediating driver but this can also be done\nwith your approach.\n\nI'm working on having a mediated PF, what is a different purpose than the\nIntel scalable I/O idea, so I will merge this patchset with my code and see\nif it works.\n\nThanks!\n\n\n> [1]\n> https://github.com/torvalds/linux/blob/master/Documentation/vfio-mediated-device.txt\n>\n> Thanks,\n> Tiwei\n>\n> Tiwei Bie (3):\n> eal: add a helper for reading string from sysfs\n> bus/mdev: add mdev bus support\n> bus/pci: add mdev support\n>\n> config/common_base | 5 +\n> config/common_linux | 1 +\n> drivers/bus/Makefile | 1 +\n> drivers/bus/mdev/Makefile | 41 +++\n> drivers/bus/mdev/linux/Makefile | 6 +\n> drivers/bus/mdev/linux/mdev.c | 117 ++++++++\n> drivers/bus/mdev/mdev.c | 310 ++++++++++++++++++++++\n> drivers/bus/mdev/meson.build | 15 ++\n> drivers/bus/mdev/private.h | 90 +++++++\n> drivers/bus/mdev/rte_bus_mdev.h | 141 ++++++++++\n> drivers/bus/mdev/rte_bus_mdev_version.map | 12 +\n> drivers/bus/meson.build | 2 +-\n> drivers/bus/pci/Makefile | 3 +\n> drivers/bus/pci/linux/Makefile | 4 +\n> drivers/bus/pci/linux/pci_vfio.c | 35 ++-\n> drivers/bus/pci/linux/pci_vfio_mdev.c | 305 +++++++++++++++++++++\n> drivers/bus/pci/meson.build | 4 +-\n> drivers/bus/pci/pci_common.c | 17 +-\n> drivers/bus/pci/private.h | 9 +\n> drivers/bus/pci/rte_bus_pci.h | 11 +-\n> lib/librte_eal/common/eal_filesystem.h | 7 +\n> lib/librte_eal/freebsd/eal/eal.c | 22 ++\n> lib/librte_eal/linux/eal/eal.c | 22 ++\n> lib/librte_eal/rte_eal_version.map | 1 +\n> mk/rte.app.mk | 1 +\n> 25 files changed, 1163 insertions(+), 19 deletions(-)\n> create mode 100644 drivers/bus/mdev/Makefile\n> create mode 100644 drivers/bus/mdev/linux/Makefile\n> create mode 100644 drivers/bus/mdev/linux/mdev.c\n> create mode 100644 drivers/bus/mdev/mdev.c\n> create mode 100644 drivers/bus/mdev/meson.build\n> create mode 100644 drivers/bus/mdev/private.h\n> create mode 100644 drivers/bus/mdev/rte_bus_mdev.h\n> create mode 100644 drivers/bus/mdev/rte_bus_mdev_version.map\n> create mode 100644 drivers/bus/pci/linux/pci_vfio_mdev.c\n>\n> --\n> 2.17.1\n>\n>", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id A214B2BC9;\n\tMon, 8 Apr 2019 10:44:20 +0200 (CEST)", "from mail-it1-f193.google.com (mail-it1-f193.google.com\n\t[209.85.166.193]) by dpdk.org (Postfix) with ESMTP id 4DEED2B95\n\tfor <dev@dpdk.org>; Mon, 8 Apr 2019 10:44:19 +0200 (CEST)", "by mail-it1-f193.google.com with SMTP id y10so19649184itc.1\n\tfor <dev@dpdk.org>; Mon, 08 Apr 2019 01:44:19 -0700 (PDT)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=netronome-com.20150623.gappssmtp.com; s=20150623;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc; bh=vOI87Jx+5LbcZLF3YAnOD2A6btnvYQsw1SWtLFKDKCA=;\n\tb=zbkiZiyZqOO3xDShAUciJumw1GfCS2oIf516x1b/NcIBPE65SgliClJqLJb38j3bO5\n\tgxbxsG5a7W1eA8k4IWoY0uMF7jzIVByzP68tq5pq9Sq8RZFEeEPslWy2jLhbAtpGqnhI\n\t0KXfo/QcttMu+/AHmnb/jlhvaMiSFDGt5ZAGklP28mDdcOql5+WZsL/8BzkbblwDY7n7\n\tAauIAyy31ssSP7f/CLvU9TOwlNQ0UPCc6U9xfkAma/DBBWOV736OD+WPCy/SqvUMJurb\n\tfvv6GiKTZv6OUgnJP60Y04EmPQzFxkUiJFK2PtV6jkYGbfaFpYfw3ZOkn7bwV2lFAMDZ\n\tiMcA==", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc;\n\tbh=vOI87Jx+5LbcZLF3YAnOD2A6btnvYQsw1SWtLFKDKCA=;\n\tb=kiq87KrbRPj0nM+QYAa365y2mQSFJYZPGCdKLkmIrlOb1kG3VtSeLHmpHzjRETzloM\n\tJqkB8Q3kP9XK1yrtJbYbxZyeExNXxQVA7dHifiPFY5nnDMS9eep0lzba19x10kVG2YIS\n\tSnjw+h3PNrZR6JABsZthMxbKjlza5S+5Nfo1dr63c/vyzy0xcyT7br6tyagaH4H35Uwl\n\tvlMRRW5zfiKJ0WpiARWe72eiDcH5oNOrnrdWfArjKliM0lmNG63CBiXQ89pyJ0Bp4VMk\n\tLTTejenmAUIuPiTmezf94mqLjzJaKGItfLadmo4i9jyOk+iE9p6Vptb1DihliHEAn7ts\n\ttnoQ==", "X-Gm-Message-State": "APjAAAWhYFnhvWbYv8YA0qaGHdq3E5n9SPfHx71DfDSZOYxPf5yf3Ksj\n\tLLslcUq92pGbId2ArDMJYSPkjmSPdCiPSF7SMINeTw==", "X-Google-Smtp-Source": "APXvYqz+0t3YXVnDQY1KtS4hYgQSvwpbLRIEXae9UE+c8gkgU12CE2QA+pLwOQI4TuuLFyZVWiP4ctOvZtaWlDV+pkQ=", "X-Received": "by 2002:a24:a00b:: with SMTP id\n\to11mr21393876ite.120.1554713058674; \n\tMon, 08 Apr 2019 01:44:18 -0700 (PDT)", "MIME-Version": "1.0", "References": "<20190403071844.21126-1-tiwei.bie@intel.com>", "In-Reply-To": "<20190403071844.21126-1-tiwei.bie@intel.com>", "From": "Alejandro Lucero <alejandro.lucero@netronome.com>", "Date": "Mon, 8 Apr 2019 09:44:07 +0100", "Message-ID": "<CAD+H993RyPiikX8VD4iCjL0CBN3Ycsd=KeD9N+Brp5QmFoKf3w@mail.gmail.com>", "To": "Tiwei Bie <tiwei.bie@intel.com>", "Cc": "dev <dev@dpdk.org>, \"Liang, Cunming\" <cunming.liang@intel.com>, \n\tBruce Richardson <bruce.richardson@intel.com>", "Content-Type": "text/plain; charset=\"UTF-8\"", "X-Content-Filtered-By": "Mailman/MimeDel 2.1.15", "Subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 94637, "web_url": "http://patches.dpdk.org/comment/94637/", "msgid": "<20190408093601.GA12313@dpdk-tbie.sh.intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20190408093601.GA12313@dpdk-tbie.sh.intel.com", "date": "2019-04-08T09:36:01", "subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "submitter": { "id": 617, "url": "http://patches.dpdk.org/api/people/617/?format=api", "name": "Tiwei Bie", "email": "tiwei.bie@intel.com" }, "content": "On Mon, Apr 08, 2019 at 09:44:07AM +0100, Alejandro Lucero wrote:\n> On Wed, Apr 3, 2019 at 8:19 AM Tiwei Bie <tiwei.bie@intel.com> wrote:\n> > Hi everyone,\n> >\n> > This is a draft implementation of the mdev (Mediated device [1])\n> > bus support in DPDK. Mdev is a way to virtualize devices in Linux\n> > kernel. Based on the device-api (mdev_type/device_api), there could\n> > be different types of mdev devices (e.g. vfio-pci). In this RFC,\n> > one mdev bus is introduced to scan the mdev devices in the system\n> > and do the probe based on the device-api.\n> >\n> > Take the mdev devices whose device-api is \"vfio-pci\" as an example,\n> > in this RFC, these devices will be probed by a mdev driver provided\n> > by PCI bus, which will plug them to the PCI bus. And they will be\n> > probed with the drivers registered on the PCI bus based on VendorID/\n> > DeviceID/... then.\n> >\n> > +----------+\n> > | mdev bus |\n> > +----+-----+\n> > |\n> > +----------------+----+------+------+\n> > | | | |\n> > mdev_vfio_pci ......\n> > (device-api: vfio-pci)\n> >\n> > There are also other ways to add mdev device support in DPDK (e.g.\n> > let PCI bus scan /sys/bus/mdev/devices directly). Comments would be\n> > appreciated!\n> \n> Hi Tiwei,\n> \n> Thanks for the patchset. I was close to send a patchset with the same mdev\n> support, but I'm glad to see your patchset first because I think it is\n> interesting to see another view of how to implemented this.\n> \n> After going through your patch I was a bit confused about how the mdev device\n> to mdev driver match was done. But then I realized the approach you are\n> following is different to my implementation, likely due to having different\n> purposes. If I understand the idea behind, you want to have same PCI PMD\n> drivers working with devices, PCI devices, created from mediated devices.\n\nExactly!\n\n> That\n> is the reason there is just one mdev driver, the one for vfio-pci mediated\n> devices type.\n> \n> My approach was different and I though having specific PMD mdev support was\n> necessary, with the PMD requiring to register a mdev driver. I can see, after\n> reading your patch, it can be perfectly possible to have the same PMDs for\n> \"pure\" PCI devices and PCI devices made from mediated devices, and if the PMD\n> requires to do something different due to the mediated devices intrinsics, then\n> explicitly supporting that per PMD. I got specific ioctl calls between the PMD\n> and the mediating driver but this can also be done with your approach.\n> \n> I'm working on having a mediated PF, what is a different purpose than the Intel\n> scalable I/O idea, so I will merge this patchset with my code and see if it\n> works. \n\nCool! Thanks!\n\n> \n> Thanks!\n> \n> \n> > [1] https://github.com/torvalds/linux/blob/master/Documentation/vfio-mediated-device.txt\n> > \n> > Thanks,\n> > Tiwei\n> > \n> > Tiwei Bie (3):\n> > eal: add a helper for reading string from sysfs\n> > bus/mdev: add mdev bus support\n> > bus/pci: add mdev support\n> > \n> > config/common_base | 5 +\n> > config/common_linux | 1 +\n> > drivers/bus/Makefile | 1 +\n> > drivers/bus/mdev/Makefile | 41 +++\n> > drivers/bus/mdev/linux/Makefile | 6 +\n> > drivers/bus/mdev/linux/mdev.c | 117 ++++++++\n> > drivers/bus/mdev/mdev.c | 310 ++++++++++++++++++++++\n> > drivers/bus/mdev/meson.build | 15 ++\n> > drivers/bus/mdev/private.h | 90 +++++++\n> > drivers/bus/mdev/rte_bus_mdev.h | 141 ++++++++++\n> > drivers/bus/mdev/rte_bus_mdev_version.map | 12 +\n> > drivers/bus/meson.build | 2 +-\n> > drivers/bus/pci/Makefile | 3 +\n> > drivers/bus/pci/linux/Makefile | 4 +\n> > drivers/bus/pci/linux/pci_vfio.c | 35 ++-\n> > drivers/bus/pci/linux/pci_vfio_mdev.c | 305 +++++++++++++++++++++\n> > drivers/bus/pci/meson.build | 4 +-\n> > drivers/bus/pci/pci_common.c | 17 +-\n> > drivers/bus/pci/private.h | 9 +\n> > drivers/bus/pci/rte_bus_pci.h | 11 +-\n> > lib/librte_eal/common/eal_filesystem.h | 7 +\n> > lib/librte_eal/freebsd/eal/eal.c | 22 ++\n> > lib/librte_eal/linux/eal/eal.c | 22 ++\n> > lib/librte_eal/rte_eal_version.map | 1 +\n> > mk/rte.app.mk | 1 +\n> > 25 files changed, 1163 insertions(+), 19 deletions(-)\n> > create mode 100644 drivers/bus/mdev/Makefile\n> > create mode 100644 drivers/bus/mdev/linux/Makefile\n> > create mode 100644 drivers/bus/mdev/linux/mdev.c\n> > create mode 100644 drivers/bus/mdev/mdev.c\n> > create mode 100644 drivers/bus/mdev/meson.build\n> > create mode 100644 drivers/bus/mdev/private.h\n> > create mode 100644 drivers/bus/mdev/rte_bus_mdev.h\n> > create mode 100644 drivers/bus/mdev/rte_bus_mdev_version.map\n> > create mode 100644 drivers/bus/pci/linux/pci_vfio_mdev.c\n> > \n> > --\n> > 2.17.1\n> \n>", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 215BD2C24;\n\tMon, 8 Apr 2019 11:36:32 +0200 (CEST)", "from mga07.intel.com (mga07.intel.com [134.134.136.100])\n\tby dpdk.org (Postfix) with ESMTP id C584E2BD8\n\tfor <dev@dpdk.org>; Mon, 8 Apr 2019 11:36:30 +0200 (CEST)", "from orsmga003.jf.intel.com ([10.7.209.27])\n\tby orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t08 Apr 2019 02:36:29 -0700", "from dpdk-tbie.sh.intel.com ([10.67.104.173])\n\tby orsmga003.jf.intel.com with ESMTP; 08 Apr 2019 02:36:28 -0700" ], "X-Amp-Result": "UNSCANNABLE", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.60,324,1549958400\"; d=\"scan'208\";a=\"140882559\"", "Date": "Mon, 8 Apr 2019 17:36:01 +0800", "From": "Tiwei Bie <tiwei.bie@intel.com>", "To": "Alejandro Lucero <alejandro.lucero@netronome.com>", "Cc": "dev <dev@dpdk.org>, \"Liang, Cunming\" <cunming.liang@intel.com>,\n\tBruce Richardson <bruce.richardson@intel.com>", "Message-ID": "<20190408093601.GA12313@dpdk-tbie.sh.intel.com>", "References": "<20190403071844.21126-1-tiwei.bie@intel.com>\n\t<CAD+H993RyPiikX8VD4iCjL0CBN3Ycsd=KeD9N+Brp5QmFoKf3w@mail.gmail.com>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=utf-8", "Content-Disposition": "inline", "In-Reply-To": "<CAD+H993RyPiikX8VD4iCjL0CBN3Ycsd=KeD9N+Brp5QmFoKf3w@mail.gmail.com>", "User-Agent": "Mutt/1.9.4 (2018-02-28)", "Subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 94773, "web_url": "http://patches.dpdk.org/comment/94773/", "msgid": "<CAHFG_=W5eX2UiiGfZzeL+12iGG2juSAJ9wyLneAhy4tDBhqMcw@mail.gmail.com>", "list_archive_url": "https://inbox.dpdk.org/dev/CAHFG_=W5eX2UiiGfZzeL+12iGG2juSAJ9wyLneAhy4tDBhqMcw@mail.gmail.com", "date": "2019-04-10T10:02:28", "subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "submitter": { "id": 587, "url": "http://patches.dpdk.org/api/people/587/?format=api", "name": "Francois Ozog", "email": "francois.ozog@linaro.org" }, "content": "Hi all,\n\nI presented an approach in Fosdem\n(https://archive.fosdem.org/2018/schedule/event/netmdev/) and feel\nhappy someone is picking up.\n\nIf we step back a little, the mdev concept is to allow userland to be\ngiven a direct control over the hardware data path on a device still\ncontrolled by the kernel.\nFrom a code base perspective, this can shrink down PMD code size b y a\nsignificant size: only 10% of the PMD code is actual data path, the\nrest being device control!\nThe concept is perfect for DPDK, SPDK and many other scenarios (AI\naccelerators).\nShould the work be triggered by DPDK community, it should be\napplicable to a broader set of communities: SPDK, VPP, ODP, AF_XDP....\n\nWe bumped into many sharing (between kernel and userland) complexities\nparticularly when a single PCI device controls two ports.\nSo let's assume we try to solve a subset of the cases: coherent IO\nmemory and a dedicated PCI space (by whatever mechanism) per port.\n\nWhat are the \"things to solve\"?\n\n1) enumeration: enumerating and \"capturing\" an mdev device (the patch I assume)\n2) bifurcation: designating the queues to capture in userland (may be\nall) with a hardware driven rule (flow director or more generic)\n3) memory management: dealing with rings and buffer management on rx\nand tx paths\n\nThe bifurcation can be as simple as : all queues in userland, or quite\nrich: TCP port 80 goes to userland while the rest (ICMP...) go to\nkernel. If the kernel gets some of the traffic there will be a routing\ninformation sharing problem to solve. We had a few experiments here.\nConclusion is its doable but many corner cases make it a big work. And\nit would be nice if the queue selection can be made very generic (and\nnot tied to flow director).\nLet's state this is for further study for now.\n\nLets focus on memory management of VFIO exposed devices.\nI haven't refreshed my knowledge of the VFIO framework so you may want\nto correct a few points...\nFirst of all, DPDK is made to switch packets and particularly between ports.\nWith VFIO, this means all devices are in the same virtual IOVA which\nis tricky to implement in the kernel.\nThere are a few strategies to do that all requiring significant mdev\nextensions and more probably a kernel infrastructure change. The good\nnews is it can be made in such a way that selected drivers implement\nthe change, not requiring all the drivers to be touched.\nAnother big question is: is the kernel allocating the memory then the\nuserland gets a map to it, or does the userland allocates the memory\nand the kernel just maintains the IOVA mapping.\nI would favor kernel allocation and userland gets a map to it (in the\nunified IOVA). One reason being that memory allocation strategy can be\nvery different from hardware to hardware:\n- driver allocates packet buffers and populate a single ring of packet per queue\n- driver allocates packet buffers of different sizes and populate\nmultiple rings per queue (for instance rings of 128, 256, 1024, 2048\nbyte arrays per queue)\n- driver allocates an unstructured memory area (say 32MB) and give it\nto hardware (no prepopulation of rings).\nSo the userland framework (DPDK, SPDK, ODP, VPP, AF_XDP,\nproprietary...) can just query for queues and rings to the kernel\ndriver that knows what has to be done for the driver. The userland\nframework just has to create the relevant objects (queues, rings,\npacket buffers) to the provided kernel information.\n\nExposing VFIO devices to DPDK and other frameworks is a major topic,\nand I suggest that at the same time enumeration is done, a broader\ndiscussion on the data path itself happens.\nData path discussion is about memory management (above) and packet\ndescriptors. Exposing hardware dependent structures in the userland is\nnot the most widely accepted wisdom.\nSo I would rather assume hardware natively produce hardware, vendor,\nOS independent descriptors. Candidates can be: DPDK mbuf, VPP vlib_buf\nor virtio 1.1. I would favor a packet descriptor that supports a\ncombination of inline offloads (VxLAN + IPSec + TSO...) : if virtio\n1.1 could be extended with some DPDK mbuf fields that would be perfect\n;-) That looks science fiction but I know that some smartNICs and\nother hardware, the hardware produced packet descriptor format can be\nflexible....\n\nCheers\n\nFF\n\n\n\nOn Mon, 8 Apr 2019 at 11:36, Tiwei Bie <tiwei.bie@intel.com> wrote:\n>\n> On Mon, Apr 08, 2019 at 09:44:07AM +0100, Alejandro Lucero wrote:\n> > On Wed, Apr 3, 2019 at 8:19 AM Tiwei Bie <tiwei.bie@intel.com> wrote:\n> > > Hi everyone,\n> > >\n> > > This is a draft implementation of the mdev (Mediated device [1])\n> > > bus support in DPDK. Mdev is a way to virtualize devices in Linux\n> > > kernel. Based on the device-api (mdev_type/device_api), there could\n> > > be different types of mdev devices (e.g. vfio-pci). In this RFC,\n> > > one mdev bus is introduced to scan the mdev devices in the system\n> > > and do the probe based on the device-api.\n> > >\n> > > Take the mdev devices whose device-api is \"vfio-pci\" as an example,\n> > > in this RFC, these devices will be probed by a mdev driver provided\n> > > by PCI bus, which will plug them to the PCI bus. And they will be\n> > > probed with the drivers registered on the PCI bus based on VendorID/\n> > > DeviceID/... then.\n> > >\n> > > +----------+\n> > > | mdev bus |\n> > > +----+-----+\n> > > |\n> > > +----------------+----+------+------+\n> > > | | | |\n> > > mdev_vfio_pci ......\n> > > (device-api: vfio-pci)\n> > >\n> > > There are also other ways to add mdev device support in DPDK (e.g.\n> > > let PCI bus scan /sys/bus/mdev/devices directly). Comments would be\n> > > appreciated!\n> >\n> > Hi Tiwei,\n> >\n> > Thanks for the patchset. I was close to send a patchset with the same mdev\n> > support, but I'm glad to see your patchset first because I think it is\n> > interesting to see another view of how to implemented this.\n> >\n> > After going through your patch I was a bit confused about how the mdev device\n> > to mdev driver match was done. But then I realized the approach you are\n> > following is different to my implementation, likely due to having different\n> > purposes. If I understand the idea behind, you want to have same PCI PMD\n> > drivers working with devices, PCI devices, created from mediated devices.\n>\n> Exactly!\n>\n> > That\n> > is the reason there is just one mdev driver, the one for vfio-pci mediated\n> > devices type.\n> >\n> > My approach was different and I though having specific PMD mdev support was\n> > necessary, with the PMD requiring to register a mdev driver. I can see, after\n> > reading your patch, it can be perfectly possible to have the same PMDs for\n> > \"pure\" PCI devices and PCI devices made from mediated devices, and if the PMD\n> > requires to do something different due to the mediated devices intrinsics, then\n> > explicitly supporting that per PMD. I got specific ioctl calls between the PMD\n> > and the mediating driver but this can also be done with your approach.\n> >\n> > I'm working on having a mediated PF, what is a different purpose than the Intel\n> > scalable I/O idea, so I will merge this patchset with my code and see if it\n> > works.\n>\n> Cool! Thanks!\n>\n> >\n> > Thanks!\n> >\n> >\n> > > [1] https://github.com/torvalds/linux/blob/master/Documentation/vfio-mediated-device.txt\n> > >\n> > > Thanks,\n> > > Tiwei\n> > >\n> > > Tiwei Bie (3):\n> > > eal: add a helper for reading string from sysfs\n> > > bus/mdev: add mdev bus support\n> > > bus/pci: add mdev support\n> > >\n> > > config/common_base | 5 +\n> > > config/common_linux | 1 +\n> > > drivers/bus/Makefile | 1 +\n> > > drivers/bus/mdev/Makefile | 41 +++\n> > > drivers/bus/mdev/linux/Makefile | 6 +\n> > > drivers/bus/mdev/linux/mdev.c | 117 ++++++++\n> > > drivers/bus/mdev/mdev.c | 310 ++++++++++++++++++++++\n> > > drivers/bus/mdev/meson.build | 15 ++\n> > > drivers/bus/mdev/private.h | 90 +++++++\n> > > drivers/bus/mdev/rte_bus_mdev.h | 141 ++++++++++\n> > > drivers/bus/mdev/rte_bus_mdev_version.map | 12 +\n> > > drivers/bus/meson.build | 2 +-\n> > > drivers/bus/pci/Makefile | 3 +\n> > > drivers/bus/pci/linux/Makefile | 4 +\n> > > drivers/bus/pci/linux/pci_vfio.c | 35 ++-\n> > > drivers/bus/pci/linux/pci_vfio_mdev.c | 305 +++++++++++++++++++++\n> > > drivers/bus/pci/meson.build | 4 +-\n> > > drivers/bus/pci/pci_common.c | 17 +-\n> > > drivers/bus/pci/private.h | 9 +\n> > > drivers/bus/pci/rte_bus_pci.h | 11 +-\n> > > lib/librte_eal/common/eal_filesystem.h | 7 +\n> > > lib/librte_eal/freebsd/eal/eal.c | 22 ++\n> > > lib/librte_eal/linux/eal/eal.c | 22 ++\n> > > lib/librte_eal/rte_eal_version.map | 1 +\n> > > mk/rte.app.mk | 1 +\n> > > 25 files changed, 1163 insertions(+), 19 deletions(-)\n> > > create mode 100644 drivers/bus/mdev/Makefile\n> > > create mode 100644 drivers/bus/mdev/linux/Makefile\n> > > create mode 100644 drivers/bus/mdev/linux/mdev.c\n> > > create mode 100644 drivers/bus/mdev/mdev.c\n> > > create mode 100644 drivers/bus/mdev/meson.build\n> > > create mode 100644 drivers/bus/mdev/private.h\n> > > create mode 100644 drivers/bus/mdev/rte_bus_mdev.h\n> > > create mode 100644 drivers/bus/mdev/rte_bus_mdev_version.map\n> > > create mode 100644 drivers/bus/pci/linux/pci_vfio_mdev.c\n> > >\n> > > --\n> > > 2.17.1\n> >\n> >\n\n\n\n--\nFrançois-Frédéric Ozog | Director Linaro Edge & Fog Computing Group\nT: +33.67221.6485\nfrancois.ozog@linaro.org | Skype: ffozog", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id C06EB5F28;\n\tWed, 10 Apr 2019 12:02:42 +0200 (CEST)", "from mail-io1-f65.google.com (mail-io1-f65.google.com\n\t[209.85.166.65]) by dpdk.org (Postfix) with ESMTP id 614335F1C\n\tfor <dev@dpdk.org>; Wed, 10 Apr 2019 12:02:41 +0200 (CEST)", "by mail-io1-f65.google.com with SMTP id x3so1548698iol.10\n\tfor <dev@dpdk.org>; Wed, 10 Apr 2019 03:02:41 -0700 (PDT)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;\n\th=mime-version:references:in-reply-to:from:date:message-id:subject:to\n\t:cc:content-transfer-encoding;\n\tbh=UF9hN0ZejhyUxaC3M21b5UQ28tcYgLmITA932pIN0Oc=;\n\tb=TQUdNPpZo/wb2y+5k3auWL6onj4xEt0/SvIQfOJ1F7Kv8LGRBAumpPSJesF1lyJ6pp\n\tcqgdk6+gSb/a6YrvP9KxN1aHKHgw5Xz6ZGLlKb6kK1dBRfmeektgpG01aqJ9wqh4RB2o\n\tWYQOi5YXYyUvHl+VTqABdlDy6ovGEXdAa3vkfyUyNVN7IwGdYV9vL0PhmQ8Gohbz5nUl\n\tTlZ0odV9jsIGqfpuANbse5d99hdGN6/YbuUbJrZVUoqFsKcou1LMDNJri6JRTKm+mZVP\n\trHo3hbJYqCRqh8flRciY5U/561tOVgY+/cRIvJjBS5Nwh03DqfLgAINeyn9sdBDX0beP\n\t5gFw==", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:references:in-reply-to:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=UF9hN0ZejhyUxaC3M21b5UQ28tcYgLmITA932pIN0Oc=;\n\tb=raIWT3SVh4CSALmzWVCWp8VNyIHD5wCnxwRfHWfkwvsuAREcr0Wwf8gNMMcIuxQOX1\n\tEUcePeoTr3zS29CxV924xq3hKw/h3Dl6j1dzZxEdi7kz3EamA8AJcF2GEoOpkOl28aXN\n\t5lewnesFccIJcJf/ScLr31/bXtw8ngvRkvGNCUKAfLZ7AtaKjjWGKiLbVTQ2H00jssdZ\n\tNqmQpZb4CFSpbQZlQ9xZBO84RfeUMRTF2H+NDm5kWj63EZ31VGuu9cTv/rMgRb454MQx\n\tnxscngtYJq/DU86JfBfEPAuAmRdxAnGCc0hlBnp4lEXIiSK73B6aD9+nZvBgt/RdTEUI\n\tplSQ==", "X-Gm-Message-State": "APjAAAVz39DT1itLUX4BFUo8uMADraD3fKTunGEKuCOXhY2lUstXHKbW\n\tV//8GOm/Xq/qIVtW4jD7nIieSZykiVdws+TlIqeL6Q==", "X-Google-Smtp-Source": "APXvYqzhOP77TIx0q9CXi3gqoLWasEqN9puXwWmCC3Tpg9Imf4+ThMUZDEPBS+etrGdh/pp2PGAJWpmjtKsmVWTBD8c=", "X-Received": "by 2002:a6b:e206:: with SMTP id z6mr8295841ioc.237.1554890560567;\n\tWed, 10 Apr 2019 03:02:40 -0700 (PDT)", "MIME-Version": "1.0", "References": "<20190403071844.21126-1-tiwei.bie@intel.com>\n\t<CAD+H993RyPiikX8VD4iCjL0CBN3Ycsd=KeD9N+Brp5QmFoKf3w@mail.gmail.com>\n\t<20190408093601.GA12313@dpdk-tbie.sh.intel.com>", "In-Reply-To": "<20190408093601.GA12313@dpdk-tbie.sh.intel.com>", "From": "Francois Ozog <francois.ozog@linaro.org>", "Date": "Wed, 10 Apr 2019 12:02:28 +0200", "Message-ID": "<CAHFG_=W5eX2UiiGfZzeL+12iGG2juSAJ9wyLneAhy4tDBhqMcw@mail.gmail.com>", "To": "Tiwei Bie <tiwei.bie@intel.com>, dev <dev@dpdk.org>", "Cc": "Alejandro Lucero <alejandro.lucero@netronome.com>, \n\t\"Liang, Cunming\" <cunming.liang@intel.com>,\n\tBruce Richardson <bruce.richardson@intel.com>, \n\tIlias Apalodimas <ilias.apalodimas@linaro.org>, brouer@redhat.com", "Content-Type": "text/plain; charset=\"UTF-8\"", "Content-Transfer-Encoding": "quoted-printable", "Subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "addressed": null }, { "id": 162150, "web_url": "http://patches.dpdk.org/comment/162150/", "msgid": "<20230703165455.45ff9d71@hermes.local>", "list_archive_url": "https://inbox.dpdk.org/dev/20230703165455.45ff9d71@hermes.local", "date": "2023-07-03T23:54:55", "subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "submitter": { "id": 27, "url": "http://patches.dpdk.org/api/people/27/?format=api", "name": "Stephen Hemminger", "email": "stephen@networkplumber.org" }, "content": "On Wed, 10 Apr 2019 12:02:28 +0200\nFrancois Ozog <francois.ozog@linaro.org> wrote:\n\n> Hi all,\n> \n> I presented an approach in Fosdem\n> (https://archive.fosdem.org/2018/schedule/event/netmdev/) and feel\n> happy someone is picking up.\n\nAfter reading through this old patch set, it looks like there are a number\nof loose ends. If it is still valid and can be used with current upstream\nLTS kernel, the please rebase and resubmit it please.", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 0B25F42DC8;\n\tTue, 4 Jul 2023 01:55:00 +0200 (CEST)", "from mails.dpdk.org (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id DACAF40F18;\n\tTue, 4 Jul 2023 01:54:59 +0200 (CEST)", "from mail-pf1-f175.google.com (mail-pf1-f175.google.com\n [209.85.210.175])\n by mails.dpdk.org (Postfix) with ESMTP id 6158040E03\n for <dev@dpdk.org>; Tue, 4 Jul 2023 01:54:58 +0200 (CEST)", "by mail-pf1-f175.google.com with SMTP id\n d2e1a72fcca58-6686708c986so4094489b3a.0\n for <dev@dpdk.org>; Mon, 03 Jul 2023 16:54:58 -0700 (PDT)", "from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218])\n by smtp.gmail.com with ESMTPSA id\n 6-20020aa79146000000b00659edece5dasm14405986pfi.49.2023.07.03.16.54.56\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Mon, 03 Jul 2023 16:54:57 -0700 (PDT)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=networkplumber-org.20221208.gappssmtp.com; s=20221208; t=1688428497;\n x=1691020497;\n h=content-transfer-encoding:mime-version:references:in-reply-to\n :message-id:subject:cc:to:from:date:from:to:cc:subject:date\n :message-id:reply-to;\n bh=TsjzK1sAt8njDpTyn3Kij5h6kdiFJ1lSIdwARrXbgSo=;\n b=cXBzqoHxJVKCuEOHRF42sK8kc0Y9EhXgOFwpvQqFIUJcglseOrr0+ID/QjKf8PJ5+G\n DqVco7xhiOazRuZBBQiXzCM3fNaNDkEIObfpBAqBciYFSfrTluVezuq57nAWX9Ue34rx\n n5uVEkXBWjfN7q6WZ/L4Ew7mzlhww8NunTaJ3ovSIlbwhQKPc7/v0IGBj5o0KXejs8uV\n RUFNozEhoDYw6LHp9/WXf17cZgsuZuHLi4U2ZQC1QdJ94/1u2XJYK0w3+3ELLnbbOcz1\n cdnUReLEMdZKV3t3YQmeTfoiniKdZh4liv1reE8IU0cxuoJp8YVXTLBsQXQ7ofBb6RAa\n yIwA==", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20221208; t=1688428497; x=1691020497;\n h=content-transfer-encoding:mime-version:references:in-reply-to\n :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc\n :subject:date:message-id:reply-to;\n bh=TsjzK1sAt8njDpTyn3Kij5h6kdiFJ1lSIdwARrXbgSo=;\n b=g9hxv5WP9fX2WJJlk3i68YB7T6swuuIdby1udmV2Pps7ogltwejjT5xkbpX7gsSPO9\n BGVIhOGWbc+jlYKfgCvBBETk6xmW+MEAVj/WqnQODbBE73hvPpyjUGc42bebIL+h9+y/\n Q5PHuMVUNOQoem8iF8tUrKDXuFA67ZBb+q1GarzBxTI/dF/f30RtKbaK8sjQncZEDVwh\n 58hdM5eyXpZ4bNCeYNX5pY9PMzgc6gGEFrsOcY/PkTlzWc3ET81sJfp58RmE3AXXRhv2\n XZBsLiY3ifHZ9O8tPb7UDrgBdwNq0Ub/JxvFSip915YlHgTP9H8lwx1qOR8AEVxeneiv\n WxoQ==", "X-Gm-Message-State": "ABy/qLa0d+u2lzrJukNCBhKYlYdUWJrvahBPMEwzzDbbN4Bn/yUdQ1SH\n CnVitjIN7/4zHsHrZCr2sZNCGg==", "X-Google-Smtp-Source": "\n APBJJlFCiz7q4uPAyMPbSJZngcC+dFXhJDzbmsaSFFJMsmt63EtPoYvSqdELaxBPpvomZFnNjmbiKg==", "X-Received": "by 2002:a05:6a00:1a8b:b0:666:d78c:33ab with SMTP id\n e11-20020a056a001a8b00b00666d78c33abmr14631266pfv.21.1688428497486;\n Mon, 03 Jul 2023 16:54:57 -0700 (PDT)", "Date": "Mon, 3 Jul 2023 16:54:55 -0700", "From": "Stephen Hemminger <stephen@networkplumber.org>", "To": "Francois Ozog <francois.ozog@linaro.org>", "Cc": "Tiwei Bie <tiwei.bie@intel.com>, dev <dev@dpdk.org>, Alejandro Lucero\n <alejandro.lucero@netronome.com>, \"Liang, Cunming\"\n <cunming.liang@intel.com>, Bruce Richardson <bruce.richardson@intel.com>,\n Ilias Apalodimas <ilias.apalodimas@linaro.org>, brouer@redhat.com", "Subject": "Re: [dpdk-dev] [RFC 0/3] Add mdev (Mediated device) support in DPDK", "Message-ID": "<20230703165455.45ff9d71@hermes.local>", "In-Reply-To": "\n <CAHFG_=W5eX2UiiGfZzeL+12iGG2juSAJ9wyLneAhy4tDBhqMcw@mail.gmail.com>", "References": "<20190403071844.21126-1-tiwei.bie@intel.com>\n <CAD+H993RyPiikX8VD4iCjL0CBN3Ycsd=KeD9N+Brp5QmFoKf3w@mail.gmail.com>\n <20190408093601.GA12313@dpdk-tbie.sh.intel.com>\n <CAHFG_=W5eX2UiiGfZzeL+12iGG2juSAJ9wyLneAhy4tDBhqMcw@mail.gmail.com>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=US-ASCII", "Content-Transfer-Encoding": "7bit", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "addressed": null } ]