Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/54802/?format=api
{ "id": 54802, "url": "http://patches.dpdk.org/api/patches/54802/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/1560505157-9769-4-git-send-email-david.marchand@redhat.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<1560505157-9769-4-git-send-email-david.marchand@redhat.com>", "list_archive_url": "https://inbox.dpdk.org/dev/1560505157-9769-4-git-send-email-david.marchand@redhat.com", "date": "2019-06-14T09:39:17", "name": "[v2,3/3] bus/pci: only consider usable devices to select IOVA mode", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": true, "hash": "d0e7173aec997aec06e71aa73cf76abc43b4b39e", "submitter": { "id": 1173, "url": "http://patches.dpdk.org/api/people/1173/?format=api", "name": "David Marchand", "email": "david.marchand@redhat.com" }, "delegate": { "id": 1, "url": "http://patches.dpdk.org/api/users/1/?format=api", "username": "tmonjalo", "first_name": "Thomas", "last_name": "Monjalon", "email": "thomas@monjalon.net" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/1560505157-9769-4-git-send-email-david.marchand@redhat.com/mbox/", "series": [ { "id": 5011, "url": "http://patches.dpdk.org/api/series/5011/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=5011", "date": "2019-06-14T09:39:14", "name": "Improve automatic selection of IOVA mode", "version": 2, "mbox": "http://patches.dpdk.org/series/5011/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/54802/comments/", "check": "success", "checks": "http://patches.dpdk.org/api/patches/54802/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 81EB11D519;\n\tFri, 14 Jun 2019 11:39:56 +0200 (CEST)", "from mx1.redhat.com (mx1.redhat.com [209.132.183.28])\n\tby dpdk.org (Postfix) with ESMTP id 2D6741D4B6\n\tfor <dev@dpdk.org>; Fri, 14 Jun 2019 11:39:54 +0200 (CEST)", "from smtp.corp.redhat.com\n\t(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 95D1A307D84B;\n\tFri, 14 Jun 2019 09:39:53 +0000 (UTC)", "from dmarchan.remote.csb (unknown [10.40.205.71])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 9422C7C576;\n\tFri, 14 Jun 2019 09:39:51 +0000 (UTC)" ], "From": "David Marchand <david.marchand@redhat.com>", "To": "dev@dpdk.org", "Cc": "benjamin.walker@intel.com, jerinj@marvell.com, anatoly.burakov@intel.com,\n\tmaxime.coquelin@redhat.com, thomas@monjalon.net", "Date": "Fri, 14 Jun 2019 11:39:17 +0200", "Message-Id": "<1560505157-9769-4-git-send-email-david.marchand@redhat.com>", "In-Reply-To": "<1560505157-9769-1-git-send-email-david.marchand@redhat.com>", "References": "<20190530174819.1160221-1-benjamin.walker@intel.com>\n\t<1560505157-9769-1-git-send-email-david.marchand@redhat.com>", "X-Scanned-By": "MIMEDefang 2.79 on 10.5.11.11", "X-Greylist": "Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.48]); Fri, 14 Jun 2019 09:39:53 +0000 (UTC)", "Subject": "[dpdk-dev] [PATCH v2 3/3] bus/pci: only consider usable devices to\n\tselect IOVA mode", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "From: Ben Walker <benjamin.walker@intel.com>\n\nWhen selecting the preferred IOVA mode of the pci bus, the current\nheuristic (\"are devices bound?\", \"are devices bound to UIO?\", \"are pmd\ndrivers supporting IOVA as VA?\" etc..) should honor the device\nwhite/blacklist so that an unwanted device does not impact the decision.\n\nThere is no reason to consider a device which has no driver available.\n\nThis applies to all OS, so implements this in common code then call a\nOS specific callback.\n\nOn Linux side:\n- the VFIO special considerations should be evaluated only if VFIO\n support is built,\n- there is no strong requirement on using VA rather than PA if a driver\n supports VA, so defaulting to DC in such a case.\n\nSigned-off-by: Ben Walker <benjamin.walker@intel.com>\nSigned-off-by: David Marchand <david.marchand@redhat.com>\n---\n drivers/bus/pci/bsd/pci.c | 9 +-\n drivers/bus/pci/linux/pci.c | 191 ++++++++++++-------------------------------\n drivers/bus/pci/pci_common.c | 65 +++++++++++++++\n drivers/bus/pci/private.h | 8 ++\n 4 files changed, 131 insertions(+), 142 deletions(-)", "diff": "diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c\nindex c7b90cb..a2de709 100644\n--- a/drivers/bus/pci/bsd/pci.c\n+++ b/drivers/bus/pci/bsd/pci.c\n@@ -376,13 +376,14 @@\n \treturn -1;\n }\n \n-/*\n- * Get iommu class of PCI devices on the bus.\n- */\n enum rte_iova_mode\n-rte_pci_get_iommu_class(void)\n+pci_device_iova_mode(const struct rte_pci_driver *pdrv __rte_unused,\n+\t\t const struct rte_pci_device *pdev)\n {\n \t/* Supports only RTE_KDRV_NIC_UIO */\n+\tif (pdev->kdrv != RTE_KDRV_NIC_UIO)\n+\t\tRTE_LOG(DEBUG, EAL, \"Unsupported kernel driver? Defaulting to IOVA as 'PA'\\n\");\n+\n \treturn RTE_IOVA_PA;\n }\n \ndiff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c\nindex b931cf9..33c8ea7 100644\n--- a/drivers/bus/pci/linux/pci.c\n+++ b/drivers/bus/pci/linux/pci.c\n@@ -500,95 +500,14 @@\n \treturn -1;\n }\n \n-/*\n- * Is pci device bound to any kdrv\n- */\n-static inline int\n-pci_one_device_is_bound(void)\n-{\n-\tstruct rte_pci_device *dev = NULL;\n-\tint ret = 0;\n-\n-\tFOREACH_DEVICE_ON_PCIBUS(dev) {\n-\t\tif (dev->kdrv == RTE_KDRV_UNKNOWN ||\n-\t\t dev->kdrv == RTE_KDRV_NONE) {\n-\t\t\tcontinue;\n-\t\t} else {\n-\t\t\tret = 1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\treturn ret;\n-}\n-\n-/*\n- * Any one of the device bound to uio\n- */\n-static inline int\n-pci_one_device_bound_uio(void)\n-{\n-\tstruct rte_pci_device *dev = NULL;\n-\tstruct rte_devargs *devargs;\n-\tint need_check;\n-\n-\tFOREACH_DEVICE_ON_PCIBUS(dev) {\n-\t\tdevargs = dev->device.devargs;\n-\n-\t\tneed_check = 0;\n-\t\tswitch (rte_pci_bus.bus.conf.scan_mode) {\n-\t\tcase RTE_BUS_SCAN_WHITELIST:\n-\t\t\tif (devargs && devargs->policy == RTE_DEV_WHITELISTED)\n-\t\t\t\tneed_check = 1;\n-\t\t\tbreak;\n-\t\tcase RTE_BUS_SCAN_UNDEFINED:\n-\t\tcase RTE_BUS_SCAN_BLACKLIST:\n-\t\t\tif (devargs == NULL ||\n-\t\t\t devargs->policy != RTE_DEV_BLACKLISTED)\n-\t\t\t\tneed_check = 1;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\tif (!need_check)\n-\t\t\tcontinue;\n-\n-\t\tif (dev->kdrv == RTE_KDRV_IGB_UIO ||\n-\t\t dev->kdrv == RTE_KDRV_UIO_GENERIC) {\n-\t\t\treturn 1;\n-\t\t}\n-\t}\n-\treturn 0;\n-}\n-\n-/*\n- * Any one of the device has iova as va\n- */\n-static inline int\n-pci_one_device_has_iova_va(void)\n-{\n-\tstruct rte_pci_device *dev = NULL;\n-\tstruct rte_pci_driver *drv = NULL;\n-\n-\tFOREACH_DRIVER_ON_PCIBUS(drv) {\n-\t\tif (drv && drv->drv_flags & RTE_PCI_DRV_IOVA_AS_VA) {\n-\t\t\tFOREACH_DEVICE_ON_PCIBUS(dev) {\n-\t\t\t\tif ((dev->kdrv == RTE_KDRV_VFIO ||\n-\t\t\t\t dev->kdrv == RTE_KDRV_NIC_MLX) &&\n-\t\t\t\t rte_pci_match(drv, dev))\n-\t\t\t\t\treturn 1;\n-\t\t\t}\n-\t\t}\n-\t}\n-\treturn 0;\n-}\n-\n #if defined(RTE_ARCH_X86)\n static bool\n-pci_one_device_iommu_support_va(struct rte_pci_device *dev)\n+pci_one_device_iommu_support_va(const struct rte_pci_device *dev)\n {\n #define VTD_CAP_MGAW_SHIFT\t16\n #define VTD_CAP_MGAW_MASK\t(0x3fULL << VTD_CAP_MGAW_SHIFT)\n #define X86_VA_WIDTH 47 /* From Documentation/x86/x86_64/mm.txt */\n-\tstruct rte_pci_addr *addr = &dev->addr;\n+\tconst struct rte_pci_addr *addr = &dev->addr;\n \tchar filename[PATH_MAX];\n \tFILE *fp;\n \tuint64_t mgaw, vtd_cap_reg = 0;\n@@ -632,80 +551,76 @@\n }\n #elif defined(RTE_ARCH_PPC_64)\n static bool\n-pci_one_device_iommu_support_va(__rte_unused struct rte_pci_device *dev)\n+pci_one_device_iommu_support_va(__rte_unused const struct rte_pci_device *dev)\n {\n \treturn false;\n }\n #else\n static bool\n-pci_one_device_iommu_support_va(__rte_unused struct rte_pci_device *dev)\n+pci_one_device_iommu_support_va(__rte_unused const struct rte_pci_device *dev)\n {\n \treturn true;\n }\n #endif\n \n-/*\n- * All devices IOMMUs support VA as IOVA\n- */\n-static bool\n-pci_devices_iommu_support_va(void)\n+enum rte_iova_mode\n+pci_device_iova_mode(const struct rte_pci_driver *pdrv,\n+\t\t const struct rte_pci_device *pdev)\n {\n-\tstruct rte_pci_device *dev = NULL;\n-\tstruct rte_pci_driver *drv = NULL;\n+\tenum rte_iova_mode iova_mode = RTE_IOVA_DC;\n+\tstatic int iommu_no_va = -1;\n \n-\tFOREACH_DRIVER_ON_PCIBUS(drv) {\n-\t\tFOREACH_DEVICE_ON_PCIBUS(dev) {\n-\t\t\tif (!rte_pci_match(drv, dev))\n-\t\t\t\tcontinue;\n-\t\t\t/*\n-\t\t\t * just one PCI device needs to be checked out because\n-\t\t\t * the IOMMU hardware is the same for all of them.\n-\t\t\t */\n-\t\t\treturn pci_one_device_iommu_support_va(dev);\n+\tswitch (pdev->kdrv) {\n+\tcase RTE_KDRV_VFIO: {\n+#ifdef VFIO_PRESENT\n+\t\tstatic int is_vfio_noiommu_enabled = -1;\n+\n+\t\tif (is_vfio_noiommu_enabled == -1) {\n+\t\t\tif (rte_vfio_noiommu_is_enabled() == 1)\n+\t\t\t\tis_vfio_noiommu_enabled = 1;\n+\t\t\telse\n+\t\t\t\tis_vfio_noiommu_enabled = 0;\n+\t\t}\n+\t\tif ((pdrv->drv_flags & RTE_PCI_DRV_IOVA_AS_VA) == 0) {\n+\t\t\tiova_mode = RTE_IOVA_PA;\n+\t\t} else if (is_vfio_noiommu_enabled != 0) {\n+\t\t\tRTE_LOG(DEBUG, EAL, \"Forcing to 'PA', vfio-noiommu mode configured\\n\");\n+\t\t\tiova_mode = RTE_IOVA_PA;\n \t\t}\n+#endif\n+\t\tbreak;\n \t}\n-\treturn true;\n-}\n \n-/*\n- * Get iommu class of PCI devices on the bus.\n- */\n-enum rte_iova_mode\n-rte_pci_get_iommu_class(void)\n-{\n-\tbool is_bound;\n-\tbool is_vfio_noiommu_enabled = true;\n-\tbool has_iova_va;\n-\tbool is_bound_uio;\n-\tbool iommu_no_va;\n-\n-\tis_bound = pci_one_device_is_bound();\n-\tif (!is_bound)\n-\t\treturn RTE_IOVA_DC;\n-\n-\thas_iova_va = pci_one_device_has_iova_va();\n-\tis_bound_uio = pci_one_device_bound_uio();\n-\tiommu_no_va = !pci_devices_iommu_support_va();\n-#ifdef VFIO_PRESENT\n-\tis_vfio_noiommu_enabled = rte_vfio_noiommu_is_enabled() == true ?\n-\t\t\t\t\ttrue : false;\n-#endif\n+\tcase RTE_KDRV_NIC_MLX:\n+\t\tif ((pdrv->drv_flags & RTE_PCI_DRV_IOVA_AS_VA) == 0)\n+\t\t\tiova_mode = RTE_IOVA_PA;\n+\t\tbreak;\n \n-\tif (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled &&\n-\t\t\t!iommu_no_va)\n-\t\treturn RTE_IOVA_VA;\n+\tcase RTE_KDRV_IGB_UIO:\n+\tcase RTE_KDRV_UIO_GENERIC:\n+\t\tiova_mode = RTE_IOVA_PA;\n+\t\tbreak;\n \n-\tif (has_iova_va) {\n-\t\tRTE_LOG(WARNING, EAL, \"Some devices want iova as va but pa will be used because.. \");\n-\t\tif (is_vfio_noiommu_enabled)\n-\t\t\tRTE_LOG(WARNING, EAL, \"vfio-noiommu mode configured\\n\");\n-\t\tif (is_bound_uio)\n-\t\t\tRTE_LOG(WARNING, EAL, \"few device bound to UIO\\n\");\n-\t\tif (iommu_no_va)\n-\t\t\tRTE_LOG(WARNING, EAL, \"IOMMU does not support IOVA as VA\\n\");\n+\tdefault:\n+\t\tRTE_LOG(DEBUG, EAL, \"Unsupported kernel driver? Defaulting to IOVA as 'PA'\\n\");\n+\t\tiova_mode = RTE_IOVA_PA;\n+\t\tbreak;\n \t}\n \n-\treturn RTE_IOVA_PA;\n+\tif (iova_mode != RTE_IOVA_PA) {\n+\t\t/*\n+\t\t * We can check this only once, because the IOMMU hardware is\n+\t\t * the same for all of them.\n+\t\t */\n+\t\tif (iommu_no_va == -1)\n+\t\t\tiommu_no_va = pci_one_device_iommu_support_va(pdev)\n+\t\t\t\t\t? 0 : 1;\n+\t\tif (iommu_no_va != 0) {\n+\t\t\tRTE_LOG(DEBUG, EAL, \"Forcing to 'PA', IOMMU does not support IOVA as 'VA'\\n\");\n+\t\t\tiova_mode = RTE_IOVA_PA;\n+\t\t}\n+\t}\n+\treturn iova_mode;\n }\n \n /* Read PCI config space. */\ndiff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c\nindex 704b9d7..d2af472 100644\n--- a/drivers/bus/pci/pci_common.c\n+++ b/drivers/bus/pci/pci_common.c\n@@ -574,6 +574,71 @@ static struct rte_devargs *pci_devargs_lookup(struct rte_pci_device *dev)\n \treturn -1;\n }\n \n+static bool\n+pci_ignore_device(const struct rte_pci_device *dev)\n+{\n+\tstruct rte_devargs *devargs = dev->device.devargs;\n+\n+\tswitch (rte_pci_bus.bus.conf.scan_mode) {\n+\tcase RTE_BUS_SCAN_WHITELIST:\n+\t\tif (devargs && devargs->policy == RTE_DEV_WHITELISTED)\n+\t\t\treturn false;\n+\t\tbreak;\n+\tcase RTE_BUS_SCAN_UNDEFINED:\n+\tcase RTE_BUS_SCAN_BLACKLIST:\n+\t\tif (devargs == NULL ||\n+\t\t devargs->policy != RTE_DEV_BLACKLISTED)\n+\t\t\treturn false;\n+\t\tbreak;\n+\t}\n+\treturn true;\n+}\n+\n+enum rte_iova_mode\n+rte_pci_get_iommu_class(void)\n+{\n+\tenum rte_iova_mode iova_mode = RTE_IOVA_DC;\n+\tconst struct rte_pci_device *dev;\n+\tconst struct rte_pci_driver *drv;\n+\tbool devices_want_va = false;\n+\tbool devices_want_pa = false;\n+\n+\tFOREACH_DEVICE_ON_PCIBUS(dev) {\n+\t\tif (pci_ignore_device(dev))\n+\t\t\tcontinue;\n+\t\tif (dev->kdrv == RTE_KDRV_UNKNOWN ||\n+\t\t dev->kdrv == RTE_KDRV_NONE)\n+\t\t\tcontinue;\n+\t\tFOREACH_DRIVER_ON_PCIBUS(drv) {\n+\t\t\tenum rte_iova_mode dev_iova_mode;\n+\n+\t\t\tif (!rte_pci_match(drv, dev))\n+\t\t\t\tcontinue;\n+\n+\t\t\tdev_iova_mode = pci_device_iova_mode(drv, dev);\n+\t\t\tRTE_LOG(DEBUG, EAL, \"PCI driver %s for device \"\n+\t\t\t\tPCI_PRI_FMT \" wants IOVA as '%s'\\n\",\n+\t\t\t\tdrv->driver.name,\n+\t\t\t\tdev->addr.domain, dev->addr.bus,\n+\t\t\t\tdev->addr.devid, dev->addr.function,\n+\t\t\t\tdev_iova_mode == RTE_IOVA_DC ? \"DC\" :\n+\t\t\t\t(dev_iova_mode == RTE_IOVA_PA ? \"PA\" : \"VA\"));\n+\t\t\tif (dev_iova_mode == RTE_IOVA_PA)\n+\t\t\t\tdevices_want_pa = true;\n+\t\t\telse if (dev_iova_mode == RTE_IOVA_VA)\n+\t\t\t\tdevices_want_va = true;\n+\t\t}\n+\t}\n+\tif (devices_want_pa) {\n+\t\tiova_mode = RTE_IOVA_PA;\n+\t\tif (devices_want_va)\n+\t\t\tRTE_LOG(WARNING, EAL, \"Some devices want 'VA' but forcing 'PA' because other devices want it\\n\");\n+\t} else if (devices_want_va) {\n+\t\tiova_mode = RTE_IOVA_VA;\n+\t}\n+\treturn iova_mode;\n+}\n+\n struct rte_pci_bus rte_pci_bus = {\n \t.bus = {\n \t\t.scan = rte_pci_scan,\ndiff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h\nindex 13c3324..8a55240 100644\n--- a/drivers/bus/pci/private.h\n+++ b/drivers/bus/pci/private.h\n@@ -173,6 +173,14 @@ int pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,\n \t const struct rte_pci_device *pci_dev);\n \n /**\n+ * OS specific callback for rte_pci_get_iommu_class\n+ *\n+ */\n+enum rte_iova_mode\n+pci_device_iova_mode(const struct rte_pci_driver *pci_drv,\n+\t\t const struct rte_pci_device *pci_dev);\n+\n+/**\n * Get iommu class of PCI devices on the bus.\n * And return their preferred iova mapping mode.\n *\n", "prefixes": [ "v2", "3/3" ] }