From patchwork Tue Jan 19 11:46:09 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Shukla X-Patchwork-Id: 9984 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 90B72919D; Tue, 19 Jan 2016 12:47:02 +0100 (CET) Received: from mail-pa0-f48.google.com (mail-pa0-f48.google.com [209.85.220.48]) by dpdk.org (Postfix) with ESMTP id 7E2E791B4 for ; Tue, 19 Jan 2016 12:47:00 +0100 (CET) Received: by mail-pa0-f48.google.com with SMTP id yy13so357636060pab.3 for ; Tue, 19 Jan 2016 03:47:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mvista-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v418Owq/KLtpV06u/JaTtFvhvgjExJjnnM0fnGOlBv8=; b=mRsH7FQCgHRvujDVv7yPD1cpiALXA2Gbp1ylGBXD8C4g5jfFliALYeXcRDivzj4ztB M8MEH4yuPdIGPA4PTrIx6vZ4h+HIlOwwRnIUX9p76i5kj7+OsfiR+E07kMngBnLVEGb+ DXc6TtS6A3PvFceIOqwea+Two++2srA6EYoSpnGdxHXHuNWo+6s5ts6K3zLiZ9V8er0n ytJ4YOkydypWKiLvIXp8xuZg1jzyNI5fGgFVu+OnQwgeXqPHfssaidiJ0sJu5sDKds3k 8hQ1PTCkjaQAaIq0XbXo9+YYQdVruGATw/8NI/qQQVTDLLGPy9whlAnmiwvEh6T1RXJA tg6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=v418Owq/KLtpV06u/JaTtFvhvgjExJjnnM0fnGOlBv8=; b=RszfI0O3RAjXi5YAfm8fmyXxPZPyE/0KaxEuiHMQnSxwxfWn2VVGe5hXynXLsG96Ds GJ/9wEBSJCNL/KoA/8HJJz6OYD5qZN0tBLbxTC0RE/Iy4f9kqICQ4+SAckD+mnE5kC6M svez65ZhD4xwYKMr/HqNVTnegW129JQnKeYw08wXDFvqYsgp+6QGbVl6ni29zIHsc1rh hDAflkgFer3q9oxjr1kNqsgvXenyVVNlDoG4UT0YplAf5a97pj+wGRnxW06Vjel+VDl5 3B2V4sgrzmLnLHx1RESzEFn9c2W50c/TkSc8yjMr2baw0PM3UGWkm3xgrnHIFVNn16Lz 1L2w== X-Gm-Message-State: AG10YOSpM9xaBVmXdc1fKjYwjMkcA8EDUIb1HVDNAs8jWGFXvpt9uy9SHR8Mrqu033THBSfJ X-Received: by 10.66.121.194 with SMTP id lm2mr33004197pab.27.1453204019930; Tue, 19 Jan 2016 03:46:59 -0800 (PST) Received: from santosh-Latitude-E5530-non-vPro.mvista.com ([111.93.218.67]) by smtp.gmail.com with ESMTPSA id 75sm41014170pfj.20.2016.01.19.03.46.56 (version=TLS1_1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 19 Jan 2016 03:46:59 -0800 (PST) From: Santosh Shukla To: dev@dpdk.org Date: Tue, 19 Jan 2016 17:16:09 +0530 Message-Id: <1453203972-24855-9-git-send-email-sshukla@mvista.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1453203972-24855-1-git-send-email-sshukla@mvista.com> References: <1453203972-24855-1-git-send-email-sshukla@mvista.com> Subject: [dpdk-dev] [PATCH v5 08/11] eal: pci: introduce RTE_KDRV_VFIO_NOIOMMUi driver mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Adding RTE_KDRV_VFIO_NOIOMMU mode in kernel driver. Also including rte_vfio_is_noiommu() helper function. This function will parse /sys/bus/pci/device// and make sure that - vfio noiommu mode set in kernel driver - pci device attached to vfio-noiommu driver only If both condition satisfies then set drv->kdrv = RTE_KDRV_VFIO_NOIOMMU Also did similar changes in virtio_rd/wr, Changes applicable for virtio spec 0.95 only. Signed-off-by: Santosh Shukla --- v4--> v5: - Removed virtio_xx_init_by_vfio and added new driver mode. - Now no need to parse vfio interface in virtio. As pci_eal module will take of vfio-noiommu driver parsing for virtio or any other future device willing to use vfio-noiommu driver. drivers/net/virtio/virtio_pci.c | 12 ++--- lib/librte_eal/common/include/rte_pci.h | 1 + lib/librte_eal/linuxapp/eal/eal_pci.c | 13 ++++-- lib/librte_eal/linuxapp/eal/eal_pci_init.h | 1 + lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 69 ++++++++++++++++++++++++++++ 5 files changed, 87 insertions(+), 9 deletions(-) diff --git a/drivers/net/virtio/virtio_pci.c b/drivers/net/virtio/virtio_pci.c index 0c29f1d..537c552 100644 --- a/drivers/net/virtio/virtio_pci.c +++ b/drivers/net/virtio/virtio_pci.c @@ -60,7 +60,7 @@ virtio_read_reg_1(struct virtio_hw *hw, uint64_t reg_offset) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_inb(dev, reg_offset, &ret); else ret = inb(VIRTIO_PCI_REG_ADDR(hw, reg_offset)); @@ -75,7 +75,7 @@ virtio_read_reg_2(struct virtio_hw *hw, uint64_t reg_offset) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_inw(dev, reg_offset, &ret); else ret = inw(VIRTIO_PCI_REG_ADDR(hw, reg_offset)); @@ -90,7 +90,7 @@ virtio_read_reg_4(struct virtio_hw *hw, uint64_t reg_offset) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_inl(dev, reg_offset, &ret); else ret = inl(VIRTIO_PCI_REG_ADDR(hw, reg_offset)); @@ -104,7 +104,7 @@ virtio_write_reg_1(struct virtio_hw *hw, uint64_t reg_offset, uint8_t value) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_outb_p(dev, reg_offset, value); else outb_p((unsigned char)value, @@ -117,7 +117,7 @@ virtio_write_reg_2(struct virtio_hw *hw, uint64_t reg_offset, uint16_t value) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_outw_p(dev, reg_offset, value); else outw_p((unsigned short)value, @@ -130,7 +130,7 @@ virtio_write_reg_4(struct virtio_hw *hw, uint64_t reg_offset, uint32_t value) struct rte_pci_device *dev; dev = hw->dev; - if (dev->kdrv == RTE_KDRV_VFIO) + if (dev->kdrv == RTE_KDRV_VFIO_NOIOMMU) ioport_outl_p(dev, reg_offset, value); else outl_p((unsigned int)value, diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h index 0c667ff..2dbc658 100644 --- a/lib/librte_eal/common/include/rte_pci.h +++ b/lib/librte_eal/common/include/rte_pci.h @@ -149,6 +149,7 @@ enum rte_kernel_driver { RTE_KDRV_VFIO, RTE_KDRV_UIO_GENERIC, RTE_KDRV_NIC_UIO, + RTE_KDRV_VFIO_NOIOMMU, RTE_KDRV_NONE, }; diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index eb503f0..2936497 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -131,6 +131,7 @@ rte_eal_pci_map_device(struct rte_pci_device *dev) /* try mapping the NIC resources using VFIO if it exists */ switch (dev->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: #ifdef VFIO_PRESENT if (pci_vfio_is_enabled()) ret = pci_vfio_map_resource(dev); @@ -158,6 +159,7 @@ rte_eal_pci_unmap_device(struct rte_pci_device *dev) /* try unmapping the NIC resources using VFIO if it exists */ switch (dev->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: RTE_LOG(ERR, EAL, "Hotplug doesn't support vfio yet\n"); break; case RTE_KDRV_IGB_UIO: @@ -353,9 +355,12 @@ pci_scan_one(const char *dirname, uint16_t domain, uint8_t bus, } if (!ret) { - if (!strcmp(driver, "vfio-pci")) - dev->kdrv = RTE_KDRV_VFIO; - else if (!strcmp(driver, "igb_uio")) + if (!strcmp(driver, "vfio-pci")) { + if (pci_vfio_is_noiommu(dev) == 0) + dev->kdrv = RTE_KDRV_VFIO_NOIOMMU; + else + dev->kdrv = RTE_KDRV_VFIO; + } else if (!strcmp(driver, "igb_uio")) dev->kdrv = RTE_KDRV_IGB_UIO; else if (!strcmp(driver, "uio_pci_generic")) dev->kdrv = RTE_KDRV_UIO_GENERIC; @@ -630,6 +635,7 @@ int rte_eal_pci_read_bar(const struct rte_pci_device *device, switch (device->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx); default: @@ -647,6 +653,7 @@ int rte_eal_pci_write_bar(const struct rte_pci_device *device, switch (device->kdrv) { case RTE_KDRV_VFIO: + case RTE_KDRV_VFIO_NOIOMMU: return pci_vfio_write_bar(intr_handle, buf, len, offset, bar_idx); default: diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/eal_pci_init.h index 3bc592b..60b95d7 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_init.h +++ b/lib/librte_eal/linuxapp/eal/eal_pci_init.h @@ -60,6 +60,7 @@ int pci_uio_write_config(const struct rte_intr_handle *intr_handle, int pci_vfio_enable(void); int pci_vfio_is_enabled(void); +int pci_vfio_is_noiommu(struct rte_pci_device *pci_dev); int pci_vfio_mp_sync_setup(void); /* access config space */ diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c index df407ef..31d688b 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c @@ -973,4 +973,73 @@ pci_vfio_is_enabled(void) { return vfio_cfg.vfio_enabled; } + +int +pci_vfio_is_noiommu(struct rte_pci_device *pci_dev) +{ + FILE *fp; + struct rte_pci_addr *loc; + const char *path = "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"; + char filename[PATH_MAX] = {0}; + char buf[PATH_MAX] = {0}; + + /* + * 1. chk vfio-noiommu mode set in kernel driver + * 2. verify pci device attached to vfio-noiommu driver + * example: + * cd /sys/bus/pci/drivers/vfio-pci//iommu_group + * > cat name + * > vfio-noiommu --> means virtio_dev attached to vfio-noiommu driver + */ + + fp = fopen(path, "r"); + if (fp == NULL) { + RTE_LOG(ERR, EAL, "can't open %s\n", path); + return -1; + } + + if (fread(buf, sizeof(char), 1, fp) != 1) { + RTE_LOG(ERR, EAL, "can't read from file %s\n", path); + fclose(fp); + return -1; + } + + if (strncmp(buf, "Y", 1) != 0) { + RTE_LOG(ERR, EAL, "[%s]: vfio: noiommu mode not set\n", path); + fclose(fp); + return -1; + } + + fclose(fp); + + /* 2. chk whether attached driver is vfio-noiommu or not */ + loc = &pci_dev->addr; + snprintf(filename, sizeof(filename), + SYSFS_PCI_DEVICES "/" PCI_PRI_FMT "/iommu_group/name", + loc->domain, loc->bus, loc->devid, loc->function); + + /* check for vfio-noiommu */ + fp = fopen(filename, "r"); + if (fp == NULL) { + RTE_LOG(ERR, EAL, "can't open %s\n", filename); + return -1; + } + + if (fread(buf, sizeof(char), sizeof("vfio-noiommu"), fp) != + sizeof("vfio-noiommu")) { + RTE_LOG(ERR, EAL, "can't read from file %s\n", filename); + fclose(fp); + return -1; + } + + if (strncmp(buf, "vfio-noiommu", strlen("vfio-noiommu")) != 0) { + RTE_LOG(ERR, EAL, "not a vfio-noiommu driver\n"); + fclose(fp); + return -1; + } + + fclose(fp); + + return 0; +} #endif