From patchwork Mon Feb 22 17:15:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?= X-Patchwork-Id: 88086 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E3776A034F; Mon, 22 Feb 2021 18:15:37 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD5E622A268; Mon, 22 Feb 2021 18:15:37 +0100 (CET) Received: from out0-154.mail.aliyun.com (out0-154.mail.aliyun.com [140.205.0.154]) by mails.dpdk.org (Postfix) with ESMTP id 0440B22A268 for ; Mon, 22 Feb 2021 18:15:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alibaba-inc.com; s=default; t=1614014132; h=From:To:Subject:Date:Message-Id; bh=2snZIOLXG7EHN06Csv0LOfwunYLYbMzb3+0YJPH/R+I=; b=DNT8TOrYEjMog4/wpccOghVLSwidW53Bvzd6yvTTQXBAhei1SxON5G+ljVwjkXwqOfYiPSjU3hdg3xjUTs7XZkCah2GGg3fZQAKsnxCuwwIF/HujC0mRJeFw8lzk3/Xkw2Dupb9KPjxMgKM/WUf67TSXFpRaJk74CDyq5R0S6H8= X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R801e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=ay29a033018047193; MF=huawei.xhw@alibaba-inc.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---.JbmAyz9_1614014121; Received: from rs3a10040.et2sqa.z1.et2sqa.tbsite.net(mailfrom:huawei.xhw@alibaba-inc.com fp:SMTPD_---.JbmAyz9_1614014121) by smtp.aliyun-inc.com(127.0.0.1); Tue, 23 Feb 2021 01:15:31 +0800 From: " =?utf-8?b?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?= " To: ferruh.yigit@intel.com, maxime.coquelin@redhat.com, david.marchand@redhat.com Cc: , , , , , " =?utf-8?b?6LCi5Y2O5LyfKOatpA==?= =?utf-8?b?5pe25q2k5Yi777yJ?= " Date: Tue, 23 Feb 2021 01:15:17 +0800 Message-Id: <1614014118-91150-2-git-send-email-huawei.xhw@alibaba-inc.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1614014118-91150-1-git-send-email-huawei.xhw@alibaba-inc.com> References: <1611890309-99135-1-git-send-email-huawei.xhw@alibaba-inc.com> <1614014118-91150-1-git-send-email-huawei.xhw@alibaba-inc.com> Subject: [dpdk-dev] [PATCH v7 1/2] bus/pci: use PCI standard sysfs entry to get PIO address X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: "huawei.xhw" Currently virtio PMD asssumes legacy device uses PIO bar. There are three ways to get PIO(PortIO) address for virtio legacy device. under igb_uio, get pio address from uio/uio# sysfs attribute under uio_pci_generic: for X86, get PIO address from /proc/ioport for other ARCH, get PIO address from standard PCI sysfs attribute Actually, igb_uio sysfs attribute exports exactly the same thing as standard PCI sysfs, i.e, pci_dev->resource[] This patch fixes these messy things, and uses standard PCI sysfs attribute. Signed-off-by: huawei xie Reviewed-by: Maxime Coquelin --- drivers/bus/pci/linux/pci.c | 77 ----------------------------------------- drivers/bus/pci/linux/pci_uio.c | 64 ++++++++++++++++++++++++---------- 2 files changed, 46 insertions(+), 95 deletions(-) diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c index 2e1808b..0f38abf 100644 --- a/drivers/bus/pci/linux/pci.c +++ b/drivers/bus/pci/linux/pci.c @@ -677,71 +677,6 @@ int rte_pci_write_config(const struct rte_pci_device *device, } } -#if defined(RTE_ARCH_X86) -static int -pci_ioport_map(struct rte_pci_device *dev, int bar __rte_unused, - struct rte_pci_ioport *p) -{ - uint16_t start, end; - FILE *fp; - char *line = NULL; - char pci_id[16]; - int found = 0; - size_t linesz; - - if (rte_eal_iopl_init() != 0) { - RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions for PCI device %s\n", - __func__, dev->name); - return -1; - } - - snprintf(pci_id, sizeof(pci_id), PCI_PRI_FMT, - dev->addr.domain, dev->addr.bus, - dev->addr.devid, dev->addr.function); - - fp = fopen("/proc/ioports", "r"); - if (fp == NULL) { - RTE_LOG(ERR, EAL, "%s(): can't open ioports\n", __func__); - return -1; - } - - while (getdelim(&line, &linesz, '\n', fp) > 0) { - char *ptr = line; - char *left; - int n; - - n = strcspn(ptr, ":"); - ptr[n] = 0; - left = &ptr[n + 1]; - - while (*left && isspace(*left)) - left++; - - if (!strncmp(left, pci_id, strlen(pci_id))) { - found = 1; - - while (*ptr && isspace(*ptr)) - ptr++; - - sscanf(ptr, "%04hx-%04hx", &start, &end); - - break; - } - } - - free(line); - fclose(fp); - - if (!found) - return -1; - - p->base = start; - RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%x\n", start); - - return 0; -} -#endif - int rte_pci_ioport_map(struct rte_pci_device *dev, int bar, struct rte_pci_ioport *p) @@ -756,14 +691,8 @@ int rte_pci_write_config(const struct rte_pci_device *device, break; #endif case RTE_PCI_KDRV_IGB_UIO: - ret = pci_uio_ioport_map(dev, bar, p); - break; case RTE_PCI_KDRV_UIO_GENERIC: -#if defined(RTE_ARCH_X86) - ret = pci_ioport_map(dev, bar, p); -#else ret = pci_uio_ioport_map(dev, bar, p); -#endif break; default: break; @@ -830,14 +759,8 @@ int rte_pci_write_config(const struct rte_pci_device *device, break; #endif case RTE_PCI_KDRV_IGB_UIO: - ret = pci_uio_ioport_unmap(p); - break; case RTE_PCI_KDRV_UIO_GENERIC: -#if defined(RTE_ARCH_X86) - ret = 0; -#else ret = pci_uio_ioport_unmap(p); -#endif break; default: break; diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c index f3305a2..01f2a40 100644 --- a/drivers/bus/pci/linux/pci_uio.c +++ b/drivers/bus/pci/linux/pci_uio.c @@ -373,10 +373,13 @@ pci_uio_ioport_map(struct rte_pci_device *dev, int bar, struct rte_pci_ioport *p) { + FILE *f = NULL; char dirname[PATH_MAX]; char filename[PATH_MAX]; - int uio_num; - unsigned long start; + char buf[BUFSIZ]; + uint64_t phys_addr, end_addr, flags; + unsigned long base; + int i; if (rte_eal_iopl_init() != 0) { RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions for PCI device %s\n", @@ -384,41 +387,66 @@ return -1; } - uio_num = pci_get_uio_dev(dev, dirname, sizeof(dirname), 0); - if (uio_num < 0) + /* open and read addresses of the corresponding resource in sysfs */ + snprintf(filename, sizeof(filename), "%s/" PCI_PRI_FMT "/resource", + rte_pci_get_sysfs_path(), dev->addr.domain, dev->addr.bus, + dev->addr.devid, dev->addr.function); + f = fopen(filename, "r"); + if (f == NULL) { + RTE_LOG(ERR, EAL, "%s(): Cannot open sysfs resource: %s\n", + __func__, strerror(errno)); return -1; + } - /* get portio start */ - snprintf(filename, sizeof(filename), - "%s/portio/port%d/start", dirname, bar); - if (eal_parse_sysfs_value(filename, &start) < 0) { - RTE_LOG(ERR, EAL, "%s(): cannot parse portio start\n", - __func__); - return -1; + for (i = 0; i < bar + 1; i++) { + if (fgets(buf, sizeof(buf), f) == NULL) { + RTE_LOG(ERR, EAL, "%s(): Cannot read sysfs resource\n", __func__); + goto error; + } } - /* ensure we don't get anything funny here, read/write will cast to - * uin16_t */ - if (start > UINT16_MAX) - return -1; + if (pci_parse_one_sysfs_resource(buf, sizeof(buf), &phys_addr, + &end_addr, &flags) < 0) + goto error; + + if (!(flags & IORESOURCE_IO)) { + RTE_LOG(ERR, EAL, "%s(): bar resource other than IO is not supported\n", __func__); + goto error; + } + base = (unsigned long)phys_addr; + RTE_LOG(INFO, EAL, "%s(): PIO BAR %08lx detected\n", __func__, base); + + if (base > UINT16_MAX) + goto error; /* FIXME only for primary process ? */ if (dev->intr_handle.type == RTE_INTR_HANDLE_UNKNOWN) { + int uio_num = pci_get_uio_dev(dev, dirname, sizeof(dirname), 0); + if (uio_num < 0) { + RTE_LOG(ERR, EAL, "cannot open %s: %s\n", + dirname, strerror(errno)); + goto error; + } snprintf(filename, sizeof(filename), "/dev/uio%u", uio_num); dev->intr_handle.fd = open(filename, O_RDWR); if (dev->intr_handle.fd < 0) { RTE_LOG(ERR, EAL, "Cannot open %s: %s\n", filename, strerror(errno)); - return -1; + goto error; } dev->intr_handle.type = RTE_INTR_HANDLE_UIO; } - RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%lx\n", start); + RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%lx\n", base); - p->base = start; + p->base = base; p->len = 0; + fclose(f); return 0; +error: + if (f) + fclose(f); + return -1; } #else int From patchwork Mon Feb 22 17:15:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?= X-Patchwork-Id: 88087 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B15C7A034F; Mon, 22 Feb 2021 18:15:47 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7BB6422A27B; Mon, 22 Feb 2021 18:15:40 +0100 (CET) Received: from out0-139.mail.aliyun.com (out0-139.mail.aliyun.com [140.205.0.139]) by mails.dpdk.org (Postfix) with ESMTP id C3A9422A274 for ; Mon, 22 Feb 2021 18:15:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alibaba-inc.com; s=default; t=1614014134; h=From:To:Subject:Date:Message-Id; bh=jzuwtI4O2kQjuRpPJpXF7KkbhAz1VnYeISFV89a56yw=; b=nv0Y3vfAYtNJYmwiT1Op+qNfXeIzvxLXZ7MqFXZQYxjRlOcTp26N2h4KsPrc8DgYn7as5ynxWxX7yXwqXwgJYt8kS601qmJz7ItWiEMCJeypHoj4ji5PbnfdOhvb2UiA1J+VgdZkBGNeXpBo9Y6KHa8i6CEnhMnrFtCs5VHbRPM= X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R201e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=ay29a033018047205; MF=huawei.xhw@alibaba-inc.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---.JbmAyz9_1614014121; Received: from rs3a10040.et2sqa.z1.et2sqa.tbsite.net(mailfrom:huawei.xhw@alibaba-inc.com fp:SMTPD_---.JbmAyz9_1614014121) by smtp.aliyun-inc.com(127.0.0.1); Tue, 23 Feb 2021 01:15:33 +0800 From: " =?utf-8?b?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?= " To: ferruh.yigit@intel.com, maxime.coquelin@redhat.com, david.marchand@redhat.com Cc: , , , , , " =?utf-8?b?6LCi5Y2O5LyfKOatpA==?= =?utf-8?b?5pe25q2k5Yi777yJ?= " Date: Tue, 23 Feb 2021 01:15:18 +0800 Message-Id: <1614014118-91150-3-git-send-email-huawei.xhw@alibaba-inc.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1614014118-91150-1-git-send-email-huawei.xhw@alibaba-inc.com> References: <1611890309-99135-1-git-send-email-huawei.xhw@alibaba-inc.com> <1614014118-91150-1-git-send-email-huawei.xhw@alibaba-inc.com> Subject: [dpdk-dev] [PATCH v7 2/2] bus/pci: support MMIO in PCI ioport accessors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: "huawei.xhw" With IO BAR, we get PIO(programmed IO) address. With MMIO BAR, we get mapped virtual address. We distinguish PIO(Programmed IO) and MMIO(memory mapped IO) by their address like how kernel does. ioread/write8/16/32 is provided to access PIO/MMIO. By the way, for virtio on arch other than x86, BAR flag indicates PIO but is mapped. Signed-off-by: huawei xie Reviewed-by: Maxime Coquelin --- drivers/bus/pci/linux/pci.c | 4 -- drivers/bus/pci/linux/pci_uio.c | 156 +++++++++++++++++++++++++++++----------- 2 files changed, 113 insertions(+), 47 deletions(-) diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c index 0f38abf..0dc99e9 100644 --- a/drivers/bus/pci/linux/pci.c +++ b/drivers/bus/pci/linux/pci.c @@ -715,8 +715,6 @@ int rte_pci_write_config(const struct rte_pci_device *device, break; #endif case RTE_PCI_KDRV_IGB_UIO: - pci_uio_ioport_read(p, data, len, offset); - break; case RTE_PCI_KDRV_UIO_GENERIC: pci_uio_ioport_read(p, data, len, offset); break; @@ -736,8 +734,6 @@ int rte_pci_write_config(const struct rte_pci_device *device, break; #endif case RTE_PCI_KDRV_IGB_UIO: - pci_uio_ioport_write(p, data, len, offset); - break; case RTE_PCI_KDRV_UIO_GENERIC: pci_uio_ioport_write(p, data, len, offset); break; diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c index 01f2a40..f566953 100644 --- a/drivers/bus/pci/linux/pci_uio.c +++ b/drivers/bus/pci/linux/pci_uio.c @@ -368,6 +368,8 @@ return -1; } +#define PIO_MAX 0x10000 + #if defined(RTE_ARCH_X86) int pci_uio_ioport_map(struct rte_pci_device *dev, int bar, @@ -381,12 +383,6 @@ unsigned long base; int i; - if (rte_eal_iopl_init() != 0) { - RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions for PCI device %s\n", - __func__, dev->name); - return -1; - } - /* open and read addresses of the corresponding resource in sysfs */ snprintf(filename, sizeof(filename), "%s/" PCI_PRI_FMT "/resource", rte_pci_get_sysfs_path(), dev->addr.domain, dev->addr.bus, @@ -408,15 +404,27 @@ &end_addr, &flags) < 0) goto error; - if (!(flags & IORESOURCE_IO)) { - RTE_LOG(ERR, EAL, "%s(): bar resource other than IO is not supported\n", __func__); - goto error; - } - base = (unsigned long)phys_addr; - RTE_LOG(INFO, EAL, "%s(): PIO BAR %08lx detected\n", __func__, base); + if (flags & IORESOURCE_IO) { + if (rte_eal_iopl_init()) { + RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions for PCI device %s\n", + __func__, dev->name); + goto error; + } - if (base > UINT16_MAX) + base = (unsigned long)phys_addr; + if (base > PIO_MAX) { + RTE_LOG(ERR, EAL, "%s(): %08lx too large PIO resource\n", __func__, base); + goto error; + } + + RTE_LOG(DEBUG, EAL, "%s(): PIO BAR %08lx detected\n", __func__, base); + } else if (flags & IORESOURCE_MEM) { + base = (unsigned long)dev->mem_resource[bar].addr; + RTE_LOG(DEBUG, EAL, "%s(): MMIO BAR %08lx detected\n", __func__, base); + } else { + RTE_LOG(ERR, EAL, "%s(): unknown BAR type\n", __func__); goto error; + } /* FIXME only for primary process ? */ if (dev->intr_handle.type == RTE_INTR_HANDLE_UNKNOWN) { @@ -517,6 +525,92 @@ } #endif +#if defined(RTE_ARCH_X86) +static inline uint8_t ioread8(void *addr) +{ + uint8_t val; + + val = (uint64_t)(uintptr_t)addr >= PIO_MAX ? + *(volatile uint8_t *)addr : + inb((unsigned long)addr); + + return val; +} + +static inline uint16_t ioread16(void *addr) +{ + uint16_t val; + + val = (uint64_t)(uintptr_t)addr >= PIO_MAX ? + *(volatile uint16_t *)addr : + inw((unsigned long)addr); + + return val; +} + +static inline uint32_t ioread32(void *addr) +{ + uint32_t val; + + val = (uint64_t)(uintptr_t)addr >= PIO_MAX ? + *(volatile uint32_t *)addr : + inl((unsigned long)addr); + + return val; +} + +static inline void iowrite8(uint8_t val, void *addr) +{ + (uint64_t)(uintptr_t)addr >= PIO_MAX ? + *(volatile uint8_t *)addr = val : + outb(val, (unsigned long)addr); +} + +static inline void iowrite16(uint16_t val, void *addr) +{ + (uint64_t)(uintptr_t)addr >= PIO_MAX ? + *(volatile uint16_t *)addr = val : + outw(val, (unsigned long)addr); +} + +static inline void iowrite32(uint32_t val, void *addr) +{ + (uint64_t)(uintptr_t)addr >= PIO_MAX ? + *(volatile uint32_t *)addr = val : + outl(val, (unsigned long)addr); +} +#else +static inline uint8_t ioread8(void *addr) +{ + return *(volatile uint8_t *)addr; +} + +static inline uint16_t ioread16(void *addr) +{ + return *(volatile uint16_t *)addr; +} + +static inline uint32_t ioread32(void *addr) +{ + return *(volatile uint32_t *)addr; +} + +static inline void iowrite8(uint8_t val, void *addr) +{ + *(volatile uint8_t *)addr = val; +} + +static inline void iowrite16(uint16_t val, void *addr) +{ + *(volatile uint16_t *)addr = val; +} + +static inline void iowrite32(uint32_t val, void *addr) +{ + *(volatile uint32_t *)addr = val; +} +#endif + void pci_uio_ioport_read(struct rte_pci_ioport *p, void *data, size_t len, off_t offset) @@ -528,25 +622,13 @@ for (d = data; len > 0; d += size, reg += size, len -= size) { if (len >= 4) { size = 4; -#if defined(RTE_ARCH_X86) - *(uint32_t *)d = inl(reg); -#else - *(uint32_t *)d = *(volatile uint32_t *)reg; -#endif + *(uint32_t *)d = ioread32((void *)reg); } else if (len >= 2) { size = 2; -#if defined(RTE_ARCH_X86) - *(uint16_t *)d = inw(reg); -#else - *(uint16_t *)d = *(volatile uint16_t *)reg; -#endif + *(uint16_t *)d = ioread16((void *)reg); } else { size = 1; -#if defined(RTE_ARCH_X86) - *d = inb(reg); -#else - *d = *(volatile uint8_t *)reg; -#endif + *d = ioread8((void *)reg); } } } @@ -562,25 +644,13 @@ for (s = data; len > 0; s += size, reg += size, len -= size) { if (len >= 4) { size = 4; -#if defined(RTE_ARCH_X86) - outl_p(*(const uint32_t *)s, reg); -#else - *(volatile uint32_t *)reg = *(const uint32_t *)s; -#endif + iowrite32(*(const uint32_t *)s, (void *)reg); } else if (len >= 2) { size = 2; -#if defined(RTE_ARCH_X86) - outw_p(*(const uint16_t *)s, reg); -#else - *(volatile uint16_t *)reg = *(const uint16_t *)s; -#endif + iowrite16(*(const uint16_t *)s, (void *)reg); } else { size = 1; -#if defined(RTE_ARCH_X86) - outb_p(*s, reg); -#else - *(volatile uint8_t *)reg = *s; -#endif + iowrite8(*s, (void *)reg); } } }