From patchwork Tue Nov 11 03:53:22 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: lxu X-Patchwork-Id: 1252 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 113187F0D; Tue, 11 Nov 2014 04:43:48 +0100 (CET) Received: from out1134-243.mail.aliyun.com (out1134-243.mail.aliyun.com [42.120.134.243]) by dpdk.org (Postfix) with ESMTP id C03E57E9D for ; Tue, 11 Nov 2014 04:43:42 +0100 (CET) X-Alimail-AntiSpam: AC=CONTINUE; BC=0.03870059|-1; FP=0|0|0|0|0|0|0|0; HT=r46d02012; MF=liang.xu@cinfotech.cn; PH=DW; RN=2; RT=2; SR=0; Received: from WS-web (liang.xu@cinfotech.cn[203.110.175.218]) by r41g03043.xy2.aliyun.com at Tue, 11 Nov 2014 11:53:22 +0800 Date: Tue, 11 Nov 2014 11:53:22 +0800 From: "XU Liang" To: "=?UTF-8?B?QnVyYWtvdiwgQW5hdG9seQ==?=" , "dev@dpdk.org" Message-ID: X-Mailer: Alimail-Mailagent revision 2667797 MIME-Version: 1.0 References: <1415347284-15468-1-git-send-email-liang.xu@cinfotech.cn> <1415619272-8281-1-git-send-email-anatoly.burakov@intel.com>, C6ECDF3AB251BE4894318F4E4512369780C07EEB@IRSMSX109.ger.corp.intel.com In-Reply-To: C6ECDF3AB251BE4894318F4E4512369780C07EEB@IRSMSX109.ger.corp.intel.com x-aliyun-mail-creator: Webmail4_2670074_hLSTW96aWxsYS81LjAgKFdpbmRvd3MgTlQgNi4xOyBXT1c2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzMzLjAuMTc1MC4xNDkgU2FmYXJpLzUzNy4zNg==2I X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] =?utf-8?q?=5BPATCH_v7=5D_eal=3A_map_PCI_memory_resourc?= =?utf-8?q?es_after_hugepages?= X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: XU Liang List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" I had finished some tests. The patch works fine. My tests are included :* single process  + uio + vfio * single process  + uio + vfio + base-virtaddr * multiple processes + uio + vfio * multiple processes + uio + vfio + base-virtaddr My unlucky multiple process application still got error without base-virtaddr when initial hugepages. See the attchments: primary.txt and secondary.txt.With base-virtaddr the patch worked, both hugepages and pci resources were mapped into base-virtaddr, My application is happy. See the attchments: base-virtaddr_primary.txt and base-virtaddr_secondary.txt. ------------------------------------------------------------------From:Burakov, Anatoly Time:2014 Nov 10 (Mon) 21 : 34To:Burakov, Anatoly , dev@dpdk.org Subject:Re: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages Nak, there are issues with the patch. There is another patch already, but I'll submit it whenever Liang verifies it works with his setup. Thanks, Anatoly -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov Sent: Monday, November 10, 2014 11:35 AM To: dev@dpdk.org Subject: [dpdk-dev] [PATCH v7] eal: map PCI memory resources after hugepages Multi-process DPDK application must mmap hugepages and pci resources into the same virtual address space. By default the virtual addresses are chosen by the primary process automatically when calling the mmap. But sometimes the chosen virtual addresses aren't usable in secondary process - for example, secondary process is linked with more libraries than primary process, and the library occupies the same address space that the primary process has requested for PCI mappings. This patch makes EAL map PCI BARs right after the hugepages (instead of location chosen by mmap) in virtual memory. Signed-off-by: Anatoly Burakov Signed-off-by: Liang Xu --- lib/librte_eal/linuxapp/eal/eal_pci.c | 19 +++++++++++++++++++ lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 9 ++++++++- lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 13 +++++++++++-- lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 6 ++++++ 4 files changed, 44 insertions(+), 3 deletions(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index 5fe3961..dae8739 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -97,6 +97,25 @@ error: return -1; } +void * +pci_find_max_end_va(void) +{ + const struct rte_memseg *seg = rte_eal_get_physmem_layout(); + const struct rte_memseg *last = seg; + unsigned i = 0; + + for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) { + if (seg->addr == NULL) + break; + + if (seg->addr > last->addr) + last = seg; + + } + return RTE_PTR_ADD(last->addr, last->len); +} + + /* map a particular resource from a file */ void * pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size) diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c index 7e62266..5090bf1 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c @@ -48,6 +48,8 @@ static int pci_parse_sysfs_value(const char *filename, uint64_t *val); +void *pci_map_addr = NULL; + #define OFF_MAX ((uint64_t)(off_t)-1) static int @@ -371,10 +373,15 @@ pci_uio_map_resource(struct rte_pci_device *dev) if (maps[j].addr != NULL) fail = 1; else { - mapaddr = pci_map_resource(NULL, fd, (off_t)offset, + if (pci_map_addr == NULL) + pci_map_addr = pci_find_max_end_va(); + + mapaddr = pci_map_resource(pci_map_addr, fd, (off_t)offset, (size_t)maps[j].size); if (mapaddr == NULL) fail = 1; + + pci_map_addr = RTE_PTR_ADD(pci_map_addr, maps[j].size); } if (fail) { diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c index c776ddc..fb6ee7a 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c @@ -720,8 +720,17 @@ pci_vfio_map_resource(struct rte_pci_device *dev) if (i == msix_bar) continue; - bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset, - reg.size); + if (internal_config.process_type == RTE_PROC_PRIMARY) { + if (pci_map_addr == NULL) + pci_map_addr = pci_find_max_end_va(); + + bar_addr = pci_map_resource(pci_map_addr, vfio_dev_fd, reg.offset, + reg.size); + pci_map_addr = RTE_PTR_ADD(pci_map_addr, reg.size); + } else { + bar_addr = pci_map_resource(maps[i].addr, vfio_dev_fd, reg.offset, + reg.size); + } if (bar_addr == NULL) { RTE_LOG(ERR, EAL, " %s mapping BAR%i failed: %s\n", pci_addr, i, diff --git a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h index d758bee..1070eb8 100644 --- a/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h +++ b/lib/librte_eal/linuxapp/eal/include/eal_pci_init.h @@ -59,6 +59,12 @@ struct mapped_pci_resource { TAILQ_HEAD(mapped_pci_res_list, mapped_pci_resource); extern struct mapped_pci_res_list *pci_res_list; +/* + * Helper function to map PCI resources right after hugepages in virtual memory + */ +extern void *pci_map_addr; +void *pci_find_max_end_va(void); + void *pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size);