From patchwork Mon Nov 10 09:53:22 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 1230 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id D5C5D7F2C; Mon, 10 Nov 2014 10:44:26 +0100 (CET) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 0F1117E8E for ; Mon, 10 Nov 2014 10:44:23 +0100 (CET) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP; 10 Nov 2014 01:54:06 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,351,1413270000"; d="scan'208";a="605210142" Received: from irsmsx102.ger.corp.intel.com ([163.33.3.155]) by orsmga001.jf.intel.com with ESMTP; 10 Nov 2014 01:53:58 -0800 Received: from irsmsx153.ger.corp.intel.com (163.33.192.75) by IRSMSX102.ger.corp.intel.com (163.33.3.155) with Microsoft SMTP Server (TLS) id 14.3.195.1; Mon, 10 Nov 2014 09:53:23 +0000 Received: from irsmsx109.ger.corp.intel.com ([169.254.13.101]) by IRSMSX153.ger.corp.intel.com ([169.254.9.23]) with mapi id 14.03.0195.001; Mon, 10 Nov 2014 09:53:23 +0000 From: "Burakov, Anatoly" To: Liang Xu , "dev@dpdk.org" Thread-Topic: [PATCH] eal: map PCI memory resources after hugepages Thread-Index: AQHP+wSiEAb9hfd8fkOi8Y61mgTq6JxZoD9Q Date: Mon, 10 Nov 2014 09:53:22 +0000 Message-ID: References: <1415417532-4363-1-git-send-email-liang.xu@cinfotech.cn> In-Reply-To: <1415417532-4363-1-git-send-email-liang.xu@cinfotech.cn> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.181] MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH] eal: map PCI memory resources after hugepages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Liang I don't think that overriding the value passed to pci_map_resource as argument is the way to go. While it results in less code, it looks weird, in my opinion at least, as I believe tracking the correctness of address being requested should be the responsibility of the caller, i.e. either UIO or VFIO code. Which is why I keep insisting that you make requested_pci_addr global to linuxapp EAL PCI section and put it into include/eal_pci_init.h. Would you mind if I made a patch for this issue based on your code? Thanks, Anatoly -----Original Message----- From: Liang Xu [mailto:liang.xu@cinfotech.cn] Sent: Saturday, November 8, 2014 3:32 AM To: dev@dpdk.org Cc: Burakov, Anatoly; thomas.monjalon@6wind.com Subject: [PATCH] eal: map PCI memory resources after hugepages A multiple process DPDK application must mmap hugepages and pci resources into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the chosen virtual addresses isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of pci resources mapped still does not refer to the parameter. In that case "EAL: pci_map_resource(): cannot mmap" will be got. This patch try to map pci resources after hugepages. So the error can be resolved by set base-virtaddr into free virtual address space. Signed-off-by: Liang Xu --- lib/librte_eal/linuxapp/eal/eal_pci.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) + default_map_addr = RTE_PTR_ADD(mapaddr, size); RTE_LOG(DEBUG, EAL, " PCI memory mapped at %p\n", mapaddr); -- 1.9.1 diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index ddb0535..502eef2 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -97,14 +97,42 @@ error: return -1; } +static void * +pci_find_max_end_va(void) +{ + const struct rte_memseg *seg = rte_eal_get_physmem_layout(); + const struct rte_memseg *last = seg; + unsigned i = 0; + + for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) { + if (seg->addr == NULL) + break; + + if (seg->addr > last->addr) + last = seg; + + } + return RTE_PTR_ADD(last->addr, last->len); } + /* map a particular resource from a file */ void * pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size) { void *mapaddr; + /* By default the PCI memory resource will be mapped after hugepages */ + static void *default_map_addr; + if (NULL == requested_addr) { + if (NULL == default_map_addr) + default_map_addr = pci_find_max_end_va(); + mapaddr = default_map_addr; + } else { + mapaddr = requested_addr; + } + /* Map the PCI memory resource of device */ - mapaddr = mmap(requested_addr, size, PROT_READ | PROT_WRITE, + mapaddr = mmap(mapaddr, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset); if (mapaddr == MAP_FAILED || (requested_addr != NULL && mapaddr != requested_addr)) { @@ -114,6 +142,8 @@ pci_map_resource(void *requested_addr, int fd, off_t offset, size_t size) strerror(errno), mapaddr); goto fail; } + if (NULL == requested_addr)