From patchwork Fri Nov 7 09:47:45 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 1185 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 19AF87F25; Fri, 7 Nov 2014 10:38:22 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 4509D7EC4 for ; Fri, 7 Nov 2014 10:38:18 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP; 07 Nov 2014 01:41:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,331,1413270000"; d="scan'208";a="618758104" Received: from irsmsx103.ger.corp.intel.com ([163.33.3.157]) by fmsmga001.fm.intel.com with ESMTP; 07 Nov 2014 01:47:47 -0800 Received: from irsmsx108.ger.corp.intel.com (163.33.3.3) by IRSMSX103.ger.corp.intel.com (163.33.3.157) with Microsoft SMTP Server (TLS) id 14.3.195.1; Fri, 7 Nov 2014 09:47:46 +0000 Received: from irsmsx109.ger.corp.intel.com ([169.254.13.101]) by IRSMSX108.ger.corp.intel.com ([169.254.11.82]) with mapi id 14.03.0195.001; Fri, 7 Nov 2014 09:47:45 +0000 From: "Burakov, Anatoly" To: lxu , "dev@dpdk.org" Thread-Topic: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured. Thread-Index: AQHP+mEdCoJJM6uhEkSpotX1GZcOhJxU6HMQ Date: Fri, 7 Nov 2014 09:47:45 +0000 Message-ID: References: <1415193919-17361-1-git-send-email-liang.xu@cinfotech.cn> <1415347284-15468-1-git-send-email-liang.xu@cinfotech.cn> In-Reply-To: <1415347284-15468-1-git-send-email-liang.xu@cinfotech.cn> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.180] MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The commit message looks fine to me, but VFIO code needs to be adjusted the same way. Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the last hugepage (because IVSHMEM memory is first to be reserved, before the main memory). My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h). My apologies for not thinking about any of this during the first 6 iterations of the patch :( Thanks, Anatoly -----Original Message----- From: lxu [mailto:liang.xu@cinfotech.cn] Sent: Friday, November 7, 2014 8:01 AM To: dev@dpdk.org Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured. A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail. This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space. Signed-off-by: lxu --- lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) + /* map uio resource into user required virtual address */ + static void * requested_addr; + if (internal_config.base_virtaddr && NULL == requested_addr) { + const struct rte_memseg * last = get_physmem_last(); + requested_addr = RTE_PTR_ADD(last->addr, last->len); + } + dev->intr_handle.fd = -1; dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN; @@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev) if (maps[j].addr != NULL) fail = 1; else { - mapaddr = pci_map_resource(NULL, fd, (off_t)offset, + mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset, (size_t)maps[j].size); if (mapaddr == NULL) fail = 1; + else if (NULL != requested_addr) + requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size); } if (fail) { -- 1.9.1 diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c index 7e62266..a2c9ab6 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c @@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf, return uio_num; } +static inline const struct rte_memseg * +get_physmem_last(void) +{ + const struct rte_memseg * seg = rte_eal_get_physmem_layout(); + const struct rte_memseg * last = seg; + unsigned i = 0; + + for (i=0; iaddr == NULL) + break; + + if(seg->addr > last->addr) + last = seg; + + } + return last; +} + /* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev) struct mapped_pci_resource *uio_res; struct pci_map *maps;