From patchwork Fri Jul 20 08:13:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Takeshi Yoshimura X-Patchwork-Id: 43224 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1ABE41C01; Fri, 20 Jul 2018 10:13:57 +0200 (CEST) Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by dpdk.org (Postfix) with ESMTP id 707D9160 for ; Fri, 20 Jul 2018 10:13:55 +0200 (CEST) Received: by mail-pg1-f193.google.com with SMTP id y5-v6so6224004pgv.1 for ; Fri, 20 Jul 2018 01:13:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Xb/L4C444IcS15/vG3E+6h6KT41mB/7ErkQyVQi0gvM=; b=mAz2hhPVIBq+1/D8eyqMq0pcULMRcGkSFUpxruVbHARqWQfHFoXgNr8K1dhgVKQWRo NiPS+WLkTvLU5SBArUmHGw8WipSxGKN8wE3BCwYsh59oMEjwW3fSAcyJXXThtR9pZUS5 wFFF60hEcY2/OJN6icqNi8ToWX9vjCtqB8gIVputtUzZJiGEBh5mz+Nx4RDZq1sfigWr wlXslra+zUixwHQrh8rbgAgEZmykrzAXWENDEOUe7pQX2x8VQoFHb23m3Sl5fvZS6etj lgQLKFNUS3idMoRYajefk+ESrjuSRaEvdjTk1Xbn5xg5CF6dZyjYyyZJVLrwaZfrta8+ E60w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Xb/L4C444IcS15/vG3E+6h6KT41mB/7ErkQyVQi0gvM=; b=QJjIg+KK/H3+/kXA5rsLm1Xu5pitMO0/Sf3FZ0QufUET4lBdCOH+TuXpBjTxLv3RZa dPJ4HXy45MmjUmId6p8BL/9InERPcFOsudOSx+KRXS/MD4GhOzWsxn9xULx8Fp+Sdp0o sFSCS+/i4z5I8X/UncS4iwpZB1MVZ9L7gwb62tuSvHt7a6SKBLJg1VgkCRLx1ZftdpGg P2W73bvv5nnr+vwFUD7h5d36X/hN4gjZz21rhD5s5nfbVsTxbLo0lPUJ9CJk1RppOZM6 9Mb5QJOGRncs8ayKgpbjXji4527VKTnTKGfdjOLbLTkzHRb8ThH86a6KzSn9yP1dRT06 pl0w== X-Gm-Message-State: AOUpUlE0yF14lBh5DBjm37MFmx0kBuyqc28bcHQaYUFFLjymS0Wc8h2l /PcWkL95q+4HMEX3JwKF30s= X-Google-Smtp-Source: AAOMgpc59Q1ipnLhOfDp+SDwd2+BT4KS1tO3L/Fytmrx0wzh/tN48UawixyCZV66Dkc6prFAH6I2cA== X-Received: by 2002:a63:8042:: with SMTP id j63-v6mr1126659pgd.230.1532074434534; Fri, 20 Jul 2018 01:13:54 -0700 (PDT) Received: from takeshi-no-air.dhcp.hakozaki.ibm.com (sg-fw-ice-redblue-p3.sagamino.jp.ibm.com. [203.141.91.12]) by smtp.gmail.com with ESMTPSA id a77-v6sm3336212pfj.38.2018.07.20.01.13.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 20 Jul 2018 01:13:53 -0700 (PDT) From: Takeshi Yoshimura To: Anatoly Burakov Cc: dev@dpdk.org, Takeshi Yoshimura Date: Fri, 20 Jul 2018 17:13:47 +0900 Message-Id: <20180720081347.6123-1-t.yoshimura8869@gmail.com> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180712030833.4887-1-t.yoshimura8869@gmail.com> References: <20180712030833.4887-1-t.yoshimura8869@gmail.com> Subject: [dpdk-dev] [PATCH v5] vfio: fix workaround of BAR mapping X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currently, VFIO will try to map around MSI-X table in the BARs. When MSI-X table (page-aligned) size is equal to (page-aligned) size of BAR, VFIO will just skip the BAR. Recent kernel versions will allow VFIO to map the entire BAR containing MSI-X tables (*), so instead of trying to map around the MSI-X vector or skipping the BAR entirely if it's not possible, we can now try mapping the entire BAR first. If mapping the entire BAR doesn't succeed, fall back to the old behavior of mapping around MSI-X table or skipping the BAR. (*): "vfio-pci: Allow mapping MSIX BAR", https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ commit/?id=a32295c612c57990d17fb0f41e7134394b2f35f6 Fixes: 90a1633b2347 ("eal/linux: allow to map BARs with MSI-X tables") Signed-off-by: Takeshi Yoshimura Reviewed-by: Anatoly Burakov --- Thanks, Anatoly. I updated the code with munmap in an error path. I also fixed the message and the wrong link. Regards, Takeshi drivers/bus/pci/linux/pci_vfio.c | 93 ++++++++++++++++++++++------------------ 1 file changed, 52 insertions(+), 41 deletions(-) diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c index aeeaa9ed8..07188c071 100644 --- a/drivers/bus/pci/linux/pci_vfio.c +++ b/drivers/bus/pci/linux/pci_vfio.c @@ -332,50 +332,59 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res, void *bar_addr; struct pci_msix_table *msix_table = &vfio_res->msix_table; struct pci_map *bar = &vfio_res->maps[bar_index]; + bool again = false; if (bar->size == 0) /* Skip this BAR */ return 0; - if (msix_table->bar_index == bar_index) { - /* - * VFIO will not let us map the MSI-X table, - * but we can map around it. - */ - uint32_t table_start = msix_table->offset; - uint32_t table_end = table_start + msix_table->size; - table_end = (table_end + ~PAGE_MASK) & PAGE_MASK; - table_start &= PAGE_MASK; - - if (table_start == 0 && table_end >= bar->size) { - /* Cannot map this BAR */ - RTE_LOG(DEBUG, EAL, "Skipping BAR%d\n", bar_index); - bar->size = 0; - bar->addr = 0; - return 0; - } - - memreg[0].offset = bar->offset; - memreg[0].size = table_start; - memreg[1].offset = bar->offset + table_end; - memreg[1].size = bar->size - table_end; - - RTE_LOG(DEBUG, EAL, - "Trying to map BAR%d that contains the MSI-X " - "table. Trying offsets: " - "0x%04lx:0x%04lx, 0x%04lx:0x%04lx\n", bar_index, - memreg[0].offset, memreg[0].size, - memreg[1].offset, memreg[1].size); - } else { - memreg[0].offset = bar->offset; - memreg[0].size = bar->size; - } - /* reserve the address using an inaccessible mapping */ bar_addr = mmap(bar->addr, bar->size, 0, MAP_PRIVATE | MAP_ANONYMOUS | additional_flags, -1, 0); - if (bar_addr != MAP_FAILED) { + if (bar_addr == MAP_FAILED) { + RTE_LOG(ERR, EAL, + "Failed to create inaccessible mapping for BAR%d\n", + bar_index); + return -1; + } + + memreg[0].offset = bar->offset; + memreg[0].size = bar->size; + do { void *map_addr = NULL; + if (again) { + /* + * VFIO did not let us map the MSI-X table, + * but we can map around it. + */ + uint32_t table_start = msix_table->offset; + uint32_t table_end = table_start + msix_table->size; + table_end = (table_end + ~PAGE_MASK) & PAGE_MASK; + table_start &= PAGE_MASK; + + if (table_start == 0 && table_end >= bar->size) { + /* Cannot map this BAR */ + RTE_LOG(DEBUG, EAL, "Skipping BAR%d\n", + bar_index); + munmap(bar_addr, bar->size); + bar->size = 0; + bar->addr = 0; + return 0; + } + + memreg[0].offset = bar->offset; + memreg[0].size = table_start; + memreg[1].offset = bar->offset + table_end; + memreg[1].size = bar->size - table_end; + + RTE_LOG(DEBUG, EAL, + "Trying to map BAR%d that contains the MSI-X " + "table. Trying offsets: " + "0x%04lx:0x%04lx, 0x%04lx:0x%04lx\n", bar_index, + memreg[0].offset, memreg[0].size, + memreg[1].offset, memreg[1].size); + } + if (memreg[0].size) { /* actual map of first part */ map_addr = pci_map_resource(bar_addr, vfio_dev_fd, @@ -384,6 +393,12 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res, MAP_FIXED); } + if (map_addr == MAP_FAILED && + msix_table->bar_index == bar_index && !again) { + again = true; + continue; + } + /* if there's a second part, try to map it */ if (map_addr != MAP_FAILED && memreg[1].offset && memreg[1].size) { @@ -404,12 +419,8 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res, bar_index); return -1; } - } else { - RTE_LOG(ERR, EAL, - "Failed to create inaccessible mapping for BAR%d\n", - bar_index); - return -1; - } + break; + } while (again); bar->addr = bar_addr; return 0;