From patchwork Tue Jul 17 08:22:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Takeshi Yoshimura X-Patchwork-Id: 43124 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id F3C022965; Tue, 17 Jul 2018 10:22:38 +0200 (CEST) Received: from mail-pl0-f65.google.com (mail-pl0-f65.google.com [209.85.160.65]) by dpdk.org (Postfix) with ESMTP id B03CB160 for ; Tue, 17 Jul 2018 10:22:37 +0200 (CEST) Received: by mail-pl0-f65.google.com with SMTP id 30-v6so128718pld.13 for ; Tue, 17 Jul 2018 01:22:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dyhudwot7o68NjFILyYZfymosFmRU2F5lQKSFH+Dc7I=; b=Egp48ffv62ZCQ0hyGoQtJOAi66JTUprf+sLdndQIh52CdEw0a5/bOHd2O78w2rti/9 t9taoeYRHuyePrLpZP0pDK67yecPWfD8US20TbfcdcUg+2p3TY+faYgt1H7vNYu3tVYJ iiIVP2Kgwxk5XEzbaSyfVhlS0PmRYSVEpKAbXNc4gM0lDt8pEjh8cd0lN4LQUlF5RphI JuVwW08Hkj4N7RvtPtdgtA30XxpjVQfWnr0ZFMSNOtmRQ0ReSE09tQMooI8YxGg4qGeQ mCXCIu5JjXyk0weJvND62tzyUhQ+fjoga8C6nQjuaEvUqhft/0IuU5CEJmxxm0HwVzNZ SDOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dyhudwot7o68NjFILyYZfymosFmRU2F5lQKSFH+Dc7I=; b=YmXcG+hNTnop1SRCScvCQaEDMTIRB3eGG8tGthIufD6iJspu293deCRhEvpN3ghZpl NlaNxGAPkl3xXJUbDb5TpjFq0ov88FaDrUttzteC8rTpHHDO9AXOlYO3TevWlKM/nRQU RvuUNZEIfjqdZHtRu9nRIQ7eFU5AajxEhrRZ3vMU3ovRTNgjTZRUpUSHNO7M18Czzrep iAog+mjz6RjUM2ZhGkaco0/5SCsAMFL2tixgiyTWvNmDq8K8Akt86UDdaK6EXH5X34cR ++aQGGQOvx3fGsZs4kycaxGGVOFd8WhfOaXYSHpGUWJC2a5GiDRvxMwgCqvbDrfoWQFI Kbng== X-Gm-Message-State: AOUpUlGiwEYLv1SSoiygXaf78Zt/OshtLzyS7sCV2/0kv/b69adz20Mq B5s1Rsm/kt+YykWtbIWE6z4= X-Google-Smtp-Source: AAOMgpepB/WBp8KHXpCInaCVy0kvDmGBJZfWURZtCN7pAoTtc0kqREDmKFuM4Ch287zUMs4ecw/KZg== X-Received: by 2002:a17:902:2006:: with SMTP id n6-v6mr646657pla.325.1531815756481; Tue, 17 Jul 2018 01:22:36 -0700 (PDT) Received: from takeshi-no-air.dhcp.hakozaki.ibm.com (sg-fw-ice-redblue-p7.sagamino.jp.ibm.com. [203.141.91.16]) by smtp.gmail.com with ESMTPSA id f17-v6sm978120pfj.122.2018.07.17.01.22.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 17 Jul 2018 01:22:35 -0700 (PDT) From: Takeshi Yoshimura To: Anatoly Burakov Cc: dev@dpdk.org, Takeshi Yoshimura Date: Tue, 17 Jul 2018 17:22:30 +0900 Message-Id: <20180717082230.3322-1-t.yoshimura8869@gmail.com> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20180712030833.4887-1-t.yoshimura8869@gmail.com> References: <20180712030833.4887-1-t.yoshimura8869@gmail.com> Subject: [dpdk-dev] [PATCH v4] vfio: fix workaround of BAR0 mapping X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The workaround of BAR0 mapping gives up and immediately returns an error if it cannot map around the MSI-X. However, recent version of VFIO allows MSIX mapping (*). I fixed not to return immediately but try mapping. In old Linux, mmap just fails and returns the same error as the code before my fix . In recent Linux, mmap succeeds and this patch enables running DPDK in specific environments (e.g., ppc64le with HGST NVMe) (*): "vfio-pci: Allow mapping MSIX BAR", https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/ commit/id=a32295c612c57990d17fb0f41e7134394b2f35f6 Fixes: 90a1633b2347 ("eal/linux: allow to map BARs with MSI-X tables") Signed-off-by: Takeshi Yoshimura Reviewed-by: Anatoly Burakov --- drivers/bus/pci/linux/pci_vfio.c | 92 ++++++++++++++++++++++------------------ 1 file changed, 51 insertions(+), 41 deletions(-) diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c index aeeaa9ed8..afdf0f6d5 100644 --- a/drivers/bus/pci/linux/pci_vfio.c +++ b/drivers/bus/pci/linux/pci_vfio.c @@ -332,50 +332,58 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res, void *bar_addr; struct pci_msix_table *msix_table = &vfio_res->msix_table; struct pci_map *bar = &vfio_res->maps[bar_index]; + bool again = false; if (bar->size == 0) /* Skip this BAR */ return 0; - if (msix_table->bar_index == bar_index) { - /* - * VFIO will not let us map the MSI-X table, - * but we can map around it. - */ - uint32_t table_start = msix_table->offset; - uint32_t table_end = table_start + msix_table->size; - table_end = (table_end + ~PAGE_MASK) & PAGE_MASK; - table_start &= PAGE_MASK; - - if (table_start == 0 && table_end >= bar->size) { - /* Cannot map this BAR */ - RTE_LOG(DEBUG, EAL, "Skipping BAR%d\n", bar_index); - bar->size = 0; - bar->addr = 0; - return 0; - } - - memreg[0].offset = bar->offset; - memreg[0].size = table_start; - memreg[1].offset = bar->offset + table_end; - memreg[1].size = bar->size - table_end; - - RTE_LOG(DEBUG, EAL, - "Trying to map BAR%d that contains the MSI-X " - "table. Trying offsets: " - "0x%04lx:0x%04lx, 0x%04lx:0x%04lx\n", bar_index, - memreg[0].offset, memreg[0].size, - memreg[1].offset, memreg[1].size); - } else { - memreg[0].offset = bar->offset; - memreg[0].size = bar->size; - } - /* reserve the address using an inaccessible mapping */ bar_addr = mmap(bar->addr, bar->size, 0, MAP_PRIVATE | MAP_ANONYMOUS | additional_flags, -1, 0); - if (bar_addr != MAP_FAILED) { + if (bar_addr == MAP_FAILED) { + RTE_LOG(ERR, EAL, + "Failed to create inaccessible mapping for BAR%d\n", + bar_index); + return -1; + } + + memreg[0].offset = bar->offset; + memreg[0].size = bar->size; + do { void *map_addr = NULL; + if (again) { + /* + * VFIO did not let us map the MSI-X table, + * but we can map around it. + */ + uint32_t table_start = msix_table->offset; + uint32_t table_end = table_start + msix_table->size; + table_end = (table_end + ~PAGE_MASK) & PAGE_MASK; + table_start &= PAGE_MASK; + + if (table_start == 0 && table_end >= bar->size) { + /* Cannot map this BAR */ + RTE_LOG(DEBUG, EAL, "Skipping BAR%d\n", + bar_index); + bar->size = 0; + bar->addr = 0; + return 0; + } + + memreg[0].offset = bar->offset; + memreg[0].size = table_start; + memreg[1].offset = bar->offset + table_end; + memreg[1].size = bar->size - table_end; + + RTE_LOG(DEBUG, EAL, + "Trying to map BAR%d that contains the MSI-X " + "table. Trying offsets: " + "0x%04lx:0x%04lx, 0x%04lx:0x%04lx\n", bar_index, + memreg[0].offset, memreg[0].size, + memreg[1].offset, memreg[1].size); + } + if (memreg[0].size) { /* actual map of first part */ map_addr = pci_map_resource(bar_addr, vfio_dev_fd, @@ -384,6 +392,12 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res, MAP_FIXED); } + if (map_addr == MAP_FAILED && + msix_table->bar_index == bar_index && !again) { + again = true; + continue; + } + /* if there's a second part, try to map it */ if (map_addr != MAP_FAILED && memreg[1].offset && memreg[1].size) { @@ -404,12 +418,8 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res, bar_index); return -1; } - } else { - RTE_LOG(ERR, EAL, - "Failed to create inaccessible mapping for BAR%d\n", - bar_index); - return -1; - } + break; + } while (again); bar->addr = bar_addr; return 0;