From patchwork Mon Mar 14 01:44:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Wu X-Patchwork-Id: 108703 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1F114A00BE; Mon, 14 Mar 2022 02:46:41 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A826840698; Mon, 14 Mar 2022 02:46:40 +0100 (CET) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2071.outbound.protection.outlook.com [40.107.236.71]) by mails.dpdk.org (Postfix) with ESMTP id 278CA40141; Mon, 14 Mar 2022 02:46:39 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RgBPsYyyyvKiOPngtE0GdBRlLbf58AXeDgNqlczRG78G8teL6hqaLweK4T+IiWScRAnOyAanwX0+nYxgu4EwRh3xzD7tlDPXfl730h4vWpMyM0q1Hl4cC62XVGCD9Q5m14twMYVYuR85mqsHaG1J2Be5bwLTQNbPuJWIblQk251EEwNQloruxYrkBloJC/POgBEgsEI4U7JbpYnJAmMb03NS84mlyO/rgCJCY2u5sO6Qy0imzdo7XXUtBjSruFE+2Z6JDtoz8pvAelQh/dGiAiimrf9T8BczDYPAs7lDKHqUrYxVJREHwU69MRAMKLF9Z19AE6RDKL8ZKOe5pyMULA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ay9CqBR9H5nblGQgMBHhdIo8zuCDey40eCVXZzu1gQ4=; b=GkERg6DtE7v/UEsCglx2hXfO60GTqcLLjuArCHiR3QQnzhZO3+kXmcL3sb/he8adZDBUePfHcbit9XyQ09WNMNE2gMsX7iVKDYE6fvKW0PE+ckAd2HIlG3tpWKmg+vzUqgYmFFE5NjFEHWvRpbzgz5ruegGYVI/HPGNulFVlUZVQlPPkiJJU84hi6otSs3nsFfahDw/UlyQuXX3+X0bC5Zer4xjBNXgJp3BxmfY62owL4pfn9SQVcQvOeazvi354qiDcgFhTkEDgPv688Hc/2wkrcz6ZJ4Xy96BstD3NgxxGs7Rre6xKVlcAnCf07PddwhU+mawFOc3j+S2WY4VOqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ay9CqBR9H5nblGQgMBHhdIo8zuCDey40eCVXZzu1gQ4=; b=sVG5R8YDO2LvzTSg9LtPXEut6x2w2HAixmoThkCCdBhqJOTUqtPgT5v1ALEkeQ5Ox79eIzJ+wlrnc7358qdnqYpQpyR70OD54opu7JsmVY+pTxQlBLR6eQwj/yZcX7W48a+2uPlWjjz5BufH3xaRHJ2ytd5KU3fpAt57UgSmENPWwa06B0CbTYdiwFGC7llCG555JxfuYa8mDuervrRdiSQ5f++gCLD0R+6i0Fka6xsa1zseYe3m537gzN6Yluw7VJPqoBdjugTzGtLtRZUu0t9hN+ly7aIfV760KYnzL+/kcgm3VCcZUi95MMrRHTRXKJEhYpP9aCBgNHclkYt6+A== Received: from BN9PR03CA0589.namprd03.prod.outlook.com (2603:10b6:408:10d::24) by BL1PR12MB5078.namprd12.prod.outlook.com (2603:10b6:208:313::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5061.24; Mon, 14 Mar 2022 01:46:27 +0000 Received: from BN8NAM11FT056.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10d:cafe::26) by BN9PR03CA0589.outlook.office365.com (2603:10b6:408:10d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5061.26 via Frontend Transport; Mon, 14 Mar 2022 01:46:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.235) by BN8NAM11FT056.mail.protection.outlook.com (10.13.177.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5061.22 via Frontend Transport; Mon, 14 Mar 2022 01:46:26 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 14 Mar 2022 01:46:16 +0000 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Sun, 13 Mar 2022 18:46:13 -0700 From: Yajun Wu To: , , , , Maxime Coquelin CC: , , , , Subject: [PATCH] vdpa/mlx5: workaround var offset within page Date: Mon, 14 Mar 2022 03:44:12 +0200 Message-ID: <20220314014414.251869-1-yajunw@nvidia.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 70b6b3f1-c453-4f42-c0a4-08da055c7436 X-MS-TrafficTypeDiagnostic: BL1PR12MB5078:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IzAA/MymrQe4nUMTqlQ+WrSiMGtiQ6zH/vR8EoekWTGtm2Ia61AhRUNSTF7V9mvWsQfT8h4OSPYVBD57G5eRin8bmbWFGGC+8RqBwHixygdszaAYYbtRos+tjBEerGC5mjZz9QKrSVrA82XsrYGCcxiQVaY/qcJqHjhpa+1ifb0twKniVy6elDlS/58MRvby5Vf4n+b5wn+3k8/C7A1p4zQyOJnItUbwJPBI5yjFXDO3JUV3SEMHNeDtynHb+f73T8foxHWZqAZ4t3mMOZodgvWISWXkMQ7ZVh0of322MiFJXK53uWBNbD6677qjl89w0jIu08AOZOoeerGqxSSQhkTYuqlAnxo4cRD673wpR0oRvFYewoZrN9fcJfS7U6ZDeBG/PYfpeeR4ef08GwAMxGF/zNhQI20b3o7AIAUyD+UKZrhZTf0KkwCYBRDWSieIqYPEXPypz9FOZpzInlRreZYcUibeZWoe59Xr7KW/EMmm9WYWu1tasQLsAP2IDiZ400VyNNC6vzGYBYHHiJ+JFU0awHHl2O5UXt0lOnoKENuAK8BUiTg6HfDaA6anYWFcAYWbnr6i503tZhO+tzkNlhJA5HnM4QP7LWGcibVHJQRedS3lu7xa/jJ7r6E+fw5swkfxRVIxYmKOqPmcge+RAq/3VixxZPyoQS3PJCkPdvoKI+lERwlDU/uF5VpT/Liq2tzXY+5jR7wlwHhY3o2TxQ== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(83380400001)(316002)(82310400004)(2906002)(86362001)(55016003)(36860700001)(40460700003)(47076005)(4326008)(8676002)(70206006)(70586007)(1076003)(7696005)(508600001)(8936002)(2616005)(356005)(110136005)(54906003)(36756003)(6286002)(336012)(426003)(5660300002)(16526019)(26005)(186003)(81166007)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Mar 2022 01:46:26.9503 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 70b6b3f1-c453-4f42-c0a4-08da055c7436 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT056.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5078 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org vDPA driver first uses kernel driver to allocate doorbell(VAR) area for each device. Then uses var->mmap_off and var->length to mmap uverbs device file as doorbell userspace virtual address. Current kernel driver provides var->mmap_off equal to page start of VAR. It's fine with x86 4K page server, because VAR physical address is only 4K aligned thus locate in 4K page start. But with aarch64 64K page server, the actual VAR physical address has offset within page(not locate in 64K page start). So vDPA driver need add this within page offset(caps.doorbell_bar_offset) to get right VAR virtual address. Fixes: 62c813706e4 ("vdpa/mlx5: map doorbell") Cc: stable@dpdk.org Signed-off-by: Yajun Wu Acked-by: Matan Azrad --- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c index 3416797d28..0748710a76 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include @@ -123,7 +124,10 @@ mlx5_vdpa_virtqs_release(struct mlx5_vdpa_priv *priv) priv->td = NULL; } if (priv->virtq_db_addr) { - claim_zero(munmap(priv->virtq_db_addr, priv->var->length)); + /* Mask out the within page offset for ummap. */ + claim_zero(munmap((void *)((uint64_t)priv->virtq_db_addr & + ~(rte_mem_page_size() - 1ULL)), + priv->var->length)); priv->virtq_db_addr = NULL; } priv->features = 0; @@ -486,6 +490,10 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv) priv->virtq_db_addr = NULL; goto error; } else { + /* Add within page offset for 64K page system. */ + priv->virtq_db_addr = (char *)priv->virtq_db_addr + + ((rte_mem_page_size() - 1) & + priv->caps.doorbell_bar_offset); DRV_LOG(DEBUG, "VAR address of doorbell mapping is %p.", priv->virtq_db_addr); }