From patchwork Thu Jan 25 03:17:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Wu X-Patchwork-Id: 136131 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E276D439BA; Thu, 25 Jan 2024 04:18:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 66DBA4029B; Thu, 25 Jan 2024 04:18:38 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2053.outbound.protection.outlook.com [40.107.243.53]) by mails.dpdk.org (Postfix) with ESMTP id 97D7D4027B; Thu, 25 Jan 2024 04:18:36 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oQ8eo076971ZGLJpP2hzuBq4x8upD3uxP0hyNFCq7oVFg2ttcx++nwKaGgOXHp/PU/U6Ii/fK9NnhqPRKiFHtgE0nSsf/GuaC00E/Kwxs9cH5H8/qQTqq2beBBDFp8Ix6zftzajG2+IXs/Zjuv6ROcdSqHk5CgB4MX0nGRSGnIbI/j0rQOmWqSVbHVR2QsRrxzsMZjFujIS6jk9ULtCffJIUvb8brRJ5Q2b023HkjdLC+4nJqh83WjjNT5la/RxP3f9PKaj1rTsuQWqnaMHiWP4sExwEIDiH6oaToWvHyW5utJm1j+piOEBGe3ociszdPtQWbO8ZHS6bqVXZE9ztUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DPVOgG7HK3ZTWExa0WdBzx0QJxaRJMCxNVPR2O0xJu8=; b=W7yxbjM58SBXMDW1q4+g0qTJgDcCAHbbclrua2HKFPaS1iXKLx5t9LljFP7eYRzCnUw4K9DEiOXrZcWlrzO8XPCKVj5WJL0xi0WEFHqILBUVah0uA1YgnhVH2n+9mID1J0EeHAPcQZC/3M61dES+A2uppsZYjxSBsJjBKeSxESGPDR/VYcicUPOPJMtOvwaiTboc4+pVoOUSvtxrHsPUoMyCG7QqtYFqMyHh+o035bZay1yrSMQFPIN5pQLRDiNRbz1lY8kYRVxHri52SkK9iuIMpJAgVCp7U07Q55PL8Gko22NRmSkYr9EVLOQnEdwES0wvH1yw2zbw9V2PrJOPjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DPVOgG7HK3ZTWExa0WdBzx0QJxaRJMCxNVPR2O0xJu8=; b=o4/t6bNlRvDKrukskusm/U5UDtzmk8luwIao1Z/4LQWzeGEnK+rgVHVbftFVSZWRZM+W6lZKZ4FVl6aWSt3nsuc2/FtHHpY6k7IG0lFgiXyp2f2ES+zy4Vj8Jj5WTbmeGcOFUwLERboeLU4yftHMWMmdnzTRSFYruaOSrYegefjCc2JGN/EMHY8voHwoq0b08b0HW0537SDDhU3boqyLn1a3VhwZm26ZXR5s0YF8iopdKW8iZMtzTkrh8pdkimPtZAH++0N5m6aBa52geApuDqWs8JePgWuaIZvEjJwi+wvZAXosOneaGn/yeEZxWE11j6fNpQr3gZQT45eJBLyF/A== Received: from CH0PR07CA0004.namprd07.prod.outlook.com (2603:10b6:610:32::9) by CH0PR12MB8529.namprd12.prod.outlook.com (2603:10b6:610:18d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22; Thu, 25 Jan 2024 03:18:34 +0000 Received: from CH1PEPF0000A34A.namprd04.prod.outlook.com (2603:10b6:610:32:cafe::97) by CH0PR07CA0004.outlook.office365.com (2603:10b6:610:32::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22 via Frontend Transport; Thu, 25 Jan 2024 03:18:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CH1PEPF0000A34A.mail.protection.outlook.com (10.167.244.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.16 via Frontend Transport; Thu, 25 Jan 2024 03:18:34 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 24 Jan 2024 19:18:14 -0800 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 24 Jan 2024 19:18:11 -0800 From: Yajun Wu To: , , Viacheslav Ovsiienko CC: , , , Subject: [PATCH] vdpa/mlx5: fix queue enable drain CQ Date: Thu, 25 Jan 2024 11:17:55 +0800 Message-ID: <20240125031755.657102-1-yajunw@nvidia.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000A34A:EE_|CH0PR12MB8529:EE_ X-MS-Office365-Filtering-Correlation-Id: 97e6a829-e47d-4557-ad84-08dc1d545075 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XGiS2pIOR7eXQmJS5oZGDVyIKwnHDgRZ8nKZUWeCPzvGAlmQiTFSsga+pV3/uwQhe/gEJnGwUch1lGyKLg3uTSAw1camzFDN7jvo4TJFuxBDCO35GD9amRSACYgkUc91dJYaLWtstH2oasl2GR/9Wi2RI7by4WTeSaLZbl8mkdxeG/EzAjDvDsu276GrWpCtPJOMKNi9BD0xbtUhAPosWMEwGnxWezdolf1x39lrwyJgVYtTjHdYgpyCK5zE97zm5wzOYeAZQ3Ey8udj7WsTFBlVQB5k5I3Qk/k3O72MMJi9WLSpRP4ZrvdhbMmtdKZ3HWqQdwSUtapSJ6fVSnQqagj0jio/CjpLqQbARG/1t+/7PJt8rzI8k7rONm+fIagXtUtVoanYFaAoq0D5jnDlV0jia7/zRucA71YDKqyaSXJxzcGBAQ3tYgVA2Dim5TY/Bag8ptmezsfZLucf2nvurHqlq8tECchcnYuoYVQPyqdTqewDKH8Muh1UdrR22lPjMlEioTLLLlOefoKcAb/lkiFjFgMKYGlyimbwZgnmmejOXqJmBbRPsJ4d6iL+zDiZ8JPNPStoElHpVaUxRK0f1xFN04fk832iUInCVFcXPDSMar46pE574PiQKx9KxyS7oNMCcpFYcrbrA5ZhU1AdhNuwuGUkCJiltGNP2gxUEED7lVfkxWrY35pd53xtlFKAHIp8lgeHvnjmi0woTTQ7s/0PQbCaWxuEB2ZBt7E0ARmtdyTdWjulyz/+nn63+Qz7n2kQ5YHyD8yIJb0u8jDItyMNhZg5gYqZuNw4HXcA/Bw= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(396003)(346002)(136003)(376002)(39860400002)(230922051799003)(64100799003)(451199024)(186009)(1800799012)(82310400011)(40470700004)(46966006)(36840700001)(4326008)(83380400001)(8936002)(55016003)(40480700001)(40460700003)(8676002)(110136005)(70206006)(70586007)(54906003)(6636002)(316002)(426003)(1076003)(6286002)(16526019)(336012)(26005)(36860700001)(7696005)(82740400003)(2906002)(86362001)(47076005)(5660300002)(356005)(7636003)(2616005)(478600001)(6666004)(41300700001)(36756003)(21314003)(2101003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 03:18:34.2396 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 97e6a829-e47d-4557-ad84-08dc1d545075 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000A34A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8529 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the case: `ethtool -L eth0 combined xxx` in VM, VQ will disable and enable without calling device close. In such case, need add drain CQ before reuse/reset event QP. Fixes: 24969c7b62 ("vdpa/mlx5: reuse event queues") Cc: stable@dpdk.org Signed-off-by: Yajun Wu Acked-by: Matan Azrad Reviewed-by: Maxime Coquelin --- drivers/vdpa/mlx5/mlx5_vdpa_event.c | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c index 9557c1042e..32430614d5 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c @@ -244,22 +244,30 @@ mlx5_vdpa_queues_complete(struct mlx5_vdpa_priv *priv) return max; } +static void +mlx5_vdpa_drain_cq_one(struct mlx5_vdpa_priv *priv, + struct mlx5_vdpa_virtq *virtq) +{ + struct mlx5_vdpa_cq *cq = &virtq->eqp.cq; + + mlx5_vdpa_queue_complete(cq); + if (cq->cq_obj.cq) { + cq->cq_obj.cqes[0].wqe_counter = rte_cpu_to_be_16(UINT16_MAX); + virtq->eqp.qp_pi = 0; + if (!cq->armed) + mlx5_vdpa_cq_arm(priv, cq); + } +} + void mlx5_vdpa_drain_cq(struct mlx5_vdpa_priv *priv) { + struct mlx5_vdpa_virtq *virtq; unsigned int i; for (i = 0; i < priv->caps.max_num_virtio_queues; i++) { - struct mlx5_vdpa_cq *cq = &priv->virtqs[i].eqp.cq; - - mlx5_vdpa_queue_complete(cq); - if (cq->cq_obj.cq) { - cq->cq_obj.cqes[0].wqe_counter = - rte_cpu_to_be_16(UINT16_MAX); - priv->virtqs[i].eqp.qp_pi = 0; - if (!cq->armed) - mlx5_vdpa_cq_arm(priv, cq); - } + virtq = &priv->virtqs[i]; + mlx5_vdpa_drain_cq_one(priv, virtq); } } @@ -632,6 +640,7 @@ mlx5_vdpa_event_qp_prepare(struct mlx5_vdpa_priv *priv, uint16_t desc_n, if (eqp->cq.cq_obj.cq != NULL && log_desc_n == eqp->cq.log_desc_n) { /* Reuse existing resources. */ eqp->cq.callfd = callfd; + mlx5_vdpa_drain_cq_one(priv, virtq); /* FW will set event qp to error state in q destroy. */ if (reset && !mlx5_vdpa_qps2rst2rts(eqp)) rte_write32(rte_cpu_to_be_32(RTE_BIT32(log_desc_n)),