From patchwork Fri Jun 30 03:15:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rma Ma X-Patchwork-Id: 129141 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0667F42D73; Fri, 30 Jun 2023 09:15:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 05BBE42C54; Fri, 30 Jun 2023 09:15:27 +0200 (CEST) Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2069.outbound.protection.outlook.com [40.107.215.69]) by mails.dpdk.org (Postfix) with ESMTP id 13BB24021F for ; Fri, 30 Jun 2023 05:15:34 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FPinwjazA0fWJJLj4TkaGXsVD1ozc7BkFGxW1ee/48PQs9kU1huQC9JuDIahv7nCR0jpAf3pdXr6MXoZE3gixqQJTkpXi5gLBS+Ft2wBsvhoaFVs3WoXR1qUtJb+gketqzwBYKMxYgGr7dBO4Q4lNV05lqY8CVNmgCQ1mTNsaIFq6VpVXFNrPNHhTSn3Jc3CSQh2e6f0n30PukulBEYiih30dTrb4GJ64L+aB3eDZ3tvdxI/B8FB4jqoN/XvkbtFEyAJiMl+mcvKWTZtFoVxXFtcCi2upAqq2bptZ9Eu1BRzWRsn+Mgm6XlTtx+yZRZY1Sd8RO6JmjsTiShWuuwu+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rGXT5dp55x53GCKMub9j80jYSeASZXKO0D8QUsjHpsQ=; b=iKlTjybaktAxv75NU+KBkfMFSz+km9PlV0fBctK4rpc5CpWnh7FvaO1Go4ULUajgH2GDcB98evanEBkjC2mFNN/QJWZ31muDeRhcEYsNA03p3kLsL2rBliffQNlBC1DBNWHth90eE9v7zvZ3dIIVxLV6INP/LzA3xeF5wuCgACU7p48AWxl8ewVbyKG4V8+/Ze5aPTE9a8T2xHsvFchZlEDA93Wq/XbqOMKzoApS2lT/syFjW8BgFD7tFh3h+EMqyJ/EieHt2W6UC1WorCzJnM+RucMSJk9EpulEiupJL10uGAk5Dz6aXVul3GiFGTiEnQlfXGp8bv1yg2wSiWp5Gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=jaguarmicro.com; dmarc=pass action=none header.from=jaguarmicro.com; dkim=pass header.d=jaguarmicro.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jaguarmicro.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rGXT5dp55x53GCKMub9j80jYSeASZXKO0D8QUsjHpsQ=; b=a3aFQiHq4XqZPadzbvLoVZFwp2xiapPxSopUeLqYxlaqJ551VJqk81Cme8nNqQ0l6oyS9OlRw0hwhiYDs5u7OGRwT0Qog7aTsIl1dL454prFxEMHtVPXKnLs9C7Bvs2XOjauu4pgaJxQI7rvw9LT9tuZOtnuVslovUKKDDHV9TOChMyFozo1Ed3ObP8+llVft9bC235wv763qOn6PYchlQdFaRa44ZAJbfza0rjLL0N9p4cno1dA+rR6vgNT2GUFZdUc8WA+GC/Q2HjYelnMK+c6ocm0SRqcsH1SNT5d4jgmiw9gBSj1Qoj1B/HQ5XasZMxYx4z4DtSDDDitlvzEjQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=jaguarmicro.com; Received: from SI2PR06MB4752.apcprd06.prod.outlook.com (2603:1096:4:14c::14) by TYZPR06MB5686.apcprd06.prod.outlook.com (2603:1096:400:282::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.23; Fri, 30 Jun 2023 03:15:30 +0000 Received: from SI2PR06MB4752.apcprd06.prod.outlook.com ([fe80::25f2:1ee4:9de:7b00]) by SI2PR06MB4752.apcprd06.prod.outlook.com ([fe80::25f2:1ee4:9de:7b00%3]) with mapi id 15.20.6521.026; Fri, 30 Jun 2023 03:15:30 +0000 From: Rma Ma To: dpdk-dev Cc: Maxime Coquelin , Chenbo Xia , Rma Ma Subject: [PATCH] vhost: add notify reply ops to avoid message deadlock Date: Fri, 30 Jun 2023 11:15:19 +0800 Message-Id: <20230630031519.1807-1-rma.ma@jaguarmicro.com> X-Mailer: git-send-email 2.17.1 X-ClientProxiedBy: SG2PR06CA0203.apcprd06.prod.outlook.com (2603:1096:4:1::35) To SI2PR06MB4752.apcprd06.prod.outlook.com (2603:1096:4:14c::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SI2PR06MB4752:EE_|TYZPR06MB5686:EE_ X-MS-Office365-Filtering-Correlation-Id: 9c591850-40d1-4baa-3bef-08db79184266 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 98Q8i7hmBppfHj9NzRlGX440OyZSEVto5n+sSoTwxKGr2ytV73x3VrUJgWfU797Z0J/e8bK+wY4TD4Pis2uz8wWlwS/9TsOBacQVfA1Kq6HyB/OWgLdAVXUD7HHJN+uxh7gersi2DNqEQNURmbv9VncMTwZfQFk8Rq8xAWCiU2idDj7heocZ6XRokwdjq4ihBQjU5svyUL3VBfLTz+1SGm5WmRV5BnxNqnZeLs21GkvClRO1ywmhIPg8UTHsoPb/pkss2VhtuSwD6z/gbEGTLSB1SMOazkQUzwcxKEFz/SzrDapOciAQHSujXMQuTL5UljZKPmyFrbm9D9FNnusJLPBP6ZWo02jtezeLZvLCLE1IYoEe+aTQEZ/Dfg5SYhwb7/sRafbCAs4BuB7y22HoJOVQ95uIKVZaip4b4D9jUNATTsd2OBsiqNGBc+JeaM/u0lYlV59DhACdkdkjsfm1uq3fsHK79HIKH46qRfTQQi1kmrtoSycUlbIo1jheGn3nLesIJ+5+K6Cp5fBbZJbTO1Rqyg4OBCY3CEyLW9P4UFZph3cYJYSHypYN/9A++1xiBK1urXBY7PiebMJGJkYlwUKVoCBR9rdcXHdEJ1kuvmx65zHZI4xKFGBFnmL6TxeH X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SI2PR06MB4752.apcprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(39830400003)(136003)(346002)(366004)(376002)(396003)(451199021)(8676002)(66476007)(66556008)(107886003)(41300700001)(66946007)(8936002)(316002)(6506007)(186003)(26005)(6916009)(1076003)(6512007)(54906003)(478600001)(6666004)(4326008)(2616005)(6486002)(52116002)(15650500001)(2906002)(5660300002)(44832011)(38350700002)(38100700002)(36756003)(86362001)(83380400001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 68++sTeKa8cX3mNIwID6+zGUxGNhlkiH8DNmAfdrOzqO7s3ejbAqSQHNQrdpcfXOsNKEGB2X8sjJdobCxqZp/tgGanbnfXOPAj6GtHSdAoL9P4yZJ8Y1/BeBIuUM7PSyGxCNZNFSyggxx2dKo2w0ZrHQbBunSp5/Bx6m2PQLotonUMoAPBN1Z3ll3c8VVyhNXvSdcnpkNTJ5MG4ho+mVtUn+R40B5dPvD/88r0IbFC72NZnZTRqIRR9PA8p2Tms9WSAhasAxncgJAfHop9yYBIDqsVUfoKAhR9U3znLmQwY5dK1HfDfMimZiSSQBzWWY0iMI3klqUmWF3UT4AICN/TyzhgDp6tGriwooEhzA3U4j7TMXHscPBNuglxZVnJ/OxLhkpvzvTAPmvwr/EQ33xx+ABTss550FtHMRKqT9lbT/uzsyXIliHWz1HxDd8OA6Wm1jJ/7oouvyJb62gl67z22ktc3n9hoanna8MTCKs/mIhlqcCjHuBnocbrETncFS8OnjdORAeTlP64i4ep/x8a3WlK1a29N+bDjEKUB1bQtBVv9dkacsxgLQA2OWvc1sJ0LUJq5Q6A5j47fy/agavis5qeiOWzaA5jvQzPJhp/FxGE6Ia7jtdgYNOk4wm7J3G36Ejx0q6g8iDC9ItvB9YgldjfP7AHk3NV442DUbqbaBPNWMDA41tBWMmbAFP1wp85lkheh+OZe30mccPR8/rKtk/8q+gqeM0Z/6zT7FKP33lETQZRWFAC/ToVo3N3XHt0YT2WmnVgc4M4BFiIVSWZEDCa3rrmnYRijhPwMgzS68lluzeJw1Vfm+6NGKONgubcIj0im7/AzN2VXPleQkqL7nBSP6Z1K9G38sBkg7zUkOrYciZ19eOKUGcDT4xYPfYBbYArPaAmxN+MYdj1JYYHAfXqsg9bxhCJwFulCJT3BEKQGYhansQg53IospyiOopJeRChMoQUgo5oU0bhdfT+b5VENI6ZT4x4gcd11YJrMv4yfXRpomMATSSBIMVdgxuTBpuOIHPTv/2Um6j0BsNOA0c/wTKNVRRU+1fXKDRH15Hl4fJynyq7gmSYNGG2jBoYPqtHP6ZWXEEVdBNNKFzI+jq4vo2RTa6KZ6IB3IB9eL3jYfbq1nzu3VvtY0o/Ylcmw+olC+ZK+0ahV48D4bys2kGXwW7WLtoXSvxzep+Xo2qP0qAf3v2nMcqCukiTM0EWnc2mRFuxAllgDQiqWd4EF0ZXSwUPeLb8Swc+XV17xKbxXa2i0Z/4STE1NgwAo6PMMTPIluocoIHlO796iFuU5EsxPgoL649+btn2VIzBMxPzbSkOjC2XkzROr51S8NMP445a2VoV4IAer4EMwspUpHvpF3ijwsWSanfTnWGzSf9P1+vVLNu1WFPz+V1KfY9gsaXcl44uZ4tKreO0Ru7dK6co2bKw448MtJ+EhyR10bx/t03D+bYWoM107nvi2bpyA8gEgjbPDh0pT7hZzesAwR9nOoo+ZZmmFYqE3SN32PN5WkPPbFTL8KIiiWZ6QEeFpNxWt880E26eDHa60x+FH4Pe/hx1DPmGKHwlO0vM9XrscGWlJXRRvSfmE0kB0E X-OriginatorOrg: jaguarmicro.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9c591850-40d1-4baa-3bef-08db79184266 X-MS-Exchange-CrossTenant-AuthSource: SI2PR06MB4752.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2023 03:15:30.4829 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 1e45a5c2-d3e1-46b3-a0e6-c5ebf6d8ba7b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0A3961c2uN63glKT3viHa2yBtNhmvNCNBzYWcwIAknRCiAcw9zERNr982ke1az7k5ai3uKmn4DbsC0YPvjVyXg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZPR06MB5686 X-Mailman-Approved-At: Fri, 30 Jun 2023 09:15:24 +0200 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Since backend and frontend message are synchronous in the same thread, there will be a probability of message deadlock. Consider each driver to determine whether to wait for response. Signed-off-by: Rma Ma --- lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/vhost_user.c | 23 ++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index 8db4ab9f4d..3d2ea3c90e 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -81,6 +81,9 @@ struct rte_vdpa_dev_ops { /** get device type: net device, blk device... */ int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type); + + /** Get the notify reply flag */ + int (*get_notify_reply_flag)(int vid, bool *need_reply); }; /** diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 901a80bbaa..aa61992939 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3365,13 +3365,14 @@ rte_vhost_backend_config_change(int vid, bool need_reply) static int vhost_user_backend_set_vring_host_notifier(struct virtio_net *dev, int index, int fd, uint64_t offset, - uint64_t size) + uint64_t size, + bool need_reply) { int ret; struct vhu_msg_context ctx = { .msg = { .request.backend = VHOST_USER_BACKEND_VRING_HOST_NOTIFIER_MSG, - .flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY, + .flags = VHOST_USER_VERSION, .size = sizeof(ctx.msg.payload.area), .payload.area = { .u64 = index & VHOST_USER_VRING_IDX_MASK, @@ -3388,7 +3389,13 @@ static int vhost_user_backend_set_vring_host_notifier(struct virtio_net *dev, ctx.fd_num = 1; } - ret = send_vhost_backend_message_process_reply(dev, &ctx); + if (!need_reply) + ret = send_vhost_backend_message(dev, &ctx); + else { + ctx.msg.flags |= VHOST_USER_NEED_REPLY; + ret = send_vhost_backend_message_process_reply(dev, &ctx); + } + if (ret < 0) VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to set host notifier (%d)\n", ret); @@ -3402,6 +3409,7 @@ int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable) int vfio_device_fd, ret = 0; uint64_t offset, size; unsigned int i, q_start, q_last; + bool need_reply; dev = get_device(vid); if (!dev) @@ -3440,6 +3448,11 @@ int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable) if (vfio_device_fd < 0) return -ENOTSUP; + if (vdpa_dev->ops->get_notify_reply_flag == NULL) + need_reply = true; + else + vdpa_dev->ops->get_notify_reply_flag(vid, &need_reply); + if (enable) { for (i = q_start; i <= q_last; i++) { if (vdpa_dev->ops->get_notify_area(vid, i, &offset, @@ -3449,7 +3462,7 @@ int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable) } if (vhost_user_backend_set_vring_host_notifier(dev, i, - vfio_device_fd, offset, size) < 0) { + vfio_device_fd, offset, size, need_reply) < 0) { ret = -EFAULT; goto disable; } @@ -3458,7 +3471,7 @@ int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable) disable: for (i = q_start; i <= q_last; i++) { vhost_user_backend_set_vring_host_notifier(dev, i, -1, - 0, 0); + 0, 0, need_reply); } }