diff mbox series

[v3] vhost: fix deadlock when message handling failed

Message ID 20220507132753.11357-1-wenwux.ma@intel.com (mailing list archive)
State Accepted, archived
Delegated to: Maxime Coquelin
Headers
Series [v3] vhost: fix deadlock when message handling failed |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/github-robot: build success github build: passed
ci/iol-mellanox-Performance success Performance Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/iol-aarch64-unit-testing success Testing PASS
ci/iol-aarch64-compile-testing success Testing PASS
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS
ci/iol-x86_64-unit-testing success Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-x86_64-compile-testing success Testing PASS

Commit Message

Ma, WenwuX May 7, 2022, 1:27 p.m. UTC 
  In vhost_user_msg_handler(), if vhost message handling
failed, we should check whether the queue is locked and
release the lock before returning. Or, it will cause a
deadlock later.

Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness notification")
Cc: stable@dpdk.org

Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
---
 lib/vhost/vhost_user.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

Comments

Chenbo Xia May 7, 2022, 6:02 a.m. UTC | #1 
> -----Original Message-----
> From: Ma, WenwuX <wenwux.ma@intel.com>
> Sent: Saturday, May 7, 2022 9:28 PM
> To: maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>;
> dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>; He,
> Xingguang <xingguang.he@intel.com>; Ma, WenwuX <wenwux.ma@intel.com>;
> stable@dpdk.org
> Subject: [PATCH v3] vhost: fix deadlock when message handling failed
> 
> In vhost_user_msg_handler(), if vhost message handling
> failed, we should check whether the queue is locked and
> release the lock before returning. Or, it will cause a
> deadlock later.
> 
> Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness notification")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> ---
>  lib/vhost/vhost_user.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 1d390677fa..4baf969ee0 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -2976,7 +2976,6 @@ vhost_user_msg_handler(int vid, int fd)
>  		return -1;
>  	}
> 
> -	ret = 0;
>  	request = ctx.msg.request.master;
>  	if (request > VHOST_USER_NONE && request < VHOST_USER_MAX &&
>  			vhost_message_str[request]) {
> @@ -3113,9 +3112,11 @@ vhost_user_msg_handler(int vid, int fd)
>  		send_vhost_reply(dev, fd, &ctx);
>  	} else if (ret == RTE_VHOST_MSG_RESULT_ERR) {
>  		VHOST_LOG_CONFIG(ERR, "(%s) vhost message handling failed.\n",
> dev->ifname);
> -		return -1;
> +		ret = -1;
> +		goto unlock;
>  	}
> 
> +	ret = 0;
>  	for (i = 0; i < dev->nr_vring; i++) {
>  		struct vhost_virtqueue *vq = dev->virtqueue[i];
>  		bool cur_ready = vq_is_ready(dev, vq);
> @@ -3126,10 +3127,11 @@ vhost_user_msg_handler(int vid, int fd)
>  		}
>  	}
> 
> +unlock:
>  	if (unlock_required)
>  		vhost_user_unlock_all_queue_pairs(dev);
> 
> -	if (!virtio_is_ready(dev))
> +	if (ret != 0 || !virtio_is_ready(dev))
>  		goto out;
> 
>  	/*
> @@ -3156,7 +3158,7 @@ vhost_user_msg_handler(int vid, int fd)
>  	}
> 
>  out:
> -	return 0;
> +	return ret;
>  }
> 
>  static int process_slave_message_reply(struct virtio_net *dev,
> --
> 2.25.1

Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
Ling, WeiX May 9, 2022, 3:13 a.m. UTC | #2 
> -----Original Message-----
> From: Wenwu Ma <wenwux.ma@intel.com>
> Sent: Saturday, May 7, 2022 9:28 PM
> To: maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>;
> dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>;
> He, Xingguang <xingguang.he@intel.com>; Ma, WenwuX
> <wenwux.ma@intel.com>; stable@dpdk.org
> Subject: [PATCH v3] vhost: fix deadlock when message handling failed
> 

Tested-by: Wei Ling <weix.ling@intel.com>
David Marchand May 17, 2022, 1:15 p.m. UTC | #3 
On Sat, May 7, 2022 at 7:30 AM Wenwu Ma <wenwux.ma@intel.com> wrote:
>
> In vhost_user_msg_handler(), if vhost message handling
> failed, we should check whether the queue is locked and
> release the lock before returning. Or, it will cause a
> deadlock later.
>
> Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness notification")
> Cc: stable@dpdk.org
>
> Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>

Thanks, lgtm.
Acked-by: David Marchand <david.marchand@redhat.com>
Maxime Coquelin May 17, 2022, 1:24 p.m. UTC | #4 
On 5/7/22 15:27, Wenwu Ma wrote:
> In vhost_user_msg_handler(), if vhost message handling
> failed, we should check whether the queue is locked and
> release the lock before returning. Or, it will cause a
> deadlock later.
> 
> Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness notification")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> ---
>   lib/vhost/vhost_user.c | 10 ++++++----
>   1 file changed, 6 insertions(+), 4 deletions(-)
> 

Applied to dpdk-next-virtio/main.

Thanks,
Maxime
Pei, Andy July 11, 2022, 8:42 a.m. UTC | #5 
HI ALL,

I see that in function vhost_user_msg_handler.
We use "ret" to store both vhost msg return code like "RTE_VHOST_MSG_RESULT_XXX" and function return value.
I wonder if it is better to use two different variable to make it easy to read.

> -----Original Message-----
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> Sent: Tuesday, May 17, 2022 9:24 PM
> To: Ma, WenwuX <WenwuX.Ma@intel.com>; Xia, Chenbo
> <Chenbo.Xia@intel.com>; dev@dpdk.org
> Cc: Hu, Jiayu <Jiayu.Hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>; He,
> Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> Subject: Re: [PATCH v3] vhost: fix deadlock when message handling failed
> 
> 
> 
> On 5/7/22 15:27, Wenwu Ma wrote:
> > In vhost_user_msg_handler(), if vhost message handling failed, we
> > should check whether the queue is locked and release the lock before
> > returning. Or, it will cause a deadlock later.
> >
> > Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness
> > notification")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> > ---
> >   lib/vhost/vhost_user.c | 10 ++++++----
> >   1 file changed, 6 insertions(+), 4 deletions(-)
> >
> 
> Applied to dpdk-next-virtio/main.
> 
> Thanks,
> Maxime
Chenbo Xia July 11, 2022, 8:54 a.m. UTC | #6 
> -----Original Message-----
> From: Pei, Andy <andy.pei@intel.com>
> Sent: Monday, July 11, 2022 4:42 PM
> To: Maxime Coquelin <maxime.coquelin@redhat.com>; Ma, WenwuX
> <wenwux.ma@intel.com>; Xia, Chenbo <chenbo.xia@intel.com>; dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>; He,
> Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> Subject: RE: [PATCH v3] vhost: fix deadlock when message handling failed
> 
> HI ALL,
> 
> I see that in function vhost_user_msg_handler.
> We use "ret" to store both vhost msg return code like
> "RTE_VHOST_MSG_RESULT_XXX" and function return value.
> I wonder if it is better to use two different variable to make it easy to
> read.

By saying 'function return value', you mean which function? Can you
elaborate more?

Thanks,
Chenbo

> 
> > -----Original Message-----
> > From: Maxime Coquelin <maxime.coquelin@redhat.com>
> > Sent: Tuesday, May 17, 2022 9:24 PM
> > To: Ma, WenwuX <WenwuX.Ma@intel.com>; Xia, Chenbo
> > <Chenbo.Xia@intel.com>; dev@dpdk.org
> > Cc: Hu, Jiayu <Jiayu.Hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>;
> He,
> > Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> > Subject: Re: [PATCH v3] vhost: fix deadlock when message handling failed
> >
> >
> >
> > On 5/7/22 15:27, Wenwu Ma wrote:
> > > In vhost_user_msg_handler(), if vhost message handling failed, we
> > > should check whether the queue is locked and release the lock before
> > > returning. Or, it will cause a deadlock later.
> > >
> > > Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness
> > > notification")
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> > > ---
> > >   lib/vhost/vhost_user.c | 10 ++++++----
> > >   1 file changed, 6 insertions(+), 4 deletions(-)
> > >
> >
> > Applied to dpdk-next-virtio/main.
> >
> > Thanks,
> > Maxime
Pei, Andy July 11, 2022, 9:02 a.m. UTC | #7 
HI Chenbo,

See below example

1.
	ret = read_vhost_message(dev, fd, &ctx);
	if (ret <= 0) {
		if (ret < 0)
			VHOST_LOG_CONFIG(dev->ifname, ERR, "vhost read message failed\n");
		else
			VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer closed\n");

		return -1;
	}

2.
	ret = vhost_user_check_and_alloc_queue_pair(dev, &ctx);
	if (ret < 0) {
		VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to alloc queue\n");
		return -1;
	}


3. ret = -1  
	} else if (ret == RTE_VHOST_MSG_RESULT_ERR) {
		VHOST_LOG_CONFIG(dev->ifname, ERR, "vhost message handling failed.\n");
		ret = -1;
		goto unlock;
	}

I mean ret is used to check the function is returned successfully or not while it is also used to store vhost msg return code like "RTE_VHOST_MSG_RESULT_XXX".




> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Monday, July 11, 2022 4:54 PM
> To: Pei, Andy <andy.pei@intel.com>; Maxime Coquelin
> <maxime.coquelin@redhat.com>; Ma, WenwuX <wenwux.ma@intel.com>;
> dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>; He,
> Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> Subject: RE: [PATCH v3] vhost: fix deadlock when message handling failed
> 
> > -----Original Message-----
> > From: Pei, Andy <andy.pei@intel.com>
> > Sent: Monday, July 11, 2022 4:42 PM
> > To: Maxime Coquelin <maxime.coquelin@redhat.com>; Ma, WenwuX
> > <wenwux.ma@intel.com>; Xia, Chenbo <chenbo.xia@intel.com>;
> > dev@dpdk.org
> > Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan
> > <yinan.wang@intel.com>; He, Xingguang <xingguang.he@intel.com>;
> > stable@dpdk.org
> > Subject: RE: [PATCH v3] vhost: fix deadlock when message handling
> > failed
> >
> > HI ALL,
> >
> > I see that in function vhost_user_msg_handler.
> > We use "ret" to store both vhost msg return code like
> > "RTE_VHOST_MSG_RESULT_XXX" and function return value.
> > I wonder if it is better to use two different variable to make it easy
> > to read.
> 
> By saying 'function return value', you mean which function? Can you elaborate
> more?
> 
> Thanks,
> Chenbo
> 
> >
> > > -----Original Message-----
> > > From: Maxime Coquelin <maxime.coquelin@redhat.com>
> > > Sent: Tuesday, May 17, 2022 9:24 PM
> > > To: Ma, WenwuX <WenwuX.Ma@intel.com>; Xia, Chenbo
> > > <Chenbo.Xia@intel.com>; dev@dpdk.org
> > > Cc: Hu, Jiayu <Jiayu.Hu@intel.com>; Wang, Yinan
> > > <yinan.wang@intel.com>;
> > He,
> > > Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> > > Subject: Re: [PATCH v3] vhost: fix deadlock when message handling
> > > failed
> > >
> > >
> > >
> > > On 5/7/22 15:27, Wenwu Ma wrote:
> > > > In vhost_user_msg_handler(), if vhost message handling failed, we
> > > > should check whether the queue is locked and release the lock
> > > > before returning. Or, it will cause a deadlock later.
> > > >
> > > > Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness
> > > > notification")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> > > > ---
> > > >   lib/vhost/vhost_user.c | 10 ++++++----
> > > >   1 file changed, 6 insertions(+), 4 deletions(-)
> > > >
> > >
> > > Applied to dpdk-next-virtio/main.
> > >
> > > Thanks,
> > > Maxime
Chenbo Xia July 11, 2022, 9:14 a.m. UTC | #8 
Hi Andy,

> -----Original Message-----
> From: Pei, Andy <andy.pei@intel.com>
> Sent: Monday, July 11, 2022 5:03 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin
> <maxime.coquelin@redhat.com>; Ma, WenwuX <wenwux.ma@intel.com>;
> dev@dpdk.org
> Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>; He,
> Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> Subject: RE: [PATCH v3] vhost: fix deadlock when message handling failed
> 
> HI Chenbo,
> 
> See below example
> 
> 1.
> 	ret = read_vhost_message(dev, fd, &ctx);
> 	if (ret <= 0) {
> 		if (ret < 0)
> 			VHOST_LOG_CONFIG(dev->ifname, ERR, "vhost read message
> failed\n");
> 		else
> 			VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer
> closed\n");
> 
> 		return -1;
> 	}
> 
> 2.
> 	ret = vhost_user_check_and_alloc_queue_pair(dev, &ctx);
> 	if (ret < 0) {
> 		VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to alloc queue\n");
> 		return -1;
> 	}
> 
> 
> 3. ret = -1
> 	} else if (ret == RTE_VHOST_MSG_RESULT_ERR) {
> 		VHOST_LOG_CONFIG(dev->ifname, ERR, "vhost message handling
> failed.\n");
> 		ret = -1;
> 		goto unlock;
> 	}
> 
> I mean ret is used to check the function is returned successfully or not
> while it is also used to store vhost msg return code like
> "RTE_VHOST_MSG_RESULT_XXX".

It makes sense to make it two, one int for return value, one enum
rte_vhost_msg_result only for message handling. IMHO, this could improve 
readability.

Thanks,
Chenbo

> 
> 
> 
> 
> > -----Original Message-----
> > From: Xia, Chenbo <chenbo.xia@intel.com>
> > Sent: Monday, July 11, 2022 4:54 PM
> > To: Pei, Andy <andy.pei@intel.com>; Maxime Coquelin
> > <maxime.coquelin@redhat.com>; Ma, WenwuX <wenwux.ma@intel.com>;
> > dev@dpdk.org
> > Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan <yinan.wang@intel.com>;
> He,
> > Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> > Subject: RE: [PATCH v3] vhost: fix deadlock when message handling failed
> >
> > > -----Original Message-----
> > > From: Pei, Andy <andy.pei@intel.com>
> > > Sent: Monday, July 11, 2022 4:42 PM
> > > To: Maxime Coquelin <maxime.coquelin@redhat.com>; Ma, WenwuX
> > > <wenwux.ma@intel.com>; Xia, Chenbo <chenbo.xia@intel.com>;
> > > dev@dpdk.org
> > > Cc: Hu, Jiayu <jiayu.hu@intel.com>; Wang, Yinan
> > > <yinan.wang@intel.com>; He, Xingguang <xingguang.he@intel.com>;
> > > stable@dpdk.org
> > > Subject: RE: [PATCH v3] vhost: fix deadlock when message handling
> > > failed
> > >
> > > HI ALL,
> > >
> > > I see that in function vhost_user_msg_handler.
> > > We use "ret" to store both vhost msg return code like
> > > "RTE_VHOST_MSG_RESULT_XXX" and function return value.
> > > I wonder if it is better to use two different variable to make it easy
> > > to read.
> >
> > By saying 'function return value', you mean which function? Can you
> elaborate
> > more?
> >
> > Thanks,
> > Chenbo
> >
> > >
> > > > -----Original Message-----
> > > > From: Maxime Coquelin <maxime.coquelin@redhat.com>
> > > > Sent: Tuesday, May 17, 2022 9:24 PM
> > > > To: Ma, WenwuX <WenwuX.Ma@intel.com>; Xia, Chenbo
> > > > <Chenbo.Xia@intel.com>; dev@dpdk.org
> > > > Cc: Hu, Jiayu <Jiayu.Hu@intel.com>; Wang, Yinan
> > > > <yinan.wang@intel.com>;
> > > He,
> > > > Xingguang <xingguang.he@intel.com>; stable@dpdk.org
> > > > Subject: Re: [PATCH v3] vhost: fix deadlock when message handling
> > > > failed
> > > >
> > > >
> > > >
> > > > On 5/7/22 15:27, Wenwu Ma wrote:
> > > > > In vhost_user_msg_handler(), if vhost message handling failed, we
> > > > > should check whether the queue is locked and release the lock
> > > > > before returning. Or, it will cause a deadlock later.
> > > > >
> > > > > Fixes: 7f31d4ea05ca ("vhost: fix lock on device readiness
> > > > > notification")
> > > > > Cc: stable@dpdk.org
> > > > >
> > > > > Signed-off-by: Wenwu Ma <wenwux.ma@intel.com>
> > > > > ---
> > > > >   lib/vhost/vhost_user.c | 10 ++++++----
> > > > >   1 file changed, 6 insertions(+), 4 deletions(-)
> > > > >
> > > >
> > > > Applied to dpdk-next-virtio/main.
> > > >
> > > > Thanks,
> > > > Maxime
diff mbox series

Patch

diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 1d390677fa..4baf969ee0 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2976,7 +2976,6 @@  vhost_user_msg_handler(int vid, int fd)
 		return -1;
 	}
 
-	ret = 0;
 	request = ctx.msg.request.master;
 	if (request > VHOST_USER_NONE && request < VHOST_USER_MAX &&
 			vhost_message_str[request]) {
@@ -3113,9 +3112,11 @@  vhost_user_msg_handler(int vid, int fd)
 		send_vhost_reply(dev, fd, &ctx);
 	} else if (ret == RTE_VHOST_MSG_RESULT_ERR) {
 		VHOST_LOG_CONFIG(ERR, "(%s) vhost message handling failed.\n", dev->ifname);
-		return -1;
+		ret = -1;
+		goto unlock;
 	}
 
+	ret = 0;
 	for (i = 0; i < dev->nr_vring; i++) {
 		struct vhost_virtqueue *vq = dev->virtqueue[i];
 		bool cur_ready = vq_is_ready(dev, vq);
@@ -3126,10 +3127,11 @@  vhost_user_msg_handler(int vid, int fd)
 		}
 	}
 
+unlock:
 	if (unlock_required)
 		vhost_user_unlock_all_queue_pairs(dev);
 
-	if (!virtio_is_ready(dev))
+	if (ret != 0 || !virtio_is_ready(dev))
 		goto out;
 
 	/*
@@ -3156,7 +3158,7 @@  vhost_user_msg_handler(int vid, int fd)
 	}
 
 out:
-	return 0;
+	return ret;
 }
 
 static int process_slave_message_reply(struct virtio_net *dev,