Message ID | 20210804083128.64981-1-zhihongx.peng@intel.com (mailing list archive) |
---|---|
State | Changes Requested, archived |
Delegated to: | Maxime Coquelin |
Headers | show |
Series | net/virtio: fix check scatter on all Rx queues | expand |
Context | Check | Description |
---|---|---|
ci/iol-x86_64-compile-testing | fail | Testing issues |
ci/iol-abi-testing | success | Testing PASS |
ci/iol-x86_64-unit-testing | success | Testing PASS |
ci/iol-mellanox-Performance | success | Performance Testing PASS |
ci/iol-aarch64-compile-testing | success | Testing PASS |
ci/iol-aarch64-unit-testing | success | Testing PASS |
ci/iol-intel-Performance | success | Performance Testing PASS |
ci/iol-intel-Functional | success | Functional Testing PASS |
ci/iol-broadcom-Functional | success | Functional Testing PASS |
ci/iol-broadcom-Performance | success | Performance Testing PASS |
ci/intel-Testing | success | Testing PASS |
ci/Intel-compilation | success | Compilation OK |
ci/github-robot | success | github build: passed |
ci/checkpatch | success | coding style OK |
Hi Zhihong, On 8/4/21 10:31 AM, zhihongx.peng@intel.com wrote: > From: Zhihong Peng <zhihongx.peng@intel.com> > > This patch fixes the wrong way to obtain virtqueue. > The end of virtqueue cannot be judged based on whether > the array is NULL. My understanding is that it is causing issue because it is confusing the control queue with a Rx queue? I so, please be more specific on the issue it is fixing in the commit message. > Fixes: 4e8169eb0d2d (net/virtio: fix Rx scatter offload) > Cc: stable@dpdk.org > > Signed-off-by: Zhihong Peng <zhihongx.peng@intel.com> > --- > drivers/net/virtio/virtio_ethdev.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c > index e58085a2c9..f2d19dc9d6 100644 > --- a/drivers/net/virtio/virtio_ethdev.c > +++ b/drivers/net/virtio/virtio_ethdev.c > @@ -873,8 +873,8 @@ virtio_check_scatter_on_all_rx_queues(struct rte_eth_dev *dev, > if (hw->vqs == NULL) > return true; > > - for (qidx = 0; (vq = hw->vqs[2 * qidx + VTNET_SQ_RQ_QUEUE_IDX]) != NULL; > - qidx++) { > + for (qidx = 0; qidx < hw->max_queue_pairs; qidx++) { > + vq = hw->vqs[2 * qidx + VTNET_SQ_RQ_QUEUE_IDX]; I agree with the change, but I would add a check to ensure vq is not NULL to be safe wrt to NULL pointer dereferencing. > rxvq = &vq->rxq; > if (rxvq->mpool == NULL) > continue; > Thanks, Maxime
On Wed, Aug 4, 2021 at 10:36 AM <zhihongx.peng@intel.com> wrote: > > From: Zhihong Peng <zhihongx.peng@intel.com> > > This patch fixes the wrong way to obtain virtqueue. > The end of virtqueue cannot be judged based on whether > the array is NULL. Indeed, good catch. I can reproduce a crash with v20.11.3 which has backport of 4e8169eb0d2d. I can not see it with main: maybe due to a lucky allocation or size requested to rte_zmalloc... ? The usecase is simple, I am surprised no validation caught it. # gdb ./build/app/dpdk-testpmd -ex 'run --vdev net_virtio_user0,path=/dev/vhost-net,iface=titi,queues=3 -a 0:0:0.0 -- -i' ... Thread 1 "dpdk-testpmd" received signal SIGSEGV, Segmentation fault. virtio_rx_mem_pool_buf_size (mp=0x110429983) at ../drivers/net/virtio/virtio_ethdev.c:873 873 return rte_pktmbuf_data_room_size(mp) - RTE_PKTMBUF_HEADROOM; Missing separate debuginfos, use: yum debuginfo-install elfutils-libelf-0.182-3.el8.x86_64 libbpf-0.2.0-1.el8.x86_64 (gdb) bt #0 virtio_rx_mem_pool_buf_size (mp=0x110429983) at ../drivers/net/virtio/virtio_ethdev.c:873 #1 0x0000000000e370d1 in virtio_check_scatter_on_all_rx_queues (frame_size=1530, dev=0x1799a40 <rte_eth_devices>) at ../drivers/net/virtio/virtio_ethdev.c:907 #2 virtio_mtu_set (dev=0x1799a40 <rte_eth_devices>, mtu=<optimized out>) at ../drivers/net/virtio/virtio_ethdev.c:938 #3 0x00000000008c30e5 in rte_eth_dev_set_mtu (port_id=port_id@entry=0, mtu=<optimized out>) at ../lib/librte_ethdev/rte_ethdev.c:3484 #4 0x00000000006a61d8 in update_jumbo_frame_offload (portid=portid@entry=0) at ../app/test-pmd/testpmd.c:3371 #5 0x00000000006a62bc in init_config_port_offloads (pid=0, socket_id=0) at ../app/test-pmd/testpmd.c:1416 #6 0x000000000061770c in init_config () at ../app/test-pmd/testpmd.c:1505 #7 main (argc=<optimized out>, argv=<optimized out>) at ../app/test-pmd/testpmd.c:3800 (gdb) f 1 #1 0x0000000000e370d1 in virtio_check_scatter_on_all_rx_queues (frame_size=1530, dev=0x1799a40 <rte_eth_devices>) at ../drivers/net/virtio/virtio_ethdev.c:907 907 buf_size = virtio_rx_mem_pool_buf_size(rxvq->mpool); (gdb) p hw->max_queue_pairs $1 = 3 (gdb) p qidx $2 = 5 (gdb) p hw->vqs[0] $3 = (struct virtqueue *) 0x17ffb03c0 (gdb) p hw->vqs[2] $4 = (struct virtqueue *) 0x17ff9dcc0 (gdb) p hw->vqs[4] $5 = (struct virtqueue *) 0x17ff8acc0 (gdb) p hw->vqs[6] $6 = (struct virtqueue *) 0x17ff77cc0 (gdb) p hw->vqs[7] $7 = (struct virtqueue *) 0x0 (gdb) p hw->vqs[8] $8 = (struct virtqueue *) 0x100004ac0 (gdb) p hw->vqs[9] $9 = (struct virtqueue *) 0x17ffb1600 (gdb) p hw->vqs[10] $10 = (struct virtqueue *) 0x17ffb18c0
On 15/09/2021 19:37, David Marchand wrote: > On Wed, Aug 4, 2021 at 10:36 AM <zhihongx.peng@intel.com> wrote: >> >> From: Zhihong Peng <zhihongx.peng@intel.com> >> >> This patch fixes the wrong way to obtain virtqueue. >> The end of virtqueue cannot be judged based on whether >> the array is NULL. > > Indeed, good catch. > > I can reproduce a crash with v20.11.3 which has backport of 4e8169eb0d2d. > I can not see it with main: maybe due to a lucky allocation or size > requested to rte_zmalloc... ? > > The usecase is simple, I am surprised no validation caught it. > > # gdb ./build/app/dpdk-testpmd -ex 'run --vdev > net_virtio_user0,path=/dev/vhost-net,iface=titi,queues=3 -a 0:0:0.0 -- > -i' > > ... > > Thread 1 "dpdk-testpmd" received signal SIGSEGV, Segmentation fault. > virtio_rx_mem_pool_buf_size (mp=0x110429983) at > ../drivers/net/virtio/virtio_ethdev.c:873 > 873 return rte_pktmbuf_data_room_size(mp) - RTE_PKTMBUF_HEADROOM; > Missing separate debuginfos, use: yum debuginfo-install > elfutils-libelf-0.182-3.el8.x86_64 libbpf-0.2.0-1.el8.x86_64 > (gdb) bt > #0 virtio_rx_mem_pool_buf_size (mp=0x110429983) at > ../drivers/net/virtio/virtio_ethdev.c:873 > #1 0x0000000000e370d1 in virtio_check_scatter_on_all_rx_queues > (frame_size=1530, dev=0x1799a40 <rte_eth_devices>) at > ../drivers/net/virtio/virtio_ethdev.c:907 > #2 virtio_mtu_set (dev=0x1799a40 <rte_eth_devices>, mtu=<optimized > out>) at ../drivers/net/virtio/virtio_ethdev.c:938 > #3 0x00000000008c30e5 in rte_eth_dev_set_mtu > (port_id=port_id@entry=0, mtu=<optimized out>) at > ../lib/librte_ethdev/rte_ethdev.c:3484 > #4 0x00000000006a61d8 in update_jumbo_frame_offload > (portid=portid@entry=0) at ../app/test-pmd/testpmd.c:3371 > #5 0x00000000006a62bc in init_config_port_offloads (pid=0, > socket_id=0) at ../app/test-pmd/testpmd.c:1416 > #6 0x000000000061770c in init_config () at ../app/test-pmd/testpmd.c:1505 > #7 main (argc=<optimized out>, argv=<optimized out>) at > ../app/test-pmd/testpmd.c:3800 > (gdb) f 1 > #1 0x0000000000e370d1 in virtio_check_scatter_on_all_rx_queues > (frame_size=1530, dev=0x1799a40 <rte_eth_devices>) at > ../drivers/net/virtio/virtio_ethdev.c:907 > 907 buf_size = virtio_rx_mem_pool_buf_size(rxvq->mpool); > (gdb) p hw->max_queue_pairs > $1 = 3 > (gdb) p qidx > $2 = 5 > (gdb) p hw->vqs[0] > $3 = (struct virtqueue *) 0x17ffb03c0 > (gdb) p hw->vqs[2] > $4 = (struct virtqueue *) 0x17ff9dcc0 > (gdb) p hw->vqs[4] > $5 = (struct virtqueue *) 0x17ff8acc0 > (gdb) p hw->vqs[6] > $6 = (struct virtqueue *) 0x17ff77cc0 > (gdb) p hw->vqs[7] > $7 = (struct virtqueue *) 0x0 > (gdb) p hw->vqs[8] > $8 = (struct virtqueue *) 0x100004ac0 > (gdb) p hw->vqs[9] > $9 = (struct virtqueue *) 0x17ffb1600 > (gdb) p hw->vqs[10] > $10 = (struct virtqueue *) 0x17ffb18c0 > > > For reference, also observed when 20.11.3 is paired with OVS https://mail.openvswitch.org/pipermail/ovs-dev/2021-September/387940.html
> -----Original Message----- > From: Kevin Traynor <ktraynor@redhat.com> > Sent: Wednesday, September 22, 2021 1:45 AM > To: David Marchand <david.marchand@redhat.com>; Peng, ZhihongX > <zhihongx.peng@intel.com> > Cc: Xia, Chenbo <chenbo.xia@intel.com>; Maxime Coquelin > <maxime.coquelin@redhat.com>; dev <dev@dpdk.org>; Ivan Ilchenko > <ivan.ilchenko@oktetlabs.ru>; dpdk stable <stable@dpdk.org>; Christian > Ehrhardt <christian.ehrhardt@canonical.com>; Xueming(Steven) Li > <xuemingl@nvidia.com>; Luca Boccassi <bluca@debian.org> > Subject: Re: [dpdk-stable] [DPDK] net/virtio: fix check scatter on all Rx > queues > > On 15/09/2021 19:37, David Marchand wrote: > > On Wed, Aug 4, 2021 at 10:36 AM <zhihongx.peng@intel.com> wrote: > >> > >> From: Zhihong Peng <zhihongx.peng@intel.com> > >> > >> This patch fixes the wrong way to obtain virtqueue. > >> The end of virtqueue cannot be judged based on whether the array is > >> NULL. > > > > Indeed, good catch. > > > > I can reproduce a crash with v20.11.3 which has backport of 4e8169eb0d2d. > > I can not see it with main: maybe due to a lucky allocation or size > > requested to rte_zmalloc... ? > > This problem was discovered through google asan, we have submitted DPDK ASan patch. http://patchwork.dpdk.org/project/dpdk/patch/20210918074155.872358-1-zhihongx.peng@intel.com/ > > The usecase is simple, I am surprised no validation caught it. > > > > # gdb ./build/app/dpdk-testpmd -ex 'run --vdev > > net_virtio_user0,path=/dev/vhost-net,iface=titi,queues=3 -a 0:0:0.0 -- > > -i' > > > > ... > > > > Thread 1 "dpdk-testpmd" received signal SIGSEGV, Segmentation fault. > > virtio_rx_mem_pool_buf_size (mp=0x110429983) at > > ../drivers/net/virtio/virtio_ethdev.c:873 > > 873 return rte_pktmbuf_data_room_size(mp) - > RTE_PKTMBUF_HEADROOM; > > Missing separate debuginfos, use: yum debuginfo-install > > elfutils-libelf-0.182-3.el8.x86_64 libbpf-0.2.0-1.el8.x86_64 > > (gdb) bt > > #0 virtio_rx_mem_pool_buf_size (mp=0x110429983) at > > ../drivers/net/virtio/virtio_ethdev.c:873 > > #1 0x0000000000e370d1 in virtio_check_scatter_on_all_rx_queues > > (frame_size=1530, dev=0x1799a40 <rte_eth_devices>) at > > ../drivers/net/virtio/virtio_ethdev.c:907 > > #2 virtio_mtu_set (dev=0x1799a40 <rte_eth_devices>, mtu=<optimized > > out>) at ../drivers/net/virtio/virtio_ethdev.c:938 > > #3 0x00000000008c30e5 in rte_eth_dev_set_mtu > > (port_id=port_id@entry=0, mtu=<optimized out>) at > > ../lib/librte_ethdev/rte_ethdev.c:3484 > > #4 0x00000000006a61d8 in update_jumbo_frame_offload > > (portid=portid@entry=0) at ../app/test-pmd/testpmd.c:3371 > > #5 0x00000000006a62bc in init_config_port_offloads (pid=0, > > socket_id=0) at ../app/test-pmd/testpmd.c:1416 > > #6 0x000000000061770c in init_config () at > > ../app/test-pmd/testpmd.c:1505 > > #7 main (argc=<optimized out>, argv=<optimized out>) at > > ../app/test-pmd/testpmd.c:3800 > > (gdb) f 1 > > #1 0x0000000000e370d1 in virtio_check_scatter_on_all_rx_queues > > (frame_size=1530, dev=0x1799a40 <rte_eth_devices>) at > > ../drivers/net/virtio/virtio_ethdev.c:907 > > 907 buf_size = virtio_rx_mem_pool_buf_size(rxvq->mpool); > > (gdb) p hw->max_queue_pairs > > $1 = 3 > > (gdb) p qidx > > $2 = 5 > > (gdb) p hw->vqs[0] > > $3 = (struct virtqueue *) 0x17ffb03c0 > > (gdb) p hw->vqs[2] > > $4 = (struct virtqueue *) 0x17ff9dcc0 > > (gdb) p hw->vqs[4] > > $5 = (struct virtqueue *) 0x17ff8acc0 > > (gdb) p hw->vqs[6] > > $6 = (struct virtqueue *) 0x17ff77cc0 > > (gdb) p hw->vqs[7] > > $7 = (struct virtqueue *) 0x0 > > (gdb) p hw->vqs[8] > > $8 = (struct virtqueue *) 0x100004ac0 > > (gdb) p hw->vqs[9] > > $9 = (struct virtqueue *) 0x17ffb1600 > > (gdb) p hw->vqs[10] > > $10 = (struct virtqueue *) 0x17ffb18c0 > > > > > > > > For reference, also observed when 20.11.3 is paired with OVS > > https://mail.openvswitch.org/pipermail/ovs-dev/2021- > September/387940.html
On Wed, Sep 22, 2021 at 10:13 AM Peng, ZhihongX <zhihongx.peng@intel.com> wrote: > > On 15/09/2021 19:37, David Marchand wrote: > > > On Wed, Aug 4, 2021 at 10:36 AM <zhihongx.peng@intel.com> wrote: > > >> > > >> From: Zhihong Peng <zhihongx.peng@intel.com> > > >> > > >> This patch fixes the wrong way to obtain virtqueue. > > >> The end of virtqueue cannot be judged based on whether the array is > > >> NULL. > > > > > > Indeed, good catch. > > > > > > I can reproduce a crash with v20.11.3 which has backport of 4e8169eb0d2d. > > > I can not see it with main: maybe due to a lucky allocation or size > > > requested to rte_zmalloc... ? > > > > > This problem was discovered through google asan, we have submitted DPDK ASan patch. > http://patchwork.dpdk.org/project/dpdk/patch/20210918074155.872358-1-zhihongx.peng@intel.com/ Can you work on a v2? Maxime had comments. https://inbox.dpdk.org/dev/e1c512c1-9766-1cd8-816b-8a77d13d53d6@redhat.com/
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c index e58085a2c9..f2d19dc9d6 100644 --- a/drivers/net/virtio/virtio_ethdev.c +++ b/drivers/net/virtio/virtio_ethdev.c @@ -873,8 +873,8 @@ virtio_check_scatter_on_all_rx_queues(struct rte_eth_dev *dev, if (hw->vqs == NULL) return true; - for (qidx = 0; (vq = hw->vqs[2 * qidx + VTNET_SQ_RQ_QUEUE_IDX]) != NULL; - qidx++) { + for (qidx = 0; qidx < hw->max_queue_pairs; qidx++) { + vq = hw->vqs[2 * qidx + VTNET_SQ_RQ_QUEUE_IDX]; rxvq = &vq->rxq; if (rxvq->mpool == NULL) continue;