[v3] kni: fix possible alloc_q starvation when mbufs are exhausted
Checks
Commit Message
In some scenarios, mbufs returned by rte_kni_rx_burst are not freed
immediately. So kni_allocate_mbufs may be failed, but we don't know.
Even worse, when alloc_q is completely exhausted, kni_net_tx in
rte_kni.ko will drop all tx packets. kni_allocate_mbufs is never
called again, even if the mbufs are eventually freed.
In this patch, we try to allocate mbufs for alloc_q when it is empty.
According to historical experience, the performance bottleneck of KNI
is offen the usleep_range of kni thread in rte_kni.ko.
The check of kni_fifo_count is trivial and the cost should be acceptable.
Fixes: 3e12a98fe397 ("kni: optimize Rx burst")
Cc: stable@dpdk.org
Signed-off-by: Yangchao Zhou <zhouyates@gmail.com>
---
lib/kni/rte_kni.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
Comments
On 12/30/2022 4:23 AM, Yangchao Zhou wrote:
> In some scenarios, mbufs returned by rte_kni_rx_burst are not freed
> immediately. So kni_allocate_mbufs may be failed, but we don't know.
>
> Even worse, when alloc_q is completely exhausted, kni_net_tx in
> rte_kni.ko will drop all tx packets. kni_allocate_mbufs is never
> called again, even if the mbufs are eventually freed.
>
> In this patch, we try to allocate mbufs for alloc_q when it is empty.
>
> According to historical experience, the performance bottleneck of KNI
> is offen the usleep_range of kni thread in rte_kni.ko.
> The check of kni_fifo_count is trivial and the cost should be acceptable.
>
Hi Yangchao,
Are you observing any performance impact with this change in you use case?
> Fixes: 3e12a98fe397 ("kni: optimize Rx burst")
> Cc: stable@dpdk.org
>
> Signed-off-by: Yangchao Zhou <zhouyates@gmail.com>
> ---
> lib/kni/rte_kni.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c
> index 8ab6c47153..bfa6a001ff 100644
> --- a/lib/kni/rte_kni.c
> +++ b/lib/kni/rte_kni.c
> @@ -634,8 +634,8 @@ rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf **mbufs, unsigned int num)
> {
> unsigned int ret = kni_fifo_get(kni->tx_q, (void **)mbufs, num);
>
> - /* If buffers removed, allocate mbufs and then put them into alloc_q */
> - if (ret)
> + /* If buffers removed or alloc_q is empty, allocate mbufs and then put them into alloc_q */
> + if (ret || (kni_fifo_count(kni->alloc_q) == 0))
> kni_allocate_mbufs(kni);
>
> return ret;
Hi Ferruh,
In my case, the traffic is not large, so I can't see the impact.
I also tested under high load(>2Mpps with 2 DPDK cores and 2 kernel threads)
and found no significant difference in performance either.
I think the reason should be that it will not
run to 'kni_fifo_count(kni->alloc_q) == 0' under high load.
On Tue, Jan 3, 2023 at 8:47 PM Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> On 12/30/2022 4:23 AM, Yangchao Zhou wrote:
> > In some scenarios, mbufs returned by rte_kni_rx_burst are not freed
> > immediately. So kni_allocate_mbufs may be failed, but we don't know.
> >
> > Even worse, when alloc_q is completely exhausted, kni_net_tx in
> > rte_kni.ko will drop all tx packets. kni_allocate_mbufs is never
> > called again, even if the mbufs are eventually freed.
> >
> > In this patch, we try to allocate mbufs for alloc_q when it is empty.
> >
> > According to historical experience, the performance bottleneck of KNI
> > is offen the usleep_range of kni thread in rte_kni.ko.
> > The check of kni_fifo_count is trivial and the cost should be acceptable.
> >
>
> Hi Yangchao,
>
> Are you observing any performance impact with this change in you use case?
>
>
> > Fixes: 3e12a98fe397 ("kni: optimize Rx burst")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Yangchao Zhou <zhouyates@gmail.com>
> > ---
> > lib/kni/rte_kni.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c
> > index 8ab6c47153..bfa6a001ff 100644
> > --- a/lib/kni/rte_kni.c
> > +++ b/lib/kni/rte_kni.c
> > @@ -634,8 +634,8 @@ rte_kni_rx_burst(struct rte_kni *kni, struct
> rte_mbuf **mbufs, unsigned int num)
> > {
> > unsigned int ret = kni_fifo_get(kni->tx_q, (void **)mbufs, num);
> >
> > - /* If buffers removed, allocate mbufs and then put them into
> alloc_q */
> > - if (ret)
> > + /* If buffers removed or alloc_q is empty, allocate mbufs and then
> put them into alloc_q */
> > + if (ret || (kni_fifo_count(kni->alloc_q) == 0))
> > kni_allocate_mbufs(kni);
> >
> > return ret;
>
>
On 1/4/2023 11:57 AM, Matt wrote:
> Hi Ferruh,
>
> In my case, the traffic is not large, so I can't see the impact.
> I also tested under high load(>2Mpps with 2 DPDK cores and 2 kernel threads)
> and found no significant difference in performance either.
> I think the reason should be that it will not
> run to 'kni_fifo_count(kni->alloc_q) == 0' under high load.
>
I agree, additional check most likely hit on the low bandwidth,
thanks for checking for performance impact.
> On Tue, Jan 3, 2023 at 8:47 PM Ferruh Yigit <ferruh.yigit@amd.com
> <mailto:ferruh.yigit@amd.com>> wrote:
>
> On 12/30/2022 4:23 AM, Yangchao Zhou wrote:
> > In some scenarios, mbufs returned by rte_kni_rx_burst are not freed
> > immediately. So kni_allocate_mbufs may be failed, but we don't know.
> >
> > Even worse, when alloc_q is completely exhausted, kni_net_tx in
> > rte_kni.ko will drop all tx packets. kni_allocate_mbufs is never
> > called again, even if the mbufs are eventually freed.
> >
> > In this patch, we try to allocate mbufs for alloc_q when it is empty.
> >
> > According to historical experience, the performance bottleneck of KNI
> > is offen the usleep_range of kni thread in rte_kni.ko.
> > The check of kni_fifo_count is trivial and the cost should be
> acceptable.
> >
>
> Hi Yangchao,
>
> Are you observing any performance impact with this change in you use
> case?
>
>
> > Fixes: 3e12a98fe397 ("kni: optimize Rx burst")
> > Cc: stable@dpdk.org <mailto:stable@dpdk.org>
> >
> > Signed-off-by: Yangchao Zhou <zhouyates@gmail.com
> <mailto:zhouyates@gmail.com>>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
04/01/2023 15:34, Ferruh Yigit:
> On 1/4/2023 11:57 AM, Matt wrote:
> > Hi Ferruh,
> >
> > In my case, the traffic is not large, so I can't see the impact.
> > I also tested under high load(>2Mpps with 2 DPDK cores and 2 kernel threads)
> > and found no significant difference in performance either.
> > I think the reason should be that it will not
> > run to 'kni_fifo_count(kni->alloc_q) == 0' under high load.
> >
>
> I agree, additional check most likely hit on the low bandwidth,
> thanks for checking for performance impact.
>
> > On Tue, Jan 3, 2023 at 8:47 PM Ferruh Yigit <ferruh.yigit@amd.com
> > <mailto:ferruh.yigit@amd.com>> wrote:
> >
> > On 12/30/2022 4:23 AM, Yangchao Zhou wrote:
> > > In some scenarios, mbufs returned by rte_kni_rx_burst are not freed
> > > immediately. So kni_allocate_mbufs may be failed, but we don't know.
> > >
> > > Even worse, when alloc_q is completely exhausted, kni_net_tx in
> > > rte_kni.ko will drop all tx packets. kni_allocate_mbufs is never
> > > called again, even if the mbufs are eventually freed.
> > >
> > > In this patch, we try to allocate mbufs for alloc_q when it is empty.
> > >
> > > According to historical experience, the performance bottleneck of KNI
> > > is offen the usleep_range of kni thread in rte_kni.ko.
> > > The check of kni_fifo_count is trivial and the cost should be
> > acceptable.
> > >
> >
> > Hi Yangchao,
> >
> > Are you observing any performance impact with this change in you use
> > case?
> >
> >
> > > Fixes: 3e12a98fe397 ("kni: optimize Rx burst")
> > > Cc: stable@dpdk.org <mailto:stable@dpdk.org>
> > >
> > > Signed-off-by: Yangchao Zhou <zhouyates@gmail.com
> > <mailto:zhouyates@gmail.com>>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
Applied, thanks.
@@ -634,8 +634,8 @@ rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf **mbufs, unsigned int num)
{
unsigned int ret = kni_fifo_get(kni->tx_q, (void **)mbufs, num);
- /* If buffers removed, allocate mbufs and then put them into alloc_q */
- if (ret)
+ /* If buffers removed or alloc_q is empty, allocate mbufs and then put them into alloc_q */
+ if (ret || (kni_fifo_count(kni->alloc_q) == 0))
kni_allocate_mbufs(kni);
return ret;