diff mbox series

kni: fix wrong mbuf alloc count in kni_allocate_mbufs

Message ID 4ebfe0d38b335a437edc9c58368153d005f562ce.1622460655.git.wangyunjian@huawei.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers
Series kni: fix wrong mbuf alloc count in kni_allocate_mbufs |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS
ci/github-robot success github build: passed
ci/iol-intel-Functional fail Functional Testing issues
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-testing fail Testing issues

Commit Message

Yunjian Wang May 31, 2021, 12:09 p.m. UTC 
  From: Yunjian Wang <wangyunjian@huawei.com>

In kni_allocate_mbufs(), we alloc mbuf for alloc_q as this code.
allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1) \
		& (MAX_MBUF_BURST_NUM - 1);
The value of allocq_free maybe zero (e.g 32 & (32 - 1) = 0), and
it will not fill the alloc_q. When the alloc_q's free count is
zero, it will drop the packet in kernel kni.

In this patch, we set the allocq_free as the min between
MAX_MBUF_BURST_NUM and the free count of the alloc_q.

Signed-off-by: Cheng Liu <liucheng11@huawei.com>
Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
---
 lib/kni/rte_kni.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Comments

Ferruh Yigit June 18, 2021, 1:37 p.m. UTC | #1 
On 5/31/2021 1:09 PM, wangyunjian wrote:
> From: Yunjian Wang <wangyunjian@huawei.com>
> 
> In kni_allocate_mbufs(), we alloc mbuf for alloc_q as this code.
> allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1) \
> 		& (MAX_MBUF_BURST_NUM - 1);
> The value of allocq_free maybe zero (e.g 32 & (32 - 1) = 0), and
> it will not fill the alloc_q. When the alloc_q's free count is
> zero, it will drop the packet in kernel kni.
> 

nack

Both 'read' & 'write' pointers can be max 'len-1',
so 'read - write - 1' can't be 'len'.
For above example first part can't be '32'.

But if you are observing a problem, can you please describe it a little more, it
may be because of something else.

> In this patch, we set the allocq_free as the min between
> MAX_MBUF_BURST_NUM and the free count of the alloc_q.
> 
> Signed-off-by: Cheng Liu <liucheng11@huawei.com>
> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> ---
>  lib/kni/rte_kni.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c
> index 9dae6a8d7c..20d8f20cef 100644
> --- a/lib/kni/rte_kni.c
> +++ b/lib/kni/rte_kni.c
> @@ -677,8 +677,9 @@ kni_allocate_mbufs(struct rte_kni *kni)
>  		return;
>  	}
>  
> -	allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1)
> -			& (MAX_MBUF_BURST_NUM - 1);
> +	allocq_free = kni_fifo_free_count(kni->alloc_q);
> +	allocq_free = (allocq_free > MAX_MBUF_BURST_NUM) ?
> +		      MAX_MBUF_BURST_NUM : allocq_free;
>  	for (i = 0; i < allocq_free; i++) {
>  		pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
>  		if (unlikely(pkts[i] == NULL)) {
>
Yunjian Wang June 21, 2021, 3:27 a.m. UTC | #2 
> -----Original Message-----
> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> Sent: Friday, June 18, 2021 9:37 PM
> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org
> Cc: liucheng (J) <liucheng11@huawei.com>; dingxiaoxiong
> <dingxiaoxiong@huawei.com>
> Subject: Re: [dpdk-dev] [PATCH] kni: fix wrong mbuf alloc count in
> kni_allocate_mbufs
> 
> On 5/31/2021 1:09 PM, wangyunjian wrote:
> > From: Yunjian Wang <wangyunjian@huawei.com>
> >
> > In kni_allocate_mbufs(), we alloc mbuf for alloc_q as this code.
> > allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1) \
> > 		& (MAX_MBUF_BURST_NUM - 1);
> > The value of allocq_free maybe zero (e.g 32 & (32 - 1) = 0), and it
> > will not fill the alloc_q. When the alloc_q's free count is zero, it
> > will drop the packet in kernel kni.
> >
> 
> nack
> 
> Both 'read' & 'write' pointers can be max 'len-1', so 'read - write - 1' can't be
> 'len'.
> For above example first part can't be '32'.
> 
> But if you are observing a problem, can you please describe it a little more, it
> may be because of something else.

The ring size is 1024. After init, write = read = 0. Then we fill kni->alloc_q to full. At this time, write = 1023, read = 0.
Then the kernel send 32 packets to userspace. At this time, write = 1023, read = 32.
And then the userspace recieve this 32 packets. Then fill the kni->alloc_q, (32 - 1023 - 1)&31 = 0, fill nothing.
...
Then the kernel send 32 packets to userspace. At this time, write = 1023, read = 992.
And then the userspace recieve this 32 packets. Then fill the kni->alloc_q, (992 - 1023 - 1)&31 = 0, fill nothing.
Then the kernel send 32 packets to userspace. The kni->alloc_q only has 31 mbufs and will drop one packet.

Absolutely, this is a special scene. Normally, it will fill some mbufs everytime, but may not enough for the kernel to use.
In this patch, we always keep the kni->alloc_q to full for the kernel to use.

Thanks

> 
> > In this patch, we set the allocq_free as the min between
> > MAX_MBUF_BURST_NUM and the free count of the alloc_q.
> >
> > Signed-off-by: Cheng Liu <liucheng11@huawei.com>
> > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > ---
> >  lib/kni/rte_kni.c | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c index
> > 9dae6a8d7c..20d8f20cef 100644
> > --- a/lib/kni/rte_kni.c
> > +++ b/lib/kni/rte_kni.c
> > @@ -677,8 +677,9 @@ kni_allocate_mbufs(struct rte_kni *kni)
> >  		return;
> >  	}
> >
> > -	allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1)
> > -			& (MAX_MBUF_BURST_NUM - 1);
> > +	allocq_free = kni_fifo_free_count(kni->alloc_q);
> > +	allocq_free = (allocq_free > MAX_MBUF_BURST_NUM) ?
> > +		      MAX_MBUF_BURST_NUM : allocq_free;
> >  	for (i = 0; i < allocq_free; i++) {
> >  		pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
> >  		if (unlikely(pkts[i] == NULL)) {
> >
Ferruh Yigit June 21, 2021, 11:26 a.m. UTC | #3 
On 6/21/2021 4:27 AM, wangyunjian wrote:
>> -----Original Message-----
>> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
>> Sent: Friday, June 18, 2021 9:37 PM
>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org
>> Cc: liucheng (J) <liucheng11@huawei.com>; dingxiaoxiong
>> <dingxiaoxiong@huawei.com>
>> Subject: Re: [dpdk-dev] [PATCH] kni: fix wrong mbuf alloc count in
>> kni_allocate_mbufs
>>
>> On 5/31/2021 1:09 PM, wangyunjian wrote:
>>> From: Yunjian Wang <wangyunjian@huawei.com>
>>>
>>> In kni_allocate_mbufs(), we alloc mbuf for alloc_q as this code.
>>> allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1) \
>>> 		& (MAX_MBUF_BURST_NUM - 1);
>>> The value of allocq_free maybe zero (e.g 32 & (32 - 1) = 0), and it
>>> will not fill the alloc_q. When the alloc_q's free count is zero, it
>>> will drop the packet in kernel kni.
>>>
>>
>> nack
>>
>> Both 'read' & 'write' pointers can be max 'len-1', so 'read - write - 1' can't be
>> 'len'.
>> For above example first part can't be '32'.
>>
>> But if you are observing a problem, can you please describe it a little more, it
>> may be because of something else.
> 
> The ring size is 1024. After init, write = read = 0. Then we fill kni->alloc_q to full. At this time, write = 1023, read = 0.
> Then the kernel send 32 packets to userspace. At this time, write = 1023, read = 32.
> And then the userspace recieve this 32 packets. Then fill the kni->alloc_q, (32 - 1023 - 1)&31 = 0, fill nothing.
> ...
> Then the kernel send 32 packets to userspace. At this time, write = 1023, read = 992.
> And then the userspace recieve this 32 packets. Then fill the kni->alloc_q, (992 - 1023 - 1)&31 = 0, fill nothing.
> Then the kernel send 32 packets to userspace. The kni->alloc_q only has 31 mbufs and will drop one packet.
> 
> Absolutely, this is a special scene. Normally, it will fill some mbufs everytime, but may not enough for the kernel to use.
> In this patch, we always keep the kni->alloc_q to full for the kernel to use.
> 

I see now, yes it is technically possible to have above scenario and it can
cause glitch in the datapath.

Below fix looks good, +1 to use 'kni_fifo_free_count()' instead of calculation
within the function which may be wrong for the 'RTE_USE_C11_MEM_MODEL' case.

Can you please add fixes line too?

> Thanks
> 
>>
>>> In this patch, we set the allocq_free as the min between
>>> MAX_MBUF_BURST_NUM and the free count of the alloc_q.
>>>
>>> Signed-off-by: Cheng Liu <liucheng11@huawei.com>
>>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
>>> ---
>>>  lib/kni/rte_kni.c | 5 +++--
>>>  1 file changed, 3 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c index
>>> 9dae6a8d7c..20d8f20cef 100644
>>> --- a/lib/kni/rte_kni.c
>>> +++ b/lib/kni/rte_kni.c
>>> @@ -677,8 +677,9 @@ kni_allocate_mbufs(struct rte_kni *kni)
>>>  		return;
>>>  	}
>>>
>>> -	allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1)
>>> -			& (MAX_MBUF_BURST_NUM - 1);
>>> +	allocq_free = kni_fifo_free_count(kni->alloc_q);
>>> +	allocq_free = (allocq_free > MAX_MBUF_BURST_NUM) ?
>>> +		      MAX_MBUF_BURST_NUM : allocq_free;
>>>  	for (i = 0; i < allocq_free; i++) {
>>>  		pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
>>>  		if (unlikely(pkts[i] == NULL)) {
>>>
>
Yunjian Wang June 22, 2021, 7:32 a.m. UTC | #4 
> -----Original Message-----
> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> Sent: Monday, June 21, 2021 7:26 PM
> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org
> Cc: liucheng (J) <liucheng11@huawei.com>; dingxiaoxiong
> <dingxiaoxiong@huawei.com>
> Subject: Re: [dpdk-dev] [PATCH] kni: fix wrong mbuf alloc count in
> kni_allocate_mbufs
> 
> On 6/21/2021 4:27 AM, wangyunjian wrote:
> >> -----Original Message-----
> >> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> >> Sent: Friday, June 18, 2021 9:37 PM
> >> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org
> >> Cc: liucheng (J) <liucheng11@huawei.com>; dingxiaoxiong
> >> <dingxiaoxiong@huawei.com>
> >> Subject: Re: [dpdk-dev] [PATCH] kni: fix wrong mbuf alloc count in
> >> kni_allocate_mbufs
> >>
> >> On 5/31/2021 1:09 PM, wangyunjian wrote:
> >>> From: Yunjian Wang <wangyunjian@huawei.com>
> >>>
> >>> In kni_allocate_mbufs(), we alloc mbuf for alloc_q as this code.
> >>> allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1) \
> >>> 		& (MAX_MBUF_BURST_NUM - 1);
> >>> The value of allocq_free maybe zero (e.g 32 & (32 - 1) = 0), and it
> >>> will not fill the alloc_q. When the alloc_q's free count is zero, it
> >>> will drop the packet in kernel kni.
> >>>
> >>
> >> nack
> >>
> >> Both 'read' & 'write' pointers can be max 'len-1', so 'read - write -
> >> 1' can't be 'len'.
> >> For above example first part can't be '32'.
> >>
> >> But if you are observing a problem, can you please describe it a
> >> little more, it may be because of something else.
> >
> > The ring size is 1024. After init, write = read = 0. Then we fill kni->alloc_q to
> full. At this time, write = 1023, read = 0.
> > Then the kernel send 32 packets to userspace. At this time, write = 1023,
> read = 32.
> > And then the userspace recieve this 32 packets. Then fill the kni->alloc_q, (32
> - 1023 - 1)&31 = 0, fill nothing.
> > ...
> > Then the kernel send 32 packets to userspace. At this time, write = 1023,
> read = 992.
> > And then the userspace recieve this 32 packets. Then fill the kni->alloc_q,
> (992 - 1023 - 1)&31 = 0, fill nothing.
> > Then the kernel send 32 packets to userspace. The kni->alloc_q only has 31
> mbufs and will drop one packet.
> >
> > Absolutely, this is a special scene. Normally, it will fill some mbufs everytime,
> but may not enough for the kernel to use.
> > In this patch, we always keep the kni->alloc_q to full for the kernel to use.
> >
> 
> I see now, yes it is technically possible to have above scenario and it can cause
> glitch in the datapath.
> 
> Below fix looks good, +1 to use 'kni_fifo_free_count()' instead of calculation
> within the function which may be wrong for the 'RTE_USE_C11_MEM_MODEL'
> case.

I compiled them on the ARM and x86 platforms with the 'RTE_USE_C11_MEM_MODEL'
case, and no error is reported.

> 
> Can you please add fixes line too?

OK, will include it in next version.

Thanks

> 
> > Thanks
> >
> >>
> >>> In this patch, we set the allocq_free as the min between
> >>> MAX_MBUF_BURST_NUM and the free count of the alloc_q.
> >>>
> >>> Signed-off-by: Cheng Liu <liucheng11@huawei.com>
> >>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> >>> ---
> >>>  lib/kni/rte_kni.c | 5 +++--
> >>>  1 file changed, 3 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c index
> >>> 9dae6a8d7c..20d8f20cef 100644
> >>> --- a/lib/kni/rte_kni.c
> >>> +++ b/lib/kni/rte_kni.c
> >>> @@ -677,8 +677,9 @@ kni_allocate_mbufs(struct rte_kni *kni)
> >>>  		return;
> >>>  	}
> >>>
> >>> -	allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1)
> >>> -			& (MAX_MBUF_BURST_NUM - 1);
> >>> +	allocq_free = kni_fifo_free_count(kni->alloc_q);
> >>> +	allocq_free = (allocq_free > MAX_MBUF_BURST_NUM) ?
> >>> +		      MAX_MBUF_BURST_NUM : allocq_free;
> >>>  	for (i = 0; i < allocq_free; i++) {
> >>>  		pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
> >>>  		if (unlikely(pkts[i] == NULL)) {
> >>>
> >
Ferruh Yigit June 22, 2021, 7:43 a.m. UTC | #5 
On 6/22/2021 8:32 AM, wangyunjian wrote:
>> -----Original Message-----
>> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
>> Sent: Monday, June 21, 2021 7:26 PM
>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org
>> Cc: liucheng (J) <liucheng11@huawei.com>; dingxiaoxiong
>> <dingxiaoxiong@huawei.com>
>> Subject: Re: [dpdk-dev] [PATCH] kni: fix wrong mbuf alloc count in
>> kni_allocate_mbufs
>>
>> On 6/21/2021 4:27 AM, wangyunjian wrote:
>>>> -----Original Message-----
>>>> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
>>>> Sent: Friday, June 18, 2021 9:37 PM
>>>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org
>>>> Cc: liucheng (J) <liucheng11@huawei.com>; dingxiaoxiong
>>>> <dingxiaoxiong@huawei.com>
>>>> Subject: Re: [dpdk-dev] [PATCH] kni: fix wrong mbuf alloc count in
>>>> kni_allocate_mbufs
>>>>
>>>> On 5/31/2021 1:09 PM, wangyunjian wrote:
>>>>> From: Yunjian Wang <wangyunjian@huawei.com>
>>>>>
>>>>> In kni_allocate_mbufs(), we alloc mbuf for alloc_q as this code.
>>>>> allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1) \
>>>>> 		& (MAX_MBUF_BURST_NUM - 1);
>>>>> The value of allocq_free maybe zero (e.g 32 & (32 - 1) = 0), and it
>>>>> will not fill the alloc_q. When the alloc_q's free count is zero, it
>>>>> will drop the packet in kernel kni.
>>>>>
>>>>
>>>> nack
>>>>
>>>> Both 'read' & 'write' pointers can be max 'len-1', so 'read - write -
>>>> 1' can't be 'len'.
>>>> For above example first part can't be '32'.
>>>>
>>>> But if you are observing a problem, can you please describe it a
>>>> little more, it may be because of something else.
>>>
>>> The ring size is 1024. After init, write = read = 0. Then we fill kni->alloc_q to
>> full. At this time, write = 1023, read = 0.
>>> Then the kernel send 32 packets to userspace. At this time, write = 1023,
>> read = 32.
>>> And then the userspace recieve this 32 packets. Then fill the kni->alloc_q, (32
>> - 1023 - 1)&31 = 0, fill nothing.
>>> ...
>>> Then the kernel send 32 packets to userspace. At this time, write = 1023,
>> read = 992.
>>> And then the userspace recieve this 32 packets. Then fill the kni->alloc_q,
>> (992 - 1023 - 1)&31 = 0, fill nothing.
>>> Then the kernel send 32 packets to userspace. The kni->alloc_q only has 31
>> mbufs and will drop one packet.
>>>
>>> Absolutely, this is a special scene. Normally, it will fill some mbufs everytime,
>> but may not enough for the kernel to use.
>>> In this patch, we always keep the kni->alloc_q to full for the kernel to use.
>>>
>>
>> I see now, yes it is technically possible to have above scenario and it can cause
>> glitch in the datapath.
>>
>> Below fix looks good, +1 to use 'kni_fifo_free_count()' instead of calculation
>> within the function which may be wrong for the 'RTE_USE_C11_MEM_MODEL'
>> case.
> 
> I compiled them on the ARM and x86 platforms with the 'RTE_USE_C11_MEM_MODEL'
> case, and no error is reported.
> 

May not be build error, but in 'RTE_USE_C11_MEM_MODEL' case 'read'/'write' are
not volatile and need to read them via C11 atomic instructions. 'allocq_free'
calculation in the 'kni_allocate_mbufs()' doesn't do that, that is why better to
replace calculation with 'kni_fifo_free_count()'.

>>
>> Can you please add fixes line too?
> 
> OK, will include it in next version.
> 

Thanks.

> Thanks
> 
>>
>>> Thanks
>>>
>>>>
>>>>> In this patch, we set the allocq_free as the min between
>>>>> MAX_MBUF_BURST_NUM and the free count of the alloc_q.
>>>>>
>>>>> Signed-off-by: Cheng Liu <liucheng11@huawei.com>
>>>>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
>>>>> ---
>>>>>  lib/kni/rte_kni.c | 5 +++--
>>>>>  1 file changed, 3 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c index
>>>>> 9dae6a8d7c..20d8f20cef 100644
>>>>> --- a/lib/kni/rte_kni.c
>>>>> +++ b/lib/kni/rte_kni.c
>>>>> @@ -677,8 +677,9 @@ kni_allocate_mbufs(struct rte_kni *kni)
>>>>>  		return;
>>>>>  	}
>>>>>
>>>>> -	allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1)
>>>>> -			& (MAX_MBUF_BURST_NUM - 1);
>>>>> +	allocq_free = kni_fifo_free_count(kni->alloc_q);
>>>>> +	allocq_free = (allocq_free > MAX_MBUF_BURST_NUM) ?
>>>>> +		      MAX_MBUF_BURST_NUM : allocq_free;
>>>>>  	for (i = 0; i < allocq_free; i++) {
>>>>>  		pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
>>>>>  		if (unlikely(pkts[i] == NULL)) {
>>>>>
>>>
>
diff mbox series

Patch

diff --git a/lib/kni/rte_kni.c b/lib/kni/rte_kni.c
index 9dae6a8d7c..20d8f20cef 100644
--- a/lib/kni/rte_kni.c
+++ b/lib/kni/rte_kni.c
@@ -677,8 +677,9 @@  kni_allocate_mbufs(struct rte_kni *kni)
 		return;
 	}
 
-	allocq_free = (kni->alloc_q->read - kni->alloc_q->write - 1)
-			& (MAX_MBUF_BURST_NUM - 1);
+	allocq_free = kni_fifo_free_count(kni->alloc_q);
+	allocq_free = (allocq_free > MAX_MBUF_BURST_NUM) ?
+		      MAX_MBUF_BURST_NUM : allocq_free;
 	for (i = 0; i < allocq_free; i++) {
 		pkts[i] = rte_pktmbuf_alloc(kni->pktmbuf_pool);
 		if (unlikely(pkts[i] == NULL)) {