diff mbox series

[v2] eal/linux: do not create user mem map repeatedly when it exists

Message ID 1595515713-24640-1-git-send-email-wangyunjian@huawei.com (mailing list archive)
State Superseded, archived
Delegated to: David Marchand
Headers
Series [v2] eal/linux: do not create user mem map repeatedly when it exists |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK
ci/travis-robot success Travis build: passed
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-testing success Testing PASS
ci/iol-mellanox-Performance success Performance Testing PASS

Commit Message

Yunjian Wang July 23, 2020, 2:48 p.m. UTC 
  From: Yunjian Wang <wangyunjian@huawei.com>

Currently, we will create new user mem map entry for the same memory
segment, but in fact it has already been added to the user mem maps.
It's not necessary to create it twice.

To resolve the issue, add support to remove the same entry in the
function compact_user_maps().

Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
Cc: stable@dpdk.org

Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
---
v2:
* Remove the same entry in the function compact_user_maps()
---
 lib/librte_eal/linux/eal_vfio.c | 5 +++++
 1 file changed, 5 insertions(+)

Comments

Burakov, Anatoly July 24, 2020, 1:25 p.m. UTC | #1 
On 23-Jul-20 3:48 PM, wangyunjian wrote:
> From: Yunjian Wang <wangyunjian@huawei.com>
> 
> Currently, we will create new user mem map entry for the same memory
> segment, but in fact it has already been added to the user mem maps.
> It's not necessary to create it twice.
> 
> To resolve the issue, add support to remove the same entry in the
> function compact_user_maps().
> 
> Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> ---
> v2:
> * Remove the same entry in the function compact_user_maps()
> ---
>   lib/librte_eal/linux/eal_vfio.c | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> index abb12a354..df99307b7 100644
> --- a/lib/librte_eal/linux/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal_vfio.c
> @@ -167,6 +167,10 @@ adjust_map(struct user_mem_map *src, struct user_mem_map *end,
>   static int
>   merge_map(struct user_mem_map *left, struct user_mem_map *right)
>   {
> +	/* merge the same maps into one */
> +	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
> +		goto out;
> +

merge_map looks for adjacent maps only, but does not handle maps that 
are wholly contained within one another ("the same map" also matches 
this definition). wouldn't it be better to check for that instead of 
*just* handling identical maps?

>   	if (left->addr + left->len != right->addr)
>   		return 0;
>   	if (left->iova + left->len != right->iova)
> @@ -174,6 +178,7 @@ merge_map(struct user_mem_map *left, struct user_mem_map *right)
>   
>   	left->len += right->len;
>   
> +out:
>   	memset(right, 0, sizeof(*right));
>   
>   	return 1;
>
Yunjian Wang July 25, 2020, 9:59 a.m. UTC | #2 
> -----Original Message-----
> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> Sent: Friday, July 24, 2020 9:25 PM
> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
> david.marchand@redhat.com
> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
> <xudingke@huawei.com>; stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
> repeatedly when it exists
> 
> On 23-Jul-20 3:48 PM, wangyunjian wrote:
> > From: Yunjian Wang <wangyunjian@huawei.com>
> >
> > Currently, we will create new user mem map entry for the same memory
> > segment, but in fact it has already been added to the user mem maps.
> > It's not necessary to create it twice.
> >
> > To resolve the issue, add support to remove the same entry in the
> > function compact_user_maps().
> >
> > Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > ---
> > v2:
> > * Remove the same entry in the function compact_user_maps()
> > ---
> >   lib/librte_eal/linux/eal_vfio.c | 5 +++++
> >   1 file changed, 5 insertions(+)
> >
> > diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> > index abb12a354..df99307b7 100644
> > --- a/lib/librte_eal/linux/eal_vfio.c
> > +++ b/lib/librte_eal/linux/eal_vfio.c
> > @@ -167,6 +167,10 @@ adjust_map(struct user_mem_map *src, struct
> user_mem_map *end,
> >   static int
> >   merge_map(struct user_mem_map *left, struct user_mem_map *right)
> >   {
> > +	/* merge the same maps into one */
> > +	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
> > +		goto out;
> > +
> 
> merge_map looks for adjacent maps only, but does not handle maps that
> are wholly contained within one another ("the same map" also matches
> this definition). wouldn't it be better to check for that instead of
> *just* handling identical maps?

What about using the initial implementation?
We don't create new user mem map entry for the same memory segment.

@@ -1828,6 +1828,13 @@  container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
 		ret = -1;
 		goto out;
 	}
+
+	/* we don't need create new user mem map entry
+	 * for the same memory segment.
+	 */
+	if (errno == EBUSY || errno == EEXIST)
+		goto out;
+
 	/* create new user mem map entry */
 	new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
 	new_map->addr = vaddr;

Thanks,
Yunjian
> 
> >   	if (left->addr + left->len != right->addr)
> >   		return 0;
> >   	if (left->iova + left->len != right->iova)
> > @@ -174,6 +178,7 @@ merge_map(struct user_mem_map *left, struct
> user_mem_map *right)
> >
> >   	left->len += right->len;
> >
> > +out:
> >   	memset(right, 0, sizeof(*right));
> >
> >   	return 1;
> >
> 
> 
> --
> Thanks,
> Anatoly
Burakov, Anatoly July 27, 2020, 9:24 a.m. UTC | #3 
On 25-Jul-20 10:59 AM, wangyunjian wrote:
>> -----Original Message-----
>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
>> Sent: Friday, July 24, 2020 9:25 PM
>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
>> david.marchand@redhat.com
>> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
>> <xudingke@huawei.com>; stable@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
>> repeatedly when it exists
>>
>> On 23-Jul-20 3:48 PM, wangyunjian wrote:
>>> From: Yunjian Wang <wangyunjian@huawei.com>
>>>
>>> Currently, we will create new user mem map entry for the same memory
>>> segment, but in fact it has already been added to the user mem maps.
>>> It's not necessary to create it twice.
>>>
>>> To resolve the issue, add support to remove the same entry in the
>>> function compact_user_maps().
>>>
>>> Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
>>> Cc: stable@dpdk.org
>>>
>>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
>>> ---
>>> v2:
>>> * Remove the same entry in the function compact_user_maps()
>>> ---
>>>    lib/librte_eal/linux/eal_vfio.c | 5 +++++
>>>    1 file changed, 5 insertions(+)
>>>
>>> diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
>>> index abb12a354..df99307b7 100644
>>> --- a/lib/librte_eal/linux/eal_vfio.c
>>> +++ b/lib/librte_eal/linux/eal_vfio.c
>>> @@ -167,6 +167,10 @@ adjust_map(struct user_mem_map *src, struct
>> user_mem_map *end,
>>>    static int
>>>    merge_map(struct user_mem_map *left, struct user_mem_map *right)
>>>    {
>>> +	/* merge the same maps into one */
>>> +	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
>>> +		goto out;
>>> +
>>
>> merge_map looks for adjacent maps only, but does not handle maps that
>> are wholly contained within one another ("the same map" also matches
>> this definition). wouldn't it be better to check for that instead of
>> *just* handling identical maps?
> 
> What about using the initial implementation?
> We don't create new user mem map entry for the same memory segment.

I don't like this implementation because it relies on particulars of how 
VFIO mapping work without explicitly specifying them. I.e. it's prone to 
breaking when changing code. That's not even mentioning that we have no 
guarantees on kernel behavior in that particular case being identical on 
all supported platforms.

I would honestly prefer an explicit compaction over implicit one.

> 
> @@ -1828,6 +1828,13 @@  container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
>   		ret = -1;
>   		goto out;
>   	}
> +
> +	/* we don't need create new user mem map entry
> +	 * for the same memory segment.
> +	 */
> +	if (errno == EBUSY || errno == EEXIST)
> +		goto out;
> +
>   	/* create new user mem map entry */
>   	new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
>   	new_map->addr = vaddr;
> 
> Thanks,
> Yunjian
>>
>>>    	if (left->addr + left->len != right->addr)
>>>    		return 0;
>>>    	if (left->iova + left->len != right->iova)
>>> @@ -174,6 +178,7 @@ merge_map(struct user_mem_map *left, struct
>> user_mem_map *right)
>>>
>>>    	left->len += right->len;
>>>
>>> +out:
>>>    	memset(right, 0, sizeof(*right));
>>>
>>>    	return 1;
>>>
>>
>>
>> --
>> Thanks,
>> Anatoly
Yunjian Wang July 30, 2020, 1:16 p.m. UTC | #4 
> -----Original Message-----
> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> Sent: Monday, July 27, 2020 5:24 PM
> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
> david.marchand@redhat.com
> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
> <xudingke@huawei.com>; stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
> repeatedly when it exists
> 
> On 25-Jul-20 10:59 AM, wangyunjian wrote:
> >> -----Original Message-----
> >> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> >> Sent: Friday, July 24, 2020 9:25 PM
> >> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
> >> david.marchand@redhat.com
> >> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
> >> <xudingke@huawei.com>; stable@dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem
> >> map repeatedly when it exists
> >>
> >> On 23-Jul-20 3:48 PM, wangyunjian wrote:
> >>> From: Yunjian Wang <wangyunjian@huawei.com>
> >>>
> >>> Currently, we will create new user mem map entry for the same memory
> >>> segment, but in fact it has already been added to the user mem maps.
> >>> It's not necessary to create it twice.
> >>>
> >>> To resolve the issue, add support to remove the same entry in the
> >>> function compact_user_maps().
> >>>
> >>> Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
> >>> Cc: stable@dpdk.org
> >>>
> >>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> >>> ---
> >>> v2:
> >>> * Remove the same entry in the function compact_user_maps()
> >>> ---
> >>>    lib/librte_eal/linux/eal_vfio.c | 5 +++++
> >>>    1 file changed, 5 insertions(+)
> >>>
> >>> diff --git a/lib/librte_eal/linux/eal_vfio.c
> >>> b/lib/librte_eal/linux/eal_vfio.c index abb12a354..df99307b7 100644
> >>> --- a/lib/librte_eal/linux/eal_vfio.c
> >>> +++ b/lib/librte_eal/linux/eal_vfio.c
> >>> @@ -167,6 +167,10 @@ adjust_map(struct user_mem_map *src, struct
> >> user_mem_map *end,
> >>>    static int
> >>>    merge_map(struct user_mem_map *left, struct user_mem_map
> *right)
> >>>    {
> >>> +	/* merge the same maps into one */
> >>> +	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
> >>> +		goto out;
> >>> +
> >>
> >> merge_map looks for adjacent maps only, but does not handle maps that
> >> are wholly contained within one another ("the same map" also matches
> >> this definition). wouldn't it be better to check for that instead of
> >> *just* handling identical maps?
> >
> > What about using the initial implementation?
> > We don't create new user mem map entry for the same memory segment.
> 
> I don't like this implementation because it relies on particulars of how VFIO
> mapping work without explicitly specifying them. I.e. it's prone to breaking
> when changing code. That's not even mentioning that we have no guarantees
> on kernel behavior in that particular case being identical on all supported
> platforms.
> 
> I would honestly prefer an explicit compaction over implicit one.

What about this implementation?

diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
index e07979936..8dcb04cd9 100644
--- a/lib/librte_eal/linux/eal_vfio.c
+++ b/lib/librte_eal/linux/eal_vfio.c
@@ -179,6 +179,19 @@ merge_map(struct user_mem_map *left, struct user_mem_map *right)
 	return 1;
 }
 
+/* try merging two same maps into one, return 1 if succeeded */
+static int
+merge_same_map(struct user_mem_map *left, struct user_mem_map *right)
+{
+	if (memcmp(left, right, sizeof(struct user_mem_map)) != 0) {
+		return 0;
+	}
+
+	memset(right, 0, sizeof(*right));
+
+	return 1;
+}
+
 static struct user_mem_map *
 find_user_mem_map(struct user_mem_maps *user_mem_maps, uint64_t addr,
 		uint64_t iova, uint64_t len)
@@ -232,7 +245,7 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
 		if (is_null_map(l) || is_null_map(r))
 			continue;
 
-		if (merge_map(l, r))
+		if (merge_map(l, r) || merge_same_map(l, r))
 			n_merged++;
 	}

Thanks,
Yunjian

> 
> >
> > @@ -1828,6 +1828,13 @@  container_dma_map(struct vfio_config
> *vfio_cfg, uint64_t vaddr, uint64_t iova,
> >   		ret = -1;
> >   		goto out;
> >   	}
> > +
> > +	/* we don't need create new user mem map entry
> > +	 * for the same memory segment.
> > +	 */
> > +	if (errno == EBUSY || errno == EEXIST)
> > +		goto out;
> > +
> >   	/* create new user mem map entry */
> >   	new_map =
> &user_mem_maps->maps[user_mem_maps->n_maps++];
> >   	new_map->addr = vaddr;
> >
> > Thanks,
> > Yunjian
> >>
> >>>    	if (left->addr + left->len != right->addr)
> >>>    		return 0;
> >>>    	if (left->iova + left->len != right->iova) @@ -174,6 +178,7 @@
> >>> merge_map(struct user_mem_map *left, struct
> >> user_mem_map *right)
> >>>
> >>>    	left->len += right->len;
> >>>
> >>> +out:
> >>>    	memset(right, 0, sizeof(*right));
> >>>
> >>>    	return 1;
> >>>
> >>
> >>
> >> --
> >> Thanks,
> >> Anatoly
> 
> 
> --
> Thanks,
> Anatoly
Burakov, Anatoly July 31, 2020, 11:55 a.m. UTC | #5 
On 30-Jul-20 2:16 PM, wangyunjian wrote:
>> -----Original Message-----
>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
>> Sent: Monday, July 27, 2020 5:24 PM
>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
>> david.marchand@redhat.com
>> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
>> <xudingke@huawei.com>; stable@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
>> repeatedly when it exists
>>
>> On 25-Jul-20 10:59 AM, wangyunjian wrote:
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
>>>> Sent: Friday, July 24, 2020 9:25 PM
>>>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
>>>> david.marchand@redhat.com
>>>> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
>>>> <xudingke@huawei.com>; stable@dpdk.org
>>>> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem
>>>> map repeatedly when it exists
>>>>
>>>> On 23-Jul-20 3:48 PM, wangyunjian wrote:
>>>>> From: Yunjian Wang <wangyunjian@huawei.com>
>>>>>
>>>>> Currently, we will create new user mem map entry for the same memory
>>>>> segment, but in fact it has already been added to the user mem maps.
>>>>> It's not necessary to create it twice.
>>>>>
>>>>> To resolve the issue, add support to remove the same entry in the
>>>>> function compact_user_maps().
>>>>>
>>>>> Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
>>>>> Cc: stable@dpdk.org
>>>>>
>>>>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
>>>>> ---
>>>>> v2:
>>>>> * Remove the same entry in the function compact_user_maps()
>>>>> ---
>>>>>     lib/librte_eal/linux/eal_vfio.c | 5 +++++
>>>>>     1 file changed, 5 insertions(+)
>>>>>
>>>>> diff --git a/lib/librte_eal/linux/eal_vfio.c
>>>>> b/lib/librte_eal/linux/eal_vfio.c index abb12a354..df99307b7 100644
>>>>> --- a/lib/librte_eal/linux/eal_vfio.c
>>>>> +++ b/lib/librte_eal/linux/eal_vfio.c
>>>>> @@ -167,6 +167,10 @@ adjust_map(struct user_mem_map *src, struct
>>>> user_mem_map *end,
>>>>>     static int
>>>>>     merge_map(struct user_mem_map *left, struct user_mem_map
>> *right)
>>>>>     {
>>>>> +	/* merge the same maps into one */
>>>>> +	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
>>>>> +		goto out;
>>>>> +
>>>>
>>>> merge_map looks for adjacent maps only, but does not handle maps that
>>>> are wholly contained within one another ("the same map" also matches
>>>> this definition). wouldn't it be better to check for that instead of
>>>> *just* handling identical maps?
>>>
>>> What about using the initial implementation?
>>> We don't create new user mem map entry for the same memory segment.
>>
>> I don't like this implementation because it relies on particulars of how VFIO
>> mapping work without explicitly specifying them. I.e. it's prone to breaking
>> when changing code. That's not even mentioning that we have no guarantees
>> on kernel behavior in that particular case being identical on all supported
>> platforms.
>>
>> I would honestly prefer an explicit compaction over implicit one.
> 
> What about this implementation?

Again, this works, but i feel like specializing it to just merge the 
exact same maps is missing an opportunity to provide a more general 
solution that merges same *and* subset maps.

> 
> diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> index e07979936..8dcb04cd9 100644
> --- a/lib/librte_eal/linux/eal_vfio.c
> +++ b/lib/librte_eal/linux/eal_vfio.c
> @@ -179,6 +179,19 @@ merge_map(struct user_mem_map *left, struct user_mem_map *right)
>   	return 1;
>   }
>   
> +/* try merging two same maps into one, return 1 if succeeded */
> +static int
> +merge_same_map(struct user_mem_map *left, struct user_mem_map *right)
> +{
> +	if (memcmp(left, right, sizeof(struct user_mem_map)) != 0) {
> +		return 0;
> +	}
> +
> +	memset(right, 0, sizeof(*right));
> +
> +	return 1;
> +}
> +
>   static struct user_mem_map *
>   find_user_mem_map(struct user_mem_maps *user_mem_maps, uint64_t addr,
>   		uint64_t iova, uint64_t len)
> @@ -232,7 +245,7 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
>   		if (is_null_map(l) || is_null_map(r))
>   			continue;
>   
> -		if (merge_map(l, r))
> +		if (merge_map(l, r) || merge_same_map(l, r))
>   			n_merged++;
>   	}
> 
> Thanks,
> Yunjian
> 
>>
>>>
>>> @@ -1828,6 +1828,13 @@  container_dma_map(struct vfio_config
>> *vfio_cfg, uint64_t vaddr, uint64_t iova,
>>>    		ret = -1;
>>>    		goto out;
>>>    	}
>>> +
>>> +	/* we don't need create new user mem map entry
>>> +	 * for the same memory segment.
>>> +	 */
>>> +	if (errno == EBUSY || errno == EEXIST)
>>> +		goto out;
>>> +
>>>    	/* create new user mem map entry */
>>>    	new_map =
>> &user_mem_maps->maps[user_mem_maps->n_maps++];
>>>    	new_map->addr = vaddr;
>>>
>>> Thanks,
>>> Yunjian
>>>>
>>>>>     	if (left->addr + left->len != right->addr)
>>>>>     		return 0;
>>>>>     	if (left->iova + left->len != right->iova) @@ -174,6 +178,7 @@
>>>>> merge_map(struct user_mem_map *left, struct
>>>> user_mem_map *right)
>>>>>
>>>>>     	left->len += right->len;
>>>>>
>>>>> +out:
>>>>>     	memset(right, 0, sizeof(*right));
>>>>>
>>>>>     	return 1;
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Anatoly
>>
>>
>> --
>> Thanks,
>> Anatoly
Yunjian Wang Aug. 5, 2020, 12:58 p.m. UTC | #6 
> -----Original Message-----
> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> Sent: Friday, July 31, 2020 7:55 PM
> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
> david.marchand@redhat.com
> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
> <xudingke@huawei.com>; stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
> repeatedly when it exists
> 
> On 30-Jul-20 2:16 PM, wangyunjian wrote:
> >> -----Original Message-----
> >> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> >> Sent: Monday, July 27, 2020 5:24 PM
> >> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
> >> david.marchand@redhat.com
> >> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
> >> <xudingke@huawei.com>; stable@dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
> >> repeatedly when it exists
> >>
> >> On 25-Jul-20 10:59 AM, wangyunjian wrote:
> >>>> -----Original Message-----
> >>>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> >>>> Sent: Friday, July 24, 2020 9:25 PM
> >>>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
> >>>> david.marchand@redhat.com
> >>>> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
> >>>> <xudingke@huawei.com>; stable@dpdk.org
> >>>> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem
> >>>> map repeatedly when it exists
> >>>>
> >>>> On 23-Jul-20 3:48 PM, wangyunjian wrote:
> >>>>> From: Yunjian Wang <wangyunjian@huawei.com>
> >>>>>
> >>>>> Currently, we will create new user mem map entry for the same memory
> >>>>> segment, but in fact it has already been added to the user mem maps.
> >>>>> It's not necessary to create it twice.
> >>>>>
> >>>>> To resolve the issue, add support to remove the same entry in the
> >>>>> function compact_user_maps().
> >>>>>
> >>>>> Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
> >>>>> Cc: stable@dpdk.org
> >>>>>
> >>>>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> >>>>> ---
> >>>>> v2:
> >>>>> * Remove the same entry in the function compact_user_maps()
> >>>>> ---
> >>>>>     lib/librte_eal/linux/eal_vfio.c | 5 +++++
> >>>>>     1 file changed, 5 insertions(+)
> >>>>>
> >>>>> diff --git a/lib/librte_eal/linux/eal_vfio.c
> >>>>> b/lib/librte_eal/linux/eal_vfio.c index abb12a354..df99307b7 100644
> >>>>> --- a/lib/librte_eal/linux/eal_vfio.c
> >>>>> +++ b/lib/librte_eal/linux/eal_vfio.c
> >>>>> @@ -167,6 +167,10 @@ adjust_map(struct user_mem_map *src,
> struct
> >>>> user_mem_map *end,
> >>>>>     static int
> >>>>>     merge_map(struct user_mem_map *left, struct user_mem_map
> >> *right)
> >>>>>     {
> >>>>> +	/* merge the same maps into one */
> >>>>> +	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
> >>>>> +		goto out;
> >>>>> +
> >>>>
> >>>> merge_map looks for adjacent maps only, but does not handle maps that
> >>>> are wholly contained within one another ("the same map" also matches
> >>>> this definition). wouldn't it be better to check for that instead of
> >>>> *just* handling identical maps?
> >>>
> >>> What about using the initial implementation?
> >>> We don't create new user mem map entry for the same memory segment.
> >>
> >> I don't like this implementation because it relies on particulars of how VFIO
> >> mapping work without explicitly specifying them. I.e. it's prone to breaking
> >> when changing code. That's not even mentioning that we have no
> guarantees
> >> on kernel behavior in that particular case being identical on all supported
> >> platforms.
> >>
> >> I would honestly prefer an explicit compaction over implicit one.
> >
> > What about this implementation?
> 
> Again, this works, but i feel like specializing it to just merge the
> exact same maps is missing an opportunity to provide a more general
> solution that merges same *and* subset maps.

Currently, the problem that I encounter is that a container has many
devices and the application will map the same memory many times.
The kernel driver returns EEXIST as long as there are overlapping memory
areas. Therefore, the application needs to ensure that the memory blocks
of the DMA do not overlap. Otherwise, it will not work normally.

Could you offer me some ideas or advise to fix it？

Thanks,
Yunjian
> 
> >
> > diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
> > index e07979936..8dcb04cd9 100644
> > --- a/lib/librte_eal/linux/eal_vfio.c
> > +++ b/lib/librte_eal/linux/eal_vfio.c
> > @@ -179,6 +179,19 @@ merge_map(struct user_mem_map *left, struct
> user_mem_map *right)
> >   	return 1;
> >   }
> >
> > +/* try merging two same maps into one, return 1 if succeeded */
> > +static int
> > +merge_same_map(struct user_mem_map *left, struct user_mem_map
> *right)
> > +{
> > +	if (memcmp(left, right, sizeof(struct user_mem_map)) != 0) {
> > +		return 0;
> > +	}
> > +
> > +	memset(right, 0, sizeof(*right));
> > +
> > +	return 1;
> > +}
> > +
> >   static struct user_mem_map *
> >   find_user_mem_map(struct user_mem_maps *user_mem_maps,
> uint64_t addr,
> >   		uint64_t iova, uint64_t len)
> > @@ -232,7 +245,7 @@ compact_user_maps(struct user_mem_maps
> *user_mem_maps)
> >   		if (is_null_map(l) || is_null_map(r))
> >   			continue;
> >
> > -		if (merge_map(l, r))
> > +		if (merge_map(l, r) || merge_same_map(l, r))
> >   			n_merged++;
> >   	}
> >
> > Thanks,
> > Yunjian
> >
> >>
> >>>
> >>> @@ -1828,6 +1828,13 @@  container_dma_map(struct vfio_config
> >> *vfio_cfg, uint64_t vaddr, uint64_t iova,
> >>>    		ret = -1;
> >>>    		goto out;
> >>>    	}
> >>> +
> >>> +	/* we don't need create new user mem map entry
> >>> +	 * for the same memory segment.
> >>> +	 */
> >>> +	if (errno == EBUSY || errno == EEXIST)
> >>> +		goto out;
> >>> +
> >>>    	/* create new user mem map entry */
> >>>    	new_map =
> >> &user_mem_maps->maps[user_mem_maps->n_maps++];
> >>>    	new_map->addr = vaddr;
> >>>
> >>> Thanks,
> >>> Yunjian
> >>>>
> >>>>>     	if (left->addr + left->len != right->addr)
> >>>>>     		return 0;
> >>>>>     	if (left->iova + left->len != right->iova) @@ -174,6 +178,7 @@
> >>>>> merge_map(struct user_mem_map *left, struct
> >>>> user_mem_map *right)
> >>>>>
> >>>>>     	left->len += right->len;
> >>>>>
> >>>>> +out:
> >>>>>     	memset(right, 0, sizeof(*right));
> >>>>>
> >>>>>     	return 1;
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Thanks,
> >>>> Anatoly
> >>
> >>
> >> --
> >> Thanks,
> >> Anatoly
> 
> 
> --
> Thanks,
> Anatoly
Burakov, Anatoly Sept. 17, 2020, 11:33 a.m. UTC | #7 
On 05-Aug-20 1:58 PM, wangyunjian wrote:
>> -----Original Message-----
>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
>> Sent: Friday, July 31, 2020 7:55 PM
>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
>> david.marchand@redhat.com
>> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
>> <xudingke@huawei.com>; stable@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
>> repeatedly when it exists
>>
>> On 30-Jul-20 2:16 PM, wangyunjian wrote:
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
>>>> Sent: Monday, July 27, 2020 5:24 PM
>>>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
>>>> david.marchand@redhat.com
>>>> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
>>>> <xudingke@huawei.com>; stable@dpdk.org
>>>> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
>>>> repeatedly when it exists
>>>>
>>>> On 25-Jul-20 10:59 AM, wangyunjian wrote:
>>>>>> -----Original Message-----
>>>>>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
>>>>>> Sent: Friday, July 24, 2020 9:25 PM
>>>>>> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
>>>>>> david.marchand@redhat.com
>>>>>> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
>>>>>> <xudingke@huawei.com>; stable@dpdk.org
>>>>>> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem
>>>>>> map repeatedly when it exists
>>>>>>
>>>>>> On 23-Jul-20 3:48 PM, wangyunjian wrote:
>>>>>>> From: Yunjian Wang <wangyunjian@huawei.com>
>>>>>>>
>>>>>>> Currently, we will create new user mem map entry for the same memory
>>>>>>> segment, but in fact it has already been added to the user mem maps.
>>>>>>> It's not necessary to create it twice.
>>>>>>>
>>>>>>> To resolve the issue, add support to remove the same entry in the
>>>>>>> function compact_user_maps().
>>>>>>>
>>>>>>> Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
>>>>>>> Cc: stable@dpdk.org
>>>>>>>
>>>>>>> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
>>>>>>> ---
>>>>>>> v2:
>>>>>>> * Remove the same entry in the function compact_user_maps()
>>>>>>> ---
>>>>>>>      lib/librte_eal/linux/eal_vfio.c | 5 +++++
>>>>>>>      1 file changed, 5 insertions(+)
>>>>>>>
>>>>>>> diff --git a/lib/librte_eal/linux/eal_vfio.c
>>>>>>> b/lib/librte_eal/linux/eal_vfio.c index abb12a354..df99307b7 100644
>>>>>>> --- a/lib/librte_eal/linux/eal_vfio.c
>>>>>>> +++ b/lib/librte_eal/linux/eal_vfio.c
>>>>>>> @@ -167,6 +167,10 @@ adjust_map(struct user_mem_map *src,
>> struct
>>>>>> user_mem_map *end,
>>>>>>>      static int
>>>>>>>      merge_map(struct user_mem_map *left, struct user_mem_map
>>>> *right)
>>>>>>>      {
>>>>>>> +	/* merge the same maps into one */
>>>>>>> +	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
>>>>>>> +		goto out;
>>>>>>> +
>>>>>>
>>>>>> merge_map looks for adjacent maps only, but does not handle maps that
>>>>>> are wholly contained within one another ("the same map" also matches
>>>>>> this definition). wouldn't it be better to check for that instead of
>>>>>> *just* handling identical maps?
>>>>>
>>>>> What about using the initial implementation?
>>>>> We don't create new user mem map entry for the same memory segment.
>>>>
>>>> I don't like this implementation because it relies on particulars of how VFIO
>>>> mapping work without explicitly specifying them. I.e. it's prone to breaking
>>>> when changing code. That's not even mentioning that we have no
>> guarantees
>>>> on kernel behavior in that particular case being identical on all supported
>>>> platforms.
>>>>
>>>> I would honestly prefer an explicit compaction over implicit one.
>>>
>>> What about this implementation?
>>
>> Again, this works, but i feel like specializing it to just merge the
>> exact same maps is missing an opportunity to provide a more general
>> solution that merges same *and* subset maps.
> 
> Currently, the problem that I encounter is that a container has many
> devices and the application will map the same memory many times.
> The kernel driver returns EEXIST as long as there are overlapping memory
> areas. Therefore, the application needs to ensure that the memory blocks
> of the DMA do not overlap. Otherwise, it will not work normally.
> 
> Could you offer me some ideas or advise to fix it？
> 

It sounds like your approach is better if that is indeed the case.
Burakov, Anatoly Sept. 17, 2020, 11:35 a.m. UTC | #8 
On 23-Jul-20 3:48 PM, wangyunjian wrote:
> From: Yunjian Wang <wangyunjian@huawei.com>
> 
> Currently, we will create new user mem map entry for the same memory
> segment, but in fact it has already been added to the user mem maps.
> It's not necessary to create it twice.
> 
> To resolve the issue, add support to remove the same entry in the
> function compact_user_maps().
> 
> Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> ---

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Yunjian Wang Oct. 15, 2020, 12:46 p.m. UTC | #9 
Hi David,

Can this patch be merged before RC1?

Thanks,
Yunjian

> -----Original Message-----
> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> Sent: Thursday, September 17, 2020 7:35 PM
> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org;
> david.marchand@redhat.com
> Cc: Lilijun (Jerry) <jerry.lilijun@huawei.com>; xudingke
> <xudingke@huawei.com>; stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
> repeatedly when it exists
> 
> On 23-Jul-20 3:48 PM, wangyunjian wrote:
> > From: Yunjian Wang <wangyunjian@huawei.com>
> >
> > Currently, we will create new user mem map entry for the same memory
> > segment, but in fact it has already been added to the user mem maps.
> > It's not necessary to create it twice.
> >
> > To resolve the issue, add support to remove the same entry in the
> > function compact_user_maps().
> >
> > Fixes: 0cbce3a167f1 ("vfio: skip DMA map failure if already mapped")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > ---
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
> --
> Thanks,
> Anatoly
David Marchand Oct. 15, 2020, 12:54 p.m. UTC | #10 
On Thu, Oct 15, 2020 at 2:46 PM wangyunjian <wangyunjian@huawei.com> wrote:
>
> Hi David,
>
> Can this patch be merged before RC1?

Trying to understand what this is about.
What is the impact from the issue you fix from an application point of view?

Getting a good title / description is important to help stable
maintainers later when judging whether they should pick it or not.
Thanks.
Yunjian Wang Oct. 16, 2020, 9:48 a.m. UTC | #11 
> -----Original Message-----
> From: David Marchand [mailto:david.marchand@redhat.com]
> Sent: Thursday, October 15, 2020 8:54 PM
> To: wangyunjian <wangyunjian@huawei.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Lilijun
> (Jerry) <jerry.lilijun@huawei.com>; xudingke <xudingke@huawei.com>;
> stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] eal/linux: do not create user mem map
> repeatedly when it exists
> 
> On Thu, Oct 15, 2020 at 2:46 PM wangyunjian <wangyunjian@huawei.com>
> wrote:
> >
> > Hi David,
> >
> > Can this patch be merged before RC1?
> 
> Trying to understand what this is about.
> What is the impact from the issue you fix from an application point of view?
> 
> Getting a good title / description is important to help stable maintainers later
> when judging whether they should pick it or not.
> Thanks.

Thanks for your suggestion, I have updated commit log and title in new version.

https://patchwork.dpdk.org/patch/81069/

Yunjian
> 
> --
> David Marchand
diff mbox series

Patch

diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_vfio.c
index abb12a354..df99307b7 100644
--- a/lib/librte_eal/linux/eal_vfio.c
+++ b/lib/librte_eal/linux/eal_vfio.c
@@ -167,6 +167,10 @@  adjust_map(struct user_mem_map *src, struct user_mem_map *end,
 static int
 merge_map(struct user_mem_map *left, struct user_mem_map *right)
 {
+	/* merge the same maps into one */
+	if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
+		goto out;
+
 	if (left->addr + left->len != right->addr)
 		return 0;
 	if (left->iova + left->len != right->iova)
@@ -174,6 +178,7 @@  merge_map(struct user_mem_map *left, struct user_mem_map *right)
 
 	left->len += right->len;
 
+out:
 	memset(right, 0, sizeof(*right));
 
 	return 1;