[1/2] doc: announce region based device mapping support

  From: Sunil Kumar Kori <skori@marvell.com>

Adding region based device mapping support, which enables pci device
to map only required memory region instead of mapping full BAR.

Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
---
 doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
 1 file changed, 13 insertions(+)
  

Please look into following deprecation notice and ack if looks okay.

> -----Original Message-----
> From: skori@marvell.com <skori@marvell.com>
> Sent: Tuesday, June 28, 2022 7:24 PM
> To: Ray Kinsella <mdr@ashroe.eu>
> Cc: dev@dpdk.org; Sunil Kumar Kori <skori@marvell.com>
> Subject: [PATCH 1/2] doc: announce region based device mapping support
> 
> From: Sunil Kumar Kori <skori@marvell.com>
> 
> Adding region based device mapping support, which enables pci device to
> map only required memory region instead of mapping full BAR.
> 
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> ---
>  doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> index 4e5b23c53d..8800a3eb41 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -125,3 +125,16 @@ Deprecation Notices
>    applications should be updated to use the ``dmadev`` library instead,
>    with the underlying HW-functionality being provided by the ``ioat`` or
>    ``idxd`` dma drivers
> +
> +* pci: Update ``rte_pci_device`` and ``rte_pci_driver`` to add region
> +based
> +  memory mapping support. There could be a requirement to mmap specific
> +memory
> +  region only. Using this mechanism, pci device can be mapped for
> +  a given BAR at a given offset of given size.
> +
> +  ``rte_pci_device`` will be added with following field
> + ``regions[PCI_MAX_RESOURCE][PCI_MAX_REGION_PER_RESOURCE];``. This
> + field will  specify the regions which are mapped for a given BAR.
> +
> +  ``rte_pci_driver`` will be added with ``rte_pci_region_map *regions``
> + and  ``valid_bars[PCI_MAX_RESOURCE]``. Using these fields, driver can
> + propagate  its region information which are required to be mmap.
> --
> 2.25.1
  

Hi Sunil,

> -----Original Message-----
> From: skori@marvell.com <skori@marvell.com>
> Sent: Tuesday, June 28, 2022 9:54 PM
> To: Ray Kinsella <mdr@ashroe.eu>
> Cc: dev@dpdk.org; Sunil Kumar Kori <skori@marvell.com>
> Subject: [PATCH 1/2] doc: announce region based device mapping support
> 
> From: Sunil Kumar Kori <skori@marvell.com>
> 
> Adding region based device mapping support, which enables pci device
> to map only required memory region instead of mapping full BAR.

Why there will be such requirement to mmap only part of BAR when the BAR
can be fully mmapped. I thought you want to enable something like sparse
mmap feature in VFIO, but seems it's a feature that let driver ask for
specific mmap option. Could you explain the use case here?

Thanks,
Chenbo

> 
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> ---
>  doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> index 4e5b23c53d..8800a3eb41 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -125,3 +125,16 @@ Deprecation Notices
>    applications should be updated to use the ``dmadev`` library instead,
>    with the underlying HW-functionality being provided by the ``ioat`` or
>    ``idxd`` dma drivers
> +
> +* pci: Update ``rte_pci_device`` and ``rte_pci_driver`` to add region
> based
> +  memory mapping support. There could be a requirement to mmap specific
> memory
> +  region only. Using this mechanism, pci device can be mapped for
> +  a given BAR at a given offset of given size.
> +
> +  ``rte_pci_device`` will be added with following field
> +  ``regions[PCI_MAX_RESOURCE][PCI_MAX_REGION_PER_RESOURCE];``. This field
> will
> +  specify the regions which are mapped for a given BAR.
> +
> +  ``rte_pci_driver`` will be added with ``rte_pci_region_map *regions``
> and
> +  ``valid_bars[PCI_MAX_RESOURCE]``. Using these fields, driver can
> propagate
> +  its region information which are required to be mmap.
> --
> 2.25.1
  

Hi Chenbo,

There are pci devices which has huge BAR memory but might not be used completely.
Like PCIe device DDR memory is exposed over BAR with 32Gb or so.
So in that case user might have a requirement to mmap only areas of its interest.
Also avoid wasting of extra memory for each pages when kernel mmaps it.

Yes, it is similar to sparse mmap in vfio but I didn't find any support in DPDK for sparse mmap. 
So I picked this approach but in either case, ABI change notification is needed, Right?

Regards
Sunil Kumar Kori

> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Thursday, July 7, 2022 6:09 PM
> To: Sunil Kumar Kori <skori@marvell.com>; Ray Kinsella <mdr@ashroe.eu>
> Cc: dev@dpdk.org
> Subject: [EXT] RE: [PATCH 1/2] doc: announce region based device mapping
> support
> 
> External Email
> 
> ----------------------------------------------------------------------
> Hi Sunil,
> 
> > -----Original Message-----
> > From: skori@marvell.com <skori@marvell.com>
> > Sent: Tuesday, June 28, 2022 9:54 PM
> > To: Ray Kinsella <mdr@ashroe.eu>
> > Cc: dev@dpdk.org; Sunil Kumar Kori <skori@marvell.com>
> > Subject: [PATCH 1/2] doc: announce region based device mapping support
> >
> > From: Sunil Kumar Kori <skori@marvell.com>
> >
> > Adding region based device mapping support, which enables pci device
> > to map only required memory region instead of mapping full BAR.
> 
> Why there will be such requirement to mmap only part of BAR when the BAR
> can be fully mmapped. I thought you want to enable something like sparse
> mmap feature in VFIO, but seems it's a feature that let driver ask for specific
> mmap option. Could you explain the use case here?
> 
> Thanks,
> Chenbo
> 
> >
> > Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> > ---
> >  doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/deprecation.rst
> > b/doc/guides/rel_notes/deprecation.rst
> > index 4e5b23c53d..8800a3eb41 100644
> > --- a/doc/guides/rel_notes/deprecation.rst
> > +++ b/doc/guides/rel_notes/deprecation.rst
> > @@ -125,3 +125,16 @@ Deprecation Notices
> >    applications should be updated to use the ``dmadev`` library instead,
> >    with the underlying HW-functionality being provided by the ``ioat`` or
> >    ``idxd`` dma drivers
> > +
> > +* pci: Update ``rte_pci_device`` and ``rte_pci_driver`` to add region
> > based
> > +  memory mapping support. There could be a requirement to mmap
> > + specific
> > memory
> > +  region only. Using this mechanism, pci device can be mapped for  a
> > + given BAR at a given offset of given size.
> > +
> > +  ``rte_pci_device`` will be added with following field
> > + ``regions[PCI_MAX_RESOURCE][PCI_MAX_REGION_PER_RESOURCE];``.
> This
> > + field
> > will
> > +  specify the regions which are mapped for a given BAR.
> > +
> > +  ``rte_pci_driver`` will be added with ``rte_pci_region_map
> > + *regions``
> > and
> > +  ``valid_bars[PCI_MAX_RESOURCE]``. Using these fields, driver can
> > propagate
> > +  its region information which are required to be mmap.
> > --
> > 2.25.1
  

On Tue, Jun 28, 2022 at 3:56 PM <skori@marvell.com> wrote:
>
> From: Sunil Kumar Kori <skori@marvell.com>
>
> Adding region based device mapping support, which enables pci device
> to map only required memory region instead of mapping full BAR.
>
> Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> ---
>  doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 4e5b23c53d..8800a3eb41 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -125,3 +125,16 @@ Deprecation Notices
>    applications should be updated to use the ``dmadev`` library instead,
>    with the underlying HW-functionality being provided by the ``ioat`` or
>    ``idxd`` dma drivers
> +
> +* pci: Update ``rte_pci_device`` and ``rte_pci_driver`` to add region based
> +  memory mapping support. There could be a requirement to mmap specific memory
> +  region only. Using this mechanism, pci device can be mapped for
> +  a given BAR at a given offset of given size.
> +
> +  ``rte_pci_device`` will be added with following field
> +  ``regions[PCI_MAX_RESOURCE][PCI_MAX_REGION_PER_RESOURCE];``. This field will
> +  specify the regions which are mapped for a given BAR.
> +
> +  ``rte_pci_driver`` will be added with ``rte_pci_region_map *regions`` and
> +  ``valid_bars[PCI_MAX_RESOURCE]``. Using these fields, driver can propagate
> +  its region information which are required to be mmap.

I don't think we need a new deprecation notice for this.
We still have Chenbo current notice:
https://git.dpdk.org/dpdk/tree/doc/guides/rel_notes/deprecation.rst#n46

Besides, my proposal hides every bus specific device representations,
and I think it covers your needs too.
See https://patches.dpdk.org/project/dpdk/patch/20220710061736.973563-1-david.marchand@redhat.com/

Did I miss something?
  

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, July 13, 2022 3:18 PM
> To: Sunil Kumar Kori <skori@marvell.com>
> Cc: Ray Kinsella <mdr@ashroe.eu>; dev <dev@dpdk.org>; Xia, Chenbo
> <chenbo.xia@intel.com>; Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [PATCH 1/2] doc: announce region based device mapping support
> 
> On Tue, Jun 28, 2022 at 3:56 PM <skori@marvell.com> wrote:
> >
> > From: Sunil Kumar Kori <skori@marvell.com>
> >
> > Adding region based device mapping support, which enables pci device
> > to map only required memory region instead of mapping full BAR.
> >
> > Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> > ---
> >  doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> > index 4e5b23c53d..8800a3eb41 100644
> > --- a/doc/guides/rel_notes/deprecation.rst
> > +++ b/doc/guides/rel_notes/deprecation.rst
> > @@ -125,3 +125,16 @@ Deprecation Notices
> >    applications should be updated to use the ``dmadev`` library instead,
> >    with the underlying HW-functionality being provided by the ``ioat``
> or
> >    ``idxd`` dma drivers
> > +
> > +* pci: Update ``rte_pci_device`` and ``rte_pci_driver`` to add region
> based
> > +  memory mapping support. There could be a requirement to mmap specific
> memory
> > +  region only. Using this mechanism, pci device can be mapped for
> > +  a given BAR at a given offset of given size.
> > +
> > +  ``rte_pci_device`` will be added with following field
> > +  ``regions[PCI_MAX_RESOURCE][PCI_MAX_REGION_PER_RESOURCE];``. This
> field will
> > +  specify the regions which are mapped for a given BAR.
> > +
> > +  ``rte_pci_driver`` will be added with ``rte_pci_region_map *regions``
> and
> > +  ``valid_bars[PCI_MAX_RESOURCE]``. Using these fields, driver can
> propagate
> > +  its region information which are required to be mmap.
> 
> I don't think we need a new deprecation notice for this.
> We still have Chenbo current notice:
> https://git.dpdk.org/dpdk/tree/doc/guides/rel_notes/deprecation.rst#n46
> 
> Besides, my proposal hides every bus specific device representations,
> and I think it covers your needs too.
> See https://patches.dpdk.org/project/dpdk/patch/20220710061736.973563-1-
> david.marchand@redhat.com/
> 
> Did I miss something?

Agree, I am about to say this :)

Sunil, with David's patchset, you can just send your patchset. David will help
clean-up the ABI for you.

Thanks,
Chenbo

> 
> 
> --
> David Marchand
  

Hi Sunil,

> -----Original Message-----
> From: Sunil Kumar Kori <skori@marvell.com>
> Sent: Wednesday, July 13, 2022 3:04 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>; Ray Kinsella <mdr@ashroe.eu>
> Cc: dev@dpdk.org
> Subject: RE: [PATCH 1/2] doc: announce region based device mapping support
> 
> Hi Chenbo,
> 
> There are pci devices which has huge BAR memory but might not be used
> completely.
> Like PCIe device DDR memory is exposed over BAR with 32Gb or so.
> So in that case user might have a requirement to mmap only areas of its
> interest.
> Also avoid wasting of extra memory for each pages when kernel mmaps it.

This problem statement makes sense to me. About the solution, pls check below.

> 
> Yes, it is similar to sparse mmap in vfio but I didn't find any support in
> DPDK for sparse mmap.
> So I picked this approach but in either case, ABI change notification is
> needed, Right?

For ABI, David & I already mentioned in another email that your notice is not
needed anymore.

What if you send the patchset to support sparse mmap in DPDK, to me, that seems
more generic. Could sparse mmap solve your problem?

Thanks,
Chenbo

> 
> Regards
> Sunil Kumar Kori
> 
> > -----Original Message-----
> > From: Xia, Chenbo <chenbo.xia@intel.com>
> > Sent: Thursday, July 7, 2022 6:09 PM
> > To: Sunil Kumar Kori <skori@marvell.com>; Ray Kinsella <mdr@ashroe.eu>
> > Cc: dev@dpdk.org
> > Subject: [EXT] RE: [PATCH 1/2] doc: announce region based device mapping
> > support
> >
> > External Email
> >
> > ----------------------------------------------------------------------
> > Hi Sunil,
> >
> > > -----Original Message-----
> > > From: skori@marvell.com <skori@marvell.com>
> > > Sent: Tuesday, June 28, 2022 9:54 PM
> > > To: Ray Kinsella <mdr@ashroe.eu>
> > > Cc: dev@dpdk.org; Sunil Kumar Kori <skori@marvell.com>
> > > Subject: [PATCH 1/2] doc: announce region based device mapping support
> > >
> > > From: Sunil Kumar Kori <skori@marvell.com>
> > >
> > > Adding region based device mapping support, which enables pci device
> > > to map only required memory region instead of mapping full BAR.
> >
> > Why there will be such requirement to mmap only part of BAR when the BAR
> > can be fully mmapped. I thought you want to enable something like sparse
> > mmap feature in VFIO, but seems it's a feature that let driver ask for
> specific
> > mmap option. Could you explain the use case here?
> >
> > Thanks,
> > Chenbo
> >
> > >
> > > Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> > > ---
> > >  doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
> > >  1 file changed, 13 insertions(+)
> > >
> > > diff --git a/doc/guides/rel_notes/deprecation.rst
> > > b/doc/guides/rel_notes/deprecation.rst
> > > index 4e5b23c53d..8800a3eb41 100644
> > > --- a/doc/guides/rel_notes/deprecation.rst
> > > +++ b/doc/guides/rel_notes/deprecation.rst
> > > @@ -125,3 +125,16 @@ Deprecation Notices
> > >    applications should be updated to use the ``dmadev`` library
> instead,
> > >    with the underlying HW-functionality being provided by the ``ioat``
> or
> > >    ``idxd`` dma drivers
> > > +
> > > +* pci: Update ``rte_pci_device`` and ``rte_pci_driver`` to add region
> > > based
> > > +  memory mapping support. There could be a requirement to mmap
> > > + specific
> > > memory
> > > +  region only. Using this mechanism, pci device can be mapped for  a
> > > + given BAR at a given offset of given size.
> > > +
> > > +  ``rte_pci_device`` will be added with following field
> > > + ``regions[PCI_MAX_RESOURCE][PCI_MAX_REGION_PER_RESOURCE];``.
> > This
> > > + field
> > > will
> > > +  specify the regions which are mapped for a given BAR.
> > > +
> > > +  ``rte_pci_driver`` will be added with ``rte_pci_region_map
> > > + *regions``
> > > and
> > > +  ``valid_bars[PCI_MAX_RESOURCE]``. Using these fields, driver can
> > > propagate
> > > +  its region information which are required to be mmap.
> > > --
> > > 2.25.1
  

Regards
Sunil Kumar Kori

> -----Original Message-----
> From: Xia, Chenbo <chenbo.xia@intel.com>
> Sent: Wednesday, July 13, 2022 12:58 PM
> To: Sunil Kumar Kori <skori@marvell.com>; Ray Kinsella <mdr@ashroe.eu>
> Cc: dev@dpdk.org; David Marchand <david.marchand@redhat.com>
> Subject: [EXT] RE: [PATCH 1/2] doc: announce region based device mapping
> support
> 
> External Email
> 
> ----------------------------------------------------------------------
> Hi Sunil,
> 
> > -----Original Message-----
> > From: Sunil Kumar Kori <skori@marvell.com>
> > Sent: Wednesday, July 13, 2022 3:04 PM
> > To: Xia, Chenbo <chenbo.xia@intel.com>; Ray Kinsella <mdr@ashroe.eu>
> > Cc: dev@dpdk.org
> > Subject: RE: [PATCH 1/2] doc: announce region based device mapping
> > support
> >
> > Hi Chenbo,
> >
> > There are pci devices which has huge BAR memory but might not be used
> > completely.
> > Like PCIe device DDR memory is exposed over BAR with 32Gb or so.
> > So in that case user might have a requirement to mmap only areas of
> > its interest.
> > Also avoid wasting of extra memory for each pages when kernel mmaps it.
> 
> This problem statement makes sense to me. About the solution, pls check
> below.
> 
> >
> > Yes, it is similar to sparse mmap in vfio but I didn't find any
> > support in DPDK for sparse mmap.
> > So I picked this approach but in either case, ABI change notification
> > is needed, Right?
> 
> For ABI, David & I already mentioned in another email that your notice is not
> needed anymore.
> 
> What if you send the patchset to support sparse mmap in DPDK, to me, that
> seems more generic. Could sparse mmap solve your problem?

I will look into sparse mmap and analyze its feasibility to solve my problem.
Will update accordingly. 

Thanks Chenbo and David.
> 
> Thanks,
> Chenbo
> 
> >
> > Regards
> > Sunil Kumar Kori
> >
> > > -----Original Message-----
> > > From: Xia, Chenbo <chenbo.xia@intel.com>
> > > Sent: Thursday, July 7, 2022 6:09 PM
> > > To: Sunil Kumar Kori <skori@marvell.com>; Ray Kinsella
> > > <mdr@ashroe.eu>
> > > Cc: dev@dpdk.org
> > > Subject: [EXT] RE: [PATCH 1/2] doc: announce region based device
> > > mapping support
> > >
> > > External Email
> > >
> > > --------------------------------------------------------------------
> > > --
> > > Hi Sunil,
> > >
> > > > -----Original Message-----
> > > > From: skori@marvell.com <skori@marvell.com>
> > > > Sent: Tuesday, June 28, 2022 9:54 PM
> > > > To: Ray Kinsella <mdr@ashroe.eu>
> > > > Cc: dev@dpdk.org; Sunil Kumar Kori <skori@marvell.com>
> > > > Subject: [PATCH 1/2] doc: announce region based device mapping
> > > > support
> > > >
> > > > From: Sunil Kumar Kori <skori@marvell.com>
> > > >
> > > > Adding region based device mapping support, which enables pci
> > > > device to map only required memory region instead of mapping full
> BAR.
> > >
> > > Why there will be such requirement to mmap only part of BAR when the
> > > BAR can be fully mmapped. I thought you want to enable something
> > > like sparse mmap feature in VFIO, but seems it's a feature that let
> > > driver ask for
> > specific
> > > mmap option. Could you explain the use case here?
> > >
> > > Thanks,
> > > Chenbo
> > >
> > > >
> > > > Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
> > > > ---
> > > >  doc/guides/rel_notes/deprecation.rst | 13 +++++++++++++
> > > >  1 file changed, 13 insertions(+)
> > > >
> > > > diff --git a/doc/guides/rel_notes/deprecation.rst
> > > > b/doc/guides/rel_notes/deprecation.rst
> > > > index 4e5b23c53d..8800a3eb41 100644
> > > > --- a/doc/guides/rel_notes/deprecation.rst
> > > > +++ b/doc/guides/rel_notes/deprecation.rst
> > > > @@ -125,3 +125,16 @@ Deprecation Notices
> > > >    applications should be updated to use the ``dmadev`` library
> > instead,
> > > >    with the underlying HW-functionality being provided by the
> > > > ``ioat``
> > or
> > > >    ``idxd`` dma drivers
> > > > +
> > > > +* pci: Update ``rte_pci_device`` and ``rte_pci_driver`` to add
> > > > +region
> > > > based
> > > > +  memory mapping support. There could be a requirement to mmap
> > > > + specific
> > > > memory
> > > > +  region only. Using this mechanism, pci device can be mapped for
> > > > + a given BAR at a given offset of given size.
> > > > +
> > > > +  ``rte_pci_device`` will be added with following field
> > > > + ``regions[PCI_MAX_RESOURCE][PCI_MAX_REGION_PER_RESOURCE];``.
> > > This
> > > > + field
> > > > will
> > > > +  specify the regions which are mapped for a given BAR.
> > > > +
> > > > +  ``rte_pci_driver`` will be added with ``rte_pci_region_map
> > > > + *regions``
> > > > and
> > > > +  ``valid_bars[PCI_MAX_RESOURCE]``. Using these fields, driver
> > > > + can
> > > > propagate
> > > > +  its region information which are required to be mmap.
> > > > --
> > > > 2.25.1