[v2,00/25] add the NFP vDPA PMD

  This patch series aims to add the NFP vDPA PMD, we also grab the common
logic into the `drivers/common/nfp` directory.

---
v2:
* Grab more logic into the `drivers/common/nfp` directory.
* Delete some logic which should be when moving logic.
---

Chaoyong He (25):
  drivers: introduce the NFP common library
  net/nfp: make VF PMD using of NFP common module
  net/nfp: rename common module name
  net/nfp: rename ctrl module name
  net/nfp: extract the cap data field
  net/nfp: extract the qcp data field
  net/nfp: extract the ctrl BAR data field
  net/nfp: extract the ctrl data field
  net/nfp: change the parameter of APIs
  net/nfp: change the parameter of reconfig
  net/nfp: extract the MAC address data field
  net/nfp: rename parameter in related logic
  drivers: add the common ctrl module
  drivers: add the nfp common module
  drivers: move queue logic to common module
  drivers: move platform module to common library
  drivers: move device module to common library
  drivers/vdpa: introduce the NFP vDPA library
  drivers: add the basic framework of vDPA PMD
  vdpa/nfp: add the logic of remap PCI memory
  vdpa/nfp: add the hardware init logic
  drivers: add the datapath update logic
  vdpa/nfp: add the notify related logic
  vdpa/nfp: add nfp vDPA device operations
  doc: add the common and vDPA document

 .mailmap                                      |   1 +
 MAINTAINERS                                   |   8 +
 doc/guides/platform/index.rst                 |   1 +
 doc/guides/platform/nfp.rst                   |  30 +
 doc/guides/rel_notes/release_23_11.rst        |   5 +
 doc/guides/vdpadevs/features/nfp.ini          |   8 +
 doc/guides/vdpadevs/index.rst                 |   1 +
 doc/guides/vdpadevs/nfp.rst                   |  54 ++
 drivers/common/nfp/meson.build                |  16 +
 drivers/common/nfp/nfp_common.c               | 226 +++++
 drivers/common/nfp/nfp_common.h               | 239 +++++
 .../nfp/nfp_common_ctrl.h}                    | 209 +---
 drivers/common/nfp/nfp_common_log.c           |   8 +
 drivers/common/nfp/nfp_common_log.h           |  16 +
 drivers/common/nfp/nfp_common_pci.c           | 275 ++++++
 drivers/common/nfp/nfp_common_pci.h           |  45 +
 .../{net/nfp/nfpcore => common/nfp}/nfp_dev.c |   3 +-
 .../{net/nfp/nfpcore => common/nfp}/nfp_dev.h |   3 +
 .../nfp/nfpcore => common/nfp}/nfp_platform.h |   0
 drivers/common/nfp/version.map                |  16 +
 drivers/meson.build                           |   1 +
 drivers/net/nfp/flower/nfp_flower.c           |  56 +-
 drivers/net/nfp/flower/nfp_flower.h           |   2 +-
 .../net/nfp/flower/nfp_flower_representor.c   |  10 +-
 drivers/net/nfp/meson.build                   |  11 +-
 drivers/net/nfp/nfd3/nfp_nfd3_dp.c            |  18 +-
 drivers/net/nfp/nfdk/nfp_nfdk_dp.c            |  22 +-
 drivers/net/nfp/nfp_cpp_bridge.h              |   2 +-
 drivers/net/nfp/nfp_ethdev.c                  |  80 +-
 drivers/net/nfp/nfp_ethdev_vf.c               |  74 +-
 drivers/net/nfp/nfp_flow.h                    |   2 +-
 drivers/net/nfp/nfp_ipsec.c                   |  86 +-
 .../nfp/{nfp_common.c => nfp_net_common.c}    | 493 ++++------
 .../nfp/{nfp_common.h => nfp_net_common.h}    | 219 +----
 .../net/nfp/{nfp_ctrl.c => nfp_net_ctrl.c}    |  21 +-
 drivers/net/nfp/nfp_net_ctrl.h                | 211 ++++
 drivers/net/nfp/nfp_rxtx.c                    |  20 +-
 drivers/net/nfp/nfpcore/nfp6000_pcie.h        |   2 +-
 drivers/net/nfp/nfpcore/nfp_cppcore.c         |   3 +-
 drivers/net/nfp/nfpcore/nfp_nsp.c             |   2 +-
 drivers/net/nfp/nfpcore/nfp_nsp_eth.c         |   3 +-
 drivers/vdpa/meson.build                      |   1 +
 drivers/vdpa/nfp/meson.build                  |  24 +
 drivers/vdpa/nfp/nfp_vdpa.c                   | 909 ++++++++++++++++++
 drivers/vdpa/nfp/nfp_vdpa_core.c              | 193 ++++
 drivers/vdpa/nfp/nfp_vdpa_core.h              |  51 +
 drivers/vdpa/nfp/nfp_vdpa_log.c               |   9 +
 drivers/vdpa/nfp/nfp_vdpa_log.h               |  21 +
 48 files changed, 2769 insertions(+), 941 deletions(-)
 create mode 100644 doc/guides/platform/nfp.rst
 create mode 100644 doc/guides/vdpadevs/features/nfp.ini
 create mode 100644 doc/guides/vdpadevs/nfp.rst
 create mode 100644 drivers/common/nfp/meson.build
 create mode 100644 drivers/common/nfp/nfp_common.c
 create mode 100644 drivers/common/nfp/nfp_common.h
 rename drivers/{net/nfp/nfp_ctrl.h => common/nfp/nfp_common_ctrl.h} (63%)
 create mode 100644 drivers/common/nfp/nfp_common_log.c
 create mode 100644 drivers/common/nfp/nfp_common_log.h
 create mode 100644 drivers/common/nfp/nfp_common_pci.c
 create mode 100644 drivers/common/nfp/nfp_common_pci.h
 rename drivers/{net/nfp/nfpcore => common/nfp}/nfp_dev.c (98%)
 rename drivers/{net/nfp/nfpcore => common/nfp}/nfp_dev.h (96%)
 rename drivers/{net/nfp/nfpcore => common/nfp}/nfp_platform.h (100%)
 create mode 100644 drivers/common/nfp/version.map
 rename drivers/net/nfp/{nfp_common.c => nfp_net_common.c} (80%)
 rename drivers/net/nfp/{nfp_common.h => nfp_net_common.h} (64%)
 rename drivers/net/nfp/{nfp_ctrl.c => nfp_net_ctrl.c} (84%)
 create mode 100644 drivers/net/nfp/nfp_net_ctrl.h
 create mode 100644 drivers/vdpa/nfp/meson.build
 create mode 100644 drivers/vdpa/nfp/nfp_vdpa.c
 create mode 100644 drivers/vdpa/nfp/nfp_vdpa_core.c
 create mode 100644 drivers/vdpa/nfp/nfp_vdpa_core.h
 create mode 100644 drivers/vdpa/nfp/nfp_vdpa_log.c
 create mode 100644 drivers/vdpa/nfp/nfp_vdpa_log.h
  

On 10/24/2023 3:28 AM, Chaoyong He wrote:
> This patch series aims to add the NFP vDPA PMD, we also grab the common
> logic into the `drivers/common/nfp` directory.
> 
> ---
> v2:
> * Grab more logic into the `drivers/common/nfp` directory.
> * Delete some logic which should be when moving logic.
> ---
> 
> Chaoyong He (25):
>   drivers: introduce the NFP common library
>   net/nfp: make VF PMD using of NFP common module
>   net/nfp: rename common module name
>   net/nfp: rename ctrl module name
>   net/nfp: extract the cap data field
>   net/nfp: extract the qcp data field
>   net/nfp: extract the ctrl BAR data field
>   net/nfp: extract the ctrl data field
>   net/nfp: change the parameter of APIs
>   net/nfp: change the parameter of reconfig
>   net/nfp: extract the MAC address data field
>   net/nfp: rename parameter in related logic
>   drivers: add the common ctrl module
>   drivers: add the nfp common module
>   drivers: move queue logic to common module
>   drivers: move platform module to common library
>   drivers: move device module to common library
>   drivers/vdpa: introduce the NFP vDPA library
>   drivers: add the basic framework of vDPA PMD
>   vdpa/nfp: add the logic of remap PCI memory
>   vdpa/nfp: add the hardware init logic
>   drivers: add the datapath update logic
>   vdpa/nfp: add the notify related logic
>   vdpa/nfp: add nfp vDPA device operations
>   doc: add the common and vDPA document
> 

Overall pretty clean set, but there are a few minor issues, commented on
patches.


Also can you please address checkpatch warnings:

  ### [PATCH] drivers: add the datapath update logic

    Warning in drivers/vdpa/nfp/nfp_vdpa.c:
    Using __atomic_xxx built-ins, prefer rte_atomic_xxx

  ### [PATCH] vdpa/nfp: add the notify related logic

    Warning in drivers/vdpa/nfp/nfp_vdpa.c:
    Using pthread functions, prefer rte_thread

  ### [PATCH] vdpa/nfp: add nfp vDPA device operations

    Warning in drivers/vdpa/nfp/nfp_vdpa.c:
    Using __atomic_xxx built-ins, prefer rte_atomic_xxx


And some typos:
  vdpa/nfp: add nfp vDPA device operations
  opetation
  

> On 10/24/2023 3:28 AM, Chaoyong He wrote:
> > This patch series aims to add the NFP vDPA PMD, we also grab the
> > common logic into the `drivers/common/nfp` directory.
> >
> > ---
> > v2:
> > * Grab more logic into the `drivers/common/nfp` directory.
> > * Delete some logic which should be when moving logic.
> > ---
> >
> > Chaoyong He (25):
> >   drivers: introduce the NFP common library
> >   net/nfp: make VF PMD using of NFP common module
> >   net/nfp: rename common module name
> >   net/nfp: rename ctrl module name
> >   net/nfp: extract the cap data field
> >   net/nfp: extract the qcp data field
> >   net/nfp: extract the ctrl BAR data field
> >   net/nfp: extract the ctrl data field
> >   net/nfp: change the parameter of APIs
> >   net/nfp: change the parameter of reconfig
> >   net/nfp: extract the MAC address data field
> >   net/nfp: rename parameter in related logic
> >   drivers: add the common ctrl module
> >   drivers: add the nfp common module
> >   drivers: move queue logic to common module
> >   drivers: move platform module to common library
> >   drivers: move device module to common library
> >   drivers/vdpa: introduce the NFP vDPA library
> >   drivers: add the basic framework of vDPA PMD
> >   vdpa/nfp: add the logic of remap PCI memory
> >   vdpa/nfp: add the hardware init logic
> >   drivers: add the datapath update logic
> >   vdpa/nfp: add the notify related logic
> >   vdpa/nfp: add nfp vDPA device operations
> >   doc: add the common and vDPA document
> >
> 
> Overall pretty clean set, but there are a few minor issues, commented on
> patches.
> 
> 
> Also can you please address checkpatch warnings:
> 
>   ### [PATCH] drivers: add the datapath update logic
> 
>     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
>     Using __atomic_xxx built-ins, prefer rte_atomic_xxx
> 

Oh, Sorry, we choose '__atomic_xxx' because we see the document in https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html?highlight=atomic%20operations%20use%20c11%20atomic%20builtins#atomic-operations-use-c11-atomic-builtins.
Maybe we misunderstood it, we will change to the `rte_atomic_xxx` in next version, thanks.

>   ### [PATCH] vdpa/nfp: add the notify related logic
> 
>     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
>     Using pthread functions, prefer rte_thread
> 

Okay, will revise in the next version.

>   ### [PATCH] vdpa/nfp: add nfp vDPA device operations
> 
>     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
>     Using __atomic_xxx built-ins, prefer rte_atomic_xxx
> 
> 
> And some typos:
>   vdpa/nfp: add nfp vDPA device operations
>   opetation

Will change in next version, thanks.
  

> > On 10/24/2023 3:28 AM, Chaoyong He wrote:
> > > This patch series aims to add the NFP vDPA PMD, we also grab the
> > > common logic into the `drivers/common/nfp` directory.
> > >
> > > ---
> > > v2:
> > > * Grab more logic into the `drivers/common/nfp` directory.
> > > * Delete some logic which should be when moving logic.
> > > ---
> > >
> > > Chaoyong He (25):
> > >   drivers: introduce the NFP common library
> > >   net/nfp: make VF PMD using of NFP common module
> > >   net/nfp: rename common module name
> > >   net/nfp: rename ctrl module name
> > >   net/nfp: extract the cap data field
> > >   net/nfp: extract the qcp data field
> > >   net/nfp: extract the ctrl BAR data field
> > >   net/nfp: extract the ctrl data field
> > >   net/nfp: change the parameter of APIs
> > >   net/nfp: change the parameter of reconfig
> > >   net/nfp: extract the MAC address data field
> > >   net/nfp: rename parameter in related logic
> > >   drivers: add the common ctrl module
> > >   drivers: add the nfp common module
> > >   drivers: move queue logic to common module
> > >   drivers: move platform module to common library
> > >   drivers: move device module to common library
> > >   drivers/vdpa: introduce the NFP vDPA library
> > >   drivers: add the basic framework of vDPA PMD
> > >   vdpa/nfp: add the logic of remap PCI memory
> > >   vdpa/nfp: add the hardware init logic
> > >   drivers: add the datapath update logic
> > >   vdpa/nfp: add the notify related logic
> > >   vdpa/nfp: add nfp vDPA device operations
> > >   doc: add the common and vDPA document
> > >
> >
> > Overall pretty clean set, but there are a few minor issues, commented
> > on patches.
> >
> >
> > Also can you please address checkpatch warnings:
> >
> >   ### [PATCH] drivers: add the datapath update logic
> >
> >     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
> >     Using __atomic_xxx built-ins, prefer rte_atomic_xxx
> >
> 
> Oh, Sorry, we choose '__atomic_xxx' because we see the document in
> https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html?highlig
> ht=atomic%20operations%20use%20c11%20atomic%20builtins#atomic-
> operations-use-c11-atomic-builtins.
> Maybe we misunderstood it, we will change to the `rte_atomic_xxx` in next
> version, thanks.

As the notes in the 'doc/guides/rel_notes/deprecation.rst':
---
  rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
  not allow for writing optimized code for all the CPU architectures supported
  in DPDK. DPDK has adopted the atomic operations from
  https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html. These
  operations must be used for patches that need to be merged in 20.08 onwards.
  This change will not introduce any performance degradation.
---
The '__atomic_xxx' is the preferred choice? So maybe it's the 'checkpatches.sh' should update?
And seems there are many logics using the '__atomic_xxx'.

Spend some time and confused about this now.
What is the right APIs I should use?
Please make it clear, thanks.

> 
> >   ### [PATCH] vdpa/nfp: add the notify related logic
> >
> >     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
> >     Using pthread functions, prefer rte_thread
> >
> 
> Okay, will revise in the next version.
> 
> >   ### [PATCH] vdpa/nfp: add nfp vDPA device operations
> >
> >     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
> >     Using __atomic_xxx built-ins, prefer rte_atomic_xxx
> >
> >
> > And some typos:
> >   vdpa/nfp: add nfp vDPA device operations
> >   opetation
> 
> Will change in next version, thanks.
  

On 10/26/2023 3:50 AM, Chaoyong He wrote:
>>> On 10/24/2023 3:28 AM, Chaoyong He wrote:
>>>> This patch series aims to add the NFP vDPA PMD, we also grab the
>>>> common logic into the `drivers/common/nfp` directory.
>>>>
>>>> ---
>>>> v2:
>>>> * Grab more logic into the `drivers/common/nfp` directory.
>>>> * Delete some logic which should be when moving logic.
>>>> ---
>>>>
>>>> Chaoyong He (25):
>>>>   drivers: introduce the NFP common library
>>>>   net/nfp: make VF PMD using of NFP common module
>>>>   net/nfp: rename common module name
>>>>   net/nfp: rename ctrl module name
>>>>   net/nfp: extract the cap data field
>>>>   net/nfp: extract the qcp data field
>>>>   net/nfp: extract the ctrl BAR data field
>>>>   net/nfp: extract the ctrl data field
>>>>   net/nfp: change the parameter of APIs
>>>>   net/nfp: change the parameter of reconfig
>>>>   net/nfp: extract the MAC address data field
>>>>   net/nfp: rename parameter in related logic
>>>>   drivers: add the common ctrl module
>>>>   drivers: add the nfp common module
>>>>   drivers: move queue logic to common module
>>>>   drivers: move platform module to common library
>>>>   drivers: move device module to common library
>>>>   drivers/vdpa: introduce the NFP vDPA library
>>>>   drivers: add the basic framework of vDPA PMD
>>>>   vdpa/nfp: add the logic of remap PCI memory
>>>>   vdpa/nfp: add the hardware init logic
>>>>   drivers: add the datapath update logic
>>>>   vdpa/nfp: add the notify related logic
>>>>   vdpa/nfp: add nfp vDPA device operations
>>>>   doc: add the common and vDPA document
>>>>
>>>
>>> Overall pretty clean set, but there are a few minor issues, commented
>>> on patches.
>>>
>>>
>>> Also can you please address checkpatch warnings:
>>>
>>>   ### [PATCH] drivers: add the datapath update logic
>>>
>>>     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
>>>     Using __atomic_xxx built-ins, prefer rte_atomic_xxx
>>>
>>
>> Oh, Sorry, we choose '__atomic_xxx' because we see the document in
>> https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html?highlig
>> ht=atomic%20operations%20use%20c11%20atomic%20builtins#atomic-
>> operations-use-c11-atomic-builtins.
>> Maybe we misunderstood it, we will change to the `rte_atomic_xxx` in next
>> version, thanks.
> 
> As the notes in the 'doc/guides/rel_notes/deprecation.rst':
> ---
>   rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
>   not allow for writing optimized code for all the CPU architectures supported
>   in DPDK. DPDK has adopted the atomic operations from
>   https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html. These
>   operations must be used for patches that need to be merged in 20.08 onwards.
>   This change will not introduce any performance degradation.
> ---
> The '__atomic_xxx' is the preferred choice? So maybe it's the 'checkpatches.sh' should update?
> And seems there are many logics using the '__atomic_xxx'.
> 
> Spend some time and confused about this now.
> What is the right APIs I should use?
> Please make it clear, thanks.
> 

Atomics usage got a few updates by time, if I remember correct:
first there was DPDK rte_atomicNN_xxx APIs,
later guidance updated to prefer compiler builtins,
and recently guidance updated to use C11 defined functions.

And now there are 'rte_atomic_xxx()' APIs, underneath they use "compiler
builtins" or "C11 functions" based on 'enable_stdatomic' config option
and of course tool-chain support.

That is the reason of complexity in the checkpatch script.

@Tyler, @David, @Honnappa, what is the latest, up to date, guidance in
the atomic APIs usage?



@Honnappa, is the deprecation notice Chaoyong highlighted still valid?
Should we update it?
  

> On 10/26/2023 3:50 AM, Chaoyong He wrote:
> >>> On 10/24/2023 3:28 AM, Chaoyong He wrote:
> >>>> This patch series aims to add the NFP vDPA PMD, we also grab the
> >>>> common logic into the `drivers/common/nfp` directory.
> >>>>
> >>>> ---
> >>>> v2:
> >>>> * Grab more logic into the `drivers/common/nfp` directory.
> >>>> * Delete some logic which should be when moving logic.
> >>>> ---
> >>>>
> >>>> Chaoyong He (25):
> >>>>   drivers: introduce the NFP common library
> >>>>   net/nfp: make VF PMD using of NFP common module
> >>>>   net/nfp: rename common module name
> >>>>   net/nfp: rename ctrl module name
> >>>>   net/nfp: extract the cap data field
> >>>>   net/nfp: extract the qcp data field
> >>>>   net/nfp: extract the ctrl BAR data field
> >>>>   net/nfp: extract the ctrl data field
> >>>>   net/nfp: change the parameter of APIs
> >>>>   net/nfp: change the parameter of reconfig
> >>>>   net/nfp: extract the MAC address data field
> >>>>   net/nfp: rename parameter in related logic
> >>>>   drivers: add the common ctrl module
> >>>>   drivers: add the nfp common module
> >>>>   drivers: move queue logic to common module
> >>>>   drivers: move platform module to common library
> >>>>   drivers: move device module to common library
> >>>>   drivers/vdpa: introduce the NFP vDPA library
> >>>>   drivers: add the basic framework of vDPA PMD
> >>>>   vdpa/nfp: add the logic of remap PCI memory
> >>>>   vdpa/nfp: add the hardware init logic
> >>>>   drivers: add the datapath update logic
> >>>>   vdpa/nfp: add the notify related logic
> >>>>   vdpa/nfp: add nfp vDPA device operations
> >>>>   doc: add the common and vDPA document
> >>>>
> >>>
> >>> Overall pretty clean set, but there are a few minor issues,
> >>> commented on patches.
> >>>
> >>>
> >>> Also can you please address checkpatch warnings:
> >>>
> >>>   ### [PATCH] drivers: add the datapath update logic
> >>>
> >>>     Warning in drivers/vdpa/nfp/nfp_vdpa.c:
> >>>     Using __atomic_xxx built-ins, prefer rte_atomic_xxx
> >>>
> >>
> >> Oh, Sorry, we choose '__atomic_xxx' because we see the document in
> >> https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html?hi
> >> ghlig
> >> ht=atomic%20operations%20use%20c11%20atomic%20builtins#atomic-
> >> operations-use-c11-atomic-builtins.
> >> Maybe we misunderstood it, we will change to the `rte_atomic_xxx` in
> >> next version, thanks.
> >
> > As the notes in the 'doc/guides/rel_notes/deprecation.rst':
> > ---
> >   rte_atomicNN_xxx: These APIs do not take memory order parameter. This
> does
> >   not allow for writing optimized code for all the CPU architectures supported
> >   in DPDK. DPDK has adopted the atomic operations from
> >   https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html.
> These
> >   operations must be used for patches that need to be merged in 20.08
> onwards.
> >   This change will not introduce any performance degradation.
> > ---
> > The '__atomic_xxx' is the preferred choice? So maybe it's the
> 'checkpatches.sh' should update?
> > And seems there are many logics using the '__atomic_xxx'.
> >
> > Spend some time and confused about this now.
> > What is the right APIs I should use?
> > Please make it clear, thanks.
> >
> 
> Atomics usage got a few updates by time, if I remember correct:
> first there was DPDK rte_atomicNN_xxx APIs, later guidance updated to prefer
> compiler builtins, and recently guidance updated to use C11 defined
> functions.
> 
> And now there are 'rte_atomic_xxx()' APIs, underneath they use "compiler
> builtins" or "C11 functions" based on 'enable_stdatomic' config option and of
> course tool-chain support.
> 
> That is the reason of complexity in the checkpatch script.
> 
> @Tyler, @David, @Honnappa, what is the latest, up to date, guidance in the
> atomic APIs usage?
> 
> 
> 
> @Honnappa, is the deprecation notice Chaoyong highlighted still valid?
> Should we update it?
> 

Oh, sorry, I have found that we should use the <rte_stdatomic.h> header file, and I have sent out the v3 patch series, and it passed the check script.
Thanks.