[v15,00/23] Add DLB PMD

  The following patch series adds support for a new eventdev PMD. The DLB
PMD adds support for the Intel Dynamic Load Balancer (DLB) hardware.
The DLB is a PCIe device that provides load-balanced, prioritized
scheduling of core-to-core communication. The device consists of
queues and arbiters that connect producer and consumer cores, and
implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The DLB hardware supports both load balanced and directed ports and
queues. Unlike other eventdev devices already in the repo,  not all
DLB ports and queues are equally capable. In particular, directed
ports are limited to a single link, and must be connected to a directed
queue.
Additionally, even though LDB ports may link multiple queues, the
number of queues that may be linked is limited by hardware. Another
difference is that DLB does not have a straightforward way of carrying
the flow_id in the queue elements (QE) that the hardware operates on.

While reviewing the code, please be aware that this PMD has full
control over the DLB hardware. Intel will be extending the DLB PMD
in the future (not as part of this first series) with a mode that we
refer to as the bifurcated PMD. The bifurcated PMD communicates with a
kernel driver to configure the device, ports, and queues, and memory
maps device MMIO so datapath operations occur purely in user-space.

The framework to support both the PF PMD and bifurcated PMD exists in
this patchset, and is why the iface.[ch] layer is present.

Major changes in V15
====================
Address comments from David Marchand:
- this patch-set is based on Nov 1, 2020 dpdk-next-eventdev
- fix docs build (doxy-api.conf.in and doxy-api-index.md)
- restore blank line in MAINTAINERS file
- move dlb announcement in release_20_11.rst after ethdev
- use headers = files() for exported meson public headers
- fix a typo in 'add documentation ..." commit message
- use eal version of cldemote
- convert a couple of printfs to LOG messages
- fix missing "~" in dlb documentation
- delay introduction of _delayed token pop functions to
  token pop commit (fixes 8 or so unused function errors)
- all patches build incrementally (gcc), and checkpatches reports
  success
- I am not able to run clang locally. If clang errors are still
  present I will ask IT to install clang on a build server tomorrow.

Major changes in V14
====================
- Fixed format errors in doc/api/doxy-api-index.md
- Delayed introduction of dlb2_consume_qe_immediate until
  add-dequeue-and-its-burst-variants.patch
- Delayed introduction of dlb2_construct_token_pop_qe until 
  add-PMD-s-token-pop-public-interface.patch
- Delayed introduction of dlb_equeue_*_delayed until
  add dequeue and its burst variants.patch

Major changes in V13
====================
- removed now unused functions dlb_umwait and dlb_umonitor

Major changes in V12
====================
- Fix CENTOS build error: use __m128i instead of __v2di with
  _mm_stream_si128

Major changes in V11
====================
- removed unused function, fixing build error
- fixed typo in port_setup commit message
- this patch series is based on dpdk-next-eventdev

Major changes in v10
=====================
- convert to use rte_power_monitor patches
- replace __builtin_ia32_movntdq() with _mm_stream_si128()
- remove unused functions in dlb_selftest.c

Major changes in v9
=====================
- fixed a build error due to __rte_cache_aligned being placed after
  the ";" character, instead of before it.

Major changes in v8 after dpdk reviews
=====================
- moved introduction of dlb in relnotes_20_11 to first patch in series
- fixed underlines in dlb.rst that were too short
- note that the code still uses its private byte-encoded versions of
  umonitor/umwait, rather than the new functions in the power
  patch that are built on top of those intrinsics. This is intentional.

Major changes in v7 after dpdk reviews
=====================
- updated MAINTAINERS file to alphabetically insert DLB
- don't create RTE_ symbols in PMD
- converted to use version.map scheme
- converted to use .._master_lcore instead of .._main_lcore
- this patch set is based on dpdk-next-eventdev

Major changes in v6 after dpdk reviews:
=====================
- fixed meson conditional build. Moved test into driver’s meson.build
  file instead of event/meson.build
- documentation is populated as associated code is introduced
- add log_register in add dynamic logging patch
- rename RTE_xxx symbol(s) as DLB2_xxx
- replaced function ptr enqueue_four with direct call to movdir64b
- remove unused port_pages
- broke up probe patch into 3 smaller patches for easier review
- changed param order of movdir64b/movntdq to match intrinsics
- added self to MAINTAINERS files
- squashed announcement of availability into last patch in series
- correct spelling errors and delete repeated words
- DPDK_21.0 -> DPDK 21 in map file
- add experimental banner to public structs and APIs
- implemented other suggestions from code reviews of DLB2 PMD. The
  software is very similar in form so some DLB2 reviews comments
  were applicable to DLB as well

Major changes in v5 after dpdk reviews and additional internal reviews
by colleagues at Intel:
================
- implement changes requested in code reviews by Gage Eads and Mike Chen
- fix a memzone leak
- convert to use eal rte-cpuflags patch from Liang Ma

Major changes in v4 after dpdk reviews and additional internal reviews
by colleagues at Intel:
================
- Remove make infrastructure
- shared code (pf/base) is now added incrementally
- flexible interface (iface.[ch]) is now added incrementally
- removed calls to rte_panic
- do not call pthread_create directly
- remove unused internal API, os_time
- convert rte_atomic to __atomic builtins
- broke out eventdev ABI changes, test/api changes, and new internal PCI
  named probe API
- relocated enqueue logic to enqueue patch

Major Changes in V3:
================
- Fixed a memory corruption issue due to not allocating enough CQ
memory for depths < 8. Hardware requires minimum allocation to be
at least 8 entries.
- Address review comments from Gage and Mattias.
- Remove versioning
- minor formatting changes

Major changes in V2:
================
- Correct ABI break that was present in V1.
- Address some of the review comments received from Mattias.
  I will address the remaining items identified by Mattias in the next
  patch delivery.
- General code cleanup based on internal code reviews

Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")

Timothy McDaniel (23):
  event/dlb: add documentation and meson infrastructure
  event/dlb: add dynamic logging
  event/dlb: add private data structures and constants
  event/dlb: add definitions shared with LKM or shared code
  event/dlb: add inline functions
  event/dlb: add eventdev probe
  event/dlb: add flexible interface
  event/dlb: add probe-time hardware init
  event/dlb: add xstats
  event/dlb: add infos get and configure
  event/dlb: add queue and port default conf
  event/dlb: add queue setup
  event/dlb: add port setup
  event/dlb: add port link
  event/dlb: add port unlink and port unlinks in progress
  event/dlb: add eventdev start
  event/dlb: add enqueue and its burst variants
  event/dlb: add dequeue and its burst variants
  event/dlb: add eventdev stop and close
  event/dlb: add PMD's token pop public interface
  event/dlb: add PMD self-tests
  event/dlb: add queue and port release
  event/dlb: add timeout ticks entry point

 MAINTAINERS                                  |    5 +
 app/test/test_eventdev.c                     |    7 +
 config/rte_config.h                          |    6 +
 doc/api/doxy-api-index.md                    |    3 +-
 doc/api/doxy-api.conf.in                     |    1 +
 doc/guides/eventdevs/dlb.rst                 |  341 ++
 doc/guides/eventdevs/index.rst               |    1 +
 doc/guides/rel_notes/release_20_11.rst       |    5 +
 drivers/event/dlb/dlb.c                      | 4079 +++++++++++++++
 drivers/event/dlb/dlb_iface.c                |   79 +
 drivers/event/dlb/dlb_iface.h                |   82 +
 drivers/event/dlb/dlb_inline_fns.h           |   36 +
 drivers/event/dlb/dlb_log.h                  |   25 +
 drivers/event/dlb/dlb_priv.h                 |  513 ++
 drivers/event/dlb/dlb_selftest.c             | 1539 ++++++
 drivers/event/dlb/dlb_user.h                 |  814 +++
 drivers/event/dlb/dlb_xstats.c               | 1217 +++++
 drivers/event/dlb/meson.build                |   22 +
 drivers/event/dlb/pf/base/dlb_hw_types.h     |  334 ++
 drivers/event/dlb/pf/base/dlb_osdep.h        |  310 ++
 drivers/event/dlb/pf/base/dlb_osdep_bitmap.h |  441 ++
 drivers/event/dlb/pf/base/dlb_osdep_list.h   |  131 +
 drivers/event/dlb/pf/base/dlb_osdep_types.h  |   31 +
 drivers/event/dlb/pf/base/dlb_regs.h         | 2368 +++++++++
 drivers/event/dlb/pf/base/dlb_resource.c     | 6904 ++++++++++++++++++++++++++
 drivers/event/dlb/pf/base/dlb_resource.h     |  876 ++++
 drivers/event/dlb/pf/dlb_main.c              |  586 +++
 drivers/event/dlb/pf/dlb_main.h              |   47 +
 drivers/event/dlb/pf/dlb_pf.c                |  750 +++
 drivers/event/dlb/rte_pmd_dlb.c              |   38 +
 drivers/event/dlb/rte_pmd_dlb.h              |   77 +
 drivers/event/dlb/version.map                |    9 +
 drivers/event/meson.build                    |    2 +-
 33 files changed, 21677 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/eventdevs/dlb.rst
 create mode 100644 drivers/event/dlb/dlb.c
 create mode 100644 drivers/event/dlb/dlb_iface.c
 create mode 100644 drivers/event/dlb/dlb_iface.h
 create mode 100644 drivers/event/dlb/dlb_inline_fns.h
 create mode 100644 drivers/event/dlb/dlb_log.h
 create mode 100644 drivers/event/dlb/dlb_priv.h
 create mode 100644 drivers/event/dlb/dlb_selftest.c
 create mode 100644 drivers/event/dlb/dlb_user.h
 create mode 100644 drivers/event/dlb/dlb_xstats.c
 create mode 100644 drivers/event/dlb/meson.build
 create mode 100644 drivers/event/dlb/pf/base/dlb_hw_types.h
 create mode 100644 drivers/event/dlb/pf/base/dlb_osdep.h
 create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_bitmap.h
 create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_list.h
 create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_types.h
 create mode 100644 drivers/event/dlb/pf/base/dlb_regs.h
 create mode 100644 drivers/event/dlb/pf/base/dlb_resource.c
 create mode 100644 drivers/event/dlb/pf/base/dlb_resource.h
 create mode 100644 drivers/event/dlb/pf/dlb_main.c
 create mode 100644 drivers/event/dlb/pf/dlb_main.h
 create mode 100644 drivers/event/dlb/pf/dlb_pf.c
 create mode 100644 drivers/event/dlb/rte_pmd_dlb.c
 create mode 100644 drivers/event/dlb/rte_pmd_dlb.h
 create mode 100644 drivers/event/dlb/version.map
  

On Sun, Nov 1, 2020 at 8:25 PM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The following patch series adds support for a new eventdev PMD. The DLB
> PMD adds support for the Intel Dynamic Load Balancer (DLB) hardware.
> The DLB is a PCIe device that provides load-balanced, prioritized
> scheduling of core-to-core communication. The device consists of
> queues and arbiters that connect producer and consumer cores, and
> implements load-balanced queueing features including:
> - Lock-free multi-producer/multi-consumer operation.
> - Multiple priority levels for varying traffic types.
> - 'Direct' traffic (i.e. multi-producer/single-consumer)
> - Simple unordered load-balanced distribution.
> - Atomic lock-free load balancing across multiple consumers.
> - Queue element reordering feature allowing ordered load-balanced
>   distribution.
>
> The DLB hardware supports both load balanced and directed ports and
> queues. Unlike other eventdev devices already in the repo,  not all
> DLB ports and queues are equally capable. In particular, directed
> ports are limited to a single link, and must be connected to a directed
> queue.
> Additionally, even though LDB ports may link multiple queues, the
> number of queues that may be linked is limited by hardware. Another
> difference is that DLB does not have a straightforward way of carrying
> the flow_id in the queue elements (QE) that the hardware operates on.
>
> While reviewing the code, please be aware that this PMD has full
> control over the DLB hardware. Intel will be extending the DLB PMD
> in the future (not as part of this first series) with a mode that we
> refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> kernel driver to configure the device, ports, and queues, and memory
> maps device MMIO so datapath operations occur purely in user-space.
>
> The framework to support both the PF PMD and bifurcated PMD exists in
> this patchset, and is why the iface.[ch] layer is present.
>
> Major changes in V15
> ====================
> Address comments from David Marchand:
> - this patch-set is based on Nov 1, 2020 dpdk-next-eventdev
> - fix docs build (doxy-api.conf.in and doxy-api-index.md)
> - restore blank line in MAINTAINERS file
> - move dlb announcement in release_20_11.rst after ethdev
> - use headers = files() for exported meson public headers
> - fix a typo in 'add documentation ..." commit message
> - use eal version of cldemote
> - convert a couple of printfs to LOG messages

There are still printf in drivers/event/dlb/pf/dlb_main.c from a grep
I did on v15.
Why were they kept?

+ I did not look too much at the PCI code in this file, but I suspect
we could factor some of it with librte_pci.


> - fix missing "~" in dlb documentation
> - delay introduction of _delayed token pop functions to
>   token pop commit (fixes 8 or so unused function errors)
> - all patches build incrementally (gcc), and checkpatches reports
>   success

There is a valid warning on patch 6:
http://mails.dpdk.org/archives/test-report/2020-November/164347.html

WARNING:REPEATED_WORD: Possible repeated word: 'of'
#1493: FILE: drivers/event/dlb/pf/base/dlb_osdep_bitmap.h:364:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,


> - I am not able to run clang locally. If clang errors are still
>   present I will ask IT to install clang on a build server tomorrow.

This time, compilation passed for each patch on my build system (using
./devtools/test-meson-builds.sh, x86, arm, ppc + doc generation).
  

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Sunday, November 1, 2020 3:11 PM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v15 00/23] Add DLB PMD
> 
> On Sun, Nov 1, 2020 at 8:25 PM Timothy McDaniel
> <timothy.mcdaniel@intel.com> wrote:
> >
> > The following patch series adds support for a new eventdev PMD. The DLB
> > PMD adds support for the Intel Dynamic Load Balancer (DLB) hardware.
> > The DLB is a PCIe device that provides load-balanced, prioritized
> > scheduling of core-to-core communication. The device consists of
> > queues and arbiters that connect producer and consumer cores, and
> > implements load-balanced queueing features including:
> > - Lock-free multi-producer/multi-consumer operation.
> > - Multiple priority levels for varying traffic types.
> > - 'Direct' traffic (i.e. multi-producer/single-consumer)
> > - Simple unordered load-balanced distribution.
> > - Atomic lock-free load balancing across multiple consumers.
> > - Queue element reordering feature allowing ordered load-balanced
> >   distribution.
> >
> > The DLB hardware supports both load balanced and directed ports and
> > queues. Unlike other eventdev devices already in the repo,  not all
> > DLB ports and queues are equally capable. In particular, directed
> > ports are limited to a single link, and must be connected to a directed
> > queue.
> > Additionally, even though LDB ports may link multiple queues, the
> > number of queues that may be linked is limited by hardware. Another
> > difference is that DLB does not have a straightforward way of carrying
> > the flow_id in the queue elements (QE) that the hardware operates on.
> >
> > While reviewing the code, please be aware that this PMD has full
> > control over the DLB hardware. Intel will be extending the DLB PMD
> > in the future (not as part of this first series) with a mode that we
> > refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> > kernel driver to configure the device, ports, and queues, and memory
> > maps device MMIO so datapath operations occur purely in user-space.
> >
> > The framework to support both the PF PMD and bifurcated PMD exists in
> > this patchset, and is why the iface.[ch] layer is present.
> >
> > Major changes in V15
> > ====================
> > Address comments from David Marchand:
> > - this patch-set is based on Nov 1, 2020 dpdk-next-eventdev
> > - fix docs build (doxy-api.conf.in and doxy-api-index.md)
> > - restore blank line in MAINTAINERS file
> > - move dlb announcement in release_20_11.rst after ethdev
> > - use headers = files() for exported meson public headers
> > - fix a typo in 'add documentation ..." commit message
> > - use eal version of cldemote
> > - convert a couple of printfs to LOG messages
> 
> There are still printf in drivers/event/dlb/pf/dlb_main.c from a grep
> I did on v15.
> Why were they kept?
> 

Sorry.  This was an oversight on my part. I thought you had identified all of the files with printf.
I should have searched all of the files, and fixed any cases that were not identified in the CR.
I will fix these in the next patch-set.

> + I did not look too much at the PCI code in this file, but I suspect
> we could factor some of it with librte_pci.
> 

Can we look at possibly doing that post rc2 merge?

> 
> > - fix missing "~" in dlb documentation
> > - delay introduction of _delayed token pop functions to
> >   token pop commit (fixes 8 or so unused function errors)
> > - all patches build incrementally (gcc), and checkpatches reports
> >   success
> 
> There is a valid warning on patch 6:
> http://mails.dpdk.org/archives/test-report/2020-November/164347.html
> 
> WARNING:REPEATED_WORD: Possible repeated word: 'of'
> #1493: FILE: drivers/event/dlb/pf/base/dlb_osdep_bitmap.h:364:
> + * Returns the bitmap's longest contiguous range of of set bits upon success,
> 

I will address with the next patch-set. Odd that I did not see the error/warning when I
ran devtools/checkpatches.sh

> 
> > - I am not able to run clang locally. If clang errors are still
> >   present I will ask IT to install clang on a build server tomorrow.
> 
> This time, compilation passed for each patch on my build system (using
> ./devtools/test-meson-builds.sh, x86, arm, ppc + doc generation).

Excellent news. Thanks for the update.

> 
> 
> --
> David Marchand
  

On Sun, Nov 1, 2020 at 10:26 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
> > + I did not look too much at the PCI code in this file, but I suspect
> > we could factor some of it with librte_pci.
> >
>
> Can we look at possibly doing that post rc2 merge?

Please try to look at dpdk existing API before adding code.
Ok for a followup cleanup.


>
> >
> > > - fix missing "~" in dlb documentation
> > > - delay introduction of _delayed token pop functions to
> > >   token pop commit (fixes 8 or so unused function errors)
> > > - all patches build incrementally (gcc), and checkpatches reports
> > >   success
> >
> > There is a valid warning on patch 6:
> > http://mails.dpdk.org/archives/test-report/2020-November/164347.html
> >
> > WARNING:REPEATED_WORD: Possible repeated word: 'of'
> > #1493: FILE: drivers/event/dlb/pf/base/dlb_osdep_bitmap.h:364:
> > + * Returns the bitmap's longest contiguous range of of set bits upon success,
> >
>
> I will address with the next patch-set. Odd that I did not see the error/warning when I
> ran devtools/checkpatches.sh

You need a fairly recent checkpatch.pl.
  

On Sun, Nov 1, 2020 at 10:26 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
> > + I did not look too much at the PCI code in this file, but I suspect
> > we could factor some of it with librte_pci.
> >
>
> Can we look at possibly doing that post rc2 merge?

Any update?
  

I'm not seeing where we can use librte_pci here.  We are already using the dpdk low level rte_pci_read_config(..) APIs.
Am I missing something?

Thanks,
Tim


> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, November 10, 2020 6:52 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Thomas Monjalon <thomas@monjalon.net>; Gaetan
> Rivet <grive@u256.net>
> Subject: Re: [dpdk-dev] [PATCH v15 00/23] Add DLB PMD
> 
> On Sun, Nov 1, 2020 at 10:26 PM McDaniel, Timothy
> <timothy.mcdaniel@intel.com> wrote:
> > > + I did not look too much at the PCI code in this file, but I suspect
> > > we could factor some of it with librte_pci.
> > >
> >
> > Can we look at possibly doing that post rc2 merge?
> 
> Any update?
> 
> 
> --
> David Marchand
  

On Tue, Nov 10, 2020 at 4:29 PM McDaniel, Timothy
<timothy.mcdaniel@intel.com> wrote:
>
> I'm not seeing where we can use librte_pci here.  We are already using the dpdk low level rte_pci_read_config(..) APIs.
> Am I missing something?

dlb_pci_find_ext_capability is a copy of rte_pci_find_ext_capability
from the pci bus.

A lot of DLB_PCI_* defines have a librte_pci equivalent.
Plus, if there are generic pci defines needed by DLB, they would
rather belong to librte_pci than a driver.
  

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, November 10, 2020 9:51 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Thomas Monjalon <thomas@monjalon.net>; Gaetan
> Rivet <grive@u256.net>
> Subject: Re: [dpdk-dev] [PATCH v15 00/23] Add DLB PMD
> 
> On Tue, Nov 10, 2020 at 4:29 PM McDaniel, Timothy
> <timothy.mcdaniel@intel.com> wrote:
> >
> > I'm not seeing where we can use librte_pci here.  We are already using the
> dpdk low level rte_pci_read_config(..) APIs.
> > Am I missing something?
> 
> dlb_pci_find_ext_capability is a copy of rte_pci_find_ext_capability
> from the pci bus.
> 
> A lot of DLB_PCI_* defines have a librte_pci equivalent.
> Plus, if there are generic pci defines needed by DLB, they would
> rather belong to librte_pci than a driver.
> 
> 
> --
> David Marchand

Thanks, David.  I missed that. I'll get to work on removing the duplication.

Tim
  

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, November 10, 2020 9:51 AM
> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
> Cc: dev <dev@dpdk.org>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Eads,
> Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Thomas Monjalon <thomas@monjalon.net>; Gaetan
> Rivet <grive@u256.net>
> Subject: Re: [dpdk-dev] [PATCH v15 00/23] Add DLB PMD
> 
> On Tue, Nov 10, 2020 at 4:29 PM McDaniel, Timothy
> <timothy.mcdaniel@intel.com> wrote:
> >
> > I'm not seeing where we can use librte_pci here.  We are already using the
> dpdk low level rte_pci_read_config(..) APIs.
> > Am I missing something?
> 
> dlb_pci_find_ext_capability is a copy of rte_pci_find_ext_capability
> from the pci bus.
> 
> A lot of DLB_PCI_* defines have a librte_pci equivalent.
> Plus, if there are generic pci defines needed by DLB, they would
> rather belong to librte_pci than a driver.
> 
> 
> --
> David Marchand

David,

I just uploaded patches that should address most of your concerns.
See https://patches.dpdk.org/patch/84034/ and https://patches.dpdk.org/patch/84031/ 

Thanks,
Tim