mbox series

[v6,0/9] ethdev: support SubFunction representor

Message ID 1613272907-22563-1-git-send-email-xuemingl@nvidia.com (mailing list archive)
Headers
Series ethdev: support SubFunction representor |

Message

Xueming Li Feb. 14, 2021, 3:21 a.m. UTC 
  SubFunction [1] is a portion of the PCI device, a SF netdev has its own
dedicated queues(txq, rxq). A SF netdev supports E-Switch representation
offload similar to existing PF and VF representors. A SF shares PCI
level resources with other SFs and/or with its parent PCI function.

From SmartNIC perspective, when PCI device is shared for multi-host,
representors for host controller and host PF is required.

This patch set introduces new representor types in addtion to existing
VF representor. Syntax:

[[c#]pf#]vf#: VF port representor/s from controller/pf
[[c#]pf#]sf#: SF port representor/s from controller/pf
#: VF representor - for backwards compatibility

"#" is number instance, list or range, valid examples:
  1, [1,3,5], [0-3], [0,2-4,6]

For backward compatibility, this patch also introduces new netdev
capability to indicate the capability of supportting SF representor.

Version history:
 RFC:
 	initial version [2]
 V2:
    - separate patch for represnetor infrastructure, controller, pf and
      sf.
    - replace representor ID macro with functions:
      rte_eth_representor_id_encode()
      rte_eth_representor_id_parse()
    - new patch to allow devargs with same PCI BDF but different
      representors.
    - other minor code updates according to comments, thanks Andrew!
    - update document
 V3:
    - improve probing of allowed devargs with same name.
    - parse single word of kvargs as key.
    - update kvargs test cases.
 V4:
    - split first representor refactor patch into
      1: add representor type
      2: refector representor list parsing
    - push the patch supporting multi-devargs for same device.
 V5:
    - add comments for parsing functions
    - update switch_representation.rst - Thanks Ajit
 V6:
    - split representor types into different patches, move to
      rte_ethdev.h
    - improvements of rte_eth_devargs_process_list() according to
      Andrew's suggestion
    - fixed PF probe failure for Intel i40e
    - replace ethdev SF capability with rte_eth_representor_info_get()
    - add new ethdev ops api to get representor info from PMD
    - replace representor ID encode/decode with conversion from
      representor info
    - change ethdev representor iterator to use new ID encoding


Xueming Li (9):
  ethdev: introduce representor type
  ethdev: support representor port list
  ethdev: support new VF representor syntax
  ethdev: support sub function representor
  ethdev: support PF index in representor
  ethdev: support multi-host in representor
  ethdev: new API to get representor info
  ethdev: representor iterator compare complete info
  kvargs: update parser to support lists

 app/test/test_kvargs.c                        |  46 ++++-
 config/rte_config.h                           |   1 +
 doc/guides/prog_guide/poll_mode_drv.rst       |  13 +-
 .../prog_guide/switch_representation.rst      |  35 +++-
 drivers/net/bnxt/bnxt_ethdev.c                |   7 +
 drivers/net/enic/enic_ethdev.c                |   6 +
 drivers/net/i40e/i40e_ethdev.c                |   7 +
 drivers/net/ixgbe/ixgbe_ethdev.c              |   7 +
 drivers/net/mlx5/linux/mlx5_os.c              |  11 ++
 lib/librte_ethdev/ethdev_driver.h             |  49 ++++-
 lib/librte_ethdev/ethdev_private.c            | 173 ++++++++++++------
 lib/librte_ethdev/ethdev_private.h            |   3 -
 lib/librte_ethdev/rte_class_eth.c             |  40 ++--
 lib/librte_ethdev/rte_ethdev.c                | 102 ++++++++++-
 lib/librte_ethdev/rte_ethdev.h                |  53 ++++++
 lib/librte_ethdev/version.map                 |   4 +
 lib/librte_kvargs/rte_kvargs.c                | 101 +++++++---
 17 files changed, 535 insertions(+), 123 deletions(-)

Comments

Stephen Hemminger Feb. 23, 2021, 1:54 a.m. UTC | #1 
On Sun, 14 Feb 2021 03:21:30 +0000
Xueming Li <xuemingl@nvidia.com> wrote:

> SubFunction [1] is a portion of the PCI device, a SF netdev has its own
> dedicated queues(txq, rxq). A SF netdev supports E-Switch representation
> offload similar to existing PF and VF representors. A SF shares PCI
> level resources with other SFs and/or with its parent PCI function.
> 
> From SmartNIC perspective, when PCI device is shared for multi-host,
> representors for host controller and host PF is required.
> 
> This patch set introduces new representor types in addtion to existing
> VF representor. Syntax:
> 
> [[c#]pf#]vf#: VF port representor/s from controller/pf
> [[c#]pf#]sf#: SF port representor/s from controller/pf
> #: VF representor - for backwards compatibility
> 
> "#" is number instance, list or range, valid examples:
>   1, [1,3,5], [0-3], [0,2-4,6]
> 
> For backward compatibility, this patch also introduces new netdev
> capability to indicate the capability of supportting SF representor.
> 
> Version history:
>  RFC:
>  	initial version [2]
>  V2:
>     - separate patch for represnetor infrastructure, controller, pf and
>       sf.
>     - replace representor ID macro with functions:
>       rte_eth_representor_id_encode()
>       rte_eth_representor_id_parse()
>     - new patch to allow devargs with same PCI BDF but different
>       representors.
>     - other minor code updates according to comments, thanks Andrew!
>     - update document
>  V3:
>     - improve probing of allowed devargs with same name.
>     - parse single word of kvargs as key.
>     - update kvargs test cases.
>  V4:
>     - split first representor refactor patch into
>       1: add representor type
>       2: refector representor list parsing
>     - push the patch supporting multi-devargs for same device.
>  V5:
>     - add comments for parsing functions
>     - update switch_representation.rst - Thanks Ajit
>  V6:
>     - split representor types into different patches, move to
>       rte_ethdev.h
>     - improvements of rte_eth_devargs_process_list() according to
>       Andrew's suggestion
>     - fixed PF probe failure for Intel i40e
>     - replace ethdev SF capability with rte_eth_representor_info_get()
>     - add new ethdev ops api to get representor info from PMD
>     - replace representor ID encode/decode with conversion from
>       representor info
>     - change ethdev representor iterator to use new ID encoding
> 
> 
> Xueming Li (9):
>   ethdev: introduce representor type
>   ethdev: support representor port list
>   ethdev: support new VF representor syntax
>   ethdev: support sub function representor
>   ethdev: support PF index in representor
>   ethdev: support multi-host in representor
>   ethdev: new API to get representor info
>   ethdev: representor iterator compare complete info
>   kvargs: update parser to support lists
> 
>  app/test/test_kvargs.c                        |  46 ++++-
>  config/rte_config.h                           |   1 +
>  doc/guides/prog_guide/poll_mode_drv.rst       |  13 +-
>  .../prog_guide/switch_representation.rst      |  35 +++-
>  drivers/net/bnxt/bnxt_ethdev.c                |   7 +
>  drivers/net/enic/enic_ethdev.c                |   6 +
>  drivers/net/i40e/i40e_ethdev.c                |   7 +
>  drivers/net/ixgbe/ixgbe_ethdev.c              |   7 +
>  drivers/net/mlx5/linux/mlx5_os.c              |  11 ++
>  lib/librte_ethdev/ethdev_driver.h             |  49 ++++-
>  lib/librte_ethdev/ethdev_private.c            | 173 ++++++++++++------
>  lib/librte_ethdev/ethdev_private.h            |   3 -
>  lib/librte_ethdev/rte_class_eth.c             |  40 ++--
>  lib/librte_ethdev/rte_ethdev.c                | 102 ++++++++++-
>  lib/librte_ethdev/rte_ethdev.h                |  53 ++++++
>  lib/librte_ethdev/version.map                 |   4 +
>  lib/librte_kvargs/rte_kvargs.c                | 101 +++++++---
>  17 files changed, 535 insertions(+), 123 deletions(-)
> 

A couple of higher level design questions:
1. How does Linux and other OS handle this in their API?
2. This solution seems quite tied into a specific implementation on your hardware.
   Is there a PCI standard for this?
3. Maybe a more general solution where these were represented as buses would
   allow for other connection methods in the future?
Wang, Haiyue Feb. 23, 2021, 11:45 a.m. UTC | #2 
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> Sent: Tuesday, February 23, 2021 09:55
> To: Xueming Li <xuemingl@nvidia.com>
> Cc: dev@dpdk.org; Viacheslav Ovsiienko <viacheslavo@nvidia.com>; Asaf Penso <asafp@nvidia.com>
> Subject: Re: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction representor
> 
> On Sun, 14 Feb 2021 03:21:30 +0000
> Xueming Li <xuemingl@nvidia.com> wrote:
> 
> > SubFunction [1] is a portion of the PCI device, a SF netdev has its own
> > dedicated queues(txq, rxq). A SF netdev supports E-Switch representation
> > offload similar to existing PF and VF representors. A SF shares PCI
> > level resources with other SFs and/or with its parent PCI function.
> >
> > From SmartNIC perspective, when PCI device is shared for multi-host,
> > representors for host controller and host PF is required.
> >
> > This patch set introduces new representor types in addtion to existing
> > VF representor. Syntax:
> >
> > [[c#]pf#]vf#: VF port representor/s from controller/pf
> > [[c#]pf#]sf#: SF port representor/s from controller/pf
> > #: VF representor - for backwards compatibility
> >
> > "#" is number instance, list or range, valid examples:
> >   1, [1,3,5], [0-3], [0,2-4,6]
> >
> > For backward compatibility, this patch also introduces new netdev
> > capability to indicate the capability of supportting SF representor.
> >
> > Version history:
> >  RFC:
> >  	initial version [2]
> >  V2:
> >     - separate patch for represnetor infrastructure, controller, pf and
> >       sf.
> >     - replace representor ID macro with functions:
> >       rte_eth_representor_id_encode()
> >       rte_eth_representor_id_parse()
> >     - new patch to allow devargs with same PCI BDF but different
> >       representors.
> >     - other minor code updates according to comments, thanks Andrew!
> >     - update document
> >  V3:
> >     - improve probing of allowed devargs with same name.
> >     - parse single word of kvargs as key.
> >     - update kvargs test cases.
> >  V4:
> >     - split first representor refactor patch into
> >       1: add representor type
> >       2: refector representor list parsing
> >     - push the patch supporting multi-devargs for same device.
> >  V5:
> >     - add comments for parsing functions
> >     - update switch_representation.rst - Thanks Ajit
> >  V6:
> >     - split representor types into different patches, move to
> >       rte_ethdev.h
> >     - improvements of rte_eth_devargs_process_list() according to
> >       Andrew's suggestion
> >     - fixed PF probe failure for Intel i40e
> >     - replace ethdev SF capability with rte_eth_representor_info_get()
> >     - add new ethdev ops api to get representor info from PMD
> >     - replace representor ID encode/decode with conversion from
> >       representor info
> >     - change ethdev representor iterator to use new ID encoding
> >
> >
> > Xueming Li (9):
> >   ethdev: introduce representor type
> >   ethdev: support representor port list
> >   ethdev: support new VF representor syntax
> >   ethdev: support sub function representor
> >   ethdev: support PF index in representor
> >   ethdev: support multi-host in representor
> >   ethdev: new API to get representor info
> >   ethdev: representor iterator compare complete info
> >   kvargs: update parser to support lists
> >
> >  app/test/test_kvargs.c                        |  46 ++++-
> >  config/rte_config.h                           |   1 +
> >  doc/guides/prog_guide/poll_mode_drv.rst       |  13 +-
> >  .../prog_guide/switch_representation.rst      |  35 +++-
> >  drivers/net/bnxt/bnxt_ethdev.c                |   7 +
> >  drivers/net/enic/enic_ethdev.c                |   6 +
> >  drivers/net/i40e/i40e_ethdev.c                |   7 +
> >  drivers/net/ixgbe/ixgbe_ethdev.c              |   7 +
> >  drivers/net/mlx5/linux/mlx5_os.c              |  11 ++
> >  lib/librte_ethdev/ethdev_driver.h             |  49 ++++-
> >  lib/librte_ethdev/ethdev_private.c            | 173 ++++++++++++------
> >  lib/librte_ethdev/ethdev_private.h            |   3 -
> >  lib/librte_ethdev/rte_class_eth.c             |  40 ++--
> >  lib/librte_ethdev/rte_ethdev.c                | 102 ++++++++++-
> >  lib/librte_ethdev/rte_ethdev.h                |  53 ++++++
> >  lib/librte_ethdev/version.map                 |   4 +
> >  lib/librte_kvargs/rte_kvargs.c                | 101 +++++++---
> >  17 files changed, 535 insertions(+), 123 deletions(-)
> >
> 
> A couple of higher level design questions:
> 1. How does Linux and other OS handle this in their API?
> 2. This solution seems quite tied into a specific implementation on your hardware.
>    Is there a PCI standard for this?

> 3. Maybe a more general solution where these were represented as buses would
>    allow for other connection methods in the future?

It should be "auxiliary bus", I think. ;-)
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/auxiliary.c

mlx5 subfunction support
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6
Xueming Li Feb. 28, 2021, 1:51 p.m. UTC | #3 
>-----Original Message-----
>From: Wang, Haiyue <haiyue.wang@intel.com>
>Sent: Tuesday, February 23, 2021 7:46 PM
>To: Stephen Hemminger <stephen@networkplumber.org>; Xueming(Steven) Li <xuemingl@nvidia.com>
>Cc: dev@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>; Asaf Penso <asafp@nvidia.com>
>Subject: RE: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction representor
>
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
>> Sent: Tuesday, February 23, 2021 09:55
>> To: Xueming Li <xuemingl@nvidia.com>
>> Cc: dev@dpdk.org; Viacheslav Ovsiienko <viacheslavo@nvidia.com>; Asaf
>> Penso <asafp@nvidia.com>
>> Subject: Re: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction
>> representor
>>
>> On Sun, 14 Feb 2021 03:21:30 +0000
>> Xueming Li <xuemingl@nvidia.com> wrote:
>>
>> > SubFunction [1] is a portion of the PCI device, a SF netdev has its
>> > own dedicated queues(txq, rxq). A SF netdev supports E-Switch
>> > representation offload similar to existing PF and VF representors. A
>> > SF shares PCI level resources with other SFs and/or with its parent PCI function.
>> >
>> > From SmartNIC perspective, when PCI device is shared for multi-host,
>> > representors for host controller and host PF is required.
>> >
>> > This patch set introduces new representor types in addtion to
>> > existing VF representor. Syntax:
>> >
>> > [[c#]pf#]vf#: VF port representor/s from controller/pf
>> > [[c#]pf#]sf#: SF port representor/s from controller/pf
>> > #: VF representor - for backwards compatibility
>> >
>> > "#" is number instance, list or range, valid examples:
>> >   1, [1,3,5], [0-3], [0,2-4,6]
>> >
>> > For backward compatibility, this patch also introduces new netdev
>> > capability to indicate the capability of supportting SF representor.
>> >
>> > Version history:
>> >  RFC:
>> >  	initial version [2]
>> >  V2:
>> >     - separate patch for represnetor infrastructure, controller, pf and
>> >       sf.
>> >     - replace representor ID macro with functions:
>> >       rte_eth_representor_id_encode()
>> >       rte_eth_representor_id_parse()
>> >     - new patch to allow devargs with same PCI BDF but different
>> >       representors.
>> >     - other minor code updates according to comments, thanks Andrew!
>> >     - update document
>> >  V3:
>> >     - improve probing of allowed devargs with same name.
>> >     - parse single word of kvargs as key.
>> >     - update kvargs test cases.
>> >  V4:
>> >     - split first representor refactor patch into
>> >       1: add representor type
>> >       2: refector representor list parsing
>> >     - push the patch supporting multi-devargs for same device.
>> >  V5:
>> >     - add comments for parsing functions
>> >     - update switch_representation.rst - Thanks Ajit
>> >  V6:
>> >     - split representor types into different patches, move to
>> >       rte_ethdev.h
>> >     - improvements of rte_eth_devargs_process_list() according to
>> >       Andrew's suggestion
>> >     - fixed PF probe failure for Intel i40e
>> >     - replace ethdev SF capability with rte_eth_representor_info_get()
>> >     - add new ethdev ops api to get representor info from PMD
>> >     - replace representor ID encode/decode with conversion from
>> >       representor info
>> >     - change ethdev representor iterator to use new ID encoding
>> >
>> >
>> > Xueming Li (9):
>> >   ethdev: introduce representor type
>> >   ethdev: support representor port list
>> >   ethdev: support new VF representor syntax
>> >   ethdev: support sub function representor
>> >   ethdev: support PF index in representor
>> >   ethdev: support multi-host in representor
>> >   ethdev: new API to get representor info
>> >   ethdev: representor iterator compare complete info
>> >   kvargs: update parser to support lists
>> >
>> >  app/test/test_kvargs.c                        |  46 ++++-
>> >  config/rte_config.h                           |   1 +
>> >  doc/guides/prog_guide/poll_mode_drv.rst       |  13 +-
>> >  .../prog_guide/switch_representation.rst      |  35 +++-
>> >  drivers/net/bnxt/bnxt_ethdev.c                |   7 +
>> >  drivers/net/enic/enic_ethdev.c                |   6 +
>> >  drivers/net/i40e/i40e_ethdev.c                |   7 +
>> >  drivers/net/ixgbe/ixgbe_ethdev.c              |   7 +
>> >  drivers/net/mlx5/linux/mlx5_os.c              |  11 ++
>> >  lib/librte_ethdev/ethdev_driver.h             |  49 ++++-
>> >  lib/librte_ethdev/ethdev_private.c            | 173 ++++++++++++------
>> >  lib/librte_ethdev/ethdev_private.h            |   3 -
>> >  lib/librte_ethdev/rte_class_eth.c             |  40 ++--
>> >  lib/librte_ethdev/rte_ethdev.c                | 102 ++++++++++-
>> >  lib/librte_ethdev/rte_ethdev.h                |  53 ++++++
>> >  lib/librte_ethdev/version.map                 |   4 +
>> >  lib/librte_kvargs/rte_kvargs.c                | 101 +++++++---
>> >  17 files changed, 535 insertions(+), 123 deletions(-)
>> >
>>
>> A couple of higher level design questions:
>> 1. How does Linux and other OS handle this in their API?

Hi Stephen, thanks for looking this!
Representor was a type of ethdev, OS independent.
SF representor is a new type besides existing VF representor type.

>> 2. This solution seems quite tied into a specific implementation on your hardware.
>>    Is there a PCI standard for this?

SF has been carved in kernel, supported by devlink. The devlink document in the link Haiyue posted should be helpful to understand the overall picture:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6

>
>> 3. Maybe a more general solution where these were represented as buses would
>>    allow for other connection methods in the future?
>
>It should be "auxiliary bus", I think. ;-) https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/auxiliary.c

Good link, Mellanox has exactly implemented SF based auxiliary bus, I'm going to post another RFC to support auxiliary bus.
But this patchset still needed to support representor of SF.

>
>mlx5 subfunction support
>https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6
Xueming Li March 31, 2021, 5:49 a.m. UTC | #4 
>-----Original Message-----
>From: Wang, Haiyue <haiyue.wang@intel.com>
>Sent: Tuesday, February 23, 2021 7:46 PM
>To: Stephen Hemminger <stephen@networkplumber.org>; Xueming(Steven) Li <xuemingl@nvidia.com>
>Cc: dev@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>; Asaf Penso <asafp@nvidia.com>
>Subject: RE: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction representor
>
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
>> Sent: Tuesday, February 23, 2021 09:55
>> To: Xueming Li <xuemingl@nvidia.com>
>> Cc: dev@dpdk.org; Viacheslav Ovsiienko <viacheslavo@nvidia.com>; Asaf
>> Penso <asafp@nvidia.com>
>> Subject: Re: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction
>> representor
>>
>> On Sun, 14 Feb 2021 03:21:30 +0000
>> Xueming Li <xuemingl@nvidia.com> wrote:
>>
>> > SubFunction [1] is a portion of the PCI device, a SF netdev has its
>> > own dedicated queues(txq, rxq). A SF netdev supports E-Switch
>> > representation offload similar to existing PF and VF representors. A
>> > SF shares PCI level resources with other SFs and/or with its parent PCI function.
>> >
>> > From SmartNIC perspective, when PCI device is shared for multi-host,
>> > representors for host controller and host PF is required.
>> >
>> > This patch set introduces new representor types in addtion to
>> > existing VF representor. Syntax:
>> >
>> > [[c#]pf#]vf#: VF port representor/s from controller/pf
>> > [[c#]pf#]sf#: SF port representor/s from controller/pf
>> > #: VF representor - for backwards compatibility
>> >
>> > "#" is number instance, list or range, valid examples:
>> >   1, [1,3,5], [0-3], [0,2-4,6]
>> >
>> > For backward compatibility, this patch also introduces new netdev
>> > capability to indicate the capability of supportting SF representor.
>> >
>> > Version history:
>> >  RFC:
>> >  	initial version [2]
>> >  V2:
>> >     - separate patch for represnetor infrastructure, controller, pf and
>> >       sf.
>> >     - replace representor ID macro with functions:
>> >       rte_eth_representor_id_encode()
>> >       rte_eth_representor_id_parse()
>> >     - new patch to allow devargs with same PCI BDF but different
>> >       representors.
>> >     - other minor code updates according to comments, thanks Andrew!
>> >     - update document
>> >  V3:
>> >     - improve probing of allowed devargs with same name.
>> >     - parse single word of kvargs as key.
>> >     - update kvargs test cases.
>> >  V4:
>> >     - split first representor refactor patch into
>> >       1: add representor type
>> >       2: refector representor list parsing
>> >     - push the patch supporting multi-devargs for same device.
>> >  V5:
>> >     - add comments for parsing functions
>> >     - update switch_representation.rst - Thanks Ajit
>> >  V6:
>> >     - split representor types into different patches, move to
>> >       rte_ethdev.h
>> >     - improvements of rte_eth_devargs_process_list() according to
>> >       Andrew's suggestion
>> >     - fixed PF probe failure for Intel i40e
>> >     - replace ethdev SF capability with rte_eth_representor_info_get()
>> >     - add new ethdev ops api to get representor info from PMD
>> >     - replace representor ID encode/decode with conversion from
>> >       representor info
>> >     - change ethdev representor iterator to use new ID encoding
>> >
>> >
>> > Xueming Li (9):
>> >   ethdev: introduce representor type
>> >   ethdev: support representor port list
>> >   ethdev: support new VF representor syntax
>> >   ethdev: support sub function representor
>> >   ethdev: support PF index in representor
>> >   ethdev: support multi-host in representor
>> >   ethdev: new API to get representor info
>> >   ethdev: representor iterator compare complete info
>> >   kvargs: update parser to support lists
>> >
>> >  app/test/test_kvargs.c                        |  46 ++++-
>> >  config/rte_config.h                           |   1 +
>> >  doc/guides/prog_guide/poll_mode_drv.rst       |  13 +-
>> >  .../prog_guide/switch_representation.rst      |  35 +++-
>> >  drivers/net/bnxt/bnxt_ethdev.c                |   7 +
>> >  drivers/net/enic/enic_ethdev.c                |   6 +
>> >  drivers/net/i40e/i40e_ethdev.c                |   7 +
>> >  drivers/net/ixgbe/ixgbe_ethdev.c              |   7 +
>> >  drivers/net/mlx5/linux/mlx5_os.c              |  11 ++
>> >  lib/librte_ethdev/ethdev_driver.h             |  49 ++++-
>> >  lib/librte_ethdev/ethdev_private.c            | 173 ++++++++++++------
>> >  lib/librte_ethdev/ethdev_private.h            |   3 -
>> >  lib/librte_ethdev/rte_class_eth.c             |  40 ++--
>> >  lib/librte_ethdev/rte_ethdev.c                | 102 ++++++++++-
>> >  lib/librte_ethdev/rte_ethdev.h                |  53 ++++++
>> >  lib/librte_ethdev/version.map                 |   4 +
>> >  lib/librte_kvargs/rte_kvargs.c                | 101 +++++++---
>> >  17 files changed, 535 insertions(+), 123 deletions(-)
>> >
>>
>> A couple of higher level design questions:
>> 1. How does Linux and other OS handle this in their API?
>> 2. This solution seems quite tied into a specific implementation on your hardware.
>>    Is there a PCI standard for this?
>
>> 3. Maybe a more general solution where these were represented as buses would
>>    allow for other connection methods in the future?
>
>It should be "auxiliary bus", I think. ;-) https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/auxiliary.c
>
>mlx5 subfunction support
>https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6

Hi,

The Auxiliary driver for DPDK posted, welcome to have a look :)
http://patchwork.dpdk.org/project/dpdk/patch/20210311130113.31382-1-xuemingl@nvidia.com/

Thanks
Wang, Haiyue March 31, 2021, 5:58 a.m. UTC | #5 
> -----Original Message-----
> From: Xueming(Steven) Li <xuemingl@nvidia.com>
> Sent: Wednesday, March 31, 2021 13:49
> To: Wang, Haiyue <haiyue.wang@intel.com>; Stephen Hemminger <stephen@networkplumber.org>
> Cc: dev@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>; Asaf Penso <asafp@nvidia.com>
> Subject: RE: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction representor
> 
> 
> 
> >-----Original Message-----
> >From: Wang, Haiyue <haiyue.wang@intel.com>
> >Sent: Tuesday, February 23, 2021 7:46 PM
> >To: Stephen Hemminger <stephen@networkplumber.org>; Xueming(Steven) Li <xuemingl@nvidia.com>
> >Cc: dev@dpdk.org; Slava Ovsiienko <viacheslavo@nvidia.com>; Asaf Penso <asafp@nvidia.com>
> >Subject: RE: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction representor
> >
> >> -----Original Message-----
> >> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> >> Sent: Tuesday, February 23, 2021 09:55
> >> To: Xueming Li <xuemingl@nvidia.com>
> >> Cc: dev@dpdk.org; Viacheslav Ovsiienko <viacheslavo@nvidia.com>; Asaf
> >> Penso <asafp@nvidia.com>
> >> Subject: Re: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction
> >> representor
> >>
> >> On Sun, 14 Feb 2021 03:21:30 +0000
> >> Xueming Li <xuemingl@nvidia.com> wrote:
> >>
> >> > SubFunction [1] is a portion of the PCI device, a SF netdev has its
> >> > own dedicated queues(txq, rxq). A SF netdev supports E-Switch
> >> > representation offload similar to existing PF and VF representors. A
> >> > SF shares PCI level resources with other SFs and/or with its parent PCI function.
> >> >
> >> > From SmartNIC perspective, when PCI device is shared for multi-host,
> >> > representors for host controller and host PF is required.
> >> >
> >> > This patch set introduces new representor types in addtion to
> >> > existing VF representor. Syntax:
> >> >
> >> > [[c#]pf#]vf#: VF port representor/s from controller/pf
> >> > [[c#]pf#]sf#: SF port representor/s from controller/pf
> >> > #: VF representor - for backwards compatibility
> >> >
> >> > "#" is number instance, list or range, valid examples:
> >> >   1, [1,3,5], [0-3], [0,2-4,6]
> >> >
> >> > For backward compatibility, this patch also introduces new netdev
> >> > capability to indicate the capability of supportting SF representor.
> >> >
> >> > Version history:
> >> >  RFC:
> >> >  	initial version [2]
> >> >  V2:
> >> >     - separate patch for represnetor infrastructure, controller, pf and
> >> >       sf.
> >> >     - replace representor ID macro with functions:
> >> >       rte_eth_representor_id_encode()
> >> >       rte_eth_representor_id_parse()
> >> >     - new patch to allow devargs with same PCI BDF but different
> >> >       representors.
> >> >     - other minor code updates according to comments, thanks Andrew!
> >> >     - update document
> >> >  V3:
> >> >     - improve probing of allowed devargs with same name.
> >> >     - parse single word of kvargs as key.
> >> >     - update kvargs test cases.
> >> >  V4:
> >> >     - split first representor refactor patch into
> >> >       1: add representor type
> >> >       2: refector representor list parsing
> >> >     - push the patch supporting multi-devargs for same device.
> >> >  V5:
> >> >     - add comments for parsing functions
> >> >     - update switch_representation.rst - Thanks Ajit
> >> >  V6:
> >> >     - split representor types into different patches, move to
> >> >       rte_ethdev.h
> >> >     - improvements of rte_eth_devargs_process_list() according to
> >> >       Andrew's suggestion
> >> >     - fixed PF probe failure for Intel i40e
> >> >     - replace ethdev SF capability with rte_eth_representor_info_get()
> >> >     - add new ethdev ops api to get representor info from PMD
> >> >     - replace representor ID encode/decode with conversion from
> >> >       representor info
> >> >     - change ethdev representor iterator to use new ID encoding
> >> >
> >> >
> >> > Xueming Li (9):
> >> >   ethdev: introduce representor type
> >> >   ethdev: support representor port list
> >> >   ethdev: support new VF representor syntax
> >> >   ethdev: support sub function representor
> >> >   ethdev: support PF index in representor
> >> >   ethdev: support multi-host in representor
> >> >   ethdev: new API to get representor info
> >> >   ethdev: representor iterator compare complete info
> >> >   kvargs: update parser to support lists
> >> >
> >> >  app/test/test_kvargs.c                        |  46 ++++-
> >> >  config/rte_config.h                           |   1 +
> >> >  doc/guides/prog_guide/poll_mode_drv.rst       |  13 +-
> >> >  .../prog_guide/switch_representation.rst      |  35 +++-
> >> >  drivers/net/bnxt/bnxt_ethdev.c                |   7 +
> >> >  drivers/net/enic/enic_ethdev.c                |   6 +
> >> >  drivers/net/i40e/i40e_ethdev.c                |   7 +
> >> >  drivers/net/ixgbe/ixgbe_ethdev.c              |   7 +
> >> >  drivers/net/mlx5/linux/mlx5_os.c              |  11 ++
> >> >  lib/librte_ethdev/ethdev_driver.h             |  49 ++++-
> >> >  lib/librte_ethdev/ethdev_private.c            | 173 ++++++++++++------
> >> >  lib/librte_ethdev/ethdev_private.h            |   3 -
> >> >  lib/librte_ethdev/rte_class_eth.c             |  40 ++--
> >> >  lib/librte_ethdev/rte_ethdev.c                | 102 ++++++++++-
> >> >  lib/librte_ethdev/rte_ethdev.h                |  53 ++++++
> >> >  lib/librte_ethdev/version.map                 |   4 +
> >> >  lib/librte_kvargs/rte_kvargs.c                | 101 +++++++---
> >> >  17 files changed, 535 insertions(+), 123 deletions(-)
> >> >
> >>
> >> A couple of higher level design questions:
> >> 1. How does Linux and other OS handle this in their API?
> >> 2. This solution seems quite tied into a specific implementation on your hardware.
> >>    Is there a PCI standard for this?
> >
> >> 3. Maybe a more general solution where these were represented as buses would
> >>    allow for other connection methods in the future?
> >
> >It should be "auxiliary bus", I think. ;-)
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/auxiliary.c
> >
> >mlx5 subfunction support
> >https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9be
> cc8799241716672f6
> 
> Hi,
> 
> The Auxiliary driver for DPDK posted, welcome to have a look :)
> http://patchwork.dpdk.org/project/dpdk/patch/20210311130113.31382-1-xuemingl@nvidia.com/

Cool. ;-)

> 
> Thanks
>