[0/9] net/cxgbe: updates for rte_flow support

  From: Karra Satwik <kaara.satwik@chelsio.com>

This series of patches contain rte_flow support for matching
Q-in-Q VLAN, IP TOS, PF, and VF fields. Also, adds Destination
MAC rewrite and Source MAC rewrite actions.

Apart from the 4-tuple (IP src/dst addresses and TCP/UDP src/dst
port addresses), there are only 40-bits available to match other
fields in packet headers. Currently, the combination of packet
header fields to match are configured via filterMode for LETCAM
filters and filterMask for HASH filters in firmware config files
(t5/t6-config.txt). Adapter needs to be reflashed with new firmware
config file everytime the combinations need to be changed. To avoid
this, a new firmware API is available to dynamically change the
combination before completing full adapter initialization. So, 2
new devargs filtermode and filtermask are added to dynamically
select the combinations during runtime.

Patch 1 adds rte_flow support for matching Q-in-Q VLAN.

Patch 2 adds rte_flow support for matching IP TOS.

Patch 3 adds rte_flow support for matching all packets on PF.

Patch 4 adds rte_flow support for matching all packets on VF.

Patch 5 adds rte_flow support for overwriting destination MAC.

Patch 6 adds Source MAC Table (SMT) support.

Patch 7 adds rte_flow support for Source MAC Rewrite.

Patch 8 adds new firmware API for validating filter spec.

Patch 9 adds devargs to control filtermode and filtermask
combinations.

Thanks,
Satwik

Karra Satwik (9):
  net/cxgbe: add rte_flow support for matching Q-in-Q VLAN
  net/cxgbe: add rte_flow support for matching IP TOS
  net/cxgbe: add rte_flow support for matching all packets on PF
  net/cxgbe: add rte_flow support for matching all packets on VF
  net/cxgbe: add rte_flow support for overwriting destination MAC
  net/cxgbe: add Source MAC Table (SMT) support
  net/cxgbe: add rte_flow support for Source MAC Rewrite
  net/cxgbe: use firmware API for validating filter spec
  net/cxgbe: add devargs to control filtermode and filtermask values

 doc/guides/nics/cxgbe.rst               | 219 +++++++++++++++++-
 drivers/net/cxgbe/Makefile              |   1 +
 drivers/net/cxgbe/base/adapter.h        |   9 +
 drivers/net/cxgbe/base/common.h         |   8 +-
 drivers/net/cxgbe/base/t4_hw.c          |  81 +++++--
 drivers/net/cxgbe/base/t4_msg.h         |  40 ++++
 drivers/net/cxgbe/base/t4_regs.h        |   4 +
 drivers/net/cxgbe/base/t4_tcb.h         |  10 +
 drivers/net/cxgbe/base/t4fw_interface.h |  55 ++++-
 drivers/net/cxgbe/cxgbe.h               |  23 ++
 drivers/net/cxgbe/cxgbe_ethdev.c        |   4 +-
 drivers/net/cxgbe/cxgbe_filter.c        | 103 ++++++++-
 drivers/net/cxgbe/cxgbe_filter.h        |   8 +-
 drivers/net/cxgbe/cxgbe_flow.c          | 241 +++++++++++++++-----
 drivers/net/cxgbe/cxgbe_main.c          | 291 +++++++++++++++++++++++-
 drivers/net/cxgbe/meson.build           |   1 +
 drivers/net/cxgbe/smt.c                 | 230 +++++++++++++++++++
 drivers/net/cxgbe/smt.h                 |  44 ++++
 18 files changed, 1275 insertions(+), 97 deletions(-)
 create mode 100644 drivers/net/cxgbe/smt.c
 create mode 100644 drivers/net/cxgbe/smt.h
  

On 3/11/2020 9:05 AM, Rahul Lakkireddy wrote:
> From: Karra Satwik <kaara.satwik@chelsio.com>
> 
> This series of patches contain rte_flow support for matching
> Q-in-Q VLAN, IP TOS, PF, and VF fields. Also, adds Destination
> MAC rewrite and Source MAC rewrite actions.
> 
> Apart from the 4-tuple (IP src/dst addresses and TCP/UDP src/dst
> port addresses), there are only 40-bits available to match other
> fields in packet headers. Currently, the combination of packet
> header fields to match are configured via filterMode for LETCAM
> filters and filterMask for HASH filters in firmware config files
> (t5/t6-config.txt). Adapter needs to be reflashed with new firmware
> config file everytime the combinations need to be changed. To avoid
> this, a new firmware API is available to dynamically change the
> combination before completing full adapter initialization. So, 2
> new devargs filtermode and filtermask are added to dynamically
> select the combinations during runtime.
> 
> Patch 1 adds rte_flow support for matching Q-in-Q VLAN.
> 
> Patch 2 adds rte_flow support for matching IP TOS.
> 
> Patch 3 adds rte_flow support for matching all packets on PF.
> 
> Patch 4 adds rte_flow support for matching all packets on VF.
> 
> Patch 5 adds rte_flow support for overwriting destination MAC.
> 
> Patch 6 adds Source MAC Table (SMT) support.
> 
> Patch 7 adds rte_flow support for Source MAC Rewrite.
> 
> Patch 8 adds new firmware API for validating filter spec.
> 
> Patch 9 adds devargs to control filtermode and filtermask
> combinations.
> 
> Thanks,
> Satwik
> 
> Karra Satwik (9):
>   net/cxgbe: add rte_flow support for matching Q-in-Q VLAN
>   net/cxgbe: add rte_flow support for matching IP TOS
>   net/cxgbe: add rte_flow support for matching all packets on PF
>   net/cxgbe: add rte_flow support for matching all packets on VF
>   net/cxgbe: add rte_flow support for overwriting destination MAC
>   net/cxgbe: add Source MAC Table (SMT) support
>   net/cxgbe: add rte_flow support for Source MAC Rewrite
>   net/cxgbe: use firmware API for validating filter spec
>   net/cxgbe: add devargs to control filtermode and filtermask values

Series applied to dpdk-next-net/master, thanks.
  

11/03/2020 10:05, Rahul Lakkireddy:
> From: Karra Satwik <kaara.satwik@chelsio.com>
> 
> This series of patches contain rte_flow support for matching
> Q-in-Q VLAN, IP TOS, PF, and VF fields. Also, adds Destination
> MAC rewrite and Source MAC rewrite actions.
> 
> Apart from the 4-tuple (IP src/dst addresses and TCP/UDP src/dst
> port addresses), there are only 40-bits available to match other
> fields in packet headers. Currently, the combination of packet
> header fields to match are configured via filterMode for LETCAM
> filters and filterMask for HASH filters in firmware config files
> (t5/t6-config.txt). Adapter needs to be reflashed with new firmware
> config file everytime the combinations need to be changed. To avoid
> this, a new firmware API is available to dynamically change the
> combination before completing full adapter initialization. So, 2
> new devargs filtermode and filtermask are added to dynamically
> select the combinations during runtime.

Please, could you explain why you are using devargs for flow matching,
instead of using the common and generic rte_flow API?
  

Hi Thomas,

On Wednesday, March 03/18/20, 2020 at 13:09:47 +0100, Thomas Monjalon wrote:
> 11/03/2020 10:05, Rahul Lakkireddy:
> > From: Karra Satwik <kaara.satwik@chelsio.com>
> > 
> > This series of patches contain rte_flow support for matching
> > Q-in-Q VLAN, IP TOS, PF, and VF fields. Also, adds Destination
> > MAC rewrite and Source MAC rewrite actions.
> > 
> > Apart from the 4-tuple (IP src/dst addresses and TCP/UDP src/dst
> > port addresses), there are only 40-bits available to match other
> > fields in packet headers. Currently, the combination of packet
> > header fields to match are configured via filterMode for LETCAM
> > filters and filterMask for HASH filters in firmware config files
> > (t5/t6-config.txt). Adapter needs to be reflashed with new firmware
> > config file everytime the combinations need to be changed. To avoid
> > this, a new firmware API is available to dynamically change the
> > combination before completing full adapter initialization. So, 2
> > new devargs filtermode and filtermask are added to dynamically
> > select the combinations during runtime.
> 
> Please, could you explain why you are using devargs for flow matching,
> instead of using the common and generic rte_flow API?
> 

The devargs are being used to configure the TCAM in hardware on
what header fields need to be matched in packets by the TCAM. The
actual filter rules are still being inserted using rte_flow API.

Apart from the 4-tuple (src/dst IP, src/dst port addresses), there
are only 40-bits available for each filter rule to match other
header fields. Hardware supports matching ethertype (16-bit),
DST MAC (9-bit MPS index), Inner VLAN (16-bit), Outer VLAN (16-bit),
IP Protocol (8-bit), IP TOS (8-bit), Ingress Physical Port (3-bit),
and PFVF (17-bit) for now. It's not possible to write a filter rule
which wants to match all the above fields, which is far beyond 40-bits
available. So, the devargs are being used to control which of the
above fields that user wants to configure the TCAM to match. Note
that once the TCAM is configured, "all" the rules in the TCAM can
only match the selected fields in the combination. They can't match
any other header fields.

For example, let's say user wants to match ethertype (16-bit),
DST MAC (9-bit MPS index), and IP protocol (8-bit) in all filter
rules. Then, they would configure the TCAM with the {ethertype,
DST MAC, and IP protocol} combination. All rules that the user
wants to insert into TCAM can only have the above header fields,
alongside the default 4-tuple (src/dst IP, src/dst port addresses).

There are 2 regions in hardware. One for matching wild-card filter
rules and the other for matching exact-match rules. The "filtermode"
devarg controls the 40-bit combination for wild-card filter rules and
the "filtermask" devarg controls the combination for exact-match rules.

Thanks,
Rahul
  

18/03/2020 14:06, Rahul Lakkireddy:
> Hi Thomas,
> 
> On Wednesday, March 03/18/20, 2020 at 13:09:47 +0100, Thomas Monjalon wrote:
> > 11/03/2020 10:05, Rahul Lakkireddy:
> > > From: Karra Satwik <kaara.satwik@chelsio.com>
> > > 
> > > This series of patches contain rte_flow support for matching
> > > Q-in-Q VLAN, IP TOS, PF, and VF fields. Also, adds Destination
> > > MAC rewrite and Source MAC rewrite actions.
> > > 
> > > Apart from the 4-tuple (IP src/dst addresses and TCP/UDP src/dst
> > > port addresses), there are only 40-bits available to match other
> > > fields in packet headers. Currently, the combination of packet
> > > header fields to match are configured via filterMode for LETCAM
> > > filters and filterMask for HASH filters in firmware config files
> > > (t5/t6-config.txt). Adapter needs to be reflashed with new firmware
> > > config file everytime the combinations need to be changed. To avoid
> > > this, a new firmware API is available to dynamically change the
> > > combination before completing full adapter initialization. So, 2
> > > new devargs filtermode and filtermask are added to dynamically
> > > select the combinations during runtime.
> > 
> > Please, could you explain why you are using devargs for flow matching,
> > instead of using the common and generic rte_flow API?
> 
> The devargs are being used to configure the TCAM in hardware on
> what header fields need to be matched in packets by the TCAM. The
> actual filter rules are still being inserted using rte_flow API.
> 
> Apart from the 4-tuple (src/dst IP, src/dst port addresses), there
> are only 40-bits available for each filter rule to match other
> header fields. Hardware supports matching ethertype (16-bit),
> DST MAC (9-bit MPS index), Inner VLAN (16-bit), Outer VLAN (16-bit),
> IP Protocol (8-bit), IP TOS (8-bit), Ingress Physical Port (3-bit),
> and PFVF (17-bit) for now. It's not possible to write a filter rule
> which wants to match all the above fields, which is far beyond 40-bits
> available. So, the devargs are being used to control which of the
> above fields that user wants to configure the TCAM to match. Note
> that once the TCAM is configured, "all" the rules in the TCAM can
> only match the selected fields in the combination. They can't match
> any other header fields.

In case a rule is not possible, are you rejecting it at the validation stage?

> For example, let's say user wants to match ethertype (16-bit),
> DST MAC (9-bit MPS index), and IP protocol (8-bit) in all filter
> rules. Then, they would configure the TCAM with the {ethertype,
> DST MAC, and IP protocol} combination. All rules that the user
> wants to insert into TCAM can only have the above header fields,
> alongside the default 4-tuple (src/dst IP, src/dst port addresses).
> 
> There are 2 regions in hardware. One for matching wild-card filter
> rules and the other for matching exact-match rules. The "filtermode"
> devarg controls the 40-bit combination for wild-card filter rules and
> the "filtermask" devarg controls the combination for exact-match rules.

I see an issue with this approach. I will explain below.

An application is written to use some flow rules.
The application requirements are expressed by the app developper
through the API (rte_flow).
In your case, the user must be aware of what the application expects
and fill the right devargs, according to what the dev wrote.
Why bothering the user with this constraint?

I understand the hardware must be prepared in advance.
I think this configuration must be done through API.
One workaround is to manage this HW limitation in a PMD-specific API.
A good solution would be to express this requirement with rte_flow.

One idea: can we use rte_flow_validate() to fill the requirements?
The PMD requirements are empty at the beginning and they are filled
with the first calls to rte_flow_validate().

Maybe we also need to express the capabilities/limitations.
Example: is there a maximum number of rules?
maximum number of protocols to match?
maximum number of bits to match?

I suppose it is not easy to implement. Comments?
  

On Wednesday, March 03/18/20, 2020 at 16:07:07 +0100, Thomas Monjalon wrote:
> 18/03/2020 14:06, Rahul Lakkireddy:
> > Hi Thomas,
> > 
> > On Wednesday, March 03/18/20, 2020 at 13:09:47 +0100, Thomas Monjalon wrote:
> > > 11/03/2020 10:05, Rahul Lakkireddy:
> > > > From: Karra Satwik <kaara.satwik@chelsio.com>
> > > > 
> > > > This series of patches contain rte_flow support for matching
> > > > Q-in-Q VLAN, IP TOS, PF, and VF fields. Also, adds Destination
> > > > MAC rewrite and Source MAC rewrite actions.
> > > > 
> > > > Apart from the 4-tuple (IP src/dst addresses and TCP/UDP src/dst
> > > > port addresses), there are only 40-bits available to match other
> > > > fields in packet headers. Currently, the combination of packet
> > > > header fields to match are configured via filterMode for LETCAM
> > > > filters and filterMask for HASH filters in firmware config files
> > > > (t5/t6-config.txt). Adapter needs to be reflashed with new firmware
> > > > config file everytime the combinations need to be changed. To avoid
> > > > this, a new firmware API is available to dynamically change the
> > > > combination before completing full adapter initialization. So, 2
> > > > new devargs filtermode and filtermask are added to dynamically
> > > > select the combinations during runtime.
> > > 
> > > Please, could you explain why you are using devargs for flow matching,
> > > instead of using the common and generic rte_flow API?
> > 
> > The devargs are being used to configure the TCAM in hardware on
> > what header fields need to be matched in packets by the TCAM. The
> > actual filter rules are still being inserted using rte_flow API.
> > 
> > Apart from the 4-tuple (src/dst IP, src/dst port addresses), there
> > are only 40-bits available for each filter rule to match other
> > header fields. Hardware supports matching ethertype (16-bit),
> > DST MAC (9-bit MPS index), Inner VLAN (16-bit), Outer VLAN (16-bit),
> > IP Protocol (8-bit), IP TOS (8-bit), Ingress Physical Port (3-bit),
> > and PFVF (17-bit) for now. It's not possible to write a filter rule
> > which wants to match all the above fields, which is far beyond 40-bits
> > available. So, the devargs are being used to control which of the
> > above fields that user wants to configure the TCAM to match. Note
> > that once the TCAM is configured, "all" the rules in the TCAM can
> > only match the selected fields in the combination. They can't match
> > any other header fields.
> 
> In case a rule is not possible, are you rejecting it at the validation stage?
> 

Yes, the rule is rejected if it can't be offloaded. Any rule
that has fields that are unsupported or haven't been configured
to be matched by TCAM are rejected. This validation is being done
in the driver's registered callback of rte_flow_validate().

> > For example, let's say user wants to match ethertype (16-bit),
> > DST MAC (9-bit MPS index), and IP protocol (8-bit) in all filter
> > rules. Then, they would configure the TCAM with the {ethertype,
> > DST MAC, and IP protocol} combination. All rules that the user
> > wants to insert into TCAM can only have the above header fields,
> > alongside the default 4-tuple (src/dst IP, src/dst port addresses).
> > 
> > There are 2 regions in hardware. One for matching wild-card filter
> > rules and the other for matching exact-match rules. The "filtermode"
> > devarg controls the 40-bit combination for wild-card filter rules and
> > the "filtermask" devarg controls the combination for exact-match rules.
> 
> I see an issue with this approach. I will explain below.
> 
> An application is written to use some flow rules.
> The application requirements are expressed by the app developper
> through the API (rte_flow).
> In your case, the user must be aware of what the application expects
> and fill the right devargs, according to what the dev wrote.
> Why bothering the user with this constraint?
> 
> I understand the hardware must be prepared in advance.
> I think this configuration must be done through API.
> One workaround is to manage this HW limitation in a PMD-specific API.
> A good solution would be to express this requirement with rte_flow.
> 

The TCAM must be configured on what fields it is expected to match
before the hardware initialization completes; i.e. during PCIe probe
itself. Devargs is the only API I know that can provide this kind
of configuration information at this early stage. Based on what
fields the TCAM has been configured to match, the resources in
hardware are allocated accordingly.

Note that this configuration is not on a per rule basis. All the
rules must only have the fields that the TCAM had been configured to
match. Any rule that doesn't adhere to this will be rejected.

> One idea: can we use rte_flow_validate() to fill the requirements?
> The PMD requirements are empty at the beginning and they are filled
> with the first calls to rte_flow_validate().
> 
> Maybe we also need to express the capabilities/limitations.
> Example: is there a maximum number of rules?
> maximum number of protocols to match?
> maximum number of bits to match?
> 
> I suppose it is not easy to implement. Comments?
> 

Perhaps the hardware's rte_flow offload capabilities can be exposed
similar to how we're exposing the Tx/Rx offloads today, via the
rte_eth_dev_info structure. Alternatively, we can add a separate
rte_flow_dev_info structure and let PMDs fill their rte_flow related
capabilities. Maybe even add a rte_flow_dev_infos_get() API for this
purpose?

Thanks,
Rahul