[RFC,v1,0/2] dts: initial checksum offload suite

Test suite for verifying checksum hardware offload through the
PMD works as expected. This is done by checking the verbose output in
testpmd while in csum forwarding mode, specifically the ol_flags
section, to ensure they match the flags in the test plan. However, 
there are a few issues I noticed while writing the suite that made
me hesitant to submit a patch:

1. SCTP hardware offload is not supported on any of the NICs I tested
on. I've tried this on mlx5, i40e, and bnxt drivers and none of them
support it. SCTP offload is used as part of almost every test case, so I
removed SCTP packets from the suite entirely. I intend to keep it that
way unless anyone is able to use the command "csum set sctp hw 0"
without an "SCTP not supported" error.
2. There are two Tx checksum test cases, which involve checking the Tx
flags section of verbose output to ensure they match the ones in the
test plan. However, the Tx flags don't appear to change at all
depending on what packet you send to testpmd, which leaves me with no
way to verify correct behavior. I'm considering removing the Tx cases
entirely, but they are a large chunk of the suite so if anyone disagrees
I can look for more of a workaround.

If anyone has any comments or advice about the issues above it is
greatly appreciated.

Dean Marx (2):
  dts: add csum HW offload to testpmd shell
  dts: checksum offload test suite

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py |  94 ++++++
 dts/tests/TestSuite_checksum_offload.py       | 288 ++++++++++++++++++
 3 files changed, 384 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_checksum_offload.py
  

Hey Dean,

Thanks for the patch, I left some thoughts on the point you raise
below and some of the larger things I noticed throughout the series.
The logic seems to all be right though and I can understand what you
were going for, so the broader ideas seem good to me.

On Mon, Aug 12, 2024 at 9:41 AM Dean Marx <dmarx@iol.unh.edu> wrote:
>
> Test suite for verifying checksum hardware offload through the
> PMD works as expected. This is done by checking the verbose output in
> testpmd while in csum forwarding mode, specifically the ol_flags
> section, to ensure they match the flags in the test plan. However,
> there are a few issues I noticed while writing the suite that made
> me hesitant to submit a patch:
>
> 1. SCTP hardware offload is not supported on any of the NICs I tested
> on. I've tried this on mlx5, i40e, and bnxt drivers and none of them
> support it. SCTP offload is used as part of almost every test case, so I
> removed SCTP packets from the suite entirely. I intend to keep it that
> way unless anyone is able to use the command "csum set sctp hw 0"
> without an "SCTP not supported" error.
> 2. There are two Tx checksum test cases, which involve checking the Tx
> flags section of verbose output to ensure they match the ones in the
> test plan. However, the Tx flags don't appear to change at all
> depending on what packet you send to testpmd, which leaves me with no
> way to verify correct behavior. I'm considering removing the Tx cases
> entirely, but they are a large chunk of the suite so if anyone disagrees
> I can look for more of a workaround.
>

As for the Tx issue, I have also noticed that these flags are
inconsistent. It looks like the Tx side of things have flags
representing if there is checksum data for the various layers, but
even then it doesn't show up in the verbose output when there is
verbose information. I think that if it isn't consistent then we can't
write a test suite that is generic enough to be run on the hardware
and it probably makes more sense to just leave it out of the suite.
I'm honestly not sure if it is a bug or just a misunderstanding with
what these flags should be doing since the Tx flags are much less
clear to me than the Rx. Regardless, it's not like it's something we
can search for with a capability or anything so we don't have a great
mechanism for getting this to work if it truly is just something where
different drivers do it differently.


> If anyone has any comments or advice about the issues above it is
> greatly appreciated.
>
> Dean Marx (2):
>   dts: add csum HW offload to testpmd shell
>   dts: checksum offload test suite
>
>  dts/framework/config/conf_yaml_schema.json    |   3 +-
>  dts/framework/remote_session/testpmd_shell.py |  94 ++++++
>  dts/tests/TestSuite_checksum_offload.py       | 288 ++++++++++++++++++
>  3 files changed, 384 insertions(+), 1 deletion(-)
>  create mode 100644 dts/tests/TestSuite_checksum_offload.py
>
> --
> 2.44.0
>