[0/3] support setting lanes

  Some speeds can be achieved with different number of lanes. For example,
100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
When use different lanes, the port cannot be up.

This patch set add support setting lanes for ethdev
In addition, modify the testpmd and hns3 driver to adapt to it.

Dengdui Huang (3):
  ethdev: support setting lanes
  net/hns3: support setting lanes
  app/testpmd: support setting lanes

 app/test-pmd/cmdline.c                      | 110 ++++----
 app/test-pmd/config.c                       |  60 +++--
 doc/guides/rel_notes/release_24_03.rst      |  11 +-
 doc/guides/testpmd_app_ug/testpmd_funcs.rst |   3 +-
 drivers/net/bnxt/bnxt_ethdev.c              |   3 +-
 drivers/net/hns3/hns3_cmd.h                 |  15 +-
 drivers/net/hns3/hns3_common.c              |   3 +-
 drivers/net/hns3/hns3_ethdev.c              | 112 ++++----
 drivers/net/hns3/hns3_ethdev.h              |   2 +
 lib/ethdev/ethdev_driver.h                  |   1 -
 lib/ethdev/ethdev_linux_ethtool.c           | 101 +++++++-
 lib/ethdev/ethdev_private.h                 |   4 +
 lib/ethdev/ethdev_trace.h                   |   4 +-
 lib/ethdev/meson.build                      |   2 +
 lib/ethdev/rte_ethdev.c                     | 272 +++++++++++++++++++-
 lib/ethdev/rte_ethdev.h                     |  99 +++++--
 lib/ethdev/version.map                      |   7 +
 17 files changed, 635 insertions(+), 174 deletions(-)
  

12/03/2024 08:52, Dengdui Huang:
> Some speeds can be achieved with different number of lanes. For example,
> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> When use different lanes, the port cannot be up.

I'm not sure what you are referring to.
I suppose it is not PCI lanes.
Please could you link to an explanation of how a port is split in lanes?
Which hardware does this?
  

On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 12/03/2024 08:52, Dengdui Huang:
> > Some speeds can be achieved with different number of lanes. For example,
> > 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> > When use different lanes, the port cannot be up.
>
> I'm not sure what you are referring to.
> I suppose it is not PCI lanes.
> Please could you link to an explanation of how a port is split in lanes?
> Which hardware does this?
>
>
>
This is a snapshot of 100Gb that the latest BCM576xx supports.
100Gb (NRZ: 25G per lane, 4 lanes) link speed
100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
100Gb (PAM4-112: 100G per lane, 1 lane) link speed

Let the user feed in lanes=< integer value> and the NIC driver decides
the matching combination speed x lanes that works. In future if a new speed
is implemented with more than 8 lanes, there wouldn't be a need
to touch this speed command. Using separate lane command would
be a better alternative to support already shipped products and only new
drivers would consider this lanes configuration, if applicable.
  

On Mon, 18 Mar 2024 14:26:33 -0700
Damodharam Ammepalli <damodharam.ammepalli@broadcom.com> wrote:

> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 12/03/2024 08:52, Dengdui Huang:  
> > > Some speeds can be achieved with different number of lanes. For example,
> > > 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> > > When use different lanes, the port cannot be up.  
> >
> > I'm not sure what you are referring to.
> > I suppose it is not PCI lanes.
> > Please could you link to an explanation of how a port is split in lanes?
> > Which hardware does this?
> >
> >
> >  
> This is a snapshot of 100Gb that the latest BCM576xx supports.
> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> 
> Let the user feed in lanes=< integer value> and the NIC driver decides
> the matching combination speed x lanes that works. In future if a new speed
> is implemented with more than 8 lanes, there wouldn't be a need
> to touch this speed command. Using separate lane command would
> be a better alternative to support already shipped products and only new
> drivers would consider this lanes configuration, if applicable.
> 

The DPDK does not need more driver specific knobs.
Shouldn't the PMD be able to auto negotiate the speed?
What does Linux do?
  

On Mon, Mar 18, 2024 at 2:42 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 18 Mar 2024 14:26:33 -0700
> Damodharam Ammepalli <damodharam.ammepalli@broadcom.com> wrote:
>
> > On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 12/03/2024 08:52, Dengdui Huang:
> > > > Some speeds can be achieved with different number of lanes. For example,
> > > > 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> > > > When use different lanes, the port cannot be up.
> > >
> > > I'm not sure what you are referring to.
> > > I suppose it is not PCI lanes.
> > > Please could you link to an explanation of how a port is split in lanes?
> > > Which hardware does this?
> > >
> > >
> > >
> > This is a snapshot of 100Gb that the latest BCM576xx supports.
> > 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> > 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> > 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> >
> > Let the user feed in lanes=< integer value> and the NIC driver decides
> > the matching combination speed x lanes that works. In future if a new speed
> > is implemented with more than 8 lanes, there wouldn't be a need
> > to touch this speed command. Using separate lane command would
> > be a better alternative to support already shipped products and only new
> > drivers would consider this lanes configuration, if applicable.
> >
>
> The DPDK does not need more driver specific knobs.
> Shouldn't the PMD be able to auto negotiate the speed?
Yes. Its possible to auto negotiate. And that's the default.
Even for the lane count, a default number can be arrived at.

> What does Linux do?
ethtool has been extended a while ago to allow configuring the number
of lanes along with speed and other settings.
But as usual, autoneg is possible.
  

18/03/2024 22:26, Damodharam Ammepalli:
> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 12/03/2024 08:52, Dengdui Huang:
> > > Some speeds can be achieved with different number of lanes. For example,
> > > 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> > > When use different lanes, the port cannot be up.
> >
> > I'm not sure what you are referring to.
> > I suppose it is not PCI lanes.
> > Please could you link to an explanation of how a port is split in lanes?
> > Which hardware does this?
> 
> This is a snapshot of 100Gb that the latest BCM576xx supports.
> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> 
> Let the user feed in lanes=< integer value> and the NIC driver decides
> the matching combination speed x lanes that works. In future if a new speed
> is implemented with more than 8 lanes, there wouldn't be a need
> to touch this speed command. Using separate lane command would
> be a better alternative to support already shipped products and only new
> drivers would consider this lanes configuration, if applicable.

Sorry it does not provide enough explanations.
What is a lane? How does it work?
Is it only for Broadcom devices? Do you know other devices?
  

On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 12/03/2024 08:52, Dengdui Huang:
>>> Some speeds can be achieved with different number of lanes. For example,
>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
>>> When use different lanes, the port cannot be up.
>>
>> I'm not sure what you are referring to.
>> I suppose it is not PCI lanes.
>> Please could you link to an explanation of how a port is split in lanes?
>> Which hardware does this?
>>
>>
>>
> This is a snapshot of 100Gb that the latest BCM576xx supports.
> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> 
> Let the user feed in lanes=< integer value> and the NIC driver decides
> the matching combination speed x lanes that works. In future if a new speed
> is implemented with more than 8 lanes, there wouldn't be a need
> to touch this speed command. Using separate lane command would
> be a better alternative to support already shipped products and only new
> drivers would consider this lanes configuration, if applicable.
> 

As far as I understand, lane is related to the physical layer of the
NIC, there are multiple copies of transmitter, receiver, modulator HW
block and each set called as a 'lane' and multiple lanes work together
to achieve desired speed. (please correct me if this is wrong).

Why not just configuring the speed is not enough? Why user needs to know
the detail and configuration of the lanes?
Will it work if driver/device configure the "speed x lane" internally
for the requested speed?

Is there a benefit to force specific lane count for a specific speed
(like power optimization, just a wild guess)?


And +1 for auto-negotiation if possible.
  

On 2024/3/20 20:31, Ferruh Yigit wrote:
> On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
>> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>>
>>> 12/03/2024 08:52, Dengdui Huang:
>>>> Some speeds can be achieved with different number of lanes. For example,
>>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
>>>> When use different lanes, the port cannot be up.
>>>
>>> I'm not sure what you are referring to.
>>> I suppose it is not PCI lanes.
>>> Please could you link to an explanation of how a port is split in lanes?
>>> Which hardware does this?
>>>
>>>
>>>
>> This is a snapshot of 100Gb that the latest BCM576xx supports.
>> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
>> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
>> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
>>
>> Let the user feed in lanes=< integer value> and the NIC driver decides
>> the matching combination speed x lanes that works. In future if a new speed
>> is implemented with more than 8 lanes, there wouldn't be a need
>> to touch this speed command. Using separate lane command would
>> be a better alternative to support already shipped products and only new
>> drivers would consider this lanes configuration, if applicable.
>>
> 
> As far as I understand, lane is related to the physical layer of the
> NIC, there are multiple copies of transmitter, receiver, modulator HW
> block and each set called as a 'lane' and multiple lanes work together
> to achieve desired speed. (please correct me if this is wrong).
> 
> Why not just configuring the speed is not enough? Why user needs to know
> the detail and configuration of the lanes?
> Will it work if driver/device configure the "speed x lane" internally
> for the requested speed?
> 
> Is there a benefit to force specific lane count for a specific speed
> (like power optimization, just a wild guess)?
> 
> 
> And +1 for auto-negotiation if possible.

As you said above,，multiple lanes work together to achieve desired speed.
For example, the following solutions can be used to implement 100G:
1、Combines four 25G lanes
2、Combines two 50G lanes
3、A single 100G lane

It is assumed that two ports are interconnected and the two ports support
the foregoing three solutions. But, we just configured the speed to 100G and
one port uses four 25G lanes by default and the other port uses two 50G lanes
by default, the port cannot be up. In this case, we need to configure the
two ports to use the same solutions (for example, uses two 50G lanes)
so that the ports can be up.

ethtool has supported lanes configuration a while ago.[1]

[1]
https://lore.kernel.org/netdev/20201010154119.3537085-1-idosch@idosch.org/T/
  

21/03/2024 03:02, huangdengdui:
> 
> On 2024/3/20 20:31, Ferruh Yigit wrote:
> > On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
> >> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>>
> >>> 12/03/2024 08:52, Dengdui Huang:
> >>>> Some speeds can be achieved with different number of lanes. For example,
> >>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> >>>> When use different lanes, the port cannot be up.
> >>>
> >>> I'm not sure what you are referring to.
> >>> I suppose it is not PCI lanes.
> >>> Please could you link to an explanation of how a port is split in lanes?
> >>> Which hardware does this?
> >>>
> >> This is a snapshot of 100Gb that the latest BCM576xx supports.
> >> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> >> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> >> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> >>
> >> Let the user feed in lanes=< integer value> and the NIC driver decides
> >> the matching combination speed x lanes that works. In future if a new speed
> >> is implemented with more than 8 lanes, there wouldn't be a need
> >> to touch this speed command. Using separate lane command would
> >> be a better alternative to support already shipped products and only new
> >> drivers would consider this lanes configuration, if applicable.
> >>
> > 
> > As far as I understand, lane is related to the physical layer of the
> > NIC, there are multiple copies of transmitter, receiver, modulator HW
> > block and each set called as a 'lane' and multiple lanes work together
> > to achieve desired speed. (please correct me if this is wrong).
> > 
> > Why not just configuring the speed is not enough? Why user needs to know
> > the detail and configuration of the lanes?
> > Will it work if driver/device configure the "speed x lane" internally
> > for the requested speed?
> > 
> > Is there a benefit to force specific lane count for a specific speed
> > (like power optimization, just a wild guess)?
> > 
> > 
> > And +1 for auto-negotiation if possible.
> 
> As you said above,，multiple lanes work together to achieve desired speed.
> For example, the following solutions can be used to implement 100G:
> 1、Combines four 25G lanes
> 2、Combines two 50G lanes
> 3、A single 100G lane
> 
> It is assumed that two ports are interconnected and the two ports support
> the foregoing three solutions. But, we just configured the speed to 100G and
> one port uses four 25G lanes by default and the other port uses two 50G lanes
> by default, the port cannot be up. In this case, we need to configure the
> two ports to use the same solutions (for example, uses two 50G lanes)
> so that the ports can be up.

Why this config is not OK? How do we know?
Really I have a very bad feeling about this feature.
  

On 2024/3/21 16:28, Thomas Monjalon wrote:
> 21/03/2024 03:02, huangdengdui:
>>
>> On 2024/3/20 20:31, Ferruh Yigit wrote:
>>> On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
>>>> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>>>>
>>>>> 12/03/2024 08:52, Dengdui Huang:
>>>>>> Some speeds can be achieved with different number of lanes. For example,
>>>>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
>>>>>> When use different lanes, the port cannot be up.
>>>>>
>>>>> I'm not sure what you are referring to.
>>>>> I suppose it is not PCI lanes.
>>>>> Please could you link to an explanation of how a port is split in lanes?
>>>>> Which hardware does this?
>>>>>
>>>> This is a snapshot of 100Gb that the latest BCM576xx supports.
>>>> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
>>>> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
>>>> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
>>>>
>>>> Let the user feed in lanes=< integer value> and the NIC driver decides
>>>> the matching combination speed x lanes that works. In future if a new speed
>>>> is implemented with more than 8 lanes, there wouldn't be a need
>>>> to touch this speed command. Using separate lane command would
>>>> be a better alternative to support already shipped products and only new
>>>> drivers would consider this lanes configuration, if applicable.
>>>>
>>>
>>> As far as I understand, lane is related to the physical layer of the
>>> NIC, there are multiple copies of transmitter, receiver, modulator HW
>>> block and each set called as a 'lane' and multiple lanes work together
>>> to achieve desired speed. (please correct me if this is wrong).
>>>
>>> Why not just configuring the speed is not enough? Why user needs to know
>>> the detail and configuration of the lanes?
>>> Will it work if driver/device configure the "speed x lane" internally
>>> for the requested speed?
>>>
>>> Is there a benefit to force specific lane count for a specific speed
>>> (like power optimization, just a wild guess)?
>>>
>>>
>>> And +1 for auto-negotiation if possible.
>>
>> As you said above,，multiple lanes work together to achieve desired speed.
>> For example, the following solutions can be used to implement 100G:
>> 1、Combines four 25G lanes
>> 2、Combines two 50G lanes
>> 3、A single 100G lane
>>
>> It is assumed that two ports are interconnected and the two ports support
>> the foregoing three solutions. But, we just configured the speed to 100G and
>> one port uses four 25G lanes by default and the other port uses two 50G lanes
>> by default, the port cannot be up. In this case, we need to configure the
>> two ports to use the same solutions (for example, uses two 50G lanes)
>> so that the ports can be up.
> 
> Why this config is not OK? How do we know?
> Really I have a very bad feeling about this feature.
> 
> 
Sorry, I don't quite understand your question.
Are you asking why cannot be up when one port uses four 25G lanes and the other port uses two 50G lanes?

100GBASE-SR2 (two 50G lanes) and 100GBASE-SR4 (four 25G lanes) have different standards at the physical layer.[1]
So it's not possible to communicate. Configuring lanes can help the driver choose the same standard.

[1]
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9844436
  

On Fri, Mar 22, 2024 at 7:58 AM huangdengdui <huangdengdui@huawei.com> wrote:
>
>
>
> On 2024/3/21 16:28, Thomas Monjalon wrote:
> > 21/03/2024 03:02, huangdengdui:
> >>
> >> On 2024/3/20 20:31, Ferruh Yigit wrote:
> >>> On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
> >>>> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>>>>
> >>>>> 12/03/2024 08:52, Dengdui Huang:
> >>>>>> Some speeds can be achieved with different number of lanes. For example,
> >>>>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> >>>>>> When use different lanes, the port cannot be up.
> >>>>>
> >>>>> I'm not sure what you are referring to.
> >>>>> I suppose it is not PCI lanes.
> >>>>> Please could you link to an explanation of how a port is split in lanes?
> >>>>> Which hardware does this?
> >>>>>
> >>>> This is a snapshot of 100Gb that the latest BCM576xx supports.
> >>>> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> >>>> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> >>>> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> >>>>
> >>>> Let the user feed in lanes=< integer value> and the NIC driver decides
> >>>> the matching combination speed x lanes that works. In future if a new speed
> >>>> is implemented with more than 8 lanes, there wouldn't be a need
> >>>> to touch this speed command. Using separate lane command would
> >>>> be a better alternative to support already shipped products and only new
> >>>> drivers would consider this lanes configuration, if applicable.
> >>>>
> >>>
> >>> As far as I understand, lane is related to the physical layer of the
> >>> NIC, there are multiple copies of transmitter, receiver, modulator HW
> >>> block and each set called as a 'lane' and multiple lanes work together
> >>> to achieve desired speed. (please correct me if this is wrong).
> >>>
> >>> Why not just configuring the speed is not enough? Why user needs to know
> >>> the detail and configuration of the lanes?
> >>> Will it work if driver/device configure the "speed x lane" internally
> >>> for the requested speed?
> >>>
> >>> Is there a benefit to force specific lane count for a specific speed
> >>> (like power optimization, just a wild guess)?
> >>>
> >>>
> >>> And +1 for auto-negotiation if possible.
> >>
> >> As you said above,，multiple lanes work together to achieve desired speed.
> >> For example, the following solutions can be used to implement 100G:
> >> 1、Combines four 25G lanes
> >> 2、Combines two 50G lanes
> >> 3、A single 100G lane
> >>
> >> It is assumed that two ports are interconnected and the two ports support
> >> the foregoing three solutions. But, we just configured the speed to 100G and
> >> one port uses four 25G lanes by default and the other port uses two 50G lanes
> >> by default, the port cannot be up. In this case, we need to configure the
> >> two ports to use the same solutions (for example, uses two 50G lanes)
> >> so that the ports can be up.
> >
> > Why this config is not OK? How do we know?
> > Really I have a very bad feeling about this feature.
> >
> >
> Sorry, I don't quite understand your question.
> Are you asking why cannot be up when one port uses four 25G lanes and the other port uses two 50G lanes?
>
> 100GBASE-SR2 (two 50G lanes) and 100GBASE-SR4 (four 25G lanes) have different standards at the physical layer.[1]
> So it's not possible to communicate. Configuring lanes can help the driver choose the same standard.

Typically, low-level drivers like FW configure this.

For example, If FW configures, 100G port as 100GBASE-SR2 then two
ethdev(port 0 and port1) will show up.
Now, assume if we expose this API and Can end user configure port 1 as
25G lines if so,
a) What happens to port0 and it states?
b) Will port2, port3 will show up after issuing this API(As end user
configured 25Gx4 for 100G)? Will application needs to hotplug to get
use ports.

> https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9844436
  

On Thu, Mar 21, 2024 at 9:39 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Fri, Mar 22, 2024 at 7:58 AM huangdengdui <huangdengdui@huawei.com> wrote:
> >
> >
> >
> > On 2024/3/21 16:28, Thomas Monjalon wrote:
> > > 21/03/2024 03:02, huangdengdui:
> > >>
> > >> On 2024/3/20 20:31, Ferruh Yigit wrote:
> > >>> On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
> > >>>> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >>>>>
> > >>>>> 12/03/2024 08:52, Dengdui Huang:
> > >>>>>> Some speeds can be achieved with different number of lanes. For example,
> > >>>>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> > >>>>>> When use different lanes, the port cannot be up.
> > >>>>>
> > >>>>> I'm not sure what you are referring to.
> > >>>>> I suppose it is not PCI lanes.
> > >>>>> Please could you link to an explanation of how a port is split in lanes?
> > >>>>> Which hardware does this?
> > >>>>>
> > >>>> This is a snapshot of 100Gb that the latest BCM576xx supports.
> > >>>> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> > >>>> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> > >>>> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> > >>>>
> > >>>> Let the user feed in lanes=< integer value> and the NIC driver decides
> > >>>> the matching combination speed x lanes that works. In future if a new speed
> > >>>> is implemented with more than 8 lanes, there wouldn't be a need
> > >>>> to touch this speed command. Using separate lane command would
> > >>>> be a better alternative to support already shipped products and only new
> > >>>> drivers would consider this lanes configuration, if applicable.
> > >>>>
> > >>>
> > >>> As far as I understand, lane is related to the physical layer of the
> > >>> NIC, there are multiple copies of transmitter, receiver, modulator HW
> > >>> block and each set called as a 'lane' and multiple lanes work together
> > >>> to achieve desired speed. (please correct me if this is wrong).
> > >>>
> > >>> Why not just configuring the speed is not enough? Why user needs to know
> > >>> the detail and configuration of the lanes?
> > >>> Will it work if driver/device configure the "speed x lane" internally
> > >>> for the requested speed?
> > >>>
> > >>> Is there a benefit to force specific lane count for a specific speed
> > >>> (like power optimization, just a wild guess)?
> > >>>
> > >>>
> > >>> And +1 for auto-negotiation if possible.
> > >>
> > >> As you said above,，multiple lanes work together to achieve desired speed.
> > >> For example, the following solutions can be used to implement 100G:
> > >> 1、Combines four 25G lanes
> > >> 2、Combines two 50G lanes
> > >> 3、A single 100G lane
> > >>
> > >> It is assumed that two ports are interconnected and the two ports support
> > >> the foregoing three solutions. But, we just configured the speed to 100G and
> > >> one port uses four 25G lanes by default and the other port uses two 50G lanes
> > >> by default, the port cannot be up. In this case, we need to configure the
> > >> two ports to use the same solutions (for example, uses two 50G lanes)
> > >> so that the ports can be up.
> > >
> > > Why this config is not OK? How do we know?
> > > Really I have a very bad feeling about this feature.
> > >
> > >
> > Sorry, I don't quite understand your question.
> > Are you asking why cannot be up when one port uses four 25G lanes and the other port uses two 50G lanes?
> >
> > 100GBASE-SR2 (two 50G lanes) and 100GBASE-SR4 (four 25G lanes) have different standards at the physical layer.[1]
> > So it's not possible to communicate. Configuring lanes can help the driver choose the same standard.
>
> Typically, low-level drivers like FW configure this.
>
> For example, If FW configures, 100G port as 100GBASE-SR2 then two
> ethdev(port 0 and port1) will show up.
> Now, assume if we expose this API and Can end user configure port 1 as
> 25G lines if so,
> a) What happens to port0 and it states?
There should be no impact to port0.

> b) Will port2, port3 will show up after issuing this API(As end user
> configured 25Gx4 for 100G)? Will application needs to hotplug to get
> use ports.
No. The port count does not change. Nor does the number of PCI
functions seen by the host. Unless designed otherwise.

Changing the lane count does not change anything in physical terms.
What changes is the modulation or the signaling scheme.
The number of lanes which can be supported is determined by
the PHY itself and the cables used and needs to be negotiated appropriately
with the remote partner - which is just like using forced Ethernet Speed
instead of auto-negotiated speeds.

I tried to search for some links, but it has not been easy.
So let me try to put something down. Sorry for the long mail.

In the above example, if the 100G port is configured for 2 lanes,
the SerDes in the PHY is configured such that each lane communicates
at 50G with the remote partner.
If the port is configured for 4 lanes, the internal SerDes is programmed
to use a lower 25G signaling scheme.

Each lane provides a parallel data transmission path and are
typically implemented using multiple pairs of copper wires or optical fibers.

When the lane configuration of the Ethernet port is changed, the port speed
does not change. But the PHY FW uses a suitable modulation scheme to
achieve the total aggregate port speed.

The choice of modulation schemes include NRZ and PAM4.

Typically 100G uses four lanes at 25 Gbps each and happens to be NRZ.
For higher Ethernet speeds, the shift has been to use PAM4 signaling
which allows twice the transmission rates compared to NRZ signaling.

So the same 100G port can be configured to send signals to the remote
partner using two lanes at 50G + 50G, or just 1 lane operating at 100G.

It gets interesting as the speeds increase.
200G can be achieved with 4 lanes at 50G or 2 lanes at 100G.

PAM4 offers improved bandwidth efficiency by encoding two bits per symbol
compared to NRZ's one bit per symbol. The Ethernet standards committee
I think has already completed work on 200G speed per lane.

BTW -
PAM4 stands for 4-Level Pulse Amplitude Modulation.
NRZ stands for Non-Return-to-Zero.

Thanks
Ajit

>
> > https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9844436
  

On Fri, Mar 22, 2024 at 10:56 AM Ajit Khaparde
<ajit.khaparde@broadcom.com> wrote:
>
> On Thu, Mar 21, 2024 at 9:39 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > On Fri, Mar 22, 2024 at 7:58 AM huangdengdui <huangdengdui@huawei.com> wrote:
> > >
> > >
> > >
> > > On 2024/3/21 16:28, Thomas Monjalon wrote:
> > > > 21/03/2024 03:02, huangdengdui:
> > > >>
> > > >> On 2024/3/20 20:31, Ferruh Yigit wrote:
> > > >>> On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
> > > >>>> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > >>>>>
> > > >>>>> 12/03/2024 08:52, Dengdui Huang:
> > > >>>>>> Some speeds can be achieved with different number of lanes. For example,
> > > >>>>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> > > >>>>>> When use different lanes, the port cannot be up.
> > > >>>>>
> > > >>>>> I'm not sure what you are referring to.
> > > >>>>> I suppose it is not PCI lanes.
> > > >>>>> Please could you link to an explanation of how a port is split in lanes?
> > > >>>>> Which hardware does this?
> > > >>>>>
> > > >>>> This is a snapshot of 100Gb that the latest BCM576xx supports.
> > > >>>> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> > > >>>> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> > > >>>> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> > > >>>>
> > > >>>> Let the user feed in lanes=< integer value> and the NIC driver decides
> > > >>>> the matching combination speed x lanes that works. In future if a new speed
> > > >>>> is implemented with more than 8 lanes, there wouldn't be a need
> > > >>>> to touch this speed command. Using separate lane command would
> > > >>>> be a better alternative to support already shipped products and only new
> > > >>>> drivers would consider this lanes configuration, if applicable.
> > > >>>>
> > > >>>
> > > >>> As far as I understand, lane is related to the physical layer of the
> > > >>> NIC, there are multiple copies of transmitter, receiver, modulator HW
> > > >>> block and each set called as a 'lane' and multiple lanes work together
> > > >>> to achieve desired speed. (please correct me if this is wrong).
> > > >>>
> > > >>> Why not just configuring the speed is not enough? Why user needs to know
> > > >>> the detail and configuration of the lanes?
> > > >>> Will it work if driver/device configure the "speed x lane" internally
> > > >>> for the requested speed?
> > > >>>
> > > >>> Is there a benefit to force specific lane count for a specific speed
> > > >>> (like power optimization, just a wild guess)?
> > > >>>
> > > >>>
> > > >>> And +1 for auto-negotiation if possible.
> > > >>
> > > >> As you said above,，multiple lanes work together to achieve desired speed.
> > > >> For example, the following solutions can be used to implement 100G:
> > > >> 1、Combines four 25G lanes
> > > >> 2、Combines two 50G lanes
> > > >> 3、A single 100G lane
> > > >>
> > > >> It is assumed that two ports are interconnected and the two ports support
> > > >> the foregoing three solutions. But, we just configured the speed to 100G and
> > > >> one port uses four 25G lanes by default and the other port uses two 50G lanes
> > > >> by default, the port cannot be up. In this case, we need to configure the
> > > >> two ports to use the same solutions (for example, uses two 50G lanes)
> > > >> so that the ports can be up.
> > > >
> > > > Why this config is not OK? How do we know?
> > > > Really I have a very bad feeling about this feature.
> > > >
> > > >
> > > Sorry, I don't quite understand your question.
> > > Are you asking why cannot be up when one port uses four 25G lanes and the other port uses two 50G lanes?
> > >
> > > 100GBASE-SR2 (two 50G lanes) and 100GBASE-SR4 (four 25G lanes) have different standards at the physical layer.[1]
> > > So it's not possible to communicate. Configuring lanes can help the driver choose the same standard.
> >
> > Typically, low-level drivers like FW configure this.
> >
> > For example, If FW configures, 100G port as 100GBASE-SR2 then two
> > ethdev(port 0 and port1) will show up.
> > Now, assume if we expose this API and Can end user configure port 1 as
> > 25G lines if so,
> > a) What happens to port0 and it states?
> There should be no impact to port0.
>
> > b) Will port2, port3 will show up after issuing this API(As end user
> > configured 25Gx4 for 100G)? Will application needs to hotplug to get
> > use ports.
> No. The port count does not change. Nor does the number of PCI
> functions seen by the host. Unless designed otherwise.
>
> Changing the lane count does not change anything in physical terms.
> What changes is the modulation or the signaling scheme.
> The number of lanes which can be supported is determined by
> the PHY itself and the cables used and needs to be negotiated appropriately
> with the remote partner - which is just like using forced Ethernet Speed
> instead of auto-negotiated speeds.

OK. It looks like platform independent then. At least cnxk driver, End
user cannot simplify change the line config parameters
while traffic is active also, it looks like other drivers need to have
SerDes training with remote partner while reconfiguring it.

At least on cnxk platform, 25Gx4 on 100G will show as 4 ethdev devices.

Having said that, If other NICs support this feature without
disturbing current port states, I don't have an objection to this API.
  

22/03/2024 06:51, Jerin Jacob:
> On Fri, Mar 22, 2024 at 10:56 AM Ajit Khaparde
> <ajit.khaparde@broadcom.com> wrote:
> >
> > On Thu, Mar 21, 2024 at 9:39 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > >
> > > On Fri, Mar 22, 2024 at 7:58 AM huangdengdui <huangdengdui@huawei.com> wrote:
> > > >
> > > >
> > > >
> > > > On 2024/3/21 16:28, Thomas Monjalon wrote:
> > > > > 21/03/2024 03:02, huangdengdui:
> > > > >>
> > > > >> On 2024/3/20 20:31, Ferruh Yigit wrote:
> > > > >>> On 3/18/2024 9:26 PM, Damodharam Ammepalli wrote:
> > > > >>>> On Mon, Mar 18, 2024 at 7:56 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > >>>>>
> > > > >>>>> 12/03/2024 08:52, Dengdui Huang:
> > > > >>>>>> Some speeds can be achieved with different number of lanes. For example,
> > > > >>>>>> 100Gbps can be achieved using two lanes of 50Gbps or four lanes of 25Gbps.
> > > > >>>>>> When use different lanes, the port cannot be up.
> > > > >>>>>
> > > > >>>>> I'm not sure what you are referring to.
> > > > >>>>> I suppose it is not PCI lanes.
> > > > >>>>> Please could you link to an explanation of how a port is split in lanes?
> > > > >>>>> Which hardware does this?
> > > > >>>>>
> > > > >>>> This is a snapshot of 100Gb that the latest BCM576xx supports.
> > > > >>>> 100Gb (NRZ: 25G per lane, 4 lanes) link speed
> > > > >>>> 100Gb (PAM4-56: 50G per lane, 2 lanes) link speed
> > > > >>>> 100Gb (PAM4-112: 100G per lane, 1 lane) link speed
> > > > >>>>
> > > > >>>> Let the user feed in lanes=< integer value> and the NIC driver decides
> > > > >>>> the matching combination speed x lanes that works. In future if a new speed
> > > > >>>> is implemented with more than 8 lanes, there wouldn't be a need
> > > > >>>> to touch this speed command. Using separate lane command would
> > > > >>>> be a better alternative to support already shipped products and only new
> > > > >>>> drivers would consider this lanes configuration, if applicable.
> > > > >>>>
> > > > >>>
> > > > >>> As far as I understand, lane is related to the physical layer of the
> > > > >>> NIC, there are multiple copies of transmitter, receiver, modulator HW
> > > > >>> block and each set called as a 'lane' and multiple lanes work together
> > > > >>> to achieve desired speed. (please correct me if this is wrong).
> > > > >>>
> > > > >>> Why not just configuring the speed is not enough? Why user needs to know
> > > > >>> the detail and configuration of the lanes?
> > > > >>> Will it work if driver/device configure the "speed x lane" internally
> > > > >>> for the requested speed?
> > > > >>>
> > > > >>> Is there a benefit to force specific lane count for a specific speed
> > > > >>> (like power optimization, just a wild guess)?
> > > > >>>
> > > > >>>
> > > > >>> And +1 for auto-negotiation if possible.
> > > > >>
> > > > >> As you said above,，multiple lanes work together to achieve desired speed.
> > > > >> For example, the following solutions can be used to implement 100G:
> > > > >> 1、Combines four 25G lanes
> > > > >> 2、Combines two 50G lanes
> > > > >> 3、A single 100G lane
> > > > >>
> > > > >> It is assumed that two ports are interconnected and the two ports support
> > > > >> the foregoing three solutions. But, we just configured the speed to 100G and
> > > > >> one port uses four 25G lanes by default and the other port uses two 50G lanes
> > > > >> by default, the port cannot be up. In this case, we need to configure the
> > > > >> two ports to use the same solutions (for example, uses two 50G lanes)
> > > > >> so that the ports can be up.
> > > > >
> > > > > Why this config is not OK? How do we know?
> > > > > Really I have a very bad feeling about this feature.
> > > > >
> > > > >
> > > > Sorry, I don't quite understand your question.
> > > > Are you asking why cannot be up when one port uses four 25G lanes and the other port uses two 50G lanes?
> > > >
> > > > 100GBASE-SR2 (two 50G lanes) and 100GBASE-SR4 (four 25G lanes) have different standards at the physical layer.[1]
> > > > So it's not possible to communicate. Configuring lanes can help the driver choose the same standard.
> > >
> > > Typically, low-level drivers like FW configure this.
> > >
> > > For example, If FW configures, 100G port as 100GBASE-SR2 then two
> > > ethdev(port 0 and port1) will show up.
> > > Now, assume if we expose this API and Can end user configure port 1 as
> > > 25G lines if so,
> > > a) What happens to port0 and it states?
> > There should be no impact to port0.
> >
> > > b) Will port2, port3 will show up after issuing this API(As end user
> > > configured 25Gx4 for 100G)? Will application needs to hotplug to get
> > > use ports.
> > No. The port count does not change. Nor does the number of PCI
> > functions seen by the host. Unless designed otherwise.
> >
> > Changing the lane count does not change anything in physical terms.
> > What changes is the modulation or the signaling scheme.
> > The number of lanes which can be supported is determined by
> > the PHY itself and the cables used and needs to be negotiated appropriately
> > with the remote partner - which is just like using forced Ethernet Speed
> > instead of auto-negotiated speeds.

Thanks for the explanation Ajit.

> OK. It looks like platform independent then. At least cnxk driver, End
> user cannot simplify change the line config parameters
> while traffic is active also, it looks like other drivers need to have
> SerDes training with remote partner while reconfiguring it.
> 
> At least on cnxk platform, 25Gx4 on 100G will show as 4 ethdev devices.

That's a strange behaviour.
Why showing 4 ports which are not independent?

> Having said that, If other NICs support this feature without
> disturbing current port states, I don't have an objection to this API.
  

On Fri, Mar 22, 2024 at 7:21 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 22/03/2024 06:51, Jerin Jacob:
> > On Fri, Mar 22, 2024 at 10:56 AM Ajit Khaparde
> > <ajit.khaparde@broadcom.com> wrote:
> > >
> > > On Thu, Mar 21, 2024 at 9:39 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > > >
> > > > On Fri, Mar 22, 2024 at 7:58 AM huangdengdui <huangdengdui@huawei.com> wrote:
> > > > >
> > > > >
> > > > >

> > > > For example, If FW configures, 100G port as 100GBASE-SR2 then two
> > > > ethdev(port 0 and port1) will show up.
> > > > Now, assume if we expose this API and Can end user configure port 1 as
> > > > 25G lines if so,
> > > > a) What happens to port0 and it states?
> > > There should be no impact to port0.
> > >
> > > > b) Will port2, port3 will show up after issuing this API(As end user
> > > > configured 25Gx4 for 100G)? Will application needs to hotplug to get
> > > > use ports.
> > > No. The port count does not change. Nor does the number of PCI
> > > functions seen by the host. Unless designed otherwise.
> > >
> > > Changing the lane count does not change anything in physical terms.
> > > What changes is the modulation or the signaling scheme.
> > > The number of lanes which can be supported is determined by
> > > the PHY itself and the cables used and needs to be negotiated appropriately
> > > with the remote partner - which is just like using forced Ethernet Speed
> > > instead of auto-negotiated speeds.
>
> Thanks for the explanation Ajit.
>
> > OK. It looks like platform independent then. At least cnxk driver, End
> > user cannot simplify change the line config parameters
> > while traffic is active also, it looks like other drivers need to have
> > SerDes training with remote partner while reconfiguring it.
> >
> > At least on cnxk platform, 25Gx4 on 100G will show as 4 ethdev devices.
>
> That's a strange behaviour.
> Why showing 4 ports which are not independent?

I checked SerDes + NIC configuration again. It supports both modes.
Show up as One port vs four ports.

>
> > Having said that, If other NICs support this feature without
> > disturbing current port states, I don't have an objection to this API.
>
>
>