[1/3] ixgbe: make link update thread periodic
Checks
Commit Message
Rather than run-to-completion, allow the link update thread to be
periodic. This will set the stage for properly handling hot-plugging.
Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
Inspired-by: Stephen Douthit <stephend@silicom-usa.com>
---
drivers/net/ixgbe/base/ixgbe_common.c | 4 +-
drivers/net/ixgbe/ixgbe_ethdev.c | 180 ++++++++++----------------
2 files changed, 71 insertions(+), 113 deletions(-)
Comments
> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Friday, May 20, 2022 2:03 AM
> To: dev@dpdk.org; Yang, Qiming <qiming.yang@intel.com>; Wu, Wenjun1
> <wenjun1.wu@intel.com>
> Cc: Stephen Douthit <stephend@silicom-usa.com>
> Subject: [PATCH 1/3] ixgbe: make link update thread periodic
>
> Rather than run-to-completion, allow the link update thread to be periodic.
> This will set the stage for properly handling hot-plugging.
Could you explain more about what's the hot-plugging issue with run-to-completion you try to fix?
>
> Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> Inspired-by: Stephen Douthit <stephend@silicom-usa.com>
> ---
> drivers/net/ixgbe/base/ixgbe_common.c | 4 +-
> drivers/net/ixgbe/ixgbe_ethdev.c | 180 ++++++++++----------------
> 2 files changed, 71 insertions(+), 113 deletions(-)
>
> diff --git a/drivers/net/ixgbe/base/ixgbe_common.c
> b/drivers/net/ixgbe/base/ixgbe_common.c
> index aa843bd5c4a5..712062306491 100644
> --- a/drivers/net/ixgbe/base/ixgbe_common.c
> +++ b/drivers/net/ixgbe/base/ixgbe_common.c
> @@ -4154,8 +4154,8 @@ s32 ixgbe_check_mac_link_generic(struct
> ixgbe_hw *hw, ixgbe_link_speed *speed,
> break;
> case ixgbe_mac_X550EM_x:
> case ixgbe_mac_X550EM_a:
> - sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP)
> &
> - IXGBE_ESDP_SDP0;
> + sfp_cage_full = !(IXGBE_READ_REG(hw, IXGBE_ESDP)
> &
> + IXGBE_ESDP_SDP0);
> break;
> default:
> /* sanity check - No SFP+ devices here */ diff --git
Looks like you change the behavior of link status check for x550.
I'm not an ixgbe expert, but I know this is not kernel driver's implementation.
So do you think this is a fix for both DPDK and kernel driver? if it is, please move this change into a separate patch and we need to reach the right expert to approve this.
> -----Original Message-----
> From: Zhang, Qi Z <qi.z.zhang@intel.com>
> Sent: Sunday, May 29, 2022 7:25 PM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> Cc: Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
>
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
>
>
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Friday, May 20, 2022 2:03 AM
> > To: dev@dpdk.org; Yang, Qiming <qiming.yang@intel.com>; Wu, Wenjun1
> > <wenjun1.wu@intel.com>
> > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > Subject: [PATCH 1/3] ixgbe: make link update thread periodic
> >
> > Rather than run-to-completion, allow the link update thread to be
> periodic.
> > This will set the stage for properly handling hot-plugging.
>
> Could you explain more about what's the hot-plugging issue with run-to-
> completion you try to fix?
>
it doesn't work right when you have SFPs. (at least not on our platform or on an
82599 dual SFP add-in card we have). run-to-completion only works 1x. if you
remove and plug in a different SFP it doesn't work. This patch series should have
been taking in context with the original SFP hotplug patch but apparently since
I can't ever seem to get the patch submission threading to do what I mean perhaps
some context has gone missing. the SFP hotplug fix has been in the queue since
Dec 2021, has been reworked several times, has gone through a change in Intel
maintainership. this patch series makes the SFP hot plugging work like the Intel
kernel driver does.
> >
> > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > Inspired-by: Stephen Douthit <stephend@silicom-usa.com>
> > ---
> > drivers/net/ixgbe/base/ixgbe_common.c | 4 +-
> > drivers/net/ixgbe/ixgbe_ethdev.c | 180 ++++++++++----------------
> > 2 files changed, 71 insertions(+), 113 deletions(-)
> >
> > diff --git a/drivers/net/ixgbe/base/ixgbe_common.c
> > b/drivers/net/ixgbe/base/ixgbe_common.c
> > index aa843bd5c4a5..712062306491 100644
> > --- a/drivers/net/ixgbe/base/ixgbe_common.c
> > +++ b/drivers/net/ixgbe/base/ixgbe_common.c
> > @@ -4154,8 +4154,8 @@ s32 ixgbe_check_mac_link_generic(struct
> > ixgbe_hw *hw, ixgbe_link_speed *speed,
> > break;
> > case ixgbe_mac_X550EM_x:
> > case ixgbe_mac_X550EM_a:
> > - sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP)
> > &
> > - IXGBE_ESDP_SDP0;
> > + sfp_cage_full = !(IXGBE_READ_REG(hw, IXGBE_ESDP)
> > &
> > + IXGBE_ESDP_SDP0);
> > break;
> > default:
> > /* sanity check - No SFP+ devices here */ diff
> > --git
>
> Looks like you change the behavior of link status check for x550.
> I'm not an ixgbe expert, but I know this is not kernel driver's
> implementation.
>
sigh. this was supposed to be part of a different patch which also had some
question about functionality. the SDP0 bit check doesn't specifically need to be
a check for a '1', since the bit reflects the state of the pin on the platform. Intel's
platform implementations have an inverter on the board to switch the state.
MOD_ABS from an SFP will be '0' when an SFP is plugged in. with an inverter in
the platform the signal will be '1' when an SFP is plugged in. there's no guidance
from Intel's platform design guide that an inverter needs to be between the SFP
and the NIC SDP pin so having it only follow Intel's platform implementations is
hard to justify.
> So do you think this is a fix for both DPDK and kernel driver? if it is, please
> move this change into a separate patch and we need to reach the right
> expert to approve this.
>
>
no, as explained above.
> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Monday, May 30, 2022 9:47 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org; Yang, Qiming
> <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> Cc: Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
>
>
>
> > -----Original Message-----
> > From: Zhang, Qi Z <qi.z.zhang@intel.com>
> > Sent: Sunday, May 29, 2022 7:25 PM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> >
> > Caution: This is an external email. Please take care when clicking
> > links or opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Friday, May 20, 2022 2:03 AM
> > > To: dev@dpdk.org; Yang, Qiming <qiming.yang@intel.com>; Wu, Wenjun1
> > > <wenjun1.wu@intel.com>
> > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > Subject: [PATCH 1/3] ixgbe: make link update thread periodic
> > >
> > > Rather than run-to-completion, allow the link update thread to be
> > periodic.
> > > This will set the stage for properly handling hot-plugging.
> >
> > Could you explain more about what's the hot-plugging issue with
> > run-to- completion you try to fix?
> >
>
> it doesn't work right when you have SFPs. (at least not on our platform or on
> an
> 82599 dual SFP add-in card we have). run-to-completion only works 1x. if
> you remove and plug in a different SFP it doesn't work. This patch series
> should have been taking in context with the original SFP hotplug patch but
> apparently since I can't ever seem to get the patch submission threading to do
> what I mean perhaps some context has gone missing. the SFP hotplug fix has
> been in the queue since Dec 2021, has been reworked several times, has gone
> through a change in Intel maintainership. this patch series makes the SFP hot
> plugging work like the Intel kernel driver does.
>
> > >
> > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > Inspired-by: Stephen Douthit <stephend@silicom-usa.com>
> > > ---
> > > drivers/net/ixgbe/base/ixgbe_common.c | 4 +-
> > > drivers/net/ixgbe/ixgbe_ethdev.c | 180 ++++++++++----------------
> > > 2 files changed, 71 insertions(+), 113 deletions(-)
> > >
> > > diff --git a/drivers/net/ixgbe/base/ixgbe_common.c
> > > b/drivers/net/ixgbe/base/ixgbe_common.c
> > > index aa843bd5c4a5..712062306491 100644
> > > --- a/drivers/net/ixgbe/base/ixgbe_common.c
> > > +++ b/drivers/net/ixgbe/base/ixgbe_common.c
> > > @@ -4154,8 +4154,8 @@ s32 ixgbe_check_mac_link_generic(struct
> > > ixgbe_hw *hw, ixgbe_link_speed *speed,
> > > break;
> > > case ixgbe_mac_X550EM_x:
> > > case ixgbe_mac_X550EM_a:
> > > - sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP)
> > > &
> > > - IXGBE_ESDP_SDP0;
> > > + sfp_cage_full = !(IXGBE_READ_REG(hw,
> > > + IXGBE_ESDP)
> > > &
> > > + IXGBE_ESDP_SDP0);
> > > break;
> > > default:
> > > /* sanity check - No SFP+ devices here */ diff
> > > --git
> >
> > Looks like you change the behavior of link status check for x550.
> > I'm not an ixgbe expert, but I know this is not kernel driver's
> > implementation.
> >
>
> sigh. this was supposed to be part of a different patch which also had some
> question about functionality. the SDP0 bit check doesn't specifically need to
> be a check for a '1', since the bit reflects the state of the pin on the platform.
> Intel's platform implementations have an inverter on the board to switch the
> state.
> MOD_ABS from an SFP will be '0' when an SFP is plugged in. with an inverter
> in the platform the signal will be '1' when an SFP is plugged in. there's no
> guidance from Intel's platform design guide that an inverter needs to be
> between the SFP and the NIC SDP pin so having it only follow Intel's platform
> implementations is hard to justify.
OK, I assume the existing code should be proved for the normal scenario (remove and plug in with the same SFP)
So how can we guarantee this change will not break something?
Could you help me to understand why we should ignore the difference when SDP0 is 1 in normal scenario?
Before change
we will continue to read the link register and return the link speed.
after change
we return SPEED_UNKNOWN immediately .
>
> > So do you think this is a fix for both DPDK and kernel driver? if it
> > is, please move this change into a separate patch and we need to
> > reach the right expert to approve this.
> >
> >
>
> no, as explained above.
> -----Original Message-----
> From: Zhang, Qi Z <qi.z.zhang@intel.com>
> Sent: Monday, May 30, 2022 10:21 PM
> To: Daly, Jeff <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> Cc: Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
>
>
>
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Monday, May 30, 2022 9:47 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org; Yang, Qiming
> > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhang, Qi Z <qi.z.zhang@intel.com>
> > > Sent: Sunday, May 29, 2022 7:25 PM
> > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> > > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> > >
> > > Caution: This is an external email. Please take care when clicking
> > > links or opening attachments.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > Sent: Friday, May 20, 2022 2:03 AM
> > > > To: dev@dpdk.org; Yang, Qiming <qiming.yang@intel.com>; Wu,
> > > > Wenjun1 <wenjun1.wu@intel.com>
> > > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > > Subject: [PATCH 1/3] ixgbe: make link update thread periodic
> > > >
> > > > Rather than run-to-completion, allow the link update thread to be
> > > periodic.
> > > > This will set the stage for properly handling hot-plugging.
> > >
> > > Could you explain more about what's the hot-plugging issue with
> > > run-to- completion you try to fix?
> > >
> >
> > it doesn't work right when you have SFPs. (at least not on our
> > platform or on an
> > 82599 dual SFP add-in card we have). run-to-completion only works 1x.
What is 1x means?
> > if you remove and plug in a different SFP it doesn't work. This patch
> > series should have been taking in context with the original SFP
> > hotplug patch but apparently since I can't ever seem to get the patch
> > submission threading to do what I mean perhaps some context has gone
> > missing. the SFP hotplug fix has been in the queue since Dec 2021,
> > has been reworked several times, has gone through a change in Intel
> > maintainership. this patch series makes the SFP hot plugging work like the
> Intel kernel driver does.
Basically, I'm asking for adding more detail information in the commit log , so people can understand why run-to-completion does not work with hot plugin
if you think some patches already be merged that helps to understand the design, you may add a commit ID as reference.
> >
> > > >
> > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > Inspired-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > ---
> > > > drivers/net/ixgbe/base/ixgbe_common.c | 4 +-
> > > > drivers/net/ixgbe/ixgbe_ethdev.c | 180 ++++++++++----------------
> > > > 2 files changed, 71 insertions(+), 113 deletions(-)
> > > >
> > > > diff --git a/drivers/net/ixgbe/base/ixgbe_common.c
> > > > b/drivers/net/ixgbe/base/ixgbe_common.c
> > > > index aa843bd5c4a5..712062306491 100644
> > > > --- a/drivers/net/ixgbe/base/ixgbe_common.c
> > > > +++ b/drivers/net/ixgbe/base/ixgbe_common.c
> > > > @@ -4154,8 +4154,8 @@ s32 ixgbe_check_mac_link_generic(struct
> > > > ixgbe_hw *hw, ixgbe_link_speed *speed,
> > > > break;
> > > > case ixgbe_mac_X550EM_x:
> > > > case ixgbe_mac_X550EM_a:
> > > > - sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP)
> > > > &
> > > > - IXGBE_ESDP_SDP0;
> > > > + sfp_cage_full = !(IXGBE_READ_REG(hw,
> > > > + IXGBE_ESDP)
> > > > &
> > > > + IXGBE_ESDP_SDP0);
> > > > break;
> > > > default:
> > > > /* sanity check - No SFP+ devices here */
> > > > diff --git
> > >
> > > Looks like you change the behavior of link status check for x550.
> > > I'm not an ixgbe expert, but I know this is not kernel driver's
> > > implementation.
> > >
> >
> > sigh. this was supposed to be part of a different patch which also
> > had some question about functionality. the SDP0 bit check doesn't
> > specifically need to be a check for a '1', since the bit reflects the state of the
> pin on the platform.
> > Intel's platform implementations have an inverter on the board to
> > switch the state.
> > MOD_ABS from an SFP will be '0' when an SFP is plugged in. with an
> > inverter in the platform the signal will be '1' when an SFP is plugged
> > in. there's no guidance from Intel's platform design guide that an
> > inverter needs to be between the SFP and the NIC SDP pin so having it
> > only follow Intel's platform implementations is hard to justify.
>
> OK, I assume the existing code should be proved for the normal scenario
> (remove and plug in with the same SFP) So how can we guarantee this change
> will not break something?
>
> Could you help me to understand why we should ignore the difference when
> SDP0 is 1 in normal scenario?
>
> Before change
> we will continue to read the link register and return the link speed.
>
> after change
> we return SPEED_UNKNOWN immediately .
>
>
>
>
> >
> > > So do you think this is a fix for both DPDK and kernel driver? if
> > > it is, please move this change into a separate patch and we need to
> > > reach the right expert to approve this.
> > >
> > >
> >
> > no, as explained above.
> -----Original Message-----
> From: Zhang, Qi Z <qi.z.zhang@intel.com>
> Sent: Monday, May 30, 2022 10:21 AM
> To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> Cc: Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
>
> Caution: This is an external email. Please take care when clicking links or
> opening attachments.
>
>
> > -----Original Message-----
> > From: Jeff Daly <jeffd@silicom-usa.com>
> > Sent: Monday, May 30, 2022 9:47 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org; Yang, Qiming
> > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhang, Qi Z <qi.z.zhang@intel.com>
> > > Sent: Sunday, May 29, 2022 7:25 PM
> > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> > > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> > >
> > > Caution: This is an external email. Please take care when clicking
> > > links or opening attachments.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > Sent: Friday, May 20, 2022 2:03 AM
> > > > To: dev@dpdk.org; Yang, Qiming <qiming.yang@intel.com>; Wu,
> > > > Wenjun1 <wenjun1.wu@intel.com>
> > > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > > Subject: [PATCH 1/3] ixgbe: make link update thread periodic
> > > >
> > > > Rather than run-to-completion, allow the link update thread to be
> > > periodic.
> > > > This will set the stage for properly handling hot-plugging.
> > >
> > > Could you explain more about what's the hot-plugging issue with
> > > run-to- completion you try to fix?
> > >
> >
> > it doesn't work right when you have SFPs. (at least not on our
> > platform or on an
> > 82599 dual SFP add-in card we have). run-to-completion only works 1x.
> > if you remove and plug in a different SFP it doesn't work. This patch
> > series should have been taking in context with the original SFP
> > hotplug patch but apparently since I can't ever seem to get the patch
> > submission threading to do what I mean perhaps some context has gone
> > missing. the SFP hotplug fix has been in the queue since Dec 2021,
> > has been reworked several times, has gone through a change in Intel
> > maintainership. this patch series makes the SFP hot plugging work like the
> Intel kernel driver does.
> >
> > > >
> > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > Inspired-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > ---
> > > > drivers/net/ixgbe/base/ixgbe_common.c | 4 +-
> > > > drivers/net/ixgbe/ixgbe_ethdev.c | 180 ++++++++++----------------
> > > > 2 files changed, 71 insertions(+), 113 deletions(-)
> > > >
> > > > diff --git a/drivers/net/ixgbe/base/ixgbe_common.c
> > > > b/drivers/net/ixgbe/base/ixgbe_common.c
> > > > index aa843bd5c4a5..712062306491 100644
> > > > --- a/drivers/net/ixgbe/base/ixgbe_common.c
> > > > +++ b/drivers/net/ixgbe/base/ixgbe_common.c
> > > > @@ -4154,8 +4154,8 @@ s32 ixgbe_check_mac_link_generic(struct
> > > > ixgbe_hw *hw, ixgbe_link_speed *speed,
> > > > break;
> > > > case ixgbe_mac_X550EM_x:
> > > > case ixgbe_mac_X550EM_a:
> > > > - sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP)
> > > > &
> > > > - IXGBE_ESDP_SDP0;
> > > > + sfp_cage_full = !(IXGBE_READ_REG(hw,
> > > > + IXGBE_ESDP)
> > > > &
> > > > + IXGBE_ESDP_SDP0);
> > > > break;
> > > > default:
> > > > /* sanity check - No SFP+ devices here */
> > > > diff --git
> > >
> > > Looks like you change the behavior of link status check for x550.
> > > I'm not an ixgbe expert, but I know this is not kernel driver's
> > > implementation.
> > >
> >
> > sigh. this was supposed to be part of a different patch which also
> > had some question about functionality. the SDP0 bit check doesn't
> > specifically need to be a check for a '1', since the bit reflects the state of
> the pin on the platform.
> > Intel's platform implementations have an inverter on the board to
> > switch the state.
> > MOD_ABS from an SFP will be '0' when an SFP is plugged in. with an
> > inverter in the platform the signal will be '1' when an SFP is plugged
> > in. there's no guidance from Intel's platform design guide that an
> > inverter needs to be between the SFP and the NIC SDP pin so having it
> > only follow Intel's platform implementations is hard to justify.
>
> OK, I assume the existing code should be proved for the normal scenario
> (remove and plug in with the same SFP) So how can we guarantee this
> change will not break something?
>
> Could you help me to understand why we should ignore the difference when
> SDP0 is 1 in normal scenario?
>
'normal scenario'? meaning on any Intel-implemented platform?
the code needs a devargs-like option so that platforms that don't implement
the inverter can still use this code in the expected way.
> Before change
> we will continue to read the link register and return the link speed.
>
> after change
> we return SPEED_UNKNOWN immediately .
>
>
>
>
> >
> > > So do you think this is a fix for both DPDK and kernel driver? if
> > > it is, please move this change into a separate patch and we need to
> > > reach the right expert to approve this.
> > >
> > >
> >
> > no, as explained above.
> -----Original Message-----
> From: Jeff Daly <jeffd@silicom-usa.com>
> Sent: Tuesday, May 31, 2022 8:25 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org; Yang, Qiming
> <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> Cc: Stephen Douthit <stephend@silicom-usa.com>
> Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
>
>
>
> > -----Original Message-----
> > From: Zhang, Qi Z <qi.z.zhang@intel.com>
> > Sent: Monday, May 30, 2022 10:21 AM
> > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> >
> > Caution: This is an external email. Please take care when clicking
> > links or opening attachments.
> >
> >
> > > -----Original Message-----
> > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > Sent: Monday, May 30, 2022 9:47 PM
> > > To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org; Yang, Qiming
> > > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Zhang, Qi Z <qi.z.zhang@intel.com>
> > > > Sent: Sunday, May 29, 2022 7:25 PM
> > > > To: Jeff Daly <jeffd@silicom-usa.com>; dev@dpdk.org; Yang, Qiming
> > > > <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>
> > > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > > Subject: RE: [PATCH 1/3] ixgbe: make link update thread periodic
> > > >
> > > > Caution: This is an external email. Please take care when clicking
> > > > links or opening attachments.
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Jeff Daly <jeffd@silicom-usa.com>
> > > > > Sent: Friday, May 20, 2022 2:03 AM
> > > > > To: dev@dpdk.org; Yang, Qiming <qiming.yang@intel.com>; Wu,
> > > > > Wenjun1 <wenjun1.wu@intel.com>
> > > > > Cc: Stephen Douthit <stephend@silicom-usa.com>
> > > > > Subject: [PATCH 1/3] ixgbe: make link update thread periodic
> > > > >
> > > > > Rather than run-to-completion, allow the link update thread to
> > > > > be
> > > > periodic.
> > > > > This will set the stage for properly handling hot-plugging.
> > > >
> > > > Could you explain more about what's the hot-plugging issue with
> > > > run-to- completion you try to fix?
> > > >
> > >
> > > it doesn't work right when you have SFPs. (at least not on our
> > > platform or on an
> > > 82599 dual SFP add-in card we have). run-to-completion only works 1x.
> > > if you remove and plug in a different SFP it doesn't work. This
> > > patch series should have been taking in context with the original
> > > SFP hotplug patch but apparently since I can't ever seem to get the
> > > patch submission threading to do what I mean perhaps some context
> > > has gone missing. the SFP hotplug fix has been in the queue since
> > > Dec 2021, has been reworked several times, has gone through a change
> > > in Intel maintainership. this patch series makes the SFP hot
> > > plugging work like the
> > Intel kernel driver does.
> > >
> > > > >
> > > > > Signed-off-by: Jeff Daly <jeffd@silicom-usa.com>
> > > > > Inspired-by: Stephen Douthit <stephend@silicom-usa.com>
> > > > > ---
> > > > > drivers/net/ixgbe/base/ixgbe_common.c | 4 +-
> > > > > drivers/net/ixgbe/ixgbe_ethdev.c | 180 ++++++++++----------------
> > > > > 2 files changed, 71 insertions(+), 113 deletions(-)
> > > > >
> > > > > diff --git a/drivers/net/ixgbe/base/ixgbe_common.c
> > > > > b/drivers/net/ixgbe/base/ixgbe_common.c
> > > > > index aa843bd5c4a5..712062306491 100644
> > > > > --- a/drivers/net/ixgbe/base/ixgbe_common.c
> > > > > +++ b/drivers/net/ixgbe/base/ixgbe_common.c
> > > > > @@ -4154,8 +4154,8 @@ s32 ixgbe_check_mac_link_generic(struct
> > > > > ixgbe_hw *hw, ixgbe_link_speed *speed,
> > > > > break;
> > > > > case ixgbe_mac_X550EM_x:
> > > > > case ixgbe_mac_X550EM_a:
> > > > > - sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP)
> > > > > &
> > > > > - IXGBE_ESDP_SDP0;
> > > > > + sfp_cage_full = !(IXGBE_READ_REG(hw,
> > > > > + IXGBE_ESDP)
> > > > > &
> > > > > + IXGBE_ESDP_SDP0);
> > > > > break;
> > > > > default:
> > > > > /* sanity check - No SFP+ devices here */
> > > > > diff --git
> > > >
> > > > Looks like you change the behavior of link status check for x550.
> > > > I'm not an ixgbe expert, but I know this is not kernel driver's
> > > > implementation.
> > > >
> > >
> > > sigh. this was supposed to be part of a different patch which also
> > > had some question about functionality. the SDP0 bit check doesn't
> > > specifically need to be a check for a '1', since the bit reflects
> > > the state of
> > the pin on the platform.
> > > Intel's platform implementations have an inverter on the board to
> > > switch the state.
> > > MOD_ABS from an SFP will be '0' when an SFP is plugged in. with an
> > > inverter in the platform the signal will be '1' when an SFP is
> > > plugged in. there's no guidance from Intel's platform design guide
> > > that an inverter needs to be between the SFP and the NIC SDP pin so
> > > having it only follow Intel's platform implementations is hard to justify.
> >
> > OK, I assume the existing code should be proved for the normal
> > scenario (remove and plug in with the same SFP) So how can we
> > guarantee this change will not break something?
> >
> > Could you help me to understand why we should ignore the difference
> > when
> > SDP0 is 1 in normal scenario?
> >
>
> 'normal scenario'? meaning on any Intel-implemented platform?
>
> the code needs a devargs-like option so that platforms that don't implement
> the inverter can still use this code in the expected way.
Yes, please move this change into a separate patch and use a devargs for SDP pin invert case.
>
> > Before change
> > we will continue to read the link register and return the link speed.
> >
> > after change
> > we return SPEED_UNKNOWN immediately .
> >
> >
> >
> >
> > >
> > > > So do you think this is a fix for both DPDK and kernel driver? if
> > > > it is, please move this change into a separate patch and we need
> > > > to reach the right expert to approve this.
> > > >
> > > >
> > >
> > > no, as explained above.
@@ -4154,8 +4154,8 @@ s32 ixgbe_check_mac_link_generic(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
break;
case ixgbe_mac_X550EM_x:
case ixgbe_mac_X550EM_a:
- sfp_cage_full = IXGBE_READ_REG(hw, IXGBE_ESDP) &
- IXGBE_ESDP_SDP0;
+ sfp_cage_full = !(IXGBE_READ_REG(hw, IXGBE_ESDP) &
+ IXGBE_ESDP_SDP0);
break;
default:
/* sanity check - No SFP+ devices here */
@@ -230,8 +230,6 @@ static int ixgbe_dev_interrupt_action(struct rte_eth_dev *dev);
static void ixgbe_dev_interrupt_handler(void *param);
static void ixgbe_dev_interrupt_delayed_handler(void *param);
static void *ixgbe_dev_setup_link_thread_handler(void *param);
-static int ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev,
- uint32_t timeout_ms);
static int ixgbe_add_rar(struct rte_eth_dev *dev,
struct rte_ether_addr *mac_addr,
@@ -1039,7 +1037,6 @@ ixgbe_swfw_lock_reset(struct ixgbe_hw *hw)
static int
eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
{
- struct ixgbe_adapter *ad = eth_dev->data->dev_private;
struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
@@ -1094,7 +1091,6 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)
return 0;
}
- rte_atomic32_clear(&ad->link_thread_running);
rte_eth_copy_pci_info(eth_dev, pci_dev);
eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
@@ -1547,7 +1543,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
{
int diag;
uint32_t tc, tcs;
- struct ixgbe_adapter *ad = eth_dev->data->dev_private;
struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_hw *hw =
@@ -1590,7 +1585,6 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev)
return 0;
}
- rte_atomic32_clear(&ad->link_thread_running);
ixgbevf_parse_devargs(eth_dev->data->dev_private,
pci_dev->device.devargs);
@@ -2558,8 +2552,11 @@ ixgbe_flow_ctrl_enable(struct rte_eth_dev *dev, struct ixgbe_hw *hw)
static int
ixgbe_dev_start(struct rte_eth_dev *dev)
{
+ struct ixgbe_adapter *ad = dev->data->dev_private;
struct ixgbe_hw *hw =
IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+ struct ixgbe_interrupt *intr =
+ IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
struct ixgbe_vf_info *vfinfo =
*IXGBE_DEV_PRIVATE_TO_P_VFDATA(dev->data->dev_private);
struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
@@ -2580,9 +2577,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
PMD_INIT_FUNC_TRACE();
- /* Stop the link setup handler before resetting the HW. */
- ixgbe_dev_wait_setup_link_complete(dev, 0);
-
/* disable uio/vfio intr/eventfd mapping */
rte_intr_disable(intr_handle);
@@ -2700,8 +2694,16 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
if (ixgbe_is_sfp(hw) && hw->phy.multispeed_fiber) {
err = hw->mac.ops.setup_sfp(hw);
- if (err)
+ intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
+ err = rte_ctrl_thread_create(&ad->link_thread_tid,
+ "ixgbe-service-tid",
+ NULL,
+ ixgbe_dev_setup_link_thread_handler,
+ dev);
+ if (err) {
+ PMD_DRV_LOG(ERR, "service_thread err");
goto error;
+ }
}
if (hw->mac.ops.get_media_type(hw) == ixgbe_media_type_copper) {
@@ -2825,12 +2827,6 @@ ixgbe_dev_start(struct rte_eth_dev *dev)
if (err)
goto error;
- /*
- * Update link status right before return, because it may
- * start link configuration process in a separate thread.
- */
- ixgbe_dev_link_update(dev, 0);
-
/* setup the macsec setting register */
if (macsec_setting->offload_en)
ixgbe_dev_macsec_register_enable(dev, macsec_setting);
@@ -2860,13 +2856,21 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
int vf;
struct ixgbe_tm_conf *tm_conf =
IXGBE_DEV_PRIVATE_TO_TM_CONF(dev->data->dev_private);
+ void *res;
+ s32 err;
if (hw->adapter_stopped)
return 0;
PMD_INIT_FUNC_TRACE();
- ixgbe_dev_wait_setup_link_complete(dev, 0);
+ /* Cancel the service thread */
+ err = pthread_cancel(adapter->link_thread_tid);
+ if (err)
+ PMD_DRV_LOG(ERR, "failed to cancel service thread %d", err);
+ err = pthread_join(adapter->link_thread_tid, &res);
+ if (err)
+ PMD_DRV_LOG(ERR, "failed to join service thread %d", err);
/* disable interrupts */
ixgbe_disable_intr(hw);
@@ -2945,7 +2949,6 @@ ixgbe_dev_set_link_up(struct rte_eth_dev *dev)
} else {
/* Turn on the laser */
ixgbe_enable_tx_laser(hw);
- ixgbe_dev_link_update(dev, 0);
}
return 0;
@@ -2976,7 +2979,6 @@ ixgbe_dev_set_link_down(struct rte_eth_dev *dev)
} else {
/* Turn off the laser */
ixgbe_disable_tx_laser(hw);
- ixgbe_dev_link_update(dev, 0);
}
return 0;
@@ -4129,54 +4131,58 @@ ixgbevf_check_link(struct ixgbe_hw *hw, ixgbe_link_speed *speed,
return ret_val;
}
-/*
- * If @timeout_ms was 0, it means that it will not return until link complete.
- * It returns 1 on complete, return 0 on timeout.
- */
-static int
-ixgbe_dev_wait_setup_link_complete(struct rte_eth_dev *dev, uint32_t timeout_ms)
-{
-#define WARNING_TIMEOUT 9000 /* 9s in total */
- struct ixgbe_adapter *ad = dev->data->dev_private;
- uint32_t timeout = timeout_ms ? timeout_ms : WARNING_TIMEOUT;
-
- while (rte_atomic32_read(&ad->link_thread_running)) {
- msec_delay(1);
- timeout--;
-
- if (timeout_ms) {
- if (!timeout)
- return 0;
- } else if (!timeout) {
- /* It will not return until link complete */
- timeout = WARNING_TIMEOUT;
- PMD_DRV_LOG(ERR, "IXGBE link thread not complete too long time!");
- }
- }
-
- return 1;
-}
-
static void *
ixgbe_dev_setup_link_thread_handler(void *param)
{
struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
- struct ixgbe_adapter *ad = dev->data->dev_private;
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
struct ixgbe_interrupt *intr =
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
- u32 speed;
- bool autoneg = false;
+ u32 speed, start, ticks, service_ms;
+ s32 err;
+ bool link_up, autoneg = false;
pthread_detach(pthread_self());
- speed = hw->phy.autoneg_advertised;
- if (!speed)
- ixgbe_get_link_capabilities(hw, &speed, &autoneg);
- ixgbe_setup_link(hw, speed, true);
+ while (1) {
+ service_ms = 100;
+ if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) {
+ speed = hw->phy.autoneg_advertised;
+
+ if (!speed)
+ ixgbe_get_link_capabilities(hw, &speed, &autoneg);
+
+ err = ixgbe_setup_link(hw, speed, true);
+
+ if (err == IXGBE_SUCCESS)
+ err = ixgbe_check_link(hw, &speed, &link_up, 0);
+
+ /* Run the service thread handler more frequently when link is
+ * down to reduce link up latency (every 200ms vs 1s)
+ *
+ * Use a number of smaller sleeps to decrease exit latency when
+ * ixgbe_dev_stop() wants this thread to join
+ */
+ if (err == IXGBE_SUCCESS && link_up) {
+ service_ms = 2000;
+ intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
+ }
+
+ if (!ixgbe_dev_link_update(dev, 0)) {
+ ixgbe_dev_link_status_print(dev);
+ rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL);
+ }
+ }
+
+ /* Call msec_delay in a loop with several smaller sleeps to
+ * provide periodic thread cancellation points
+ */
+ start = rte_get_timer_cycles();
+ ticks = (uint64_t)service_ms * rte_get_timer_hz() / 1E3;
+ while ((rte_get_timer_cycles() - start) < ticks)
+ msec_delay(100);
+ }
- intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
- rte_atomic32_clear(&ad->link_thread_running);
return NULL;
}
@@ -4219,11 +4225,8 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
int wait_to_complete, int vf)
{
struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
- struct ixgbe_adapter *ad = dev->data->dev_private;
struct rte_eth_link link;
ixgbe_link_speed link_speed = IXGBE_LINK_SPEED_UNKNOWN;
- struct ixgbe_interrupt *intr =
- IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
bool link_up;
int diag;
int wait = 1;
@@ -4238,9 +4241,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
hw->mac.get_link_status = true;
- if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
- return rte_eth_linkstatus_set(dev, &link);
-
/* check if it needs to wait to complete, if lsc interrupt is enabled */
if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc != 0)
wait = 0;
@@ -4255,7 +4255,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
else
diag = ixgbe_check_link(hw, &link_speed, &link_up, wait);
- if (diag != 0) {
+ if (diag != 0 || !link_up) {
link.link_speed = RTE_ETH_SPEED_NUM_100M;
link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
return rte_eth_linkstatus_set(dev, &link);
@@ -4267,32 +4267,6 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
link_up = 0;
}
- if (link_up == 0) {
- if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
- ixgbe_dev_wait_setup_link_complete(dev, 0);
- if (rte_atomic32_test_and_set(&ad->link_thread_running)) {
- /* To avoid race condition between threads, set
- * the IXGBE_FLAG_NEED_LINK_CONFIG flag only
- * when there is no link thread running.
- */
- intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
- if (rte_ctrl_thread_create(&ad->link_thread_tid,
- "ixgbe-link-handler",
- NULL,
- ixgbe_dev_setup_link_thread_handler,
- dev) < 0) {
- PMD_DRV_LOG(ERR,
- "Create link thread failed!");
- rte_atomic32_clear(&ad->link_thread_running);
- }
- } else {
- PMD_DRV_LOG(ERR,
- "Other link thread is running now!");
- }
- }
- return rte_eth_linkstatus_set(dev, &link);
- }
-
link.link_status = RTE_ETH_LINK_UP;
link.link_duplex = RTE_ETH_LINK_FULL_DUPLEX;
@@ -4566,6 +4540,8 @@ ixgbe_dev_link_status_print(struct rte_eth_dev *dev)
static int
ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
{
+ struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+ struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_interrupt *intr =
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
int64_t timeout;
@@ -4605,16 +4581,14 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
if (rte_eal_alarm_set(timeout * 1000,
ixgbe_dev_interrupt_delayed_handler, (void *)dev) < 0)
PMD_DRV_LOG(ERR, "Error setting alarm");
- else {
- /* remember original mask */
- intr->mask_original = intr->mask;
+ else
/* only disable lsc interrupt */
intr->mask &= ~IXGBE_EIMS_LSC;
- }
}
PMD_DRV_LOG(DEBUG, "enable intr immediately");
ixgbe_enable_intr(dev);
+ rte_intr_ack(intr_handle);
return 0;
}
@@ -4637,8 +4611,6 @@ static void
ixgbe_dev_interrupt_delayed_handler(void *param)
{
struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
- struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
- struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
struct ixgbe_interrupt *intr =
IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
struct ixgbe_hw *hw =
@@ -4668,13 +4640,10 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
intr->flags &= ~IXGBE_FLAG_MACSEC;
}
- /* restore original mask */
- intr->mask = intr->mask_original;
- intr->mask_original = 0;
+ if (dev->data->dev_conf.intr_conf.lsc != 0)
+ intr->mask |= IXGBE_EICR_LSC;
- PMD_DRV_LOG(DEBUG, "enable intr in delayed handler S[%08x]", eicr);
ixgbe_enable_intr(dev);
- rte_intr_ack(intr_handle);
}
/**
@@ -5316,9 +5285,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
PMD_INIT_FUNC_TRACE();
- /* Stop the link setup handler before resetting the HW. */
- ixgbe_dev_wait_setup_link_complete(dev, 0);
-
err = hw->mac.ops.reset_hw(hw);
/**
@@ -5398,12 +5364,6 @@ ixgbevf_dev_start(struct rte_eth_dev *dev)
/* Re-enable interrupt for VF */
ixgbevf_intr_enable(dev);
- /*
- * Update link status right before return, because it may
- * start link configuration process in a separate thread.
- */
- ixgbevf_dev_link_update(dev, 0);
-
hw->adapter_stopped = false;
return 0;
@@ -5422,8 +5382,6 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
PMD_INIT_FUNC_TRACE();
- ixgbe_dev_wait_setup_link_complete(dev, 0);
-
ixgbevf_intr_disable(dev);
dev->data->dev_started = 0;