[dpdk-dev,v3,1/3] fm10k: enable FTAG based forwarding

  This patch enables reading sglort info into mbuf for RX and inserting
an FTAG at the beginning of the packet for TX. The vlan_tci_outer field
selected from rte_mbuf structure for sglort is not used in fm10k now.
In FTAG based forwarding mode, the switch will forward packets according
to glort info in FTAG rather than mac and vlan table.

To activate this feature, user needs to turn ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
to y in common_linuxapp or common_bsdapp. Currently this feature is supported
only on PF, because FM10K_PFVTCTL register is read-only for VF.

Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
---
 config/common_bsdapp               |  1 +
 config/common_linuxapp             |  1 +
 drivers/net/fm10k/fm10k_ethdev.c   | 12 ++++++++++++
 drivers/net/fm10k/fm10k_rxtx.c     | 17 +++++++++++++++++
 drivers/net/fm10k/fm10k_rxtx_vec.c |  9 +++++++++
 5 files changed, 40 insertions(+)
  

On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> This patch enables reading sglort info into mbuf for RX and inserting
> an FTAG at the beginning of the packet for TX. The vlan_tci_outer field
> selected from rte_mbuf structure for sglort is not used in fm10k now.
> In FTAG based forwarding mode, the switch will forward packets according
> to glort info in FTAG rather than mac and vlan table.
> 
> To activate this feature, user needs to turn ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> to y in common_linuxapp or common_bsdapp. Currently this feature is supported
> only on PF, because FM10K_PFVTCTL register is read-only for VF.
> 
> Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>

Any comments on this patch?

My thoughts: is there a way in which this could be done without adding in a new
build time config option?

/Bruce

> ---
>  config/common_bsdapp               |  1 +
>  config/common_linuxapp             |  1 +
>  drivers/net/fm10k/fm10k_ethdev.c   | 12 ++++++++++++
>  drivers/net/fm10k/fm10k_rxtx.c     | 17 +++++++++++++++++
>  drivers/net/fm10k/fm10k_rxtx_vec.c |  9 +++++++++
>  5 files changed, 40 insertions(+)
> 
> diff --git a/config/common_bsdapp b/config/common_bsdapp
> index ed7c31c..451f81a 100644
> --- a/config/common_bsdapp
> +++ b/config/common_bsdapp
> @@ -208,6 +208,7 @@ CONFIG_RTE_LIBRTE_FM10K_DEBUG_TX=n
>  CONFIG_RTE_LIBRTE_FM10K_DEBUG_TX_FREE=n
>  CONFIG_RTE_LIBRTE_FM10K_DEBUG_DRIVER=n
>  CONFIG_RTE_LIBRTE_FM10K_RX_OLFLAGS_ENABLE=y
> +CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD=n
>  
>  #
>  # Compile burst-oriented Mellanox ConnectX-3 (MLX4) PMD
> diff --git a/config/common_linuxapp b/config/common_linuxapp
> index 74bc515..c928bce 100644
> --- a/config/common_linuxapp
> +++ b/config/common_linuxapp
> @@ -207,6 +207,7 @@ CONFIG_RTE_LIBRTE_FM10K_DEBUG_TX_FREE=n
>  CONFIG_RTE_LIBRTE_FM10K_DEBUG_DRIVER=n
>  CONFIG_RTE_LIBRTE_FM10K_RX_OLFLAGS_ENABLE=y
>  CONFIG_RTE_LIBRTE_FM10K_INC_VECTOR=y
> +CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD=n
>  
>  #
>  # Compile burst-oriented Mellanox ConnectX-3 (MLX4) PMD
> diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
> index e4aed94..65d355e 100644
> --- a/drivers/net/fm10k/fm10k_ethdev.c
> +++ b/drivers/net/fm10k/fm10k_ethdev.c
> @@ -668,6 +668,18 @@ fm10k_dev_tx_init(struct rte_eth_dev *dev)
>  			PMD_INIT_LOG(ERR, "failed to disable queue %d", i);
>  			return -1;
>  		}
> +#ifdef RTE_LIBRTE_FM10K_FTAG_FWD
> +		/* Enable use of FTAG bit in TX descriptor, PFVTCTL
> +		 * register is read-only for VF.
> +		 */
> +		if (hw->mac.type == fm10k_mac_pf)
> +			FM10K_WRITE_REG(hw, FM10K_PFVTCTL(i),
> +					FM10K_PFVTCTL_FTAG_DESC_ENABLE);
> +		else {
> +			PMD_INIT_LOG(ERR, "FTAG is not supported in VF.");
> +			return -ENOTSUP;
> +		}
> +#endif
>  
>  		/* set location and size for descriptor ring */
>  		FM10K_WRITE_REG(hw, FM10K_TDBAL(i),
> diff --git a/drivers/net/fm10k/fm10k_rxtx.c b/drivers/net/fm10k/fm10k_rxtx.c
> index e958865..f87987d 100644
> --- a/drivers/net/fm10k/fm10k_rxtx.c
> +++ b/drivers/net/fm10k/fm10k_rxtx.c
> @@ -152,6 +152,13 @@ fm10k_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
>  		 */
>  		mbuf->ol_flags |= PKT_RX_VLAN_PKT;
>  		mbuf->vlan_tci = desc.w.vlan;
> +#ifdef RTE_LIBRTE_FM10K_FTAG_FWD
> +		/**
> +		 * mbuf->vlan_tci_outer is an idle field in fm10k driver,
> +		 * so it can be selected to store sglort value.
> +		 */
> +		mbuf->vlan_tci_outer = rte_le_to_cpu_16(desc.w.sglort);
> +#endif
>  
>  		rx_pkts[count] = mbuf;
>  		if (++next_dd == q->nb_desc) {
> @@ -307,6 +314,13 @@ fm10k_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
>  		 */
>  		mbuf->ol_flags |= PKT_RX_VLAN_PKT;
>  		first_seg->vlan_tci = desc.w.vlan;
> +#ifdef RTE_LIBRTE_FM10K_FTAG_FWD
> +		/**
> +		 * mbuf->vlan_tci_outer is an idle field in fm10k driver,
> +		 * so it can be selected to store sglort value.
> +		 */
> +		first_seg->vlan_tci_outer = rte_le_to_cpu_16(desc.w.sglort);
> +#endif
>  
>  		/* Prefetch data of first segment, if configured to do so. */
>  		rte_packet_prefetch((char *)first_seg->buf_addr +
> @@ -432,6 +446,9 @@ static inline void tx_xmit_pkt(struct fm10k_tx_queue *q, struct rte_mbuf *mb)
>  	q->nb_free -= mb->nb_segs;
>  
>  	q->hw_ring[q->next_free].flags = 0;
> +#ifdef RTE_LIBRTE_FM10K_FTAG_FWD
> +	q->hw_ring[q->next_free].flags |= FM10K_TXD_FLAG_FTAG;
> +#endif
>  	/* set checksum flags on first descriptor of packet. SCTP checksum
>  	 * offload is not supported, but we do not explicitly check for this
>  	 * case in favor of greatly simplified processing. */
> diff --git a/drivers/net/fm10k/fm10k_rxtx_vec.c b/drivers/net/fm10k/fm10k_rxtx_vec.c
> index 2a57eef..0b0f2e3 100644
> --- a/drivers/net/fm10k/fm10k_rxtx_vec.c
> +++ b/drivers/net/fm10k/fm10k_rxtx_vec.c
> @@ -198,7 +198,12 @@ fm10k_rx_vec_condition_check(struct rte_eth_dev *dev)
>  	    rxmode->header_split == 1)
>  		return -1;
>  
> +#ifdef RTE_LIBRTE_FM10K_FTAG_FWD
> +	return -1;
> +#else
>  	return 0;
> +#endif
> +
>  #else
>  	RTE_SET_USED(dev);
>  	return -1;
> @@ -648,7 +653,11 @@ fm10k_tx_vec_condition_check(struct fm10k_tx_queue *txq)
>  	if ((txq->txq_flags & FM10K_SIMPLE_TX_FLAG) != FM10K_SIMPLE_TX_FLAG)
>  		return -1;
>  
> +#ifdef RTE_LIBRTE_FM10K_FTAG_FWD
> +	return -1;
> +#else
>  	return 0;
> +#endif
>  }
>  
>  static inline void
> -- 
> 1.9.3
>
  

2016-02-24 15:42, Bruce Richardson:
> On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> > This patch enables reading sglort info into mbuf for RX and inserting
> > an FTAG at the beginning of the packet for TX. The vlan_tci_outer field
> > selected from rte_mbuf structure for sglort is not used in fm10k now.
> > In FTAG based forwarding mode, the switch will forward packets according
> > to glort info in FTAG rather than mac and vlan table.
> > 
> > To activate this feature, user needs to turn ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> > to y in common_linuxapp or common_bsdapp. Currently this feature is supported
> > only on PF, because FM10K_PFVTCTL register is read-only for VF.
> > 
> > Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
> 
> Any comments on this patch?
> 
> My thoughts: is there a way in which this could be done without adding in a new
> build time config option?

Bruce, it's simpler to explain that build time options are forbidden to
enable such options.
Or the terrific kid's approach: one day, the Big Build-Option Eater will come
and will eat every undecided features! ;)
  

On Wed, Feb 24, 2016 at 05:37:45PM +0100, Thomas Monjalon wrote:
> 2016-02-24 15:42, Bruce Richardson:
> > On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> > > This patch enables reading sglort info into mbuf for RX and inserting
> > > an FTAG at the beginning of the packet for TX. The vlan_tci_outer field
> > > selected from rte_mbuf structure for sglort is not used in fm10k now.
> > > In FTAG based forwarding mode, the switch will forward packets according
> > > to glort info in FTAG rather than mac and vlan table.
> > > 
> > > To activate this feature, user needs to turn ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> > > to y in common_linuxapp or common_bsdapp. Currently this feature is supported
> > > only on PF, because FM10K_PFVTCTL register is read-only for VF.
> > > 
> > > Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
> > 
> > Any comments on this patch?
> > 
> > My thoughts: is there a way in which this could be done without adding in a new
> > build time config option?
> 
> Bruce, it's simpler to explain that build time options are forbidden to
> enable such options.
> Or the terrific kid's approach: one day, the Big Build-Option Eater will come
> and will eat every undecided features! ;)

Good-cop, bad-cop, guess who I'm playing? :-)

/Bruce
  

Hi, Bruce, Thomas,

Best Regards,
Mark

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> Sent: Thursday, February 25, 2016 12:38 AM
> To: Richardson, Bruce; Wang, Xiao W
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> forwarding
> 
> 2016-02-24 15:42, Bruce Richardson:
> > On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> > > This patch enables reading sglort info into mbuf for RX and inserting
> > > an FTAG at the beginning of the packet for TX. The vlan_tci_outer field
> > > selected from rte_mbuf structure for sglort is not used in fm10k now.
> > > In FTAG based forwarding mode, the switch will forward packets
> according
> > > to glort info in FTAG rather than mac and vlan table.
> > >
> > > To activate this feature, user needs to turn
> ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> > > to y in common_linuxapp or common_bsdapp. Currently this feature is
> supported
> > > only on PF, because FM10K_PFVTCTL register is read-only for VF.
> > >
> > > Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
> >
> > Any comments on this patch?
> >
> > My thoughts: is there a way in which this could be done without adding in a
> new
> > build time config option?
> 
> Bruce, it's simpler to explain that build time options are forbidden to
> enable such options.
> Or the terrific kid's approach: one day, the Big Build-Option Eater will come
> and will eat every undecided features! ;)

This feature is trying to use FTAG (a unique tech in fm10k) instead of mac/vlan
to forward packets. App need a way to tell PMD driver that which forwarding
style it would like to use. 
So, the best option is to let packets carry a flag in mbuf to tell drive in fast path. 
You can see that this is unique to fm10k and we thought community won't like to see 
this flag introduced into mbuf. If you do agree, we can introduce a new flag.
So, we step backwards and assume customer will use static configurations to enable
this feature. After it is enabled, we'll assume app will use FTAG for all packets.
  

On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> Hi, Bruce, Thomas,
> 
> Best Regards,
> Mark
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> > Sent: Thursday, February 25, 2016 12:38 AM
> > To: Richardson, Bruce; Wang, Xiao W
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> > forwarding
> > 
> > 2016-02-24 15:42, Bruce Richardson:
> > > On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> > > > This patch enables reading sglort info into mbuf for RX and inserting
> > > > an FTAG at the beginning of the packet for TX. The vlan_tci_outer field
> > > > selected from rte_mbuf structure for sglort is not used in fm10k now.
> > > > In FTAG based forwarding mode, the switch will forward packets
> > according
> > > > to glort info in FTAG rather than mac and vlan table.
> > > >
> > > > To activate this feature, user needs to turn
> > ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> > > > to y in common_linuxapp or common_bsdapp. Currently this feature is
> > supported
> > > > only on PF, because FM10K_PFVTCTL register is read-only for VF.
> > > >
> > > > Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
> > >
> > > Any comments on this patch?
> > >
> > > My thoughts: is there a way in which this could be done without adding in a
> > new
> > > build time config option?
> > 
> > Bruce, it's simpler to explain that build time options are forbidden to
> > enable such options.
> > Or the terrific kid's approach: one day, the Big Build-Option Eater will come
> > and will eat every undecided features! ;)
> 
> This feature is trying to use FTAG (a unique tech in fm10k) instead of mac/vlan
> to forward packets. App need a way to tell PMD driver that which forwarding
> style it would like to use. 

Why not just specify this in the port configuration at setup time?

> So, the best option is to let packets carry a flag in mbuf to tell drive in fast path. 
> You can see that this is unique to fm10k and we thought community won't like to see 
> this flag introduced into mbuf. If you do agree, we can introduce a new flag.

Why does it need to be specified per-mbuf? The existing config flag added is
global, so a per-mbuf flag shouldn't be needed to get equivalent behaviour.

> So, we step backwards and assume customer will use static configurations to enable
> this feature. After it is enabled, we'll assume app will use FTAG for all packets.

Yes, but instead of compile time option, why not port config-time option instead?

/Bruce
  

Hi, Bruce,

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Thursday, February 25, 2016 9:35 PM
> To: Chen, Jing D <jing.d.chen@intel.com>
> Cc: Thomas Monjalon <thomas.monjalon@6wind.com>; Wang, Xiao W
> <xiao.w.wang@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> forwarding
> 
> On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> > Hi, Bruce, Thomas,
> >
> > Best Regards,
> > Mark
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas
> Monjalon
> > > Sent: Thursday, February 25, 2016 12:38 AM
> > > To: Richardson, Bruce; Wang, Xiao W
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> > > forwarding
> > >
> > > 2016-02-24 15:42, Bruce Richardson:
> > > > On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> > > > > This patch enables reading sglort info into mbuf for RX and
> > > > > inserting an FTAG at the beginning of the packet for TX. The
> > > > > vlan_tci_outer field selected from rte_mbuf structure for sglort is not
> used in fm10k now.
> > > > > In FTAG based forwarding mode, the switch will forward packets
> > > according
> > > > > to glort info in FTAG rather than mac and vlan table.
> > > > >
> > > > > To activate this feature, user needs to turn
> > > ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> > > > > to y in common_linuxapp or common_bsdapp. Currently this feature
> > > > > is
> > > supported
> > > > > only on PF, because FM10K_PFVTCTL register is read-only for VF.
> > > > >
> > > > > Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
> > > >
> > > > Any comments on this patch?
> > > >
> > > > My thoughts: is there a way in which this could be done without
> > > > adding in a
> > > new
> > > > build time config option?
> > >
> > > Bruce, it's simpler to explain that build time options are forbidden
> > > to enable such options.
> > > Or the terrific kid's approach: one day, the Big Build-Option Eater
> > > will come and will eat every undecided features! ;)
> >
> > This feature is trying to use FTAG (a unique tech in fm10k) instead of
> > mac/vlan to forward packets. App need a way to tell PMD driver that
> > which forwarding style it would like to use.
> 
> Why not just specify this in the port configuration at setup time?
> 

Please educate me. I think the port configuration flags are also common to all PMD
Drivers. Is it possible to add a flag like "RTE_USE_FTAG" and pass to PMD driver?

> > So, the best option is to let packets carry a flag in mbuf to tell drive in fast
> path.
> > You can see that this is unique to fm10k and we thought community
> > won't like to see this flag introduced into mbuf. If you do agree, we can
> introduce a new flag.
> 
> Why does it need to be specified per-mbuf? The existing config flag added is
> global, so a per-mbuf flag shouldn't be needed to get equivalent behaviour.
> 
> > So, we step backwards and assume customer will use static
> > configurations to enable this feature. After it is enabled, we'll assume app
> will use FTAG for all packets.
> 
> Yes, but instead of compile time option, why not port config-time option
> instead?
> 
> /Bruce
  

On Thu, Feb 25, 2016 at 03:45:45PM +0000, Chen, Jing D wrote:
> Hi, Bruce,
> 
> > -----Original Message-----
> > From: Richardson, Bruce
> > Sent: Thursday, February 25, 2016 9:35 PM
> > To: Chen, Jing D <jing.d.chen@intel.com>
> > Cc: Thomas Monjalon <thomas.monjalon@6wind.com>; Wang, Xiao W
> > <xiao.w.wang@intel.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> > forwarding
> > 
> > On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> > > Hi, Bruce, Thomas,
> > >
> > > Best Regards,
> > > Mark
> > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas
> > Monjalon
> > > > Sent: Thursday, February 25, 2016 12:38 AM
> > > > To: Richardson, Bruce; Wang, Xiao W
> > > > Cc: dev@dpdk.org
> > > > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> > > > forwarding
> > > >
> > > > 2016-02-24 15:42, Bruce Richardson:
> > > > > On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> > > > > > This patch enables reading sglort info into mbuf for RX and
> > > > > > inserting an FTAG at the beginning of the packet for TX. The
> > > > > > vlan_tci_outer field selected from rte_mbuf structure for sglort is not
> > used in fm10k now.
> > > > > > In FTAG based forwarding mode, the switch will forward packets
> > > > according
> > > > > > to glort info in FTAG rather than mac and vlan table.
> > > > > >
> > > > > > To activate this feature, user needs to turn
> > > > ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> > > > > > to y in common_linuxapp or common_bsdapp. Currently this feature
> > > > > > is
> > > > supported
> > > > > > only on PF, because FM10K_PFVTCTL register is read-only for VF.
> > > > > >
> > > > > > Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
> > > > >
> > > > > Any comments on this patch?
> > > > >
> > > > > My thoughts: is there a way in which this could be done without
> > > > > adding in a
> > > > new
> > > > > build time config option?
> > > >
> > > > Bruce, it's simpler to explain that build time options are forbidden
> > > > to enable such options.
> > > > Or the terrific kid's approach: one day, the Big Build-Option Eater
> > > > will come and will eat every undecided features! ;)
> > >
> > > This feature is trying to use FTAG (a unique tech in fm10k) instead of
> > > mac/vlan to forward packets. App need a way to tell PMD driver that
> > > which forwarding style it would like to use.
> > 
> > Why not just specify this in the port configuration at setup time?
> > 
> 
> Please educate me. I think the port configuration flags are also common to all PMD
> Drivers. Is it possible to add a flag like "RTE_USE_FTAG" and pass to PMD driver?
> 
They are. 
For something PMD specific, like FTAG, it's always a challenge, and I don't know
off the top of my head if there is a simple option. However, given the choice
between an mbuf flag and a port config flag, I'd always choose the former.
Other alternatives would be to have a fm10k specific API in the fm10k driver alone.

I'll let Thomas as ethdev maintainer comment if he has other suggestions as to
how to handle this case. I suspect this won't be the first device-specific 
piece of functionality we need to deal with.

/Bruce
  

Hi,

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Friday, February 26, 2016 12:14 AM
> To: Chen, Jing D <jing.d.chen@intel.com>
> Cc: Thomas Monjalon <thomas.monjalon@6wind.com>; Wang, Xiao W
> <xiao.w.wang@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based forwarding
> 
> On Thu, Feb 25, 2016 at 03:45:45PM +0000, Chen, Jing D wrote:
> > Hi, Bruce,
> >
> > > -----Original Message-----
> > > From: Richardson, Bruce
> > > Sent: Thursday, February 25, 2016 9:35 PM
> > > To: Chen, Jing D <jing.d.chen@intel.com>
> > > Cc: Thomas Monjalon <thomas.monjalon@6wind.com>; Wang, Xiao W
> > > <xiao.w.wang@intel.com>; dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> > > forwarding
> > >
> > > On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> > > > Hi, Bruce, Thomas,
> > > >
> > > > Best Regards,
> > > > Mark
> > > >
> > > > > -----Original Message-----
> > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas
> > > Monjalon
> > > > > Sent: Thursday, February 25, 2016 12:38 AM
> > > > > To: Richardson, Bruce; Wang, Xiao W
> > > > > Cc: dev@dpdk.org
> > > > > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based
> > > > > forwarding
> > > > >
> > > > > 2016-02-24 15:42, Bruce Richardson:
> > > > > > On Thu, Feb 04, 2016 at 11:38:47AM +0800, Wang Xiao W wrote:
> > > > > > > This patch enables reading sglort info into mbuf for RX and
> > > > > > > inserting an FTAG at the beginning of the packet for TX. The
> > > > > > > vlan_tci_outer field selected from rte_mbuf structure for
> > > > > > > sglort is not
> > > used in fm10k now.
> > > > > > > In FTAG based forwarding mode, the switch will forward
> > > > > > > packets
> > > > > according
> > > > > > > to glort info in FTAG rather than mac and vlan table.
> > > > > > >
> > > > > > > To activate this feature, user needs to turn
> > > > > ``CONFIG_RTE_LIBRTE_FM10K_FTAG_FWD``
> > > > > > > to y in common_linuxapp or common_bsdapp. Currently this
> > > > > > > feature is
> > > > > supported
> > > > > > > only on PF, because FM10K_PFVTCTL register is read-only for VF.
> > > > > > >
> > > > > > > Signed-off-by: Wang Xiao W <xiao.w.wang@intel.com>
> > > > > >
> > > > > > Any comments on this patch?
> > > > > >
> > > > > > My thoughts: is there a way in which this could be done
> > > > > > without adding in a
> > > > > new
> > > > > > build time config option?
> > > > >
> > > > > Bruce, it's simpler to explain that build time options are
> > > > > forbidden to enable such options.
> > > > > Or the terrific kid's approach: one day, the Big Build-Option
> > > > > Eater will come and will eat every undecided features! ;)
> > > >
> > > > This feature is trying to use FTAG (a unique tech in fm10k)
> > > > instead of mac/vlan to forward packets. App need a way to tell PMD
> > > > driver that which forwarding style it would like to use.
> > >
> > > Why not just specify this in the port configuration at setup time?
> > >
> >
> > Please educate me. I think the port configuration flags are also
> > common to all PMD Drivers. Is it possible to add a flag like "RTE_USE_FTAG"
> and pass to PMD driver?
> >
> They are.
> For something PMD specific, like FTAG, it's always a challenge, and I don't
> know off the top of my head if there is a simple option. However, given the
> choice between an mbuf flag and a port config flag, I'd always choose the
> former.
> Other alternatives would be to have a fm10k specific API in the fm10k driver
> alone.
> 
> I'll let Thomas as ethdev maintainer comment if he has other suggestions as to
> how to handle this case. I suspect this won't be the first device-specific piece of
> functionality we need to deal with.
> 
> /Bruce

Whatever method we choose, we have to find a way for the user to express his need
for FTAG, it maybe a build time config option, or a port config flag (no such flag now),
or a fast path flag in mbuf (no such flag now) etc. For the customer Topsec's use case,
they use FTAG for all the TX packets, so all the above methods (per build config, per
port config, per mbuf config) can meet their need. Since the pmd frame work is for
common, it's hard to add new fields only for one specific NIC, so I add a build time
config and make an introduction in the doc.

Thanks for the discussion, Thomas, do you have any suggestions?

BTW, I have found the patch needs to be rebased on head of dpdk-next-net/rel_16_04 since several
fm10k patches were applied recently.

Best Regards,
Xiao
  

2016-02-26 04:31, Wang, Xiao W:
> From: Richardson, Bruce
> > On Thu, Feb 25, 2016 at 03:45:45PM +0000, Chen, Jing D wrote:
> > > From: Richardson, Bruce
> > > > On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> > > > > This feature is trying to use FTAG (a unique tech in fm10k)
> > > > > instead of mac/vlan to forward packets. App need a way to tell PMD
> > > > > driver that which forwarding style it would like to use.
> > > >
> > > > Why not just specify this in the port configuration at setup time?
> > > >
> > >
> > > Please educate me. I think the port configuration flags are also
> > > common to all PMD Drivers. Is it possible to add a flag like "RTE_USE_FTAG"
> > and pass to PMD driver?
> > >
> > They are.
> > For something PMD specific, like FTAG, it's always a challenge, and I don't
> > know off the top of my head if there is a simple option. However, given the
> > choice between an mbuf flag and a port config flag, I'd always choose the
> > former.
> > Other alternatives would be to have a fm10k specific API in the fm10k driver
> > alone.
> > 
> > I'll let Thomas as ethdev maintainer comment if he has other suggestions as to
> > how to handle this case. I suspect this won't be the first device-specific piece of
> > functionality we need to deal with.
> > 
> > /Bruce
> 
> Whatever method we choose, we have to find a way for the user to express his need
> for FTAG, it maybe a build time config option, or a port config flag (no such flag now),
> or a fast path flag in mbuf (no such flag now) etc. For the customer Topsec's use case,
> they use FTAG for all the TX packets, so all the above methods (per build config, per
> port config, per mbuf config) can meet their need. Since the pmd frame work is for
> common, it's hard to add new fields only for one specific NIC, so I add a build time
> config and make an introduction in the doc.
> 
> Thanks for the discussion, Thomas, do you have any suggestions?

I don't understand why you say this feature is specific to fm10k. Can we
imagine another NIC having this capability?
I think it must be an port configuration, as Bruce suggested.
What about a field in struct rte_eth_conf?
  

Hi,

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Friday, February 26, 2016 5:06 PM
> To: Wang, Xiao W <xiao.w.wang@intel.com>
> Cc: Richardson, Bruce <bruce.richardson@intel.com>; Chen, Jing D
> <jing.d.chen@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based forwarding
> 
> 2016-02-26 04:31, Wang, Xiao W:
> > From: Richardson, Bruce
> > > On Thu, Feb 25, 2016 at 03:45:45PM +0000, Chen, Jing D wrote:
> > > > From: Richardson, Bruce
> > > > > On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> > > > > > This feature is trying to use FTAG (a unique tech in fm10k)
> > > > > > instead of mac/vlan to forward packets. App need a way to tell
> > > > > > PMD driver that which forwarding style it would like to use.
> > > > >
> > > > > Why not just specify this in the port configuration at setup time?
> > > > >
> > > >
> > > > Please educate me. I think the port configuration flags are also
> > > > common to all PMD Drivers. Is it possible to add a flag like
> "RTE_USE_FTAG"
> > > and pass to PMD driver?
> > > >
> > > They are.
> > > For something PMD specific, like FTAG, it's always a challenge, and
> > > I don't know off the top of my head if there is a simple option.
> > > However, given the choice between an mbuf flag and a port config
> > > flag, I'd always choose the former.
> > > Other alternatives would be to have a fm10k specific API in the
> > > fm10k driver alone.
> > >
> > > I'll let Thomas as ethdev maintainer comment if he has other
> > > suggestions as to how to handle this case. I suspect this won't be
> > > the first device-specific piece of functionality we need to deal with.
> > >
> > > /Bruce
> >
> > Whatever method we choose, we have to find a way for the user to
> > express his need for FTAG, it maybe a build time config option, or a
> > port config flag (no such flag now), or a fast path flag in mbuf (no
> > such flag now) etc. For the customer Topsec's use case, they use FTAG
> > for all the TX packets, so all the above methods (per build config,
> > per port config, per mbuf config) can meet their need. Since the pmd
> > frame work is for common, it's hard to add new fields only for one specific
> NIC, so I add a build time config and make an introduction in the doc.
> >
> > Thanks for the discussion, Thomas, do you have any suggestions?
> 
> I don't understand why you say this feature is specific to fm10k. Can we
> imagine another NIC having this capability?

As you know, fm10k has a switch logic between the Mac and Phy, every packets
Sent out from the host will be switched inside the NIC, other NICs don't have
a switch inside, and the FTAG feature is related to the switch function.

As introduced in the second patch:
The FM10K family of NICs support the addition of a Fabric Tag (FTAG) to carry
special information. The FTAG is placed at the beginning of the frame, it contains
information such as where the packet comes from and goes, and the vlan tag. In
FTAG based forwarding mode, the switch logic forwards packets according to
glort (global resource tag) information, rather than the mac and vlan table.
So this is a feature specific to fm10k.

Best Regards,
Xiao
> I think it must be an port configuration, as Bruce suggested.
> What about a field in struct rte_eth_conf?
  

On Fri, Feb 26, 2016 at 09:24:06AM +0000, Wang, Xiao W wrote:
> Hi,
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Sent: Friday, February 26, 2016 5:06 PM
> > To: Wang, Xiao W <xiao.w.wang@intel.com>
> > Cc: Richardson, Bruce <bruce.richardson@intel.com>; Chen, Jing D
> > <jing.d.chen@intel.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based forwarding
> > 
> > 2016-02-26 04:31, Wang, Xiao W:
> > > From: Richardson, Bruce
> > > > On Thu, Feb 25, 2016 at 03:45:45PM +0000, Chen, Jing D wrote:
> > > > > From: Richardson, Bruce
> > > > > > On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> > > > > > > This feature is trying to use FTAG (a unique tech in fm10k)
> > > > > > > instead of mac/vlan to forward packets. App need a way to tell
> > > > > > > PMD driver that which forwarding style it would like to use.
> > > > > >
> > > > > > Why not just specify this in the port configuration at setup time?
> > > > > >
> > > > >
> > > > > Please educate me. I think the port configuration flags are also
> > > > > common to all PMD Drivers. Is it possible to add a flag like
> > "RTE_USE_FTAG"
> > > > and pass to PMD driver?
> > > > >
> > > > They are.
> > > > For something PMD specific, like FTAG, it's always a challenge, and
> > > > I don't know off the top of my head if there is a simple option.
> > > > However, given the choice between an mbuf flag and a port config
> > > > flag, I'd always choose the former.
> > > > Other alternatives would be to have a fm10k specific API in the
> > > > fm10k driver alone.
> > > >
> > > > I'll let Thomas as ethdev maintainer comment if he has other
> > > > suggestions as to how to handle this case. I suspect this won't be
> > > > the first device-specific piece of functionality we need to deal with.
> > > >
> > > > /Bruce
> > >
> > > Whatever method we choose, we have to find a way for the user to
> > > express his need for FTAG, it maybe a build time config option, or a
> > > port config flag (no such flag now), or a fast path flag in mbuf (no
> > > such flag now) etc. For the customer Topsec's use case, they use FTAG
> > > for all the TX packets, so all the above methods (per build config,
> > > per port config, per mbuf config) can meet their need. Since the pmd
> > > frame work is for common, it's hard to add new fields only for one specific
> > NIC, so I add a build time config and make an introduction in the doc.
> > >
> > > Thanks for the discussion, Thomas, do you have any suggestions?
> > 
> > I don't understand why you say this feature is specific to fm10k. Can we
> > imagine another NIC having this capability?
> 
> As you know, fm10k has a switch logic between the Mac and Phy, every packets
> Sent out from the host will be switched inside the NIC, other NICs don't have
> a switch inside, and the FTAG feature is related to the switch function.
> 
> As introduced in the second patch:
> The FM10K family of NICs support the addition of a Fabric Tag (FTAG) to carry
> special information. The FTAG is placed at the beginning of the frame, it contains
> information such as where the packet comes from and goes, and the vlan tag. In
> FTAG based forwarding mode, the switch logic forwards packets according to
> glort (global resource tag) information, rather than the mac and vlan table.
> So this is a feature specific to fm10k.
> 
> Best Regards,
> Xiao

If it is fm10k specific, how about just adding a public function to the fm10k
driver to turn it on. The user app will be non-portable across NICs, but
that's the price of using nic-specific features.

/Bruce

> > I think it must be an port configuration, as Bruce suggested.
> > What about a field in struct rte_eth_conf?
  

On Fri, Feb 26, 2016 at 3:48 PM, Bruce Richardson
<bruce.richardson@intel.com> wrote:
> On Fri, Feb 26, 2016 at 09:24:06AM +0000, Wang, Xiao W wrote:
>> Hi,
>> > > Thanks for the discussion, Thomas, do you have any suggestions?
>> >
>> > I don't understand why you say this feature is specific to fm10k. Can we
>> > imagine another NIC having this capability?
>>
>> As you know, fm10k has a switch logic between the Mac and Phy, every packets
>> Sent out from the host will be switched inside the NIC, other NICs don't have
>> a switch inside, and the FTAG feature is related to the switch function.
>>
>> As introduced in the second patch:
>> The FM10K family of NICs support the addition of a Fabric Tag (FTAG) to carry
>> special information. The FTAG is placed at the beginning of the frame, it contains
>> information such as where the packet comes from and goes, and the vlan tag. In
>> FTAG based forwarding mode, the switch logic forwards packets according to
>> glort (global resource tag) information, rather than the mac and vlan table.
>> So this is a feature specific to fm10k.
>
> If it is fm10k specific, how about just adding a public function to the fm10k
> driver to turn it on. The user app will be non-portable across NICs, but
> that's the price of using nic-specific features.

What about using a devargs ?
Something like :
-w xxxx:xx:xx.x,enable_ftag=1

The application still needs to know about this to enable it, but that
sounds better to me.
The only issue is that it can't work with hotplug at the moment.
  

On Fri, Feb 26, 2016 at 04:00:49PM +0100, David Marchand wrote:
> On Fri, Feb 26, 2016 at 3:48 PM, Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> > On Fri, Feb 26, 2016 at 09:24:06AM +0000, Wang, Xiao W wrote:
> >> Hi,
> >> > > Thanks for the discussion, Thomas, do you have any suggestions?
> >> >
> >> > I don't understand why you say this feature is specific to fm10k. Can we
> >> > imagine another NIC having this capability?
> >>
> >> As you know, fm10k has a switch logic between the Mac and Phy, every packets
> >> Sent out from the host will be switched inside the NIC, other NICs don't have
> >> a switch inside, and the FTAG feature is related to the switch function.
> >>
> >> As introduced in the second patch:
> >> The FM10K family of NICs support the addition of a Fabric Tag (FTAG) to carry
> >> special information. The FTAG is placed at the beginning of the frame, it contains
> >> information such as where the packet comes from and goes, and the vlan tag. In
> >> FTAG based forwarding mode, the switch logic forwards packets according to
> >> glort (global resource tag) information, rather than the mac and vlan table.
> >> So this is a feature specific to fm10k.
> >
> > If it is fm10k specific, how about just adding a public function to the fm10k
> > driver to turn it on. The user app will be non-portable across NICs, but
> > that's the price of using nic-specific features.
> 
> What about using a devargs ?
> Something like :
> -w xxxx:xx:xx.x,enable_ftag=1
> 
> The application still needs to know about this to enable it, but that
> sounds better to me.
> The only issue is that it can't work with hotplug at the moment.
>
Seems a good enough solution to me. Xiao, any other thoughts?

/Bruce
  

2016-02-26 09:24, Wang, Xiao W:
> Hi,
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Sent: Friday, February 26, 2016 5:06 PM
> > To: Wang, Xiao W <xiao.w.wang@intel.com>
> > Cc: Richardson, Bruce <bruce.richardson@intel.com>; Chen, Jing D
> > <jing.d.chen@intel.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based forwarding
> > 
> > 2016-02-26 04:31, Wang, Xiao W:
> > > From: Richardson, Bruce
> > > > On Thu, Feb 25, 2016 at 03:45:45PM +0000, Chen, Jing D wrote:
> > > > > From: Richardson, Bruce
> > > > > > On Thu, Feb 25, 2016 at 10:04:02AM +0000, Chen, Jing D wrote:
> > > > > > > This feature is trying to use FTAG (a unique tech in fm10k)
> > > > > > > instead of mac/vlan to forward packets. App need a way to tell
> > > > > > > PMD driver that which forwarding style it would like to use.
> > > > > >
> > > > > > Why not just specify this in the port configuration at setup time?
> > > > > >
> > > > >
> > > > > Please educate me. I think the port configuration flags are also
> > > > > common to all PMD Drivers. Is it possible to add a flag like
> > "RTE_USE_FTAG"
> > > > and pass to PMD driver?
> > > > >
> > > > They are.
> > > > For something PMD specific, like FTAG, it's always a challenge, and
> > > > I don't know off the top of my head if there is a simple option.
> > > > However, given the choice between an mbuf flag and a port config
> > > > flag, I'd always choose the former.
> > > > Other alternatives would be to have a fm10k specific API in the
> > > > fm10k driver alone.
> > > >
> > > > I'll let Thomas as ethdev maintainer comment if he has other
> > > > suggestions as to how to handle this case. I suspect this won't be
> > > > the first device-specific piece of functionality we need to deal with.
> > > >
> > > > /Bruce
> > >
> > > Whatever method we choose, we have to find a way for the user to
> > > express his need for FTAG, it maybe a build time config option, or a
> > > port config flag (no such flag now), or a fast path flag in mbuf (no
> > > such flag now) etc. For the customer Topsec's use case, they use FTAG
> > > for all the TX packets, so all the above methods (per build config,
> > > per port config, per mbuf config) can meet their need. Since the pmd
> > > frame work is for common, it's hard to add new fields only for one specific
> > NIC, so I add a build time config and make an introduction in the doc.
> > >
> > > Thanks for the discussion, Thomas, do you have any suggestions?
> > 
> > I don't understand why you say this feature is specific to fm10k. Can we
> > imagine another NIC having this capability?
> 
> As you know, fm10k has a switch logic between the Mac and Phy, every packets
> Sent out from the host will be switched inside the NIC, other NICs don't have
> a switch inside, and the FTAG feature is related to the switch function.
> 
> As introduced in the second patch:
> The FM10K family of NICs support the addition of a Fabric Tag (FTAG) to carry
> special information. The FTAG is placed at the beginning of the frame, it contains
> information such as where the packet comes from and goes, and the vlan tag. In
> FTAG based forwarding mode, the switch logic forwards packets according to
> glort (global resource tag) information, rather than the mac and vlan table.
> So this is a feature specific to fm10k.

No I still don't think you should consider it specific to fm10k.
But yes it is specific to a switch device.
I'm OK to have a structure for switch configuration in ethdev.
  

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Saturday, February 27, 2016 12:33 AM
> To: David Marchand <david.marchand@6wind.com>
> Cc: Wang, Xiao W <xiao.w.wang@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3 1/3] fm10k: enable FTAG based forwarding
> 
> On Fri, Feb 26, 2016 at 04:00:49PM +0100, David Marchand wrote:
> > On Fri, Feb 26, 2016 at 3:48 PM, Bruce Richardson
> > <bruce.richardson@intel.com> wrote:
> > > On Fri, Feb 26, 2016 at 09:24:06AM +0000, Wang, Xiao W wrote:
> > >> Hi,
> > >> > > Thanks for the discussion, Thomas, do you have any suggestions?
> > >> >
> > >> > I don't understand why you say this feature is specific to fm10k.
> > >> > Can we imagine another NIC having this capability?
> > >>
> > >> As you know, fm10k has a switch logic between the Mac and Phy,
> > >> every packets Sent out from the host will be switched inside the
> > >> NIC, other NICs don't have a switch inside, and the FTAG feature is related
> to the switch function.
> > >>
> > >> As introduced in the second patch:
> > >> The FM10K family of NICs support the addition of a Fabric Tag
> > >> (FTAG) to carry special information. The FTAG is placed at the
> > >> beginning of the frame, it contains information such as where the
> > >> packet comes from and goes, and the vlan tag. In FTAG based
> > >> forwarding mode, the switch logic forwards packets according to glort
> (global resource tag) information, rather than the mac and vlan table.
> > >> So this is a feature specific to fm10k.
> > >
> > > If it is fm10k specific, how about just adding a public function to
> > > the fm10k driver to turn it on. The user app will be non-portable
> > > across NICs, but that's the price of using nic-specific features.
> >
> > What about using a devargs ?
> > Something like :
> > -w xxxx:xx:xx.x,enable_ftag=1
> >
> > The application still needs to know about this to enable it, but that
> > sounds better to me.
> > The only issue is that it can't work with hotplug at the moment.
> >
> Seems a good enough solution to me. Xiao, any other thoughts?
> 
> /Bruce

I also agree with the devargs solution, in this way, the build time config can
be removed and we don't need to add extra fields into ethdev.
I'll rework the patch, thanks for the suggestions.

Best Regards,
Xiao
  

v4:
* Removed the build time config option, used devargs to config FTAG.
* Rebased on head of dpdk-next-net/rel_16_04 branch.

v3:
* Removed "\n" in PMD_INIT_LOG.
* Returned "-ENOTSUP" instead of -1 in VF FTAG use case.

v2:
* Gave an error message for VF FTAG use case.
* Added a notice in the doc to emphasize that application should ensure
  an appropriate FTAG for every frame in FTAG based forwarding mode.

Wang Xiao W (3):
  fm10k: enable FTAG based forwarding
  doc: add introduction for fm10k FTAG based forwarding
  doc: update release note for fm10k FTAG support

 doc/guides/nics/fm10k.rst              | 16 +++++++++++++-
 doc/guides/rel_notes/release_16_04.rst |  2 ++
 drivers/net/fm10k/fm10k.h              |  2 ++
 drivers/net/fm10k/fm10k_ethdev.c       | 38 +++++++++++++++++++++++++++++++++-
 drivers/net/fm10k/fm10k_rxtx.c         | 15 ++++++++++++++
 drivers/net/fm10k/fm10k_rxtx_vec.c     |  4 ++++
 6 files changed, 75 insertions(+), 2 deletions(-)