[dpdk-dev] ixgbe: fix icc issue with mbuf initializer

  When using Intel C++ compiler(icc) 14.0.1.106 or the older icc 13.x
version, the mbuf initializer variable was not getting configured
correctly, as the mb_def variable was not set correctly. This is due
to an issue with icc (DPD200249565 which already been fixed in
icc 14.0.2 and newer compiler release) where it incorrectly calculates
the field offsets with initializers when zero-sized fields
are used in a structure.
To work around this, the code in ixgbe_rxq_vec_setup does not setup the
fields using an initializer, but instead assigns the values individually
in code
NOTE: There is no performance impact to this change as the queue
setup functions are not data-plane APIs, but are only used at app
initialization.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
  

Hello Bruce,

On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
bruce.richardson@intel.com> wrote:

> diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> index e813e43..b57c588 100644
> --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> @@ -730,16 +730,15 @@ static struct ixgbe_txq_ops vec_txq_ops = {
>  int
>  ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
>  {
> -       struct rte_mbuf mb_def = {
> -               .nb_segs = 1,
> -               .data_off = RTE_PKTMBUF_HEADROOM,
> -#ifdef RTE_MBUF_REFCNT
> -               { .refcnt = 1, }
> -#endif
> -       };
> +       struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
>
> +       mb_def.nb_segs = 1;
> +       mb_def.data_off = RTE_PKTMBUF_HEADROOM;
>         mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
>         mb_def.port = rxq->port_id;
> +#ifdef RTE_MBUF_REFCNT
> +       mb_def.refcnt = 1;
> +#endif
>         rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
>         return 0;
>  }
>

I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
this "refcnt" field.
This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
configs.
But I suppose this is fine at init time (since the union will initialize
properly the field).

By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
From my point of view, there is not much use of a refcnt that is not atomic
:-).
  

On Mon, Nov 03, 2014 at 01:31:10PM +0100, David Marchand wrote:
> Hello Bruce,
> 
> On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
> bruce.richardson@intel.com> wrote:
> 
> > diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > index e813e43..b57c588 100644
> > --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > @@ -730,16 +730,15 @@ static struct ixgbe_txq_ops vec_txq_ops = {
> >  int
> >  ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
> >  {
> > -       struct rte_mbuf mb_def = {
> > -               .nb_segs = 1,
> > -               .data_off = RTE_PKTMBUF_HEADROOM,
> > -#ifdef RTE_MBUF_REFCNT
> > -               { .refcnt = 1, }
> > -#endif
> > -       };
> > +       struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
> >
> > +       mb_def.nb_segs = 1;
> > +       mb_def.data_off = RTE_PKTMBUF_HEADROOM;
> >         mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
> >         mb_def.port = rxq->port_id;
> > +#ifdef RTE_MBUF_REFCNT
> > +       mb_def.refcnt = 1;
> > +#endif
> >         rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
> >         return 0;
> >  }
> >
> 
> I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
> this "refcnt" field.
> This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
> configs.
> But I suppose this is fine at init time (since the union will initialize
> properly the field).

It's a good point, I'll update patch to use the appropriate macro which will clean up the code a bit.

/Bruce

> 
> By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
> From my point of view, there is not much use of a refcnt that is not atomic
> :-).
> 
> 
> -- 
> David Marchand
  

2014-11-03 12:47, Bruce Richardson:
> On Mon, Nov 03, 2014 at 01:31:10PM +0100, David Marchand wrote:
> > On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
> > > +#ifdef RTE_MBUF_REFCNT
> > > +       mb_def.refcnt = 1;
> > > +#endif
> > 
> > I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
> > this "refcnt" field.
> > This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
> > configs.
> > But I suppose this is fine at init time (since the union will initialize
> > properly the field).
> 
> It's a good point, I'll update patch to use the appropriate macro which will clean up the code a bit.

> > By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
> > From my point of view, there is not much use of a refcnt that is not atomic
> > :-).

Bruce, I think it's a good question but you didn't answer.
Maybe we should remove this option to keep only atomic mode.
  

On Mon, Nov 03, 2014 at 01:59:16PM +0100, Thomas Monjalon wrote:
> 2014-11-03 12:47, Bruce Richardson:
> > On Mon, Nov 03, 2014 at 01:31:10PM +0100, David Marchand wrote:
> > > On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
> > > > +#ifdef RTE_MBUF_REFCNT
> > > > +       mb_def.refcnt = 1;
> > > > +#endif
> > > 
> > > I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
> > > this "refcnt" field.
> > > This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
> > > configs.
> > > But I suppose this is fine at init time (since the union will initialize
> > > properly the field).
> > 
> > It's a good point, I'll update patch to use the appropriate macro which will clean up the code a bit.
> 
> > > By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
> > > From my point of view, there is not much use of a refcnt that is not atomic
> > > :-).
> 
> Bruce, I think it's a good question but you didn't answer.
> Maybe we should remove this option to keep only atomic mode.
> 

I didn't answer just because it wasn't directly relevant to the patch. It was not
meant as a snub. :-)

As for why the option is there, it's purely for performance, I suspect. The
cost of doing increments and decrements using atomic operations is far higher
than doing a read-modify-write on a single core. However, the downside is
obviously that you need to know what you are doing if you disable atomic refcnts
and, given that atomic is the default, I reckon we can probably get rid of the
option permanently - unless someone has a use case where they turn off the option,
and can't take the performance hit of the atomic instructions.

As a further asside, if we get the proposed changes made to the zero-copy vhost
implementation - discussed previously[1] - we should hopefully be able to get rid
of the refcnt option too, and leave it permanently enabled.

/Bruce

[1] Discussed in this thread: http://permalink.gmane.org/gmane.comp.networking.dpdk.devel/7098