[dpdk-dev,4/5] mbuf: add userdata pointer field

  While some applications may store metadata about packets in the packet
mbuf headroom, this is not a workable solution for packet metadata which
is either:
* larger than the headroom (or headroom is needed for adding pkt headers)
* needs to be shared or copied among packets

To support these use cases in applications, we reserve a general
"userdata" pointer field inside the second cache-line of the mbuf. This
is better than having the application store the pointer to the external
metadata in the packet headroom, as it saves an additional cache-line
from being used.

Apart from storing metadata, this field also provides a general 8-byte
scratch space inside the mbuf for any other application uses that are
applicable.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h | 3 ++-
 lib/librte_mbuf/rte_mbuf.h                                    | 3 +++
 2 files changed, 5 insertions(+), 1 deletion(-)
  

On Wed, Sep 17, 2014 at 11:01:41AM +0100, Bruce Richardson wrote:
> While some applications may store metadata about packets in the packet
> mbuf headroom, this is not a workable solution for packet metadata which
> is either:
> * larger than the headroom (or headroom is needed for adding pkt headers)
> * needs to be shared or copied among packets
> 
> To support these use cases in applications, we reserve a general
> "userdata" pointer field inside the second cache-line of the mbuf. This
> is better than having the application store the pointer to the external
> metadata in the packet headroom, as it saves an additional cache-line
> from being used.
> 
> Apart from storing metadata, this field also provides a general 8-byte
> scratch space inside the mbuf for any other application uses that are
> applicable.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h | 3 ++-
>  lib/librte_mbuf/rte_mbuf.h                                    | 3 +++
>  2 files changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> index 25ed672..d27e891 100644
> --- a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> +++ b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> @@ -117,7 +117,8 @@ struct rte_kni_mbuf {
>  	uint16_t data_len;      /**< Amount of data in segment buffer. */
>  	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
>  	char pad3[8];
> -	void *pool __attribute__((__aligned__(64)));
> +	void *pad4 __attribute__((__aligned__(64)));
> +	void *pool;
I don't see a comment about this in the changelog, only about the userdata
pointer being added below.


>  	void *next;
>  };
>  
> diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h
> index 8e27d2e..b1acfc3 100644
> --- a/lib/librte_mbuf/rte_mbuf.h
> +++ b/lib/librte_mbuf/rte_mbuf.h
> @@ -172,6 +172,9 @@ struct rte_mbuf {
>  
>  	/* second cache line - fields only used in slow path or on TX */
>  	MARKER cacheline1 __rte_cache_aligned;
> +
> +	void *userdata;           /**< Can be used for external metadata */
> +
Do you want to make this a void* or a char[8]?  I ask because if people are
going to use is as a scratch space (rather than a pointer), they get a suprise
when they build this on 32 bit systems, and their 8 byte scratch space is
reduced to 4 bytes.

Neil

>  	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
>  	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
>  
> -- 
> 1.9.3
> 
>
  

> -----Original Message-----
> From: Neil Horman [mailto:nhorman@tuxdriver.com]
> Sent: Wednesday, September 17, 2014 4:35 PM
> To: Richardson, Bruce
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH 4/5] mbuf: add userdata pointer field
> 
> On Wed, Sep 17, 2014 at 11:01:41AM +0100, Bruce Richardson wrote:
> > While some applications may store metadata about packets in the packet
> > mbuf headroom, this is not a workable solution for packet metadata which
> > is either:
> > * larger than the headroom (or headroom is needed for adding pkt headers)
> > * needs to be shared or copied among packets
> >
> > To support these use cases in applications, we reserve a general
> > "userdata" pointer field inside the second cache-line of the mbuf. This
> > is better than having the application store the pointer to the external
> > metadata in the packet headroom, as it saves an additional cache-line
> > from being used.
> >
> > Apart from storing metadata, this field also provides a general 8-byte
> > scratch space inside the mbuf for any other application uses that are
> > applicable.
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> >  lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h | 3 ++-
> >  lib/librte_mbuf/rte_mbuf.h                                    | 3 +++
> >  2 files changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> > index 25ed672..d27e891 100644
> > --- a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> > +++ b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> > @@ -117,7 +117,8 @@ struct rte_kni_mbuf {
> >  	uint16_t data_len;      /**< Amount of data in segment buffer. */
> >  	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
> >  	char pad3[8];
> > -	void *pool __attribute__((__aligned__(64)));
> > +	void *pad4 __attribute__((__aligned__(64)));
> > +	void *pool;
> I don't see a comment about this in the changelog, only about the userdata
> pointer being added below.

Yes, this is the userdata pointer - just added as padding here, since it's not actually needed by the kernel-side KNI module.

> 
> 
> >  	void *next;
> >  };
> >
> > diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h
> > index 8e27d2e..b1acfc3 100644
> > --- a/lib/librte_mbuf/rte_mbuf.h
> > +++ b/lib/librte_mbuf/rte_mbuf.h
> > @@ -172,6 +172,9 @@ struct rte_mbuf {
> >
> >  	/* second cache line - fields only used in slow path or on TX */
> >  	MARKER cacheline1 __rte_cache_aligned;
> > +
> > +	void *userdata;           /**< Can be used for external metadata */
> > +
> Do you want to make this a void* or a char[8]?  I ask because if people are
> going to use is as a scratch space (rather than a pointer), they get a suprise
> when they build this on 32 bit systems, and their 8 byte scratch space is
> reduced to 4 bytes.

I think this is better as a pointer, as that is how it is likely to be used. As for 32-bit, I'm torn between wanting to just update the comment and feeling the need to update the code to actually make the thing 8-byte on 32-bit! Changing the comment to be more accurate is easier, unions are ugly looking in the structure IMHO, so maybe I'll just mark the following field (pool) as always 8-byte aligned....

/Bruce

> 
> Neil
> 
> >  	struct rte_mempool *pool; /**< Pool from which mbuf was allocated.
> */
> >  	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
> >
> > --
> > 1.9.3
> >
> >
  

On Wed, Sep 17, 2014 at 04:02:07PM +0000, Richardson, Bruce wrote:
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Wednesday, September 17, 2014 4:35 PM
> > To: Richardson, Bruce
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH 4/5] mbuf: add userdata pointer field
> > 
> > On Wed, Sep 17, 2014 at 11:01:41AM +0100, Bruce Richardson wrote:
> > > While some applications may store metadata about packets in the packet
> > > mbuf headroom, this is not a workable solution for packet metadata which
> > > is either:
> > > * larger than the headroom (or headroom is needed for adding pkt headers)
> > > * needs to be shared or copied among packets
> > >
> > > To support these use cases in applications, we reserve a general
> > > "userdata" pointer field inside the second cache-line of the mbuf. This
> > > is better than having the application store the pointer to the external
> > > metadata in the packet headroom, as it saves an additional cache-line
> > > from being used.
> > >
> > > Apart from storing metadata, this field also provides a general 8-byte
> > > scratch space inside the mbuf for any other application uses that are
> > > applicable.
> > >
> > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > > ---
> > >  lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h | 3 ++-
> > >  lib/librte_mbuf/rte_mbuf.h                                    | 3 +++
> > >  2 files changed, 5 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> > b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> > > index 25ed672..d27e891 100644
> > > --- a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> > > +++ b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
> > > @@ -117,7 +117,8 @@ struct rte_kni_mbuf {
> > >  	uint16_t data_len;      /**< Amount of data in segment buffer. */
> > >  	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
> > >  	char pad3[8];
> > > -	void *pool __attribute__((__aligned__(64)));
> > > +	void *pad4 __attribute__((__aligned__(64)));
> > > +	void *pool;
> > I don't see a comment about this in the changelog, only about the userdata
> > pointer being added below.
> 
> Yes, this is the userdata pointer - just added as padding here, since it's not actually needed by the kernel-side KNI module.
> 
Ah, then maybe merge it with the pad3 pointer, to make it look a bit better?

> > 
> > 
> > >  	void *next;
> > >  };
> > >
> > > diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h
> > > index 8e27d2e..b1acfc3 100644
> > > --- a/lib/librte_mbuf/rte_mbuf.h
> > > +++ b/lib/librte_mbuf/rte_mbuf.h
> > > @@ -172,6 +172,9 @@ struct rte_mbuf {
> > >
> > >  	/* second cache line - fields only used in slow path or on TX */
> > >  	MARKER cacheline1 __rte_cache_aligned;
> > > +
> > > +	void *userdata;           /**< Can be used for external metadata */
> > > +
> > Do you want to make this a void* or a char[8]?  I ask because if people are
> > going to use is as a scratch space (rather than a pointer), they get a suprise
> > when they build this on 32 bit systems, and their 8 byte scratch space is
> > reduced to 4 bytes.
> 
> I think this is better as a pointer, as that is how it is likely to be used. As for 32-bit, I'm torn between wanting to just update the comment and feeling the need to update the code to actually make the thing 8-byte on 32-bit! Changing the comment to be more accurate is easier, unions are ugly looking in the structure IMHO, so maybe I'll just mark the following field (pool) as always 8-byte aligned....
> 
Ok, however you want to bring it into alignment, as long as its clear that its
either 8 bytes always, or a variable size based on the arch.

Best
Neil

> /Bruce
> 
> > 
> > Neil
> > 
> > >  	struct rte_mempool *pool; /**< Pool from which mbuf was allocated.
> > */
> > >  	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
> > >
> > > --
> > > 1.9.3
> > >
> > >
>