[v3] eal: add generic thread-local-storage functions

  Add support for tls functionality in EAL.

The following functions are added:
rte_tls_create_key - function to create a tls data key.
rte_tls_delete_key - function to delete a tls data key.
rte_tls_set_thread_value - function to set value bound to the tls key
rte_tls_get_thread_value - function to get value bound to the tls key

tls key will be defied by the new type rte_tls_key_t

Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
---
v3: switch from pthread shim to generic eal implementation [DmitryK]
---
 lib/librte_eal/include/meson.build |  1 +
 lib/librte_eal/include/rte_tls.h   | 88 ++++++++++++++++++++++++++++++++++++++
 lib/librte_eal/rte_eal_exports.def |  5 +++
 lib/librte_eal/unix/eal_unix_tls.c | 79 ++++++++++++++++++++++++++++++++++
 lib/librte_eal/unix/meson.build    |  1 +
 lib/librte_eal/version.map         |  6 +++
 lib/librte_eal/windows/eal_tls.c   | 72 +++++++++++++++++++++++++++++++
 lib/librte_eal/windows/meson.build |  1 +
 8 files changed, 253 insertions(+)
 create mode 100644 lib/librte_eal/include/rte_tls.h
 create mode 100644 lib/librte_eal/unix/eal_unix_tls.c
 create mode 100644 lib/librte_eal/windows/eal_tls.c
  

On Thu, 17 Dec 2020 19:49:13 +0200, Tal Shnaiderman wrote:
> Add support for tls functionality in EAL.
> 
> The following functions are added:
> rte_tls_create_key - function to create a tls data key.
> rte_tls_delete_key - function to delete a tls data key.
> rte_tls_set_thread_value - function to set value bound to the tls key
> rte_tls_get_thread_value - function to get value bound to the tls key
> 
> tls key will be defied by the new type rte_tls_key_t
> 
> Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> ---
> v3: switch from pthread shim to generic eal implementation [DmitryK]

Hi Tal,

Unix code can be placed in common/ directory, so that it can be eventually
used on Windows with external pthread implementation.

See more comments inline.

> +++ b/lib/librte_eal/include/rte_tls.h
> @@ -0,0 +1,88 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2020 Mellanox Technologies, Ltd
> + */
> +
> +#include <rte_compat.h>
> +
> +#ifndef _RTE_TLS_H_
> +#define _RTE_TLS_H_
> +
> +/**
> + * @file
> + *
> + * TLS functions
> + *
> + * Simple TLS functionality supplied by eal.
> + */

These functions are supposed to be the first part of an API that will be
used instead of naked pthread in DPDK. Maybe more generic names are in order,
like rte_thread.h and rte_thread_tls_create/destroy/get/set(). In particular,
rte_tls_*() is confusing compared to rte_lcore_*().

> +
> +/**
> + * Opaque pointer for tls key.
> + */
> +typedef struct eal_tls_key *rte_tls_key_t;

"_t" suffix is reserved by POSIX, "rte_" prefix is sufficient.

> +
> +/**
> + * Function to create a tls data key visible to all threads in the process

Typos: "TLS", "EAL".

> + * function need to be called once to create a key usable by all threads.
> + * rte_tls_key_t is an opaque pointer used to store the allocated key.
> + * and optional destructor can be set to be called when a thread expires.
> + *
> + * @param key
> + *   The rte_tls_key_t will cantain the allocated key

Typo: "cantain". I'd say: "Pointer to store the allocated rte_tls_key".

> + * @param destructor
> + *   The function to be called when the thread expires

Typo: no period (line-break will be removed by Doxygen).

> + *   Not supported on Windows OS.

Some drivers (net/mlx5, bus/dpaa, bus/fsmlc) rely on this feature.
Admittedly, it would be hard to implement. You know net/mlx5 well, how will
it be affected? Do you plan to stop relying on this feature or implement it
in the future? Anyway, LGTM for now.

> + *
> + * @return
> + *   On success, zero.
> + *   On failure, a negative error number -ENOMEM or -ENOEXEC

Let's not make POSIX codes part of the API.

> + */
> +__rte_experimental
> +int
> +rte_tls_create_key(rte_tls_key_t *key, void (*destructor)(void *));

[...]
> +int
> +rte_tls_create_key(rte_tls_key_t *key, void (*destructor)(void *))
> +{
> +	int err;
> +
> +	*key = malloc(sizeof(struct eal_tls_key));
> +	if ((*key) == NULL) {
> +		RTE_LOG(DEBUG, EAL, "Cannot allocate tls key.");
> +		return -ENOMEM;
> +	}
> +	err = pthread_key_create(&((*key)->thread_index), destructor);
> +	if (err) {
> +		RTE_LOG(DEBUG, EAL, "pthread_key_create failed: %s\n",
> +			 rte_strerror(err));

Use strerror() for OS error codes here and below.

> +		free(*key);
> +		return -ENOEXEC;
> +	}
> +	return 0;
> +}
> +

[...]
> +int
> +rte_tls_set_thread_value(rte_tls_key_t key, const void *value)
> +{
> +	if (!key)
> +		return -EINVAL;
> +	/* discard const qualifier */
> +	char *p = (char *) (uintptr_t) value;

Please declare all variables at the top of a block.

> +
> +	if (!TlsSetValue(key->thread_index, p)) {
> +		RTE_LOG_WIN32_ERR("TlsSetValue()");
> +		return -ENOEXEC;
> +	}
> +	return 0;
> +}
> +
> +void *
> +rte_tls_get_thread_value(rte_tls_key_t key)
> +{
> +	if (!key)
> +		rte_errno = EINVAL;
> +	void *output = TlsGetValue(key->thread_index);

Same as above.

> +	if (GetLastError() != ERROR_SUCCESS) {
> +		RTE_LOG_WIN32_ERR("TlsGetValue()");
> +		rte_errno = ENOEXEC;
> +		return NULL;
> +	}
> +	return output;
> +}
  

> Subject: Re: [PATCH v3] eal: add generic thread-local-storage functions
> 
> External email: Use caution opening links or attachments
> 
> 
> On Thu, 17 Dec 2020 19:49:13 +0200, Tal Shnaiderman wrote:
> > Add support for tls functionality in EAL.
> >
> > The following functions are added:
> > rte_tls_create_key - function to create a tls data key.
> > rte_tls_delete_key - function to delete a tls data key.
> > rte_tls_set_thread_value - function to set value bound to the tls key
> > rte_tls_get_thread_value - function to get value bound to the tls key
> >
> > tls key will be defied by the new type rte_tls_key_t
> >
> > Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> > ---
> > v3: switch from pthread shim to generic eal implementation [DmitryK]
> 
> Hi Tal,
> 
> Unix code can be placed in common/ directory, so that it can be eventually
> used on Windows with external pthread implementation.
> 
> See more comments inline.
> 
> > +++ b/lib/librte_eal/include/rte_tls.h
> > @@ -0,0 +1,88 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2020 Mellanox Technologies, Ltd  */
> > +
> > +#include <rte_compat.h>
> > +
> > +#ifndef _RTE_TLS_H_
> > +#define _RTE_TLS_H_
> > +
> > +/**
> > + * @file
> > + *
> > + * TLS functions
> > + *
> > + * Simple TLS functionality supplied by eal.
> > + */
> 
> These functions are supposed to be the first part of an API that will be used
> instead of naked pthread in DPDK. Maybe more generic names are in order,
> like rte_thread.h and rte_thread_tls_create/destroy/get/set(). In particular,
> rte_tls_*() is confusing compared to rte_lcore_*().
> 
> > +
> > +/**
> > + * Opaque pointer for tls key.
> > + */
> > +typedef struct eal_tls_key *rte_tls_key_t;
> 
> "_t" suffix is reserved by POSIX, "rte_" prefix is sufficient.
> 
> > +
> > +/**
> > + * Function to create a tls data key visible to all threads in the
> > +process
> 
> Typos: "TLS", "EAL".
> 

Didn't find the EAL typo.

> > + * function need to be called once to create a key usable by all threads.
> > + * rte_tls_key_t is an opaque pointer used to store the allocated key.
> > + * and optional destructor can be set to be called when a thread expires.
> > + *
> > + * @param key
> > + *   The rte_tls_key_t will cantain the allocated key
> 
> Typo: "cantain". I'd say: "Pointer to store the allocated rte_tls_key".
> 
> > + * @param destructor
> > + *   The function to be called when the thread expires
> 
> Typo: no period (line-break will be removed by Doxygen).
> 
> > + *   Not supported on Windows OS.
> 
> Some drivers (net/mlx5, bus/dpaa, bus/fsmlc) rely on this feature.
> Admittedly, it would be hard to implement. You know net/mlx5 well, how
> will it be affected? Do you plan to stop relying on this feature or implement it
> in the future? Anyway, LGTM for now.

In net/mlx5 we will not relay on the destructor for the Windows flow and plan to do the housekeeping in the PMD, to summarize we save HANDLE for each thread using rte_tls_set_thread_value in the PMD destruction flow we call the needed release function for each terminated thread.

I used the opaque pointer in case future development might want to implement It without breaking the API, I assumed additions to the TLS key struct on Windows side will be needed.

Thanks for the review, I'll send a v4 after fixing your comments.

> 
> > + *
> > + * @return
> > + *   On success, zero.
> > + *   On failure, a negative error number -ENOMEM or -ENOEXEC
> 
> Let's not make POSIX codes part of the API.
> 
> > + */
> > +__rte_experimental
> > +int
> > +rte_tls_create_key(rte_tls_key_t *key, void (*destructor)(void *));
> 
> [...]
> > +int
> > +rte_tls_create_key(rte_tls_key_t *key, void (*destructor)(void *)) {
> > +     int err;
> > +
> > +     *key = malloc(sizeof(struct eal_tls_key));
> > +     if ((*key) == NULL) {
> > +             RTE_LOG(DEBUG, EAL, "Cannot allocate tls key.");
> > +             return -ENOMEM;
> > +     }
> > +     err = pthread_key_create(&((*key)->thread_index), destructor);
> > +     if (err) {
> > +             RTE_LOG(DEBUG, EAL, "pthread_key_create failed: %s\n",
> > +                      rte_strerror(err));
> 
> Use strerror() for OS error codes here and below.
> 
> > +             free(*key);
> > +             return -ENOEXEC;
> > +     }
> > +     return 0;
> > +}
> > +
> 
> [...]
> > +int
> > +rte_tls_set_thread_value(rte_tls_key_t key, const void *value) {
> > +     if (!key)
> > +             return -EINVAL;
> > +     /* discard const qualifier */
> > +     char *p = (char *) (uintptr_t) value;
> 
> Please declare all variables at the top of a block.
> 
> > +
> > +     if (!TlsSetValue(key->thread_index, p)) {
> > +             RTE_LOG_WIN32_ERR("TlsSetValue()");
> > +             return -ENOEXEC;
> > +     }
> > +     return 0;
> > +}
> > +
> > +void *
> > +rte_tls_get_thread_value(rte_tls_key_t key) {
> > +     if (!key)
> > +             rte_errno = EINVAL;
> > +     void *output = TlsGetValue(key->thread_index);
> 
> Same as above.
> 
> > +     if (GetLastError() != ERROR_SUCCESS) {
> > +             RTE_LOG_WIN32_ERR("TlsGetValue()");
> > +             rte_errno = ENOEXEC;
> > +             return NULL;
> > +     }
> > +     return output;
> > +}