mbox series

[v6,00/10] eal: Add new API for threading

Message ID 1617413948-10504-1-git-send-email-navasile@linux.microsoft.com (mailing list archive)
Headers show
Series eal: Add new API for threading | expand

Message

Narcisa Ana Maria Vasile April 3, 2021, 1:38 a.m. UTC
From: Narcisa Vasile <navasile@microsoft.com>

EAL thread API

**Problem Statement**
DPDK currently uses the pthread interface to create and manage threads.
Windows does not support the POSIX thread programming model, so it currently
relies on a header file that hides the Windows calls under
pthread matched interfaces. Given that EAL should isolate the environment
specifics from the applications and libraries and mediate
all the communication with the operating systems, a new EAL interface
is needed for thread management.

**Goals**
* Introduce a generic EAL API for threading support that will remove
  the current Windows pthread.h shim.
* Replace references to pthread_* across the DPDK codebase with the new
  RTE_THREAD_* API.
* Allow users to choose between using the RTE_THREAD_* API or a
  3rd party thread library through a configuration option.

**Design plan**
New API main files:
* rte_thread.h (librte_eal/include)
* rte_thread_types.h (librte_eal/include)
* rte_thread_windows_types.h (librte_eal/windows/include)
* rte_thread.c (librte_eal/windows)
* rte_thread.c (librte_eal/common)

For flexibility, the user is offered the option of either using the RTE_THREAD_* API or
a 3rd party thread library, through a meson flag “use_external_thread_lib”.
By default, this flag is set to FALSE, which means Windows libraries and applications
will use the RTE_THREAD_* API for managing threads.

If compiling on Windows and the “use_external_thread_lib” is *not* set,
the following files will be parsed: 
* include/rte_thread.h
* windows/include/rte_thread_windows_types.h
* windows/rte_thread.c
In all other cases, the compilation/parsing includes the following files:
* include/rte_thread.h 
* include/rte_thread_types.h
* common/rte_thread.c

**A schematic example of the design**
--------------------------------------------------
lib/librte_eal/include/rte_thread.h
int rte_thread_create();

lib/librte_eal/common/rte_thread.c
int rte_thread_create() 
{
	return pthread_create();
}

lib/librte_eal/windows/rte_thread.c
int rte_thread_create() 
{
	return CreateThread();
}

lib/librte_eal/windows/meson.build
if get_option('use_external_thread_lib')
	sources += 'librte_eal/common/rte_thread.c'
else
	sources += 'librte_eal/windows/rte_thread.c'
endif
-----------------------------------------------------

**Thread attributes**

When or after a thread is created, specific characteristics of the thread
can be adjusted. Given that the thread characteristics that are of interest
for DPDK applications are affinity and priority, the following structure
that represents thread attributes has been defined:

typedef struct
{
	enum rte_thread_priority priority;
	rte_cpuset_t cpuset;
} rte_thread_attr_t;

The *rte_thread_create()* function can optionally receive an rte_thread_attr_t
object that will cause the thread to be created with the affinity and priority
described by the attributes object. If no rte_thread_attr_t is passed
(parameter is NULL), the default affinity and priority are used.
An rte_thread_attr_t object can also be set to the default values
by calling *rte_thread_attr_init()*.

*Priority* is represented through an enum that currently advertises
two values for priority:
	- RTE_THREAD_PRIORITY_NORMAL
	- RTE_THREAD_PRIORITY_REALTIME_CRITICAL
The enum can be extended to allow for multiple priority levels.
rte_thread_set_priority      - sets the priority of a thread
rte_thread_attr_set_priority - updates an rte_thread_attr_t object
                               with a new value for priority

The user can choose thread priority through an EAL parameter,
when starting an application.  If EAL parameter is not used,
the per-platform default value for thread priority is used.
Otherwise administrator has an option to set one of available options:
 --thread-prio normal
 --thread-prio realtime

Example:
./dpdk-l2fwd -l 0-3 -n 4 –thread-prio normal -- -q 8 -p ffff

*Affinity* is described by the already known “rte_cpuset_t” type.
rte_thread_attr_set/get_affinity - sets/gets the affinity field in a
                                   rte_thread_attr_t object
rte_thread_set/get_affinity      – sets/gets the affinity of a thread

**Errors**
A translation function that maps Windows error codes to errno-style
error codes is provided. 

**Future work**
Note that this patchset was focused on introducing new API that will
remove the Windows pthread.h shim. In DPDK, there are still a few references
to pthread_* that were not implemented in the shim.
The long term plan is for EAL to provide full threading support:
* Adding support for conditional variables
* Additional functionality offered by pthread_* (such as pthread_setname_np, etc.)
* Static mutex initializers are not used on Windows. If we must continue
  using them, they need to be platform dependent and an implementation will
  need to be provided for Windows.

v6:
- improve error-translation function
- call the error translation function in rte_thread_value_get()

v5:
- update cover letter with more details on the priority argument

v4:
- fix function description
- rebase

v3:
- rebase

v2:
- revert changes that break ABI 
- break up changes into smaller patches
- fix coding style issues
- fix issues with errors
- fix parameter type in examples/kni.c

Narcisa Vasile (10):
  eal: add thread id and simple thread functions
  eal: add thread attributes
  windows/eal: translate Windows errors to errno-style errors
  eal: implement functions for thread affinity management
  eal: implement thread priority management functions
  eal: add thread lifetime management
  eal: implement functions for mutex management
  eal: implement functions for thread barrier management
  eal: add EAL argument for setting thread priority
  Enable the new EAL thread API

 app/test/process.h                            |   6 +-
 app/test/test_lcores.c                        |  16 +-
 app/test/test_link_bonding.c                  |  10 +-
 app/test/test_lpm_perf.c                      |  12 +-
 config/meson.build                            |   4 +
 drivers/bus/dpaa/base/qbman/bman_driver.c     |   6 +-
 drivers/bus/dpaa/base/qbman/dpaa_sys.c        |  14 +-
 drivers/bus/dpaa/base/qbman/process.c         |   6 +-
 drivers/bus/fslmc/portal/dpaa2_hw_dpio.c      |   6 +-
 drivers/compress/mlx5/mlx5_compress.c         |  10 +-
 drivers/event/dlb/pf/base/dlb_osdep.h         |   2 +-
 drivers/event/dlb2/pf/base/dlb2_osdep.h       |   2 +-
 drivers/net/af_xdp/rte_eth_af_xdp.c           |  18 +-
 drivers/net/ark/ark_ethdev.c                  |   4 +-
 drivers/net/atlantic/atl_ethdev.c             |   4 +-
 drivers/net/atlantic/atl_types.h              |   5 +-
 .../net/atlantic/hw_atl/hw_atl_utils_fw2x.c   |  24 +-
 drivers/net/axgbe/axgbe_dev.c                 |   8 +-
 drivers/net/axgbe/axgbe_ethdev.c              |   8 +-
 drivers/net/axgbe/axgbe_ethdev.h              |   8 +-
 drivers/net/axgbe/axgbe_i2c.c                 |   4 +-
 drivers/net/axgbe/axgbe_mdio.c                |   8 +-
 drivers/net/axgbe/axgbe_phy_impl.c            |   6 +-
 drivers/net/bnxt/bnxt.h                       |  16 +-
 drivers/net/bnxt/bnxt_cpr.c                   |   4 +-
 drivers/net/bnxt/bnxt_ethdev.c                |  52 +-
 drivers/net/bnxt/bnxt_irq.c                   |   8 +-
 drivers/net/bnxt/bnxt_reps.c                  |  10 +-
 drivers/net/bnxt/tf_ulp/bnxt_ulp.c            |  34 +-
 drivers/net/bnxt/tf_ulp/bnxt_ulp.h            |   4 +-
 drivers/net/bnxt/tf_ulp/ulp_fc_mgr.c          |  24 +-
 drivers/net/bnxt/tf_ulp/ulp_fc_mgr.h          |   2 +-
 drivers/net/ena/base/ena_plat_dpdk.h          |   8 +-
 drivers/net/enic/enic.h                       |   2 +-
 drivers/net/hns3/hns3_ethdev.h                |   2 +-
 drivers/net/hns3/hns3_ethdev_vf.c             |   2 +-
 drivers/net/hns3/hns3_mbx.c                   |   2 +-
 drivers/net/ice/ice_dcf_parent.c              |   4 +-
 drivers/net/ipn3ke/ipn3ke_representor.c       |   6 +-
 drivers/net/ixgbe/ixgbe_ethdev.c              |   2 +-
 drivers/net/ixgbe/ixgbe_ethdev.h              |   2 +-
 drivers/net/kni/rte_eth_kni.c                 |   6 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   2 +-
 drivers/net/mlx5/mlx5.c                       |  20 +-
 drivers/net/mlx5/mlx5.h                       |   2 +-
 drivers/net/mlx5/mlx5_txpp.c                  |   8 +-
 drivers/net/mlx5/windows/mlx5_flow_os.c       |  10 +-
 drivers/net/mlx5/windows/mlx5_os.c            |   2 +-
 drivers/net/qede/base/bcm_osal.h              |   8 +-
 drivers/net/vhost/rte_eth_vhost.c             |  22 +-
 .../net/virtio/virtio_user/virtio_user_dev.c  |  30 +-
 .../net/virtio/virtio_user/virtio_user_dev.h  |   2 +-
 drivers/raw/ifpga/ifpga_rawdev.c              |   8 +-
 drivers/vdpa/ifc/ifcvf_vdpa.c                 |  36 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  24 +-
 drivers/vdpa/mlx5/mlx5_vdpa.h                 |   6 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c           |  67 +--
 examples/kni/main.c                           |   6 +-
 examples/vhost/main.c                         |   2 +-
 examples/vhost_blk/vhost_blk.c                |  10 +-
 lib/librte_eal/common/eal_common_options.c    |  36 +-
 lib/librte_eal/common/eal_common_proc.c       |  48 +-
 lib/librte_eal/common/eal_common_thread.c     |  43 +-
 lib/librte_eal/common/eal_common_trace.c      |   2 +-
 lib/librte_eal/common/eal_internal_cfg.h      |   2 +
 lib/librte_eal/common/eal_options.h           |   2 +
 lib/librte_eal/common/eal_private.h           |   2 +-
 lib/librte_eal/common/malloc_mp.c             |  32 +-
 lib/librte_eal/common/meson.build             |   1 +
 lib/librte_eal/common/rte_thread.c            | 342 +++++++++++
 lib/librte_eal/freebsd/eal.c                  |  37 +-
 lib/librte_eal/freebsd/eal_alarm.c            |  12 +-
 lib/librte_eal/freebsd/eal_interrupts.c       |   4 +-
 lib/librte_eal/freebsd/eal_thread.c           |   8 +-
 lib/librte_eal/include/meson.build            |   1 +
 lib/librte_eal/include/rte_lcore.h            |   8 +-
 lib/librte_eal/include/rte_per_lcore.h        |   2 -
 lib/librte_eal/include/rte_thread.h           | 342 ++++++++++-
 lib/librte_eal/include/rte_thread_types.h     |  20 +
 lib/librte_eal/linux/eal.c                    |  42 +-
 lib/librte_eal/linux/eal_alarm.c              |  10 +-
 lib/librte_eal/linux/eal_interrupts.c         |   4 +-
 lib/librte_eal/linux/eal_thread.c             |   8 +-
 lib/librte_eal/linux/eal_timer.c              |   2 +-
 lib/librte_eal/rte_eal_exports.def            |  20 +
 lib/librte_eal/unix/meson.build               |   1 -
 lib/librte_eal/unix/rte_thread.c              |  92 ---
 lib/librte_eal/version.map                    |  21 +
 lib/librte_eal/windows/eal.c                  |  26 +-
 lib/librte_eal/windows/eal_interrupts.c       |   6 +-
 lib/librte_eal/windows/eal_lcore.c            | 170 ++++--
 lib/librte_eal/windows/eal_thread.c           |  24 +-
 lib/librte_eal/windows/eal_windows.h          |  20 +-
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/pthread.h      | 186 ------
 .../include/rte_windows_thread_types.h        |  19 +
 lib/librte_eal/windows/include/sched.h        |   2 +-
 lib/librte_eal/windows/meson.build            |   7 +-
 lib/librte_eal/windows/rte_thread.c           | 532 +++++++++++++++++-
 lib/librte_ethdev/rte_ethdev.c                |   4 +-
 lib/librte_ethdev/rte_ethdev_core.h           |   3 +-
 lib/librte_ethdev/rte_flow.c                  |   4 +-
 .../rte_event_eth_rx_adapter.c                |   6 +-
 lib/librte_vhost/fd_man.c                     |  40 +-
 lib/librte_vhost/fd_man.h                     |   6 +-
 lib/librte_vhost/socket.c                     | 124 ++--
 lib/librte_vhost/vhost.c                      |  10 +-
 meson_options.txt                             |   2 +
 108 files changed, 2048 insertions(+), 934 deletions(-)
 create mode 100644 lib/librte_eal/common/rte_thread.c
 create mode 100644 lib/librte_eal/include/rte_thread_types.h
 delete mode 100644 lib/librte_eal/unix/rte_thread.c
 delete mode 100644 lib/librte_eal/windows/include/pthread.h
 create mode 100644 lib/librte_eal/windows/include/rte_windows_thread_types.h

Comments

David Marchand April 12, 2021, 7:56 a.m. UTC | #1
On Sat, Apr 3, 2021 at 3:39 AM Narcisa Ana Maria Vasile
<navasile@linux.microsoft.com> wrote:
>
> From: Narcisa Vasile <navasile@microsoft.com>
>
> EAL thread API
>
> **Problem Statement**
> DPDK currently uses the pthread interface to create and manage threads.
> Windows does not support the POSIX thread programming model, so it currently
> relies on a header file that hides the Windows calls under
> pthread matched interfaces. Given that EAL should isolate the environment
> specifics from the applications and libraries and mediate
> all the communication with the operating systems, a new EAL interface
> is needed for thread management.
>
> **Goals**
> * Introduce a generic EAL API for threading support that will remove
>   the current Windows pthread.h shim.
> * Replace references to pthread_* across the DPDK codebase with the new
>   RTE_THREAD_* API.
> * Allow users to choose between using the RTE_THREAD_* API or a
>   3rd party thread library through a configuration option.
>
> **Design plan**
> New API main files:
> * rte_thread.h (librte_eal/include)
> * rte_thread_types.h (librte_eal/include)
> * rte_thread_windows_types.h (librte_eal/windows/include)
> * rte_thread.c (librte_eal/windows)
> * rte_thread.c (librte_eal/common)
>
> For flexibility, the user is offered the option of either using the RTE_THREAD_* API or
> a 3rd party thread library, through a meson flag “use_external_thread_lib”.
> By default, this flag is set to FALSE, which means Windows libraries and applications
> will use the RTE_THREAD_* API for managing threads.
>
> If compiling on Windows and the “use_external_thread_lib” is *not* set,
> the following files will be parsed:
> * include/rte_thread.h
> * windows/include/rte_thread_windows_types.h
> * windows/rte_thread.c
> In all other cases, the compilation/parsing includes the following files:
> * include/rte_thread.h
> * include/rte_thread_types.h
> * common/rte_thread.c
>
> **A schematic example of the design**
> --------------------------------------------------
> lib/librte_eal/include/rte_thread.h
> int rte_thread_create();
>
> lib/librte_eal/common/rte_thread.c
> int rte_thread_create()
> {
>         return pthread_create();
> }
>
> lib/librte_eal/windows/rte_thread.c
> int rte_thread_create()
> {
>         return CreateThread();
> }
>
> lib/librte_eal/windows/meson.build
> if get_option('use_external_thread_lib')
>         sources += 'librte_eal/common/rte_thread.c'
> else
>         sources += 'librte_eal/windows/rte_thread.c'
> endif
> -----------------------------------------------------
>
> **Thread attributes**
>
> When or after a thread is created, specific characteristics of the thread
> can be adjusted. Given that the thread characteristics that are of interest
> for DPDK applications are affinity and priority, the following structure
> that represents thread attributes has been defined:
>
> typedef struct
> {
>         enum rte_thread_priority priority;
>         rte_cpuset_t cpuset;
> } rte_thread_attr_t;
>
> The *rte_thread_create()* function can optionally receive an rte_thread_attr_t
> object that will cause the thread to be created with the affinity and priority
> described by the attributes object. If no rte_thread_attr_t is passed
> (parameter is NULL), the default affinity and priority are used.
> An rte_thread_attr_t object can also be set to the default values
> by calling *rte_thread_attr_init()*.
>
> *Priority* is represented through an enum that currently advertises
> two values for priority:
>         - RTE_THREAD_PRIORITY_NORMAL
>         - RTE_THREAD_PRIORITY_REALTIME_CRITICAL
> The enum can be extended to allow for multiple priority levels.
> rte_thread_set_priority      - sets the priority of a thread
> rte_thread_attr_set_priority - updates an rte_thread_attr_t object
>                                with a new value for priority
>
> The user can choose thread priority through an EAL parameter,
> when starting an application.  If EAL parameter is not used,
> the per-platform default value for thread priority is used.
> Otherwise administrator has an option to set one of available options:
>  --thread-prio normal
>  --thread-prio realtime
>
> Example:
> ./dpdk-l2fwd -l 0-3 -n 4 –thread-prio normal -- -q 8 -p ffff
>
> *Affinity* is described by the already known “rte_cpuset_t” type.
> rte_thread_attr_set/get_affinity - sets/gets the affinity field in a
>                                    rte_thread_attr_t object
> rte_thread_set/get_affinity      – sets/gets the affinity of a thread
>
> **Errors**
> A translation function that maps Windows error codes to errno-style
> error codes is provided.
>
> **Future work**
> Note that this patchset was focused on introducing new API that will
> remove the Windows pthread.h shim. In DPDK, there are still a few references
> to pthread_* that were not implemented in the shim.
> The long term plan is for EAL to provide full threading support:
> * Adding support for conditional variables
> * Additional functionality offered by pthread_* (such as pthread_setname_np, etc.)
> * Static mutex initializers are not used on Windows. If we must continue
>   using them, they need to be platform dependent and an implementation will
>   need to be provided for Windows.

Thanks for taking on this huge work.

There is no review on this series and I don't have time for it.
This is a core part of the EAL API.
Seeing how rc1 is at the end of this week, my recommendation is to
postpone to 21.08.
Narcisa Ana Maria Vasile April 16, 2021, 8:30 p.m. UTC | #2
On Mon, Apr 12, 2021 at 09:56:14AM +0200, David Marchand wrote:
> On Sat, Apr 3, 2021 at 3:39 AM Narcisa Ana Maria Vasile
> <navasile@linux.microsoft.com> wrote:
> >
> > From: Narcisa Vasile <navasile@microsoft.com>
> >
> > EAL thread API
> >
> Thanks for taking on this huge work.
> 
> There is no review on this series and I don't have time for it.
> This is a core part of the EAL API.
> Seeing how rc1 is at the end of this week, my recommendation is to
> postpone to 21.08.
> 
  Thanks David! I think we've seen a lot of great contributions recently
  and maintainers had a lot of patches to look at. The reviewers will probably
  have more time to analyze this in the next period.

> 
> -- 
> David Marchand