eal: restrict ctrl threads to startup cpu affinity
Checks
Commit Message
Spawning the ctrl threads on anything that is not part of the eal
coremask is not that polite to the rest of the system.
Rather than introduce yet another eal options for this, let's take
the startup cpu affinity as a reference and remove the eal coremask
from it.
If no cpu is left, then we default to the master core.
The cpuset is computed once at init before the original cpu affinity.
Fixes: d651ee4919cd ("eal: set affinity for control threads")
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/librte_eal/common/eal_common_options.c | 28 ++++++++++++++++++++++++++++
lib/librte_eal/common/eal_common_thread.c | 21 ++++-----------------
lib/librte_eal/common/eal_internal_cfg.h | 3 +++
3 files changed, 35 insertions(+), 17 deletions(-)
Comments
On Wed, Feb 13, 2019 at 5:14 PM David Marchand <david.marchand@redhat.com>
wrote:
> Spawning the ctrl threads on anything that is not part of the eal
> coremask is not that polite to the rest of the system.
>
> Rather than introduce yet another eal options for this, let's take
> the startup cpu affinity as a reference and remove the eal coremask
> from it.
> If no cpu is left, then we default to the master core.
>
> The cpuset is computed once at init before the original cpu affinity.
>
Need to fix this last sentence...
> Fixes: d651ee4919cd ("eal: set affinity for control threads")
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/librte_eal/common/eal_common_options.c | 28
> ++++++++++++++++++++++++++++
> lib/librte_eal/common/eal_common_thread.c | 21 ++++-----------------
> lib/librte_eal/common/eal_internal_cfg.h | 3 +++
> 3 files changed, 35 insertions(+), 17 deletions(-)
>
> diff --git a/lib/librte_eal/common/eal_common_options.c
> b/lib/librte_eal/common/eal_common_options.c
> index 6c96f45..b766252 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -1360,6 +1361,31 @@ static int xdigit2val(unsigned char c)
> cfg->lcore_count -= removed;
> }
>
> +static void
> +compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
> +{
> + rte_cpuset_t *cpuset = &internal_cfg->ctrl_cpuset;
> + rte_cpuset_t default_set;
> + unsigned int lcore_id;
> +
> + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
> + if (eal_cpu_detected(lcore_id) &&
> + rte_lcore_has_role(lcore_id, ROLE_OFF)) {
> + CPU_SET(lcore_id, cpuset);
> + }
> + }
> +
> + if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
> + &default_set) < 0)
> + CPU_ZERO(&default_set);
> +
> + CPU_AND(cpuset, cpuset, &default_set);
>
CPU_AND is different on Freebsd.
*CPU*_*AND*(*cpuset*_*t* **dst*, *cpuset*_*t* **src*);
The *CPU*_*AND*() macro removes CPUs absent from *src* from
*dst*. (It is the
*cpuset(9)*
<https://www.freebsd.org/cgi/man.cgi?query=cpuset&sektion=9&apropos=0&manpath=FreeBSD+11.0-RELEASE+and+Ports>
equivalent of the scalar: *dst* &= *src*.) *CPU*_*AND*_*ATOMIC*() is
similar, with the same atomic semantics as *CPU*_*OR*_*ATOMIC*().
Will fix in v2.
On 13-Feb-19 4:13 PM, David Marchand wrote:
> Spawning the ctrl threads on anything that is not part of the eal
> coremask is not that polite to the rest of the system.
>
> Rather than introduce yet another eal options for this, let's take
> the startup cpu affinity as a reference and remove the eal coremask
> from it.
> If no cpu is left, then we default to the master core.
>
> The cpuset is computed once at init before the original cpu affinity.
>
> Fixes: d651ee4919cd ("eal: set affinity for control threads")
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Hi David,
Maybe i didn't have enough coffee today and i'm missing something here,
but how is this different? Removing the coremask cores from the cpuset
will effectively "spawn the ctrl threads on anything that is not part of
the EAL coremask" (which is "not that polite to the rest of the
system"), unless the application was run with taskset.
Is "taskset" the key point here? I.e. by default, we're still "not
polite", unless the user asks nicely? :)
On Thu, Feb 14, 2019 at 10:39 AM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
> On 13-Feb-19 4:13 PM, David Marchand wrote:
> > Spawning the ctrl threads on anything that is not part of the eal
> > coremask is not that polite to the rest of the system.
> >
> > Rather than introduce yet another eal options for this, let's take
> > the startup cpu affinity as a reference and remove the eal coremask
> > from it.
> > If no cpu is left, then we default to the master core.
> >
> > The cpuset is computed once at init before the original cpu affinity.
> >
> > Fixes: d651ee4919cd ("eal: set affinity for control threads")
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> Hi David,
>
> Maybe i didn't have enough coffee today and i'm missing something here,
> but how is this different? Removing the coremask cores from the cpuset
> will effectively "spawn the ctrl threads on anything that is not part of
> the EAL coremask" (which is "not that polite to the rest of the
> system"), unless the application was run with taskset.
>
> Is "taskset" the key point here? I.e. by default, we're still "not
> polite", unless the user asks nicely? :)
>
Eheh, sorry, yes.
A bit more context then, if you want to clearly pin cpu resources for the
processes on your system (let's say having virtual machines and a popular
vswitch), I can only think of two solutions.
Either you have something to configure your processes to have them call
sched_setaffinity/pthread_set_affinity_np, or you use taskset to get them
"jailed" without them caring.
Before the incriminated commit, we were keeping all threads on the coremask
that had been passed, but as Olivier said, we would end up with ctrl
threads spanwed on core running dataplane threads as well.
Now, the ctrl threads can be spawned anywhere on all & ~coremask, with no
way to configure this.
I considered adding a new eal option, but I think relying on the current
cpu affinity is a better default behavior and I can't see drawbacks at the
moment.
On 14-Feb-19 9:53 AM, David Marchand wrote:
> On Thu, Feb 14, 2019 at 10:39 AM Burakov, Anatoly
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>
> On 13-Feb-19 4:13 PM, David Marchand wrote:
> > Spawning the ctrl threads on anything that is not part of the eal
> > coremask is not that polite to the rest of the system.
> >
> > Rather than introduce yet another eal options for this, let's take
> > the startup cpu affinity as a reference and remove the eal coremask
> > from it.
> > If no cpu is left, then we default to the master core.
> >
> > The cpuset is computed once at init before the original cpu affinity.
> >
> > Fixes: d651ee4919cd ("eal: set affinity for control threads")
> > Signed-off-by: David Marchand <david.marchand@redhat.com
> <mailto:david.marchand@redhat.com>>
> > ---
>
> Hi David,
>
> Maybe i didn't have enough coffee today and i'm missing something here,
> but how is this different? Removing the coremask cores from the cpuset
> will effectively "spawn the ctrl threads on anything that is not
> part of
> the EAL coremask" (which is "not that polite to the rest of the
> system"), unless the application was run with taskset.
>
> Is "taskset" the key point here? I.e. by default, we're still "not
> polite", unless the user asks nicely? :)
>
>
> Eheh, sorry, yes.
> A bit more context then, if you want to clearly pin cpu resources for
> the processes on your system (let's say having virtual machines and a
> popular vswitch), I can only think of two solutions.
> Either you have something to configure your processes to have them call
> sched_setaffinity/pthread_set_affinity_np, or you use taskset to get
> them "jailed" without them caring.
>
> Before the incriminated commit, we were keeping all threads on the
> coremask that had been passed, but as Olivier said, we would end up with
> ctrl threads spanwed on core running dataplane threads as well.
>
> Now, the ctrl threads can be spawned anywhere on all & ~coremask, with
> no way to configure this.
> I considered adding a new eal option, but I think relying on the current
> cpu affinity is a better default behavior and I can't see drawbacks at
> the moment.
>
>
> --
> David Marchand
>
OK, that makes sense. However, i feel this behavior (both old and new,
for that matter) should be better documented somewhere in the EAL docs.
On Thu, Feb 14, 2019 at 11:05 AM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:
> On 14-Feb-19 9:53 AM, David Marchand wrote:
>
> > A bit more context then, if you want to clearly pin cpu resources for
> > the processes on your system (let's say having virtual machines and a
> > popular vswitch), I can only think of two solutions.
> > Either you have something to configure your processes to have them call
> > sched_setaffinity/pthread_set_affinity_np, or you use taskset to get
> > them "jailed" without them caring.
> >
> > Before the incriminated commit, we were keeping all threads on the
> > coremask that had been passed, but as Olivier said, we would end up with
> > ctrl threads spanwed on core running dataplane threads as well.
> >
> > Now, the ctrl threads can be spawned anywhere on all & ~coremask, with
> > no way to configure this.
> > I considered adding a new eal option, but I think relying on the current
> > cpu affinity is a better default behavior and I can't see drawbacks at
> > the moment.
>
> OK, that makes sense. However, i feel this behavior (both old and new,
> for that matter) should be better documented somewhere in the EAL docs.
>
>
I'll see what I can add in the doc for v2.
On Wed, Feb 13, 2019 at 5:14 PM David Marchand <david.marchand@redhat.com>
wrote:
> Spawning the ctrl threads on anything that is not part of the eal
> coremask is not that polite to the rest of the system.
>
> Rather than introduce yet another eal options for this, let's take
> the startup cpu affinity as a reference and remove the eal coremask
> from it.
> If no cpu is left, then we default to the master core.
>
> The cpuset is computed once at init before the original cpu affinity.
>
> Fixes: d651ee4919cd ("eal: set affinity for control threads")
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/librte_eal/common/eal_common_options.c | 28
> ++++++++++++++++++++++++++++
> lib/librte_eal/common/eal_common_thread.c | 21 ++++-----------------
> lib/librte_eal/common/eal_internal_cfg.h | 3 +++
> 3 files changed, 35 insertions(+), 17 deletions(-)
>
> diff --git a/lib/librte_eal/common/eal_common_options.c
> b/lib/librte_eal/common/eal_common_options.c
> index 6c96f45..b766252 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -1360,6 +1361,31 @@ static int xdigit2val(unsigned char c)
> cfg->lcore_count -= removed;
> }
>
> +static void
> +compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
> +{
> + rte_cpuset_t *cpuset = &internal_cfg->ctrl_cpuset;
> + rte_cpuset_t default_set;
> + unsigned int lcore_id;
> +
> + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
> + if (eal_cpu_detected(lcore_id) &&
> + rte_lcore_has_role(lcore_id, ROLE_OFF)) {
> + CPU_SET(lcore_id, cpuset);
> + }
> + }
> +
> + if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
> + &default_set) < 0)
> + CPU_ZERO(&default_set);
>
Now that I am testing on Freebsd, I can see this block I took from the
existing "auto detect" function is just wrong.
pthread_(g|s)etaffinity_np return a > 0 error value when failing.
@@ -217,6 +217,7 @@ struct device_option {
internal_cfg->create_uio_dev = 0;
internal_cfg->iova_mode = RTE_IOVA_DC;
internal_cfg->user_mbuf_pool_ops_name = NULL;
+ CPU_ZERO(&internal_cfg->ctrl_cpuset);
internal_cfg->init_complete = 0;
}
@@ -1360,6 +1361,31 @@ static int xdigit2val(unsigned char c)
cfg->lcore_count -= removed;
}
+static void
+compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
+{
+ rte_cpuset_t *cpuset = &internal_cfg->ctrl_cpuset;
+ rte_cpuset_t default_set;
+ unsigned int lcore_id;
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (eal_cpu_detected(lcore_id) &&
+ rte_lcore_has_role(lcore_id, ROLE_OFF)) {
+ CPU_SET(lcore_id, cpuset);
+ }
+ }
+
+ if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
+ &default_set) < 0)
+ CPU_ZERO(&default_set);
+
+ CPU_AND(cpuset, cpuset, &default_set);
+
+ /* if no detected cpu is off, use master core */
+ if (!CPU_COUNT(cpuset))
+ CPU_SET(rte_get_master_lcore(), cpuset);
+}
+
int
eal_cleanup_config(struct internal_config *internal_cfg)
{
@@ -1393,6 +1419,8 @@ static int xdigit2val(unsigned char c)
lcore_config[cfg->master_lcore].core_role = ROLE_RTE;
}
+ compute_ctrl_threads_cpuset(internal_cfg);
+
/* if no memory amounts were requested, this will result in 0 and
* will be overridden later, right after eal_hugepage_info_init() */
for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
@@ -16,6 +16,7 @@
#include <rte_memory.h>
#include <rte_log.h>
+#include "eal_internal_cfg.h"
#include "eal_private.h"
#include "eal_thread.h"
@@ -168,10 +169,9 @@ static void *rte_thread_init(void *arg)
const pthread_attr_t *attr,
void *(*start_routine)(void *), void *arg)
{
+ rte_cpuset_t *cpuset = &internal_config.ctrl_cpuset;
struct rte_thread_ctrl_params *params;
- unsigned int lcore_id;
- rte_cpuset_t cpuset;
- int cpu_found, ret;
+ int ret;
params = malloc(sizeof(*params));
if (!params)
@@ -195,20 +195,7 @@ static void *rte_thread_init(void *arg)
"Cannot set name for ctrl thread\n");
}
- cpu_found = 0;
- CPU_ZERO(&cpuset);
- for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
- if (eal_cpu_detected(lcore_id) &&
- rte_lcore_has_role(lcore_id, ROLE_OFF)) {
- CPU_SET(lcore_id, &cpuset);
- cpu_found = 1;
- }
- }
- /* if no detected cpu is off, use master core */
- if (!cpu_found)
- CPU_SET(rte_get_master_lcore(), &cpuset);
-
- ret = pthread_setaffinity_np(*thread, sizeof(cpuset), &cpuset);
+ ret = pthread_setaffinity_np(*thread, sizeof(*cpuset), cpuset);
if (ret < 0)
goto fail;
@@ -13,6 +13,8 @@
#include <rte_eal.h>
#include <rte_pci_dev_feature_defs.h>
+#include "eal_thread.h"
+
#define MAX_HUGEPAGE_SIZES 3 /**< support up to 3 page sizes */
/*
@@ -73,6 +75,7 @@ struct internal_config {
unsigned num_hugepage_sizes; /**< how many sizes on this system */
struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES];
enum rte_iova_mode iova_mode ; /**< Set IOVA mode on this system */
+ rte_cpuset_t ctrl_cpuset; /**< cpuset for ctrl threads */
volatile unsigned int init_complete;
/**< indicates whether EAL has completed initialization */
};