[v9] app/testpmd: support multi-process
Checks
Commit Message
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
---
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 14 ++++-
app/test-pmd/parameters.c | 12 ++++
app/test-pmd/testpmd.c | 108 +++++++++++++++++++++++----------
app/test-pmd/testpmd.h | 3 +
doc/guides/rel_notes/release_21_05.rst | 1 +
doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++++++
7 files changed, 196 insertions(+), 34 deletions(-)
Comments
On 4/16/2021 2:52 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
Hi Connor,
Thanks for the update. +Anatoly as multi-process maintainer, and ethdev maintainers.
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> ---
> v9:
> * Updated release notes and rst doc.
> * Deleted deprecated codes.
> * move macro and variable.
>
> v8:
> * Added warning info about queue numbers and process numbers.
>
> v7:
> * Fixed compiling error for unexpected unindent.
>
> v6:
> * Add rte flow description for multiple process.
>
> v5:
> * Fixed run_app.rst for multiple process description.
> * Fix compiling error.
>
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
> ---
> app/test-pmd/cmdline.c | 6 ++
> app/test-pmd/config.c | 14 ++++-
> app/test-pmd/parameters.c | 12 ++++
> app/test-pmd/testpmd.c | 108 +++++++++++++++++++++++----------
> app/test-pmd/testpmd.h | 3 +
> doc/guides/rel_notes/release_21_05.rst | 1 +
> doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++++++
> 7 files changed, 196 insertions(+), 34 deletions(-)
>
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> index 5bf1497..e465824 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
> __rte_unused void *data)
> {
> struct cmd_set_flush_rx *res = parsed_result;
> +
> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
> + printf("multi-process doesn't support to flush rx queues.\n");
> + return;
> + }
> +
> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
> }
>
> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
> index 40b2b29..c982c87 100644
> --- a/app/test-pmd/config.c
> +++ b/app/test-pmd/config.c
> @@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
> queueid_t rxq;
> queueid_t nb_q;
> streamid_t sm_id;
> + int start;
> + int end;
>
> nb_q = nb_rxq;
> if (nb_q > nb_txq)
> @@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
> init_fwd_streams();
>
> setup_fwd_config_of_each_lcore(&cur_fwd_config);
> - rxp = 0; rxq = 0;
> +
> + if (proc_id > 0 && nb_q % num_procs)
> + printf("Warning! queue numbers should be multiple of "
> + "processes, or packet loss will happen.\n");
> +
> + start = proc_id * nb_q / num_procs;
> + end = start + nb_q / num_procs;
> + rxp = 0;
> + rxq = start;
Can you put some comment above, on what/why is done, something similar to you
already put into the documentation?
> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
> struct fwd_stream *fs;
>
> @@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
> continue;
> rxp = 0;
> rxq++;
> + if (rxq >= end)
> + rxq = start;
> }
> }
>
> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
> index f3954c1..d86cc21 100644
> --- a/app/test-pmd/parameters.c
> +++ b/app/test-pmd/parameters.c
> @@ -508,6 +508,9 @@ parse_link_speed(int n)
> void
> launch_args_parse(int argc, char** argv)
> {
> +#define PARAM_PROC_ID "proc-id"
> +#define PARAM_NUM_PROCS "num-procs"
> +
> int n, opt;
> char **argvopt;
> int opt_idx;
> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
> { "rx-mq-mode", 1, 0, 0 },
> { "record-core-cycles", 0, 0, 0 },
> { "record-burst-stats", 0, 0, 0 },
> + { PARAM_NUM_PROCS, 1, 0, 0 },
> + { PARAM_PROC_ID, 1, 0, 0 },
> { 0, 0, 0, 0 },
> };
>
> @@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
> record_core_cycles = 1;
> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
> record_burst_stats = 1;
> +
To be consistent for rest, this empty line can be removed.
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_NUM_PROCS, 9) == 0)
> + num_procs = atoi(optarg);
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_PROC_ID, 7) == 0)
> + proc_id = atoi(optarg);
> break;
> case 'h':
> usage(argv[0]);
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 96d2e0f..01d0d82 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
> */
> uint32_t eth_link_speed;
>
> +/*
> + * Id of the current process.
> + */
Can you please add more info, like this being related to the primary/secondary
process support and used to configure the queues to be polled.
> +int proc_id;
> +
> +/*
> + * Number of processes.
> + */
> +unsigned int num_procs = 1;
> +
Ditto.
<...>
> @@ -2511,21 +2537,28 @@ start_port(portid_t pid)
> return -1;
> }
> /* configure port */
> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_configure(pi,
> + nb_rxq + nb_hairpinq,
> nb_txq + nb_hairpinq,
> &(port->dev_conf));
> - if (diag != 0) {
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> - printf("Fail to configure port %d\n", pi);
> - /* try to reconfigure port next time */
> - port->need_reconfig = 1;
> - return -1;
> + if (diag != 0) {
> + if (rte_atomic16_cmpset(
> + &(port->port_status),
> + RTE_PORT_HANDLING,
> + RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set "
> + "back to stopped\n", pi);
> + printf("Fail to configure port %d\n",
> + pi);
> + /* try to reconfigure port next time */
> + port->need_reconfig = 1;
> + return -1;
> + }
Just an idea,
I am a little worried about the complexity this new process type checks are
added to the multiple locations. What do you think hiding process type checks
behind new functions, like:
eth_dev_configure_mp(...) {
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
return rte_eth_dev_configure(..)
}
return 0;
}
and replace it with 'rte_eth_dev_configure()':
-diag = rte_eth_dev_configure(...)
+diag = eth_dev_configure_mp(...)
Do you think does this help reducing the complexity added by the multi-process
support?
> }
> }
> - if (port->need_reconfig_queues > 0) {
> + if (port->need_reconfig_queues > 0 &&
> + rte_eal_process_type() == RTE_PROC_PRIMARY) {
According our coding convention, we don't allign with paranthesis but put double
tabs.
And what about creating an inline function for primary process check, like
is_proc_primary(), that may help.
> port->need_reconfig_queues = 0;
> /* setup tx queues */
> for (qi = 0; qi < nb_txq; qi++) {
> @@ -2626,17 +2659,20 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - diag = rte_eth_dev_start(pi);
> - if (diag < 0) {
> - printf("Fail to start port %d: %s\n", pi,
> - rte_strerror(-diag));
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_start(pi);
> + if (diag < 0) {
> + printf("Fail to start port %d: %s\n", pi,
> + rte_strerror(-diag));
>
> - /* Fail to setup rx queue, return */
> - if (rte_atomic16_cmpset(&(port->port_status),
> + /* Fail to setup rx queue, return */
> + if (rte_atomic16_cmpset(&(port->port_status),
> RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
Syntax is wrong here after change, but if you go with above idea to hide process
type check within functions, you won't need to change these lines at all.
<...>
> @@ -3885,8 +3925,10 @@ main(int argc, char** argv)
> }
> }
>
> - if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
> + if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
> + pmd_test_exit();
Why need to add 'pmd_test_exit()' for the multi-process support, if there is a
special case, please add comment for it.
Hi, Ferrruh,
All is fixed in v10, please check it out, thanks.
在 2021/4/17 0:30, Ferruh Yigit 写道:
> On 4/16/2021 2:52 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>
> Hi Connor,
>
> Thanks for the update. +Anatoly as multi-process maintainer, and ethdev
> maintainers.
>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> ---
>> v9:
>> * Updated release notes and rst doc.
>> * Deleted deprecated codes.
>> * move macro and variable.
>>
>> v8:
>> * Added warning info about queue numbers and process numbers.
>>
>> v7:
>> * Fixed compiling error for unexpected unindent.
>>
>> v6:
>> * Add rte flow description for multiple process.
>>
>> v5:
>> * Fixed run_app.rst for multiple process description.
>> * Fix compiling error.
>>
>> v4:
>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>
>> v3:
>> * Fixed compiling error using gcc10.0.
>>
>> v2:
>> * Added document for this patch.
>> ---
>> app/test-pmd/cmdline.c | 6 ++
>> app/test-pmd/config.c | 14 ++++-
>> app/test-pmd/parameters.c | 12 ++++
>> app/test-pmd/testpmd.c | 108
>> +++++++++++++++++++++++----------
>> app/test-pmd/testpmd.h | 3 +
>> doc/guides/rel_notes/release_21_05.rst | 1 +
>> doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++++++
>> 7 files changed, 196 insertions(+), 34 deletions(-)
>>
>> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
>> index 5bf1497..e465824 100644
>> --- a/app/test-pmd/cmdline.c
>> +++ b/app/test-pmd/cmdline.c
>> @@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
>> __rte_unused void *data)
>> {
>> struct cmd_set_flush_rx *res = parsed_result;
>> +
>> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
>> + printf("multi-process doesn't support to flush rx queues.\n");
>> + return;
>> + }
>> +
>> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
>> }
>> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
>> index 40b2b29..c982c87 100644
>> --- a/app/test-pmd/config.c
>> +++ b/app/test-pmd/config.c
>> @@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
>> queueid_t rxq;
>> queueid_t nb_q;
>> streamid_t sm_id;
>> + int start;
>> + int end;
>> nb_q = nb_rxq;
>> if (nb_q > nb_txq)
>> @@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
>> init_fwd_streams();
>> setup_fwd_config_of_each_lcore(&cur_fwd_config);
>> - rxp = 0; rxq = 0;
>> +
>> + if (proc_id > 0 && nb_q % num_procs)
>> + printf("Warning! queue numbers should be multiple of "
>> + "processes, or packet loss will happen.\n");
>> +
>> + start = proc_id * nb_q / num_procs;
>> + end = start + nb_q / num_procs;
>> + rxp = 0;
>> + rxq = start;
>
> Can you put some comment above, on what/why is done, something similar
> to you already put into the documentation?
>
>> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
>> struct fwd_stream *fs;
>> @@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
>> continue;
>> rxp = 0;
>> rxq++;
>> + if (rxq >= end)
>> + rxq = start;
>> }
>> }
>> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
>> index f3954c1..d86cc21 100644
>> --- a/app/test-pmd/parameters.c
>> +++ b/app/test-pmd/parameters.c
>> @@ -508,6 +508,9 @@ parse_link_speed(int n)
>> void
>> launch_args_parse(int argc, char** argv)
>> {
>> +#define PARAM_PROC_ID "proc-id"
>> +#define PARAM_NUM_PROCS "num-procs"
>> +
>> int n, opt;
>> char **argvopt;
>> int opt_idx;
>> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
>> { "rx-mq-mode", 1, 0, 0 },
>> { "record-core-cycles", 0, 0, 0 },
>> { "record-burst-stats", 0, 0, 0 },
>> + { PARAM_NUM_PROCS, 1, 0, 0 },
>> + { PARAM_PROC_ID, 1, 0, 0 },
>> { 0, 0, 0, 0 },
>> };
>> @@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
>> record_core_cycles = 1;
>> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
>> record_burst_stats = 1;
>> +
>
> To be consistent for rest, this empty line can be removed.
>
>> + if (strncmp(lgopts[opt_idx].name,
>> + PARAM_NUM_PROCS, 9) == 0)
>> + num_procs = atoi(optarg);
>> + if (strncmp(lgopts[opt_idx].name,
>> + PARAM_PROC_ID, 7) == 0)
>> + proc_id = atoi(optarg);
>> break;
>> case 'h':
>> usage(argv[0]);
>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>> index 96d2e0f..01d0d82 100644
>> --- a/app/test-pmd/testpmd.c
>> +++ b/app/test-pmd/testpmd.c
>> @@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode =
>> ETH_MQ_RX_VMDQ_DCB_RSS;
>> */
>> uint32_t eth_link_speed;
>> +/*
>> + * Id of the current process.
>> + */
>
> Can you please add more info, like this being related to the
> primary/secondary process support and used to configure the queues to be
> polled.
>
>> +int proc_id;
>> +
>> +/*
>> + * Number of processes.
>> + */
>> +unsigned int num_procs = 1;
>> +
>
> Ditto.
>
> <...>
>
>> @@ -2511,21 +2537,28 @@ start_port(portid_t pid)
>> return -1;
>> }
>> /* configure port */
>> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
>> + diag = rte_eth_dev_configure(pi,
>> + nb_rxq + nb_hairpinq,
>> nb_txq + nb_hairpinq,
>> &(port->dev_conf));
>> - if (diag != 0) {
>> - if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back "
>> - "to stopped\n", pi);
>> - printf("Fail to configure port %d\n", pi);
>> - /* try to reconfigure port next time */
>> - port->need_reconfig = 1;
>> - return -1;
>> + if (diag != 0) {
>> + if (rte_atomic16_cmpset(
>> + &(port->port_status),
>> + RTE_PORT_HANDLING,
>> + RTE_PORT_STOPPED) == 0)
>> + printf("Port %d can not be set "
>> + "back to stopped\n", pi);
>> + printf("Fail to configure port %d\n",
>> + pi);
>> + /* try to reconfigure port next time */
>> + port->need_reconfig = 1;
>> + return -1;
>> + }
>
> Just an idea,
> I am a little worried about the complexity this new process type checks
> are added to the multiple locations. What do you think hiding process
> type checks behind new functions, like:
> eth_dev_configure_mp(...) {
> if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> return rte_eth_dev_configure(..)
> }
> return 0;
> }
>
> and replace it with 'rte_eth_dev_configure()':
>
> -diag = rte_eth_dev_configure(...)
> +diag = eth_dev_configure_mp(...)
>
> Do you think does this help reducing the complexity added by the
> multi-process support?
>
>> }
>> }
>> - if (port->need_reconfig_queues > 0) {
>> + if (port->need_reconfig_queues > 0 &&
>> + rte_eal_process_type() == RTE_PROC_PRIMARY) {
>
> According our coding convention, we don't allign with paranthesis but
> put double tabs.
>
> And what about creating an inline function for primary process check,
> like is_proc_primary(), that may help.
>
>> port->need_reconfig_queues = 0;
>> /* setup tx queues */
>> for (qi = 0; qi < nb_txq; qi++) {
>> @@ -2626,17 +2659,20 @@ start_port(portid_t pid)
>> cnt_pi++;
>> /* start port */
>> - diag = rte_eth_dev_start(pi);
>> - if (diag < 0) {
>> - printf("Fail to start port %d: %s\n", pi,
>> - rte_strerror(-diag));
>> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
>> + diag = rte_eth_dev_start(pi);
>> + if (diag < 0) {
>> + printf("Fail to start port %d: %s\n", pi,
>> + rte_strerror(-diag));
>> - /* Fail to setup rx queue, return */
>> - if (rte_atomic16_cmpset(&(port->port_status),
>> + /* Fail to setup rx queue, return */
>> + if (rte_atomic16_cmpset(&(port->port_status),
>> RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>
> Syntax is wrong here after change, but if you go with above idea to hide
> process type check within functions, you won't need to change these
> lines at all.
>
> <...>
>
>> @@ -3885,8 +3925,10 @@ main(int argc, char** argv)
>> }
>> }
>> - if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
>> + if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
>> + pmd_test_exit();
>
> Why need to add 'pmd_test_exit()' for the multi-process support, if
> there is a special case, please add comment for it.
>
>
> .
@@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
@@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process.
+ */
+int proc_id;
+
+/*
+ * Number of processes.
+ */
+unsigned int num_procs = 1;
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +987,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1074,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2022,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2537,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,17 +2659,20 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
- if (diag < 0) {
- printf("Fail to start port %d: %s\n", pi,
- rte_strerror(-diag));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d: %s\n", pi,
+ rte_strerror(-diag));
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
- continue;
+ printf("Port %d can not be set back to "
+ "stopped\n",
+ pi);
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2765,7 +2801,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2871,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3099,7 +3138,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3430,7 +3469,8 @@ update_jumbo_frame_offload(portid_t portid)
/* If JUMBO_FRAME is set MTU conversion done by ethdev layer,
* if unset do it here
*/
- if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
+ if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
ret = rte_eth_dev_set_mtu(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
@@ -3621,6 +3661,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3786,10 +3830,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3885,8 +3925,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -630,6 +630,9 @@ extern struct mplsoudp_decap_conf mplsoudp_decap_conf;
extern enum rte_eth_rx_mq_mode rx_mq_mode;
+extern int proc_id;
+extern unsigned int num_procs;
+
static inline unsigned int
lcore_num(void)
{
@@ -185,6 +185,7 @@ New Features
``show port (port_id) rxq (queue_id) desc used count``
* Added command to dump internal representation information of single flow.
``flow dump (port_id) rule (rule_id)``
+ * Added support multi-process for testpmd.
Removed Items
@@ -551,3 +551,89 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.