diff mbox series

[v9] app/testpmd: support multi-process

Message ID 1618537978-35479-1-git-send-email-humin29@huawei.com (mailing list archive)
State Superseded
Delegated to: Ferruh Yigit
Headers show
Series [v9] app/testpmd: support multi-process | expand

Checks

Context Check Description
ci/iol-testing success Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/intel-Testing success Testing PASS
ci/Intel-compilation success Compilation OK
ci/checkpatch success coding style OK

Commit Message

Min Hu (Connor) April 16, 2021, 1:52 a.m. UTC
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0

the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1

Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
---
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.

v8:
* Added warning info about queue numbers and process numbers.

v7:
* Fixed compiling error for unexpected unindent.

v6:
* Add rte flow description for multiple process.

v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.

v4:
* Fixed minimum vlaue of Rxq or Txq in doc.

v3:
* Fixed compiling error using gcc10.0.

v2:
* Added document for this patch.
---
 app/test-pmd/cmdline.c                 |   6 ++
 app/test-pmd/config.c                  |  14 ++++-
 app/test-pmd/parameters.c              |  12 ++++
 app/test-pmd/testpmd.c                 | 108 +++++++++++++++++++++++----------
 app/test-pmd/testpmd.h                 |   3 +
 doc/guides/rel_notes/release_21_05.rst |   1 +
 doc/guides/testpmd_app_ug/run_app.rst  |  86 ++++++++++++++++++++++++++
 7 files changed, 196 insertions(+), 34 deletions(-)

Comments

Ferruh Yigit April 16, 2021, 4:30 p.m. UTC | #1
On 4/16/2021 2:52 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
> 
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
> 

Hi Connor,

Thanks for the update. +Anatoly as multi-process maintainer, and ethdev maintainers.

> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> ---
> v9:
> * Updated release notes and rst doc.
> * Deleted deprecated codes.
> * move macro and variable.
> 
> v8:
> * Added warning info about queue numbers and process numbers.
> 
> v7:
> * Fixed compiling error for unexpected unindent.
> 
> v6:
> * Add rte flow description for multiple process.
> 
> v5:
> * Fixed run_app.rst for multiple process description.
> * Fix compiling error.
> 
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
> 
> v3:
> * Fixed compiling error using gcc10.0.
> 
> v2:
> * Added document for this patch.
> ---
>   app/test-pmd/cmdline.c                 |   6 ++
>   app/test-pmd/config.c                  |  14 ++++-
>   app/test-pmd/parameters.c              |  12 ++++
>   app/test-pmd/testpmd.c                 | 108 +++++++++++++++++++++++----------
>   app/test-pmd/testpmd.h                 |   3 +
>   doc/guides/rel_notes/release_21_05.rst |   1 +
>   doc/guides/testpmd_app_ug/run_app.rst  |  86 ++++++++++++++++++++++++++
>   7 files changed, 196 insertions(+), 34 deletions(-)
> 
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> index 5bf1497..e465824 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
>   		__rte_unused void *data)
>   {
>   	struct cmd_set_flush_rx *res = parsed_result;
> +
> +	if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
> +		printf("multi-process doesn't support to flush rx queues.\n");
> +		return;
> +	}
> +
>   	no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
>   }
>   
> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
> index 40b2b29..c982c87 100644
> --- a/app/test-pmd/config.c
> +++ b/app/test-pmd/config.c
> @@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
>   	queueid_t  rxq;
>   	queueid_t  nb_q;
>   	streamid_t  sm_id;
> +	int start;
> +	int end;
>   
>   	nb_q = nb_rxq;
>   	if (nb_q > nb_txq)
> @@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
>   	init_fwd_streams();
>   
>   	setup_fwd_config_of_each_lcore(&cur_fwd_config);
> -	rxp = 0; rxq = 0;
> +
> +	if (proc_id > 0 && nb_q % num_procs)
> +		printf("Warning! queue numbers should be multiple of "
> +			"processes, or packet loss will happen.\n");
> +
> +	start = proc_id * nb_q / num_procs;
> +	end = start + nb_q / num_procs;
> +	rxp = 0;
> +	rxq = start;

Can you put some comment above, on what/why is done, something similar to you 
already put into the documentation?

>   	for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
>   		struct fwd_stream *fs;
>   
> @@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
>   			continue;
>   		rxp = 0;
>   		rxq++;
> +		if (rxq >= end)
> +			rxq = start;
>   	}
>   }
>   
> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
> index f3954c1..d86cc21 100644
> --- a/app/test-pmd/parameters.c
> +++ b/app/test-pmd/parameters.c
> @@ -508,6 +508,9 @@ parse_link_speed(int n)
>   void
>   launch_args_parse(int argc, char** argv)
>   {
> +#define PARAM_PROC_ID "proc-id"
> +#define PARAM_NUM_PROCS "num-procs"
> +
>   	int n, opt;
>   	char **argvopt;
>   	int opt_idx;
> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
>   		{ "rx-mq-mode",                 1, 0, 0 },
>   		{ "record-core-cycles",         0, 0, 0 },
>   		{ "record-burst-stats",         0, 0, 0 },
> +		{ PARAM_NUM_PROCS,              1, 0, 0 },
> +		{ PARAM_PROC_ID,                1, 0, 0 },
>   		{ 0, 0, 0, 0 },
>   	};
>   
> @@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
>   				record_core_cycles = 1;
>   			if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
>   				record_burst_stats = 1;
> +

To be consistent for rest, this empty line can be removed.

> +			if (strncmp(lgopts[opt_idx].name,
> +				    PARAM_NUM_PROCS, 9) == 0)
> +				num_procs = atoi(optarg);
> +			if (strncmp(lgopts[opt_idx].name,
> +				    PARAM_PROC_ID, 7) == 0)
> +				proc_id = atoi(optarg);
>   			break;
>   		case 'h':
>   			usage(argv[0]);
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 96d2e0f..01d0d82 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
>    */
>   uint32_t eth_link_speed;
>   
> +/*
> + * Id of the current process.
> + */

Can you please add more info, like this being related to the primary/secondary 
process support and used to configure the queues to be polled.

> +int proc_id;
> +
> +/*
> + * Number of processes.
> + */
> +unsigned int num_procs = 1;
> +

Ditto.

<...>

> @@ -2511,21 +2537,28 @@ start_port(portid_t pid)
>   				return -1;
>   			}
>   			/* configure port */
> -			diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
> +			if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +				diag = rte_eth_dev_configure(pi,
> +						     nb_rxq + nb_hairpinq,
>   						     nb_txq + nb_hairpinq,
>   						     &(port->dev_conf));
> -			if (diag != 0) {
> -				if (rte_atomic16_cmpset(&(port->port_status),
> -				RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> -					printf("Port %d can not be set back "
> -							"to stopped\n", pi);
> -				printf("Fail to configure port %d\n", pi);
> -				/* try to reconfigure port next time */
> -				port->need_reconfig = 1;
> -				return -1;
> +				if (diag != 0) {
> +					if (rte_atomic16_cmpset(
> +							&(port->port_status),
> +							RTE_PORT_HANDLING,
> +							RTE_PORT_STOPPED) == 0)
> +						printf("Port %d can not be set "
> +						       "back to stopped\n", pi);
> +					printf("Fail to configure port %d\n",
> +						pi);
> +					/* try to reconfigure port next time */
> +					port->need_reconfig = 1;
> +					return -1;
> +				}

Just an idea,
I am a little worried about the complexity this new process type checks are 
added to the multiple locations. What do you think hiding process type checks 
behind new functions, like:
eth_dev_configure_mp(...) {
   if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
     return rte_eth_dev_configure(..)
   }
   return 0;
}

and replace it with 'rte_eth_dev_configure()':

  -diag = rte_eth_dev_configure(...)
  +diag = eth_dev_configure_mp(...)

Do you think does this help reducing the complexity added by the multi-process 
support?

>   			}
>   		}
> -		if (port->need_reconfig_queues > 0) {
> +		if (port->need_reconfig_queues > 0 &&
> +		    rte_eal_process_type() == RTE_PROC_PRIMARY) {

According our coding convention, we don't allign with paranthesis but put double 
tabs.

And what about creating an inline function for primary process check, like 
is_proc_primary(), that may help.

>   			port->need_reconfig_queues = 0;
>   			/* setup tx queues */
>   			for (qi = 0; qi < nb_txq; qi++) {
> @@ -2626,17 +2659,20 @@ start_port(portid_t pid)
>   		cnt_pi++;
>   
>   		/* start port */
> -		diag = rte_eth_dev_start(pi);
> -		if (diag < 0) {
> -			printf("Fail to start port %d: %s\n", pi,
> -			       rte_strerror(-diag));
> +		if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +			diag = rte_eth_dev_start(pi);
> +			if (diag < 0) {
> +				printf("Fail to start port %d: %s\n", pi,
> +				       rte_strerror(-diag));
>   
> -			/* Fail to setup rx queue, return */
> -			if (rte_atomic16_cmpset(&(port->port_status),
> +				/* Fail to setup rx queue, return */
> +				if (rte_atomic16_cmpset(&(port->port_status),
>   				RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)

Syntax is wrong here after change, but if you go with above idea to hide process 
type check within functions, you won't need to change these lines at all.

<...>

> @@ -3885,8 +3925,10 @@ main(int argc, char** argv)
>   		}
>   	}
>   
> -	if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
> +	if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
> +		pmd_test_exit();

Why need to add 'pmd_test_exit()' for the multi-process support, if there is a 
special case, please add comment for it.
Min Hu (Connor) April 17, 2021, 6:12 a.m. UTC | #2
Hi, Ferrruh,
	All is fixed in v10, please check it out, thanks.

在 2021/4/17 0:30, Ferruh Yigit 写道:
> On 4/16/2021 2:52 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
> 
> Hi Connor,
> 
> Thanks for the update. +Anatoly as multi-process maintainer, and ethdev 
> maintainers.
> 
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> ---
>> v9:
>> * Updated release notes and rst doc.
>> * Deleted deprecated codes.
>> * move macro and variable.
>>
>> v8:
>> * Added warning info about queue numbers and process numbers.
>>
>> v7:
>> * Fixed compiling error for unexpected unindent.
>>
>> v6:
>> * Add rte flow description for multiple process.
>>
>> v5:
>> * Fixed run_app.rst for multiple process description.
>> * Fix compiling error.
>>
>> v4:
>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>
>> v3:
>> * Fixed compiling error using gcc10.0.
>>
>> v2:
>> * Added document for this patch.
>> ---
>>   app/test-pmd/cmdline.c                 |   6 ++
>>   app/test-pmd/config.c                  |  14 ++++-
>>   app/test-pmd/parameters.c              |  12 ++++
>>   app/test-pmd/testpmd.c                 | 108 
>> +++++++++++++++++++++++----------
>>   app/test-pmd/testpmd.h                 |   3 +
>>   doc/guides/rel_notes/release_21_05.rst |   1 +
>>   doc/guides/testpmd_app_ug/run_app.rst  |  86 ++++++++++++++++++++++++++
>>   7 files changed, 196 insertions(+), 34 deletions(-)
>>
>> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
>> index 5bf1497..e465824 100644
>> --- a/app/test-pmd/cmdline.c
>> +++ b/app/test-pmd/cmdline.c
>> @@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
>>           __rte_unused void *data)
>>   {
>>       struct cmd_set_flush_rx *res = parsed_result;
>> +
>> +    if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
>> +        printf("multi-process doesn't support to flush rx queues.\n");
>> +        return;
>> +    }
>> +
>>       no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
>>   }
>> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
>> index 40b2b29..c982c87 100644
>> --- a/app/test-pmd/config.c
>> +++ b/app/test-pmd/config.c
>> @@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
>>       queueid_t  rxq;
>>       queueid_t  nb_q;
>>       streamid_t  sm_id;
>> +    int start;
>> +    int end;
>>       nb_q = nb_rxq;
>>       if (nb_q > nb_txq)
>> @@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
>>       init_fwd_streams();
>>       setup_fwd_config_of_each_lcore(&cur_fwd_config);
>> -    rxp = 0; rxq = 0;
>> +
>> +    if (proc_id > 0 && nb_q % num_procs)
>> +        printf("Warning! queue numbers should be multiple of "
>> +            "processes, or packet loss will happen.\n");
>> +
>> +    start = proc_id * nb_q / num_procs;
>> +    end = start + nb_q / num_procs;
>> +    rxp = 0;
>> +    rxq = start;
> 
> Can you put some comment above, on what/why is done, something similar 
> to you already put into the documentation?
> 
>>       for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
>>           struct fwd_stream *fs;
>> @@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
>>               continue;
>>           rxp = 0;
>>           rxq++;
>> +        if (rxq >= end)
>> +            rxq = start;
>>       }
>>   }
>> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
>> index f3954c1..d86cc21 100644
>> --- a/app/test-pmd/parameters.c
>> +++ b/app/test-pmd/parameters.c
>> @@ -508,6 +508,9 @@ parse_link_speed(int n)
>>   void
>>   launch_args_parse(int argc, char** argv)
>>   {
>> +#define PARAM_PROC_ID "proc-id"
>> +#define PARAM_NUM_PROCS "num-procs"
>> +
>>       int n, opt;
>>       char **argvopt;
>>       int opt_idx;
>> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
>>           { "rx-mq-mode",                 1, 0, 0 },
>>           { "record-core-cycles",         0, 0, 0 },
>>           { "record-burst-stats",         0, 0, 0 },
>> +        { PARAM_NUM_PROCS,              1, 0, 0 },
>> +        { PARAM_PROC_ID,                1, 0, 0 },
>>           { 0, 0, 0, 0 },
>>       };
>> @@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
>>                   record_core_cycles = 1;
>>               if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
>>                   record_burst_stats = 1;
>> +
> 
> To be consistent for rest, this empty line can be removed.
> 
>> +            if (strncmp(lgopts[opt_idx].name,
>> +                    PARAM_NUM_PROCS, 9) == 0)
>> +                num_procs = atoi(optarg);
>> +            if (strncmp(lgopts[opt_idx].name,
>> +                    PARAM_PROC_ID, 7) == 0)
>> +                proc_id = atoi(optarg);
>>               break;
>>           case 'h':
>>               usage(argv[0]);
>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>> index 96d2e0f..01d0d82 100644
>> --- a/app/test-pmd/testpmd.c
>> +++ b/app/test-pmd/testpmd.c
>> @@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode = 
>> ETH_MQ_RX_VMDQ_DCB_RSS;
>>    */
>>   uint32_t eth_link_speed;
>> +/*
>> + * Id of the current process.
>> + */
> 
> Can you please add more info, like this being related to the 
> primary/secondary process support and used to configure the queues to be 
> polled.
> 
>> +int proc_id;
>> +
>> +/*
>> + * Number of processes.
>> + */
>> +unsigned int num_procs = 1;
>> +
> 
> Ditto.
> 
> <...>
> 
>> @@ -2511,21 +2537,28 @@ start_port(portid_t pid)
>>                   return -1;
>>               }
>>               /* configure port */
>> -            diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>> +            if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
>> +                diag = rte_eth_dev_configure(pi,
>> +                             nb_rxq + nb_hairpinq,
>>                                nb_txq + nb_hairpinq,
>>                                &(port->dev_conf));
>> -            if (diag != 0) {
>> -                if (rte_atomic16_cmpset(&(port->port_status),
>> -                RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> -                    printf("Port %d can not be set back "
>> -                            "to stopped\n", pi);
>> -                printf("Fail to configure port %d\n", pi);
>> -                /* try to reconfigure port next time */
>> -                port->need_reconfig = 1;
>> -                return -1;
>> +                if (diag != 0) {
>> +                    if (rte_atomic16_cmpset(
>> +                            &(port->port_status),
>> +                            RTE_PORT_HANDLING,
>> +                            RTE_PORT_STOPPED) == 0)
>> +                        printf("Port %d can not be set "
>> +                               "back to stopped\n", pi);
>> +                    printf("Fail to configure port %d\n",
>> +                        pi);
>> +                    /* try to reconfigure port next time */
>> +                    port->need_reconfig = 1;
>> +                    return -1;
>> +                }
> 
> Just an idea,
> I am a little worried about the complexity this new process type checks 
> are added to the multiple locations. What do you think hiding process 
> type checks behind new functions, like:
> eth_dev_configure_mp(...) {
>    if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
>      return rte_eth_dev_configure(..)
>    }
>    return 0;
> }
> 
> and replace it with 'rte_eth_dev_configure()':
> 
>   -diag = rte_eth_dev_configure(...)
>   +diag = eth_dev_configure_mp(...)
> 
> Do you think does this help reducing the complexity added by the 
> multi-process support?
> 
>>               }
>>           }
>> -        if (port->need_reconfig_queues > 0) {
>> +        if (port->need_reconfig_queues > 0 &&
>> +            rte_eal_process_type() == RTE_PROC_PRIMARY) {
> 
> According our coding convention, we don't allign with paranthesis but 
> put double tabs.
> 
> And what about creating an inline function for primary process check, 
> like is_proc_primary(), that may help.
> 
>>               port->need_reconfig_queues = 0;
>>               /* setup tx queues */
>>               for (qi = 0; qi < nb_txq; qi++) {
>> @@ -2626,17 +2659,20 @@ start_port(portid_t pid)
>>           cnt_pi++;
>>           /* start port */
>> -        diag = rte_eth_dev_start(pi);
>> -        if (diag < 0) {
>> -            printf("Fail to start port %d: %s\n", pi,
>> -                   rte_strerror(-diag));
>> +        if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
>> +            diag = rte_eth_dev_start(pi);
>> +            if (diag < 0) {
>> +                printf("Fail to start port %d: %s\n", pi,
>> +                       rte_strerror(-diag));
>> -            /* Fail to setup rx queue, return */
>> -            if (rte_atomic16_cmpset(&(port->port_status),
>> +                /* Fail to setup rx queue, return */
>> +                if (rte_atomic16_cmpset(&(port->port_status),
>>                   RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> 
> Syntax is wrong here after change, but if you go with above idea to hide 
> process type check within functions, you won't need to change these 
> lines at all.
> 
> <...>
> 
>> @@ -3885,8 +3925,10 @@ main(int argc, char** argv)
>>           }
>>       }
>> -    if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
>> +    if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
>> +        pmd_test_exit();
> 
> Why need to add 'pmd_test_exit()' for the multi-process support, if 
> there is a special case, please add comment for it.
> 
> 
> .
diff mbox series

Patch

diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 5bf1497..e465824 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5354,6 +5354,12 @@  cmd_set_flush_rx_parsed(void *parsed_result,
 		__rte_unused void *data)
 {
 	struct cmd_set_flush_rx *res = parsed_result;
+
+	if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+		printf("multi-process doesn't support to flush rx queues.\n");
+		return;
+	}
+
 	no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
 }
 
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 40b2b29..c982c87 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2860,6 +2860,8 @@  rss_fwd_config_setup(void)
 	queueid_t  rxq;
 	queueid_t  nb_q;
 	streamid_t  sm_id;
+	int start;
+	int end;
 
 	nb_q = nb_rxq;
 	if (nb_q > nb_txq)
@@ -2877,7 +2879,15 @@  rss_fwd_config_setup(void)
 	init_fwd_streams();
 
 	setup_fwd_config_of_each_lcore(&cur_fwd_config);
-	rxp = 0; rxq = 0;
+
+	if (proc_id > 0 && nb_q % num_procs)
+		printf("Warning! queue numbers should be multiple of "
+			"processes, or packet loss will happen.\n");
+
+	start = proc_id * nb_q / num_procs;
+	end = start + nb_q / num_procs;
+	rxp = 0;
+	rxq = start;
 	for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
 		struct fwd_stream *fs;
 
@@ -2894,6 +2904,8 @@  rss_fwd_config_setup(void)
 			continue;
 		rxp = 0;
 		rxq++;
+		if (rxq >= end)
+			rxq = start;
 	}
 }
 
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..d86cc21 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@  parse_link_speed(int n)
 void
 launch_args_parse(int argc, char** argv)
 {
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
 	int n, opt;
 	char **argvopt;
 	int opt_idx;
@@ -625,6 +628,8 @@  launch_args_parse(int argc, char** argv)
 		{ "rx-mq-mode",                 1, 0, 0 },
 		{ "record-core-cycles",         0, 0, 0 },
 		{ "record-burst-stats",         0, 0, 0 },
+		{ PARAM_NUM_PROCS,              1, 0, 0 },
+		{ PARAM_PROC_ID,                1, 0, 0 },
 		{ 0, 0, 0, 0 },
 	};
 
@@ -1391,6 +1396,13 @@  launch_args_parse(int argc, char** argv)
 				record_core_cycles = 1;
 			if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
 				record_burst_stats = 1;
+
+			if (strncmp(lgopts[opt_idx].name,
+				    PARAM_NUM_PROCS, 9) == 0)
+				num_procs = atoi(optarg);
+			if (strncmp(lgopts[opt_idx].name,
+				    PARAM_PROC_ID, 7) == 0)
+				proc_id = atoi(optarg);
 			break;
 		case 'h':
 			usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 96d2e0f..01d0d82 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -518,6 +518,16 @@  enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
  */
 uint32_t eth_link_speed;
 
+/*
+ * Id of the current process.
+ */
+int proc_id;
+
+/*
+ * Number of processes.
+ */
+unsigned int num_procs = 1;
+
 /* Forward function declarations */
 static void setup_attached_port(portid_t pi);
 static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +987,11 @@  mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
 	mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
 	mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		rte_mp = rte_mempool_lookup(pool_name);
+		goto err;
+	}
+
 	TESTPMD_LOG(INFO,
 		"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
 		pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1074,14 @@  mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
 
 err:
 	if (rte_mp == NULL) {
-		rte_exit(EXIT_FAILURE,
-			"Creation of mbuf pool for socket %u failed: %s\n",
-			socket_id, rte_strerror(rte_errno));
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+			rte_exit(EXIT_FAILURE,
+				"Creation of mbuf pool for socket %u failed: %s\n",
+				socket_id, rte_strerror(rte_errno));
+		else
+			rte_exit(EXIT_FAILURE,
+				"Get mbuf pool for socket %u failed: %s\n",
+				socket_id, rte_strerror(rte_errno));
 	} else if (verbose_level > 0) {
 		rte_mempool_dump(stdout, rte_mp);
 	}
@@ -2002,6 +2022,12 @@  flush_fwd_rx_queues(void)
 	uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
 	uint64_t timer_period;
 
+	if (num_procs > 1) {
+		printf("multi-process not support for flushing fwd rx "
+		       "queues, skip the below lines and return.\n");
+		return;
+	}
+
 	/* convert to number of cycles */
 	timer_period = rte_get_timer_hz(); /* 1 second timeout */
 
@@ -2511,21 +2537,28 @@  start_port(portid_t pid)
 				return -1;
 			}
 			/* configure port */
-			diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+			if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+				diag = rte_eth_dev_configure(pi,
+						     nb_rxq + nb_hairpinq,
 						     nb_txq + nb_hairpinq,
 						     &(port->dev_conf));
-			if (diag != 0) {
-				if (rte_atomic16_cmpset(&(port->port_status),
-				RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
-					printf("Port %d can not be set back "
-							"to stopped\n", pi);
-				printf("Fail to configure port %d\n", pi);
-				/* try to reconfigure port next time */
-				port->need_reconfig = 1;
-				return -1;
+				if (diag != 0) {
+					if (rte_atomic16_cmpset(
+							&(port->port_status),
+							RTE_PORT_HANDLING,
+							RTE_PORT_STOPPED) == 0)
+						printf("Port %d can not be set "
+						       "back to stopped\n", pi);
+					printf("Fail to configure port %d\n",
+						pi);
+					/* try to reconfigure port next time */
+					port->need_reconfig = 1;
+					return -1;
+				}
 			}
 		}
-		if (port->need_reconfig_queues > 0) {
+		if (port->need_reconfig_queues > 0 &&
+		    rte_eal_process_type() == RTE_PROC_PRIMARY) {
 			port->need_reconfig_queues = 0;
 			/* setup tx queues */
 			for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,17 +2659,20 @@  start_port(portid_t pid)
 		cnt_pi++;
 
 		/* start port */
-		diag = rte_eth_dev_start(pi);
-		if (diag < 0) {
-			printf("Fail to start port %d: %s\n", pi,
-			       rte_strerror(-diag));
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+			diag = rte_eth_dev_start(pi);
+			if (diag < 0) {
+				printf("Fail to start port %d: %s\n", pi,
+				       rte_strerror(-diag));
 
-			/* Fail to setup rx queue, return */
-			if (rte_atomic16_cmpset(&(port->port_status),
+				/* Fail to setup rx queue, return */
+				if (rte_atomic16_cmpset(&(port->port_status),
 				RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
-				printf("Port %d can not be set back to "
-							"stopped\n", pi);
-			continue;
+					printf("Port %d can not be set back to "
+								"stopped\n",
+						pi);
+				continue;
+			}
 		}
 
 		if (rte_atomic16_cmpset(&(port->port_status),
@@ -2765,7 +2801,8 @@  stop_port(portid_t pid)
 		if (port->flow_list)
 			port_flow_flush(pi);
 
-		if (rte_eth_dev_stop(pi) != 0)
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+		    rte_eth_dev_stop(pi) != 0)
 			RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
 				pi);
 
@@ -2834,8 +2871,10 @@  close_port(portid_t pid)
 			continue;
 		}
 
-		port_flow_flush(pi);
-		rte_eth_dev_close(pi);
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+			port_flow_flush(pi);
+			rte_eth_dev_close(pi);
+		}
 	}
 
 	remove_invalid_ports();
@@ -3099,7 +3138,7 @@  pmd_test_exit(void)
 		}
 	}
 	for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
-		if (mempools[i])
+		if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
 			rte_mempool_free(mempools[i]);
 	}
 
@@ -3430,7 +3469,8 @@  update_jumbo_frame_offload(portid_t portid)
 	/* If JUMBO_FRAME is set MTU conversion done by ethdev layer,
 	 * if unset do it here
 	 */
-	if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
+	if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0 &&
+	     rte_eal_process_type() == RTE_PROC_PRIMARY) {
 		ret = rte_eth_dev_set_mtu(portid,
 				port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
 		if (ret)
@@ -3621,6 +3661,10 @@  init_port_dcb_config(portid_t pid,
 	int retval;
 	uint16_t i;
 
+	if (num_procs > 1) {
+		printf("The multi-process feature doesn't support dcb.\n");
+		return -ENOTSUP;
+	}
 	rte_port = &ports[pid];
 
 	memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3786,10 +3830,6 @@  main(int argc, char** argv)
 		rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
 			 rte_strerror(rte_errno));
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
-		rte_exit(EXIT_FAILURE,
-			 "Secondary process type not supported.\n");
-
 	ret = register_eth_event_callback();
 	if (ret != 0)
 		rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3885,8 +3925,10 @@  main(int argc, char** argv)
 		}
 	}
 
-	if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+	if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+		pmd_test_exit();
 		rte_exit(EXIT_FAILURE, "Start ports failed\n");
+	}
 
 	/* set all ports to promiscuous mode by default */
 	RTE_ETH_FOREACH_DEV(port_id) {
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 36d8535..e04016c 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -630,6 +630,9 @@  extern struct mplsoudp_decap_conf mplsoudp_decap_conf;
 
 extern enum rte_eth_rx_mq_mode rx_mq_mode;
 
+extern int proc_id;
+extern unsigned int num_procs;
+
 static inline unsigned int
 lcore_num(void)
 {
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 3bd7757..df01d4f 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -185,6 +185,7 @@  New Features
     ``show port (port_id) rxq (queue_id) desc used count``
   * Added command to dump internal representation information of single flow.
     ``flow dump (port_id) rule (rule_id)``
+  * Added support multi-process for testpmd.
 
 
 Removed Items
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index ec1dc7d..d714951 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -551,3 +551,89 @@  The command line options are:
     bit 1 - two hairpin ports paired
     bit 0 - two hairpin ports loop
     The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+	primary process:
+	sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+        --num-procs=2 --proc-id=0
+
+	secondary process:
+	sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+        --num-procs=2 --proc-id=1
+
+The command line options are:
+
+*   ``--rxq=N``
+
+    Set the number of Rx queues per port to N. N is the sum of queues used by primary
+    and secondary process. Primary process and secondary process should have separate
+    queues, and each should occupy at least one queue. Where N should be the multiple
+    of number of processes.
+
+*   ``--txq=N``
+
+    Set the number of Tx queues per port to N. N is the sum of queues used by primary
+    and secondary process. Primary process and secondary process should have separate
+    queues, and each should occupy at least one queue. Where N should be the multiple
+    of number of processes.
+
+*   ``--num-procs=N``
+
+    The number of processes which will be used.
+
+*   ``--proc-id=id``
+
+    The id of the current process (id < num-procs). id should be different in primary
+    process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs;
+end(queue end id) = start + nb_q / num_procs;
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.