On 4/19/2021 2:03 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
Hi Connor,
I put some comments on the documentation, can you please check them?
Meanwhile, John, Marko can you please help on the doc review?
Thanks,
ferruh
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
<...>
> @@ -2626,16 +2701,16 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - diag = rte_eth_dev_start(pi);
> + diag = eth_dev_start_mp(pi);
> if (diag < 0) {
> printf("Fail to start port %d: %s\n", pi,
> rte_strerror(-diag));
>
> /* Fail to setup rx queue, return */
> if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back to "
> - "stopped\n", pi);
> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set back to stopped\n",
> + pi);
Indentation is wrong.
<...>
> --- a/doc/guides/testpmd_app_ug/run_app.rst
> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> @@ -543,3 +543,89 @@ The command line options are:
> bit 1 - two hairpin ports paired
> bit 0 - two hairpin ports loop
> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
> +
> +
> +Testpmd Support Multi Process Command-line Options
> +--------------------------------------------------
What do you think making this a sub-section of the "Testpmd Command-line Options"?
And the section name can be "Testpmd Multi-Process Command-line Options"
> +
> +The following are the command-line options for the testpmd applications(support
Space before '('.
> +multi process).They must be separated from the EAL options, shown in the previous
> +section, with a ``--`` separator:
If this becomes sub-section of the "Testpmd Command-line Options", above will
become duplication, and I think it can be simplified as:
"The following are the command-line options for testpmd multi-process support:"
> +
> +.. code-block:: console
> +
> + primary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=0
> +
> + secondary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=1
> +
> +The command line options are:
> +
> +* ``--rxq=N``
> +
> + Set the number of Rx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
> +
> +* ``--txq=N``
> +
> + Set the number of Tx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
> +* ``--proc-id=id``
> +
> + The id of the current process (id < num-procs). id should be different in primary
> + process and secondary process.
> +
The famous question, does it start from '0' or '1'?
> +Calculation rule for queue:
> +All queues are allocated to different processes based on proc_num and proc_id.
Can highlight proc_num and proc_id, as ``proc_num`` and ``proc_id``.
> +Calculation rule for the Testpmd to allocate queues to each process:
Not sure if testpmd should start with uppercase.
> +start(queue start id) = proc_id * nb_q / num_procs；
> +end(queue end id) = start + nb_q / num_procs；
> +
Can you put above into a code-block.
> +For example, if supports 4 txq and rxq
".., if testpmp supports 4 Tx and Rx queues"
> +the 0~1 for primary process
> +the 2~3 for secondary process
> +
> +The number of rings should be a multiple of the number of processes. If not,
> +redundant queues will exist after queues are allocated to processes. After RSS is
> +enabled, packet loss occurs when traffic is sent to all processes at the same time.
> +Some traffic enters redundant queues and cannot be forwarded.
> +
> +Most dev ops is supported in primary and secondary process. While secondary process
"Most of the device operations are supported in ..."?
"While the secondary process ..."?
> +is not permitted to allocate or release shared memory, so some ops are not supported
> +as follows:
> +``dev_configure``
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
> +
This list displayed as single line in the html output, if the intention is to
have them listed as been in the source, may need following update:
-as follows:
+as follows::
+
> +So, any command from testpmd which calls those APIs will not be supported in secondary
> +process, like:
> +``port config all rxq|txq|rxd|txd <value>``
> +``port config <port_id> rx_offload xxx on/off ``
> +``port config <port_id> tx_offload xxx on/off``
> +etc.
Same here for the list view.
> +
> +RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
You may use "Flow API", instead of RTE_FLOW, so it becomes: "Flow API is supported."
> +stats supported, stats will not change when one quit and start, As they share the same
> +buffer to store the stats.
The above stats related sentences looks like squeezed within the flow API
related description, it may be good to move it.
> Flow rules are maintained in process level: primary and secondary
> +has its own flow list(but one flow list in HW). The two can see all the queues, so setting
Space before '('.
> +the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
> +from others is not permitted.
I understand what you mean, but wording can be improved, can you please try to
reword it?
> +
> +RSS supported, Primary process and secondary process has separate queues to use, RSS
"RSS is supported"? And start 'primary' with lowercase.
> +will work in their own queues whether primary and secondary process.
>
There is a chance that when RSS is enabled, the packet may end up in other
process' queues?
Hi, Ferruh,
在 2021/4/19 21:42, Ferruh Yigit 写道:
> On 4/19/2021 2:03 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>
> Hi Connor,
>
> I put some comments on the documentation, can you please check them?
>
> Meanwhile, John, Marko can you please help on the doc review?
>
> Thanks,
> ferruh
>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> <...>
>
>> @@ -2626,16 +2701,16 @@ start_port(portid_t pid)
>> cnt_pi++;
>> /* start port */
>> - diag = rte_eth_dev_start(pi);
>> + diag = eth_dev_start_mp(pi);
>> if (diag < 0) {
>> printf("Fail to start port %d: %s\n", pi,
>> rte_strerror(-diag));
>> /* Fail to setup rx queue, return */
>> if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back to "
>> - "stopped\n", pi);
>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> + printf("Port %d can not be set back to stopped\n",
>> + pi);
>
> Indentation is wrong.
>
> <...>
>
>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>> @@ -543,3 +543,89 @@ The command line options are:
>> bit 1 - two hairpin ports paired
>> bit 0 - two hairpin ports loop
>> The default value is 0. Hairpin will use single port mode and
>> implicit Tx flow mode.
>> +
>> +
>> +Testpmd Support Multi Process Command-line Options
>> +--------------------------------------------------
>
> What do you think making this a sub-section of the "Testpmd Command-line
> Options"?
>
> And the section name can be "Testpmd Multi-Process Command-line Options"
>
>> +
>> +The following are the command-line options for the testpmd
>> applications(support
>
> Space before '('.
>
>> +multi process).They must be separated from the EAL options, shown in
>> the previous
>> +section, with a ``--`` separator:
>
> If this becomes sub-section of the "Testpmd Command-line Options", above
> will become duplication, and I think it can be simplified as:
>
> "The following are the command-line options for testpmd multi-process
> support:"
>
>
>> +
>> +.. code-block:: console
>> +
>> + primary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4
>> --txq=4 \
>> + --num-procs=2 --proc-id=0
>> +
>> + secondary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4
>> --txq=4 \
>> + --num-procs=2 --proc-id=1
>> +
>> +The command line options are:
>> +
>> +* ``--rxq=N``
>> +
>> + Set the number of Rx queues per port to N. N is the sum of queues
>> used by primary
>> + and secondary process. Primary process and secondary process
>> should have separate
>> + queues, and each should occupy at least one queue. Where N should
>> be the multiple
>> + of number of processes.
>> +
>> +* ``--txq=N``
>> +
>> + Set the number of Tx queues per port to N. N is the sum of queues
>> used by primary
>> + and secondary process. Primary process and secondary process
>> should have separate
>> + queues, and each should occupy at least one queue. Where N should
>> be the multiple
>> + of number of processes.
>> +
>> +* ``--num-procs=N``
>> +
>> + The number of processes which will be used.
>> +
>> +* ``--proc-id=id``
>> +
>> + The id of the current process (id < num-procs). id should be
>> different in primary
>> + process and secondary process.
>> +
>
> The famous question, does it start from '0' or '1'?
>
>> +Calculation rule for queue:
>> +All queues are allocated to different processes based on proc_num and
>> proc_id.
>
> Can highlight proc_num and proc_id, as ``proc_num`` and ``proc_id``.
>
>> +Calculation rule for the Testpmd to allocate queues to each process:
>
> Not sure if testpmd should start with uppercase.
>
>> +start(queue start id) = proc_id * nb_q / num_procs；
>> +end(queue end id) = start + nb_q / num_procs；
>> +
>
> Can you put above into a code-block.
>
Have already put above into a code-block in previous version.
>> +For example, if supports 4 txq and rxq
>
> ".., if testpmp supports 4 Tx and Rx queues"
>
>> +the 0~1 for primary process
>> +the 2~3 for secondary process
>> +
>> +The number of rings should be a multiple of the number of processes.
>> If not,
>> +redundant queues will exist after queues are allocated to processes.
>> After RSS is
>> +enabled, packet loss occurs when traffic is sent to all processes at
>> the same time.
>> +Some traffic enters redundant queues and cannot be forwarded.
>> +
>> +Most dev ops is supported in primary and secondary process. While
>> secondary process
>
> "Most of the device operations are supported in ..."?
> "While the secondary process ..."?
>
>> +is not permitted to allocate or release shared memory, so some ops
>> are not supported
>> +as follows:
>> +``dev_configure``
>> +``dev_start``
>> +``dev_stop``
>> +``rx_queue_setup``
>> +``tx_queue_setup``
>> +``rx_queue_release``
>> +``tx_queue_release``
>> +
>
> This list displayed as single line in the html output, if the intention
> is to have them listed as been in the source, may need following update:
>
> -as follows:
> +as follows::
> +
>
>> +So, any command from testpmd which calls those APIs will not be
>> supported in secondary
>> +process, like:
>> +``port config all rxq|txq|rxd|txd <value>``
>> +``port config <port_id> rx_offload xxx on/off ``
>> +``port config <port_id> tx_offload xxx on/off``
>> +etc.
>
> Same here for the list view.
>
>> +
>> +RTE_FLOW supported, it applies only on its own process on SW side,
>> but all on HW size.
>
> You may use "Flow API", instead of RTE_FLOW, so it becomes: "Flow API is
> supported."
>
>> +stats supported, stats will not change when one quit and start, As
>> they share the same
>> +buffer to store the stats.
>
> The above stats related sentences looks like squeezed within the flow
> API related description, it may be good to move it.
>
>> Flow rules are maintained in process level: primary and secondary
>> +has its own flow list(but one flow list in HW). The two can see all
>> the queues, so setting
>
> Space before '('.
>
>> +the flow rules for the other is OK. Of course, io(receive or transmit
>> packets) in the queue
>> +from others is not permitted.
>
> I understand what you mean, but wording can be improved, can you please
> try to reword it?
>
>> +
>> +RSS supported, Primary process and secondary process has separate
>> queues to use, RSS
>
> "RSS is supported"? And start 'primary' with lowercase.
>
>> +will work in their own queues whether primary and secondary process.
>>
>
> There is a chance that when RSS is enabled, the packet may end up in
> other process' queues?
When RSS is enabled, packet could end up in other process's queues, but
primary and secondary receives packerts from their own queues, and
they do not interfere with each other, do not conflict with each other.
> .
@@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -2859,6 +2859,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2876,7 +2878,22 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2893,6 +2910,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
@@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ return rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (is_proc_primary())
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2583,24 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d\n", pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2548,8 +2623,8 @@ start_port(portid_t pid)
if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING,
RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d tx queues\n",
pi);
/* try to reconfigure queues next time */
@@ -2626,16 +2701,16 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
continue;
}
@@ -2765,7 +2840,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2909,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3101,7 +3178,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3432,7 +3509,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3622,6 +3699,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3787,10 +3868,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -630,6 +630,15 @@ extern struct mplsoudp_decap_conf mplsoudp_decap_conf;
extern enum rte_eth_rx_mq_mode rx_mq_mode;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
@@ -192,6 +192,7 @@ New Features
``show port (port_id) rxq (queue_id) desc used count``
* Added command to dump internal representation information of single flow.
``flow dump (port_id) rule (rule_id)``
+ * Added support multi-process for testpmd.
Removed Items
@@ -543,3 +543,89 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.