[RFC,2/5] ethdev: introduce the affinity field in Tx queue API
Checks
Commit Message
For the multiple hardware ports connect to a single DPDK port (mhpsdp),
the previous patch introduces the new rte flow item to match the port
affinity of the received packets.
This patch adds the tx_affinity setting in Tx queue API, the affinity value
reflects packets be sent to which hardware port.
Adds the new tx_affinity field into the padding hole of rte_eth_txconf
structure, the size of rte_eth_txconf keeps the same. Adds a suppress
type for structure change in the ABI check file.
This patch adds the testpmd command line:
testpmd> port config (port_id) txq (queue_id) affinity (value)
For example, there're two hardware ports connects to a single DPDK
port (port id 0), and affinity 1 stood for hard port 1 and affinity
2 stood for hardware port 2, used the below command to config
tx affinity for each TxQ:
port config 0 txq 0 affinity 1
port config 0 txq 1 affinity 1
port config 0 txq 2 affinity 2
port config 0 txq 3 affinity 2
These commands config the TxQ index 0 and TxQ index 1 with affinity 1,
uses TxQ 0 or TxQ 1 send packets, these packets will be sent from the
hardware port 1, and similar with hardware port 2 if sending packets
with TxQ 2 or TxQ 3.
Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
---
app/test-pmd/cmdline.c | 84 +++++++++++++++++++++
devtools/libabigail.abignore | 5 ++
doc/guides/rel_notes/release_22_03.rst | 4 +
doc/guides/testpmd_app_ug/testpmd_funcs.rst | 13 ++++
lib/ethdev/rte_ethdev.h | 1 +
5 files changed, 107 insertions(+)
Comments
Hi Jiawei,
> -----Original Message-----
> From: Jiawei(Jonny) Wang <jiaweiw@nvidia.com>
> Sent: Wednesday, 21 December 2022 12:30
>
> For the multiple hardware ports connect to a single DPDK port (mhpsdp),
> the previous patch introduces the new rte flow item to match the port
> affinity of the received packets.
>
> This patch adds the tx_affinity setting in Tx queue API, the affinity value
> reflects packets be sent to which hardware port.
>
> Adds the new tx_affinity field into the padding hole of rte_eth_txconf
> structure, the size of rte_eth_txconf keeps the same. Adds a suppress
> type for structure change in the ABI check file.
>
> This patch adds the testpmd command line:
> testpmd> port config (port_id) txq (queue_id) affinity (value)
>
> For example, there're two hardware ports connects to a single DPDK
> port (port id 0), and affinity 1 stood for hard port 1 and affinity
> 2 stood for hardware port 2, used the below command to config
> tx affinity for each TxQ:
> port config 0 txq 0 affinity 1
> port config 0 txq 1 affinity 1
> port config 0 txq 2 affinity 2
> port config 0 txq 3 affinity 2
>
> These commands config the TxQ index 0 and TxQ
index 1 with affinity 1,
> uses TxQ 0 or TxQ 1 send packets, these packets will be sent from the
> hardware port 1, and similar with hardware port 2 if sending packets
> with TxQ 2 or TxQ 3.
>
> Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
> ---
Acked-by: Ori Kam <orika@nvidia.com>
Best,
Ori
21/12/2022 11:29, Jiawei Wang:
> For the multiple hardware ports connect to a single DPDK port (mhpsdp),
> the previous patch introduces the new rte flow item to match the port
> affinity of the received packets.
>
> This patch adds the tx_affinity setting in Tx queue API, the affinity value
> reflects packets be sent to which hardware port.
I think "affinity" means we would like packet to be sent
on a specific hardware port, but it is not mandatory.
Is it the meaning you want? Or should it be a mandatory port?
> Adds the new tx_affinity field into the padding hole of rte_eth_txconf
> structure, the size of rte_eth_txconf keeps the same. Adds a suppress
> type for structure change in the ABI check file.
>
> This patch adds the testpmd command line:
> testpmd> port config (port_id) txq (queue_id) affinity (value)
>
> For example, there're two hardware ports connects to a single DPDK
connects -> connected
> port (port id 0), and affinity 1 stood for hard port 1 and affinity
> 2 stood for hardware port 2, used the below command to config
> tx affinity for each TxQ:
> port config 0 txq 0 affinity 1
> port config 0 txq 1 affinity 1
> port config 0 txq 2 affinity 2
> port config 0 txq 3 affinity 2
>
> These commands config the TxQ index 0 and TxQ index 1 with affinity 1,
> uses TxQ 0 or TxQ 1 send packets, these packets will be sent from the
> hardware port 1, and similar with hardware port 2 if sending packets
> with TxQ 2 or TxQ 3.
[...]
> @@ -212,6 +212,10 @@ API Changes
> +* ethdev: added a new field:
> +
> + - Tx affinity per-queue ``rte_eth_txconf.tx_affinity``
Adding a new field is not an API change because
existing applications don't need to update their code
if they don't care this new field.
I think you can remove this note.
> --- a/lib/ethdev/rte_ethdev.h
> +++ b/lib/ethdev/rte_ethdev.h
> @@ -1138,6 +1138,7 @@ struct rte_eth_txconf {
> less free descriptors than this value. */
>
> uint8_t tx_deferred_start; /**< Do not start queue with rte_eth_dev_start(). */
> + uint8_t tx_affinity; /**< Drives the setting of affinity per-queue. */
Why "Drives"? It is the setting, right?
rte_eth_txconf is per-queue so no need to repeat.
I think a good comment here would be to mention it is a physical port index for mhpsdp.
Another good comment would be to specify how ports are numbered.
Hi,
>
> 21/12/2022 11:29, Jiawei Wang:
> > For the multiple hardware ports connect to a single DPDK port
> > (mhpsdp), the previous patch introduces the new rte flow item to match
> > the port affinity of the received packets.
> >
> > This patch adds the tx_affinity setting in Tx queue API, the affinity
> > value reflects packets be sent to which hardware port.
>
> I think "affinity" means we would like packet to be sent on a specific hardware
> port, but it is not mandatory.
> Is it the meaning you want? Or should it be a mandatory port?
>
Right, it's optional setting not mandatory.
> > Adds the new tx_affinity field into the padding hole of rte_eth_txconf
> > structure, the size of rte_eth_txconf keeps the same. Adds a suppress
> > type for structure change in the ABI check file.
> >
> > This patch adds the testpmd command line:
> > testpmd> port config (port_id) txq (queue_id) affinity (value)
> >
> > For example, there're two hardware ports connects to a single DPDK
>
> connects -> connected
>
OK, will fix in next version.
> > port (port id 0), and affinity 1 stood for hard port 1 and affinity
> > 2 stood for hardware port 2, used the below command to config tx
> > affinity for each TxQ:
> > port config 0 txq 0 affinity 1
> > port config 0 txq 1 affinity 1
> > port config 0 txq 2 affinity 2
> > port config 0 txq 3 affinity 2
> >
> > These commands config the TxQ index 0 and TxQ index 1 with affinity 1,
> > uses TxQ 0 or TxQ 1 send packets, these packets will be sent from the
> > hardware port 1, and similar with hardware port 2 if sending packets
> > with TxQ 2 or TxQ 3.
>
> [...]
> > @@ -212,6 +212,10 @@ API Changes
> > +* ethdev: added a new field:
> > +
> > + - Tx affinity per-queue ``rte_eth_txconf.tx_affinity``
>
> Adding a new field is not an API change because existing applications don't
> need to update their code if they don't care this new field.
> I think you can remove this note.
>
OK, will remove in next version.
> > --- a/lib/ethdev/rte_ethdev.h
> > +++ b/lib/ethdev/rte_ethdev.h
> > @@ -1138,6 +1138,7 @@ struct rte_eth_txconf {
> > less free descriptors than this value. */
> >
> > uint8_t tx_deferred_start; /**< Do not start queue with
> > rte_eth_dev_start(). */
> > + uint8_t tx_affinity; /**< Drives the setting of affinity per-queue.
> > +*/
>
> Why "Drives"? It is the setting, right?
> rte_eth_txconf is per-queue so no need to repeat.
> I think a good comment here would be to mention it is a physical port index for
> mhpsdp.
> Another good comment would be to specify how ports are numbered.
>
OK, will update the comment for this new setting.
Thanks.
18/01/2023 15:44, Jiawei(Jonny) Wang:
> > 21/12/2022 11:29, Jiawei Wang:
> > > For the multiple hardware ports connect to a single DPDK port
> > > (mhpsdp), the previous patch introduces the new rte flow item to match
> > > the port affinity of the received packets.
> > >
> > > This patch adds the tx_affinity setting in Tx queue API, the affinity
> > > value reflects packets be sent to which hardware port.
> >
> > I think "affinity" means we would like packet to be sent on a specific hardware
> > port, but it is not mandatory.
> > Is it the meaning you want? Or should it be a mandatory port?
>
> Right, it's optional setting not mandatory.
I think there is a misunderstanding.
I mean that "affinity" with port 0 may suggest that we try
to send to port 0 but sometimes the packet will be sent to port 1.
And I think you want the packet to be always sent to port 0
if affinity is 0, right?
If yes, I think the word "affinity" does not convey the right idea.
And again, the naming should give the idea that we are talking about
multiple ports merged in one DPDK port.
> > > Adds the new tx_affinity field into the padding hole of rte_eth_txconf
> > > structure, the size of rte_eth_txconf keeps the same. Adds a suppress
> > > type for structure change in the ABI check file.
> > >
> > > This patch adds the testpmd command line:
> > > testpmd> port config (port_id) txq (queue_id) affinity (value)
> > >
> > > For example, there're two hardware ports connects to a single DPDK
> >
> > connects -> connected
>
> OK, will fix in next version.
>
> > > port (port id 0), and affinity 1 stood for hard port 1 and affinity
> > > 2 stood for hardware port 2, used the below command to config tx
> > > affinity for each TxQ:
> > > port config 0 txq 0 affinity 1
> > > port config 0 txq 1 affinity 1
> > > port config 0 txq 2 affinity 2
> > > port config 0 txq 3 affinity 2
> > >
> > > These commands config the TxQ index 0 and TxQ index 1 with affinity 1,
> > > uses TxQ 0 or TxQ 1 send packets, these packets will be sent from the
> > > hardware port 1, and similar with hardware port 2 if sending packets
> > > with TxQ 2 or TxQ 3.
> >
> > [...]
> > > @@ -212,6 +212,10 @@ API Changes
> > > +* ethdev: added a new field:
> > > +
> > > + - Tx affinity per-queue ``rte_eth_txconf.tx_affinity``
> >
> > Adding a new field is not an API change because existing applications don't
> > need to update their code if they don't care this new field.
> > I think you can remove this note.
>
> OK, will remove in next version.
>
> > > --- a/lib/ethdev/rte_ethdev.h
> > > +++ b/lib/ethdev/rte_ethdev.h
> > > @@ -1138,6 +1138,7 @@ struct rte_eth_txconf {
> > > less free descriptors than this value. */
> > >
> > > uint8_t tx_deferred_start; /**< Do not start queue with
> > > rte_eth_dev_start(). */
> > > + uint8_t tx_affinity; /**< Drives the setting of affinity per-queue.
> > > +*/
> >
> > Why "Drives"? It is the setting, right?
> > rte_eth_txconf is per-queue so no need to repeat.
> > I think a good comment here would be to mention it is a physical port index for
> > mhpsdp.
> > Another good comment would be to specify how ports are numbered.
>
> OK, will update the comment for this new setting.
>
> Thanks.
Hi,
> 18/01/2023 15:44, Jiawei(Jonny) Wang:
> > > 21/12/2022 11:29, Jiawei Wang:
> > > > For the multiple hardware ports connect to a single DPDK port
> > > > (mhpsdp), the previous patch introduces the new rte flow item to
> > > > match the port affinity of the received packets.
> > > >
> > > > This patch adds the tx_affinity setting in Tx queue API, the
> > > > affinity value reflects packets be sent to which hardware port.
> > >
> > > I think "affinity" means we would like packet to be sent on a
> > > specific hardware port, but it is not mandatory.
> > > Is it the meaning you want? Or should it be a mandatory port?
> >
> > Right, it's optional setting not mandatory.
>
> I think there is a misunderstanding.
> I mean that "affinity" with port 0 may suggest that we try to send to port 0 but
> sometimes the packet will be sent to port 1.
>
> And I think you want the packet to be always sent to port 0 if affinity is 0, right?
>
These packets should be always sent to port 0 if 'affinity' be set with hardware port 0.
'affinity is 0' -> 0 means that no affinity be set and traffic should be kept the same behavior
as before, for example, routing between different hardware ports.
> If yes, I think the word "affinity" does not convey the right idea.
> And again, the naming should give the idea that we are talking about multiple
> ports merged in one DPDK port.
>
OK, how about 'tx_mhpsdp_hwport?
'mhpsdp' as mentioned before, 'hwport' means for one 'hardware port'.
> > > > Adds the new tx_affinity field into the padding hole of
> > > > rte_eth_txconf structure, the size of rte_eth_txconf keeps the
> > > > same. Adds a suppress type for structure change in the ABI check file.
> > > >
> > > > This patch adds the testpmd command line:
> > > > testpmd> port config (port_id) txq (queue_id) affinity (value)
> > > >
> > > > For example, there're two hardware ports connects to a single DPDK
> > >
> > > connects -> connected
> >
> > OK, will fix in next version.
> >
> > > > port (port id 0), and affinity 1 stood for hard port 1 and
> > > > affinity
> > > > 2 stood for hardware port 2, used the below command to config tx
> > > > affinity for each TxQ:
> > > > port config 0 txq 0 affinity 1
> > > > port config 0 txq 1 affinity 1
> > > > port config 0 txq 2 affinity 2
> > > > port config 0 txq 3 affinity 2
> > > >
> > > > These commands config the TxQ index 0 and TxQ index 1 with
> > > > affinity 1, uses TxQ 0 or TxQ 1 send packets, these packets will
> > > > be sent from the hardware port 1, and similar with hardware port 2
> > > > if sending packets with TxQ 2 or TxQ 3.
> > >
> > > [...]
> > > > @@ -212,6 +212,10 @@ API Changes
> > > > +* ethdev: added a new field:
> > > > +
> > > > + - Tx affinity per-queue ``rte_eth_txconf.tx_affinity``
> > >
> > > Adding a new field is not an API change because existing
> > > applications don't need to update their code if they don't care this new field.
> > > I think you can remove this note.
> >
> > OK, will remove in next version.
> >
> > > > --- a/lib/ethdev/rte_ethdev.h
> > > > +++ b/lib/ethdev/rte_ethdev.h
> > > > @@ -1138,6 +1138,7 @@ struct rte_eth_txconf {
> > > > less free descriptors than this value. */
> > > >
> > > > uint8_t tx_deferred_start; /**< Do not start queue with
> > > > rte_eth_dev_start(). */
> > > > + uint8_t tx_affinity; /**< Drives the setting of affinity per-queue.
> > > > +*/
> > >
> > > Why "Drives"? It is the setting, right?
> > > rte_eth_txconf is per-queue so no need to repeat.
> > > I think a good comment here would be to mention it is a physical
> > > port index for mhpsdp.
> > > Another good comment would be to specify how ports are numbered.
> >
> > OK, will update the comment for this new setting.
> >
> > Thanks.
>
>
@@ -764,6 +764,10 @@ static void cmd_help_long_parsed(void *parsed_result,
"port cleanup (port_id) txq (queue_id) (free_cnt)\n"
" Cleanup txq mbufs for a specific Tx queue\n\n"
+
+ "port config <port_id> txq <queue_id> affinity <value>\n"
+ " Set the port affinity value "
+ "on a specific Tx queue\n\n"
);
}
@@ -12621,6 +12625,85 @@ static cmdline_parse_inst_t cmd_show_port_flow_transfer_proxy = {
}
};
+/* *** configure port txq affinity value *** */
+struct cmd_config_tx_affinity {
+ cmdline_fixed_string_t port;
+ cmdline_fixed_string_t config;
+ portid_t portid;
+ cmdline_fixed_string_t txq;
+ uint16_t qid;
+ cmdline_fixed_string_t affinity;
+ uint16_t value;
+};
+
+static void
+cmd_config_tx_affinity_parsed(void *parsed_result,
+ __rte_unused struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_config_tx_affinity *res = parsed_result;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(res->portid, ENABLED_WARN))
+ return;
+
+ if (res->portid == (portid_t)RTE_PORT_ALL) {
+ printf("Invalid port id\n");
+ return;
+ }
+
+ port = &ports[res->portid];
+
+ if (strcmp(res->txq, "txq")) {
+ printf("Unknown parameter\n");
+ return;
+ }
+ if (tx_queue_id_is_invalid(res->qid))
+ return;
+
+ port->txq[res->qid].conf.tx_affinity = res->value;
+
+ cmd_reconfig_device_queue(res->portid, 0, 1);
+}
+
+cmdline_parse_token_string_t cmd_config_tx_affinity_port =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_affinity,
+ port, "port");
+cmdline_parse_token_string_t cmd_config_tx_affinity_config =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_affinity,
+ config, "config");
+cmdline_parse_token_num_t cmd_config_tx_affinity_portid =
+ TOKEN_NUM_INITIALIZER(struct cmd_config_tx_affinity,
+ portid, RTE_UINT16);
+cmdline_parse_token_string_t cmd_config_tx_affinity_txq =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_affinity,
+ txq, "txq");
+cmdline_parse_token_num_t cmd_config_tx_affinity_qid =
+ TOKEN_NUM_INITIALIZER(struct cmd_config_tx_affinity,
+ qid, RTE_UINT16);
+cmdline_parse_token_string_t cmd_config_tx_affinity_affine =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_affinity,
+ affinity, "affinity");
+cmdline_parse_token_num_t cmd_config_tx_affinity_value =
+ TOKEN_NUM_INITIALIZER(struct cmd_config_tx_affinity,
+ value, RTE_UINT16);
+
+static cmdline_parse_inst_t cmd_config_tx_affinity = {
+ .f = cmd_config_tx_affinity_parsed,
+ .data = (void *)0,
+ .help_str = "port config <port_id> txq <queue_id> affinity <value>",
+ .tokens = {
+ (void *)&cmd_config_tx_affinity_port,
+ (void *)&cmd_config_tx_affinity_config,
+ (void *)&cmd_config_tx_affinity_portid,
+ (void *)&cmd_config_tx_affinity_txq,
+ (void *)&cmd_config_tx_affinity_qid,
+ (void *)&cmd_config_tx_affinity_affine,
+ (void *)&cmd_config_tx_affinity_value,
+ NULL,
+ },
+};
+
/* ******************************************************************************** */
/* list of instructions */
@@ -12851,6 +12934,7 @@ static cmdline_parse_ctx_t builtin_ctx[] = {
(cmdline_parse_inst_t *)&cmd_show_capability,
(cmdline_parse_inst_t *)&cmd_set_flex_is_pattern,
(cmdline_parse_inst_t *)&cmd_set_flex_spec_pattern,
+ (cmdline_parse_inst_t *)&cmd_config_tx_affinity,
NULL,
};
@@ -20,6 +20,11 @@
[suppress_file]
soname_regexp = ^librte_.*mlx.*glue\.
+; Ignore fields inserted in middle padding of rte_eth_txconf
+[suppress_type]
+ name = rte_eth_txconf
+ has_data_member_inserted_between = {offset_after(tx_deferred_start), offset_of(offloads)}
+
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Experimental APIs exceptions ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -212,6 +212,10 @@ API Changes
* ethdev: Old public macros and enumeration constants without ``RTE_ETH_`` prefix,
which are kept for backward compatibility, are marked as deprecated.
+* ethdev: added a new field:
+
+ - Tx affinity per-queue ``rte_eth_txconf.tx_affinity``
+
* cryptodev: The asymmetric session handling was modified to use a single
mempool object. An API ``rte_cryptodev_asym_session_pool_create`` was added
to create a mempool with element size big enough to hold the generic asymmetric
@@ -1605,6 +1605,19 @@ Enable or disable a per queue Tx offloading only on a specific Tx queue::
This command should be run when the port is stopped, or else it will fail.
+config per queue Tx affinity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Configure a affinity value per queue Tx offloading only on a specific Tx queue::
+
+ testpmd> port (port_id) txq (queue_id) affinity (value)
+
+* ``affinity``: reflects packet can be sent to which hardware port.
+ uses it on multiple hardware ports connect to
+ a single DPDK port (mhpsdp).
+
+This command should be run when the port is stopped, or else it will fail.
+
Config VXLAN Encap outer layers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1138,6 +1138,7 @@ struct rte_eth_txconf {
less free descriptors than this value. */
uint8_t tx_deferred_start; /**< Do not start queue with rte_eth_dev_start(). */
+ uint8_t tx_affinity; /**< Drives the setting of affinity per-queue. */
/**
* Per-queue Tx offloads to be set using RTE_ETH_TX_OFFLOAD_* flags.
* Only offloads set on tx_queue_offload_capa or tx_offload_capa