On 8/10/2017 3:50 AM, Jiayu Hu wrote:
> The GRO library provides two reassembly modes: lightweight mode and
> heavyweight mode. This patch is to support the heavyweight mode in
> csum forwarding engine.
>
> With the command "gro (heavymode|lightmode) (on|off) <port id>", users
> can select the lightweight mode or the heavyweight mode to use. With
> the command "gro flush interval <num>", users can set the interval of
> flushing GROed packets from the reassembly tables for the heavyweight
> mode.
>
> Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
> ---
> app/test-pmd/cmdline.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++---
> app/test-pmd/config.c | 33 +++++++++++++--------
> app/test-pmd/csumonly.c | 29 ++++++++++++++----
> app/test-pmd/testpmd.c | 17 +++++++++++
> app/test-pmd/testpmd.h | 12 +++++++-
> 5 files changed, 147 insertions(+), 23 deletions(-)
>
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> index cd8c358..3224ce1 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -423,7 +423,7 @@ static void cmd_help_long_parsed(void *parsed_result,
> "tso show (portid)"
> " Display the status of TCP Segmentation Offload.\n\n"
>
> - "gro (on|off) (port_id)"
> + "gro (heavymode|lightmode) (on|off) (port_id)\n"
Not specific / limited to gro, but I have a few testpmd usability questions:
1) to update some settings, there are two root level commands already:
a) set ... (like: "set fwd mac", "set promisc #P on")
b) port config ... (like: "port config #P speed auto duplex auto")
I don't know what it difference between above two, but thinking as "port
config" is to configure ports and "set" is to set rest of testpmd config
makes sense to me. (but "set vf .." and "set port .." doesn't fit to
this statement)
Instead of adding "gro" root level command, why not add this under "port
config", like: "port config #P gro on" ? (or "set port #P gro on")
2) Should each configuration set have a corresponding show command?
How a user can see the current gro setting?
Do we need a "show port gro" ?
3) Where to place #P in the command:
There are inconsistencies about where to put it, like:
"port config #P l2-tunnel enable"
"port config mtu #P value"
"port #P rxq #Q start"
or
"show port info #P"
"show port #P rss-hash ip4 key"
It can be good to define a place for it for consistency.
4) Using "port" keyword in some commands before port_id:
Like: "set link-down port #P", "reset port #P mirror-rule value"
Mostly we don't have it:
"show txq info #P #Q", "set bonding mode #M #P", "set stat_qmap rx #P
#Q", "vlan set strip on #P"
It can be good to define one or other.
And I guess having need to use "port" keyword can be an indication that
command should move under "port" command:
"reset port #P mirror-rule #id" can be "port reset #P mirror-rule #id"
Thanks,
ferruh
Hi Ferruh,
On Thu, Aug 10, 2017 at 10:50:02AM +0100, Ferruh Yigit wrote:
> On 8/10/2017 3:50 AM, Jiayu Hu wrote:
> > The GRO library provides two reassembly modes: lightweight mode and
> > heavyweight mode. This patch is to support the heavyweight mode in
> > csum forwarding engine.
> >
> > With the command "gro (heavymode|lightmode) (on|off) <port id>", users
> > can select the lightweight mode or the heavyweight mode to use. With
> > the command "gro flush interval <num>", users can set the interval of
> > flushing GROed packets from the reassembly tables for the heavyweight
> > mode.
> >
> > Signed-off-by: Jiayu Hu <jiayu.hu@intel.com>
> > ---
> > app/test-pmd/cmdline.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++---
> > app/test-pmd/config.c | 33 +++++++++++++--------
> > app/test-pmd/csumonly.c | 29 ++++++++++++++----
> > app/test-pmd/testpmd.c | 17 +++++++++++
> > app/test-pmd/testpmd.h | 12 +++++++-
> > 5 files changed, 147 insertions(+), 23 deletions(-)
> >
> > diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> > index cd8c358..3224ce1 100644
> > --- a/app/test-pmd/cmdline.c
> > +++ b/app/test-pmd/cmdline.c
> > @@ -423,7 +423,7 @@ static void cmd_help_long_parsed(void *parsed_result,
> > "tso show (portid)"
> > " Display the status of TCP Segmentation Offload.\n\n"
> >
> > - "gro (on|off) (port_id)"
> > + "gro (heavymode|lightmode) (on|off) (port_id)\n"
>
> Not specific / limited to gro, but I have a few testpmd usability questions:
>
> 1) to update some settings, there are two root level commands already:
> a) set ... (like: "set fwd mac", "set promisc #P on")
> b) port config ... (like: "port config #P speed auto duplex auto")
>
> I don't know what it difference between above two, but thinking as "port
> config" is to configure ports and "set" is to set rest of testpmd config
> makes sense to me. (but "set vf .." and "set port .." doesn't fit to
> this statement)
>
> Instead of adding "gro" root level command, why not add this under "port
> config", like: "port config #P gro on" ? (or "set port #P gro on")
Yes, using 'gro' as root command will make the usage of testpmd inconsistent.
I will change the GSO command in the next patch. Besides, there are other root
commands, like 'tso', and maybe we need to change them too.
>
>
> 2) Should each configuration set have a corresponding show command?
>
> How a user can see the current gro setting?
> Do we need a "show port gro" ?
Yes, I will add one in the next patch.
>
>
> 3) Where to place #P in the command:
> There are inconsistencies about where to put it, like:
> "port config #P l2-tunnel enable"
> "port config mtu #P value"
> "port #P rxq #Q start"
>
> or
>
> "show port info #P"
> "show port #P rss-hash ip4 key"
>
> It can be good to define a place for it for consistency.
Make sense.
>
>
> 4) Using "port" keyword in some commands before port_id:
> Like: "set link-down port #P", "reset port #P mirror-rule value"
>
> Mostly we don't have it:
> "show txq info #P #Q", "set bonding mode #M #P", "set stat_qmap rx #P
> #Q", "vlan set strip on #P"
>
> It can be good to define one or other.
Agree, and I will add "port" to GRO related commands.
Thanks,
Jiayu
>
> And I guess having need to use "port" keyword can be an indication that
> command should move under "port" command:
> "reset port #P mirror-rule #id" can be "port reset #P mirror-rule #id"
>
>
> Thanks,
> ferruh
@@ -423,7 +423,7 @@ static void cmd_help_long_parsed(void *parsed_result,
"tso show (portid)"
" Display the status of TCP Segmentation Offload.\n\n"
- "gro (on|off) (port_id)"
+ "gro (heavymode|lightmode) (on|off) (port_id)\n"
" Enable or disable Generic Receive Offload in"
" csum forwarding engine.\n\n"
@@ -431,6 +431,10 @@ static void cmd_help_long_parsed(void *parsed_result,
" Set max flow number and max packet number per-flow"
" for GRO.\n\n"
+ "gro flush interval (num)\n"
+ " Set the interval of flushing GROed packets from"
+ " reassembly tables.\n\n"
+
"set fwd (%s)\n"
" Set packet forwarding mode.\n\n"
@@ -3853,6 +3857,7 @@ cmdline_parse_inst_t cmd_tunnel_tso_show = {
struct cmd_gro_result {
cmdline_fixed_string_t cmd_keyword;
cmdline_fixed_string_t mode;
+ cmdline_fixed_string_t onoff;
uint8_t port_id;
};
@@ -3864,7 +3869,7 @@ cmd_enable_gro_parsed(void *parsed_result,
struct cmd_gro_result *res;
res = parsed_result;
- setup_gro(res->mode, res->port_id);
+ setup_gro(res->mode, res->onoff, res->port_id);
}
cmdline_parse_token_string_t cmd_gro_keyword =
@@ -3872,7 +3877,10 @@ cmdline_parse_token_string_t cmd_gro_keyword =
cmd_keyword, "gro");
cmdline_parse_token_string_t cmd_gro_mode =
TOKEN_STRING_INITIALIZER(struct cmd_gro_result,
- mode, "on#off");
+ mode, "heavymode#lightmode");
+cmdline_parse_token_string_t cmd_gro_onoff =
+ TOKEN_STRING_INITIALIZER(struct cmd_gro_result,
+ onoff, "on#off");
cmdline_parse_token_num_t cmd_gro_pid =
TOKEN_NUM_INITIALIZER(struct cmd_gro_result,
port_id, UINT8);
@@ -3880,15 +3888,77 @@ cmdline_parse_token_num_t cmd_gro_pid =
cmdline_parse_inst_t cmd_enable_gro = {
.f = cmd_enable_gro_parsed,
.data = NULL,
- .help_str = "gro (on|off) (port_id)",
+ .help_str = "gro (heavymode|lightmode) (on|off) <port_id>",
.tokens = {
(void *)&cmd_gro_keyword,
(void *)&cmd_gro_mode,
+ (void *)&cmd_gro_onoff,
(void *)&cmd_gro_pid,
NULL,
},
};
+/* *** SET FLUSH INTERVAL FOR THE HEAVYWEIGHT MODE GRO *** */
+struct cmd_gro_flush_result {
+ cmdline_fixed_string_t cmd_keyword;
+ cmdline_fixed_string_t cmd_flush;
+ cmdline_fixed_string_t cmd_interval;
+ uint32_t cmd_num;
+};
+
+static void
+cmd_gro_flush_parsed(void *parsed_result,
+ __attribute__((unused)) struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ struct cmd_gro_flush_result *res;
+
+ res = parsed_result;
+ if (test_done == 0) {
+ printf("Before set flushing interval for the heavyweight"
+ " mode GRO, please stop forwarding first\n");
+ return;
+ }
+
+ if (!strcmp(res->cmd_interval, "interval")) {
+ if (res->cmd_num > GRO_DEFAULT_FLUSH_INTERVAL) {
+ printf("The interval value should be in the range"
+ " of 0 to %u. Revert to the default"
+ " value %u\n",
+ GRO_MAX_FLUSH_INTERVAL,
+ GRO_DEFAULT_FLUSH_INTERVAL);
+ gro_flush_interval = GRO_DEFAULT_FLUSH_INTERVAL;
+ } else
+ gro_flush_interval = res->cmd_num;
+ }
+}
+
+cmdline_parse_token_string_t cmd_gro_flush_keyword =
+ TOKEN_STRING_INITIALIZER(struct cmd_gro_flush_result,
+ cmd_keyword, "gro");
+cmdline_parse_token_string_t cmd_gro_flush_flush =
+ TOKEN_STRING_INITIALIZER(struct cmd_gro_flush_result,
+ cmd_flush, "flush");
+cmdline_parse_token_string_t cmd_gro_flush_interval =
+ TOKEN_STRING_INITIALIZER(struct cmd_gro_flush_result,
+ cmd_interval, "interval");
+cmdline_parse_token_num_t cmd_gro_flush_num =
+ TOKEN_NUM_INITIALIZER(struct cmd_gro_flush_result,
+ cmd_num, UINT32);
+
+cmdline_parse_inst_t cmd_gro_flush = {
+ .f = cmd_gro_flush_parsed,
+ .data = NULL,
+ .help_str = "gro flush interval <num>",
+ .tokens = {
+ (void *)&cmd_gro_flush_keyword,
+ (void *)&cmd_gro_flush_flush,
+ (void *)&cmd_gro_flush_interval,
+ (void *)&cmd_gro_flush_num,
+ NULL,
+ },
+};
+
/* *** SET MAX FLOW NUMBER AND ITEM NUM PER FLOW FOR GRO *** */
struct cmd_gro_set_result {
cmdline_fixed_string_t gro;
@@ -14251,6 +14321,7 @@ cmdline_parse_ctx_t main_ctx[] = {
(cmdline_parse_inst_t *)&cmd_tunnel_tso_show,
(cmdline_parse_inst_t *)&cmd_enable_gro,
(cmdline_parse_inst_t *)&cmd_gro_set,
+ (cmdline_parse_inst_t *)&cmd_gro_flush,
(cmdline_parse_inst_t *)&cmd_link_flow_control_set,
(cmdline_parse_inst_t *)&cmd_link_flow_control_set_rx,
(cmdline_parse_inst_t *)&cmd_link_flow_control_set_tx,
@@ -2420,7 +2420,7 @@ set_tx_pkt_segments(unsigned *seg_lengths, unsigned nb_segs)
}
void
-setup_gro(const char *mode, uint8_t port_id)
+setup_gro(const char *mode, const char *onoff, uint8_t port_id)
{
if (!rte_eth_dev_is_valid_port(port_id)) {
printf("invalid port id %u\n", port_id);
@@ -2431,20 +2431,27 @@ setup_gro(const char *mode, uint8_t port_id)
" please stop forwarding first\n");
return;
}
- if (strcmp(mode, "on") == 0) {
- if (gro_ports[port_id].enable) {
- printf("port %u has enabled GRO\n", port_id);
+ if (strcmp(onoff, "on") == 0) {
+ if (gro_ports[port_id].enable != 0) {
+ printf("port %u has enabled GRO. Please"
+ " disable GRO first\n", port_id);
return;
}
- gro_ports[port_id].enable = 1;
- gro_ports[port_id].param.gro_types = RTE_GRO_TCP_IPV4;
-
- if (gro_ports[port_id].param.max_flow_num == 0)
- gro_ports[port_id].param.max_flow_num =
- GRO_DEFAULT_FLOW_NUM;
- if (gro_ports[port_id].param.max_item_per_flow == 0)
- gro_ports[port_id].param.max_item_per_flow =
- GRO_DEFAULT_ITEM_NUM_PER_FLOW;
+ if (strcmp(mode, "heavymode") == 0)
+ gro_ports[port_id].enable = GRO_HEAVYMODE;
+ else {
+ gro_ports[port_id].enable = GRO_LIGHTMODE;
+ gro_ports[port_id].param.gro_types = RTE_GRO_TCP_IPV4;
+
+ if (gro_ports[port_id].param.max_flow_num == 0) {
+ gro_ports[port_id].param.max_flow_num =
+ GRO_DEFAULT_FLOW_NUM;
+ }
+ if (gro_ports[port_id].param.max_item_per_flow == 0) {
+ gro_ports[port_id].param.max_item_per_flow =
+ GRO_DEFAULT_ITEM_NUM_PER_FLOW;
+ }
+ }
} else {
if (gro_ports[port_id].enable == 0) {
printf("port %u has disabled GRO\n", port_id);
@@ -631,6 +631,9 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
struct rte_mbuf *m, *p;
struct ether_hdr *eth_hdr;
void *l3_hdr = NULL, *outer_l3_hdr = NULL; /* can be IPv4 or IPv6 */
+ void *gro_ctx;
+ uint16_t gro_pkts_num;
+ uint8_t gro_enable;
uint16_t nb_rx;
uint16_t nb_tx;
uint16_t nb_prep;
@@ -657,17 +660,33 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
nb_pkt_per_burst);
if (unlikely(nb_rx == 0))
return;
- if (unlikely(gro_ports[fs->rx_port].enable))
- nb_rx = rte_gro_reassemble_burst(pkts_burst,
- nb_rx,
- &(gro_ports[fs->rx_port].param));
-
#ifdef RTE_TEST_PMD_RECORD_BURST_STATS
fs->rx_burst_stats.pkt_burst_spread[nb_rx]++;
#endif
fs->rx_packets += nb_rx;
rx_bad_ip_csum = 0;
rx_bad_l4_csum = 0;
+ gro_enable = gro_ports[fs->rx_port].enable;
+
+ if (unlikely(gro_enable == GRO_HEAVYMODE)) {
+ gro_ctx = current_fwd_lcore()->gro_ctx;
+ nb_rx = rte_gro_reassemble(pkts_burst, nb_rx, gro_ctx);
+
+ if (fs->gro_times++ >= gro_flush_interval) {
+ gro_pkts_num = rte_gro_get_pkt_count(gro_ctx);
+ if (gro_pkts_num > MAX_PKT_BURST - nb_rx)
+ gro_pkts_num = MAX_PKT_BURST - nb_rx;
+
+ nb_rx += rte_gro_timeout_flush(gro_ctx, 0,
+ gro_ports[fs->rx_port].param.gro_types,
+ &pkts_burst[nb_rx],
+ gro_pkts_num);
+ fs->gro_times = 0;
+ }
+ } else if (unlikely(gro_enable == GRO_LIGHTMODE)) {
+ nb_rx = rte_gro_reassemble_burst(pkts_burst, nb_rx,
+ &(gro_ports[fs->rx_port].param));
+ }
txp = &ports[fs->tx_port];
testpmd_ol_flags = txp->tx_ol_flags;
@@ -386,6 +386,7 @@ uint8_t bitrate_enabled;
#endif
struct gro_status gro_ports[RTE_MAX_ETHPORTS];
+uint32_t gro_flush_interval = GRO_DEFAULT_FLUSH_INTERVAL;
/* Forward function declarations */
static void map_port_queue_stats_mapping_registers(uint8_t pi, struct rte_port *port);
@@ -570,6 +571,7 @@ init_config(void)
unsigned int nb_mbuf_per_pool;
lcoreid_t lc_id;
uint8_t port_per_socket[RTE_MAX_NUMA_NODES];
+ struct rte_gro_param gro_param;
memset(port_per_socket,0,RTE_MAX_NUMA_NODES);
@@ -671,6 +673,20 @@ init_config(void)
rte_exit(EXIT_FAILURE, "FAIL from init_fwd_streams()\n");
fwd_config_setup();
+
+ /* create a gro context for each lcore */
+ gro_param.gro_types = RTE_GRO_TCP_IPV4;
+ gro_param.max_flow_num = GRO_MAX_FLUSH_INTERVAL;
+ gro_param.max_item_per_flow = MAX_PKT_BURST;
+ for (lc_id = 0; lc_id < nb_lcores; lc_id++) {
+ gro_param.socket_id = rte_lcore_to_socket_id(
+ fwd_lcores_cpuids[lc_id]);
+ fwd_lcores[lc_id]->gro_ctx = rte_gro_ctx_create(&gro_param);
+ if (fwd_lcores[lc_id]->gro_ctx == NULL) {
+ rte_exit(EXIT_FAILURE,
+ "rte_gro_ctx_create() failed\n");
+ }
+ }
}
@@ -1165,6 +1181,7 @@ start_packet_forwarding(int with_tx_first)
fwd_streams[sm_id]->fwd_dropped = 0;
fwd_streams[sm_id]->rx_bad_ip_csum = 0;
fwd_streams[sm_id]->rx_bad_l4_csum = 0;
+ fwd_streams[sm_id]->gro_times = 0;
#ifdef RTE_TEST_PMD_RECORD_BURST_STATS
memset(&fwd_streams[sm_id]->rx_burst_stats, 0,
@@ -120,6 +120,7 @@ struct fwd_stream {
unsigned int fwd_dropped; /**< received packets not forwarded */
unsigned int rx_bad_ip_csum ; /**< received packets has bad ip checksum */
unsigned int rx_bad_l4_csum ; /**< received packets has bad l4 checksum */
+ unsigned int gro_times; /**< reassembly times in heavyweight mode */
#ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
uint64_t core_cycles; /**< used for RX and TX processing */
#endif
@@ -206,6 +207,7 @@ struct rte_port {
*/
struct fwd_lcore {
struct rte_mempool *mbp; /**< The mbuf pool to use by this core */
+ void *gro_ctx; /**< GRO context */
streamid_t stream_idx; /**< index of 1st stream in "fwd_streams" */
streamid_t stream_nb; /**< number of streams in "fwd_streams" */
lcoreid_t cpuid_idx; /**< index of logical core in CPU id table */
@@ -434,13 +436,21 @@ extern struct ether_addr peer_eth_addrs[RTE_MAX_ETHPORTS];
extern uint32_t burst_tx_delay_time; /**< Burst tx delay time(us) for mac-retry. */
extern uint32_t burst_tx_retry_num; /**< Burst tx retry number for mac-retry. */
+#define GRO_HEAVYMODE 0x1
+#define GRO_LIGHTMODE 0x2
+
#define GRO_DEFAULT_FLOW_NUM 4
#define GRO_DEFAULT_ITEM_NUM_PER_FLOW DEF_PKT_BURST
+
+#define GRO_DEFAULT_FLUSH_INTERVAL 4
+#define GRO_MAX_FLUSH_INTERVAL 8
+
struct gro_status {
struct rte_gro_param param;
uint8_t enable;
};
extern struct gro_status gro_ports[RTE_MAX_ETHPORTS];
+extern uint32_t gro_flush_interval;
static inline unsigned int
lcore_num(void)
@@ -640,7 +650,7 @@ void get_2tuple_filter(uint8_t port_id, uint16_t index);
void get_5tuple_filter(uint8_t port_id, uint16_t index);
int rx_queue_id_is_invalid(queueid_t rxq_id);
int tx_queue_id_is_invalid(queueid_t txq_id);
-void setup_gro(const char *mode, uint8_t port_id);
+void setup_gro(const char *mode, const char *onoff, uint8_t port_id);
/* Functions to manage the set of filtered Multicast MAC addresses */
void mcast_addr_add(uint8_t port_id, struct ether_addr *mc_addr);