From patchwork Wed Jan 10 06:49:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: wei.guo.simon@gmail.com X-Patchwork-Id: 33351 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7B9858E01; Wed, 10 Jan 2018 07:50:08 +0100 (CET) Received: from mail-pl0-f66.google.com (mail-pl0-f66.google.com [209.85.160.66]) by dpdk.org (Postfix) with ESMTP id 62A327CC4 for ; Wed, 10 Jan 2018 07:50:07 +0100 (CET) Received: by mail-pl0-f66.google.com with SMTP id 1so6760140plv.0 for ; Tue, 09 Jan 2018 22:50:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=El7xVHFANG31ib9weWOI52xikUyxvNZWEEFGfTHt+OA=; b=vR7KeMdFI3E00lp1HXn5UTxDUb2CamEM+/+JV4TAYyuRkDpIo6HVWG24+jCKfCeXq9 bFFlJfd6xQpSHZNTVQp2lEePPTqC9TbYKrIJJ5XSsNJ8pZZLL3//8pDbOh5AMt5yeMog 0OoL3j83tXleFO3++SH7gt0C9bdeGxLrzfvzK6zCFqbk4m2vBn7zY01aDfekHAOy0jWt iIF2NZHnAB7s6umrEYzFPYqTpeiZqbrZ3L21O+7h+sXgFKhUNIMggII01JHhTHjyUkhg rCvanhzA2c04QhexJp0PmdjCPsO0NfCi2TKtZyxopy+kj2sy9Cz0mGGKDRuiI6bqZXot 7WGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=El7xVHFANG31ib9weWOI52xikUyxvNZWEEFGfTHt+OA=; b=qh1n7zOPHBVD1BWptOaxjor8Rh10g7fEd2Ja+ua56J0qye3L16hCz+sfV9rWNNx0Nh 4piSIVBr5k2ZByKmHYeJ0zyN1epE2sIKpWQlWwg/Fj3LlQycJnlZDuJs4TMMur4H8jul F4DSaXTMQse8hn1DFMoNrOleo01qLpXQBclal3Q5nT37EOGxQePMv41CYkwYrL3TXnXq YSDlx4zcEeTO8TkNMkOo5py3Jg3GJaeFF2K8JSOu6N838d4purGl3D0lheEksk3qahgT 5V3hupS5YyllLIW/eP0h1Hb1xgtZKvbXivbPSByk+QzmpQH7esHBJhWkqfpscCU22QKj jQkA== X-Gm-Message-State: AKGB3mLjbZs2nvpZoB4Nancc8znARvutS8Da/blj+quJwQYUQj0jzbMC Yd8F8vsdd4uwzxL+y53t3zg= X-Google-Smtp-Source: ACJfBot/DJPryRCpkPiOTDkiQo3fqdMZ2R3MLsslL1WTrrXXFdhMKyWWVFHCleOF+pGlcv8lrgh/fQ== X-Received: by 10.84.234.200 with SMTP id i8mr18559235plt.161.1515567006501; Tue, 09 Jan 2018 22:50:06 -0800 (PST) Received: from simonLocalRHEL7.cn.ibm.com ([112.73.0.88]) by smtp.gmail.com with ESMTPSA id d2sm34443141pfe.164.2018.01.09.22.50.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Jan 2018 22:50:05 -0800 (PST) From: wei.guo.simon@gmail.com To: Wenzhuo Lu Cc: dev@dpdk.org, Simon Guo Date: Wed, 10 Jan 2018 14:49:53 +0800 Message-Id: <1515566993-23135-1-git-send-email-wei.guo.simon@gmail.com> X-Mailer: git-send-email 1.8.3.1 Subject: [dpdk-dev] [PATCH v3] app/testpmd: add option ring-bind-lcpu to bind Q with CPU X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Simon Guo Currently the rx/tx queue is allocated from the buffer pool on socket of: - port's socket if --port-numa-config specified - or ring-numa-config setting per port All the above will "bind" queue to single socket per port configuration. But it can actually archieve better performance if one port's queue can be spread across multiple NUMA nodes, and the rx/tx queue is allocated per lcpu socket. This patch adds a new option "--ring-bind-lcpu"(no parameter). With this, testpmd can utilize the PCI-e bus bandwidth on another NUMA nodes. When --port-numa-config or --ring-numa-config option is specified, this --ring-bind-lcpu option will be suppressed. Test result: 64bytes package, running in PowerPC with Mellanox CX-4 card, single port(100G), with 8 cores, fw mode: - Without this patch: 52.5Mpps throughput - With this patch: 66Mpps throughput ~25% improvement Signed-off-by: Simon Guo --- app/test-pmd/parameters.c | 14 +++++- app/test-pmd/testpmd.c | 111 +++++++++++++++++++++++++++++++++------------- app/test-pmd/testpmd.h | 7 +++ 3 files changed, 100 insertions(+), 32 deletions(-) diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index 304b98d..1dba92e 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -104,6 +104,10 @@ "(flag: 1 for RX; 2 for TX; 3 for RX and TX).\n"); printf(" --socket-num=N: set socket from which all memory is allocated " "in NUMA mode.\n"); + printf(" --ring-bind-lcpu: " + "specify TX/RX rings will be allocated on local socket of lcpu." + "It will be ignored if ring-numa-config or port-numa-config is used. " + "As a result, it allows one port binds to multiple NUMA nodes.\n"); printf(" --mbuf-size=N: set the data size of mbuf to N bytes.\n"); printf(" --total-num-mbufs=N: set the number of mbufs to be allocated " "in mbuf pools.\n"); @@ -544,6 +548,7 @@ { "interactive", 0, 0, 0 }, { "cmdline-file", 1, 0, 0 }, { "auto-start", 0, 0, 0 }, + { "ring-bind-lcpu", 0, 0, 0 }, { "eth-peers-configfile", 1, 0, 0 }, { "eth-peer", 1, 0, 0 }, #endif @@ -676,6 +681,10 @@ stats_period = n; break; } + if (!strcmp(lgopts[opt_idx].name, "ring-bind-lcpu")) { + ring_bind_lcpu |= RBL_BIND_LOCAL_MASK; + break; + } if (!strcmp(lgopts[opt_idx].name, "eth-peers-configfile")) { if (init_peer_eth_addrs(optarg) != 0) @@ -739,11 +748,14 @@ if (parse_portnuma_config(optarg)) rte_exit(EXIT_FAILURE, "invalid port-numa configuration\n"); + ring_bind_lcpu |= RBL_PORT_NUMA_MASK; } - if (!strcmp(lgopts[opt_idx].name, "ring-numa-config")) + if (!strcmp(lgopts[opt_idx].name, "ring-numa-config")) { if (parse_ringnuma_config(optarg)) rte_exit(EXIT_FAILURE, "invalid ring-numa configuration\n"); + ring_bind_lcpu |= RBL_RING_NUMA_MASK; + } if (!strcmp(lgopts[opt_idx].name, "socket-num")) { n = atoi(optarg); if (!new_socket_id((uint8_t)n)) { diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 9414d0e..75553b9 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -68,6 +68,9 @@ uint8_t interactive = 0; uint8_t auto_start = 0; uint8_t tx_first; + +uint8_t ring_bind_lcpu; + char cmdline_filename[PATH_MAX] = {0}; /* @@ -1410,6 +1413,43 @@ static int eth_event_callback(portid_t port_id, return 1; } +static int find_local_socket(queueid_t qi, int is_rxq) +{ + /* + * try to find the local socket with following logic: + * 1) Find the correct stream for the queue; + * 2) Find the correct lcore for the stream; + * 3) Find the correct socket for the lcore; + */ + int i, j, socket = NUMA_NO_CONFIG; + + /* find the stream based on queue no*/ + for (i = 0; i < nb_fwd_streams; i++) { + if (is_rxq) { + if (fwd_streams[i]->rx_queue == qi) + break; + } else { + if (fwd_streams[i]->tx_queue == qi) + break; + } + } + if (i == nb_fwd_streams) + return NUMA_NO_CONFIG; + + /* find the lcore based on stream idx */ + for (j = 0; j < nb_lcores; j++) { + if (fwd_lcores[j]->stream_idx == i) + break; + } + if (j == nb_lcores) + return NUMA_NO_CONFIG; + + /* find the socket for the lcore */ + socket = rte_lcore_to_socket_id(fwd_lcores_cpuids[j]); + + return socket; +} + int start_port(portid_t pid) { @@ -1469,14 +1509,18 @@ static int eth_event_callback(portid_t port_id, port->need_reconfig_queues = 0; /* setup tx queues */ for (qi = 0; qi < nb_txq; qi++) { + int socket = port->socket_id; if ((numa_support) && (txring_numa[pi] != NUMA_NO_CONFIG)) - diag = rte_eth_tx_queue_setup(pi, qi, - nb_txd,txring_numa[pi], - &(port->tx_conf)); - else - diag = rte_eth_tx_queue_setup(pi, qi, - nb_txd,port->socket_id, + socket = txring_numa[pi]; + else if (ring_bind_lcpu) { + int ret = find_local_socket(qi, 0); + if (ret != NUMA_NO_CONFIG) + socket = ret; + } + + diag = rte_eth_tx_queue_setup(pi, qi, + nb_txd, socket, &(port->tx_conf)); if (diag == 0) @@ -1495,35 +1539,28 @@ static int eth_event_callback(portid_t port_id, } /* setup rx queues */ for (qi = 0; qi < nb_rxq; qi++) { + int socket = port->socket_id; if ((numa_support) && - (rxring_numa[pi] != NUMA_NO_CONFIG)) { - struct rte_mempool * mp = - mbuf_pool_find(rxring_numa[pi]); - if (mp == NULL) { - printf("Failed to setup RX queue:" - "No mempool allocation" - " on the socket %d\n", - rxring_numa[pi]); - return -1; - } - - diag = rte_eth_rx_queue_setup(pi, qi, - nb_rxd,rxring_numa[pi], - &(port->rx_conf),mp); - } else { - struct rte_mempool *mp = - mbuf_pool_find(port->socket_id); - if (mp == NULL) { - printf("Failed to setup RX queue:" + (rxring_numa[pi] != NUMA_NO_CONFIG)) + socket = rxring_numa[pi]; + else if (ring_bind_lcpu) { + int ret = find_local_socket(qi, 1); + if (ret != NUMA_NO_CONFIG) + socket = ret; + } + + struct rte_mempool *mp = + mbuf_pool_find(socket); + if (mp == NULL) { + printf("Failed to setup RX queue:" "No mempool allocation" " on the socket %d\n", - port->socket_id); - return -1; - } - diag = rte_eth_rx_queue_setup(pi, qi, - nb_rxd,port->socket_id, - &(port->rx_conf), mp); + socket); + return -1; } + diag = rte_eth_rx_queue_setup(pi, qi, + nb_rxd, socket, + &(port->rx_conf), mp); if (diag == 0) continue; @@ -2414,6 +2451,18 @@ uint8_t port_is_bonding_slave(portid_t slave_pid) "but nb_txq=%d will prevent to fully test it.\n", nb_rxq, nb_txq); + if (ring_bind_lcpu & RBL_BIND_LOCAL_MASK) { + if (ring_bind_lcpu & (RBL_RING_NUMA_MASK | RBL_PORT_NUMA_MASK)) { + printf("ring-bind-lcpu option is suppressed by " + "ring-numa-config or port-numa-config option\n"); + ring_bind_lcpu = 0; + } else { + printf("RingBuffer bind with local CPU selected\n"); + ring_bind_lcpu = 1; + } + } else + ring_bind_lcpu = 0; + init_config(); if (start_port(RTE_PORT_ALL) != 0) rte_exit(EXIT_FAILURE, "Start ports failed\n"); diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index 2a266fd..99e55b2 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -328,6 +328,13 @@ struct queue_stats_mappings { extern uint8_t interactive; extern uint8_t auto_start; extern uint8_t tx_first; + +/* for --ring-bind-lcpu option checking, RBL means Ring Bind Lcpu related */ +#define RBL_BIND_LOCAL_MASK (1 << 0) +#define RBL_RING_NUMA_MASK (1 << 1) +#define RBL_PORT_NUMA_MASK (1 << 2) +extern uint8_t ring_bind_lcpu; + extern char cmdline_filename[PATH_MAX]; /**< offline commands file */ extern uint8_t numa_support; /**< set by "--numa" parameter */ extern uint16_t port_topology; /**< set by "--port-topology" parameter */