From patchwork Fri Jun 30 12:37:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: wei.guo.simon@gmail.com X-Patchwork-Id: 26089 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 96BF14CC5; Fri, 30 Jun 2017 14:38:11 +0200 (CEST) Received: from mail-pf0-f193.google.com (mail-pf0-f193.google.com [209.85.192.193]) by dpdk.org (Postfix) with ESMTP id C64D7235 for ; Fri, 30 Jun 2017 14:38:09 +0200 (CEST) Received: by mail-pf0-f193.google.com with SMTP id z6so16968172pfk.3 for ; Fri, 30 Jun 2017 05:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=MSGhQMQs+FR2MV5q84NRiK1ljAXkhez9UwdcptWpSrE=; b=tvyQVwp2bPrq00dbNo/9ShgLTQH1Vpn8Sh9uiPbojFLuK+Uu2UaMPRnGpap/HdUiIs nsxEmvTzXPhs9NrhRQWZwkQaFI9VHNHCD2LanjdHkhZoEjBRbOuhtlvmdm+U9gsiE92s v+obL6IC5E7/V8Yodg5p5DOKrmSHDnK2jCcu6WXExMOoGhVk8BgABJxXb7bl+okO14bK D4rehI3B1sXk6WI5FhNmX7qz+AJfh5+0PQAdo7+8ryDBzc4YSYse4cVgPXmIhT2PRW/O 90PT4uTvNCaGejQokHyBt/8mwQ0Pru6VQ1Kc2UR0BPlLoTCEe9pXAKd/ffrufw7PB+cM E+6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=MSGhQMQs+FR2MV5q84NRiK1ljAXkhez9UwdcptWpSrE=; b=EOSEKCplwYTWUy3q0bTXrDsdoBytEa/FeBKUqeMYnzj/FD/zhXovNh119fSQaE5hi4 eVCB8XMqFtTA9u7R5gH8+B4p7Ung6VJc2FAuYxmdwUZfOEshHCccarOoipXiAeyh4Arz XWcX6lwFdduSs2jPEuautuQiI8m2b/uRROMS1+9eYIwmmuJUW3yZt2q3991chqgLs4fM fajkwLonN1e5aFUcoC6GfwhvHV/QCYA58iGZ1sACxGTlIahLYkKzzHMs8K1akIcok3fa VA6J1DQTgNpSvLoJYn1ZpFmnHaSdsOBK9fqbzmX49nsIALRfCS8egm2mvRu8hpb4NICU s2BQ== X-Gm-Message-State: AKS2vOxcOiXxTNa7jY1xAP0nYARy3gfSdWNl4Yl+e+XmmUiJ19Vpmt61 A5KaDV+ul14ZZA== X-Received: by 10.84.131.163 with SMTP id d32mr24181579pld.16.1498826288965; Fri, 30 Jun 2017 05:38:08 -0700 (PDT) Received: from simonLocalRHEL7.x64.cn.ibm.com ([116.230.182.250]) by smtp.gmail.com with ESMTPSA id 188sm13538617pgc.49.2017.06.30.05.38.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Jun 2017 05:38:08 -0700 (PDT) From: wei.guo.simon@gmail.com To: Jingjing Wu Cc: dev@dpdk.org, Simon Guo Date: Fri, 30 Jun 2017 20:37:54 +0800 Message-Id: <1498826274-1769-1-git-send-email-wei.guo.simon@gmail.com> X-Mailer: git-send-email 1.8.3.1 Subject: [dpdk-dev] [RFC PATCH] app/testpmd: add option ring-bind-lcpu to bind Q with CPU X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Simon Guo Currently the rx/tx queue is allocated from the buffer pool on socket of: - port's socket if --port-numa-config specified - or ring-numa-config setting per port All the above will "bind" queue to single socket per port configuration. But it can actually archieve better performance if one port's queue can be spread across multiple NUMA nodes, and the rx/tx queue is allocated per lcpu socket. With this patch, testpmd can utilize the PCI-e bus bandwidth on another NUMA nodes. with 64bytes package, When running in PowerPC with Mellanox CX-4 card, single port(100G), with 8 cores, fw mode: - Without this patch: 52.5Mpps throughput - With this patch: 66Mpps throughput Signed-off-by: Simon Guo --- app/test-pmd/parameters.c | 10 +++++ app/test-pmd/testpmd.c | 102 ++++++++++++++++++++++++++++++++-------------- app/test-pmd/testpmd.h | 1 + 3 files changed, 82 insertions(+), 31 deletions(-) diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index fbe6284..d3ab946 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -130,6 +130,11 @@ "(flag: 1 for RX; 2 for TX; 3 for RX and TX).\n"); printf(" --socket-num=N: set socket from which all memory is allocated " "in NUMA mode.\n"); + printf(" --ring-bind-lcpu: " + " specify TX/RX rings will be allocated on local socket of lcpu." + " It will overriden ring-numa-config or port-numa-config if success." + " If local ring buffer is unavailable, it will use --ring-numa-config or port-numa-config instead." + " It allows one port binds to multiple NUMA nodes. \n"); printf(" --mbuf-size=N: set the data size of mbuf to N bytes.\n"); printf(" --total-num-mbufs=N: set the number of mbufs to be allocated " "in mbuf pools.\n"); @@ -563,6 +568,7 @@ { "interactive", 0, 0, 0 }, { "cmdline-file", 1, 0, 0 }, { "auto-start", 0, 0, 0 }, + { "ring-bind-lcpu", 0, 0, 0 }, { "eth-peers-configfile", 1, 0, 0 }, { "eth-peer", 1, 0, 0 }, #endif @@ -674,6 +680,10 @@ printf("Auto-start selected\n"); auto_start = 1; } + if (!strcmp(lgopts[opt_idx].name, "ring-bind-lcpu")) { + printf("RingBuffer bind with local CPU selected\n"); + ring_bind_lcpu = 1; + } if (!strcmp(lgopts[opt_idx].name, "eth-peers-configfile")) { if (init_peer_eth_addrs(optarg) != 0) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index b29328a..92eed8f 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -98,6 +98,7 @@ /* use master core for command line ? */ uint8_t interactive = 0; uint8_t auto_start = 0; +uint8_t ring_bind_lcpu = 0; char cmdline_filename[PATH_MAX] = {0}; /* @@ -1395,6 +1396,46 @@ static void eth_event_callback(uint8_t port_id, return 1; } +static int find_local_socket(queueid_t qi, int is_rxq) +{ + /* + * try to find the local mp with following logic: + * 1) Find the correct stream for the queue; + * 2) Find the correct lcore for the stream; + * 3) Find the correct socket for the lcore; + * 4) Find the correct mp for the scoket; + * + * If failed, failover to the old implementation. + */ + int i, j, socket = NUMA_NO_CONFIG; + + /* find the stream based on queue no*/ + for (i = 0; i < nb_fwd_streams; i++) { + if (is_rxq) { + if (fwd_streams[i]->rx_queue == qi) + break; + } else { + if (fwd_streams[i]->tx_queue == qi) + break; + } + } + if (i == nb_fwd_streams) + return NUMA_NO_CONFIG; + + /* find the lcore based on stream idx */ + for (j = 0; j < nb_lcores; j++) { + if (fwd_lcores[j]->stream_idx == i) + break; + } + if (j == nb_lcores) + return NUMA_NO_CONFIG; + + /* find the scoket for the lcore */ + socket = rte_lcore_to_socket_id(fwd_lcores_cpuids[j]); + + return socket; +} + int start_port(portid_t pid) { @@ -1445,14 +1486,19 @@ static void eth_event_callback(uint8_t port_id, port->need_reconfig_queues = 0; /* setup tx queues */ for (qi = 0; qi < nb_txq; qi++) { + int socket = port->socket_id; if ((numa_support) && (txring_numa[pi] != NUMA_NO_CONFIG)) - diag = rte_eth_tx_queue_setup(pi, qi, - nb_txd,txring_numa[pi], - &(port->tx_conf)); - else - diag = rte_eth_tx_queue_setup(pi, qi, - nb_txd,port->socket_id, + socket = txring_numa[pi]; + + if (ring_bind_lcpu) { + int ret = find_local_socket(qi, 0); + if (ret != NUMA_NO_CONFIG) + socket = ret; + } + + diag = rte_eth_tx_queue_setup(pi, qi, + nb_txd, socket, &(port->tx_conf)); if (diag == 0) @@ -1471,35 +1517,29 @@ static void eth_event_callback(uint8_t port_id, } /* setup rx queues */ for (qi = 0; qi < nb_rxq; qi++) { + int socket = port->socket_id; if ((numa_support) && - (rxring_numa[pi] != NUMA_NO_CONFIG)) { - struct rte_mempool * mp = - mbuf_pool_find(rxring_numa[pi]); - if (mp == NULL) { - printf("Failed to setup RX queue:" - "No mempool allocation" - " on the socket %d\n", - rxring_numa[pi]); - return -1; - } - - diag = rte_eth_rx_queue_setup(pi, qi, - nb_rxd,rxring_numa[pi], - &(port->rx_conf),mp); - } else { - struct rte_mempool *mp = - mbuf_pool_find(port->socket_id); - if (mp == NULL) { - printf("Failed to setup RX queue:" + (rxring_numa[pi] != NUMA_NO_CONFIG)) + socket = rxring_numa[pi]; + + if (ring_bind_lcpu) { + int ret = find_local_socket(qi, 1); + if (ret != NUMA_NO_CONFIG) + socket = ret; + } + + struct rte_mempool *mp = + mbuf_pool_find(socket); + if (mp == NULL) { + printf("Failed to setup RX queue:" "No mempool allocation" " on the socket %d\n", - port->socket_id); - return -1; - } - diag = rte_eth_rx_queue_setup(pi, qi, - nb_rxd,port->socket_id, - &(port->rx_conf), mp); + socket); + return -1; } + diag = rte_eth_rx_queue_setup(pi, qi, + nb_rxd,socket, + &(port->rx_conf), mp); if (diag == 0) continue; diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index 364502d..c45b2b8 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -299,6 +299,7 @@ struct queue_stats_mappings { extern uint16_t verbose_level; /**< Drives messages being displayed, if any. */ extern uint8_t interactive; extern uint8_t auto_start; +extern uint8_t ring_bind_lcpu; extern char cmdline_filename[PATH_MAX]; /**< offline commands file */ extern uint8_t numa_support; /**< set by "--numa" parameter */ extern uint16_t port_topology; /**< set by "--port-topology" parameter */