From patchwork Tue Jan 24 09:06:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "De Lara Guarch, Pablo" X-Patchwork-Id: 19931 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 2435A108F; Tue, 24 Jan 2017 10:04:45 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 14BE8108D for ; Tue, 24 Jan 2017 10:04:41 +0100 (CET) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga101.fm.intel.com with ESMTP; 24 Jan 2017 01:04:40 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,277,1477983600"; d="scan'208";a="34651954" Received: from silpixa00381631.ir.intel.com (HELO silpixa00381631.ger.corp.intel.com) ([10.237.222.122]) by orsmga002.jf.intel.com with ESMTP; 24 Jan 2017 01:04:37 -0800 From: Pablo de Lara To: dev@dpdk.org Cc: Pablo de Lara Date: Tue, 24 Jan 2017 09:06:14 +0000 Message-Id: <1485248774-91712-1-git-send-email-pablo.de.lara.guarch@intel.com> X-Mailer: git-send-email 2.7.4 Subject: [dpdk-dev] [PATCH] examples/flow-distributor: rename to server_node_efd X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" To avoid confusion with distributor app, this commit renames the flow-distributor sample app to server_node_efd, since it shows how to use the EFD library and it is based on a server/nodes model. Signed-off-by: Pablo de Lara --- MAINTAINERS | 4 +- doc/api/examples.dox | 8 +- doc/guides/sample_app_ug/flow_distributor.rst | 494 -------- doc/guides/sample_app_ug/img/flow_distributor.svg | 1254 --------------------- doc/guides/sample_app_ug/img/server_node_efd.svg | 1254 +++++++++++++++++++++ doc/guides/sample_app_ug/index.rst | 4 +- doc/guides/sample_app_ug/server_node_efd.rst | 494 ++++++++ examples/flow_distributor/Makefile | 44 - examples/flow_distributor/distributor/Makefile | 57 - examples/flow_distributor/distributor/args.c | 200 ---- examples/flow_distributor/distributor/args.h | 39 - examples/flow_distributor/distributor/init.c | 371 ------ examples/flow_distributor/distributor/init.h | 76 -- examples/flow_distributor/distributor/main.c | 362 ------ examples/flow_distributor/node/Makefile | 48 - examples/flow_distributor/node/node.c | 417 ------- examples/flow_distributor/shared/common.h | 99 -- examples/server_node_efd/Makefile | 44 + examples/server_node_efd/node/Makefile | 48 + examples/server_node_efd/node/node.c | 417 +++++++ examples/server_node_efd/server/Makefile | 57 + examples/server_node_efd/server/args.c | 200 ++++ examples/server_node_efd/server/args.h | 39 + examples/server_node_efd/server/init.c | 371 ++++++ examples/server_node_efd/server/init.h | 76 ++ examples/server_node_efd/server/main.c | 362 ++++++ examples/server_node_efd/shared/common.h | 99 ++ 27 files changed, 3470 insertions(+), 3468 deletions(-) delete mode 100644 doc/guides/sample_app_ug/flow_distributor.rst delete mode 100644 doc/guides/sample_app_ug/img/flow_distributor.svg create mode 100644 doc/guides/sample_app_ug/img/server_node_efd.svg create mode 100644 doc/guides/sample_app_ug/server_node_efd.rst delete mode 100644 examples/flow_distributor/Makefile delete mode 100644 examples/flow_distributor/distributor/Makefile delete mode 100644 examples/flow_distributor/distributor/args.c delete mode 100644 examples/flow_distributor/distributor/args.h delete mode 100644 examples/flow_distributor/distributor/init.c delete mode 100644 examples/flow_distributor/distributor/init.h delete mode 100644 examples/flow_distributor/distributor/main.c delete mode 100644 examples/flow_distributor/node/Makefile delete mode 100644 examples/flow_distributor/node/node.c delete mode 100644 examples/flow_distributor/shared/common.h create mode 100644 examples/server_node_efd/Makefile create mode 100644 examples/server_node_efd/node/Makefile create mode 100644 examples/server_node_efd/node/node.c create mode 100644 examples/server_node_efd/server/Makefile create mode 100644 examples/server_node_efd/server/args.c create mode 100644 examples/server_node_efd/server/args.h create mode 100644 examples/server_node_efd/server/init.c create mode 100644 examples/server_node_efd/server/init.h create mode 100644 examples/server_node_efd/server/main.c create mode 100644 examples/server_node_efd/shared/common.h diff --git a/MAINTAINERS b/MAINTAINERS index f071138..cfbec21 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -551,8 +551,8 @@ M: Pablo de Lara Guarch F: lib/librte_efd/ F: doc/guides/prog_guide/efd_lib.rst F: app/test/test_efd* -F: examples/flow_distributor/ -F: doc/guides/sample_app_ug/flow_distributor.rst +F: examples/server_node_efd/ +F: doc/guides/sample_app_ug/server_node_efd.rst Hashes M: Bruce Richardson diff --git a/doc/api/examples.dox b/doc/api/examples.dox index c13e574..9a48f5f 100644 --- a/doc/api/examples.dox +++ b/doc/api/examples.dox @@ -52,10 +52,6 @@ @example load_balancer/init.c @example load_balancer/main.c @example load_balancer/runtime.c -@example flow_distributor/distributor/args.c -@example flow_distributor/distributor/init.c -@example flow_distributor/distributor/main.c -@example flow_distributor/node/node.c @example multi_process/client_server_mp/mp_client/client.c @example multi_process/client_server_mp/mp_server/args.c @example multi_process/client_server_mp/mp_server/init.c @@ -94,6 +90,10 @@ @example quota_watermark/qw/init.c @example quota_watermark/qw/main.c @example rxtx_callbacks/main.c +@example server_node_efd/server/args.c +@example server_node_efd/server/init.c +@example server_node_efd/server/main.c +@example server_node_efd/node/node.c @example skeleton/basicfwd.c @example tep_termination/main.c @example tep_termination/vxlan.c diff --git a/doc/guides/sample_app_ug/flow_distributor.rst b/doc/guides/sample_app_ug/flow_distributor.rst deleted file mode 100644 index a12df76..0000000 --- a/doc/guides/sample_app_ug/flow_distributor.rst +++ /dev/null @@ -1,494 +0,0 @@ -.. BSD LICENSE - Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Intel Corporation nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -Flow Distributor Sample Application -=================================== - -This sample application demonstrates the use of EFD library as a flow-level -load balancer, for more information about the EFD Library please refer to the -DPDK programmer's guide. - -This sample application is a variant of the -:ref:`client-server sample application ` -where a specific target node is specified for every and each flow -(not in a round-robin fashion as the original load balancing sample application). - -Overview --------- - -The architecture of the EFD flow-based load balancer sample application is -presented in the following figure. - -.. _figure_efd_sample_app_overview: - -.. figure:: img/flow_distributor.* - - Using EFD as a Flow-Level Load Balancer - -As shown in :numref:`figure_efd_sample_app_overview`, -the sample application consists of a front-end node (distributor) -using the EFD library to create a load-balancing table for flows, -for each flow a target backend worker node is specified. The EFD table does not -store the flow key (unlike a regular hash table), and hence, it can -individually load-balance millions of flows (number of targets * maximum number -of flows fit in a flow table per target) while still fitting in CPU cache. - -It should be noted that although they are referred to as nodes, the frontend -distributor and worker nodes are processes running on the same platform. - -Front-end Distributor -~~~~~~~~~~~~~~~~~~~~~ - -Upon initializing, the frontend distributor node (process) creates a flow -distributor table (based on the EFD library) which is populated with flow -information and its intended target node. - -The sample application assigns a specific target node_id (process) for each of -the IP destination addresses as follows: - -.. code-block:: c - - node_id = i % num_nodes; /* Target node id is generated */ - ip_dst = rte_cpu_to_be_32(i); /* Specific ip destination address is - assigned to this target node */ - -then the pair of is inserted into the flow distribution table. - -The main loop of the the distributor node receives a burst of packets, then for -each packet, a flow key (IP destination address) is extracted. The flow -distributor table is looked up and the target node id is returned. Packets are -then enqueued to the specified target node id. - -It should be noted that flow distributor table is not a membership test table. -I.e. if the key has already been inserted the target node id will be correct, -but for new keys the flow distributor table will return a value (which can be -valid). - -Backend Worker Nodes -~~~~~~~~~~~~~~~~~~~~ - -Upon initializing, the worker node (process) creates a flow table (a regular -hash table that stores the key default size 1M flows) which is populated with -only the flow information that is serviced at this node. This flow key is -essential to point out new keys that have not been inserted before. - -The worker node's main loop is simply receiving packets then doing a hash table -lookup. If a match occurs then statistics are updated for flows serviced by -this node. If no match is found in the local hash table then this indicates -that this is a new flow, which is dropped. - - -Compiling the Application -------------------------- - -The sequence of steps used to build the application is: - -#. Export the required environment variables: - - .. code-block:: console - - export RTE_SDK=/path/to/rte_sdk - export RTE_TARGET=x86_64-native-linuxapp-gcc - -#. Build the application executable file: - - .. code-block:: console - - cd ${RTE_SDK}/examples/flow_distributor/ - make - - For more details on how to build the DPDK libraries and sample - applications, - please refer to the *DPDK Getting Started Guide.* - - -Running the Application ------------------------ - -The application has two binaries to be run: the front-end distributor -and the back-end node. - -The frontend distributor (distributor) has the following command line options:: - - ./distributor [EAL options] -- -p PORTMASK -n NUM_NODES -f NUM_FLOWS - -Where, - -* ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure -* ``-n NUM_NODES:`` Number of back-end nodes that will be used -* ``-f NUM_FLOWS:`` Number of flows to be added in the EFD table (1 million, by default) - -The back-end node (node) has the following command line options:: - - ./node [EAL options] -- -n NODE_ID - -Where, - -* ``-n NODE_ID:`` Node ID, which cannot be equal or higher than NUM_MODES - - -First, the distributor app must be launched, with the number of nodes that will be run. -Once it has been started, the node instances can be run, with different NODE_ID. -These instances have to be run as secondary processes, with ``--proc-type=secondary`` -in the EAL options, which will attach to the primary process memory, and therefore, -they can access the queues created by the primary process to distribute packets. - -To successfully run the application, the command line used to start the -application has to be in sync with the traffic flows configured on the traffic -generator side. - -For examples of application command lines and traffic generator flows, please -refer to the DPDK Test Report. For more details on how to set up and run the -sample applications provided with DPDK package, please refer to the -:ref:`DPDK Getting Started Guide for Linux ` and -:ref:`DPDK Getting Started Guide for FreeBSD `. - - -Explanation ------------ - -As described in previous sections, there are two processes in this example. - -The first process, the front-end distributor, creates and populates the EFD table, -which is used to distribute packets to nodes, which the number of flows -specified in the command line (1 million, by default). - - -.. code-block:: c - - static void - create_flow_distributor_table(void) - { - uint8_t socket_id = rte_socket_id(); - - /* create table */ - efd_table = rte_efd_create("flow table", num_flows * 2, sizeof(uint32_t), - 1 << socket_id, socket_id); - - if (efd_table == NULL) - rte_exit(EXIT_FAILURE, "Problem creating the flow table\n"); - } - - static void - populate_flow_distributor_table(void) - { - unsigned int i; - int32_t ret; - uint32_t ip_dst; - uint8_t socket_id = rte_socket_id(); - uint64_t node_id; - - /* Add flows in table */ - for (i = 0; i < num_flows; i++) { - node_id = i % num_nodes; - - ip_dst = rte_cpu_to_be_32(i); - ret = rte_efd_update(efd_table, socket_id, - (void *)&ip_dst, (efd_value_t)node_id); - if (ret < 0) - rte_exit(EXIT_FAILURE, "Unable to add entry %u in " - "flow distributor table\n", i); - } - - printf("EFD table: Adding 0x%x keys\n", num_flows); - } - -After initialization, packets are received from the enabled ports, and the IPv4 -address from the packets is used as a key to look up in the EFD table, -which tells the node where the packet has to be distributed. - -.. code-block:: c - - static void - process_packets(uint32_t port_num __rte_unused, struct rte_mbuf *pkts[], - uint16_t rx_count, unsigned int socket_id) - { - uint16_t i; - uint8_t node; - efd_value_t data[EFD_BURST_MAX]; - const void *key_ptrs[EFD_BURST_MAX]; - - struct ipv4_hdr *ipv4_hdr; - uint32_t ipv4_dst_ip[EFD_BURST_MAX]; - - for (i = 0; i < rx_count; i++) { - /* Handle IPv4 header.*/ - ipv4_hdr = rte_pktmbuf_mtod_offset(pkts[i], struct ipv4_hdr *, - sizeof(struct ether_hdr)); - ipv4_dst_ip[i] = ipv4_hdr->dst_addr; - key_ptrs[i] = (void *)&ipv4_dst_ip[i]; - } - - rte_efd_lookup_bulk(efd_table, socket_id, rx_count, - (const void **) key_ptrs, data); - for (i = 0; i < rx_count; i++) { - node = (uint8_t) ((uintptr_t)data[i]); - - if (node >= num_nodes) { - /* - * Node is out of range, which means that - * flow has not been inserted - */ - flow_dist_stats.drop++; - rte_pktmbuf_free(pkts[i]); - } else { - flow_dist_stats.distributed++; - enqueue_rx_packet(node, pkts[i]); - } - } - - for (i = 0; i < num_nodes; i++) - flush_rx_queue(i); - } - -The burst of packets received is enqueued in temporary buffers (per node), -and enqueued in the shared ring between the distributor and the node. -After this, a new burst of packets is received and this process is -repeated infinitely. - -.. code-block:: c - - static void - flush_rx_queue(uint16_t node) - { - uint16_t j; - struct node *cl; - - if (cl_rx_buf[node].count == 0) - return; - - cl = &nodes[node]; - if (rte_ring_enqueue_bulk(cl->rx_q, (void **)cl_rx_buf[node].buffer, - cl_rx_buf[node].count) != 0){ - for (j = 0; j < cl_rx_buf[node].count; j++) - rte_pktmbuf_free(cl_rx_buf[node].buffer[j]); - cl->stats.rx_drop += cl_rx_buf[node].count; - } else - cl->stats.rx += cl_rx_buf[node].count; - - cl_rx_buf[node].count = 0; - } - -The second process, the back-end node, receives the packets from the shared -ring with the distributor and send them out, if they belong to the node. - -At initialization, it attaches to the distributor process memory, to have -access to the shared ring, parameters and statistics. - -.. code-block:: c - - rx_ring = rte_ring_lookup(get_rx_queue_name(node_id)); - if (rx_ring == NULL) - rte_exit(EXIT_FAILURE, "Cannot get RX ring - " - "is distributor process running?\n"); - - mp = rte_mempool_lookup(PKTMBUF_POOL_NAME); - if (mp == NULL) - rte_exit(EXIT_FAILURE, "Cannot get mempool for mbufs\n"); - - mz = rte_memzone_lookup(MZ_SHARED_INFO); - if (mz == NULL) - rte_exit(EXIT_FAILURE, "Cannot get port info structure\n"); - info = mz->addr; - tx_stats = &(info->tx_stats[node_id]); - filter_stats = &(info->filter_stats[node_id]); - -Then, the hash table that contains the flows that will be handled -by the node is created and populated. - -.. code-block:: c - - static struct rte_hash * - create_hash_table(const struct shared_info *info) - { - uint32_t num_flows_node = info->num_flows / info->num_nodes; - char name[RTE_HASH_NAMESIZE]; - struct rte_hash *h; - - /* create table */ - struct rte_hash_parameters hash_params = { - .entries = num_flows_node * 2, /* table load = 50% */ - .key_len = sizeof(uint32_t), /* Store IPv4 dest IP address */ - .socket_id = rte_socket_id(), - .hash_func_init_val = 0, - }; - - snprintf(name, sizeof(name), "hash_table_%d", node_id); - hash_params.name = name; - h = rte_hash_create(&hash_params); - - if (h == NULL) - rte_exit(EXIT_FAILURE, - "Problem creating the hash table for node %d\n", - node_id); - return h; - } - - static void - populate_hash_table(const struct rte_hash *h, const struct shared_info *info) - { - unsigned int i; - int32_t ret; - uint32_t ip_dst; - uint32_t num_flows_node = 0; - uint64_t target_node; - - /* Add flows in table */ - for (i = 0; i < info->num_flows; i++) { - target_node = i % info->num_nodes; - if (target_node != node_id) - continue; - - ip_dst = rte_cpu_to_be_32(i); - - ret = rte_hash_add_key(h, (void *) &ip_dst); - if (ret < 0) - rte_exit(EXIT_FAILURE, "Unable to add entry %u " - "in hash table\n", i); - else - num_flows_node++; - - } - - printf("Hash table: Adding 0x%x keys\n", num_flows_node); - } - -After initialization, packets are dequeued from the shared ring -(from the distributor) and, like in the distributor process, -the IPv4 address from the packets is used as a key to look up in the hash table. -If there is a hit, packet is stored in a buffer, to be eventually transmitted -in one of the enabled ports. If key is not there, packet is dropped, since the -flow is not handled by the node. - -.. code-block:: c - - static inline void - handle_packets(struct rte_hash *h, struct rte_mbuf **bufs, uint16_t num_packets) - { - struct ipv4_hdr *ipv4_hdr; - uint32_t ipv4_dst_ip[PKT_READ_SIZE]; - const void *key_ptrs[PKT_READ_SIZE]; - unsigned int i; - int32_t positions[PKT_READ_SIZE] = {0}; - - for (i = 0; i < num_packets; i++) { - /* Handle IPv4 header.*/ - ipv4_hdr = rte_pktmbuf_mtod_offset(bufs[i], struct ipv4_hdr *, - sizeof(struct ether_hdr)); - ipv4_dst_ip[i] = ipv4_hdr->dst_addr; - key_ptrs[i] = &ipv4_dst_ip[i]; - } - /* Check if packets belongs to any flows handled by this node */ - rte_hash_lookup_bulk(h, key_ptrs, num_packets, positions); - - for (i = 0; i < num_packets; i++) { - if (likely(positions[i] >= 0)) { - filter_stats->passed++; - transmit_packet(bufs[i]); - } else { - filter_stats->drop++; - /* Drop packet, as flow is not handled by this node */ - rte_pktmbuf_free(bufs[i]); - } - } - } - -Finally, note that both processes updates statistics, such as transmitted, received -and dropped packets, which are shown and refreshed by the distributor app. - -.. code-block:: c - - static void - do_stats_display(void) - { - unsigned int i, j; - const char clr[] = {27, '[', '2', 'J', '\0'}; - const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'}; - uint64_t port_tx[RTE_MAX_ETHPORTS], port_tx_drop[RTE_MAX_ETHPORTS]; - uint64_t node_tx[MAX_NODES], node_tx_drop[MAX_NODES]; - - /* to get TX stats, we need to do some summing calculations */ - memset(port_tx, 0, sizeof(port_tx)); - memset(port_tx_drop, 0, sizeof(port_tx_drop)); - memset(node_tx, 0, sizeof(node_tx)); - memset(node_tx_drop, 0, sizeof(node_tx_drop)); - - for (i = 0; i < num_nodes; i++) { - const struct tx_stats *tx = &info->tx_stats[i]; - - for (j = 0; j < info->num_ports; j++) { - const uint64_t tx_val = tx->tx[info->id[j]]; - const uint64_t drop_val = tx->tx_drop[info->id[j]]; - - port_tx[j] += tx_val; - port_tx_drop[j] += drop_val; - node_tx[i] += tx_val; - node_tx_drop[i] += drop_val; - } - } - - /* Clear screen and move to top left */ - printf("%s%s", clr, topLeft); - - printf("PORTS\n"); - printf("-----\n"); - for (i = 0; i < info->num_ports; i++) - printf("Port %u: '%s'\t", (unsigned int)info->id[i], - get_printable_mac_addr(info->id[i])); - printf("\n\n"); - for (i = 0; i < info->num_ports; i++) { - printf("Port %u - rx: %9"PRIu64"\t" - "tx: %9"PRIu64"\n", - (unsigned int)info->id[i], info->rx_stats.rx[i], - port_tx[i]); - } - - printf("\nFLOW DISTRIBUTOR\n"); - printf("-----\n"); - printf("distributed: %9"PRIu64", drop: %9"PRIu64"\n", - flow_dist_stats.distributed, flow_dist_stats.drop); - - printf("\nNODES\n"); - printf("-------\n"); - for (i = 0; i < num_nodes; i++) { - const unsigned long long rx = nodes[i].stats.rx; - const unsigned long long rx_drop = nodes[i].stats.rx_drop; - const struct filter_stats *filter = &info->filter_stats[i]; - - printf("Node %2u - rx: %9llu, rx_drop: %9llu\n" - " tx: %9"PRIu64", tx_drop: %9"PRIu64"\n" - " filter_passed: %9"PRIu64", " - "filter_drop: %9"PRIu64"\n", - i, rx, rx_drop, node_tx[i], node_tx_drop[i], - filter->passed, filter->drop); - } - - printf("\n"); - } diff --git a/doc/guides/sample_app_ug/img/flow_distributor.svg b/doc/guides/sample_app_ug/img/flow_distributor.svg deleted file mode 100644 index 9aee30b..0000000 --- a/doc/guides/sample_app_ug/img/flow_distributor.svg +++ /dev/null @@ -1,1254 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Page-1 - - - Rectangle.58 - Key 1 - - - - - - - - - - Key 1 - - Rectangle.59 - Action 1 - - - - - - - - - - Action 1 - - Rectangle.60 - Key 2 - - - - - - - - - - Key 2 - - Rectangle.61 - Action 2 - - - - - - - - - - Action 2 - - Rectangle.62 - - - - - - - - - - Rectangle.63 - - - - - - - - - - Rectangle.64 - - - - - - - - - - Rectangle.65 - - - - - - - - - - Rectangle.66 - - - - - - - - - - Rectangle.67 - - - - - - - - - - Rectangle.68 - - - - - - - - - - Rectangle.69 - - - - - - - - - - Rectangle.70 - Key x - - - - - - - - - - Key x - - Rectangle.71 - Action x - - - - - - - - - - Action x - - Rectangle.72 - Key y - - - - - - - - - - Key y - - Rectangle.73 - Action y - - - - - - - - - - Action y - - Rectangle.74 - Key z - - - - - - - - - - Key z - - Rectangle.75 - Action z - - - - - - - - - - Action z - - Rectangle.76 - - - - - - - - - - Rectangle.77 - - - - - - - - - - Rectangle.78 - - - - - - - - - - Rectangle.79 - - - - - - - - - - Rectangle.80 - Key N - - - - - - - - - - Key N - - Rectangle.81 - Action N - - - - - - - - - - Action N - - Rectangle.82 - - - - - - - - - - Sheet.28 - Local Table for N Specific Flows Serviced at Node 1 - - - - Local Table for N Specific Flows Serviced at Node 1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Load balancer - - Sheet.35 - - - - - - - - - - - Sheet.36 - - - - - - - - Rectangle - - - - - - - - - - Sheet.38 - EFD Table - - - - EFD Table - - Rectangle.39 - Group_id - - - - - - - - - - Group_id - - Rectangle.40 - Hash index - - - - - - - - - - Hash index - - Rectangle.41 - - - - - - - - - - Rectangle.42 - - - - - - - - - - Rectangle.43 - - - - - - - - - - Rectangle.44 - - - - - - - - - - Rectangle.45 - - - - - - - - - - Rectangle.46 - - - - - - - - - - Sheet.47 - - - - Sheet.48 - Supports X*N Flows - - - - Supports X*N Flows - - Sheet.49 - Frontend Server or Load Balancer - - - - Frontend Serveror Load Balancer - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Server - - Sheet.52 - - - - - - - Sheet.53 - - - - - - - Sheet.54 - - - - - - - - - - - - Sheet.59 - - - - Sheet.60 - Backend Server 1 - - - - Backend Server 1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Server.61 - - Sheet.62 - - - - - - - Sheet.63 - - - - - - - Sheet.64 - - - - - - - - - - - - Sheet.65 - Backend Server 2 - - - - Backend Server 2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Server.66 - - Sheet.67 - - - - - - - Sheet.68 - - - - - - - Sheet.69 - - - - - - - - - - - - Sheet.70 - Backend Server X - - - - Backend Server X - - Sheet.71 - - - - Sheet.72 - - - - Rectangle.73 - Key 1 - - - - - - - - - - Key 1 - - Rectangle.74 - Action 1 - - - - - - - - - - Action 1 - - Rectangle.75 - Key 2 - - - - - - - - - - Key 2 - - Rectangle.76 - Action 2 - - - - - - - - - - Action 2 - - Rectangle.77 - - - - - - - - - - Rectangle.78 - - - - - - - - - - Rectangle.79 - - - - - - - - - - Rectangle.80 - - - - - - - - - - Rectangle.81 - - - - - - - - - - Rectangle.82 - - - - - - - - - - Rectangle.83 - - - - - - - - - - Rectangle.84 - - - - - - - - - - Rectangle.85 - Key x - - - - - - - - - - Key x - - Rectangle.86 - Action x - - - - - - - - - - Action x - - Rectangle.87 - Key y - - - - - - - - - - Key y - - Rectangle.88 - Action y - - - - - - - - - - Action y - - Rectangle.89 - Key z - - - - - - - - - - Key z - - Rectangle.90 - Action z - - - - - - - - - - Action z - - Rectangle.91 - - - - - - - - - - Rectangle.92 - - - - - - - - - - Rectangle.93 - - - - - - - - - - Rectangle.94 - - - - - - - - - - Rectangle.95 - Key N - - - - - - - - - - Key N - - Rectangle.96 - Action N - - - - - - - - - - Action N - - Rectangle.97 - - - - - - - - - - Sheet.98 - Local Table for N Specific Flows Serviced at Node X - - - - Local Table for N Specific Flows Serviced at Node X - - Sheet.99 - - - - Sheet.100 - - - - Sheet.101 - - - - Sheet.102 - Supports N Flows - - - - Supports N Flows - - diff --git a/doc/guides/sample_app_ug/img/server_node_efd.svg b/doc/guides/sample_app_ug/img/server_node_efd.svg new file mode 100644 index 0000000..9aee30b --- /dev/null +++ b/doc/guides/sample_app_ug/img/server_node_efd.svg @@ -0,0 +1,1254 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Page-1 + + + Rectangle.58 + Key 1 + + + + + + + + + + Key 1 + + Rectangle.59 + Action 1 + + + + + + + + + + Action 1 + + Rectangle.60 + Key 2 + + + + + + + + + + Key 2 + + Rectangle.61 + Action 2 + + + + + + + + + + Action 2 + + Rectangle.62 + + + + + + + + + + Rectangle.63 + + + + + + + + + + Rectangle.64 + + + + + + + + + + Rectangle.65 + + + + + + + + + + Rectangle.66 + + + + + + + + + + Rectangle.67 + + + + + + + + + + Rectangle.68 + + + + + + + + + + Rectangle.69 + + + + + + + + + + Rectangle.70 + Key x + + + + + + + + + + Key x + + Rectangle.71 + Action x + + + + + + + + + + Action x + + Rectangle.72 + Key y + + + + + + + + + + Key y + + Rectangle.73 + Action y + + + + + + + + + + Action y + + Rectangle.74 + Key z + + + + + + + + + + Key z + + Rectangle.75 + Action z + + + + + + + + + + Action z + + Rectangle.76 + + + + + + + + + + Rectangle.77 + + + + + + + + + + Rectangle.78 + + + + + + + + + + Rectangle.79 + + + + + + + + + + Rectangle.80 + Key N + + + + + + + + + + Key N + + Rectangle.81 + Action N + + + + + + + + + + Action N + + Rectangle.82 + + + + + + + + + + Sheet.28 + Local Table for N Specific Flows Serviced at Node 1 + + + + Local Table for N Specific Flows Serviced at Node 1 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Load balancer + + Sheet.35 + + + + + + + + + + + Sheet.36 + + + + + + + + Rectangle + + + + + + + + + + Sheet.38 + EFD Table + + + + EFD Table + + Rectangle.39 + Group_id + + + + + + + + + + Group_id + + Rectangle.40 + Hash index + + + + + + + + + + Hash index + + Rectangle.41 + + + + + + + + + + Rectangle.42 + + + + + + + + + + Rectangle.43 + + + + + + + + + + Rectangle.44 + + + + + + + + + + Rectangle.45 + + + + + + + + + + Rectangle.46 + + + + + + + + + + Sheet.47 + + + + Sheet.48 + Supports X*N Flows + + + + Supports X*N Flows + + Sheet.49 + Frontend Server or Load Balancer + + + + Frontend Serveror Load Balancer + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Server + + Sheet.52 + + + + + + + Sheet.53 + + + + + + + Sheet.54 + + + + + + + + + + + + Sheet.59 + + + + Sheet.60 + Backend Server 1 + + + + Backend Server 1 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Server.61 + + Sheet.62 + + + + + + + Sheet.63 + + + + + + + Sheet.64 + + + + + + + + + + + + Sheet.65 + Backend Server 2 + + + + Backend Server 2 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Server.66 + + Sheet.67 + + + + + + + Sheet.68 + + + + + + + Sheet.69 + + + + + + + + + + + + Sheet.70 + Backend Server X + + + + Backend Server X + + Sheet.71 + + + + Sheet.72 + + + + Rectangle.73 + Key 1 + + + + + + + + + + Key 1 + + Rectangle.74 + Action 1 + + + + + + + + + + Action 1 + + Rectangle.75 + Key 2 + + + + + + + + + + Key 2 + + Rectangle.76 + Action 2 + + + + + + + + + + Action 2 + + Rectangle.77 + + + + + + + + + + Rectangle.78 + + + + + + + + + + Rectangle.79 + + + + + + + + + + Rectangle.80 + + + + + + + + + + Rectangle.81 + + + + + + + + + + Rectangle.82 + + + + + + + + + + Rectangle.83 + + + + + + + + + + Rectangle.84 + + + + + + + + + + Rectangle.85 + Key x + + + + + + + + + + Key x + + Rectangle.86 + Action x + + + + + + + + + + Action x + + Rectangle.87 + Key y + + + + + + + + + + Key y + + Rectangle.88 + Action y + + + + + + + + + + Action y + + Rectangle.89 + Key z + + + + + + + + + + Key z + + Rectangle.90 + Action z + + + + + + + + + + Action z + + Rectangle.91 + + + + + + + + + + Rectangle.92 + + + + + + + + + + Rectangle.93 + + + + + + + + + + Rectangle.94 + + + + + + + + + + Rectangle.95 + Key N + + + + + + + + + + Key N + + Rectangle.96 + Action N + + + + + + + + + + Action N + + Rectangle.97 + + + + + + + + + + Sheet.98 + Local Table for N Specific Flows Serviced at Node X + + + + Local Table for N Specific Flows Serviced at Node X + + Sheet.99 + + + + Sheet.100 + + + + Sheet.101 + + + + Sheet.102 + Supports N Flows + + + + Supports N Flows + + diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst index 260f6a5..99ad8ea 100644 --- a/doc/guides/sample_app_ug/index.rst +++ b/doc/guides/sample_app_ug/index.rst @@ -57,7 +57,7 @@ Sample Applications User Guides l3_forward_virtual link_status_intr load_balancer - flow_distributor + server_node_efd multi_process qos_metering qos_scheduler @@ -133,6 +133,8 @@ Sample Applications User Guides :numref:`figure_ptpclient_highlevel` :ref:`figure_ptpclient_highlevel` +:numref:`figure_efd_sample_app_overview` :ref:`figure_efd_sample_app_overview` + **Tables** :numref:`table_qos_metering_1` :ref:`table_qos_metering_1` diff --git a/doc/guides/sample_app_ug/server_node_efd.rst b/doc/guides/sample_app_ug/server_node_efd.rst new file mode 100644 index 0000000..9b69cfe --- /dev/null +++ b/doc/guides/sample_app_ug/server_node_efd.rst @@ -0,0 +1,494 @@ +.. BSD LICENSE + Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Server-Node EFD Sample Application +================================== + +This sample application demonstrates the use of EFD library as a flow-level +load balancer, for more information about the EFD Library please refer to the +DPDK programmer's guide. + +This sample application is a variant of the +:ref:`client-server sample application ` +where a specific target node is specified for every and each flow +(not in a round-robin fashion as the original load balancing sample application). + +Overview +-------- + +The architecture of the EFD flow-based load balancer sample application is +presented in the following figure. + +.. _figure_efd_sample_app_overview: + +.. figure:: img/server_node_efd.* + + Using EFD as a Flow-Level Load Balancer + +As shown in :numref:`figure_efd_sample_app_overview`, +the sample application consists of a front-end node (server) +using the EFD library to create a load-balancing table for flows, +for each flow a target backend worker node is specified. The EFD table does not +store the flow key (unlike a regular hash table), and hence, it can +individually load-balance millions of flows (number of targets * maximum number +of flows fit in a flow table per target) while still fitting in CPU cache. + +It should be noted that although they are referred to as nodes, the frontend +server and worker nodes are processes running on the same platform. + +Front-end Server +~~~~~~~~~~~~~~~~ + +Upon initializing, the frontend server node (process) creates a flow +distributor table (based on the EFD library) which is populated with flow +information and its intended target node. + +The sample application assigns a specific target node_id (process) for each of +the IP destination addresses as follows: + +.. code-block:: c + + node_id = i % num_nodes; /* Target node id is generated */ + ip_dst = rte_cpu_to_be_32(i); /* Specific ip destination address is + assigned to this target node */ + +then the pair of is inserted into the flow distribution table. + +The main loop of the server process receives a burst of packets, then for +each packet, a flow key (IP destination address) is extracted. The flow +distributor table is looked up and the target node id is returned. Packets are +then enqueued to the specified target node id. + +It should be noted that flow distributor table is not a membership test table. +I.e. if the key has already been inserted the target node id will be correct, +but for new keys the flow distributor table will return a value (which can be +valid). + +Backend Worker Nodes +~~~~~~~~~~~~~~~~~~~~ + +Upon initializing, the worker node (process) creates a flow table (a regular +hash table that stores the key default size 1M flows) which is populated with +only the flow information that is serviced at this node. This flow key is +essential to point out new keys that have not been inserted before. + +The worker node's main loop is simply receiving packets then doing a hash table +lookup. If a match occurs then statistics are updated for flows serviced by +this node. If no match is found in the local hash table then this indicates +that this is a new flow, which is dropped. + + +Compiling the Application +------------------------- + +The sequence of steps used to build the application is: + +#. Export the required environment variables: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + export RTE_TARGET=x86_64-native-linuxapp-gcc + +#. Build the application executable file: + + .. code-block:: console + + cd ${RTE_SDK}/examples/server_node_efd/ + make + + For more details on how to build the DPDK libraries and sample + applications, + please refer to the *DPDK Getting Started Guide.* + + +Running the Application +----------------------- + +The application has two binaries to be run: the front-end server +and the back-end node. + +The frontend server (server) has the following command line options:: + + ./server [EAL options] -- -p PORTMASK -n NUM_NODES -f NUM_FLOWS + +Where, + +* ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure +* ``-n NUM_NODES:`` Number of back-end nodes that will be used +* ``-f NUM_FLOWS:`` Number of flows to be added in the EFD table (1 million, by default) + +The back-end node (node) has the following command line options:: + + ./node [EAL options] -- -n NODE_ID + +Where, + +* ``-n NODE_ID:`` Node ID, which cannot be equal or higher than NUM_MODES + + +First, the server app must be launched, with the number of nodes that will be run. +Once it has been started, the node instances can be run, with different NODE_ID. +These instances have to be run as secondary processes, with ``--proc-type=secondary`` +in the EAL options, which will attach to the primary process memory, and therefore, +they can access the queues created by the primary process to distribute packets. + +To successfully run the application, the command line used to start the +application has to be in sync with the traffic flows configured on the traffic +generator side. + +For examples of application command lines and traffic generator flows, please +refer to the DPDK Test Report. For more details on how to set up and run the +sample applications provided with DPDK package, please refer to the +:ref:`DPDK Getting Started Guide for Linux ` and +:ref:`DPDK Getting Started Guide for FreeBSD `. + + +Explanation +----------- + +As described in previous sections, there are two processes in this example. + +The first process, the front-end server, creates and populates the EFD table, +which is used to distribute packets to nodes, which the number of flows +specified in the command line (1 million, by default). + + +.. code-block:: c + + static void + create_efd_table(void) + { + uint8_t socket_id = rte_socket_id(); + + /* create table */ + efd_table = rte_efd_create("flow table", num_flows * 2, sizeof(uint32_t), + 1 << socket_id, socket_id); + + if (efd_table == NULL) + rte_exit(EXIT_FAILURE, "Problem creating the flow table\n"); + } + + static void + populate_efd_table(void) + { + unsigned int i; + int32_t ret; + uint32_t ip_dst; + uint8_t socket_id = rte_socket_id(); + uint64_t node_id; + + /* Add flows in table */ + for (i = 0; i < num_flows; i++) { + node_id = i % num_nodes; + + ip_dst = rte_cpu_to_be_32(i); + ret = rte_efd_update(efd_table, socket_id, + (void *)&ip_dst, (efd_value_t)node_id); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Unable to add entry %u in " + "EFD table\n", i); + } + + printf("EFD table: Adding 0x%x keys\n", num_flows); + } + +After initialization, packets are received from the enabled ports, and the IPv4 +address from the packets is used as a key to look up in the EFD table, +which tells the node where the packet has to be distributed. + +.. code-block:: c + + static void + process_packets(uint32_t port_num __rte_unused, struct rte_mbuf *pkts[], + uint16_t rx_count, unsigned int socket_id) + { + uint16_t i; + uint8_t node; + efd_value_t data[EFD_BURST_MAX]; + const void *key_ptrs[EFD_BURST_MAX]; + + struct ipv4_hdr *ipv4_hdr; + uint32_t ipv4_dst_ip[EFD_BURST_MAX]; + + for (i = 0; i < rx_count; i++) { + /* Handle IPv4 header.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(pkts[i], struct ipv4_hdr *, + sizeof(struct ether_hdr)); + ipv4_dst_ip[i] = ipv4_hdr->dst_addr; + key_ptrs[i] = (void *)&ipv4_dst_ip[i]; + } + + rte_efd_lookup_bulk(efd_table, socket_id, rx_count, + (const void **) key_ptrs, data); + for (i = 0; i < rx_count; i++) { + node = (uint8_t) ((uintptr_t)data[i]); + + if (node >= num_nodes) { + /* + * Node is out of range, which means that + * flow has not been inserted + */ + flow_dist_stats.drop++; + rte_pktmbuf_free(pkts[i]); + } else { + flow_dist_stats.distributed++; + enqueue_rx_packet(node, pkts[i]); + } + } + + for (i = 0; i < num_nodes; i++) + flush_rx_queue(i); + } + +The burst of packets received is enqueued in temporary buffers (per node), +and enqueued in the shared ring between the server and the node. +After this, a new burst of packets is received and this process is +repeated infinitely. + +.. code-block:: c + + static void + flush_rx_queue(uint16_t node) + { + uint16_t j; + struct node *cl; + + if (cl_rx_buf[node].count == 0) + return; + + cl = &nodes[node]; + if (rte_ring_enqueue_bulk(cl->rx_q, (void **)cl_rx_buf[node].buffer, + cl_rx_buf[node].count) != 0){ + for (j = 0; j < cl_rx_buf[node].count; j++) + rte_pktmbuf_free(cl_rx_buf[node].buffer[j]); + cl->stats.rx_drop += cl_rx_buf[node].count; + } else + cl->stats.rx += cl_rx_buf[node].count; + + cl_rx_buf[node].count = 0; + } + +The second process, the back-end node, receives the packets from the shared +ring with the server and send them out, if they belong to the node. + +At initialization, it attaches to the server process memory, to have +access to the shared ring, parameters and statistics. + +.. code-block:: c + + rx_ring = rte_ring_lookup(get_rx_queue_name(node_id)); + if (rx_ring == NULL) + rte_exit(EXIT_FAILURE, "Cannot get RX ring - " + "is server process running?\n"); + + mp = rte_mempool_lookup(PKTMBUF_POOL_NAME); + if (mp == NULL) + rte_exit(EXIT_FAILURE, "Cannot get mempool for mbufs\n"); + + mz = rte_memzone_lookup(MZ_SHARED_INFO); + if (mz == NULL) + rte_exit(EXIT_FAILURE, "Cannot get port info structure\n"); + info = mz->addr; + tx_stats = &(info->tx_stats[node_id]); + filter_stats = &(info->filter_stats[node_id]); + +Then, the hash table that contains the flows that will be handled +by the node is created and populated. + +.. code-block:: c + + static struct rte_hash * + create_hash_table(const struct shared_info *info) + { + uint32_t num_flows_node = info->num_flows / info->num_nodes; + char name[RTE_HASH_NAMESIZE]; + struct rte_hash *h; + + /* create table */ + struct rte_hash_parameters hash_params = { + .entries = num_flows_node * 2, /* table load = 50% */ + .key_len = sizeof(uint32_t), /* Store IPv4 dest IP address */ + .socket_id = rte_socket_id(), + .hash_func_init_val = 0, + }; + + snprintf(name, sizeof(name), "hash_table_%d", node_id); + hash_params.name = name; + h = rte_hash_create(&hash_params); + + if (h == NULL) + rte_exit(EXIT_FAILURE, + "Problem creating the hash table for node %d\n", + node_id); + return h; + } + + static void + populate_hash_table(const struct rte_hash *h, const struct shared_info *info) + { + unsigned int i; + int32_t ret; + uint32_t ip_dst; + uint32_t num_flows_node = 0; + uint64_t target_node; + + /* Add flows in table */ + for (i = 0; i < info->num_flows; i++) { + target_node = i % info->num_nodes; + if (target_node != node_id) + continue; + + ip_dst = rte_cpu_to_be_32(i); + + ret = rte_hash_add_key(h, (void *) &ip_dst); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Unable to add entry %u " + "in hash table\n", i); + else + num_flows_node++; + + } + + printf("Hash table: Adding 0x%x keys\n", num_flows_node); + } + +After initialization, packets are dequeued from the shared ring +(from the server) and, like in the server process, +the IPv4 address from the packets is used as a key to look up in the hash table. +If there is a hit, packet is stored in a buffer, to be eventually transmitted +in one of the enabled ports. If key is not there, packet is dropped, since the +flow is not handled by the node. + +.. code-block:: c + + static inline void + handle_packets(struct rte_hash *h, struct rte_mbuf **bufs, uint16_t num_packets) + { + struct ipv4_hdr *ipv4_hdr; + uint32_t ipv4_dst_ip[PKT_READ_SIZE]; + const void *key_ptrs[PKT_READ_SIZE]; + unsigned int i; + int32_t positions[PKT_READ_SIZE] = {0}; + + for (i = 0; i < num_packets; i++) { + /* Handle IPv4 header.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(bufs[i], struct ipv4_hdr *, + sizeof(struct ether_hdr)); + ipv4_dst_ip[i] = ipv4_hdr->dst_addr; + key_ptrs[i] = &ipv4_dst_ip[i]; + } + /* Check if packets belongs to any flows handled by this node */ + rte_hash_lookup_bulk(h, key_ptrs, num_packets, positions); + + for (i = 0; i < num_packets; i++) { + if (likely(positions[i] >= 0)) { + filter_stats->passed++; + transmit_packet(bufs[i]); + } else { + filter_stats->drop++; + /* Drop packet, as flow is not handled by this node */ + rte_pktmbuf_free(bufs[i]); + } + } + } + +Finally, note that both processes updates statistics, such as transmitted, received +and dropped packets, which are shown and refreshed by the server app. + +.. code-block:: c + + static void + do_stats_display(void) + { + unsigned int i, j; + const char clr[] = {27, '[', '2', 'J', '\0'}; + const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'}; + uint64_t port_tx[RTE_MAX_ETHPORTS], port_tx_drop[RTE_MAX_ETHPORTS]; + uint64_t node_tx[MAX_NODES], node_tx_drop[MAX_NODES]; + + /* to get TX stats, we need to do some summing calculations */ + memset(port_tx, 0, sizeof(port_tx)); + memset(port_tx_drop, 0, sizeof(port_tx_drop)); + memset(node_tx, 0, sizeof(node_tx)); + memset(node_tx_drop, 0, sizeof(node_tx_drop)); + + for (i = 0; i < num_nodes; i++) { + const struct tx_stats *tx = &info->tx_stats[i]; + + for (j = 0; j < info->num_ports; j++) { + const uint64_t tx_val = tx->tx[info->id[j]]; + const uint64_t drop_val = tx->tx_drop[info->id[j]]; + + port_tx[j] += tx_val; + port_tx_drop[j] += drop_val; + node_tx[i] += tx_val; + node_tx_drop[i] += drop_val; + } + } + + /* Clear screen and move to top left */ + printf("%s%s", clr, topLeft); + + printf("PORTS\n"); + printf("-----\n"); + for (i = 0; i < info->num_ports; i++) + printf("Port %u: '%s'\t", (unsigned int)info->id[i], + get_printable_mac_addr(info->id[i])); + printf("\n\n"); + for (i = 0; i < info->num_ports; i++) { + printf("Port %u - rx: %9"PRIu64"\t" + "tx: %9"PRIu64"\n", + (unsigned int)info->id[i], info->rx_stats.rx[i], + port_tx[i]); + } + + printf("\nSERVER\n"); + printf("-----\n"); + printf("distributed: %9"PRIu64", drop: %9"PRIu64"\n", + flow_dist_stats.distributed, flow_dist_stats.drop); + + printf("\nNODES\n"); + printf("-------\n"); + for (i = 0; i < num_nodes; i++) { + const unsigned long long rx = nodes[i].stats.rx; + const unsigned long long rx_drop = nodes[i].stats.rx_drop; + const struct filter_stats *filter = &info->filter_stats[i]; + + printf("Node %2u - rx: %9llu, rx_drop: %9llu\n" + " tx: %9"PRIu64", tx_drop: %9"PRIu64"\n" + " filter_passed: %9"PRIu64", " + "filter_drop: %9"PRIu64"\n", + i, rx, rx_drop, node_tx[i], node_tx_drop[i], + filter->passed, filter->drop); + } + + printf("\n"); + } diff --git a/examples/flow_distributor/Makefile b/examples/flow_distributor/Makefile deleted file mode 100644 index 5bae706..0000000 --- a/examples/flow_distributor/Makefile +++ /dev/null @@ -1,44 +0,0 @@ -# BSD LICENSE -# -# Copyright(c) 2016-2017 Intel Corporation. All rights reserved. -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions -# are met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in -# the documentation and/or other materials provided with the -# distribution. -# * Neither the name of Intel Corporation nor the names of its -# contributors may be used to endorse or promote products derived -# from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -ifeq ($(RTE_SDK),) -$(error "Please define RTE_SDK environment variable") -endif - -# Default target, can be overridden by command line or environment -RTE_TARGET ?= x86_64-native-linuxapp-gcc - -include $(RTE_SDK)/mk/rte.vars.mk - -DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += distributor -DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += node - -include $(RTE_SDK)/mk/rte.extsubdir.mk diff --git a/examples/flow_distributor/distributor/Makefile b/examples/flow_distributor/distributor/Makefile deleted file mode 100644 index 8714151..0000000 --- a/examples/flow_distributor/distributor/Makefile +++ /dev/null @@ -1,57 +0,0 @@ -# BSD LICENSE -# -# Copyright(c) 2016-2017 Intel Corporation. All rights reserved. -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions -# are met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in -# the documentation and/or other materials provided with the -# distribution. -# * Neither the name of Intel Corporation nor the names of its -# contributors may be used to endorse or promote products derived -# from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -ifeq ($(RTE_SDK),) -$(error "Please define RTE_SDK environment variable") -endif - -# Default target, can be overridden by command line or environment -RTE_TARGET ?= x86_64-native-linuxapp-gcc - -include $(RTE_SDK)/mk/rte.vars.mk - -ifneq ($(CONFIG_RTE_EXEC_ENV), "linuxapp") -$(error This application can only operate in a linuxapp environment, \ -please change the definition of the RTE_TARGET environment variable) -endif - -# binary name -APP = distributor - -# all source are stored in SRCS-y -SRCS-y := main.c init.c args.c - -INC := $(wildcard *.h) - -CFLAGS += $(WERROR_FLAGS) -O3 -CFLAGS += -I$(SRCDIR)/../shared - -include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/examples/flow_distributor/distributor/args.c b/examples/flow_distributor/distributor/args.c deleted file mode 100644 index ee29203..0000000 --- a/examples/flow_distributor/distributor/args.c +++ /dev/null @@ -1,200 +0,0 @@ -/*- - * BSD LICENSE - * - * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -#include -#include -#include -#include -#include -#include - -#include -#include - -#include "common.h" -#include "args.h" -#include "init.h" - -/* 1M flows by default */ -#define DEFAULT_NUM_FLOWS 0x100000 - -/* global var for number of nodes - extern in header */ -uint8_t num_nodes; -/* global var for number of flows - extern in header */ -uint32_t num_flows = DEFAULT_NUM_FLOWS; - -static const char *progname; - -/** - * Prints out usage information to stdout - */ -static void -usage(void) -{ - printf("%s [EAL options] -- -p PORTMASK -n NUM_NODES -f NUM_FLOWS\n" - " -p PORTMASK: hexadecimal bitmask of ports to use\n" - " -n NUM_NODES: number of node processes to use\n" - " -f NUM_FLOWS: number of flows to be added in the EFD table\n", - progname); -} - -/** - * The ports to be used by the application are passed in - * the form of a bitmask. This function parses the bitmask - * and places the port numbers to be used into the port[] - * array variable - */ -static int -parse_portmask(uint8_t max_ports, const char *portmask) -{ - char *end = NULL; - unsigned long pm; - uint8_t count = 0; - - if (portmask == NULL || *portmask == '\0') - return -1; - - /* convert parameter to a number and verify */ - pm = strtoul(portmask, &end, 16); - if (end == NULL || *end != '\0' || pm == 0) - return -1; - - /* loop through bits of the mask and mark ports */ - while (pm != 0) { - if (pm & 0x01) { /* bit is set in mask, use port */ - if (count >= max_ports) - printf("WARNING: requested port %u not present" - " - ignoring\n", (unsigned int)count); - else - info->id[info->num_ports++] = count; - } - pm = (pm >> 1); - count++; - } - - return 0; -} - -/** - * Take the number of nodes parameter passed to the app - * and convert to a number to store in the num_nodes variable - */ -static int -parse_num_nodes(const char *nodes) -{ - char *end = NULL; - unsigned long temp; - - if (nodes == NULL || *nodes == '\0') - return -1; - - temp = strtoul(nodes, &end, 10); - if (end == NULL || *end != '\0' || temp == 0) - return -1; - - num_nodes = (uint8_t)temp; - return 0; -} - -static int -parse_num_flows(const char *flows) -{ - char *end = NULL; - - /* parse hexadecimal string */ - num_flows = strtoul(flows, &end, 16); - if ((flows[0] == '\0') || (end == NULL) || (*end != '\0')) - return -1; - - if (num_flows == 0) - return -1; - - return 0; -} - -/** - * The application specific arguments follow the DPDK-specific - * arguments which are stripped by the DPDK init. This function - * processes these application arguments, printing usage info - * on error. - */ -int -parse_app_args(uint8_t max_ports, int argc, char *argv[]) -{ - int option_index, opt; - char **argvopt = argv; - static struct option lgopts[] = { /* no long options */ - {NULL, 0, 0, 0 } - }; - progname = argv[0]; - - while ((opt = getopt_long(argc, argvopt, "n:f:p:", lgopts, - &option_index)) != EOF) { - switch (opt) { - case 'p': - if (parse_portmask(max_ports, optarg) != 0) { - usage(); - return -1; - } - break; - case 'n': - if (parse_num_nodes(optarg) != 0) { - usage(); - return -1; - } - break; - case 'f': - if (parse_num_flows(optarg) != 0) { - usage(); - return -1; - } - break; - default: - printf("ERROR: Unknown option '%c'\n", opt); - usage(); - return -1; - } - } - - if (info->num_ports == 0 || num_nodes == 0) { - usage(); - return -1; - } - - if (info->num_ports % 2 != 0) { - printf("ERROR: application requires an even " - "number of ports to use\n"); - return -1; - } - return 0; -} diff --git a/examples/flow_distributor/distributor/args.h b/examples/flow_distributor/distributor/args.h deleted file mode 100644 index cacf395..0000000 --- a/examples/flow_distributor/distributor/args.h +++ /dev/null @@ -1,39 +0,0 @@ -/*- - * BSD LICENSE - * - * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -#ifndef _ARGS_H_ -#define _ARGS_H_ - -int parse_app_args(uint8_t max_ports, int argc, char *argv[]); - -#endif /* ifndef _ARGS_H_ */ diff --git a/examples/flow_distributor/distributor/init.c b/examples/flow_distributor/distributor/init.c deleted file mode 100644 index 3b0aa85..0000000 --- a/examples/flow_distributor/distributor/init.c +++ /dev/null @@ -1,371 +0,0 @@ -/*- - * BSD LICENSE - * - * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "common.h" -#include "args.h" -#include "init.h" - -#define MBUFS_PER_NODE 1536 -#define MBUFS_PER_PORT 1536 -#define MBUF_CACHE_SIZE 512 - -#define RTE_MP_RX_DESC_DEFAULT 512 -#define RTE_MP_TX_DESC_DEFAULT 512 -#define NODE_QUEUE_RINGSIZE 128 - -#define NO_FLAGS 0 - -/* The mbuf pool for packet rx */ -struct rte_mempool *pktmbuf_pool; - -/* array of info/queues for nodes */ -struct node *nodes; - -/* Flow distributor table */ -struct rte_efd_table *efd_table; - -/* Shared info between distributor and nodes */ -struct shared_info *info; - -/** - * Initialise the mbuf pool for packet reception for the NIC, and any other - * buffer pools needed by the app - currently none. - */ -static int -init_mbuf_pools(void) -{ - const unsigned int num_mbufs = (num_nodes * MBUFS_PER_NODE) + - (info->num_ports * MBUFS_PER_PORT); - - /* - * Don't pass single-producer/single-consumer flags to mbuf create as it - * seems faster to use a cache instead - */ - printf("Creating mbuf pool '%s' [%u mbufs] ...\n", - PKTMBUF_POOL_NAME, num_mbufs); - pktmbuf_pool = rte_pktmbuf_pool_create(PKTMBUF_POOL_NAME, num_mbufs, - MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id()); - - return pktmbuf_pool == NULL; /* 0 on success */ -} - -/** - * Initialise an individual port: - * - configure number of rx and tx rings - * - set up each rx ring, to pull from the main mbuf pool - * - set up each tx ring - * - start the port and report its status to stdout - */ -static int -init_port(uint8_t port_num) -{ - /* for port configuration all features are off by default */ - const struct rte_eth_conf port_conf = { - .rxmode = { - .mq_mode = ETH_MQ_RX_RSS - } - }; - const uint16_t rx_rings = 1, tx_rings = num_nodes; - const uint16_t rx_ring_size = RTE_MP_RX_DESC_DEFAULT; - const uint16_t tx_ring_size = RTE_MP_TX_DESC_DEFAULT; - - uint16_t q; - int retval; - - printf("Port %u init ... ", (unsigned int)port_num); - fflush(stdout); - - /* - * Standard DPDK port initialisation - config port, then set up - * rx and tx rings. - */ - retval = rte_eth_dev_configure(port_num, rx_rings, tx_rings, &port_conf); - if (retval != 0) - return retval; - - for (q = 0; q < rx_rings; q++) { - retval = rte_eth_rx_queue_setup(port_num, q, rx_ring_size, - rte_eth_dev_socket_id(port_num), - NULL, pktmbuf_pool); - if (retval < 0) - return retval; - } - - for (q = 0; q < tx_rings; q++) { - retval = rte_eth_tx_queue_setup(port_num, q, tx_ring_size, - rte_eth_dev_socket_id(port_num), - NULL); - if (retval < 0) - return retval; - } - - rte_eth_promiscuous_enable(port_num); - - retval = rte_eth_dev_start(port_num); - if (retval < 0) - return retval; - - printf("done:\n"); - - return 0; -} - -/** - * Set up the DPDK rings which will be used to pass packets, via - * pointers, between the multi-process distributor and node processes. - * Each node needs one RX queue. - */ -static int -init_shm_rings(void) -{ - unsigned int i; - unsigned int socket_id; - const char *q_name; - const unsigned int ringsize = NODE_QUEUE_RINGSIZE; - - nodes = rte_malloc("node details", - sizeof(*nodes) * num_nodes, 0); - if (nodes == NULL) - rte_exit(EXIT_FAILURE, "Cannot allocate memory for " - "node program details\n"); - - for (i = 0; i < num_nodes; i++) { - /* Create an RX queue for each node */ - socket_id = rte_socket_id(); - q_name = get_rx_queue_name(i); - nodes[i].rx_q = rte_ring_create(q_name, - ringsize, socket_id, - RING_F_SP_ENQ | RING_F_SC_DEQ); - if (nodes[i].rx_q == NULL) - rte_exit(EXIT_FAILURE, "Cannot create rx ring queue " - "for node %u\n", i); - } - return 0; -} - -/* - * Create flow distributor table which will contain all the flows - * that will be distributed among the nodes - */ -static void -create_flow_distributor_table(void) -{ - uint8_t socket_id = rte_socket_id(); - - /* create table */ - efd_table = rte_efd_create("flow table", num_flows * 2, sizeof(uint32_t), - 1 << socket_id, socket_id); - - if (efd_table == NULL) - rte_exit(EXIT_FAILURE, "Problem creating the flow table\n"); -} - -static void -populate_flow_distributor_table(void) -{ - unsigned int i; - int32_t ret; - uint32_t ip_dst; - uint8_t socket_id = rte_socket_id(); - uint64_t node_id; - - /* Add flows in table */ - for (i = 0; i < num_flows; i++) { - node_id = i % num_nodes; - - ip_dst = rte_cpu_to_be_32(i); - ret = rte_efd_update(efd_table, socket_id, - (void *)&ip_dst, (efd_value_t)node_id); - if (ret < 0) - rte_exit(EXIT_FAILURE, "Unable to add entry %u in " - "flow distributor table\n", i); - } - - printf("EFD table: Adding 0x%x keys\n", num_flows); -} - -/* Check the link status of all ports in up to 9s, and print them finally */ -static void -check_all_ports_link_status(uint8_t port_num, uint32_t port_mask) -{ -#define CHECK_INTERVAL 100 /* 100ms */ -#define MAX_CHECK_TIME 90 /* 9s (90 * 100ms) in total */ - uint8_t portid, count, all_ports_up, print_flag = 0; - struct rte_eth_link link; - - printf("\nChecking link status"); - fflush(stdout); - for (count = 0; count <= MAX_CHECK_TIME; count++) { - all_ports_up = 1; - for (portid = 0; portid < port_num; portid++) { - if ((port_mask & (1 << info->id[portid])) == 0) - continue; - memset(&link, 0, sizeof(link)); - rte_eth_link_get_nowait(info->id[portid], &link); - /* print link status if flag set */ - if (print_flag == 1) { - if (link.link_status) - printf("Port %d Link Up - speed %u " - "Mbps - %s\n", info->id[portid], - (unsigned int)link.link_speed, - (link.link_duplex == ETH_LINK_FULL_DUPLEX) ? - ("full-duplex") : ("half-duplex\n")); - else - printf("Port %d Link Down\n", - (uint8_t)info->id[portid]); - continue; - } - /* clear all_ports_up flag if any link down */ - if (link.link_status == ETH_LINK_DOWN) { - all_ports_up = 0; - break; - } - } - /* after finally printing all link status, get out */ - if (print_flag == 1) - break; - - if (all_ports_up == 0) { - printf("."); - fflush(stdout); - rte_delay_ms(CHECK_INTERVAL); - } - - /* set the print_flag if all ports up or timeout */ - if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) { - print_flag = 1; - printf("done\n"); - } - } -} - -/** - * Main init function for the multi-process distributor app, - * calls subfunctions to do each stage of the initialisation. - */ -int -init(int argc, char *argv[]) -{ - int retval; - const struct rte_memzone *mz; - uint8_t i, total_ports; - - /* init EAL, parsing EAL args */ - retval = rte_eal_init(argc, argv); - if (retval < 0) - return -1; - argc -= retval; - argv += retval; - - /* get total number of ports */ - total_ports = rte_eth_dev_count(); - - /* set up array for port data */ - mz = rte_memzone_reserve(MZ_SHARED_INFO, sizeof(*info), - rte_socket_id(), NO_FLAGS); - if (mz == NULL) - rte_exit(EXIT_FAILURE, "Cannot reserve memory zone " - "for port information\n"); - memset(mz->addr, 0, sizeof(*info)); - info = mz->addr; - - /* parse additional, application arguments */ - retval = parse_app_args(total_ports, argc, argv); - if (retval != 0) - return -1; - - /* initialise mbuf pools */ - retval = init_mbuf_pools(); - if (retval != 0) - rte_exit(EXIT_FAILURE, "Cannot create needed mbuf pools\n"); - - /* now initialise the ports we will use */ - for (i = 0; i < info->num_ports; i++) { - retval = init_port(info->id[i]); - if (retval != 0) - rte_exit(EXIT_FAILURE, "Cannot initialise port %u\n", - (unsigned int) i); - } - - check_all_ports_link_status(info->num_ports, (~0x0)); - - /* initialise the node queues/rings for inter-eu comms */ - init_shm_rings(); - - /* Create the flow distributor table */ - create_flow_distributor_table(); - - /* Populate the flow distributor table */ - populate_flow_distributor_table(); - - /* Share the total number of nodes */ - info->num_nodes = num_nodes; - - /* Share the total number of flows */ - info->num_flows = num_flows; - return 0; -} diff --git a/examples/flow_distributor/distributor/init.h b/examples/flow_distributor/distributor/init.h deleted file mode 100644 index d11aacf..0000000 --- a/examples/flow_distributor/distributor/init.h +++ /dev/null @@ -1,76 +0,0 @@ -/*- - * BSD LICENSE - * - * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -#ifndef _INIT_H_ -#define _INIT_H_ - -/* - * #include - * #include "args.h" - */ - -/* - * Define a node structure with all needed info, including - * stats from the nodes. - */ -struct node { - struct rte_ring *rx_q; - unsigned int node_id; - /* these stats hold how many packets the node will actually receive, - * and how many packets were dropped because the node's queue was full. - * The port-info stats, in contrast, record how many packets were received - * or transmitted on an actual NIC port. - */ - struct { - uint64_t rx; - uint64_t rx_drop; - } stats; -}; - -extern struct rte_efd_table *efd_table; -extern struct node *nodes; - -/* - * shared information between distributor and nodes: number of clients, - * port numbers, rx and tx stats etc. - */ -extern struct shared_info *info; - -extern struct rte_mempool *pktmbuf_pool; -extern uint8_t num_nodes; -extern unsigned int num_sockets; -extern uint32_t num_flows; - -int init(int argc, char *argv[]); - -#endif /* ifndef _INIT_H_ */ diff --git a/examples/flow_distributor/distributor/main.c b/examples/flow_distributor/distributor/main.c deleted file mode 100644 index f97f003..0000000 --- a/examples/flow_distributor/distributor/main.c +++ /dev/null @@ -1,362 +0,0 @@ -/*- - * BSD LICENSE - * - * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "common.h" -#include "args.h" -#include "init.h" - -/* - * When doing reads from the NIC or the node queues, - * use this batch size - */ -#define PACKET_READ_SIZE 32 - -/* - * Local buffers to put packets in, used to send packets in bursts to the - * nodes - */ -struct node_rx_buf { - struct rte_mbuf *buffer[PACKET_READ_SIZE]; - uint16_t count; -}; - -struct flow_distributor_stats { - uint64_t distributed; - uint64_t drop; -} flow_dist_stats; - -/* One buffer per node rx queue - dynamically allocate array */ -static struct node_rx_buf *cl_rx_buf; - -static const char * -get_printable_mac_addr(uint8_t port) -{ - static const char err_address[] = "00:00:00:00:00:00"; - static char addresses[RTE_MAX_ETHPORTS][sizeof(err_address)]; - struct ether_addr mac; - - if (unlikely(port >= RTE_MAX_ETHPORTS)) - return err_address; - if (unlikely(addresses[port][0] == '\0')) { - rte_eth_macaddr_get(port, &mac); - snprintf(addresses[port], sizeof(addresses[port]), - "%02x:%02x:%02x:%02x:%02x:%02x\n", - mac.addr_bytes[0], mac.addr_bytes[1], - mac.addr_bytes[2], mac.addr_bytes[3], - mac.addr_bytes[4], mac.addr_bytes[5]); - } - return addresses[port]; -} - -/* - * This function displays the recorded statistics for each port - * and for each node. It uses ANSI terminal codes to clear - * screen when called. It is called from a single non-master - * thread in the distributor process, when the process is run with more - * than one lcore enabled. - */ -static void -do_stats_display(void) -{ - unsigned int i, j; - const char clr[] = {27, '[', '2', 'J', '\0'}; - const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'}; - uint64_t port_tx[RTE_MAX_ETHPORTS], port_tx_drop[RTE_MAX_ETHPORTS]; - uint64_t node_tx[MAX_NODES], node_tx_drop[MAX_NODES]; - - /* to get TX stats, we need to do some summing calculations */ - memset(port_tx, 0, sizeof(port_tx)); - memset(port_tx_drop, 0, sizeof(port_tx_drop)); - memset(node_tx, 0, sizeof(node_tx)); - memset(node_tx_drop, 0, sizeof(node_tx_drop)); - - for (i = 0; i < num_nodes; i++) { - const struct tx_stats *tx = &info->tx_stats[i]; - - for (j = 0; j < info->num_ports; j++) { - const uint64_t tx_val = tx->tx[info->id[j]]; - const uint64_t drop_val = tx->tx_drop[info->id[j]]; - - port_tx[j] += tx_val; - port_tx_drop[j] += drop_val; - node_tx[i] += tx_val; - node_tx_drop[i] += drop_val; - } - } - - /* Clear screen and move to top left */ - printf("%s%s", clr, topLeft); - - printf("PORTS\n"); - printf("-----\n"); - for (i = 0; i < info->num_ports; i++) - printf("Port %u: '%s'\t", (unsigned int)info->id[i], - get_printable_mac_addr(info->id[i])); - printf("\n\n"); - for (i = 0; i < info->num_ports; i++) { - printf("Port %u - rx: %9"PRIu64"\t" - "tx: %9"PRIu64"\n", - (unsigned int)info->id[i], info->rx_stats.rx[i], - port_tx[i]); - } - - printf("\nFLOW DISTRIBUTOR\n"); - printf("-----\n"); - printf("distributed: %9"PRIu64", drop: %9"PRIu64"\n", - flow_dist_stats.distributed, flow_dist_stats.drop); - - printf("\nNODES\n"); - printf("-------\n"); - for (i = 0; i < num_nodes; i++) { - const unsigned long long rx = nodes[i].stats.rx; - const unsigned long long rx_drop = nodes[i].stats.rx_drop; - const struct filter_stats *filter = &info->filter_stats[i]; - - printf("Node %2u - rx: %9llu, rx_drop: %9llu\n" - " tx: %9"PRIu64", tx_drop: %9"PRIu64"\n" - " filter_passed: %9"PRIu64", " - "filter_drop: %9"PRIu64"\n", - i, rx, rx_drop, node_tx[i], node_tx_drop[i], - filter->passed, filter->drop); - } - - printf("\n"); -} - -/* - * The function called from each non-master lcore used by the process. - * The test_and_set function is used to randomly pick a single lcore on which - * the code to display the statistics will run. Otherwise, the code just - * repeatedly sleeps. - */ -static int -sleep_lcore(__attribute__((unused)) void *dummy) -{ - /* Used to pick a display thread - static, so zero-initialised */ - static rte_atomic32_t display_stats; - - /* Only one core should display stats */ - if (rte_atomic32_test_and_set(&display_stats)) { - const unsigned int sleeptime = 1; - - printf("Core %u displaying statistics\n", rte_lcore_id()); - - /* Longer initial pause so above printf is seen */ - sleep(sleeptime * 3); - - /* Loop forever: sleep always returns 0 or <= param */ - while (sleep(sleeptime) <= sleeptime) - do_stats_display(); - } - return 0; -} - -/* - * Function to set all the node statistic values to zero. - * Called at program startup. - */ -static void -clear_stats(void) -{ - unsigned int i; - - for (i = 0; i < num_nodes; i++) - nodes[i].stats.rx = nodes[i].stats.rx_drop = 0; -} - -/* - * send a burst of traffic to a node, assuming there are packets - * available to be sent to this node - */ -static void -flush_rx_queue(uint16_t node) -{ - uint16_t j; - struct node *cl; - - if (cl_rx_buf[node].count == 0) - return; - - cl = &nodes[node]; - if (rte_ring_enqueue_bulk(cl->rx_q, (void **)cl_rx_buf[node].buffer, - cl_rx_buf[node].count) != 0){ - for (j = 0; j < cl_rx_buf[node].count; j++) - rte_pktmbuf_free(cl_rx_buf[node].buffer[j]); - cl->stats.rx_drop += cl_rx_buf[node].count; - } else - cl->stats.rx += cl_rx_buf[node].count; - - cl_rx_buf[node].count = 0; -} - -/* - * marks a packet down to be sent to a particular node process - */ -static inline void -enqueue_rx_packet(uint8_t node, struct rte_mbuf *buf) -{ - cl_rx_buf[node].buffer[cl_rx_buf[node].count++] = buf; -} - -/* - * This function takes a group of packets and routes them - * individually to the node process. Very simply round-robins the packets - * without checking any of the packet contents. - */ -static void -process_packets(uint32_t port_num __rte_unused, struct rte_mbuf *pkts[], - uint16_t rx_count, unsigned int socket_id) -{ - uint16_t i; - uint8_t node; - efd_value_t data[RTE_EFD_BURST_MAX]; - const void *key_ptrs[RTE_EFD_BURST_MAX]; - - struct ipv4_hdr *ipv4_hdr; - uint32_t ipv4_dst_ip[RTE_EFD_BURST_MAX]; - - for (i = 0; i < rx_count; i++) { - /* Handle IPv4 header.*/ - ipv4_hdr = rte_pktmbuf_mtod_offset(pkts[i], struct ipv4_hdr *, - sizeof(struct ether_hdr)); - ipv4_dst_ip[i] = ipv4_hdr->dst_addr; - key_ptrs[i] = (void *)&ipv4_dst_ip[i]; - } - - rte_efd_lookup_bulk(efd_table, socket_id, rx_count, - (const void **) key_ptrs, data); - for (i = 0; i < rx_count; i++) { - node = (uint8_t) ((uintptr_t)data[i]); - - if (node >= num_nodes) { - /* - * Node is out of range, which means that - * flow has not been inserted - */ - flow_dist_stats.drop++; - rte_pktmbuf_free(pkts[i]); - } else { - flow_dist_stats.distributed++; - enqueue_rx_packet(node, pkts[i]); - } - } - - for (i = 0; i < num_nodes; i++) - flush_rx_queue(i); -} - -/* - * Function called by the master lcore of the DPDK process. - */ -static void -do_packet_forwarding(void) -{ - unsigned int port_num = 0; /* indexes the port[] array */ - unsigned int socket_id = rte_socket_id(); - - for (;;) { - struct rte_mbuf *buf[PACKET_READ_SIZE]; - uint16_t rx_count; - - /* read a port */ - rx_count = rte_eth_rx_burst(info->id[port_num], 0, - buf, PACKET_READ_SIZE); - info->rx_stats.rx[port_num] += rx_count; - - /* Now process the NIC packets read */ - if (likely(rx_count > 0)) - process_packets(port_num, buf, rx_count, socket_id); - - /* move to next port */ - if (++port_num == info->num_ports) - port_num = 0; - } -} - -int -main(int argc, char *argv[]) -{ - /* initialise the system */ - if (init(argc, argv) < 0) - return -1; - RTE_LOG(INFO, APP, "Finished Process Init.\n"); - - cl_rx_buf = calloc(num_nodes, sizeof(cl_rx_buf[0])); - - /* clear statistics */ - clear_stats(); - - /* put all other cores to sleep bar master */ - rte_eal_mp_remote_launch(sleep_lcore, NULL, SKIP_MASTER); - - do_packet_forwarding(); - return 0; -} diff --git a/examples/flow_distributor/node/Makefile b/examples/flow_distributor/node/Makefile deleted file mode 100644 index 8cf7b65..0000000 --- a/examples/flow_distributor/node/Makefile +++ /dev/null @@ -1,48 +0,0 @@ -# BSD LICENSE -# -# Copyright(c) 2016-2017 Intel Corporation. All rights reserved. -# All rights reserved. -# -# Redistribution and use in source and binary forms, with or without -# modification, are permitted provided that the following conditions -# are met: -# -# * Redistributions of source code must retain the above copyright -# notice, this list of conditions and the following disclaimer. -# * Redistributions in binary form must reproduce the above copyright -# notice, this list of conditions and the following disclaimer in -# the documentation and/or other materials provided with the -# distribution. -# * Neither the name of Intel Corporation nor the names of its -# contributors may be used to endorse or promote products derived -# from this software without specific prior written permission. -# -# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -ifeq ($(RTE_SDK),) -$(error "Please define RTE_SDK environment variable") -endif - -# Default target, can be overridden by command line or environment -include $(RTE_SDK)/mk/rte.vars.mk - -# binary name -APP = node - -# all source are stored in SRCS-y -SRCS-y := node.c - -CFLAGS += $(WERROR_FLAGS) -O3 -CFLAGS += -I$(SRCDIR)/../shared - -include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/examples/flow_distributor/node/node.c b/examples/flow_distributor/node/node.c deleted file mode 100644 index 1f1e7e7..0000000 --- a/examples/flow_distributor/node/node.c +++ /dev/null @@ -1,417 +0,0 @@ -/*- - * BSD LICENSE - * - * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "common.h" - -/* Number of packets to attempt to read from queue */ -#define PKT_READ_SIZE ((uint16_t)32) - -/* - * Our node id number - tells us which rx queue to read, and NIC TX - * queue to write to. - */ -static uint8_t node_id; - -#define MBQ_CAPACITY 32 - -/* maps input ports to output ports for packets */ -static uint8_t output_ports[RTE_MAX_ETHPORTS]; - -/* buffers up a set of packet that are ready to send */ -struct rte_eth_dev_tx_buffer *tx_buffer[RTE_MAX_ETHPORTS]; - -/* shared data from distributor. We update statistics here */ -static struct tx_stats *tx_stats; - -static struct filter_stats *filter_stats; - -/* - * print a usage message - */ -static void -usage(const char *progname) -{ - printf("Usage: %s [EAL args] -- -n \n\n", progname); -} - -/* - * Convert the node id number from a string to an int. - */ -static int -parse_node_num(const char *node) -{ - char *end = NULL; - unsigned long temp; - - if (node == NULL || *node == '\0') - return -1; - - temp = strtoul(node, &end, 10); - if (end == NULL || *end != '\0') - return -1; - - node_id = (uint8_t)temp; - return 0; -} - -/* - * Parse the application arguments to the node app. - */ -static int -parse_app_args(int argc, char *argv[]) -{ - int option_index, opt; - char **argvopt = argv; - const char *progname = NULL; - static struct option lgopts[] = { /* no long options */ - {NULL, 0, 0, 0 } - }; - progname = argv[0]; - - while ((opt = getopt_long(argc, argvopt, "n:", lgopts, - &option_index)) != EOF) { - switch (opt) { - case 'n': - if (parse_node_num(optarg) != 0) { - usage(progname); - return -1; - } - break; - default: - usage(progname); - return -1; - } - } - return 0; -} - -/* - * Tx buffer error callback - */ -static void -flush_tx_error_callback(struct rte_mbuf **unsent, uint16_t count, - void *userdata) { - int i; - uint8_t port_id = (uintptr_t)userdata; - - tx_stats->tx_drop[port_id] += count; - - /* free the mbufs which failed from transmit */ - for (i = 0; i < count; i++) - rte_pktmbuf_free(unsent[i]); - -} - -static void -configure_tx_buffer(uint8_t port_id, uint16_t size) -{ - int ret; - - /* Initialize TX buffers */ - tx_buffer[port_id] = rte_zmalloc_socket("tx_buffer", - RTE_ETH_TX_BUFFER_SIZE(size), 0, - rte_eth_dev_socket_id(port_id)); - if (tx_buffer[port_id] == NULL) - rte_exit(EXIT_FAILURE, "Cannot allocate buffer for tx " - "on port %u\n", (unsigned int) port_id); - - rte_eth_tx_buffer_init(tx_buffer[port_id], size); - - ret = rte_eth_tx_buffer_set_err_callback(tx_buffer[port_id], - flush_tx_error_callback, (void *)(intptr_t)port_id); - if (ret < 0) - rte_exit(EXIT_FAILURE, "Cannot set error callback for " - "tx buffer on port %u\n", (unsigned int) port_id); -} - -/* - * set up output ports so that all traffic on port gets sent out - * its paired port. Index using actual port numbers since that is - * what comes in the mbuf structure. - */ -static void -configure_output_ports(const struct shared_info *info) -{ - int i; - - if (info->num_ports > RTE_MAX_ETHPORTS) - rte_exit(EXIT_FAILURE, "Too many ethernet ports. " - "RTE_MAX_ETHPORTS = %u\n", - (unsigned int)RTE_MAX_ETHPORTS); - for (i = 0; i < info->num_ports - 1; i += 2) { - uint8_t p1 = info->id[i]; - uint8_t p2 = info->id[i+1]; - - output_ports[p1] = p2; - output_ports[p2] = p1; - - configure_tx_buffer(p1, MBQ_CAPACITY); - configure_tx_buffer(p2, MBQ_CAPACITY); - - } -} - -/* - * Create the hash table that will contain the flows that - * the node will handle, which will be used to decide if packet - * is transmitted or dropped. - */ -static struct rte_hash * -create_hash_table(const struct shared_info *info) -{ - uint32_t num_flows_node = info->num_flows / info->num_nodes; - char name[RTE_HASH_NAMESIZE]; - struct rte_hash *h; - - /* create table */ - struct rte_hash_parameters hash_params = { - .entries = num_flows_node * 2, /* table load = 50% */ - .key_len = sizeof(uint32_t), /* Store IPv4 dest IP address */ - .socket_id = rte_socket_id(), - .hash_func_init_val = 0, - }; - - snprintf(name, sizeof(name), "hash_table_%d", node_id); - hash_params.name = name; - h = rte_hash_create(&hash_params); - - if (h == NULL) - rte_exit(EXIT_FAILURE, - "Problem creating the hash table for node %d\n", - node_id); - return h; -} - -static void -populate_hash_table(const struct rte_hash *h, const struct shared_info *info) -{ - unsigned int i; - int32_t ret; - uint32_t ip_dst; - uint32_t num_flows_node = 0; - uint64_t target_node; - - /* Add flows in table */ - for (i = 0; i < info->num_flows; i++) { - target_node = i % info->num_nodes; - if (target_node != node_id) - continue; - - ip_dst = rte_cpu_to_be_32(i); - - ret = rte_hash_add_key(h, (void *) &ip_dst); - if (ret < 0) - rte_exit(EXIT_FAILURE, "Unable to add entry %u " - "in hash table\n", i); - else - num_flows_node++; - - } - - printf("Hash table: Adding 0x%x keys\n", num_flows_node); -} - -/* - * This function performs routing of packets - * Just sends each input packet out an output port based solely on the input - * port it arrived on. - */ -static inline void -transmit_packet(struct rte_mbuf *buf) -{ - int sent; - const uint8_t in_port = buf->port; - const uint8_t out_port = output_ports[in_port]; - struct rte_eth_dev_tx_buffer *buffer = tx_buffer[out_port]; - - sent = rte_eth_tx_buffer(out_port, node_id, buffer, buf); - if (sent) - tx_stats->tx[out_port] += sent; - -} - -static inline void -handle_packets(struct rte_hash *h, struct rte_mbuf **bufs, uint16_t num_packets) -{ - struct ipv4_hdr *ipv4_hdr; - uint32_t ipv4_dst_ip[PKT_READ_SIZE]; - const void *key_ptrs[PKT_READ_SIZE]; - unsigned int i; - int32_t positions[PKT_READ_SIZE] = {0}; - - for (i = 0; i < num_packets; i++) { - /* Handle IPv4 header.*/ - ipv4_hdr = rte_pktmbuf_mtod_offset(bufs[i], struct ipv4_hdr *, - sizeof(struct ether_hdr)); - ipv4_dst_ip[i] = ipv4_hdr->dst_addr; - key_ptrs[i] = &ipv4_dst_ip[i]; - } - /* Check if packets belongs to any flows handled by this node */ - rte_hash_lookup_bulk(h, key_ptrs, num_packets, positions); - - for (i = 0; i < num_packets; i++) { - if (likely(positions[i] >= 0)) { - filter_stats->passed++; - transmit_packet(bufs[i]); - } else { - filter_stats->drop++; - /* Drop packet, as flow is not handled by this node */ - rte_pktmbuf_free(bufs[i]); - } - } -} - -/* - * Application main function - loops through - * receiving and processing packets. Never returns - */ -int -main(int argc, char *argv[]) -{ - const struct rte_memzone *mz; - struct rte_ring *rx_ring; - struct rte_hash *h; - struct rte_mempool *mp; - struct shared_info *info; - int need_flush = 0; /* indicates whether we have unsent packets */ - int retval; - void *pkts[PKT_READ_SIZE]; - uint16_t sent; - - retval = rte_eal_init(argc, argv); - if (retval < 0) - return -1; - argc -= retval; - argv += retval; - - if (parse_app_args(argc, argv) < 0) - rte_exit(EXIT_FAILURE, "Invalid command-line arguments\n"); - - if (rte_eth_dev_count() == 0) - rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n"); - - rx_ring = rte_ring_lookup(get_rx_queue_name(node_id)); - if (rx_ring == NULL) - rte_exit(EXIT_FAILURE, "Cannot get RX ring - " - "is distributor process running?\n"); - - mp = rte_mempool_lookup(PKTMBUF_POOL_NAME); - if (mp == NULL) - rte_exit(EXIT_FAILURE, "Cannot get mempool for mbufs\n"); - - mz = rte_memzone_lookup(MZ_SHARED_INFO); - if (mz == NULL) - rte_exit(EXIT_FAILURE, "Cannot get port info structure\n"); - info = mz->addr; - tx_stats = &(info->tx_stats[node_id]); - filter_stats = &(info->filter_stats[node_id]); - - configure_output_ports(info); - - h = create_hash_table(info); - - populate_hash_table(h, info); - - RTE_LOG(INFO, APP, "Finished Process Init.\n"); - - printf("\nNode process %d handling packets\n", node_id); - printf("[Press Ctrl-C to quit ...]\n"); - - for (;;) { - uint16_t rx_pkts = PKT_READ_SIZE; - uint8_t port; - - /* - * Try dequeuing max possible packets first, if that fails, - * get the most we can. Loop body should only execute once, - * maximum - */ - while (rx_pkts > 0 && - unlikely(rte_ring_dequeue_bulk(rx_ring, pkts, - rx_pkts) != 0)) - rx_pkts = (uint16_t)RTE_MIN(rte_ring_count(rx_ring), - PKT_READ_SIZE); - - if (unlikely(rx_pkts == 0)) { - if (need_flush) - for (port = 0; port < info->num_ports; port++) { - sent = rte_eth_tx_buffer_flush( - info->id[port], - node_id, - tx_buffer[port]); - if (unlikely(sent)) - tx_stats->tx[port] += sent; - } - need_flush = 0; - continue; - } - - handle_packets(h, (struct rte_mbuf **)pkts, rx_pkts); - - need_flush = 1; - } -} diff --git a/examples/flow_distributor/shared/common.h b/examples/flow_distributor/shared/common.h deleted file mode 100644 index 5dcffd6..0000000 --- a/examples/flow_distributor/shared/common.h +++ /dev/null @@ -1,99 +0,0 @@ -/*- - * BSD LICENSE - * - * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -#ifndef _COMMON_H_ -#define _COMMON_H_ - -#include -#include - -#define MAX_NODES 16 -/* - * Shared port info, including statistics information for display by distributor. - * Structure will be put in a memzone. - * - All port id values share one cache line as this data will be read-only - * during operation. - * - All rx statistic values share cache lines, as this data is written only - * by the distributor process. (rare reads by stats display) - * - The tx statistics have values for all ports per cache line, but the stats - * themselves are written by the nodes, so we have a distinct set, on different - * cache lines for each node to use. - */ -struct rx_stats { - uint64_t rx[RTE_MAX_ETHPORTS]; -} __rte_cache_aligned; - -struct tx_stats { - uint64_t tx[RTE_MAX_ETHPORTS]; - uint64_t tx_drop[RTE_MAX_ETHPORTS]; -} __rte_cache_aligned; - -struct filter_stats { - uint64_t drop; - uint64_t passed; -} __rte_cache_aligned; - -struct shared_info { - uint8_t num_nodes; - uint8_t num_ports; - uint32_t num_flows; - uint8_t id[RTE_MAX_ETHPORTS]; - struct rx_stats rx_stats; - struct tx_stats tx_stats[MAX_NODES]; - struct filter_stats filter_stats[MAX_NODES]; -}; - -/* define common names for structures shared between distributor and node */ -#define MP_NODE_RXQ_NAME "MProc_Node_%u_RX" -#define PKTMBUF_POOL_NAME "MProc_pktmbuf_pool" -#define MZ_SHARED_INFO "MProc_shared_info" - -/* - * Given the rx queue name template above, get the queue name - */ -static inline const char * -get_rx_queue_name(unsigned int id) -{ - /* - * Buffer for return value. Size calculated by %u being replaced - * by maximum 3 digits (plus an extra byte for safety) - */ - static char buffer[sizeof(MP_NODE_RXQ_NAME) + 2]; - - snprintf(buffer, sizeof(buffer) - 1, MP_NODE_RXQ_NAME, id); - return buffer; -} - -#define RTE_LOGTYPE_APP RTE_LOGTYPE_USER1 - -#endif diff --git a/examples/server_node_efd/Makefile b/examples/server_node_efd/Makefile new file mode 100644 index 0000000..6977ef9 --- /dev/null +++ b/examples/server_node_efd/Makefile @@ -0,0 +1,44 @@ +# BSD LICENSE +# +# Copyright(c) 2016-2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += server +DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += node + +include $(RTE_SDK)/mk/rte.extsubdir.mk diff --git a/examples/server_node_efd/node/Makefile b/examples/server_node_efd/node/Makefile new file mode 100644 index 0000000..8cf7b65 --- /dev/null +++ b/examples/server_node_efd/node/Makefile @@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2016-2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +include $(RTE_SDK)/mk/rte.vars.mk + +# binary name +APP = node + +# all source are stored in SRCS-y +SRCS-y := node.c + +CFLAGS += $(WERROR_FLAGS) -O3 +CFLAGS += -I$(SRCDIR)/../shared + +include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/examples/server_node_efd/node/node.c b/examples/server_node_efd/node/node.c new file mode 100644 index 0000000..a6c0c70 --- /dev/null +++ b/examples/server_node_efd/node/node.c @@ -0,0 +1,417 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common.h" + +/* Number of packets to attempt to read from queue */ +#define PKT_READ_SIZE ((uint16_t)32) + +/* + * Our node id number - tells us which rx queue to read, and NIC TX + * queue to write to. + */ +static uint8_t node_id; + +#define MBQ_CAPACITY 32 + +/* maps input ports to output ports for packets */ +static uint8_t output_ports[RTE_MAX_ETHPORTS]; + +/* buffers up a set of packet that are ready to send */ +struct rte_eth_dev_tx_buffer *tx_buffer[RTE_MAX_ETHPORTS]; + +/* shared data from server. We update statistics here */ +static struct tx_stats *tx_stats; + +static struct filter_stats *filter_stats; + +/* + * print a usage message + */ +static void +usage(const char *progname) +{ + printf("Usage: %s [EAL args] -- -n \n\n", progname); +} + +/* + * Convert the node id number from a string to an int. + */ +static int +parse_node_num(const char *node) +{ + char *end = NULL; + unsigned long temp; + + if (node == NULL || *node == '\0') + return -1; + + temp = strtoul(node, &end, 10); + if (end == NULL || *end != '\0') + return -1; + + node_id = (uint8_t)temp; + return 0; +} + +/* + * Parse the application arguments to the node app. + */ +static int +parse_app_args(int argc, char *argv[]) +{ + int option_index, opt; + char **argvopt = argv; + const char *progname = NULL; + static struct option lgopts[] = { /* no long options */ + {NULL, 0, 0, 0 } + }; + progname = argv[0]; + + while ((opt = getopt_long(argc, argvopt, "n:", lgopts, + &option_index)) != EOF) { + switch (opt) { + case 'n': + if (parse_node_num(optarg) != 0) { + usage(progname); + return -1; + } + break; + default: + usage(progname); + return -1; + } + } + return 0; +} + +/* + * Tx buffer error callback + */ +static void +flush_tx_error_callback(struct rte_mbuf **unsent, uint16_t count, + void *userdata) { + int i; + uint8_t port_id = (uintptr_t)userdata; + + tx_stats->tx_drop[port_id] += count; + + /* free the mbufs which failed from transmit */ + for (i = 0; i < count; i++) + rte_pktmbuf_free(unsent[i]); + +} + +static void +configure_tx_buffer(uint8_t port_id, uint16_t size) +{ + int ret; + + /* Initialize TX buffers */ + tx_buffer[port_id] = rte_zmalloc_socket("tx_buffer", + RTE_ETH_TX_BUFFER_SIZE(size), 0, + rte_eth_dev_socket_id(port_id)); + if (tx_buffer[port_id] == NULL) + rte_exit(EXIT_FAILURE, "Cannot allocate buffer for tx " + "on port %u\n", (unsigned int) port_id); + + rte_eth_tx_buffer_init(tx_buffer[port_id], size); + + ret = rte_eth_tx_buffer_set_err_callback(tx_buffer[port_id], + flush_tx_error_callback, (void *)(intptr_t)port_id); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Cannot set error callback for " + "tx buffer on port %u\n", (unsigned int) port_id); +} + +/* + * set up output ports so that all traffic on port gets sent out + * its paired port. Index using actual port numbers since that is + * what comes in the mbuf structure. + */ +static void +configure_output_ports(const struct shared_info *info) +{ + int i; + + if (info->num_ports > RTE_MAX_ETHPORTS) + rte_exit(EXIT_FAILURE, "Too many ethernet ports. " + "RTE_MAX_ETHPORTS = %u\n", + (unsigned int)RTE_MAX_ETHPORTS); + for (i = 0; i < info->num_ports - 1; i += 2) { + uint8_t p1 = info->id[i]; + uint8_t p2 = info->id[i+1]; + + output_ports[p1] = p2; + output_ports[p2] = p1; + + configure_tx_buffer(p1, MBQ_CAPACITY); + configure_tx_buffer(p2, MBQ_CAPACITY); + + } +} + +/* + * Create the hash table that will contain the flows that + * the node will handle, which will be used to decide if packet + * is transmitted or dropped. + */ +static struct rte_hash * +create_hash_table(const struct shared_info *info) +{ + uint32_t num_flows_node = info->num_flows / info->num_nodes; + char name[RTE_HASH_NAMESIZE]; + struct rte_hash *h; + + /* create table */ + struct rte_hash_parameters hash_params = { + .entries = num_flows_node * 2, /* table load = 50% */ + .key_len = sizeof(uint32_t), /* Store IPv4 dest IP address */ + .socket_id = rte_socket_id(), + .hash_func_init_val = 0, + }; + + snprintf(name, sizeof(name), "hash_table_%d", node_id); + hash_params.name = name; + h = rte_hash_create(&hash_params); + + if (h == NULL) + rte_exit(EXIT_FAILURE, + "Problem creating the hash table for node %d\n", + node_id); + return h; +} + +static void +populate_hash_table(const struct rte_hash *h, const struct shared_info *info) +{ + unsigned int i; + int32_t ret; + uint32_t ip_dst; + uint32_t num_flows_node = 0; + uint64_t target_node; + + /* Add flows in table */ + for (i = 0; i < info->num_flows; i++) { + target_node = i % info->num_nodes; + if (target_node != node_id) + continue; + + ip_dst = rte_cpu_to_be_32(i); + + ret = rte_hash_add_key(h, (void *) &ip_dst); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Unable to add entry %u " + "in hash table\n", i); + else + num_flows_node++; + + } + + printf("Hash table: Adding 0x%x keys\n", num_flows_node); +} + +/* + * This function performs routing of packets + * Just sends each input packet out an output port based solely on the input + * port it arrived on. + */ +static inline void +transmit_packet(struct rte_mbuf *buf) +{ + int sent; + const uint8_t in_port = buf->port; + const uint8_t out_port = output_ports[in_port]; + struct rte_eth_dev_tx_buffer *buffer = tx_buffer[out_port]; + + sent = rte_eth_tx_buffer(out_port, node_id, buffer, buf); + if (sent) + tx_stats->tx[out_port] += sent; + +} + +static inline void +handle_packets(struct rte_hash *h, struct rte_mbuf **bufs, uint16_t num_packets) +{ + struct ipv4_hdr *ipv4_hdr; + uint32_t ipv4_dst_ip[PKT_READ_SIZE]; + const void *key_ptrs[PKT_READ_SIZE]; + unsigned int i; + int32_t positions[PKT_READ_SIZE] = {0}; + + for (i = 0; i < num_packets; i++) { + /* Handle IPv4 header.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(bufs[i], struct ipv4_hdr *, + sizeof(struct ether_hdr)); + ipv4_dst_ip[i] = ipv4_hdr->dst_addr; + key_ptrs[i] = &ipv4_dst_ip[i]; + } + /* Check if packets belongs to any flows handled by this node */ + rte_hash_lookup_bulk(h, key_ptrs, num_packets, positions); + + for (i = 0; i < num_packets; i++) { + if (likely(positions[i] >= 0)) { + filter_stats->passed++; + transmit_packet(bufs[i]); + } else { + filter_stats->drop++; + /* Drop packet, as flow is not handled by this node */ + rte_pktmbuf_free(bufs[i]); + } + } +} + +/* + * Application main function - loops through + * receiving and processing packets. Never returns + */ +int +main(int argc, char *argv[]) +{ + const struct rte_memzone *mz; + struct rte_ring *rx_ring; + struct rte_hash *h; + struct rte_mempool *mp; + struct shared_info *info; + int need_flush = 0; /* indicates whether we have unsent packets */ + int retval; + void *pkts[PKT_READ_SIZE]; + uint16_t sent; + + retval = rte_eal_init(argc, argv); + if (retval < 0) + return -1; + argc -= retval; + argv += retval; + + if (parse_app_args(argc, argv) < 0) + rte_exit(EXIT_FAILURE, "Invalid command-line arguments\n"); + + if (rte_eth_dev_count() == 0) + rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n"); + + rx_ring = rte_ring_lookup(get_rx_queue_name(node_id)); + if (rx_ring == NULL) + rte_exit(EXIT_FAILURE, "Cannot get RX ring - " + "is server process running?\n"); + + mp = rte_mempool_lookup(PKTMBUF_POOL_NAME); + if (mp == NULL) + rte_exit(EXIT_FAILURE, "Cannot get mempool for mbufs\n"); + + mz = rte_memzone_lookup(MZ_SHARED_INFO); + if (mz == NULL) + rte_exit(EXIT_FAILURE, "Cannot get port info structure\n"); + info = mz->addr; + tx_stats = &(info->tx_stats[node_id]); + filter_stats = &(info->filter_stats[node_id]); + + configure_output_ports(info); + + h = create_hash_table(info); + + populate_hash_table(h, info); + + RTE_LOG(INFO, APP, "Finished Process Init.\n"); + + printf("\nNode process %d handling packets\n", node_id); + printf("[Press Ctrl-C to quit ...]\n"); + + for (;;) { + uint16_t rx_pkts = PKT_READ_SIZE; + uint8_t port; + + /* + * Try dequeuing max possible packets first, if that fails, + * get the most we can. Loop body should only execute once, + * maximum + */ + while (rx_pkts > 0 && + unlikely(rte_ring_dequeue_bulk(rx_ring, pkts, + rx_pkts) != 0)) + rx_pkts = (uint16_t)RTE_MIN(rte_ring_count(rx_ring), + PKT_READ_SIZE); + + if (unlikely(rx_pkts == 0)) { + if (need_flush) + for (port = 0; port < info->num_ports; port++) { + sent = rte_eth_tx_buffer_flush( + info->id[port], + node_id, + tx_buffer[port]); + if (unlikely(sent)) + tx_stats->tx[port] += sent; + } + need_flush = 0; + continue; + } + + handle_packets(h, (struct rte_mbuf **)pkts, rx_pkts); + + need_flush = 1; + } +} diff --git a/examples/server_node_efd/server/Makefile b/examples/server_node_efd/server/Makefile new file mode 100644 index 0000000..a2f2f36 --- /dev/null +++ b/examples/server_node_efd/server/Makefile @@ -0,0 +1,57 @@ +# BSD LICENSE +# +# Copyright(c) 2016-2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +ifneq ($(CONFIG_RTE_EXEC_ENV), "linuxapp") +$(error This application can only operate in a linuxapp environment, \ +please change the definition of the RTE_TARGET environment variable) +endif + +# binary name +APP = server + +# all source are stored in SRCS-y +SRCS-y := main.c init.c args.c + +INC := $(wildcard *.h) + +CFLAGS += $(WERROR_FLAGS) -O3 +CFLAGS += -I$(SRCDIR)/../shared + +include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/examples/server_node_efd/server/args.c b/examples/server_node_efd/server/args.c new file mode 100644 index 0000000..ee29203 --- /dev/null +++ b/examples/server_node_efd/server/args.c @@ -0,0 +1,200 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "common.h" +#include "args.h" +#include "init.h" + +/* 1M flows by default */ +#define DEFAULT_NUM_FLOWS 0x100000 + +/* global var for number of nodes - extern in header */ +uint8_t num_nodes; +/* global var for number of flows - extern in header */ +uint32_t num_flows = DEFAULT_NUM_FLOWS; + +static const char *progname; + +/** + * Prints out usage information to stdout + */ +static void +usage(void) +{ + printf("%s [EAL options] -- -p PORTMASK -n NUM_NODES -f NUM_FLOWS\n" + " -p PORTMASK: hexadecimal bitmask of ports to use\n" + " -n NUM_NODES: number of node processes to use\n" + " -f NUM_FLOWS: number of flows to be added in the EFD table\n", + progname); +} + +/** + * The ports to be used by the application are passed in + * the form of a bitmask. This function parses the bitmask + * and places the port numbers to be used into the port[] + * array variable + */ +static int +parse_portmask(uint8_t max_ports, const char *portmask) +{ + char *end = NULL; + unsigned long pm; + uint8_t count = 0; + + if (portmask == NULL || *portmask == '\0') + return -1; + + /* convert parameter to a number and verify */ + pm = strtoul(portmask, &end, 16); + if (end == NULL || *end != '\0' || pm == 0) + return -1; + + /* loop through bits of the mask and mark ports */ + while (pm != 0) { + if (pm & 0x01) { /* bit is set in mask, use port */ + if (count >= max_ports) + printf("WARNING: requested port %u not present" + " - ignoring\n", (unsigned int)count); + else + info->id[info->num_ports++] = count; + } + pm = (pm >> 1); + count++; + } + + return 0; +} + +/** + * Take the number of nodes parameter passed to the app + * and convert to a number to store in the num_nodes variable + */ +static int +parse_num_nodes(const char *nodes) +{ + char *end = NULL; + unsigned long temp; + + if (nodes == NULL || *nodes == '\0') + return -1; + + temp = strtoul(nodes, &end, 10); + if (end == NULL || *end != '\0' || temp == 0) + return -1; + + num_nodes = (uint8_t)temp; + return 0; +} + +static int +parse_num_flows(const char *flows) +{ + char *end = NULL; + + /* parse hexadecimal string */ + num_flows = strtoul(flows, &end, 16); + if ((flows[0] == '\0') || (end == NULL) || (*end != '\0')) + return -1; + + if (num_flows == 0) + return -1; + + return 0; +} + +/** + * The application specific arguments follow the DPDK-specific + * arguments which are stripped by the DPDK init. This function + * processes these application arguments, printing usage info + * on error. + */ +int +parse_app_args(uint8_t max_ports, int argc, char *argv[]) +{ + int option_index, opt; + char **argvopt = argv; + static struct option lgopts[] = { /* no long options */ + {NULL, 0, 0, 0 } + }; + progname = argv[0]; + + while ((opt = getopt_long(argc, argvopt, "n:f:p:", lgopts, + &option_index)) != EOF) { + switch (opt) { + case 'p': + if (parse_portmask(max_ports, optarg) != 0) { + usage(); + return -1; + } + break; + case 'n': + if (parse_num_nodes(optarg) != 0) { + usage(); + return -1; + } + break; + case 'f': + if (parse_num_flows(optarg) != 0) { + usage(); + return -1; + } + break; + default: + printf("ERROR: Unknown option '%c'\n", opt); + usage(); + return -1; + } + } + + if (info->num_ports == 0 || num_nodes == 0) { + usage(); + return -1; + } + + if (info->num_ports % 2 != 0) { + printf("ERROR: application requires an even " + "number of ports to use\n"); + return -1; + } + return 0; +} diff --git a/examples/server_node_efd/server/args.h b/examples/server_node_efd/server/args.h new file mode 100644 index 0000000..cacf395 --- /dev/null +++ b/examples/server_node_efd/server/args.h @@ -0,0 +1,39 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _ARGS_H_ +#define _ARGS_H_ + +int parse_app_args(uint8_t max_ports, int argc, char *argv[]); + +#endif /* ifndef _ARGS_H_ */ diff --git a/examples/server_node_efd/server/init.c b/examples/server_node_efd/server/init.c new file mode 100644 index 0000000..82457b4 --- /dev/null +++ b/examples/server_node_efd/server/init.c @@ -0,0 +1,371 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common.h" +#include "args.h" +#include "init.h" + +#define MBUFS_PER_NODE 1536 +#define MBUFS_PER_PORT 1536 +#define MBUF_CACHE_SIZE 512 + +#define RTE_MP_RX_DESC_DEFAULT 512 +#define RTE_MP_TX_DESC_DEFAULT 512 +#define NODE_QUEUE_RINGSIZE 128 + +#define NO_FLAGS 0 + +/* The mbuf pool for packet rx */ +struct rte_mempool *pktmbuf_pool; + +/* array of info/queues for nodes */ +struct node *nodes; + +/* EFD table */ +struct rte_efd_table *efd_table; + +/* Shared info between server and nodes */ +struct shared_info *info; + +/** + * Initialise the mbuf pool for packet reception for the NIC, and any other + * buffer pools needed by the app - currently none. + */ +static int +init_mbuf_pools(void) +{ + const unsigned int num_mbufs = (num_nodes * MBUFS_PER_NODE) + + (info->num_ports * MBUFS_PER_PORT); + + /* + * Don't pass single-producer/single-consumer flags to mbuf create as it + * seems faster to use a cache instead + */ + printf("Creating mbuf pool '%s' [%u mbufs] ...\n", + PKTMBUF_POOL_NAME, num_mbufs); + pktmbuf_pool = rte_pktmbuf_pool_create(PKTMBUF_POOL_NAME, num_mbufs, + MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id()); + + return pktmbuf_pool == NULL; /* 0 on success */ +} + +/** + * Initialise an individual port: + * - configure number of rx and tx rings + * - set up each rx ring, to pull from the main mbuf pool + * - set up each tx ring + * - start the port and report its status to stdout + */ +static int +init_port(uint8_t port_num) +{ + /* for port configuration all features are off by default */ + const struct rte_eth_conf port_conf = { + .rxmode = { + .mq_mode = ETH_MQ_RX_RSS + } + }; + const uint16_t rx_rings = 1, tx_rings = num_nodes; + const uint16_t rx_ring_size = RTE_MP_RX_DESC_DEFAULT; + const uint16_t tx_ring_size = RTE_MP_TX_DESC_DEFAULT; + + uint16_t q; + int retval; + + printf("Port %u init ... ", (unsigned int)port_num); + fflush(stdout); + + /* + * Standard DPDK port initialisation - config port, then set up + * rx and tx rings. + */ + retval = rte_eth_dev_configure(port_num, rx_rings, tx_rings, &port_conf); + if (retval != 0) + return retval; + + for (q = 0; q < rx_rings; q++) { + retval = rte_eth_rx_queue_setup(port_num, q, rx_ring_size, + rte_eth_dev_socket_id(port_num), + NULL, pktmbuf_pool); + if (retval < 0) + return retval; + } + + for (q = 0; q < tx_rings; q++) { + retval = rte_eth_tx_queue_setup(port_num, q, tx_ring_size, + rte_eth_dev_socket_id(port_num), + NULL); + if (retval < 0) + return retval; + } + + rte_eth_promiscuous_enable(port_num); + + retval = rte_eth_dev_start(port_num); + if (retval < 0) + return retval; + + printf("done:\n"); + + return 0; +} + +/** + * Set up the DPDK rings which will be used to pass packets, via + * pointers, between the multi-process server and node processes. + * Each node needs one RX queue. + */ +static int +init_shm_rings(void) +{ + unsigned int i; + unsigned int socket_id; + const char *q_name; + const unsigned int ringsize = NODE_QUEUE_RINGSIZE; + + nodes = rte_malloc("node details", + sizeof(*nodes) * num_nodes, 0); + if (nodes == NULL) + rte_exit(EXIT_FAILURE, "Cannot allocate memory for " + "node program details\n"); + + for (i = 0; i < num_nodes; i++) { + /* Create an RX queue for each node */ + socket_id = rte_socket_id(); + q_name = get_rx_queue_name(i); + nodes[i].rx_q = rte_ring_create(q_name, + ringsize, socket_id, + RING_F_SP_ENQ | RING_F_SC_DEQ); + if (nodes[i].rx_q == NULL) + rte_exit(EXIT_FAILURE, "Cannot create rx ring queue " + "for node %u\n", i); + } + return 0; +} + +/* + * Create EFD table which will contain all the flows + * that will be distributed among the nodes + */ +static void +create_efd_table(void) +{ + uint8_t socket_id = rte_socket_id(); + + /* create table */ + efd_table = rte_efd_create("flow table", num_flows * 2, sizeof(uint32_t), + 1 << socket_id, socket_id); + + if (efd_table == NULL) + rte_exit(EXIT_FAILURE, "Problem creating the flow table\n"); +} + +static void +populate_efd_table(void) +{ + unsigned int i; + int32_t ret; + uint32_t ip_dst; + uint8_t socket_id = rte_socket_id(); + uint64_t node_id; + + /* Add flows in table */ + for (i = 0; i < num_flows; i++) { + node_id = i % num_nodes; + + ip_dst = rte_cpu_to_be_32(i); + ret = rte_efd_update(efd_table, socket_id, + (void *)&ip_dst, (efd_value_t)node_id); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Unable to add entry %u in " + "EFD table\n", i); + } + + printf("EFD table: Adding 0x%x keys\n", num_flows); +} + +/* Check the link status of all ports in up to 9s, and print them finally */ +static void +check_all_ports_link_status(uint8_t port_num, uint32_t port_mask) +{ +#define CHECK_INTERVAL 100 /* 100ms */ +#define MAX_CHECK_TIME 90 /* 9s (90 * 100ms) in total */ + uint8_t portid, count, all_ports_up, print_flag = 0; + struct rte_eth_link link; + + printf("\nChecking link status"); + fflush(stdout); + for (count = 0; count <= MAX_CHECK_TIME; count++) { + all_ports_up = 1; + for (portid = 0; portid < port_num; portid++) { + if ((port_mask & (1 << info->id[portid])) == 0) + continue; + memset(&link, 0, sizeof(link)); + rte_eth_link_get_nowait(info->id[portid], &link); + /* print link status if flag set */ + if (print_flag == 1) { + if (link.link_status) + printf("Port %d Link Up - speed %u " + "Mbps - %s\n", info->id[portid], + (unsigned int)link.link_speed, + (link.link_duplex == ETH_LINK_FULL_DUPLEX) ? + ("full-duplex") : ("half-duplex\n")); + else + printf("Port %d Link Down\n", + (uint8_t)info->id[portid]); + continue; + } + /* clear all_ports_up flag if any link down */ + if (link.link_status == ETH_LINK_DOWN) { + all_ports_up = 0; + break; + } + } + /* after finally printing all link status, get out */ + if (print_flag == 1) + break; + + if (all_ports_up == 0) { + printf("."); + fflush(stdout); + rte_delay_ms(CHECK_INTERVAL); + } + + /* set the print_flag if all ports up or timeout */ + if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) { + print_flag = 1; + printf("done\n"); + } + } +} + +/** + * Main init function for the multi-process server app, + * calls subfunctions to do each stage of the initialisation. + */ +int +init(int argc, char *argv[]) +{ + int retval; + const struct rte_memzone *mz; + uint8_t i, total_ports; + + /* init EAL, parsing EAL args */ + retval = rte_eal_init(argc, argv); + if (retval < 0) + return -1; + argc -= retval; + argv += retval; + + /* get total number of ports */ + total_ports = rte_eth_dev_count(); + + /* set up array for port data */ + mz = rte_memzone_reserve(MZ_SHARED_INFO, sizeof(*info), + rte_socket_id(), NO_FLAGS); + if (mz == NULL) + rte_exit(EXIT_FAILURE, "Cannot reserve memory zone " + "for port information\n"); + memset(mz->addr, 0, sizeof(*info)); + info = mz->addr; + + /* parse additional, application arguments */ + retval = parse_app_args(total_ports, argc, argv); + if (retval != 0) + return -1; + + /* initialise mbuf pools */ + retval = init_mbuf_pools(); + if (retval != 0) + rte_exit(EXIT_FAILURE, "Cannot create needed mbuf pools\n"); + + /* now initialise the ports we will use */ + for (i = 0; i < info->num_ports; i++) { + retval = init_port(info->id[i]); + if (retval != 0) + rte_exit(EXIT_FAILURE, "Cannot initialise port %u\n", + (unsigned int) i); + } + + check_all_ports_link_status(info->num_ports, (~0x0)); + + /* initialise the node queues/rings for inter-eu comms */ + init_shm_rings(); + + /* Create the EFD table */ + create_efd_table(); + + /* Populate the EFD table */ + populate_efd_table(); + + /* Share the total number of nodes */ + info->num_nodes = num_nodes; + + /* Share the total number of flows */ + info->num_flows = num_flows; + return 0; +} diff --git a/examples/server_node_efd/server/init.h b/examples/server_node_efd/server/init.h new file mode 100644 index 0000000..8dc5885 --- /dev/null +++ b/examples/server_node_efd/server/init.h @@ -0,0 +1,76 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _INIT_H_ +#define _INIT_H_ + +/* + * #include + * #include "args.h" + */ + +/* + * Define a node structure with all needed info, including + * stats from the nodes. + */ +struct node { + struct rte_ring *rx_q; + unsigned int node_id; + /* these stats hold how many packets the node will actually receive, + * and how many packets were dropped because the node's queue was full. + * The port-info stats, in contrast, record how many packets were received + * or transmitted on an actual NIC port. + */ + struct { + uint64_t rx; + uint64_t rx_drop; + } stats; +}; + +extern struct rte_efd_table *efd_table; +extern struct node *nodes; + +/* + * shared information between server and nodes: number of nodes, + * port numbers, rx and tx stats etc. + */ +extern struct shared_info *info; + +extern struct rte_mempool *pktmbuf_pool; +extern uint8_t num_nodes; +extern unsigned int num_sockets; +extern uint32_t num_flows; + +int init(int argc, char *argv[]); + +#endif /* ifndef _INIT_H_ */ diff --git a/examples/server_node_efd/server/main.c b/examples/server_node_efd/server/main.c new file mode 100644 index 0000000..1a54d1b --- /dev/null +++ b/examples/server_node_efd/server/main.c @@ -0,0 +1,362 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common.h" +#include "args.h" +#include "init.h" + +/* + * When doing reads from the NIC or the node queues, + * use this batch size + */ +#define PACKET_READ_SIZE 32 + +/* + * Local buffers to put packets in, used to send packets in bursts to the + * nodes + */ +struct node_rx_buf { + struct rte_mbuf *buffer[PACKET_READ_SIZE]; + uint16_t count; +}; + +struct efd_stats { + uint64_t distributed; + uint64_t drop; +} flow_dist_stats; + +/* One buffer per node rx queue - dynamically allocate array */ +static struct node_rx_buf *cl_rx_buf; + +static const char * +get_printable_mac_addr(uint8_t port) +{ + static const char err_address[] = "00:00:00:00:00:00"; + static char addresses[RTE_MAX_ETHPORTS][sizeof(err_address)]; + struct ether_addr mac; + + if (unlikely(port >= RTE_MAX_ETHPORTS)) + return err_address; + if (unlikely(addresses[port][0] == '\0')) { + rte_eth_macaddr_get(port, &mac); + snprintf(addresses[port], sizeof(addresses[port]), + "%02x:%02x:%02x:%02x:%02x:%02x\n", + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); + } + return addresses[port]; +} + +/* + * This function displays the recorded statistics for each port + * and for each node. It uses ANSI terminal codes to clear + * screen when called. It is called from a single non-master + * thread in the server process, when the process is run with more + * than one lcore enabled. + */ +static void +do_stats_display(void) +{ + unsigned int i, j; + const char clr[] = {27, '[', '2', 'J', '\0'}; + const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'}; + uint64_t port_tx[RTE_MAX_ETHPORTS], port_tx_drop[RTE_MAX_ETHPORTS]; + uint64_t node_tx[MAX_NODES], node_tx_drop[MAX_NODES]; + + /* to get TX stats, we need to do some summing calculations */ + memset(port_tx, 0, sizeof(port_tx)); + memset(port_tx_drop, 0, sizeof(port_tx_drop)); + memset(node_tx, 0, sizeof(node_tx)); + memset(node_tx_drop, 0, sizeof(node_tx_drop)); + + for (i = 0; i < num_nodes; i++) { + const struct tx_stats *tx = &info->tx_stats[i]; + + for (j = 0; j < info->num_ports; j++) { + const uint64_t tx_val = tx->tx[info->id[j]]; + const uint64_t drop_val = tx->tx_drop[info->id[j]]; + + port_tx[j] += tx_val; + port_tx_drop[j] += drop_val; + node_tx[i] += tx_val; + node_tx_drop[i] += drop_val; + } + } + + /* Clear screen and move to top left */ + printf("%s%s", clr, topLeft); + + printf("PORTS\n"); + printf("-----\n"); + for (i = 0; i < info->num_ports; i++) + printf("Port %u: '%s'\t", (unsigned int)info->id[i], + get_printable_mac_addr(info->id[i])); + printf("\n\n"); + for (i = 0; i < info->num_ports; i++) { + printf("Port %u - rx: %9"PRIu64"\t" + "tx: %9"PRIu64"\n", + (unsigned int)info->id[i], info->rx_stats.rx[i], + port_tx[i]); + } + + printf("\nSERVER\n"); + printf("-----\n"); + printf("distributed: %9"PRIu64", drop: %9"PRIu64"\n", + flow_dist_stats.distributed, flow_dist_stats.drop); + + printf("\nNODES\n"); + printf("-------\n"); + for (i = 0; i < num_nodes; i++) { + const unsigned long long rx = nodes[i].stats.rx; + const unsigned long long rx_drop = nodes[i].stats.rx_drop; + const struct filter_stats *filter = &info->filter_stats[i]; + + printf("Node %2u - rx: %9llu, rx_drop: %9llu\n" + " tx: %9"PRIu64", tx_drop: %9"PRIu64"\n" + " filter_passed: %9"PRIu64", " + "filter_drop: %9"PRIu64"\n", + i, rx, rx_drop, node_tx[i], node_tx_drop[i], + filter->passed, filter->drop); + } + + printf("\n"); +} + +/* + * The function called from each non-master lcore used by the process. + * The test_and_set function is used to randomly pick a single lcore on which + * the code to display the statistics will run. Otherwise, the code just + * repeatedly sleeps. + */ +static int +sleep_lcore(__attribute__((unused)) void *dummy) +{ + /* Used to pick a display thread - static, so zero-initialised */ + static rte_atomic32_t display_stats; + + /* Only one core should display stats */ + if (rte_atomic32_test_and_set(&display_stats)) { + const unsigned int sleeptime = 1; + + printf("Core %u displaying statistics\n", rte_lcore_id()); + + /* Longer initial pause so above printf is seen */ + sleep(sleeptime * 3); + + /* Loop forever: sleep always returns 0 or <= param */ + while (sleep(sleeptime) <= sleeptime) + do_stats_display(); + } + return 0; +} + +/* + * Function to set all the node statistic values to zero. + * Called at program startup. + */ +static void +clear_stats(void) +{ + unsigned int i; + + for (i = 0; i < num_nodes; i++) + nodes[i].stats.rx = nodes[i].stats.rx_drop = 0; +} + +/* + * send a burst of traffic to a node, assuming there are packets + * available to be sent to this node + */ +static void +flush_rx_queue(uint16_t node) +{ + uint16_t j; + struct node *cl; + + if (cl_rx_buf[node].count == 0) + return; + + cl = &nodes[node]; + if (rte_ring_enqueue_bulk(cl->rx_q, (void **)cl_rx_buf[node].buffer, + cl_rx_buf[node].count) != 0){ + for (j = 0; j < cl_rx_buf[node].count; j++) + rte_pktmbuf_free(cl_rx_buf[node].buffer[j]); + cl->stats.rx_drop += cl_rx_buf[node].count; + } else + cl->stats.rx += cl_rx_buf[node].count; + + cl_rx_buf[node].count = 0; +} + +/* + * marks a packet down to be sent to a particular node process + */ +static inline void +enqueue_rx_packet(uint8_t node, struct rte_mbuf *buf) +{ + cl_rx_buf[node].buffer[cl_rx_buf[node].count++] = buf; +} + +/* + * This function takes a group of packets and routes them + * individually to the node process. Very simply round-robins the packets + * without checking any of the packet contents. + */ +static void +process_packets(uint32_t port_num __rte_unused, struct rte_mbuf *pkts[], + uint16_t rx_count, unsigned int socket_id) +{ + uint16_t i; + uint8_t node; + efd_value_t data[RTE_EFD_BURST_MAX]; + const void *key_ptrs[RTE_EFD_BURST_MAX]; + + struct ipv4_hdr *ipv4_hdr; + uint32_t ipv4_dst_ip[RTE_EFD_BURST_MAX]; + + for (i = 0; i < rx_count; i++) { + /* Handle IPv4 header.*/ + ipv4_hdr = rte_pktmbuf_mtod_offset(pkts[i], struct ipv4_hdr *, + sizeof(struct ether_hdr)); + ipv4_dst_ip[i] = ipv4_hdr->dst_addr; + key_ptrs[i] = (void *)&ipv4_dst_ip[i]; + } + + rte_efd_lookup_bulk(efd_table, socket_id, rx_count, + (const void **) key_ptrs, data); + for (i = 0; i < rx_count; i++) { + node = (uint8_t) ((uintptr_t)data[i]); + + if (node >= num_nodes) { + /* + * Node is out of range, which means that + * flow has not been inserted + */ + flow_dist_stats.drop++; + rte_pktmbuf_free(pkts[i]); + } else { + flow_dist_stats.distributed++; + enqueue_rx_packet(node, pkts[i]); + } + } + + for (i = 0; i < num_nodes; i++) + flush_rx_queue(i); +} + +/* + * Function called by the master lcore of the DPDK process. + */ +static void +do_packet_forwarding(void) +{ + unsigned int port_num = 0; /* indexes the port[] array */ + unsigned int socket_id = rte_socket_id(); + + for (;;) { + struct rte_mbuf *buf[PACKET_READ_SIZE]; + uint16_t rx_count; + + /* read a port */ + rx_count = rte_eth_rx_burst(info->id[port_num], 0, + buf, PACKET_READ_SIZE); + info->rx_stats.rx[port_num] += rx_count; + + /* Now process the NIC packets read */ + if (likely(rx_count > 0)) + process_packets(port_num, buf, rx_count, socket_id); + + /* move to next port */ + if (++port_num == info->num_ports) + port_num = 0; + } +} + +int +main(int argc, char *argv[]) +{ + /* initialise the system */ + if (init(argc, argv) < 0) + return -1; + RTE_LOG(INFO, APP, "Finished Process Init.\n"); + + cl_rx_buf = calloc(num_nodes, sizeof(cl_rx_buf[0])); + + /* clear statistics */ + clear_stats(); + + /* put all other cores to sleep bar master */ + rte_eal_mp_remote_launch(sleep_lcore, NULL, SKIP_MASTER); + + do_packet_forwarding(); + return 0; +} diff --git a/examples/server_node_efd/shared/common.h b/examples/server_node_efd/shared/common.h new file mode 100644 index 0000000..8a13479 --- /dev/null +++ b/examples/server_node_efd/shared/common.h @@ -0,0 +1,99 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _COMMON_H_ +#define _COMMON_H_ + +#include +#include + +#define MAX_NODES 16 +/* + * Shared port info, including statistics information for display by server. + * Structure will be put in a memzone. + * - All port id values share one cache line as this data will be read-only + * during operation. + * - All rx statistic values share cache lines, as this data is written only + * by the server process. (rare reads by stats display) + * - The tx statistics have values for all ports per cache line, but the stats + * themselves are written by the nodes, so we have a distinct set, on different + * cache lines for each node to use. + */ +struct rx_stats { + uint64_t rx[RTE_MAX_ETHPORTS]; +} __rte_cache_aligned; + +struct tx_stats { + uint64_t tx[RTE_MAX_ETHPORTS]; + uint64_t tx_drop[RTE_MAX_ETHPORTS]; +} __rte_cache_aligned; + +struct filter_stats { + uint64_t drop; + uint64_t passed; +} __rte_cache_aligned; + +struct shared_info { + uint8_t num_nodes; + uint8_t num_ports; + uint32_t num_flows; + uint8_t id[RTE_MAX_ETHPORTS]; + struct rx_stats rx_stats; + struct tx_stats tx_stats[MAX_NODES]; + struct filter_stats filter_stats[MAX_NODES]; +}; + +/* define common names for structures shared between server and node */ +#define MP_NODE_RXQ_NAME "MProc_Node_%u_RX" +#define PKTMBUF_POOL_NAME "MProc_pktmbuf_pool" +#define MZ_SHARED_INFO "MProc_shared_info" + +/* + * Given the rx queue name template above, get the queue name + */ +static inline const char * +get_rx_queue_name(unsigned int id) +{ + /* + * Buffer for return value. Size calculated by %u being replaced + * by maximum 3 digits (plus an extra byte for safety) + */ + static char buffer[sizeof(MP_NODE_RXQ_NAME) + 2]; + + snprintf(buffer, sizeof(buffer) - 1, MP_NODE_RXQ_NAME, id); + return buffer; +} + +#define RTE_LOGTYPE_APP RTE_LOGTYPE_USER1 + +#endif