Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/52449/?format=api
{ "id": 52449, "url": "http://patches.dpdk.org/api/patches/52449/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/20190409063344.36564-3-vipin.varghese@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20190409063344.36564-3-vipin.varghese@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20190409063344.36564-3-vipin.varghese@intel.com", "date": "2019-04-09T06:33:44", "name": "[v8,2/2] doc: add guide for debug and troubleshoot", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": true, "hash": "88e128925a3c8ee83541de6d64d15e6acafd50d0", "submitter": { "id": 882, "url": "http://patches.dpdk.org/api/people/882/?format=api", "name": "Varghese, Vipin", "email": "vipin.varghese@intel.com" }, "delegate": { "id": 1, "url": "http://patches.dpdk.org/api/users/1/?format=api", "username": "tmonjalo", "first_name": "Thomas", "last_name": "Monjalon", "email": "thomas@monjalon.net" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/20190409063344.36564-3-vipin.varghese@intel.com/mbox/", "series": [ { "id": 4192, "url": "http://patches.dpdk.org/api/series/4192/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=4192", "date": "2019-04-09T06:33:42", "name": "guide to debug and troubleshoot.", "version": 8, "mbox": "http://patches.dpdk.org/series/4192/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/52449/comments/", "check": "success", "checks": "http://patches.dpdk.org/api/patches/52449/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 709DF559A;\n\tTue, 9 Apr 2019 08:33:37 +0200 (CEST)", "from mga04.intel.com (mga04.intel.com [192.55.52.120])\n\tby dpdk.org (Postfix) with ESMTP id 0DEB75323\n\tfor <dev@dpdk.org>; Tue, 9 Apr 2019 08:33:33 +0200 (CEST)", "from fmsmga005.fm.intel.com ([10.253.24.32])\n\tby fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t08 Apr 2019 23:33:33 -0700", "from unknown (HELO saesrv02-S2600CWR.intel.com) ([10.224.122.203])\n\tby fmsmga005.fm.intel.com with ESMTP; 08 Apr 2019 23:33:30 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.60,328,1549958400\"; d=\"scan'208\";a=\"336190515\"", "From": "Vipin Varghese <vipin.varghese@intel.com>", "To": "dev@dpdk.org, marko.kovacevic@intel.com, john.mcnamara@intel.com,\n\tshreyansh.jain@nxp.com", "Cc": "keith.wiles@intel.com, amit.tamboli@intel.com, sanjay.padubidri@intel.com,\n\tamol.patel@intel.com, ferruh.yigit@intel.com,\n\tVipin Varghese <vipin.varghese@intel.com>", "Date": "Tue, 9 Apr 2019 12:03:44 +0530", "Message-Id": "<20190409063344.36564-3-vipin.varghese@intel.com>", "X-Mailer": "git-send-email 2.17.1", "In-Reply-To": "<20190409063344.36564-1-vipin.varghese@intel.com>", "References": "<20190225171222.64134-3-vipin.varghese@intel.com>\n\t<20190409063344.36564-1-vipin.varghese@intel.com>", "Subject": "[dpdk-dev] [PATCH v8 2/2] doc: add guide for debug and troubleshoot", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "Add user guide on debugging and troubleshooting for common\nissues and bottleneck found in the sample application model.\n\nSigned-off-by: Vipin Varghese <vipin.varghese@intel.com>\nAcked-by: Marko Kovacevic <marko.kovacevic@intel.com>\nAcked-by: John McNamara <john.mcnamara@intel.com>\n---\n doc/guides/howto/debug_troubleshoot_guide.rst | 464 ++++++++++++++++++\n doc/guides/howto/index.rst | 1 +\n 2 files changed, 465 insertions(+)\n create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst", "diff": "diff --git a/doc/guides/howto/debug_troubleshoot_guide.rst b/doc/guides/howto/debug_troubleshoot_guide.rst\nnew file mode 100644\nindex 000000000..6abddcd32\n--- /dev/null\n+++ b/doc/guides/howto/debug_troubleshoot_guide.rst\n@@ -0,0 +1,464 @@\n+.. SPDX-License-Identifier: BSD-3-Clause\n+ Copyright(c) 2018 Intel Corporation.\n+\n+Debug & Troubleshoot guide\n+==========================\n+\n+DPDK applications can be designed to have simple or complex pipeline processing\n+stages making use of single or multiple threads. Applications can use poll mode\n+hardware devices which helps in offloading CPU cycles too. It is common to find\n+solutions designed with\n+\n+* single or multiple primary processes\n+\n+* single primary and single secondary\n+\n+* single primary and multiple secondaries\n+\n+In all the above cases, it is tedious to isolate, debug, and understand various\n+behaviors which occur randomly or periodically. The goal of the guide is to\n+consolidate a few commonly seen issues for reference. Then, isolate to identify\n+the root cause through step by step debug at various stages.\n+\n+.. note::\n+\n+ It is difficult to cover all possible issues; in a single attempt. With\n+ feedback and suggestions from the community, more cases can be covered.\n+\n+\n+Application Overview\n+--------------------\n+\n+By making use of the application model as a reference, we can discuss multiple\n+causes of issues in the guide. Let us assume the sample makes use of a single\n+primary process, with various processing stages running on multiple cores. The\n+application may also make uses of Poll Mode Driver, and libraries like service\n+cores, mempool, mbuf, eventdev, cryptodev, QoS, and ethdev.\n+\n+The overview of an application modeled using PMD is shown in\n+:numref:`dtg_sample_app_model`.\n+\n+.. _dtg_sample_app_model:\n+\n+.. figure:: img/dtg_sample_app_model.*\n+\n+ Overview of pipeline stage of an application\n+\n+\n+Bottleneck Analysis\n+-------------------\n+\n+A couple of factors that lead the design decision could be the platform, scale\n+factor, and target. This distinct preference leads to multiple combinations,\n+that are built using PMD and libraries of DPDK. While the compiler, library\n+mode, and optimization flags are the components are to be constant, that\n+affects the application too.\n+\n+\n+Is there mismatch in packet (received < desired) rate?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+RX Port and associated core :numref:`dtg_rx_rate`.\n+\n+.. _dtg_rx_rate:\n+\n+.. figure:: img/dtg_rx_rate.*\n+\n+ RX packet rate compared against received rate.\n+\n+#. Is the configuration for the RX setup correctly?\n+\n+ * Identify if port Speed and Duplex is matching to desired values with\n+ ``rte_eth_link_get``.\n+\n+ * Check ``DEV_RX_OFFLOAD_JUMBO_FRAME`` is set with ``rte_eth_dev_info_get``.\n+\n+ * Check promiscuous mode if the drops do not occur for unique MAC address\n+ with ``rte_eth_promiscuous_get``.\n+\n+#. Is the drop isolated to certain NIC only?\n+\n+ * Make use of ``rte_eth_dev_stats`` to identify the drops cause.\n+\n+ * If there are mbuf drops, check nb_desc for RX descriptor as it might not\n+ be sufficient for the application.\n+\n+ * If ``rte_eth_dev_stats`` shows drops are on specific RX queues, ensure RX\n+ lcore threads has enough cycles for ``rte_eth_rx_burst`` on the port queue\n+ pair.\n+\n+ * If there are redirect to a specific port queue pair with, ensure RX lcore\n+ threads gets enough cycles.\n+\n+ * Check the RSS configuration ``rte_eth_dev_rss_hash_conf_get`` if the\n+ spread is not even and causing drops.\n+\n+ * If PMD stats are not updating, then there might be offload or configuration\n+ which is dropping the incoming traffic.\n+\n+#. Is there drops still seen?\n+\n+ * If there are multiple port queue pair, it might be the RX thread, RX\n+ distributor, or event RX adapter not having enough cycles.\n+\n+ * If there are drops seen for RX adapter or RX distributor, try using\n+ ``rte_prefetch_non_temporal`` which intimates the core that the mbuf in the\n+ cache is temporary.\n+\n+\n+Is there packet drops at receive or transmit?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+RX-TX port and associated cores :numref:`dtg_rx_tx_drop`.\n+\n+.. _dtg_rx_tx_drop:\n+\n+.. figure:: img/dtg_rx_tx_drop.*\n+\n+ RX-TX drops\n+\n+#. At RX\n+\n+ * Identify if there are multiple RX queue configured for port by\n+ ``nb_rx_queues`` using ``rte_eth_dev_info_get``.\n+\n+ * Using ``rte_eth_dev_stats`` fetch drops in q_errors, check if RX thread\n+ is configured to fetch packets from the port queue pair.\n+\n+ * Using ``rte_eth_dev_stats`` shows drops in ``rx_nombuf``, check if RX\n+ thread has enough cycles to consume the packets from the queue.\n+\n+#. At TX\n+\n+ * If the TX rate is falling behind the application fill rate, identify if\n+ there are enough descriptors with ``rte_eth_dev_info_get`` for TX.\n+\n+ * Check the ``nb_pkt`` in ``rte_eth_tx_burst`` is done for multiple packets.\n+\n+ * Check ``rte_eth_tx_burst`` invokes the vector function call for the PMD.\n+\n+ * If oerrors are getting incremented, TX packet validations are failing.\n+ Check if there queue specific offload failures.\n+\n+ * If the drops occur for large size packets, check MTU and multi-segment\n+ support configured for NIC.\n+\n+\n+Is there object drops in producer point for the ring library?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Producer point for ring :numref:`dtg_producer_ring`.\n+\n+.. _dtg_producer_ring:\n+\n+.. figure:: img/dtg_producer_ring.*\n+\n+ Producer point for Rings\n+\n+#. Performance issue isolation at producer\n+\n+ * Use ``rte_ring_dump`` to validate for all single producer flag is set to\n+ ``RING_F_SP_ENQ``.\n+\n+ * There should be sufficient ``rte_ring_free_count`` at any point in time.\n+\n+ * Extreme stalls in dequeue stage of the pipeline will cause\n+ ``rte_ring_full`` to be true.\n+\n+\n+Is there object drops in consumer point for the ring library?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Consumer point for ring :numref:`dtg_consumer_ring`.\n+\n+.. _dtg_consumer_ring:\n+\n+.. figure:: img/dtg_consumer_ring.*\n+\n+ Consumer point for Rings\n+\n+#. Performance issue isolation at consumer\n+\n+ * Use ``rte_ring_dump`` to validate for all single consumer flag is set to\n+ ``RING_F_SC_DEQ``.\n+\n+ * If the desired burst dequeue falls behind the actual dequeue, the enqueue\n+ stage is not filling up the ring as required.\n+\n+ * Extreme stall in the enqueue will lead to ``rte_ring_empty`` to be true.\n+\n+\n+Is there a variance in packet or object processing rate in the pipeline?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Memory objects close to NUMA :numref:`dtg_mempool`.\n+\n+.. _dtg_mempool:\n+\n+.. figure:: img/dtg_mempool.*\n+\n+ Memory objects have to be close to the device per NUMA.\n+\n+#. Stall in processing pipeline can be attributes of MBUF release delays.\n+ These can be narrowed down to\n+\n+ * Heavy processing cycles at single or multiple processing stages.\n+\n+ * Cache is spread due to the increased stages in the pipeline.\n+\n+ * CPU thread responsible for TX is not able to keep up with the burst of\n+ traffic.\n+\n+ * Extra cycles to linearize multi-segment buffer and software offload like\n+ checksum, TSO, and VLAN strip.\n+\n+ * Packet buffer copy in fast path also results in stalls in MBUF release if\n+ not done selectively.\n+\n+ * Application logic sets ``rte_pktmbuf_refcnt_set`` to higher than the\n+ desired value and frequently uses ``rte_pktmbuf_prefree_seg`` and does\n+ not release MBUF back to mempool.\n+\n+#. Lower performance between the pipeline processing stages can be\n+\n+ * The NUMA instance for packets or objects from NIC, mempool, and ring\n+ should be the same.\n+\n+ * Drops on a specific socket are due to insufficient objects in the pool.\n+ Use ``rte_mempool_get_count`` or ``rte_mempool_avail_count`` to monitor\n+ when drops occurs.\n+\n+ * Try prefetching the content in processing pipeline logic to minimize the\n+ stalls.\n+\n+#. Performance issue can be due to special cases\n+\n+ * Check if MBUF continuous with ``rte_pktmbuf_is_contiguous`` as certain\n+ offload requires the same.\n+\n+ * Use ``rte_mempool_cache_create`` for user threads require access to\n+ mempool objects.\n+\n+ * If the variance is absent for larger huge pages, then try rte_mem_lock_page\n+ on the objects, packets, lookup tables to isolate the issue.\n+\n+\n+Is there a variance in cryptodev performance?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Crypto device and PMD :numref:`dtg_crypto`.\n+\n+.. _dtg_crypto:\n+\n+.. figure:: img/dtg_crypto.*\n+\n+ CRYPTO and interaction with PMD device.\n+\n+#. Performance issue isolation for enqueue\n+\n+ * Ensure cryptodev, resources and enqueue is running on NUMA cores.\n+\n+ * Isolate if the cause of errors for err_count using ``rte_cryptodev_stats``.\n+\n+ * Parallelize enqueue thread for varied multiple queue pair.\n+\n+#. Performance issue isolation for dequeue\n+\n+ * Ensure cryptodev, resources and dequeue are running on NUMA cores.\n+\n+ * Isolate if the cause of errors for err_count using ``rte_cryptodev_stats``.\n+\n+ * Parallelize dequeue thread for varied multiple queue pair.\n+\n+#. Performance issue isolation for crypto operation\n+\n+ * If the cryptodev software-assist is in use, ensure the library is built\n+ with right (SIMD) flags or check if the queue pair using CPU ISA for\n+ feature_flags AVX|SSE|NEON using ``rte_cryptodev_info_get``.\n+\n+ * If the cryptodev hardware-assist is in use, ensure both firmware and\n+ drivers are up to date.\n+\n+#. Configuration issue isolation\n+\n+ * Identify cryptodev instances with ``rte_cryptodev_count`` and\n+ ``rte_cryptodev_info_get``.\n+\n+\n+Is user functions performance is not as expected?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Custom worker function :numref:`dtg_distributor_worker`.\n+\n+.. _dtg_distributor_worker:\n+\n+.. figure:: img/dtg_distributor_worker.*\n+\n+ Custom worker function performance drops.\n+\n+#. Performance issue isolation\n+\n+ * The functions running on CPU cores without context switches are the\n+ performing scenarios. Identify lcore with ``rte_lcore`` and lcore index\n+ mapping with CPU using ``rte_lcore_index``.\n+\n+ * The functions running on CPU cores without context switches are the\n+ performing scenarios. Identify lcore with ``rte_lcore`` and lcore index\n+ mapping with CPU using ``rte_lcore_index``.\n+\n+ * Use ``rte_thread_get_affinity`` to isolate functions running on the same\n+ CPU core.\n+\n+#. Configuration issue isolation\n+\n+ * Identify core role using ``rte_eal_lcore_role`` to identify RTE, OFF and\n+ SERVICE. Check performance functions are mapped to run on the cores.\n+\n+ * For high-performance execution logic ensure running it on correct NUMA\n+ and non-master core.\n+\n+ * Analyze run logic with ``rte_dump_stack``, ``rte_dump_registers`` and\n+ ``rte_memdump`` for more insights.\n+\n+ * Make use of objdump to ensure opcode is matching to the desired state.\n+\n+\n+Is the execution cycles for dynamic service functions are not frequent?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+service functions on service cores :numref:`dtg_service`.\n+\n+.. _dtg_service:\n+\n+.. figure:: img/dtg_service.*\n+\n+ functions running on service cores\n+\n+#. Performance issue isolation\n+\n+ * Services configured for parallel execution should have\n+ ``rte_service_lcore_count`` should be equal to\n+ ``rte_service_lcore_count_services``.\n+\n+ * A service to run parallel on all cores should return\n+ ``RTE_SERVICE_CAP_MT_SAFE`` for ``rte_service_probe_capability`` and\n+ ``rte_service_map_lcore_get`` returns unique lcore.\n+\n+ * If service function execution cycles for dynamic service functions are\n+ not frequent?\n+\n+ * If services share the lcore, overall execution should fit budget.\n+\n+#. Configuration issue isolation\n+\n+ * Check if service is running with ``rte_service_runstate_get``.\n+\n+ * Generic debug via ``rte_service_dump``.\n+\n+\n+Is there a bottleneck in the performance of eventdev?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+#. Check for generic configuration\n+\n+ * Ensure the event devices created are right NUMA using\n+ ``rte_event_dev_count`` and ``rte_event_dev_socket_id``.\n+\n+ * Check for event stages if the events are looped back into the same queue.\n+\n+ * If the failure is on the enqueue stage for events, check if queue depth\n+ with ``rte_event_dev_info_get``.\n+\n+#. If there are performance drops in the enqueue stage\n+\n+ * Use ``rte_event_dev_dump`` to dump the eventdev information.\n+\n+ * Periodically checks stats for queue and port to identify the starvation.\n+\n+ * Check the in-flight events for the desired queue for enqueue and dequeue.\n+\n+\n+Is there a variance in traffic manager?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Traffic Manager on TX interface :numref:`dtg_qos_tx`.\n+\n+.. _dtg_qos_tx:\n+\n+.. figure:: img/dtg_qos_tx.*\n+\n+ Traffic Manager just before TX.\n+\n+#. Identify the cause for a variance from expected behavior, is due to\n+ insufficient CPU cycles. Use ``rte_tm_capabilities_get`` to fetch features\n+ for hierarchies, WRED and priority schedulers to be offloaded hardware.\n+\n+#. Undesired flow drops can be narrowed down to WRED, priority, and rates\n+ limiters.\n+\n+#. Isolate the flow in which the undesired drops occur. Use\n+ ``rte_tn_get_number_of_leaf_node`` and flow table to ping down the leaf\n+ where drops occur.\n+\n+#. Check the stats using ``rte_tm_stats_update`` and ``rte_tm_node_stats_read``\n+ for drops for hierarchy, schedulers and WRED configurations.\n+\n+\n+Is the packet not in the unexpected format?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Packet capture before and after processing :numref:`dtg_pdump`.\n+\n+.. _dtg_pdump:\n+\n+.. figure:: img/dtg_pdump.*\n+\n+ Capture points of Traffic at RX-TX.\n+\n+#. To isolate the possible packet corruption in the processing pipeline,\n+ carefully staged capture packets are to be implemented.\n+\n+ * First, isolate at NIC entry and exit.\n+\n+ Use pdump in primary to allow secondary to access port-queue pair. The\n+ packets get copied over in RX|TX callback by the secondary process using\n+ ring buffers.\n+\n+ * Second, isolate at pipeline entry and exit.\n+\n+ Using hooks or callbacks capture the packet middle of the pipeline stage\n+ to copy the packets, which can be shared to the secondary debug process\n+ via user-defined custom rings.\n+\n+.. note::\n+\n+ Use similar analysis to objects and metadata corruption.\n+\n+\n+Does the issue still persist?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The issue can be further narrowed down to the following causes.\n+\n+#. If there are vendor or application specific metadata, check for errors due\n+ to META data error flags. Dumping private meta-data in the objects can give\n+ insight into details for debugging.\n+\n+#. If there are multi-process for either data or configuration, check for\n+ possible errors in the secondary process where the configuration fails and\n+ possible data corruption in the data plane.\n+\n+#. Random drops in the RX or TX when opening other application is an indication\n+ of the effect of a noisy neighbor. Try using the cache allocation technique\n+ to minimize the effect between applications.\n+\n+\n+How to develop a custom code to debug?\n+--------------------------------------\n+\n+#. For an application that runs as the primary process only, debug functionality\n+ is added in the same process. These can be invoked by timer call-back,\n+ service core and signal handler.\n+\n+#. For the application that runs as multiple processes. debug functionality in\n+ a standalone secondary process.\ndiff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst\nindex a642a2be1..9527fa84d 100644\n--- a/doc/guides/howto/index.rst\n+++ b/doc/guides/howto/index.rst\n@@ -18,3 +18,4 @@ HowTo Guides\n virtio_user_as_exceptional_path\n packet_capture_framework\n telemetry\n+ debug_troubleshoot_guide\n", "prefixes": [ "v8", "2/2" ] }