From patchwork Thu Dec 5 17:31:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 63602 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2A52DA04F2; Thu, 5 Dec 2019 18:34:12 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 773C91BF81; Thu, 5 Dec 2019 18:34:11 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 8E2511BF7D for ; Thu, 5 Dec 2019 18:34:09 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Dec 2019 09:34:08 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,282,1571727600"; d="scan'208";a="223711791" Received: from silpixa00399953.ir.intel.com (HELO silpixa00399953.ger.corp.intel.com) ([10.237.222.53]) by orsmga002.jf.intel.com with ESMTP; 05 Dec 2019 09:34:07 -0800 From: Ciara Power To: dev@dpdk.org Cc: Bruce Richardson Date: Thu, 5 Dec 2019 17:31:22 +0000 Message-Id: <20191205173128.64543-1-ciara.power@intel.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 Subject: [dpdk-dev] [RFC 0/6] replace telemetry with process_info X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Bruce Richardson This patchset proposes a new library, called "process-info" for now, to replace the existing telemetry library in DPDK. (Name subject to change if someone can propose a better one). The existing telemetry library provides useful capabilities if used: - Creates a unix socket on the system to allow external programs connect and gather stats about the DPDK process. - Supports outputting the xstats for various network cards on the system. - Can be used to output any other information exported to the metrics library, e.g. by applications. - Uses JSON message format, which is widely supported by other languages and systems. - Is supported by a plugin to collectd. However, the library also has some issues and limitations that could be improved upon: - Has a dependency on libjansson for JSON processing, so is disabled by default. - Tied entirely to the metrics library for statistics. - No support for sending non-stats data, e.g. something as simple as DPDK version string. - All data gathering functions are in the library itself, which also means… - No support for libraries or drivers to present their own information via the library. We therefore propose to keep the good points of the existing library, but change the way things work to rectify the downsides. This leads to the following design choices in the library: - Keep the existing idea of using a unix socket for connection (just simplifying the connection handling). - We would like to use JSON format, where possible, but the jansson library dependency is problematic. However, creating JSON-encoded data is easier than trying to parse JSON in C code, so we can keep the JSON output format for processing by e.g. collectd and other tools, while simplifying the input to be plain text commands: - Commands in this RFC are designed to all start with "/" for consistency - Any parameters to the commands, e.g. the specific port to get stats for, are placed after a comma "," - Have the library only handle socket creation and input handling. All data gathering should be provided by functions external to the library registered by other components, e.g. have ethdev library provide the function to query NIC xstats, etc. - Have the library directly initialized by EAL by default, so that unless an app explicitly does not want the support, monitoring is available on all DPDK apps. The obvious question that remains to be answered here is: "why a new library rather than just fixing the old one?" The answer is that we have conflicts between the final two design choices above, which require that the library be built early in the build as other libraries will call into it to register callbacks, and the desire to keep backward compatibility e.g. for use with collectd plugin, which requires the existing library code be kept around and built - as it is now - at the end of the build process since it calls into other DPDK libraries. We therefore cannot have one library that meets both requirements, hence the replacement which allows us to maintain backward compatibility by just leaving the old lib in place until e.g. 20.11. This is also why the new library is called "process_info", since the name "telemetry" is already taken. Suggestions for a better name welcome. Bruce Richardson (4): process-info: introduce process-info library eal: integrate process-info library usertools: add process-info python script ethdev: add callback support for process-info Ciara Power (2): rawdev: add callback support for process-info examples/l3fwd-power: enable use of process-info config/common_base | 5 + examples/l3fwd-power/main.c | 83 ++----- lib/Makefile | 3 +- lib/librte_eal/common/eal_common_options.c | 75 ++++++ lib/librte_eal/common/eal_internal_cfg.h | 1 + lib/librte_eal/common/eal_options.h | 5 + lib/librte_eal/freebsd/eal/Makefile | 1 + lib/librte_eal/freebsd/eal/eal.c | 14 ++ lib/librte_eal/linux/eal/Makefile | 1 + lib/librte_eal/linux/eal/eal.c | 15 ++ lib/librte_eal/meson.build | 2 +- lib/librte_ethdev/Makefile | 2 +- lib/librte_ethdev/rte_ethdev.c | 73 ++++++ lib/librte_process_info/Makefile | 26 ++ lib/librte_process_info/meson.build | 8 + lib/librte_process_info/process_info.c | 231 ++++++++++++++++++ lib/librte_process_info/rte_process_info.h | 25 ++ .../rte_process_info_version.map | 6 + lib/librte_rawdev/Makefile | 3 +- lib/librte_rawdev/meson.build | 1 + lib/librte_rawdev/rte_rawdev.c | 82 +++++++ lib/meson.build | 1 + mk/rte.app.mk | 1 + usertools/test_new_telemetry.py | 28 +++ 24 files changed, 630 insertions(+), 62 deletions(-) create mode 100644 lib/librte_process_info/Makefile create mode 100644 lib/librte_process_info/meson.build create mode 100644 lib/librte_process_info/process_info.c create mode 100644 lib/librte_process_info/rte_process_info.h create mode 100644 lib/librte_process_info/rte_process_info_version.map create mode 100755 usertools/test_new_telemetry.py