Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/139419/?format=api
{ "id": 139419, "url": "http://patches.dpdk.org/api/patches/139419/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/20240416134620.64277-3-rjarry@redhat.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20240416134620.64277-3-rjarry@redhat.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20240416134620.64277-3-rjarry@redhat.com", "date": "2024-04-16T13:46:22", "name": "[v2] usertools: add telemetry exporter", "commit_ref": null, "pull_url": null, "state": "new", "archived": false, "hash": "69177473c1546a085e99c43a70ca9fd896da0c26", "submitter": { "id": 2850, "url": "http://patches.dpdk.org/api/people/2850/?format=api", "name": "Robin Jarry", "email": "rjarry@redhat.com" }, "delegate": { "id": 1, "url": "http://patches.dpdk.org/api/users/1/?format=api", "username": "tmonjalo", "first_name": "Thomas", "last_name": "Monjalon", "email": "thomas@monjalon.net" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/20240416134620.64277-3-rjarry@redhat.com/mbox/", "series": [ { "id": 31756, "url": "http://patches.dpdk.org/api/series/31756/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=31756", "date": "2024-04-16T13:46:22", "name": "[v2] usertools: add telemetry exporter", "version": 2, "mbox": "http://patches.dpdk.org/series/31756/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/139419/comments/", "check": "warning", "checks": "http://patches.dpdk.org/api/patches/139419/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 6CB3E43E74;\n\tTue, 16 Apr 2024 15:47:00 +0200 (CEST)", "from mails.dpdk.org (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 5676C40268;\n\tTue, 16 Apr 2024 15:47:00 +0200 (CEST)", "from us-smtp-delivery-124.mimecast.com\n (us-smtp-delivery-124.mimecast.com [170.10.129.124])\n by mails.dpdk.org (Postfix) with ESMTP id 5D25240262\n for <dev@dpdk.org>; Tue, 16 Apr 2024 15:46:58 +0200 (CEST)", "from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com\n [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-499-JkaUv5Y-N0eR5wFPNdSBpg-1; Tue, 16 Apr 2024 09:46:56 -0400", "from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com\n [10.11.54.4])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E685C801FAF\n for <dev@dpdk.org>; Tue, 16 Apr 2024 13:46:55 +0000 (UTC)", "from localhost.localdomain (unknown [10.39.208.36])\n by smtp.corp.redhat.com (Postfix) with ESMTP id D0E252026962;\n Tue, 16 Apr 2024 13:46:54 +0000 (UTC)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1713275217;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=lsyor7zNNUuXiqGz5nISeH5xE+o28DgBjeSa7sSs/mg=;\n b=cjZXF8VwhcHLH2ofFhLZq4k1IX9d2ir3V4zH6nEJjNLf5vKgTfOy0paFtuIv4QaBdZBiUe\n xsD6oBN2jFKLrQCE3/n5S1//7BwBKZ0j7gu0COKc1TFrC/nuHABMXw8dL0gCEDvdAfWU03\n UAnuQXdy+rkLMQ680h9eGrvTZNHHgt0=", "X-MC-Unique": "JkaUv5Y-N0eR5wFPNdSBpg-1", "From": "Robin Jarry <rjarry@redhat.com>", "To": "dev@dpdk.org", "Cc": "Anthony Harivel <aharivel@redhat.com>", "Subject": "[PATCH v2] usertools: add telemetry exporter", "Date": "Tue, 16 Apr 2024 15:46:22 +0200", "Message-ID": "<20240416134620.64277-3-rjarry@redhat.com>", "In-Reply-To": "<20230926163442.844006-2-rjarry@redhat.com>", "References": "<20230926163442.844006-2-rjarry@redhat.com>", "MIME-Version": "1.0", "X-Scanned-By": "MIMEDefang 3.4.1 on 10.11.54.4", "X-Mimecast-Spam-Score": "0", "X-Mimecast-Originator": "redhat.com", "Content-Transfer-Encoding": "8bit", "Content-Type": "text/plain; charset=\"US-ASCII\"; x-default=true", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "For now the telemetry socket is local to the machine running a DPDK\napplication. Also, there is no official \"schema\" for the exposed\nmetrics. Add a framework and a script to collect and expose these\nmetrics to telemetry and observability agree gators such as Prometheus,\nCarbon or Influxdb. The exposed data must be done with end-users in\nmind, some DPDK terminology or internals may not make sense to everyone.\n\nThe script only serves as an entry point and does not know anything\nabout any specific metrics nor JSON data structures exposed in the\ntelemetry socket.\n\nIt uses dynamically loaded endpoint exporters which are basic python\nfiles that must implement two functions:\n\n def info() -> dict[MetricName, MetricInfo]:\n Mapping of metric names to their description and type.\n\n def metrics(sock: TelemetrySocket) -> list[MetricValue]:\n Request data from sock and return it as metric values. A metric\n value is a 3-tuple: (name: str, value: any, labels: dict). Each\n name must be present in info().\n\nThe sock argument passed to metrics() has a single method:\n\n def cmd(self, uri: str, arg: any = None) -> dict | list:\n Request JSON data to the telemetry socket and parse it to python\n values.\n\nThe main script invokes endpoints and exports the data into an output\nformat. For now, only two formats are implemented:\n\n* openmetrics/prometheus: text based format exported via a local HTTP\n server.\n* carbon/graphite: binary (python pickle) format exported to a distant\n carbon TCP server.\n\nAs a starting point, 3 built-in endpoints are implemented:\n\n* counters: ethdev hardware counters\n* cpu: lcore usage\n* memory: overall memory usage\n\nThe goal is to keep all built-in endpoints in the DPDK repository so\nthat they can be updated along with the telemetry JSON data structures.\n\nExample output for the openmetrics:// format:\n\n ~# dpdk-telemetry-exporter.py -o openmetrics://:9876 &\n INFO using endpoint: counters (from .../telemetry-endpoints/counters.py)\n INFO using endpoint: cpu (from .../telemetry-endpoints/cpu.py)\n INFO using endpoint: memory (from .../telemetry-endpoints/memory.py)\n INFO listening on port 9876\n [1] 838829\n\n ~$ curl http://127.0.0.1:9876/\n # HELP dpdk_cpu_total_cycles Total number of CPU cycles.\n # TYPE dpdk_cpu_total_cycles counter\n # HELP dpdk_cpu_busy_cycles Number of busy CPU cycles.\n # TYPE dpdk_cpu_busy_cycles counter\n dpdk_cpu_total_cycles{cpu=\"73\", numa=\"0\"} 4353385274702980\n dpdk_cpu_busy_cycles{cpu=\"73\", numa=\"0\"} 6215932860\n dpdk_cpu_total_cycles{cpu=\"9\", numa=\"0\"} 4353385274745740\n dpdk_cpu_busy_cycles{cpu=\"9\", numa=\"0\"} 6215932860\n dpdk_cpu_total_cycles{cpu=\"8\", numa=\"0\"} 4353383451895540\n dpdk_cpu_busy_cycles{cpu=\"8\", numa=\"0\"} 6171923160\n dpdk_cpu_total_cycles{cpu=\"72\", numa=\"0\"} 4353385274817320\n dpdk_cpu_busy_cycles{cpu=\"72\", numa=\"0\"} 6215932860\n # HELP dpdk_memory_total_bytes The total size of reserved memory in bytes.\n # TYPE dpdk_memory_total_bytes gauge\n # HELP dpdk_memory_used_bytes The currently used memory in bytes.\n # TYPE dpdk_memory_used_bytes gauge\n dpdk_memory_total_bytes 1073741824\n dpdk_memory_used_bytes 794197376\n\nLink: https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format\nLink: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#text-format\nLink: https://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol\nLink: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/prometheus\nSigned-off-by: Robin Jarry <rjarry@redhat.com>\n---\n\nNotes:\n v2:\n \n * Refuse to run if no endpoints are enabled.\n * Handle endpoint errors gracefully without failing the whole query.\n\n usertools/dpdk-telemetry-exporter.py | 405 ++++++++++++++++++++++\n usertools/meson.build | 6 +\n usertools/telemetry-endpoints/counters.py | 47 +++\n usertools/telemetry-endpoints/cpu.py | 29 ++\n usertools/telemetry-endpoints/memory.py | 37 ++\n 5 files changed, 524 insertions(+)\n create mode 100755 usertools/dpdk-telemetry-exporter.py\n create mode 100644 usertools/telemetry-endpoints/counters.py\n create mode 100644 usertools/telemetry-endpoints/cpu.py\n create mode 100644 usertools/telemetry-endpoints/memory.py", "diff": "diff --git a/usertools/dpdk-telemetry-exporter.py b/usertools/dpdk-telemetry-exporter.py\nnew file mode 100755\nindex 000000000000..f8d873ad856c\n--- /dev/null\n+++ b/usertools/dpdk-telemetry-exporter.py\n@@ -0,0 +1,405 @@\n+#!/usr/bin/env python3\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright (c) 2023 Robin Jarry\n+\n+r'''\n+DPDK telemetry exporter.\n+\n+It uses dynamically loaded endpoint exporters which are basic python files that\n+must implement two functions:\n+\n+ def info() -> dict[MetricName, MetricInfo]:\n+ \"\"\"\n+ Mapping of metric names to their description and type.\n+ \"\"\"\n+\n+ def metrics(sock: TelemetrySocket) -> list[MetricValue]:\n+ \"\"\"\n+ Request data from sock and return it as metric values. A metric value\n+ is a 3-tuple: (name: str, value: any, labels: dict). Each name must be\n+ present in info().\n+ \"\"\"\n+\n+The sock argument passed to metrics() has a single method:\n+\n+ def cmd(self, uri, arg=None) -> dict | list:\n+ \"\"\"\n+ Request JSON data to the telemetry socket and parse it to python\n+ values.\n+ \"\"\"\n+\n+See existing endpoints for examples.\n+\n+The exporter supports multiple output formats:\n+\n+prometheus://ADDRESS:PORT\n+openmetrics://ADDRESS:PORT\n+ Expose the enabled endpoints via a local HTTP server listening on the\n+ specified address and port. GET requests on that server are served with\n+ text/plain responses in the prometheus/openmetrics format.\n+\n+ More details:\n+ https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format\n+\n+carbon://ADDRESS:PORT\n+graphite://ADDRESS:PORT\n+ Export all enabled endpoints to the specified TCP ADDRESS:PORT in the pickle\n+ carbon format.\n+\n+ More details:\n+ https://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol\n+'''\n+\n+import argparse\n+import importlib.util\n+import json\n+import logging\n+import os\n+import pickle\n+import re\n+import socket\n+import struct\n+import sys\n+import time\n+import typing\n+from http import HTTPStatus, server\n+from urllib.parse import urlparse\n+\n+LOG = logging.getLogger(__name__)\n+# Use local endpoints path only when running from source\n+LOCAL = os.path.join(os.path.dirname(__file__), \"telemetry-endpoints\")\n+DEFAULT_LOAD_PATHS = []\n+if os.path.isdir(LOCAL):\n+ DEFAULT_LOAD_PATHS.append(LOCAL)\n+DEFAULT_LOAD_PATHS += [\n+ \"/usr/local/share/dpdk/telemetry-endpoints\",\n+ \"/usr/share/dpdk/telemetry-endpoints\",\n+]\n+DEFAULT_OUTPUT = \"openmetrics://:9876\"\n+\n+\n+def main():\n+ logging.basicConfig(\n+ stream=sys.stdout,\n+ level=logging.INFO,\n+ format=\"%(asctime)s %(levelname)s %(message)s\",\n+ datefmt=\"%Y-%m-%d %H:%M:%S\",\n+ )\n+ parser = argparse.ArgumentParser(\n+ description=__doc__,\n+ formatter_class=argparse.RawDescriptionHelpFormatter,\n+ )\n+ parser.add_argument(\n+ \"-o\",\n+ \"--output\",\n+ metavar=\"FORMAT://PARAMETERS\",\n+ default=urlparse(DEFAULT_OUTPUT),\n+ type=urlparse,\n+ help=f\"\"\"\n+ Output format (default: \"{DEFAULT_OUTPUT}\"). Depending on the format,\n+ URL elements have different meanings. By default, the exporter starts a\n+ local HTTP server on port 9876 that serves requests in the\n+ prometheus/openmetrics plain text format.\n+ \"\"\",\n+ )\n+ parser.add_argument(\n+ \"-p\",\n+ \"--load-path\",\n+ dest=\"load_paths\",\n+ type=lambda v: v.split(os.pathsep),\n+ default=DEFAULT_LOAD_PATHS,\n+ help=f\"\"\"\n+ The list of paths from which to disvover endpoints.\n+ (default: \"{os.pathsep.join(DEFAULT_LOAD_PATHS)}\").\n+ \"\"\",\n+ )\n+ parser.add_argument(\n+ \"-e\",\n+ \"--endpoint\",\n+ dest=\"endpoints\",\n+ metavar=\"ENDPOINT\",\n+ action=\"append\",\n+ help=\"\"\"\n+ Telemetry endpoint to export (by default, all discovered endpoints are\n+ enabled). This option can be specified more than once.\n+ \"\"\",\n+ )\n+ parser.add_argument(\n+ \"-l\",\n+ \"--list\",\n+ action=\"store_true\",\n+ help=\"\"\"\n+ Only list detected endpoints and exit.\n+ \"\"\",\n+ )\n+ parser.add_argument(\n+ \"-s\",\n+ \"--socket-path\",\n+ default=\"/run/dpdk/rte/dpdk_telemetry.v2\",\n+ help=\"\"\"\n+ The DPDK telemetry socket path (default: \"%(default)s\").\n+ \"\"\",\n+ )\n+ args = parser.parse_args()\n+ output = OUTPUT_FORMATS.get(args.output.scheme)\n+ if output is None:\n+ parser.error(f\"unsupported output format: {args.output.scheme}://\")\n+\n+ try:\n+ endpoints = load_endpoints(args.load_paths, args.endpoints)\n+ if args.list:\n+ return\n+ except Exception as e:\n+ parser.error(str(e))\n+\n+ output(args, endpoints)\n+\n+\n+class TelemetrySocket:\n+ \"\"\"\n+ Abstraction of the DPDK telemetry socket.\n+ \"\"\"\n+\n+ def __init__(self, path: str):\n+ self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET)\n+ self.sock.connect(path)\n+ data = json.loads(self.sock.recv(1024).decode())\n+ self.max_output_len = data[\"max_output_len\"]\n+\n+ def cmd(\n+ self, uri: str, arg: typing.Any = None\n+ ) -> typing.Optional[typing.Union[dict, list]]:\n+ \"\"\"\n+ Request JSON data to the telemetry socket and parse it to python\n+ values.\n+ \"\"\"\n+ if arg is not None:\n+ u = f\"{uri},{arg}\"\n+ else:\n+ u = uri\n+ self.sock.send(u.encode(\"utf-8\"))\n+ data = self.sock.recv(self.max_output_len)\n+ return json.loads(data.decode(\"utf-8\"))[uri]\n+\n+ def __enter__(self):\n+ return self\n+\n+ def __exit__(self, *args, **kwargs):\n+ self.sock.close()\n+\n+\n+MetricDescription = str\n+MetricType = str\n+MetricName = str\n+MetricLabels = typing.Dict[str, typing.Any]\n+MetricInfo = typing.Tuple[MetricDescription, MetricType]\n+MetricValue = typing.Tuple[MetricName, typing.Any, MetricLabels]\n+\n+\n+class TelemetryEndpoint:\n+ \"\"\"\n+ Placeholder class only used for typing annotations.\n+ \"\"\"\n+\n+ @staticmethod\n+ def info() -> typing.Dict[MetricName, MetricInfo]:\n+ \"\"\"\n+ Mapping of metric names to their description and type.\n+ \"\"\"\n+ raise NotImplementedError()\n+\n+ @staticmethod\n+ def metrics(sock: TelemetrySocket) -> typing.List[MetricValue]:\n+ \"\"\"\n+ Request data from sock and return it as metric values. Each metric\n+ name must be present in info().\n+ \"\"\"\n+ raise NotImplementedError()\n+\n+\n+def load_endpoints(\n+ paths: typing.List[str], names: typing.List[str]\n+) -> typing.List[TelemetryEndpoint]:\n+ \"\"\"\n+ Load selected telemetry endpoints from the specified paths.\n+ \"\"\"\n+\n+ endpoints = {}\n+ dwb = sys.dont_write_bytecode\n+ sys.dont_write_bytecode = True # never generate .pyc files for endpoints\n+\n+ for p in paths:\n+ if not os.path.isdir(p):\n+ continue\n+ for fname in os.listdir(p):\n+ f = os.path.join(p, fname)\n+ if os.path.isdir(f):\n+ continue\n+ try:\n+ name, _ = os.path.splitext(fname)\n+ if names is not None and name not in names:\n+ # not selected by user\n+ continue\n+ if name in endpoints:\n+ # endpoint with same name already loaded\n+ continue\n+ spec = importlib.util.spec_from_file_location(name, f)\n+ module = importlib.util.module_from_spec(spec)\n+ spec.loader.exec_module(module)\n+ endpoints[name] = module\n+ except Exception:\n+ LOG.exception(\"parsing endpoint: %s\", f)\n+\n+ if not endpoints:\n+ raise Exception(\"no telemetry endpoints detected/selected\")\n+\n+ sys.dont_write_bytecode = dwb\n+\n+ modules = []\n+ info = {}\n+ for name, module in sorted(endpoints.items()):\n+ LOG.info(\"using endpoint: %s (from %s)\", name, module.__file__)\n+ try:\n+ for metric, (description, type_) in module.info().items():\n+ info[(name, metric)] = (description, type_)\n+ modules.append(module)\n+ except Exception:\n+ LOG.exception(\"getting endpoint info: %s\", name)\n+ return modules\n+\n+\n+def serve_openmetrics(\n+ args: argparse.Namespace, endpoints: typing.List[TelemetryEndpoint]\n+):\n+ \"\"\"\n+ Start an HTTP server and serve requests in the openmetrics/prometheus\n+ format.\n+ \"\"\"\n+ listen = (args.output.hostname or \"\", int(args.output.port or 80))\n+ with server.HTTPServer(listen, OpenmetricsHandler) as httpd:\n+ httpd.dpdk_socket_path = args.socket_path\n+ httpd.telemetry_endpoints = endpoints\n+ LOG.info(\"listening on port %s\", httpd.server_port)\n+ try:\n+ httpd.serve_forever()\n+ except KeyboardInterrupt:\n+ LOG.info(\"shutting down\")\n+\n+\n+class OpenmetricsHandler(server.BaseHTTPRequestHandler):\n+ \"\"\"\n+ Basic HTTP handler that returns prometheus/openmetrics formatted responses.\n+ \"\"\"\n+\n+ CONTENT_TYPE = \"text/plain; version=0.0.4; charset=utf-8\"\n+\n+ def escape(self, value: typing.Any) -> str:\n+ \"\"\"\n+ Escape a metric label value.\n+ \"\"\"\n+ value = str(value)\n+ value = value.replace('\"', '\\\\\"')\n+ value = value.replace(\"\\\\\", \"\\\\\\\\\")\n+ return value.replace(\"\\n\", \"\\\\n\")\n+\n+ def do_GET(self):\n+ \"\"\"\n+ Called uppon GET requests.\n+ \"\"\"\n+ try:\n+ lines = []\n+ metrics_names = set()\n+ with TelemetrySocket(self.server.dpdk_socket_path) as sock:\n+ for e in self.server.telemetry_endpoints:\n+ info = e.info()\n+ metrics_lines = []\n+ try:\n+ metrics = e.metrics(sock)\n+ except Exception:\n+ LOG.exception(\"%s: metrics collection failed\", e.__name__)\n+ continue\n+ for name, value, labels in metrics:\n+ fullname = re.sub(r\"\\W\", \"_\", f\"dpdk_{e.__name__}_{name}\")\n+ labels = \", \".join(\n+ f'{k}=\"{self.escape(v)}\"' for k, v in labels.items()\n+ )\n+ if labels:\n+ labels = f\"{{{labels}}}\"\n+ metrics_lines.append(f\"{fullname}{labels} {value}\")\n+ if fullname not in metrics_names:\n+ metrics_names.add(fullname)\n+ desc, metric_type = info[name]\n+ lines += [\n+ f\"# HELP {fullname} {desc}\",\n+ f\"# TYPE {fullname} {metric_type}\",\n+ ]\n+ lines += metrics_lines\n+ if not lines:\n+ self.send_error(HTTPStatus.INTERNAL_SERVER_ERROR)\n+ LOG.error(\n+ \"%s %s: no metrics collected\",\n+ self.address_string(),\n+ self.requestline,\n+ )\n+ body = \"\\n\".join(lines).encode(\"utf-8\") + b\"\\n\"\n+ self.send_response(HTTPStatus.OK)\n+ self.send_header(\"Content-Type\", self.CONTENT_TYPE)\n+ self.send_header(\"Content-Length\", str(len(body)))\n+ self.end_headers()\n+ self.wfile.write(body)\n+ LOG.info(\"%s %s\", self.address_string(), self.requestline)\n+\n+ except (FileNotFoundError, ConnectionRefusedError):\n+ self.send_error(HTTPStatus.SERVICE_UNAVAILABLE)\n+ LOG.exception(\n+ \"%s %s: telemetry socket not available\",\n+ self.address_string(),\n+ self.requestline,\n+ )\n+ except Exception:\n+ self.send_error(HTTPStatus.INTERNAL_SERVER_ERROR)\n+ LOG.exception(\"%s %s\", self.address_string(), self.requestline)\n+\n+ def log_message(self, fmt, *args):\n+ pass # disable built-in logger\n+\n+\n+def export_carbon(args: argparse.Namespace, endpoints: typing.List[TelemetryEndpoint]):\n+ \"\"\"\n+ Collect all metrics and export them to a carbon server in the pickle format.\n+ \"\"\"\n+ addr = (args.output.hostname or \"\", int(args.output.port or 80))\n+ with TelemetrySocket(args.socket_path) as dpdk:\n+ with socket.socket() as carbon:\n+ carbon.connect(addr)\n+ all_metrics = []\n+ for e in endpoints:\n+ try:\n+ metrics = e.metrics(dpdk)\n+ except Exception:\n+ LOG.exception(\"%s: metrics collection failed\", e.__name__)\n+ continue\n+ for name, value, labels in metrics:\n+ fullname = re.sub(r\"\\W\", \".\", f\"dpdk.{e.__name__}.{name}\")\n+ for key, val in labels.items():\n+ val = str(val).replace(\";\", \"\")\n+ fullname += f\";{key}={val}\"\n+ all_metrics.append((fullname, (time.time(), value)))\n+ if not all_metrics:\n+ raise Exception(\"no metrics collected\")\n+ payload = pickle.dumps(all_metrics, protocol=2)\n+ header = struct.pack(\"!L\", len(payload))\n+ buf = header + payload\n+ carbon.sendall(buf)\n+\n+\n+OUTPUT_FORMATS = {\n+ \"openmetrics\": serve_openmetrics,\n+ \"prometheus\": serve_openmetrics,\n+ \"carbon\": export_carbon,\n+ \"graphite\": export_carbon,\n+}\n+\n+\n+if __name__ == \"__main__\":\n+ main()\ndiff --git a/usertools/meson.build b/usertools/meson.build\nindex 740b4832f36d..eb48e2f4403f 100644\n--- a/usertools/meson.build\n+++ b/usertools/meson.build\n@@ -11,5 +11,11 @@ install_data([\n 'dpdk-telemetry.py',\n 'dpdk-hugepages.py',\n 'dpdk-rss-flows.py',\n+ 'dpdk-telemetry-exporter.py',\n ],\n install_dir: 'bin')\n+\n+install_subdir(\n+ 'telemetry-endpoints',\n+ install_dir: 'share/dpdk',\n+ strip_directory: false)\ndiff --git a/usertools/telemetry-endpoints/counters.py b/usertools/telemetry-endpoints/counters.py\nnew file mode 100644\nindex 000000000000..e17cffb43b2c\n--- /dev/null\n+++ b/usertools/telemetry-endpoints/counters.py\n@@ -0,0 +1,47 @@\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright (c) 2023 Robin Jarry\n+\n+RX_PACKETS = \"rx_packets\"\n+RX_BYTES = \"rx_bytes\"\n+RX_MISSED = \"rx_missed\"\n+RX_NOMBUF = \"rx_nombuf\"\n+RX_ERRORS = \"rx_errors\"\n+TX_PACKETS = \"tx_packets\"\n+TX_BYTES = \"tx_bytes\"\n+TX_ERRORS = \"tx_errors\"\n+\n+\n+def info() -> \"dict[Name, tuple[Description, Type]]\":\n+ return {\n+ RX_PACKETS: (\"Number of successfully received packets.\", \"counter\"),\n+ RX_BYTES: (\"Number of successfully received bytes.\", \"counter\"),\n+ RX_MISSED: (\n+ \"Number of packets dropped by the HW because Rx queues are full.\",\n+ \"counter\",\n+ ),\n+ RX_NOMBUF: (\"Number of Rx mbuf allocation failures.\", \"counter\"),\n+ RX_ERRORS: (\"Number of erroneous received packets.\", \"counter\"),\n+ TX_PACKETS: (\"Number of successfully transmitted packets.\", \"counter\"),\n+ TX_BYTES: (\"Number of successfully transmitted bytes.\", \"counter\"),\n+ TX_ERRORS: (\"Number of packet transmission failures.\", \"counter\"),\n+ }\n+\n+\n+def metrics(sock: \"TelemetrySocket\") -> \"list[tuple[Name, Value, Labels]]\":\n+ out = []\n+ for port_id in sock.cmd(\"/ethdev/list\"):\n+ port = sock.cmd(\"/ethdev/info\", port_id)\n+ stats = sock.cmd(\"/ethdev/stats\", port_id)\n+ labels = {\"port\": port[\"name\"]}\n+ out += [\n+ (RX_PACKETS, stats[\"ipackets\"], labels),\n+ (RX_PACKETS, stats[\"ipackets\"], labels),\n+ (RX_BYTES, stats[\"ibytes\"], labels),\n+ (RX_MISSED, stats[\"imissed\"], labels),\n+ (RX_NOMBUF, stats[\"rx_nombuf\"], labels),\n+ (RX_ERRORS, stats[\"ierrors\"], labels),\n+ (TX_PACKETS, stats[\"opackets\"], labels),\n+ (TX_BYTES, stats[\"obytes\"], labels),\n+ (TX_ERRORS, stats[\"oerrors\"], labels),\n+ ]\n+ return out\ndiff --git a/usertools/telemetry-endpoints/cpu.py b/usertools/telemetry-endpoints/cpu.py\nnew file mode 100644\nindex 000000000000..d38d8d6e2558\n--- /dev/null\n+++ b/usertools/telemetry-endpoints/cpu.py\n@@ -0,0 +1,29 @@\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright (c) 2023 Robin Jarry\n+\n+CPU_TOTAL = \"total_cycles\"\n+CPU_BUSY = \"busy_cycles\"\n+\n+\n+def info() -> \"dict[Name, tuple[Description, Type]]\":\n+ return {\n+ CPU_TOTAL: (\"Total number of CPU cycles.\", \"counter\"),\n+ CPU_BUSY: (\"Number of busy CPU cycles.\", \"counter\"),\n+ }\n+\n+\n+def metrics(sock: \"TelemetrySocket\") -> \"list[tuple[Name, Value, Labels]]\":\n+ out = []\n+ for lcore_id in sock.cmd(\"/eal/lcore/list\"):\n+ lcore = sock.cmd(\"/eal/lcore/info\", lcore_id)\n+ cpu = \",\".join(str(c) for c in lcore.get(\"cpuset\", []))\n+ total = lcore.get(\"total_cycles\")\n+ busy = lcore.get(\"busy_cycles\", 0)\n+ if not (cpu and total):\n+ continue\n+ labels = {\"cpu\": cpu, \"numa\": lcore.get(\"socket\", 0)}\n+ out += [\n+ (CPU_TOTAL, total, labels),\n+ (CPU_BUSY, busy, labels),\n+ ]\n+ return out\ndiff --git a/usertools/telemetry-endpoints/memory.py b/usertools/telemetry-endpoints/memory.py\nnew file mode 100644\nindex 000000000000..32cce1e59382\n--- /dev/null\n+++ b/usertools/telemetry-endpoints/memory.py\n@@ -0,0 +1,37 @@\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright (c) 2023 Robin Jarry\n+\n+MEM_TOTAL = \"total_bytes\"\n+MEM_USED = \"used_bytes\"\n+\n+\n+def info() -> \"dict[Name, tuple[Description, Type]]\":\n+ return {\n+ MEM_TOTAL: (\"The total size of reserved memory in bytes.\", \"gauge\"),\n+ MEM_USED: (\"The currently used memory in bytes.\", \"gauge\"),\n+ }\n+\n+\n+def metrics(sock: \"TelemetrySocket\") -> \"list[tuple[Name, Value, Labels]]\":\n+ zones = {}\n+ used = 0\n+ for zone in sock.cmd(\"/eal/memzone_list\") or []:\n+ z = sock.cmd(\"/eal/memzone_info\", zone)\n+ start = int(z[\"Hugepage_base\"], 16)\n+ end = start + (z[\"Hugepage_size\"] * z[\"Hugepage_used\"])\n+ used += z[\"Length\"]\n+ for s, e in list(zones.items()):\n+ if s < start < e < end:\n+ zones[s] = end\n+ break\n+ if start < s < end < e:\n+ del zones[s]\n+ zones[start] = e\n+ break\n+ else:\n+ zones[start] = end\n+\n+ return [\n+ (MEM_TOTAL, sum(end - start for (start, end) in zones.items()), {}),\n+ (MEM_USED, max(0, used), {}),\n+ ]\n", "prefixes": [ "v2" ] }