Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/133494/?format=api
http://patches.dpdk.org/api/patches/133494/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/20231027110117.70995-3-bruce.richardson@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20231027110117.70995-3-bruce.richardson@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20231027110117.70995-3-bruce.richardson@intel.com", "date": "2023-10-27T11:01:10", "name": "[v7,2/9] buildtools: script to generate cmdline boilerplate", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": true, "hash": "2c89fdf8bfc5cb85aac1fed67737a1db386f81cb", "submitter": { "id": 20, "url": "http://patches.dpdk.org/api/people/20/?format=api", "name": "Bruce Richardson", "email": "bruce.richardson@intel.com" }, "delegate": { "id": 24651, "url": "http://patches.dpdk.org/api/users/24651/?format=api", "username": "dmarchand", "first_name": "David", "last_name": "Marchand", "email": "david.marchand@redhat.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/20231027110117.70995-3-bruce.richardson@intel.com/mbox/", "series": [ { "id": 30025, "url": "http://patches.dpdk.org/api/series/30025/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=30025", "date": "2023-10-27T11:01:08", "name": "document and simplify use of cmdline", "version": 7, "mbox": "http://patches.dpdk.org/series/30025/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/133494/comments/", "check": "success", "checks": "http://patches.dpdk.org/api/patches/133494/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 72E7043215;\n\tFri, 27 Oct 2023 13:01:44 +0200 (CEST)", "from mails.dpdk.org (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 2D42D42831;\n\tFri, 27 Oct 2023 13:01:35 +0200 (CEST)", "from mgamail.intel.com (mgamail.intel.com [192.198.163.8])\n by mails.dpdk.org (Postfix) with ESMTP id 2194240F16\n for <dev@dpdk.org>; Fri, 27 Oct 2023 13:01:32 +0200 (CEST)", "from orsmga004.jf.intel.com ([10.7.209.38])\n by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 27 Oct 2023 04:01:32 -0700", "from silpixa00401385.ir.intel.com ([10.237.214.154])\n by orsmga004.jf.intel.com with ESMTP; 27 Oct 2023 04:01:30 -0700" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple;\n d=intel.com; i=@intel.com; q=dns/txt; s=Intel;\n t=1698404494; x=1729940494;\n h=from:to:cc:subject:date:message-id:in-reply-to:\n references:mime-version:content-transfer-encoding;\n bh=cKoyzD6rwHZe2+y4k2RcrY4M6hWlBHfTuV4CDrL3/i0=;\n b=aB3dH9cqvayT3D/K6O03pzmHXgCnucA9S19VeZ5uDXKquwAxjpnvEVPi\n cSZlL1b+Dbk5xmrSJ62pzwaQBV8FGz/fLs71Ibhp4BX2pRC8y5oMm8Nlx\n LqCyIXE1OMCXYAko5tcw9AOV0RLdMnP5YYtA61pTYG9gG3vhpudoIBXEg\n reHD/UH0dtGuFdQ3sMkjBJ4FySJnyq27eQys7oSHJ63Xl1KxkPy6banYJ\n Ocyfqf3klj+p9ML/BOt04V1CMoPDlinYj1NZ/UhQf+TjhJe6k812ZNMWk\n qB+DxKxtIZMA90CmRVkcGaK6WIu8fH1eYYY5pyRIFAqFEsOCk+cx42zMf A==;", "X-IronPort-AV": [ "E=McAfee;i=\"6600,9927,10875\"; a=\"546680\"", "E=Sophos;i=\"6.03,256,1694761200\";\n d=\"scan'208\";a=\"546680\"", "E=McAfee;i=\"6600,9927,10875\"; a=\"883157839\"", "E=Sophos;i=\"6.03,256,1694761200\"; d=\"scan'208\";a=\"883157839\"" ], "X-ExtLoop1": "1", "From": "Bruce Richardson <bruce.richardson@intel.com>", "To": "dev@dpdk.org", "Cc": "david.marchand@redhat.com, rjarry@redhat.com,\n Bruce Richardson <bruce.richardson@intel.com>", "Subject": "[PATCH v7 2/9] buildtools: script to generate cmdline boilerplate", "Date": "Fri, 27 Oct 2023 12:01:10 +0100", "Message-Id": "<20231027110117.70995-3-bruce.richardson@intel.com>", "X-Mailer": "git-send-email 2.39.2", "In-Reply-To": "<20231027110117.70995-1-bruce.richardson@intel.com>", "References": "<20230802170052.955323-1-bruce.richardson@intel.com>\n <20231027110117.70995-1-bruce.richardson@intel.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "Provide a \"dpdk-cmdline-gen.py\" script for application developers to\nquickly generate the boilerplate code necessary for using the cmdline\nlibrary.\n\nExample of use:\nThe script takes an input file with a list of commands the user wants in\nthe app, where the parameter variables are tagged with the type.\nFor example:\n\n\t$ cat commands.list\n\tlist\n\tadd <UINT16>x <UINT16>y\n\techo <STRING>message\n\tadd socket <STRING>path\n\tquit\n\nWhen run through the script as \"./dpdk-cmdline-gen.py commands.list\",\nthe output will be the contents of a header file with all the\nboilerplate necessary for a commandline instance with those commands.\n\nIf the flag --stubs is passed, an output header filename must also be\npassed, in which case both a header file with the definitions and a C\nfile with function stubs in it is written to disk. The separation is so\nthat the header file can be rewritten at any future point to add more\ncommands, while the C file can be kept as-is and extended by the user\nwith any additional functions needed.\n\nSigned-off-by: Bruce Richardson <bruce.richardson@intel.com>\n---\n buildtools/dpdk-cmdline-gen.py | 195 ++++++++++++++++++++++++++++++\n buildtools/meson.build | 7 ++\n doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-\n 3 files changed, 332 insertions(+), 1 deletion(-)\n create mode 100755 buildtools/dpdk-cmdline-gen.py", "diff": "diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py\nnew file mode 100755\nindex 0000000000..c208121363\n--- /dev/null\n+++ b/buildtools/dpdk-cmdline-gen.py\n@@ -0,0 +1,195 @@\n+#!/usr/bin/env python3\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright(c) 2023 Intel Corporation\n+#\n+\"\"\"\n+Script to automatically generate boilerplate for using DPDK cmdline library.\n+\"\"\"\n+\n+import argparse\n+import sys\n+\n+PARSE_FN_PARAMS = \"void *parsed_result, struct cmdline *cl, void *data\"\n+PARSE_FN_BODY = \"\"\"\n+ /* TODO: command action */\n+ RTE_SET_USED(parsed_result);\n+ RTE_SET_USED(cl);\n+ RTE_SET_USED(data);\n+\"\"\"\n+NUMERIC_TYPES = [\n+ \"UINT8\",\n+ \"UINT16\",\n+ \"UINT32\",\n+ \"UINT64\",\n+ \"INT8\",\n+ \"INT16\",\n+ \"INT32\",\n+ \"INT64\",\n+]\n+\n+\n+def process_command(lineno, tokens, comment):\n+ \"\"\"Generate the structures and definitions for a single command.\"\"\"\n+ out = []\n+ cfile_out = []\n+\n+ if tokens[0].startswith(\"<\"):\n+ raise ValueError(f\"Error line {lineno + 1}: command must start with a literal string\")\n+\n+ name_tokens = []\n+ for t in tokens:\n+ if t.startswith(\"<\"):\n+ break\n+ name_tokens.append(t)\n+ name = \"_\".join(name_tokens)\n+\n+ result_struct = []\n+ initializers = []\n+ token_list = []\n+ for t in tokens:\n+ if t.startswith(\"<\"):\n+ t_type, t_name = t[1:].split(\">\")\n+ t_val = \"NULL\"\n+ else:\n+ t_type = \"STRING\"\n+ t_name = t\n+ t_val = f'\"{t}\"'\n+\n+ if t_type == \"STRING\":\n+ result_struct.append(f\"\\tcmdline_fixed_string_t {t_name};\")\n+ initializers.append(\n+ f\"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\\n\"\n+ f\"\\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});\"\n+ )\n+ elif t_type in NUMERIC_TYPES:\n+ result_struct.append(f\"\\t{t_type.lower()}_t {t_name};\")\n+ initializers.append(\n+ f\"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\\n\"\n+ f\"\\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});\"\n+ )\n+ elif t_type in [\"IP\", \"IP_ADDR\", \"IPADDR\"]:\n+ result_struct.append(f\"\\tcmdline_ipaddr_t {t_name};\")\n+ initializers.append(\n+ f\"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\\n\"\n+ f\"\\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});\"\n+ )\n+ else:\n+ raise TypeError(f\"Error line {lineno + 1}: unknown token type '{t_type}'\")\n+ token_list.append(f\"cmd_{name}_{t_name}_tok\")\n+\n+ out.append(f'/* Auto-generated handling for command \"{\" \".join(tokens)}\" */')\n+ # output function prototype\n+ func_sig = f\"void\\ncmd_{name}_parsed({PARSE_FN_PARAMS})\"\n+ out.append(f\"extern {func_sig};\\n\")\n+ # output result data structure\n+ out.append(f\"struct cmd_{name}_result {{\\n\" + \"\\n\".join(result_struct) + \"\\n};\\n\")\n+ # output the initializer tokens\n+ out.append(\"\\n\".join(initializers) + \"\\n\")\n+ # output the instance structure\n+ inst_elems = \"\\n\".join([f\"\\t\\t(void *)&{t},\" for t in token_list])\n+ out.append(\n+ f\"\"\"\\\n+static cmdline_parse_inst_t cmd_{name} = {{\n+\\t.f = cmd_{name}_parsed,\n+\\t.data = NULL,\n+\\t.help_str = \"{comment}\",\n+\\t.tokens = {{\n+{inst_elems}\n+\\t\\tNULL,\n+\\t}}\n+}};\n+\"\"\"\n+ )\n+ # output function template if C file being written\n+ cfile_out.append(f\"{func_sig}\\n{{{PARSE_FN_BODY}}}\\n\")\n+\n+ # return the instance structure name\n+ return (f\"cmd_{name}\", out, cfile_out)\n+\n+\n+def process_commands(infile, hfile, cfile, ctxname):\n+ \"\"\"Generate boilerplate output for a list of commands from infile.\"\"\"\n+ instances = []\n+\n+ hfile.write(\n+ f\"\"\"\\\n+/* File autogenerated by {sys.argv[0]} */\n+#ifndef GENERATED_COMMANDS_H\n+#define GENERATED_COMMANDS_H\n+#include <rte_common.h>\n+#include <cmdline.h>\n+#include <cmdline_parse_string.h>\n+#include <cmdline_parse_num.h>\n+#include <cmdline_parse_ipaddr.h>\n+\n+\"\"\"\n+ )\n+\n+ for lineno, line in enumerate(infile.readlines()):\n+ if line.lstrip().startswith(\"#\"):\n+ continue\n+ if \"#\" not in line:\n+ line = line + \"#\" # ensure split always works, even if no help text\n+ tokens, comment = line.split(\"#\", 1)\n+ cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())\n+ hfile.write(\"\\n\".join(h_out))\n+ if cfile:\n+ cfile.write(\"\\n\".join(c_out))\n+ instances.append(cmd_inst)\n+\n+ inst_join_str = \",\\n\\t&\"\n+ hfile.write(\n+ f\"\"\"\n+static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{\n+\\t&{inst_join_str.join(instances)},\n+\\tNULL\n+}};\n+\n+#endif /* GENERATED_COMMANDS_H */\n+\"\"\"\n+ )\n+\n+\n+def main():\n+ \"\"\"Application main entry point.\"\"\"\n+ ap = argparse.ArgumentParser(description=__doc__)\n+ ap.add_argument(\n+ \"--stubs\",\n+ action=\"store_true\",\n+ help=\"Produce C file with empty function stubs for each command\",\n+ )\n+ ap.add_argument(\n+ \"--output-file\",\n+ \"-o\",\n+ default=\"-\",\n+ help=\"Output header filename [default to stdout]\",\n+ )\n+ ap.add_argument(\n+ \"--context-name\",\n+ default=\"ctx\",\n+ help=\"Name given to the cmdline context variable in the output header [default=ctx]\",\n+ )\n+ ap.add_argument(\"infile\", type=argparse.FileType(\"r\"), help=\"File with list of commands\")\n+ args = ap.parse_args()\n+\n+ if not args.stubs:\n+ if args.output_file == \"-\":\n+ process_commands(args.infile, sys.stdout, None, args.context_name)\n+ else:\n+ with open(args.output_file, \"w\") as hfile:\n+ process_commands(args.infile, hfile, None, args.context_name)\n+ else:\n+ if not args.output_file.endswith(\".h\"):\n+ ap.error(\n+ \"-o/--output-file: specify an output filename ending with .h when creating stubs\"\n+ )\n+\n+ cfilename = args.output_file[:-2] + \".c\"\n+ with open(args.output_file, \"w\") as hfile:\n+ with open(cfilename, \"w\") as cfile:\n+ cfile.write(f'#include \"{args.output_file}\"\\n\\n')\n+ process_commands(args.infile, hfile, cfile, args.context_name)\n+\n+\n+if __name__ == \"__main__\":\n+ main()\ndiff --git a/buildtools/meson.build b/buildtools/meson.build\nindex 948ac17dd2..72447b60a0 100644\n--- a/buildtools/meson.build\n+++ b/buildtools/meson.build\n@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')\n get_numa_count_cmd = py3 + files('get-numa-count.py')\n get_test_suites_cmd = py3 + files('get-test-suites.py')\n has_hugepages_cmd = py3 + files('has-hugepages.py')\n+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')\n+\n+# install any build tools that end-users might want also\n+install_data([\n+ 'dpdk-cmdline-gen.py',\n+ ],\n+ install_dir: 'bin')\n \n # select library and object file format\n pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]\ndiff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst\nindex 40f49a30cc..0b96b770e2 100644\n--- a/doc/guides/prog_guide/cmdline.rst\n+++ b/doc/guides/prog_guide/cmdline.rst\n@@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste\n \n 6. Within your main application code, create a new command-line instance passing in the context.\n \n-The next few subsections will cover each of these steps in more detail,\n+Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK,\n+and found in the ``buildtools`` folder in the source tree.\n+This section covers adding a command-line using this script to generate the boiler plate,\n+while the following section,\n+`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually.\n+\n+Creating a Command List File\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application.\n+While these can be piped to it via standard input, using a list file is probably best.\n+\n+The format of the list file must be:\n+\n+* Comment lines start with '#' as first non-whitespace character\n+\n+* One command per line\n+\n+* Variable fields are prefixed by the type-name in angle-brackets, for example:\n+\n+ * ``<STRING>message``\n+\n+ * ``<UINT16>port_id``\n+\n+ * ``<IP>src_ip``\n+\n+* The help text for a command is given in the form of a comment on the same line as the command\n+\n+An example list file, with a variety of (unrelated) commands, is shown below::\n+\n+ # example list file\n+ list # show all entries\n+ add <UINT16>x <UINT16>y # add x and y\n+ echo <STRING>message # print message to screen\n+ add socket <STRING>path # add unix socket with the given path\n+ quit # close the application\n+\n+Running the Generator Script\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter.\n+The script will output the generated C code to standard output,\n+the contents of which are in the form of a C header file.\n+Optionally, an output filename may be specified via the ``-o/--output-file`` argument.\n+\n+The generated content includes:\n+\n+* The result structure definitions for each command\n+\n+* The token initializers for each structure field\n+\n+* An \"extern\" function prototype for the callback for each command\n+\n+* A parse context for each command, including the per-command comments as help string\n+\n+* A command-line context array definition, suitable for passing to ``cmdline_new``\n+\n+If so desired, the script can also output function stubs for the callback functions for each command.\n+This behaviour is triggered by passing the ``--stubs`` flag to the script.\n+In this case, an output file must be provided with a filename ending in \".h\",\n+and the callback stubs will be written to an equivalent \".c\" file.\n+\n+.. note::\n+\n+ The stubs are written to a separate file,\n+ to allow continuous use of the script to regenerate the command-line header,\n+ without overwriting any code the user has added to the callback functions.\n+ This makes it easy to incrementally add new commands to an existing application.\n+\n+Providing the Function Callbacks\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+As discussed above, the script output is a header file, containing structure definitions,\n+but the callback functions themselves obviously have to be provided by the user.\n+These callback functions must be provided as non-static functions in a C file,\n+and named ``cmd_<cmdname>_parsed``.\n+The function prototypes can be seen in the generated output header.\n+\n+The \"cmdname\" part of the function name is built up by combining the non-variable initial tokens in the command.\n+So, given the commands in our worked example below: ``quit`` and ``show port stats <n>``,\n+the callback functions would be:\n+\n+.. code:: c\n+\n+ void\n+ cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)\n+ {\n+ ...\n+ }\n+\n+ void\n+ cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)\n+ {\n+ ...\n+ }\n+\n+These functions must be provided by the developer, but, as stated above,\n+stub functions may be generated by the script automatically using the ``--stubs`` parameter.\n+\n+The same \"cmdname\" stem is used in the naming of the generated structures too.\n+To get at the results structure for each command above,\n+the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result``\n+or ``struct cmd_show_port_stats_result`` respectively.\n+\n+Integrating with the Application\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+To integrate the script output with the application,\n+we must ``#include`` the generated header into our applications C file,\n+and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``.\n+The first parameter to the function call should be the context array in the generated header file,\n+``ctx`` by default. (Modifiable via script parameter).\n+\n+The callback functions may be in this same file, or in a separate one -\n+they just need to be available to the linker at build-time.\n+\n+Limitations of the Script Approach\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The script approach works for most commands that a user may wish to add to an application.\n+However, it does not support the full range of functions possible with the DPDK command-line library.\n+For example,\n+it is not possible using the script to multiplex multiple commands into a single callback function.\n+To use this functionality, the user should follow the instructions in the next section\n+`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance.\n+\n+Worked Example of Adding Command-line to an Application\n+-------------------------------------------------------\n+\n+The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail,\n working through an example to add two commands to a command-line instance.\n Those two commands will be:\n \n", "prefixes": [ "v7", "2/9" ] }{ "id": 133494, "url": "