Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/132743/?format=api
{ "id": 132743, "url": "http://patches.dpdk.org/api/patches/132743/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/20231017121318.146007-3-bruce.richardson@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20231017121318.146007-3-bruce.richardson@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20231017121318.146007-3-bruce.richardson@intel.com", "date": "2023-10-17T12:13:11", "name": "[v5,2/9] buildtools: script to generate cmdline boilerplate", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "2fa777849f75d47fc447a40a1f963f0549b8eafa", "submitter": { "id": 20, "url": "http://patches.dpdk.org/api/people/20/?format=api", "name": "Bruce Richardson", "email": "bruce.richardson@intel.com" }, "delegate": { "id": 1, "url": "http://patches.dpdk.org/api/users/1/?format=api", "username": "tmonjalo", "first_name": "Thomas", "last_name": "Monjalon", "email": "thomas@monjalon.net" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/20231017121318.146007-3-bruce.richardson@intel.com/mbox/", "series": [ { "id": 29878, "url": "http://patches.dpdk.org/api/series/29878/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=29878", "date": "2023-10-17T12:13:09", "name": "document and simplify use of cmdline", "version": 5, "mbox": "http://patches.dpdk.org/series/29878/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/132743/comments/", "check": "success", "checks": "http://patches.dpdk.org/api/patches/132743/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 3526D4318A;\n\tTue, 17 Oct 2023 14:13:41 +0200 (CEST)", "from mails.dpdk.org (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 6BB5D41143;\n\tTue, 17 Oct 2023 14:13:31 +0200 (CEST)", "from mgamail.intel.com (mgamail.intel.com [134.134.136.31])\n by mails.dpdk.org (Postfix) with ESMTP id DDC1240EE7\n for <dev@dpdk.org>; Tue, 17 Oct 2023 14:13:28 +0200 (CEST)", "from fmviesa001.fm.intel.com ([10.60.135.141])\n by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 17 Oct 2023 05:13:28 -0700", "from unknown (HELO silpixa00401385.ir.intel.com) ([10.237.214.41])\n by fmviesa001.fm.intel.com with ESMTP; 17 Oct 2023 05:13:32 -0700" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple;\n d=intel.com; i=@intel.com; q=dns/txt; s=Intel;\n t=1697544809; x=1729080809;\n h=from:to:cc:subject:date:message-id:in-reply-to:\n references:mime-version:content-transfer-encoding;\n bh=AxfBmh/UdGXX/fl1cyuNn+kdIKfYdje4ZRsn0Tm3Tgw=;\n b=J+rH/3hYEm88NRP8n0XwbES/p6z1LT7i+XZ5QvYv6ui57dIcHtjHtKGe\n cs0Zou+1CiEi3v4JIlRrdL1cATBJEgLnNVOjwBcgnzZ6yiWaX7DnrEfVw\n utwAtFpyV4odv0smc7Du3NZoTk3W4H+SiGbPwevJSzM+ZhoKKbcxKi0c9\n AXqgdwFwqxlUvOqt4vlbtaeR76P9K5oc4gE+Slw9UUB6PEAxS/cnGscLs\n Ag+N96xkGS9Q1vLb5tDw7Kzs5PnlD8qUO9r5kSLkf3qrnFXCBLyLm5WRG\n 7rGeNfjT/KwLg+M9SAinGD3cfUYBTgsLlMAt9J/1K+oenm7FPn0Beae17 Q==;", "X-IronPort-AV": [ "E=McAfee;i=\"6600,9927,10865\"; a=\"449984769\"", "E=Sophos;i=\"6.03,232,1694761200\"; d=\"scan'208\";a=\"449984769\"", "E=Sophos;i=\"6.03,232,1694761200\";\n d=\"scan'208\";a=\"4068842\"" ], "X-ExtLoop1": "1", "From": "Bruce Richardson <bruce.richardson@intel.com>", "To": "dev@dpdk.org", "Cc": "david.marchand@redhat.com, rjarry@redhat.com,\n Bruce Richardson <bruce.richardson@intel.com>", "Subject": "[PATCH v5 2/9] buildtools: script to generate cmdline boilerplate", "Date": "Tue, 17 Oct 2023 13:13:11 +0100", "Message-Id": "<20231017121318.146007-3-bruce.richardson@intel.com>", "X-Mailer": "git-send-email 2.39.2", "In-Reply-To": "<20231017121318.146007-1-bruce.richardson@intel.com>", "References": "<20230802170052.955323-1-bruce.richardson@intel.com>\n <20231017121318.146007-1-bruce.richardson@intel.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "Provide a \"dpdk-cmdline-gen.py\" script for application developers to\nquickly generate the boilerplate code necessary for using the cmdline\nlibrary.\n\nExample of use:\nThe script takes an input file with a list of commands the user wants in\nthe app, where the parameter variables are tagged with the type.\nFor example:\n\n\t$ cat commands.list\n\tlist\n\tadd <UINT16>x <UINT16>y\n\techo <STRING>message\n\tadd socket <STRING>path\n\tquit\n\nWhen run through the script as \"./dpdk-cmdline-gen.py commands.list\",\nthe output will be the contents of a header file with all the\nboilerplate necessary for a commandline instance with those commands.\n\nIf the flag --stubs is passed, an output header filename must also be\npassed, in which case both a header file with the definitions and a C\nfile with function stubs in it is written to disk. The separation is so\nthat the header file can be rewritten at any future point to add more\ncommands, while the C file can be kept as-is and extended by the user\nwith any additional functions needed.\n\nSigned-off-by: Bruce Richardson <bruce.richardson@intel.com>\n---\n buildtools/dpdk-cmdline-gen.py | 190 ++++++++++++++++++++++++++++++\n buildtools/meson.build | 7 ++\n doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-\n 3 files changed, 327 insertions(+), 1 deletion(-)\n create mode 100755 buildtools/dpdk-cmdline-gen.py", "diff": "diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py\nnew file mode 100755\nindex 0000000000..6cb7610de4\n--- /dev/null\n+++ b/buildtools/dpdk-cmdline-gen.py\n@@ -0,0 +1,190 @@\n+#!/usr/bin/env python3\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright(c) 2023 Intel Corporation\n+#\n+\"\"\"\n+Script to automatically generate boilerplate for using DPDK cmdline library.\n+\"\"\"\n+\n+import argparse\n+import sys\n+\n+PARSE_FN_PARAMS = \"void *parsed_result, struct cmdline *cl, void *data\"\n+PARSE_FN_BODY = \"\"\"\n+ /* TODO: command action */\n+ RTE_SET_USED(parsed_result);\n+ RTE_SET_USED(cl);\n+ RTE_SET_USED(data);\n+\"\"\"\n+NUMERIC_TYPES = [\n+ \"UINT8\",\n+ \"UINT16\",\n+ \"UINT32\",\n+ \"UINT64\",\n+ \"INT8\",\n+ \"INT16\",\n+ \"INT32\",\n+ \"INT64\",\n+]\n+\n+\n+def process_command(lineno, tokens, comment):\n+ \"\"\"Generate the structures and definitions for a single command.\"\"\"\n+ out = []\n+ cfile_out = []\n+\n+ if tokens[0].startswith(\"<\"):\n+ raise ValueError(f\"Error line {lineno + 1}: command must start with a literal string\")\n+\n+ name_tokens = []\n+ for t in tokens:\n+ if t.startswith(\"<\"):\n+ break\n+ name_tokens.append(t)\n+ name = \"_\".join(name_tokens)\n+\n+ result_struct = []\n+ initializers = []\n+ token_list = []\n+ for t in tokens:\n+ if t.startswith(\"<\"):\n+ t_type, t_name = t[1:].split(\">\")\n+ t_val = \"NULL\"\n+ else:\n+ t_type = \"STRING\"\n+ t_name = t\n+ t_val = f'\"{t}\"'\n+\n+ if t_type == \"STRING\":\n+ result_struct.append(f\"\\tcmdline_fixed_string_t {t_name};\")\n+ initializers.append(\n+ f\"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\\n\"\n+ + f\"\\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});\"\n+ )\n+ elif t_type in NUMERIC_TYPES:\n+ result_struct.append(f\"\\t{t_type.lower()}_t {t_name};\")\n+ initializers.append(\n+ f\"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\\n\"\n+ + f\"\\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});\"\n+ )\n+ elif t_type in [\"IP\", \"IP_ADDR\", \"IPADDR\"]:\n+ result_struct.append(f\"\\tcmdline_ipaddr_t {t_name};\")\n+ initializers.append(\n+ f\"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\\n\"\n+ + f\"\\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});\"\n+ )\n+ else:\n+ raise TypeError(f\"Error line {lineno + 1}: unknown token type '{t_type}'\")\n+ token_list.append(f\"cmd_{name}_{t_name}_tok\")\n+\n+ out.append(f'/* Auto-generated handling for command \"{\" \".join(tokens)}\" */')\n+ # output function prototype\n+ func_sig = f\"void\\ncmd_{name}_parsed({PARSE_FN_PARAMS})\"\n+ out.append(f\"extern {func_sig};\\n\")\n+ # output result data structure\n+ out.append(f\"struct cmd_{name}_result {{\\n\" + \"\\n\".join(result_struct) + \"\\n};\\n\")\n+ # output the initializer tokens\n+ out.append(\"\\n\".join(initializers) + \"\\n\")\n+ # output the instance structure\n+ out.append(\n+ f\"static cmdline_parse_inst_t cmd_{name} = {{\\n\"\n+ + f\"\\t.f = cmd_{name}_parsed,\\n\"\n+ + \"\\t.data = NULL,\\n\"\n+ + f'\\t.help_str = \"{comment}\",\\n'\n+ + \"\\t.tokens = {\"\n+ )\n+ for t in token_list:\n+ out.append(f\"\\t\\t(void *)&{t},\")\n+ out.append(\"\\t\\tNULL\\n\" + \"\\t}\\n\" + \"};\\n\")\n+ # output function template if C file being written\n+ cfile_out.append(f\"{func_sig}\\n{{{PARSE_FN_BODY}}}\\n\")\n+\n+ # return the instance structure name\n+ return (f\"cmd_{name}\", out, cfile_out)\n+\n+\n+def process_commands(infile, hfile, cfile, ctxname):\n+ \"\"\"Generate boilerplate output for a list of commands from infile.\"\"\"\n+ instances = []\n+\n+ hfile.write(\n+ f\"\"\"/* File autogenerated by {sys.argv[0]} */\n+#ifndef GENERATED_COMMANDS_H\n+#define GENERATED_COMMANDS_H\n+#include <rte_common.h>\n+#include <cmdline.h>\n+#include <cmdline_parse_string.h>\n+#include <cmdline_parse_num.h>\n+#include <cmdline_parse_ipaddr.h>\n+\n+\"\"\"\n+ )\n+\n+ for lineno, line in enumerate(infile.readlines()):\n+ if line.lstrip().startswith(\"#\"):\n+ continue\n+ if \"#\" not in line:\n+ line = line + \"#\" # ensure split always works, even if no help text\n+ tokens, comment = line.split(\"#\", 1)\n+ cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())\n+ hfile.write(\"\\n\".join(h_out))\n+ if cfile:\n+ cfile.write(\"\\n\".join(c_out))\n+ instances.append(cmd_inst)\n+\n+ inst_join_str = \",\\n\\t&\"\n+ hfile.write(\n+ f\"\"\"\n+static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{\n+\\t&{inst_join_str.join(instances)},\n+\\tNULL\n+}};\n+\n+#endif /* GENERATED_COMMANDS_H */\n+\"\"\"\n+ )\n+\n+\n+def main():\n+ \"\"\"Application main entry point.\"\"\"\n+ ap = argparse.ArgumentParser(description=__doc__)\n+ ap.add_argument(\n+ \"--stubs\",\n+ action=\"store_true\",\n+ help=\"Produce C file with empty function stubs for each command\",\n+ )\n+ ap.add_argument(\n+ \"--output-file\",\n+ \"-o\",\n+ default=\"-\",\n+ help=\"Output header filename [default to stdout]\",\n+ )\n+ ap.add_argument(\n+ \"--context-name\",\n+ default=\"ctx\",\n+ help=\"Name given to the cmdline context variable in the output header [default=ctx]\",\n+ )\n+ ap.add_argument(\"infile\", type=argparse.FileType(\"r\"), help=\"File with list of commands\")\n+ args = ap.parse_args()\n+\n+ if not args.stubs:\n+ if args.output_file == \"-\":\n+ process_commands(args.infile, sys.stdout, None, args.context_name)\n+ else:\n+ with open(args.output_file, \"w\") as hfile:\n+ process_commands(args.infile, hfile, None, args.context_name)\n+ else:\n+ if not args.output_file.endswith(\".h\"):\n+ ap.error(\n+ \"-o/--output-file: specify an output filename ending with .h when creating stubs\"\n+ )\n+\n+ cfilename = args.output_file[:-2] + \".c\"\n+ with open(args.output_file, \"w\") as hfile:\n+ with open(cfilename, \"w\") as cfile:\n+ cfile.write(f'#include \"{args.output_file}\"\\n\\n')\n+ process_commands(args.infile, hfile, cfile, args.context_name)\n+\n+\n+if __name__ == \"__main__\":\n+ main()\ndiff --git a/buildtools/meson.build b/buildtools/meson.build\nindex 948ac17dd2..72447b60a0 100644\n--- a/buildtools/meson.build\n+++ b/buildtools/meson.build\n@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')\n get_numa_count_cmd = py3 + files('get-numa-count.py')\n get_test_suites_cmd = py3 + files('get-test-suites.py')\n has_hugepages_cmd = py3 + files('has-hugepages.py')\n+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')\n+\n+# install any build tools that end-users might want also\n+install_data([\n+ 'dpdk-cmdline-gen.py',\n+ ],\n+ install_dir: 'bin')\n \n # select library and object file format\n pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]\ndiff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst\nindex 40f49a30cc..0b96b770e2 100644\n--- a/doc/guides/prog_guide/cmdline.rst\n+++ b/doc/guides/prog_guide/cmdline.rst\n@@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste\n \n 6. Within your main application code, create a new command-line instance passing in the context.\n \n-The next few subsections will cover each of these steps in more detail,\n+Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK,\n+and found in the ``buildtools`` folder in the source tree.\n+This section covers adding a command-line using this script to generate the boiler plate,\n+while the following section,\n+`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually.\n+\n+Creating a Command List File\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application.\n+While these can be piped to it via standard input, using a list file is probably best.\n+\n+The format of the list file must be:\n+\n+* Comment lines start with '#' as first non-whitespace character\n+\n+* One command per line\n+\n+* Variable fields are prefixed by the type-name in angle-brackets, for example:\n+\n+ * ``<STRING>message``\n+\n+ * ``<UINT16>port_id``\n+\n+ * ``<IP>src_ip``\n+\n+* The help text for a command is given in the form of a comment on the same line as the command\n+\n+An example list file, with a variety of (unrelated) commands, is shown below::\n+\n+ # example list file\n+ list # show all entries\n+ add <UINT16>x <UINT16>y # add x and y\n+ echo <STRING>message # print message to screen\n+ add socket <STRING>path # add unix socket with the given path\n+ quit # close the application\n+\n+Running the Generator Script\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter.\n+The script will output the generated C code to standard output,\n+the contents of which are in the form of a C header file.\n+Optionally, an output filename may be specified via the ``-o/--output-file`` argument.\n+\n+The generated content includes:\n+\n+* The result structure definitions for each command\n+\n+* The token initializers for each structure field\n+\n+* An \"extern\" function prototype for the callback for each command\n+\n+* A parse context for each command, including the per-command comments as help string\n+\n+* A command-line context array definition, suitable for passing to ``cmdline_new``\n+\n+If so desired, the script can also output function stubs for the callback functions for each command.\n+This behaviour is triggered by passing the ``--stubs`` flag to the script.\n+In this case, an output file must be provided with a filename ending in \".h\",\n+and the callback stubs will be written to an equivalent \".c\" file.\n+\n+.. note::\n+\n+ The stubs are written to a separate file,\n+ to allow continuous use of the script to regenerate the command-line header,\n+ without overwriting any code the user has added to the callback functions.\n+ This makes it easy to incrementally add new commands to an existing application.\n+\n+Providing the Function Callbacks\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+As discussed above, the script output is a header file, containing structure definitions,\n+but the callback functions themselves obviously have to be provided by the user.\n+These callback functions must be provided as non-static functions in a C file,\n+and named ``cmd_<cmdname>_parsed``.\n+The function prototypes can be seen in the generated output header.\n+\n+The \"cmdname\" part of the function name is built up by combining the non-variable initial tokens in the command.\n+So, given the commands in our worked example below: ``quit`` and ``show port stats <n>``,\n+the callback functions would be:\n+\n+.. code:: c\n+\n+ void\n+ cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)\n+ {\n+ ...\n+ }\n+\n+ void\n+ cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)\n+ {\n+ ...\n+ }\n+\n+These functions must be provided by the developer, but, as stated above,\n+stub functions may be generated by the script automatically using the ``--stubs`` parameter.\n+\n+The same \"cmdname\" stem is used in the naming of the generated structures too.\n+To get at the results structure for each command above,\n+the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result``\n+or ``struct cmd_show_port_stats_result`` respectively.\n+\n+Integrating with the Application\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+To integrate the script output with the application,\n+we must ``#include`` the generated header into our applications C file,\n+and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``.\n+The first parameter to the function call should be the context array in the generated header file,\n+``ctx`` by default. (Modifiable via script parameter).\n+\n+The callback functions may be in this same file, or in a separate one -\n+they just need to be available to the linker at build-time.\n+\n+Limitations of the Script Approach\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The script approach works for most commands that a user may wish to add to an application.\n+However, it does not support the full range of functions possible with the DPDK command-line library.\n+For example,\n+it is not possible using the script to multiplex multiple commands into a single callback function.\n+To use this functionality, the user should follow the instructions in the next section\n+`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance.\n+\n+Worked Example of Adding Command-line to an Application\n+-------------------------------------------------------\n+\n+The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail,\n working through an example to add two commands to a command-line instance.\n Those two commands will be:\n \n", "prefixes": [ "v5", "2/9" ] }