From patchwork Fri Jul 29 10:55:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114435 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 54AD8A04FD; Fri, 29 Jul 2022 12:56:01 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2C25B42C17; Fri, 29 Jul 2022 12:55:59 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id E0CF442C15 for ; Fri, 29 Jul 2022 12:55:56 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id A6FB91C1C7B; Fri, 29 Jul 2022 12:55:55 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6BUAcCMq4J4n; Fri, 29 Jul 2022 12:55:52 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id D9BF613125D; Fri, 29 Jul 2022 12:55:51 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 1/9] dts: add project tools config Date: Fri, 29 Jul 2022 10:55:42 +0000 Message-Id: <20220729105550.1382664-2-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org .gitignore contains standard Python-related files. Apart from that, add configuration for Python tools used in DTS: Poetry, dependency and package manager Black, formatter Pylama, static analysis Isort, import sorting .editorconfig modifies the line length to 88, which is the default Black uses. It seems to be the best of all worlds. [0] [0] https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/.editorconfig | 7 + dts/.gitignore | 14 ++ dts/README.md | 15 ++ dts/poetry.lock | 474 +++++++++++++++++++++++++++++++++++++++++++++ dts/pylama.ini | 8 + dts/pyproject.toml | 43 ++++ 6 files changed, 561 insertions(+) create mode 100644 dts/.editorconfig create mode 100644 dts/.gitignore create mode 100644 dts/README.md create mode 100644 dts/poetry.lock create mode 100644 dts/pylama.ini create mode 100644 dts/pyproject.toml diff --git a/dts/.editorconfig b/dts/.editorconfig new file mode 100644 index 0000000000..657f959030 --- /dev/null +++ b/dts/.editorconfig @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# See https://editorconfig.org/ for syntax reference. +# + +[*.py] +max_line_length = 88 diff --git a/dts/.gitignore b/dts/.gitignore new file mode 100644 index 0000000000..9c49935b6f --- /dev/null +++ b/dts/.gitignore @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# + +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] +*$py.class + +# IDE files +.idea + +# DTS results +output diff --git a/dts/README.md b/dts/README.md new file mode 100644 index 0000000000..d8f88f97fe --- /dev/null +++ b/dts/README.md @@ -0,0 +1,15 @@ +# Poetry + +The typical style of python dependency management, requirements.txt, has a few +issues. The advantages of Poetry include specifying what python version is required and +forcing you to specify versions, enforced by a lockfile, both of which help prevent +broken dependencies. Another benefit is the use of pyproject.toml, which has become the +standard config file for python projects, improving project organization. + +# Python Version + +The Python Version required by DTS is specified in +[DTS python config file](./pyproject.toml) in the **[tool.poetry.dependencies]** +section. Poetry doesn't install Python, so you may need to satisfy this requirement if +your Python is not up to date. A tool such as [Pyenv](https://github.com/pyenv/pyenv) +is a good way to get Python, though not the only one. diff --git a/dts/poetry.lock b/dts/poetry.lock new file mode 100644 index 0000000000..c76913f141 --- /dev/null +++ b/dts/poetry.lock @@ -0,0 +1,474 @@ +[[package]] +name = "attrs" +version = "21.4.0" +description = "Classes Without Boilerplate" +category = "main" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" + +[package.extras] +dev = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six", "mypy", "pytest-mypy-plugins", "zope.interface", "furo", "sphinx", "sphinx-notfound-page", "pre-commit", "cloudpickle"] +docs = ["furo", "sphinx", "zope.interface", "sphinx-notfound-page"] +tests = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six", "mypy", "pytest-mypy-plugins", "zope.interface", "cloudpickle"] +tests_no_zope = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "six", "mypy", "pytest-mypy-plugins", "cloudpickle"] + +[[package]] +name = "black" +version = "22.6.0" +description = "The uncompromising code formatter." +category = "dev" +optional = false +python-versions = ">=3.6.2" + +[package.dependencies] +click = ">=8.0.0" +mypy-extensions = ">=0.4.3" +pathspec = ">=0.9.0" +platformdirs = ">=2" +tomli = {version = ">=1.1.0", markers = "python_full_version < \"3.11.0a7\""} + +[package.extras] +colorama = ["colorama (>=0.4.3)"] +d = ["aiohttp (>=3.7.4)"] +jupyter = ["ipython (>=7.8.0)", "tokenize-rt (>=3.2.0)"] +uvloop = ["uvloop (>=0.15.2)"] + +[[package]] +name = "click" +version = "8.1.3" +description = "Composable command line interface toolkit" +category = "dev" +optional = false +python-versions = ">=3.7" + +[package.dependencies] +colorama = {version = "*", markers = "platform_system == \"Windows\""} + +[[package]] +name = "colorama" +version = "0.4.5" +description = "Cross-platform colored terminal text." +category = "dev" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" + +[[package]] +name = "isort" +version = "5.10.1" +description = "A Python utility / library to sort Python imports." +category = "dev" +optional = false +python-versions = ">=3.6.1,<4.0" + +[package.extras] +pipfile_deprecated_finder = ["pipreqs", "requirementslib"] +requirements_deprecated_finder = ["pipreqs", "pip-api"] +colors = ["colorama (>=0.4.3,<0.5.0)"] +plugins = ["setuptools"] + +[[package]] +name = "jsonpatch" +version = "1.32" +description = "Apply JSON-Patches (RFC 6902)" +category = "main" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" + +[package.dependencies] +jsonpointer = ">=1.9" + +[[package]] +name = "jsonpointer" +version = "2.3" +description = "Identify specific nodes in a JSON document (RFC 6901)" +category = "main" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" + +[[package]] +name = "jsonschema" +version = "4.7.2" +description = "An implementation of JSON Schema validation for Python" +category = "main" +optional = false +python-versions = ">=3.7" + +[package.dependencies] +attrs = ">=17.4.0" +pyrsistent = ">=0.14.0,<0.17.0 || >0.17.0,<0.17.1 || >0.17.1,<0.17.2 || >0.17.2" + +[package.extras] +format = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3987", "uri-template", "webcolors (>=1.11)"] +format-nongpl = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3986-validator (>0.1.0)", "uri-template", "webcolors (>=1.11)"] + +[[package]] +name = "mccabe" +version = "0.7.0" +description = "McCabe checker, plugin for flake8" +category = "dev" +optional = false +python-versions = ">=3.6" + +[[package]] +name = "mypy" +version = "0.961" +description = "Optional static typing for Python" +category = "dev" +optional = false +python-versions = ">=3.6" + +[package.dependencies] +mypy-extensions = ">=0.4.3" +tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""} +typing-extensions = ">=3.10" + +[package.extras] +dmypy = ["psutil (>=4.0)"] +python2 = ["typed-ast (>=1.4.0,<2)"] +reports = ["lxml"] + +[[package]] +name = "mypy-extensions" +version = "0.4.3" +description = "Experimental type system extensions for programs checked with the mypy typechecker." +category = "dev" +optional = false +python-versions = "*" + +[[package]] +name = "pathspec" +version = "0.9.0" +description = "Utility library for gitignore style pattern matching of file paths." +category = "dev" +optional = false +python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7" + +[[package]] +name = "pexpect" +version = "4.8.0" +description = "Pexpect allows easy control of interactive console applications." +category = "main" +optional = false +python-versions = "*" + +[package.dependencies] +ptyprocess = ">=0.5" + +[[package]] +name = "platformdirs" +version = "2.5.2" +description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"." +category = "dev" +optional = false +python-versions = ">=3.7" + +[package.extras] +docs = ["furo (>=2021.7.5b38)", "proselint (>=0.10.2)", "sphinx-autodoc-typehints (>=1.12)", "sphinx (>=4)"] +test = ["appdirs (==1.4.4)", "pytest-cov (>=2.7)", "pytest-mock (>=3.6)", "pytest (>=6)"] + +[[package]] +name = "ptyprocess" +version = "0.7.0" +description = "Run a subprocess in a pseudo terminal" +category = "main" +optional = false +python-versions = "*" + +[[package]] +name = "pycodestyle" +version = "2.8.0" +description = "Python style guide checker" +category = "dev" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" + +[[package]] +name = "pydocstyle" +version = "6.1.1" +description = "Python docstring style checker" +category = "dev" +optional = false +python-versions = ">=3.6" + +[package.dependencies] +snowballstemmer = "*" + +[package.extras] +toml = ["toml"] + +[[package]] +name = "pyflakes" +version = "2.4.0" +description = "passive checker of Python programs" +category = "dev" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" + +[[package]] +name = "pylama" +version = "8.3.8" +description = "Code audit tool for python" +category = "dev" +optional = false +python-versions = ">=3.7" + +[package.dependencies] +mccabe = ">=0.6.1" +pycodestyle = ">=2.8.0" +pydocstyle = ">=6.1.1" +pyflakes = ">=2.4.0" + +[package.extras] +all = ["pylint", "eradicate", "radon", "mypy", "vulture"] +eradicate = ["eradicate"] +mypy = ["mypy"] +pylint = ["pylint"] +radon = ["radon"] +tests = ["pytest", "pytest-mypy", "eradicate (>=2.0.0)", "radon (>=5.1.0)", "mypy", "pylint (>=2.11.1)", "pylama-quotes", "vulture", "types-setuptools"] +vulture = ["vulture"] + +[[package]] +name = "pyrsistent" +version = "0.18.1" +description = "Persistent/Functional/Immutable data structures" +category = "main" +optional = false +python-versions = ">=3.7" + +[[package]] +name = "pyyaml" +version = "6.0" +description = "YAML parser and emitter for Python" +category = "main" +optional = false +python-versions = ">=3.6" + +[[package]] +name = "scapy" +version = "2.4.5" +description = "Scapy: interactive packet manipulation tool" +category = "main" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4" + +[package.extras] +basic = ["ipython"] +complete = ["ipython", "pyx", "cryptography (>=2.0)", "matplotlib"] +docs = ["sphinx (>=3.0.0)", "sphinx_rtd_theme (>=0.4.3)", "tox (>=3.0.0)"] + +[[package]] +name = "snowballstemmer" +version = "2.2.0" +description = "This package provides 29 stemmers for 28 languages generated from Snowball algorithms." +category = "dev" +optional = false +python-versions = "*" + +[[package]] +name = "tomli" +version = "2.0.1" +description = "A lil' TOML parser" +category = "dev" +optional = false +python-versions = ">=3.7" + +[[package]] +name = "types-pyyaml" +version = "6.0.10" +description = "Typing stubs for PyYAML" +category = "main" +optional = false +python-versions = "*" + +[[package]] +name = "typing-extensions" +version = "4.3.0" +description = "Backported and Experimental Type Hints for Python 3.7+" +category = "dev" +optional = false +python-versions = ">=3.7" + +[[package]] +name = "warlock" +version = "2.0.1" +description = "Python object model built on JSON schema and JSON patch." +category = "main" +optional = false +python-versions = ">=3.7,<4.0" + +[package.dependencies] +jsonpatch = ">=1,<2" +jsonschema = ">=4,<5" + +[metadata] +lock-version = "1.1" +python-versions = "^3.10" +content-hash = "91d2a7dca306da3c93ce79f27c43bebd4b951ece5ef9ca6f256568286ef48f4b" + +[metadata.files] +attrs = [ + {file = "attrs-21.4.0-py2.py3-none-any.whl", hash = "sha256:2d27e3784d7a565d36ab851fe94887c5eccd6a463168875832a1be79c82828b4"}, + {file = "attrs-21.4.0.tar.gz", hash = "sha256:626ba8234211db98e869df76230a137c4c40a12d72445c45d5f5b716f076e2fd"}, +] +black = [] +click = [ + {file = "click-8.1.3-py3-none-any.whl", hash = "sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48"}, + {file = "click-8.1.3.tar.gz", hash = "sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e"}, +] +colorama = [ + {file = "colorama-0.4.5-py2.py3-none-any.whl", hash = "sha256:854bf444933e37f5824ae7bfc1e98d5bce2ebe4160d46b5edf346a89358e99da"}, + {file = "colorama-0.4.5.tar.gz", hash = "sha256:e6c6b4334fc50988a639d9b98aa429a0b57da6e17b9a44f0451f930b6967b7a4"}, +] +isort = [ + {file = "isort-5.10.1-py3-none-any.whl", hash = "sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7"}, + {file = "isort-5.10.1.tar.gz", hash = "sha256:e8443a5e7a020e9d7f97f1d7d9cd17c88bcb3bc7e218bf9cf5095fe550be2951"}, +] +jsonpatch = [ + {file = "jsonpatch-1.32-py2.py3-none-any.whl", hash = "sha256:26ac385719ac9f54df8a2f0827bb8253aa3ea8ab7b3368457bcdb8c14595a397"}, + {file = "jsonpatch-1.32.tar.gz", hash = "sha256:b6ddfe6c3db30d81a96aaeceb6baf916094ffa23d7dd5fa2c13e13f8b6e600c2"}, +] +jsonpointer = [ + {file = "jsonpointer-2.3-py2.py3-none-any.whl", hash = "sha256:51801e558539b4e9cd268638c078c6c5746c9ac96bc38152d443400e4f3793e9"}, + {file = "jsonpointer-2.3.tar.gz", hash = "sha256:97cba51526c829282218feb99dab1b1e6bdf8efd1c43dc9d57be093c0d69c99a"}, +] +jsonschema = [] +mccabe = [ + {file = "mccabe-0.7.0-py2.py3-none-any.whl", hash = "sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e"}, + {file = "mccabe-0.7.0.tar.gz", hash = "sha256:348e0240c33b60bbdf4e523192ef919f28cb2c3d7d5c7794f74009290f236325"}, +] +mypy = [ + {file = "mypy-0.961-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:697540876638ce349b01b6786bc6094ccdaba88af446a9abb967293ce6eaa2b0"}, + {file = "mypy-0.961-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:b117650592e1782819829605a193360a08aa99f1fc23d1d71e1a75a142dc7e15"}, + {file = "mypy-0.961-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:bdd5ca340beffb8c44cb9dc26697628d1b88c6bddf5c2f6eb308c46f269bb6f3"}, + {file = "mypy-0.961-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:3e09f1f983a71d0672bbc97ae33ee3709d10c779beb613febc36805a6e28bb4e"}, + {file = "mypy-0.961-cp310-cp310-win_amd64.whl", hash = "sha256:e999229b9f3198c0c880d5e269f9f8129c8862451ce53a011326cad38b9ccd24"}, + {file = "mypy-0.961-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:b24be97351084b11582fef18d79004b3e4db572219deee0212078f7cf6352723"}, + {file = "mypy-0.961-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f4a21d01fc0ba4e31d82f0fff195682e29f9401a8bdb7173891070eb260aeb3b"}, + {file = "mypy-0.961-cp36-cp36m-win_amd64.whl", hash = "sha256:439c726a3b3da7ca84a0199a8ab444cd8896d95012c4a6c4a0d808e3147abf5d"}, + {file = "mypy-0.961-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:5a0b53747f713f490affdceef835d8f0cb7285187a6a44c33821b6d1f46ed813"}, + {file = "mypy-0.961-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:0e9f70df36405c25cc530a86eeda1e0867863d9471fe76d1273c783df3d35c2e"}, + {file = "mypy-0.961-cp37-cp37m-win_amd64.whl", hash = "sha256:b88f784e9e35dcaa075519096dc947a388319cb86811b6af621e3523980f1c8a"}, + {file = "mypy-0.961-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:d5aaf1edaa7692490f72bdb9fbd941fbf2e201713523bdb3f4038be0af8846c6"}, + {file = "mypy-0.961-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:9f5f5a74085d9a81a1f9c78081d60a0040c3efb3f28e5c9912b900adf59a16e6"}, + {file = "mypy-0.961-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:f4b794db44168a4fc886e3450201365c9526a522c46ba089b55e1f11c163750d"}, + {file = "mypy-0.961-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:64759a273d590040a592e0f4186539858c948302c653c2eac840c7a3cd29e51b"}, + {file = "mypy-0.961-cp38-cp38-win_amd64.whl", hash = "sha256:63e85a03770ebf403291ec50097954cc5caf2a9205c888ce3a61bd3f82e17569"}, + {file = "mypy-0.961-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:5f1332964963d4832a94bebc10f13d3279be3ce8f6c64da563d6ee6e2eeda932"}, + {file = "mypy-0.961-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:006be38474216b833eca29ff6b73e143386f352e10e9c2fbe76aa8549e5554f5"}, + {file = "mypy-0.961-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:9940e6916ed9371809b35b2154baf1f684acba935cd09928952310fbddaba648"}, + {file = "mypy-0.961-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:a5ea0875a049de1b63b972456542f04643daf320d27dc592d7c3d9cd5d9bf950"}, + {file = "mypy-0.961-cp39-cp39-win_amd64.whl", hash = "sha256:1ece702f29270ec6af25db8cf6185c04c02311c6bb21a69f423d40e527b75c56"}, + {file = "mypy-0.961-py3-none-any.whl", hash = "sha256:03c6cc893e7563e7b2949b969e63f02c000b32502a1b4d1314cabe391aa87d66"}, + {file = "mypy-0.961.tar.gz", hash = "sha256:f730d56cb924d371c26b8eaddeea3cc07d78ff51c521c6d04899ac6904b75492"}, +] +mypy-extensions = [ + {file = "mypy_extensions-0.4.3-py2.py3-none-any.whl", hash = "sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d"}, + {file = "mypy_extensions-0.4.3.tar.gz", hash = "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"}, +] +pathspec = [ + {file = "pathspec-0.9.0-py2.py3-none-any.whl", hash = "sha256:7d15c4ddb0b5c802d161efc417ec1a2558ea2653c2e8ad9c19098201dc1c993a"}, + {file = "pathspec-0.9.0.tar.gz", hash = "sha256:e564499435a2673d586f6b2130bb5b95f04a3ba06f81b8f895b651a3c76aabb1"}, +] +pexpect = [ + {file = "pexpect-4.8.0-py2.py3-none-any.whl", hash = "sha256:0b48a55dcb3c05f3329815901ea4fc1537514d6ba867a152b581d69ae3710937"}, + {file = "pexpect-4.8.0.tar.gz", hash = "sha256:fc65a43959d153d0114afe13997d439c22823a27cefceb5ff35c2178c6784c0c"}, +] +platformdirs = [ + {file = "platformdirs-2.5.2-py3-none-any.whl", hash = "sha256:027d8e83a2d7de06bbac4e5ef7e023c02b863d7ea5d079477e722bb41ab25788"}, + {file = "platformdirs-2.5.2.tar.gz", hash = "sha256:58c8abb07dcb441e6ee4b11d8df0ac856038f944ab98b7be6b27b2a3c7feef19"}, +] +ptyprocess = [ + {file = "ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35"}, + {file = "ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220"}, +] +pycodestyle = [ + {file = "pycodestyle-2.8.0-py2.py3-none-any.whl", hash = "sha256:720f8b39dde8b293825e7ff02c475f3077124006db4f440dcbc9a20b76548a20"}, + {file = "pycodestyle-2.8.0.tar.gz", hash = "sha256:eddd5847ef438ea1c7870ca7eb78a9d47ce0cdb4851a5523949f2601d0cbbe7f"}, +] +pydocstyle = [ + {file = "pydocstyle-6.1.1-py3-none-any.whl", hash = "sha256:6987826d6775056839940041beef5c08cc7e3d71d63149b48e36727f70144dc4"}, + {file = "pydocstyle-6.1.1.tar.gz", hash = "sha256:1d41b7c459ba0ee6c345f2eb9ae827cab14a7533a88c5c6f7e94923f72df92dc"}, +] +pyflakes = [ + {file = "pyflakes-2.4.0-py2.py3-none-any.whl", hash = "sha256:3bb3a3f256f4b7968c9c788781e4ff07dce46bdf12339dcda61053375426ee2e"}, + {file = "pyflakes-2.4.0.tar.gz", hash = "sha256:05a85c2872edf37a4ed30b0cce2f6093e1d0581f8c19d7393122da7e25b2b24c"}, +] +pylama = [ + {file = "pylama-8.3.8-py3-none-any.whl", hash = "sha256:aff89423f7de118713f638c7f937fa83a5873e3bdf06d413661d9cb8dc5f3a7b"}, + {file = "pylama-8.3.8.tar.gz", hash = "sha256:2dd852fe9312ea6012466cf17ff179668fc3d2716856fcfaaee8ce7876d83620"}, +] +pyrsistent = [ + {file = "pyrsistent-0.18.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:df46c854f490f81210870e509818b729db4488e1f30f2a1ce1698b2295a878d1"}, + {file = "pyrsistent-0.18.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5d45866ececf4a5fff8742c25722da6d4c9e180daa7b405dc0a2a2790d668c26"}, + {file = "pyrsistent-0.18.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4ed6784ceac462a7d6fcb7e9b663e93b9a6fb373b7f43594f9ff68875788e01e"}, + {file = "pyrsistent-0.18.1-cp310-cp310-win32.whl", hash = "sha256:e4f3149fd5eb9b285d6bfb54d2e5173f6a116fe19172686797c056672689daf6"}, + {file = "pyrsistent-0.18.1-cp310-cp310-win_amd64.whl", hash = "sha256:636ce2dc235046ccd3d8c56a7ad54e99d5c1cd0ef07d9ae847306c91d11b5fec"}, + {file = "pyrsistent-0.18.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:e92a52c166426efbe0d1ec1332ee9119b6d32fc1f0bbfd55d5c1088070e7fc1b"}, + {file = "pyrsistent-0.18.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d7a096646eab884bf8bed965bad63ea327e0d0c38989fc83c5ea7b8a87037bfc"}, + {file = "pyrsistent-0.18.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cdfd2c361b8a8e5d9499b9082b501c452ade8bbf42aef97ea04854f4a3f43b22"}, + {file = "pyrsistent-0.18.1-cp37-cp37m-win32.whl", hash = "sha256:7ec335fc998faa4febe75cc5268a9eac0478b3f681602c1f27befaf2a1abe1d8"}, + {file = "pyrsistent-0.18.1-cp37-cp37m-win_amd64.whl", hash = "sha256:6455fc599df93d1f60e1c5c4fe471499f08d190d57eca040c0ea182301321286"}, + {file = "pyrsistent-0.18.1-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:fd8da6d0124efa2f67d86fa70c851022f87c98e205f0594e1fae044e7119a5a6"}, + {file = "pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7bfe2388663fd18bd8ce7db2c91c7400bf3e1a9e8bd7d63bf7e77d39051b85ec"}, + {file = "pyrsistent-0.18.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0e3e1fcc45199df76053026a51cc59ab2ea3fc7c094c6627e93b7b44cdae2c8c"}, + {file = "pyrsistent-0.18.1-cp38-cp38-win32.whl", hash = "sha256:b568f35ad53a7b07ed9b1b2bae09eb15cdd671a5ba5d2c66caee40dbf91c68ca"}, + {file = "pyrsistent-0.18.1-cp38-cp38-win_amd64.whl", hash = "sha256:d1b96547410f76078eaf66d282ddca2e4baae8964364abb4f4dcdde855cd123a"}, + {file = "pyrsistent-0.18.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:f87cc2863ef33c709e237d4b5f4502a62a00fab450c9e020892e8e2ede5847f5"}, + {file = "pyrsistent-0.18.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6bc66318fb7ee012071b2792024564973ecc80e9522842eb4e17743604b5e045"}, + {file = "pyrsistent-0.18.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:914474c9f1d93080338ace89cb2acee74f4f666fb0424896fcfb8d86058bf17c"}, + {file = "pyrsistent-0.18.1-cp39-cp39-win32.whl", hash = "sha256:1b34eedd6812bf4d33814fca1b66005805d3640ce53140ab8bbb1e2651b0d9bc"}, + {file = "pyrsistent-0.18.1-cp39-cp39-win_amd64.whl", hash = "sha256:e24a828f57e0c337c8d8bb9f6b12f09dfdf0273da25fda9e314f0b684b415a07"}, + {file = "pyrsistent-0.18.1.tar.gz", hash = "sha256:d4d61f8b993a7255ba714df3aca52700f8125289f84f704cf80916517c46eb96"}, +] +pyyaml = [ + {file = "PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53"}, + {file = "PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c"}, + {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc"}, + {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b"}, + {file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"}, + {file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"}, + {file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"}, + {file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"}, + {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f"}, + {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"}, + {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4"}, + {file = "PyYAML-6.0-cp36-cp36m-win32.whl", hash = "sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293"}, + {file = "PyYAML-6.0-cp36-cp36m-win_amd64.whl", hash = "sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57"}, + {file = "PyYAML-6.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c"}, + {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0"}, + {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4"}, + {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9"}, + {file = "PyYAML-6.0-cp37-cp37m-win32.whl", hash = "sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737"}, + {file = "PyYAML-6.0-cp37-cp37m-win_amd64.whl", hash = "sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d"}, + {file = "PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b"}, + {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba"}, + {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34"}, + {file = "PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287"}, + {file = "PyYAML-6.0-cp38-cp38-win32.whl", hash = "sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78"}, + {file = "PyYAML-6.0-cp38-cp38-win_amd64.whl", hash = "sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07"}, + {file = "PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b"}, + {file = "PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174"}, + {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803"}, + {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3"}, + {file = "PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0"}, + {file = "PyYAML-6.0-cp39-cp39-win32.whl", hash = "sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb"}, + {file = "PyYAML-6.0-cp39-cp39-win_amd64.whl", hash = "sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c"}, + {file = "PyYAML-6.0.tar.gz", hash = "sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2"}, +] +scapy = [ + {file = "scapy-2.4.5.tar.gz", hash = "sha256:bc707e3604784496b6665a9e5b2a69c36cc9fb032af4864b29051531b24c8593"}, +] +snowballstemmer = [ + {file = "snowballstemmer-2.2.0-py2.py3-none-any.whl", hash = "sha256:c8e1716e83cc398ae16824e5572ae04e0d9fc2c6b985fb0f900f5f0c96ecba1a"}, + {file = "snowballstemmer-2.2.0.tar.gz", hash = "sha256:09b16deb8547d3412ad7b590689584cd0fe25ec8db3be37788be3810cbf19cb1"}, +] +tomli = [ + {file = "tomli-2.0.1-py3-none-any.whl", hash = "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc"}, + {file = "tomli-2.0.1.tar.gz", hash = "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"}, +] +types-pyyaml = [] +typing-extensions = [] +warlock = [ + {file = "warlock-2.0.1-py3-none-any.whl", hash = "sha256:448df959cec31904f686ac8c6b1dfab80f0cdabce3d303be517dd433eeebf012"}, + {file = "warlock-2.0.1.tar.gz", hash = "sha256:99abbf9525b2a77f2cde896d3a9f18a5b4590db063db65e08207694d2e0137fc"}, +] diff --git a/dts/pylama.ini b/dts/pylama.ini new file mode 100644 index 0000000000..23fc709b5a --- /dev/null +++ b/dts/pylama.ini @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 University of New Hampshire +# + +[pylama] +format = pylint +linters = pep8,pycodestyle,pylint +ignore = F0401,C0111,E731,E266,E501,E203 diff --git a/dts/pyproject.toml b/dts/pyproject.toml new file mode 100644 index 0000000000..710c247b97 --- /dev/null +++ b/dts/pyproject.toml @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 University of New Hampshire +# + +[tool.poetry] +name = "dts" +version = "0.1.0" +description = "" +authors = ["Owen Hilyard ", "dts@dpdk.org"] + +[tool.poetry.dependencies] +python = "^3.10" +pexpect = "^4.8.0" +warlock = "^2.0.1" +PyYAML = "^6.0" +types-PyYAML = "^6.0.8" +scapy = "^2.4.5" + +[tool.poetry.dev-dependencies] +mypy = "^0.961" +black = "^22.6.0" +isort = "^5.10.1" +pylama = "^8.3.8" + +[tool.poetry.scripts] +dts = "main:main" + +[build-system] +requires = ["poetry-core>=1.0.0"] +build-backend = "poetry.core.masonry.api" + +[tool.isort] +profile = "black" + +[tool.mypy] +ignore_missing_imports = true +disallow_untyped_defs = true +disallow_untyped_calls = true +python_version = "3.10" +disallow_any_unimported = true +check_untyped_defs = true +strict_optional = true +strict_equality = true From patchwork Fri Jul 29 10:55:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114436 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 42864A04FD; Fri, 29 Jul 2022 12:56:08 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0BD7542C24; Fri, 29 Jul 2022 12:56:00 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id A6D4842C15 for ; Fri, 29 Jul 2022 12:55:57 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id A1D141C207A; Fri, 29 Jul 2022 12:55:56 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bpgen4rGPg4p; Fri, 29 Jul 2022 12:55:55 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id 5B67C13125E; Fri, 29 Jul 2022 12:55:52 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 2/9] dts: add developer tools Date: Fri, 29 Jul 2022 10:55:43 +0000 Message-Id: <20220729105550.1382664-3-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The Dockerfile contains basic image for CI and developers. There's also an integration of the Dockerfile with Visual Studio. The formatter script uses Black and Isort to format the Python code. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/.devcontainer/devcontainer.json | 30 ++++++++++++ dts/Dockerfile | 38 +++++++++++++++ dts/README.md | 74 ++++++++++++++++++++++++++++- dts/format.sh | 45 ++++++++++++++++++ 4 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 dts/.devcontainer/devcontainer.json create mode 100644 dts/Dockerfile create mode 100755 dts/format.sh diff --git a/dts/.devcontainer/devcontainer.json b/dts/.devcontainer/devcontainer.json new file mode 100644 index 0000000000..41ca28fc17 --- /dev/null +++ b/dts/.devcontainer/devcontainer.json @@ -0,0 +1,30 @@ +// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at: +// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile +{ + "name": "Existing Dockerfile", + + // Sets the run context to one level up instead of the .devcontainer folder. + "context": "..", + + // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename. + "dockerFile": "../Dockerfile", + + // Use 'forwardPorts' to make a list of ports inside the container available locally. + // "forwardPorts": [], + + // Uncomment the next line to run commands after the container is created - for example installing curl. + "postCreateCommand": "poetry install", + + "extensions": [ + "ms-python.vscode-pylance", + ] + + // Uncomment when using a ptrace-based debugger like C++, Go, and Rust + // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ], + + // Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker. + // "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ], + + // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root. + // "remoteUser": "vscode" +} diff --git a/dts/Dockerfile b/dts/Dockerfile new file mode 100644 index 0000000000..6700aa45b8 --- /dev/null +++ b/dts/Dockerfile @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 University of New Hampshire +# + +FROM ubuntu:22.04 AS base + +RUN apt-get -y update && apt-get -y upgrade && \ + apt-get -y install --no-install-recommends \ + python3 \ + python3-pip \ + python3-pexpect \ + python3-poetry \ + python3-cachecontrol \ + openssh-client + + +FROM base AS runner + +# This container is intended to be used as the base for automated systems. +# It bakes DTS into the container during the build. + +RUN mkdir /dts +COPY ./pyproject.toml /dts/pyproject.toml +COPY ./poetry.lock /dts/poetry.lock +WORKDIR /dts +RUN poetry install --no-dev +COPY . /dts + +CMD ["poetry", "run", "python", "main.py"] + +FROM base AS dev + +# This container is intended to be used as a development environment. + +RUN apt-get -y install --no-install-recommends \ + vim emacs git + +WORKDIR /dts diff --git a/dts/README.md b/dts/README.md index d8f88f97fe..55a272d767 100644 --- a/dts/README.md +++ b/dts/README.md @@ -12,4 +12,76 @@ The Python Version required by DTS is specified in [DTS python config file](./pyproject.toml) in the **[tool.poetry.dependencies]** section. Poetry doesn't install Python, so you may need to satisfy this requirement if your Python is not up to date. A tool such as [Pyenv](https://github.com/pyenv/pyenv) -is a good way to get Python, though not the only one. +is a good way to get Python, though not the only one. However, DTS includes a +development environment in the form of a Docker image. + +# Expected Environment + +The expected execution and development environments for DTS are the same, +the container defined by [Dockerfile](./Dockerfile). Using a container for the +development environment helps with a few things. + +1. It helps enforce the boundary between the tester and the traffic + generator/sut, something which has experienced issues in the past. +2. It makes creating containers to run DTS inside automated tooling + much easier, since they can be based off of a known-working environment + that will be updated as DTS is. +3. It abstracts DTS from the server it is running on. This means that the + bare-metal os can be whatever corporate policy or your personal preferences + dictate, and DTS does not have to try to support all 15 distros that + are supported by DPDK CI. +4. It makes automated testing for DTS easier, since new dependencies can be + sent in with the patches. +5. It fixes the issue of undocumented dependencies, where some test suites + require python libraries that are not installed. +6. Allows everyone to use the same python version easily, even if they are + using an LTS distro or Windows. +7. Allows you to run the tester on Windows while developing via Docker for + Windows. + +## Tips for setting up a development environment + +### Getting a docker shell + +These commands will give you a bash shell inside the container with all the python +dependencies installed. This will place you inside a python virtual +environment. DTS is mounted via a volume, which is essentially a symlink +from the host to the container. This enables you to edit and run inside the container +and then delete the container when you are done, keeping your work. + +```shell +docker build --target dev -t dpdk-dts . +docker run -v $(pwd):/dts -it dpdk-dts bash +$ poetry install +$ poetry shell +``` + +### Vim/Emacs + +Any editor in the ubuntu repos should be easy to use. You can add your normal +config files as a volume, enabling you to use your preferred settings. + +```shell +apt install vim +apt install emacs +``` + +### Visual Studio Code + +VSCode has first-class support for developing with containers. You may need to run the +non-docker setup commands in the integrated terminal. DTS contains a .devcontainer +config, so if you open the folder in vscode it should prompt you to use the dev +container assuming you have the plugin installed. Please refer to +[VS Development Containers Docs](https://code.visualstudio.com/docs/remote/containers) +to set it all up. + +### Other + +Searching for '$IDE dev containers' will probably lead you in the right +direction. + +# Python Formatting + +The tools used to format Python code in DTS are Black and Isort. There's a shell +script, function.sh, which runs the formatters. Poetry will install these tools, +so once you have that set up, you should run it before submitting patches. diff --git a/dts/format.sh b/dts/format.sh new file mode 100755 index 0000000000..7d72335470 --- /dev/null +++ b/dts/format.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 University of New Hampshire +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# + +function main() { + # The directory to work on is either passed in as argument 1, + # or is the current working directory + DIRECTORY=${1:-$(pwd)} + LINE_LENGTH=88 + + BLACK_VERSION=$(awk '/\[tool.poetry.dev-dependencies\]/,/$^/' pyproject.toml |\ + grep black | grep -o '[0-9][^"]*') + + PYTHON_VERSION=$(awk '/\[tool.poetry.dependencies\]/,/$^/' pyproject.toml |\ + grep python | grep -o '[0-9][^"]*' | tr -d '.') + + isort \ + --overwrite-in-place \ + --profile black \ + -j "$(nproc)" \ + --line-length $LINE_LENGTH \ + --python-version auto \ + "$DIRECTORY" + + black \ + --line-length $LINE_LENGTH \ + --required-version "${BLACK_VERSION}" \ + --target-version "py${PYTHON_VERSION}" \ + --safe \ + "$DIRECTORY" +} + +function help() { + echo "usage: format.sh " +} + +if [ "$1" == "-h" ] || [ "$1" == "--help" ]; then + help + exit 0 +fi + +main "$1" + From patchwork Fri Jul 29 10:55:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114438 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AC43BA04FD; Fri, 29 Jul 2022 12:56:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 47D6742C32; Fri, 29 Jul 2022 12:56:04 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id D65F342C1F for ; Fri, 29 Jul 2022 12:55:59 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id 96DE713125D; Fri, 29 Jul 2022 12:55:58 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e11Yfoh1oXAC; Fri, 29 Jul 2022 12:55:56 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id CC4C613125F; Fri, 29 Jul 2022 12:55:52 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 3/9] dts: add basic logging facility Date: Fri, 29 Jul 2022 10:55:44 +0000 Message-Id: <20220729105550.1382664-4-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The logging module provides loggers distinguished by two attributes, a custom format and a verbosity switch. The loggers log to both console and more verbosely to files. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/framework/__init__.py | 3 + dts/framework/logger.py | 124 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+) create mode 100644 dts/framework/__init__.py create mode 100644 dts/framework/logger.py diff --git a/dts/framework/__init__.py b/dts/framework/__init__.py new file mode 100644 index 0000000000..3c30bccf43 --- /dev/null +++ b/dts/framework/__init__.py @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# diff --git a/dts/framework/logger.py b/dts/framework/logger.py new file mode 100644 index 0000000000..920ce0fb15 --- /dev/null +++ b/dts/framework/logger.py @@ -0,0 +1,124 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +import logging +import os.path +from typing import TypedDict + +""" +DTS logger module with several log level. DTS framework and TestSuite log +will saved into different log files. +""" +verbose = False +date_fmt = "%d/%m/%Y %H:%M:%S" +stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + + +class LoggerDictType(TypedDict): + logger: "DTSLOG" + name: str + node: str + + +# List for saving all using loggers +global Loggers +Loggers: list[LoggerDictType] = [] + + +def set_verbose() -> None: + global verbose + verbose = True + + +class DTSLOG(logging.LoggerAdapter): + """ + DTS log class for framework and testsuite. + """ + + node: str + logger: logging.Logger + sh: logging.StreamHandler + fh: logging.FileHandler + verbose_handler: logging.FileHandler + + def __init__(self, logger: logging.Logger, node: str = "suite"): + global log_dir + + self.logger = logger + self.logger.setLevel(1) # 1 means log everything + + self.node = node + + # add handler to emit to stdout + sh = logging.StreamHandler() + sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) + + sh.setLevel(logging.DEBUG) # file handler default level + global verbose + if verbose is True: + sh.setLevel(logging.DEBUG) + else: + sh.setLevel(logging.INFO) # console handler defaultlevel + + self.logger.addHandler(sh) + self.sh = sh + + if not os.path.exists("output"): + os.mkdir("output") + + fh = logging.FileHandler(f"output/{node}.log") + fh.setFormatter( + logging.Formatter( + fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s", + datefmt=date_fmt, + ) + ) + + fh.setLevel(1) # We want all the logs we can get in the file + self.logger.addHandler(fh) + self.fh = fh + + # This outputs EVERYTHING, intended for post-mortem debugging + # Also optimized for processing via AWK (awk -F '|' ...) + verbose_handler = logging.FileHandler(f"output/{node}.verbose.log") + verbose_handler.setFormatter( + logging.Formatter( + fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|%(funcName)s|" + "%(process)d|%(thread)d|%(threadName)s|%(message)s", + datefmt=date_fmt, + ) + ) + + verbose_handler.setLevel(1) # We want all the logs we can get in the file + self.logger.addHandler(verbose_handler) + self.verbose_handler = verbose_handler + + super(DTSLOG, self).__init__(self.logger, dict(node=self.node)) + + def logger_exit(self) -> None: + """ + Remove stream handler and logfile handler. + """ + for handler in (self.sh, self.fh, self.verbose_handler): + handler.flush() + self.logger.removeHandler(handler) + + +def getLogger(name: str, node: str = "suite") -> DTSLOG: + """ + Get logger handler and if there's no handler for specified Node will create one. + """ + global Loggers + # return saved logger + logger: LoggerDictType + for logger in Loggers: + if logger["name"] == name and logger["node"] == node: + return logger["logger"] + + # return new logger + dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node) + Loggers.append({"logger": dts_logger, "name": name, "node": node}) + return dts_logger From patchwork Fri Jul 29 10:55:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114437 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2F6E0A04FD; Fri, 29 Jul 2022 12:56:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 42C1442C2E; Fri, 29 Jul 2022 12:56:03 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id A5F4F42C1E for ; Fri, 29 Jul 2022 12:55:59 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id 977B913125E; Fri, 29 Jul 2022 12:55:58 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eLFpnnTmpH_a; Fri, 29 Jul 2022 12:55:57 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id 4A5DD1C124E; Fri, 29 Jul 2022 12:55:53 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 4/9] dts: add ssh pexpect library Date: Fri, 29 Jul 2022 10:55:45 +0000 Message-Id: <20220729105550.1382664-5-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The library uses the pexpect python library and implements connection to a node and two ways to interact with the node: 1. Send a string with specified prompt which will be matched after the string has been sent to the node. 2. Send a command to be executed. No prompt is specified here. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/framework/exception.py | 57 ++++++++++ dts/framework/ssh_pexpect.py | 205 +++++++++++++++++++++++++++++++++++ dts/framework/utils.py | 12 ++ 3 files changed, 274 insertions(+) create mode 100644 dts/framework/exception.py create mode 100644 dts/framework/ssh_pexpect.py create mode 100644 dts/framework/utils.py diff --git a/dts/framework/exception.py b/dts/framework/exception.py new file mode 100644 index 0000000000..35e81a4d99 --- /dev/null +++ b/dts/framework/exception.py @@ -0,0 +1,57 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +""" +User-defined exceptions used across the framework. +""" + + +class TimeoutException(Exception): + """ + Command execution timeout. + """ + + command: str + output: str + + def __init__(self, command: str, output: str): + self.command = command + self.output = output + + def __str__(self) -> str: + return f"TIMEOUT on {self.command}" + + def get_output(self) -> str: + return self.output + + +class SSHConnectionException(Exception): + """ + SSH connection error. + """ + + host: str + + def __init__(self, host: str): + self.host = host + + def __str__(self) -> str: + return f"Error trying to connect with {self.host}" + + +class SSHSessionDeadException(Exception): + """ + SSH session is not alive. + It can no longer be used. + """ + + host: str + + def __init__(self, host: str): + self.host = host + + def __str__(self) -> str: + return f"SSH session with {self.host} has died" diff --git a/dts/framework/ssh_pexpect.py b/dts/framework/ssh_pexpect.py new file mode 100644 index 0000000000..e8f64515c0 --- /dev/null +++ b/dts/framework/ssh_pexpect.py @@ -0,0 +1,205 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +import time +from typing import Optional + +from pexpect import pxssh + +from .exception import SSHConnectionException, SSHSessionDeadException, TimeoutException +from .logger import DTSLOG +from .utils import GREEN, RED + +""" +The module handles ssh sessions to TG and SUT. +It implements the send_expect function to send commands and get output data. +""" + + +class SSHPexpect: + username: str + password: str + node: str + logger: DTSLOG + magic_prompt: str + + def __init__( + self, + node: str, + username: str, + password: Optional[str], + logger: DTSLOG, + ): + self.magic_prompt = "MAGIC PROMPT" + self.logger = logger + + self.node = node + self.username = username + self.password = password or "" + self.logger.info(f"ssh {self.username}@{self.node}") + + self._connect_host() + + def _connect_host(self) -> None: + """ + Create connection to assigned node. + """ + retry_times = 10 + try: + if ":" in self.node: + while retry_times: + self.ip = self.node.split(":")[0] + self.port = int(self.node.split(":")[1]) + self.session = pxssh.pxssh(encoding="utf-8") + try: + self.session.login( + self.ip, + self.username, + self.password, + original_prompt="[$#>]", + port=self.port, + login_timeout=20, + password_regex=r"(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)", + ) + except Exception as e: + print(e) + time.sleep(2) + retry_times -= 1 + print("retry %d times connecting..." % (10 - retry_times)) + else: + break + else: + raise Exception("connect to %s:%s failed" % (self.ip, self.port)) + else: + self.session = pxssh.pxssh(encoding="utf-8") + self.session.login( + self.node, + self.username, + self.password, + original_prompt="[$#>]", + password_regex=r"(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)", + ) + self.logger.info(f"Connection to {self.node} succeeded") + self.send_expect("stty -echo", "#") + self.send_expect("stty columns 1000", "#") + except Exception as e: + print(RED(str(e))) + if getattr(self, "port", None): + suggestion = ( + "\nSuggession: Check if the firewall on [ %s ] " % self.ip + + "is stopped\n" + ) + print(GREEN(suggestion)) + + raise SSHConnectionException(self.node) + + def send_expect_base(self, command: str, expected: str, timeout: float) -> str: + self.clean_session() + self.session.PROMPT = expected + self.__sendline(command) + self.__prompt(command, timeout) + + before = self.get_output_before() + return before + + def send_expect( + self, command: str, expected: str, timeout: float = 15, verify: bool = False + ) -> str | int: + + try: + ret = self.send_expect_base(command, expected, timeout) + if verify: + ret_status = self.send_expect_base("echo $?", expected, timeout) + if not int(ret_status): + return ret + else: + self.logger.error("Command: %s failure!" % command) + self.logger.error(ret) + return int(ret_status) + else: + return ret + except Exception as e: + print( + RED( + "Exception happened in [%s] and output is [%s]" + % (command, self.get_output_before()) + ) + ) + raise e + + def send_command(self, command: str, timeout: float = 1) -> str: + try: + self.clean_session() + self.__sendline(command) + except Exception as e: + raise e + + output = self.get_session_before(timeout=timeout) + self.session.PROMPT = self.session.UNIQUE_PROMPT + self.session.prompt(0.1) + + return output + + def clean_session(self) -> None: + self.get_session_before(timeout=0.01) + + def get_session_before(self, timeout: float = 15) -> str: + """ + Get all output before timeout + """ + self.session.PROMPT = self.magic_prompt + try: + self.session.prompt(timeout) + except Exception as e: + pass + + before = self.get_output_all() + self.__flush() + + return before + + def __flush(self) -> None: + """ + Clear all session buffer + """ + self.session.buffer = "" + self.session.before = "" + + def __prompt(self, command: str, timeout: float) -> None: + if not self.session.prompt(timeout): + raise TimeoutException(command, self.get_output_all()) from None + + def __sendline(self, command: str) -> None: + if not self.isalive(): + raise SSHSessionDeadException(self.node) + if len(command) == 2 and command.startswith("^"): + self.session.sendcontrol(command[1]) + else: + self.session.sendline(command) + + def get_output_before(self) -> str: + if not self.isalive(): + raise SSHSessionDeadException(self.node) + before: list[str] = self.session.before.rsplit("\r\n", 1) + if before[0] == "[PEXPECT]": + before[0] = "" + + return before[0] + + def get_output_all(self) -> str: + output: str = self.session.before + output.replace("[PEXPECT]", "") + return output + + def close(self, force: bool = False) -> None: + if force is True: + self.session.close() + else: + if self.isalive(): + self.session.logout() + + def isalive(self) -> bool: + return self.session.isalive() diff --git a/dts/framework/utils.py b/dts/framework/utils.py new file mode 100644 index 0000000000..db87349827 --- /dev/null +++ b/dts/framework/utils.py @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +def RED(text: str) -> str: + return f"\u001B[31;1m{str(text)}\u001B[0m" + + +def GREEN(text: str) -> str: + return f"\u001B[32;1m{str(text)}\u001B[0m" From patchwork Fri Jul 29 10:55:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114439 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D38A6A04FD; Fri, 29 Jul 2022 12:56:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4F3A442C29; Fri, 29 Jul 2022 12:56:05 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id 2D22142C27 for ; Fri, 29 Jul 2022 12:56:01 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id 7976613125F; Fri, 29 Jul 2022 12:56:00 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wxfAGhli0blN; Fri, 29 Jul 2022 12:55:59 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id BDE961C1C73; Fri, 29 Jul 2022 12:55:53 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 5/9] dts: add ssh connection extension Date: Fri, 29 Jul 2022 10:55:46 +0000 Message-Id: <20220729105550.1382664-6-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The class adds logging and history records to existing pexpect methods. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/framework/ssh_connection.py | 70 +++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 dts/framework/ssh_connection.py diff --git a/dts/framework/ssh_connection.py b/dts/framework/ssh_connection.py new file mode 100644 index 0000000000..bbf7c8ef01 --- /dev/null +++ b/dts/framework/ssh_connection.py @@ -0,0 +1,70 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +import dataclasses +from typing import Any, Optional + +from .logger import DTSLOG +from .ssh_pexpect import SSHPexpect + + +@dataclasses.dataclass(slots=True, frozen=True) +class HistoryRecord: + command: str + name: str + output: str | int + + +class SSHConnection(object): + """ + Module for create session to node. + """ + + name: str + history: list[HistoryRecord] + logger: DTSLOG + session: SSHPexpect | Any + + def __init__( + self, + node: str, + session_name: str, + logger: DTSLOG, + username: str, + password: Optional[str] = "", + ): + self.session = SSHPexpect(node, username, password, logger) + self.name = session_name + self.history = [] + self.logger = logger + + def send_expect( + self, cmds: str, expected: str, timeout: float = 15, verify: bool = False + ) -> str | int: + self.logger.info(cmds) + out = self.session.send_expect(cmds, expected, timeout, verify) + if isinstance(out, str): + self.logger.debug(out.replace(cmds, "")) + self.history.append(HistoryRecord(command=cmds, name=self.name, output=out)) + return out + + def send_command(self, cmds: str, timeout: float = 1) -> str: + self.logger.info(cmds) + out = self.session.send_command(cmds, timeout) + self.logger.debug(out.replace(cmds, "")) + self.history.append(HistoryRecord(command=cmds, name=self.name, output=out)) + return out + + def get_session_before(self, timeout: float = 15) -> str: + out = self.session.get_session_before(timeout) + self.logger.debug(out) + return out + + def close(self, force: bool = False) -> None: + if getattr(self, "logger", None): + self.logger.logger_exit() + + self.session.close(force) From patchwork Fri Jul 29 10:55:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114440 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C8D3EA04FD; Fri, 29 Jul 2022 12:56:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5BC8742C39; Fri, 29 Jul 2022 12:56:06 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id D874A42C27 for ; Fri, 29 Jul 2022 12:56:01 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id 2FD261C124E; Fri, 29 Jul 2022 12:56:01 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id viu1AnUoyx3O; Fri, 29 Jul 2022 12:55:59 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id 7FBCB1C1C7A; Fri, 29 Jul 2022 12:55:54 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 6/9] dts: add config parser module Date: Fri, 29 Jul 2022 10:55:47 +0000 Message-Id: <20220729105550.1382664-7-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Owen Hilyard The configuration is split into two parts, one defining the parameters of the test run and the other defining the topology to be used. The format of the configuration is YAML. It is validated according to a json schema which also servers as detailed documentation of the various configuration fields. This means that the complete set of allowed values are tied to the schema as a source of truth. This enables making changes to parts of DTS that interface with config files without a high risk of breaking someone's configuration. This configuration system uses immutable objects to represent the configuration, making IDE/LSP autocomplete work properly. There are two ways to specify the configuration file path, an environment variable or a command line argument, applied in that order. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/conf.yaml | 7 ++ dts/framework/config/__init__.py | 99 ++++++++++++++++++++++ dts/framework/config/conf_yaml_schema.json | 73 ++++++++++++++++ dts/framework/exception.py | 14 +++ dts/framework/settings.py | 65 ++++++++++++++ 5 files changed, 258 insertions(+) create mode 100644 dts/conf.yaml create mode 100644 dts/framework/config/__init__.py create mode 100644 dts/framework/config/conf_yaml_schema.json create mode 100644 dts/framework/settings.py diff --git a/dts/conf.yaml b/dts/conf.yaml new file mode 100644 index 0000000000..cb12ea3d0f --- /dev/null +++ b/dts/conf.yaml @@ -0,0 +1,7 @@ +executions: + - system_under_test: "SUT 1" +nodes: + - name: "SUT 1" + hostname: "SUT IP address or hostname" + user: root + password: "Leave blank to use SSH keys" diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py new file mode 100644 index 0000000000..a0fdffcd77 --- /dev/null +++ b/dts/framework/config/__init__.py @@ -0,0 +1,99 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2021 Intel Corporation +# Copyright(c) 2022 University of New Hampshire +# + +""" +Generic port and topology nodes configuration file load function +""" +import json +import os.path +import pathlib +from dataclasses import dataclass +from typing import Any, Optional + +import warlock +import yaml + +from framework.settings import SETTINGS + + +# Slots enables some optimizations, by pre-allocating space for the defined +# attributes in the underlying data structure. +# +# Frozen makes the object immutable. This enables further optimizations, +# and makes it thread safe should we every want to move in that direction. +@dataclass(slots=True, frozen=True) +class NodeConfiguration: + name: str + hostname: str + user: str + password: Optional[str] + + @staticmethod + def from_dict(d: dict) -> "NodeConfiguration": + return NodeConfiguration( + name=d["name"], + hostname=d["hostname"], + user=d["user"], + password=d.get("password"), + ) + + +@dataclass(slots=True, frozen=True) +class ExecutionConfiguration: + system_under_test: NodeConfiguration + + @staticmethod + def from_dict(d: dict, node_map: dict) -> "ExecutionConfiguration": + sut_name = d["system_under_test"] + assert sut_name in node_map, f"Unknown SUT {sut_name} in execution {d}" + + return ExecutionConfiguration( + system_under_test=node_map[sut_name], + ) + + +@dataclass(slots=True, frozen=True) +class Configuration: + executions: list[ExecutionConfiguration] + + @staticmethod + def from_dict(d: dict) -> "Configuration": + nodes: list[NodeConfiguration] = list( + map(NodeConfiguration.from_dict, d["nodes"]) + ) + assert len(nodes) > 0, "There must be a node to test" + + node_map = {node.name: node for node in nodes} + assert len(nodes) == len(node_map), "Duplicate node names are not allowed" + + executions: list[ExecutionConfiguration] = list( + map( + ExecutionConfiguration.from_dict, d["executions"], [node_map for _ in d] + ) + ) + + return Configuration(executions=executions) + + +def load_config() -> Configuration: + """ + Loads the configuration file and the configuration file schema, + validates the configuration file, and creates a configuration object. + """ + with open(SETTINGS.config_file_path, "r") as f: + config_data = yaml.safe_load(f) + + schema_path = os.path.join( + pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json" + ) + + with open(schema_path, "r") as f: + schema = json.load(f) + config: dict[str, Any] = warlock.model_factory(schema, name="_Config")(config_data) + config_obj: Configuration = Configuration.from_dict(dict(config)) + return config_obj + + +CONFIGURATION = load_config() diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json new file mode 100644 index 0000000000..04b2bec3a5 --- /dev/null +++ b/dts/framework/config/conf_yaml_schema.json @@ -0,0 +1,73 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema", + "title": "DPDK DTS Config Schema", + "definitions": { + "node_name": { + "type": "string", + "description": "A unique identifier for a node" + }, + "node_role": { + "type": "string", + "description": "The role a node plays in DTS", + "enum": [ + "system_under_test", + "traffic_generator" + ] + } + }, + "type": "object", + "properties": { + "nodes": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "A unique identifier for this node" + }, + "hostname": { + "type": "string", + "description": "A hostname from which the node running DTS can access this node. This can also be an IP address." + }, + "user": { + "type": "string", + "description": "The user to access this node with." + }, + "password": { + "type": "string", + "description": "The password to use on this node. SSH keys are preferred." + } + }, + "additionalProperties": false, + "required": [ + "name", + "hostname", + "user" + ] + }, + "minimum": 1 + }, + "executions": { + "type": "array", + "items": { + "type": "object", + "properties": { + "system_under_test": { + "$ref": "#/definitions/node_name" + } + }, + "additionalProperties": false, + "required": [ + "system_under_test" + ] + }, + "minimum": 1 + } + }, + "required": [ + "executions", + "nodes" + ], + "additionalProperties": false +} diff --git a/dts/framework/exception.py b/dts/framework/exception.py index 35e81a4d99..8466990aa5 100644 --- a/dts/framework/exception.py +++ b/dts/framework/exception.py @@ -55,3 +55,17 @@ def __init__(self, host: str): def __str__(self) -> str: return f"SSH session with {self.host} has died" + + +class ConfigParseException(Exception): + """ + Configuration file parse failure exception. + """ + + config: str + + def __init__(self, conf_file: str): + self.config = conf_file + + def __str__(self) -> str: + return f"Failed to parse config file [{self.config}]" diff --git a/dts/framework/settings.py b/dts/framework/settings.py new file mode 100644 index 0000000000..4793d550ac --- /dev/null +++ b/dts/framework/settings.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2021 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +import argparse +import os +from dataclasses import dataclass +from typing import Any + + +class _EnvironmentArgument(argparse.Action): + def __init__( + self, env_var: str, required: bool = True, default: Any = None, **kwargs + ): + env_var_value = os.environ.get(env_var) + default = env_var_value or default + super(_EnvironmentArgument, self).__init__( + default=default, required=default is None and required, **kwargs + ) + + def __call__( + self, + parser: argparse.ArgumentParser, + namespace: argparse.Namespace, + values: Any, + option_string: str = None, + ) -> None: + setattr(namespace, self.dest, values) + + +def _env_arg(envvar: str) -> Any: + def wrapper(**kwargs) -> _EnvironmentArgument: + return _EnvironmentArgument(envvar, **kwargs) + + return wrapper + + +@dataclass(slots=True, frozen=True) +class _Settings: + config_file_path: str + + +def _get_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="DPDK test framework.") + + parser.add_argument( + "--config-file", + action=_env_arg("DTS_CFG_FILE"), + default="conf.yaml", + help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets", + ) + + return parser + + +def _get_settings() -> _Settings: + args = _get_parser().parse_args() + return _Settings( + config_file_path=args.config_file, + ) + + +SETTINGS: _Settings = _get_settings() From patchwork Fri Jul 29 10:55:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114441 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D32BA04FD; Fri, 29 Jul 2022 12:56:42 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 98DEA42C30; Fri, 29 Jul 2022 12:56:11 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id A532142C15 for ; Fri, 29 Jul 2022 12:56:02 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id F3B5A1C1C73; Fri, 29 Jul 2022 12:56:01 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zULRDdVf3ZBi; Fri, 29 Jul 2022 12:56:01 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id 3585613125C; Fri, 29 Jul 2022 12:55:55 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 7/9] dts: add Node base class Date: Fri, 29 Jul 2022 10:55:48 +0000 Message-Id: <20220729105550.1382664-8-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The base class implements basic node management methods - connect and execute commands. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/framework/node.py | 99 +++++++++++++++++++++++++++++++++++++++ dts/framework/settings.py | 11 +++++ 2 files changed, 110 insertions(+) create mode 100644 dts/framework/node.py diff --git a/dts/framework/node.py b/dts/framework/node.py new file mode 100644 index 0000000000..e5c5454ebe --- /dev/null +++ b/dts/framework/node.py @@ -0,0 +1,99 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +from typing import Optional + +from .config import NodeConfiguration +from .logger import DTSLOG, getLogger +from .settings import SETTINGS +from .ssh_connection import SSHConnection + +""" +A node is a generic host that DTS connects to and manages. +""" + + +class Node(object): + """ + Basic module for node management. This module implements methods that + manage a node, such as information gathering (of CPU/PCI/NIC) and + environment setup. + """ + + _config: NodeConfiguration + logger: DTSLOG + main_session: SSHConnection + name: str + _other_sessions: list[SSHConnection] + + def __init__(self, node_config: NodeConfiguration): + self._config = node_config + self.name = node_config.name + + self.logger = getLogger(self.name) + self.logger.info(f"Created node: {self.name}") + self.main_session = SSHConnection( + self.get_ip_address(), + self.name, + self.logger, + self.get_username(), + self.get_password(), + ) + + def get_ip_address(self) -> str: + """ + Get SUT's ip address. + """ + return self._config.hostname + + def get_password(self) -> Optional[str]: + """ + Get SUT's login password. + """ + return self._config.password + + def get_username(self) -> str: + """ + Get SUT's login username. + """ + return self._config.user + + def send_expect( + self, + command: str, + expected: str, + timeout: float = SETTINGS.timeout, + verify: bool = False, + trim_whitespace: bool = True, + ) -> str | int: + """ + Send commands to node and return string before expected string. If + there's no expected string found before timeout, TimeoutException will + be raised. + + By default, it will trim the whitespace from the expected string. This + behavior can be turned off via the trim_whitespace argument. + """ + + if trim_whitespace: + expected = expected.strip() + + return self.main_session.send_expect(command, expected, timeout, verify) + + def send_command(self, cmds: str, timeout: float = SETTINGS.timeout) -> str: + """ + Send commands to node and return string before timeout. + """ + + return self.main_session.send_command(cmds, timeout) + + def node_exit(self) -> None: + """ + Recover all resource before node exit + """ + if self.main_session: + self.main_session.close() + self.logger.logger_exit() diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 4793d550ac..07952b9f9e 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -40,6 +40,7 @@ def wrapper(**kwargs) -> _EnvironmentArgument: @dataclass(slots=True, frozen=True) class _Settings: config_file_path: str + timeout: float def _get_parser() -> argparse.ArgumentParser: @@ -52,6 +53,15 @@ def _get_parser() -> argparse.ArgumentParser: help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets", ) + parser.add_argument( + "-t", + "--timeout", + action=_env_arg("DTS_TIMEOUT"), + default=15, + required=False, + help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.", + ) + return parser @@ -59,6 +69,7 @@ def _get_settings() -> _Settings: args = _get_parser().parse_args() return _Settings( config_file_path=args.config_file, + timeout=float(args.timeout), ) From patchwork Fri Jul 29 10:55:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114442 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 35935A04FD; Fri, 29 Jul 2022 12:56:49 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8EE6642C40; Fri, 29 Jul 2022 12:56:12 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id 6C33E42C22 for ; Fri, 29 Jul 2022 12:56:03 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id B96FA13125C; Fri, 29 Jul 2022 12:56:02 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MuO70snVFK8o; Fri, 29 Jul 2022 12:56:01 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id A833F1C1C7D; Fri, 29 Jul 2022 12:55:55 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 8/9] dts: add dts workflow module Date: Fri, 29 Jul 2022 10:55:49 +0000 Message-Id: <20220729105550.1382664-9-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The module implements methods needed to run DTS. It handles the creation of objects and eventually the whole DTS workflow, such as running node setups, test gathering, setup and execution and various cleanups. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/framework/dts.py | 70 +++++++++++++++++++++++++++++++++++++++ dts/framework/settings.py | 11 ++++++ dts/framework/utils.py | 19 +++++++++++ 3 files changed, 100 insertions(+) create mode 100644 dts/framework/dts.py diff --git a/dts/framework/dts.py b/dts/framework/dts.py new file mode 100644 index 0000000000..1938ea6af8 --- /dev/null +++ b/dts/framework/dts.py @@ -0,0 +1,70 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2019 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +import sys +from typing import Iterable, Optional + +import framework.logger as logger + +from .config import CONFIGURATION +from .logger import getLogger +from .node import Node +from .settings import SETTINGS +from .utils import check_dts_python_version + +log_handler: Optional[logger.DTSLOG] = None + + +def run_all() -> None: + """ + Main process of DTS, it will run all test suites in the config file. + """ + + global log_handler + + # check the python version of the server that run dts + check_dts_python_version() + + # init log_handler handler + if SETTINGS.verbose is True: + logger.set_verbose() + + log_handler = getLogger("dts") + + nodes = {} + # This try/finally block means "Run the try block, if there is an exception, + # run the finally block before passing it upward. If there is not an exception, + # run the finally block after the try block is finished." This helps avoid the + # problem of python's interpreter exit context, which essentially prevents you + # from making certain system calls. This makes cleaning up resources difficult, + # since most of the resources in DTS are network-based, which is restricted. + # + # An except block SHOULD NOT be added to this. A failure at this level should + # deliver a full stack trace for debugging, since the only place that exceptions + # should be caught and handled is in the testing code. + try: + # for all Execution sections + for execution in CONFIGURATION.executions: + sut_config = execution.system_under_test + if sut_config.name not in nodes: + nodes[sut_config.name] = Node(sut_config) + + finally: + quit_execution(nodes.values()) + + +def quit_execution(sut_nodes: Iterable[Node]) -> None: + """ + Close session to SUT and TG before quit. + Return exit status when failure occurred. + """ + for sut_node in sut_nodes: + # close all session + sut_node.node_exit() + + if log_handler is not None: + log_handler.info("DTS ended") + sys.exit(0) diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 07952b9f9e..c9621d4e3d 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -41,6 +41,7 @@ def wrapper(**kwargs) -> _EnvironmentArgument: class _Settings: config_file_path: str timeout: float + verbose: bool def _get_parser() -> argparse.ArgumentParser: @@ -62,6 +63,15 @@ def _get_parser() -> argparse.ArgumentParser: help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.", ) + parser.add_argument( + "-v", + "--verbose", + action=_env_arg("DTS_VERBOSE"), + default="N", + required=False, + help="[DTS_VERBOSE] Set to 'Y' to enable verbose output, logging all messages to the console.", + ) + return parser @@ -70,6 +80,7 @@ def _get_settings() -> _Settings: return _Settings( config_file_path=args.config_file, timeout=float(args.timeout), + verbose=(args.verbose == "Y"), ) diff --git a/dts/framework/utils.py b/dts/framework/utils.py index db87349827..2a174831d0 100644 --- a/dts/framework/utils.py +++ b/dts/framework/utils.py @@ -4,9 +4,28 @@ # Copyright(c) 2022 University of New Hampshire # +import sys + + def RED(text: str) -> str: return f"\u001B[31;1m{str(text)}\u001B[0m" def GREEN(text: str) -> str: return f"\u001B[32;1m{str(text)}\u001B[0m" + + +def check_dts_python_version() -> None: + if sys.version_info.major < 3 or ( + sys.version_info.major == 3 and sys.version_info.minor < 10 + ): + print( + RED( + ( + "WARNING: DTS execution node's python version is lower than" + "python 3.10, is deprecated and will not work in future releases." + ) + ), + file=sys.stderr, + ) + print(RED("Please use Python >= 3.10 instead"), file=sys.stderr) From patchwork Fri Jul 29 10:55:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 114443 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0EF99A04FD; Fri, 29 Jul 2022 12:56:56 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7860F42C48; Fri, 29 Jul 2022 12:56:13 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id 0130D42C30 for ; Fri, 29 Jul 2022 12:56:04 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id 4CB6E1C1C7A; Fri, 29 Jul 2022 12:56:03 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gUK6TC7DWzIx; Fri, 29 Jul 2022 12:56:02 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id 7AA4C1C2079; Fri, 29 Jul 2022 12:55:56 +0200 (CEST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v4 9/9] dts: add dts executable script Date: Fri, 29 Jul 2022 10:55:50 +0000 Message-Id: <20220729105550.1382664-10-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The script is an interface to run DTS. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš Reviewed-by: Lijuan Tu --- dts/main.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100755 dts/main.py diff --git a/dts/main.py b/dts/main.py new file mode 100755 index 0000000000..a700707650 --- /dev/null +++ b/dts/main.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022 University of New Hampshire +# + +""" +A test framework for testing DPDK. +""" + +import logging + +from framework import dts + + +def main() -> None: + dts.run_all() + + +# Main program begins here +if __name__ == "__main__": + logging.raiseExceptions = True + main()