From patchwork Fri Feb 23 07:54:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 678 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7DBE643B9B; Fri, 23 Feb 2024 08:55:07 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 079624068A; Fri, 23 Feb 2024 08:55:07 +0100 (CET) Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) by mails.dpdk.org (Postfix) with ESMTP id 1B321402ED for ; Fri, 23 Feb 2024 08:55:05 +0100 (CET) Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-5649c25369aso531632a12.2 for ; Thu, 22 Feb 2024 23:55:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1708674904; x=1709279704; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hoPZJlnbMWUwbRjLwuLA+NFmXXD5GsJPVQM0oIuWXpg=; b=iDCrkw5rvhcQhQ1gcMkA5lHFhhX8gLMcF5YWWe/Jv7qw06piTbBKHXUftRCryATeHK qkyAnhdnzl5FSRl+rUtXRzOYQrZEjVni9/7F9/K/58D0v9Q4hYS+LmsxVHGjKY7Zm0VN cPTbZcHt7Y4D8FCfm7lpj7+sZ/boF05tGTpsinQr5+4bJD7Cfgz4Kzipcs+VPQoYDB3y 2bVQufWq/PG+OchlTdxGpTLHlq+jc2UEQRHLmwmtY9mic7n2JSBk4Mlr5Kqo86QEWXEg fvzaegTS/epX5KdQPxNjez5dDcA2xYq/zFxe4b3EFk9vUN5BlnWmj0bcSogiPTePO7/F eWZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708674904; x=1709279704; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hoPZJlnbMWUwbRjLwuLA+NFmXXD5GsJPVQM0oIuWXpg=; b=Dk3SMXzwKnj+yFQdCumxh1yj6RX8G1z1Zwj+cx6y9K2zDZxQxn2I84oAxMNOYMZnTM TnIOvPdOvbRJZgkpi1wFB1pO++sPJUPutfDvtUUNu1vtXLUAX5y6uKY09GwR7MrB9rMY mBZ2kM3sZBWieAXKza4yMnLU0AWeBRQQUz57AKJc5pCG1vN/jUFu3vu2picBBdkqhLEw 3PzmJMzLeerBUL+t9wQDgAi0rfPU1+IDtvtOEyZU75YNT8/N/qUplYi0zIZsRf7kEqmh 79yQRrbP80G4eX5CcGkn8o/0nf5Q7d2K30HQtBY/slScf8QeCHHkfpTLDsIVivxBuGp/ iPFQ== X-Gm-Message-State: AOJu0Yz4cbatMvOORKQClh2E5uT+i2fkN7U2OoL96D2d2s23DNViuaNX uAdRhtS2G99rHcUHGRvt7j8JVDwZuTB0eCLL8nd6YDZ6LCYqpkjEmFDp8xzrQrvwVKzxe+7elzx 6wCE= X-Google-Smtp-Source: AGHT+IFKgdIKxkotH7GHI0L5+ta5c56JppvaJKJwqQ8i0J+mYSXs2vpVW5wp4I4zb+s3Rkx3GHWxmg== X-Received: by 2002:a17:906:57c9:b0:a3e:7e1f:449e with SMTP id u9-20020a17090657c900b00a3e7e1f449emr100254ejr.16.1708674904621; Thu, 22 Feb 2024 23:55:04 -0800 (PST) Received: from localhost.localdomain ([84.245.120.62]) by smtp.gmail.com with ESMTPSA id th7-20020a1709078e0700b00a3e059c5c5fsm6660235ejc.188.2024.02.22.23.55.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Feb 2024 23:55:04 -0800 (PST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v3 0/7] test case blocking and logging Date: Fri, 23 Feb 2024 08:54:55 +0100 Message-Id: <20240223075502.60485-1-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231220103331.60888-1-juraj.linkes@pantheon.tech> References: <20231220103331.60888-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org We currently don't record test case results that couldn't be executed because of a previous failure, such as when a test suite setup failed, resulting in no executed test cases. In order to record the test cases that couldn't be executed, we must know the lists of test suites and test cases ahead of the actual test suite execution, as an error could occur before we even start executing test suites. In addition, the patch series contains two refactors and one improvement. The first refactor is closely related. The dts.py was renamed to runner.py and given a clear purpose - running the test suites and all other orchestration needed to run test suites. The logic for this was not all in the original dts.py module and it was brought there. The runner is also responsible for recording results, which is the blocked test cases are recorded. The other refactor, logging, is related to the first refactor. The logging module was simplified while extending capabilities. Each test suite logs into its own log file in addition to the main log file which the runner must handle (as it knows when we start executing particular test suites). The runner also handles the switching between execution stages for the purposes of logging. The one aforementioned improvement is in unifying how we specify test cases in the conf.yaml file and in the environment variable/command line argument. v2: Rebase and update of the whole patch. There are changes in all parts of the code, mainly improving the design and logic. Also added the last patch which improves test suite specification on the cmdline. v3: Improved variables in _get_test_suite_class along with docstring. Fixed smoke test suite not being added into the list of test suites to be executed. Juraj Linkeš (7): dts: convert dts.py methods to class dts: move test suite execution logic to DTSRunner dts: filter test suites in executions dts: reorganize test result dts: block all test cases when earlier setup fails dts: refactor logging configuration dts: improve test suite and case filtering doc/guides/tools/dts.rst | 14 +- dts/framework/config/__init__.py | 36 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/dts.py | 338 --------- dts/framework/logger.py | 236 +++--- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 706 ++++++++++++++++++ dts/framework/settings.py | 188 +++-- dts/framework/test_result.py | 565 ++++++++------ dts/framework/test_suite.py | 239 +----- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 9 +- dts/pyproject.toml | 3 + dts/tests/TestSuite_smoke_tests.py | 2 +- 19 files changed, 1360 insertions(+), 1028 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py