From patchwork Tue Mar 12 17:25:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeremy Spewock X-Patchwork-Id: 138226 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B36C243C94; Tue, 12 Mar 2024 18:26:33 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A037A42D90; Tue, 12 Mar 2024 18:26:33 +0100 (CET) Received: from mail-pg1-f226.google.com (mail-pg1-f226.google.com [209.85.215.226]) by mails.dpdk.org (Postfix) with ESMTP id 11D2F42D89 for ; Tue, 12 Mar 2024 18:26:32 +0100 (CET) Received: by mail-pg1-f226.google.com with SMTP id 41be03b00d2f7-5dcc4076c13so61016a12.0 for ; Tue, 12 Mar 2024 10:26:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1710264391; x=1710869191; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aslyKnAynBtHbwjUDaSheVxbl6+MwYCHgzOkaqUlbFI=; b=B0G35cWWX9Xa8ARxNatBQNNJUNrB108KL7aSqZDSlBlxX1vHTN2E9DW6BvneuGK9aC 8ws+6RGs1OYUgMbe6bgbZMt8aTy9wWtL4cRpoOEGRAV3KYxk4BxIb9QTemtxLKq+UIv+ gccaTAP3Edga6ACqmGFaXeN2xNI1N1aG2bJCY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710264391; x=1710869191; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aslyKnAynBtHbwjUDaSheVxbl6+MwYCHgzOkaqUlbFI=; b=eug+tp11LDjroHn1x3squBmRd2FPzj3Aifn9YeqeOQnqp0ixX++LTT/9XBQkBqYTr5 AiO2IoPp6YEBhTdtlKm8b2/9vYnqOqbiHJKHmtfRpGDemOyLgZCccne1X5asky0OpVsT ljifXvQp14HGDUYvFvhKD8nkEzCGKANscYeIIvGMNSCyMzOlKdgTCjYJB71ezpoXqFlC tRyY/x8dOv+X0C/S3Cr+YWfoMZTroUu3VcsiBEzrSlAupGSKC3/vTDQct7lnpc4lgpoi FThdf+eHthWvwKAIJ3/mHwAxy3HK2El9sVBn46ehShp3b3J09hF8/ZhKT3UxobGAsGs/ 23dw== X-Gm-Message-State: AOJu0YyTIwm40sWx+7fzKpO4qdXW8rcx/6l7iAa2diLu01vTtx1yt4mO 0FS6SV0l60C3/2fNvRVeeNRbgbFa7pxaRPR2qHRLvcQxduA+Af6REi+lGSBLRUFzmlDcC6nOv5O ravXW2xyBGncyE7XNoDcNClZPhL4Hb2sVdJymn6g1WXCdC7Jf X-Google-Smtp-Source: AGHT+IGtSlwPhLXSKMfaUKEcIBB7+DWqOZSV0o0Wq/ry/LLoy9qqni6WYW3qXFstbS93k07+3s3YyvXklNNc X-Received: by 2002:a17:90a:53e4:b0:29a:72b:df85 with SMTP id y91-20020a17090a53e400b0029a072bdf85mr306815pjh.7.1710264391188; Tue, 12 Mar 2024 10:26:31 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [2606:4100:3880:1234::84]) by smtp-relay.gmail.com with ESMTPS id n3-20020a17090ade8300b0029af7a6e1a8sm395550pjv.4.2024.03.12.10.26.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2024 10:26:31 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1271:90f9:1b64:f6e6:867f]) by postal.iol.unh.edu (Postfix) with ESMTP id 519716052471; Tue, 12 Mar 2024 13:26:30 -0400 (EDT) From: jspewock@iol.unh.edu To: Honnappa.Nagarahalli@arm.com, juraj.linkes@pantheon.tech, thomas@monjalon.net, wathsala.vithanage@arm.com, probb@iol.unh.edu, paul.szczepanek@arm.com, yoan.picchi@foss.arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, Jeremy Spewock Subject: [PATCH v1 1/2] dts: Improve output gathering in interactive shells Date: Tue, 12 Mar 2024 13:25:57 -0400 Message-ID: <20240312172558.11844-2-jspewock@iol.unh.edu> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240312172558.11844-1-jspewock@iol.unh.edu> References: <20240312172558.11844-1-jspewock@iol.unh.edu> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Jeremy Spewock The current implementation of consuming output from interactive shells relies on being able to find an expected prompt somewhere within the output buffer after sending the command. This is useful in situations where the prompt does not appear in the output itself, but in some practical cases (such as the starting of an XML-RPC server for scapy) the prompt exists in one of the commands sent to the shell and this can cause the command to exit early and creates a race condition between the server starting and the first command being sent to the server. This patch addresses this problem by searching for a line that strictly ends with the provided prompt, rather than one that simply contains it, so that the detection that a command is finished is more consistent. It also adds a catch to detect when a command times out before finding the prompt so that the exception can be wrapped into a more explicit one and display the output that it did manage to gather before timing out. Bugzilla ID: 1359 Fixes: 88489c0501af ("dts: add smoke tests") Signed-off-by: Jeremy Spewock --- dts/framework/exception.py | 7 +++++ .../remote_session/interactive_shell.py | 26 +++++++++++++------ 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/dts/framework/exception.py b/dts/framework/exception.py index 658eee2c38..cce1e0231a 100644 --- a/dts/framework/exception.py +++ b/dts/framework/exception.py @@ -146,6 +146,13 @@ def __str__(self) -> str: return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}" +class InteractiveCommandExecutionError(DTSError): + """An unsuccessful execution of a remote command in an interactive environment.""" + + #: + severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR + + class RemoteDirectoryExistsError(DTSError): """A directory that exists on a remote node.""" diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py index 5cfe202e15..2bcfdcb3c7 100644 --- a/dts/framework/remote_session/interactive_shell.py +++ b/dts/framework/remote_session/interactive_shell.py @@ -20,6 +20,7 @@ from paramiko import Channel, SSHClient, channel # type: ignore[import] +from framework.exception import InteractiveCommandExecutionError from framework.logger import DTSLogger from framework.settings import SETTINGS @@ -124,6 +125,10 @@ def send_command(self, command: str, prompt: str | None = None) -> str: Returns: All output in the buffer before expected string. + + Raises: + InteractiveCommandExecutionError: If command was sent but prompt could not be found in + the output. """ self._logger.info(f"Sending: '{command}'") if prompt is None: @@ -131,14 +136,19 @@ def send_command(self, command: str, prompt: str | None = None) -> str: self._stdin.write(f"{command}{self._command_extra_chars}\n") self._stdin.flush() out: str = "" - for line in self._stdout: - out += line - if prompt in line and not line.rstrip().endswith( - command.rstrip() - ): # ignore line that sent command - break - self._logger.debug(f"Got output: {out}") - return out + try: + for line in self._stdout: + out += line + if line.rstrip().endswith(prompt): + break + except TimeoutError: + raise InteractiveCommandExecutionError( + f"Failed to find the prompt ({prompt}) at the end of a line in the output from the" + f" command ({command}). Got:\n{out}" + ) + else: + self._logger.debug(f"Got output: {out}") + return out def close(self) -> None: """Properly free all resources.""" From patchwork Tue Mar 12 17:25:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeremy Spewock X-Patchwork-Id: 138227 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 326B943C94; Tue, 12 Mar 2024 18:26:39 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CEFE042D9F; Tue, 12 Mar 2024 18:26:34 +0100 (CET) Received: from mail-yw1-f228.google.com (mail-yw1-f228.google.com [209.85.128.228]) by mails.dpdk.org (Postfix) with ESMTP id 1E34D42D90 for ; Tue, 12 Mar 2024 18:26:32 +0100 (CET) Received: by mail-yw1-f228.google.com with SMTP id 00721157ae682-60a046c5262so40372707b3.2 for ; Tue, 12 Mar 2024 10:26:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1710264391; x=1710869191; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eNnbsEuPm6Ao9V2u9FX2LI7//LDOOF21WP+ap5evX1k=; b=S4lN2sPtsIvmmxjmkRxgRlUgUCWTvQegDvSj8sxtgc9lBbhAudbnu/scP1DPGKS3tn RzTDVJK7W8Jxs7L+dAFupLCR9Y+yZkrHdSxdo+pl/3rqgKIZm5E/odLJP362OkTiS0xU 4eIWX69/Az3Ru9uSCDJd0b/TL1vCqdtP0WYog= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710264391; x=1710869191; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eNnbsEuPm6Ao9V2u9FX2LI7//LDOOF21WP+ap5evX1k=; b=ZQ9+k1aKMTr4nTlLpTM7lwNpbbU6Sihr2lik1h1N8xizEIAv629YswX6lYfnUHrx+8 waZNVdIWY/o1vYV2iN7z5tG6VCcu09p6n1TYGVtJlQRnrRkX+MkEeIOHp0Kx+fRD7lR8 GYBPXv78t0iQzShEn39wSDwcKWj46P2PHQnQaMz/iJZMib+yv1wtuIoIBV4SnI6D3FAD xLxkeq3I5hgRe1sWlrQFmpydXSlhMiGyNSfgiujPih1Js9auIn7XOhz0B23BZODv0YpO trHRKIZT4WgtgD6YRML4AOOMOqyzS1Yz3hg7lBdPT4n7RJ+pTyEOyCpqZ41Ui2EAERGj Z6Zw== X-Gm-Message-State: AOJu0YwgZSCVH8F6glzyjqSwW3WG6N9Ohha11aW/rRN2zk4ZSJBXaJLm fWfq6GQ2R+Om9PUwJcENM2lXSw841gfbGkBKNv1vyR7kKtt4b4II4dCJXuLmU3/9aYebervIXAF 9U4P0SMxG5Er31gTAbQE+ysdlmfvmN1hx X-Google-Smtp-Source: AGHT+IEEp6uwh0IL3oNmaNaTqpQgXkCpyoORcdf92UI5R531s+olFjqE2+EmzhB7iJ+Pm5dgnsuKvgBotzkR X-Received: by 2002:a0d:e502:0:b0:608:d1b3:30ac with SMTP id o2-20020a0de502000000b00608d1b330acmr121602ywe.30.1710264391445; Tue, 12 Mar 2024 10:26:31 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [132.177.123.84]) by smtp-relay.gmail.com with ESMTPS id dc14-20020a05690c0f0e00b0060a1d21048fsm206851ywb.1.2024.03.12.10.26.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Mar 2024 10:26:31 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1271:90f9:1b64:f6e6:867f]) by postal.iol.unh.edu (Postfix) with ESMTP id 023A36052472; Tue, 12 Mar 2024 13:26:31 -0400 (EDT) From: jspewock@iol.unh.edu To: Honnappa.Nagarahalli@arm.com, juraj.linkes@pantheon.tech, thomas@monjalon.net, wathsala.vithanage@arm.com, probb@iol.unh.edu, paul.szczepanek@arm.com, yoan.picchi@foss.arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, Jeremy Spewock Subject: [PATCH v1 2/2] dts: Add missing docstring from XML-RPC server Date: Tue, 12 Mar 2024 13:25:58 -0400 Message-ID: <20240312172558.11844-3-jspewock@iol.unh.edu> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240312172558.11844-1-jspewock@iol.unh.edu> References: <20240312172558.11844-1-jspewock@iol.unh.edu> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Jeremy Spewock When this XML-RPC server implementation was added, the docstring had to be shortened in order to reduce the chances of this race condition being encountered. Now that this race condition issue is resolved, the full docstring can be restored. Signed-off-by: Jeremy Spewock --- .../testbed_model/traffic_generator/scapy.py | 46 ++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/dts/framework/testbed_model/traffic_generator/scapy.py b/dts/framework/testbed_model/traffic_generator/scapy.py index 5b60f66237..1b46e613a4 100644 --- a/dts/framework/testbed_model/traffic_generator/scapy.py +++ b/dts/framework/testbed_model/traffic_generator/scapy.py @@ -125,9 +125,53 @@ def scapy_send_packets(xmlrpc_packets: list[xmlrpc.client.Binary], send_iface: s class QuittableXMLRPCServer(SimpleXMLRPCServer): - """Basic XML-RPC server. + r"""Basic XML-RPC server. The server may be augmented by functions serializable by the :mod:`marshal` module. + + Example: + :: + + def hello_world(): + # to be sent to the XML-RPC server + print("Hello World!") + + # start the XML-RPC server on the remote node + # the example assumes you're already connect to a tg_node + # this is done by starting a Python shell on the remote node + from framework.remote_session import PythonShell + session = tg_node.create_interactive_shell(PythonShell, timeout=5, privileged=True) + + # then importing the modules needed to run the server + # and the modules for any functions later added to the server + session.send_command("import xmlrpc") + session.send_command("from xmlrpc.server import SimpleXMLRPCServer") + + # sending the source code of this class to the Python shell + from xmlrpc.server import SimpleXMLRPCServer + src = inspect.getsource(QuittableXMLRPCServer) + src = "\n".join([l for l in src.splitlines() if not l.isspace() and l != ""]) + spacing = "\n" * 4 + session.send_command(spacing + src + spacing) + + # then starting the server with: + command = "s = QuittableXMLRPCServer(('0.0.0.0', {listen_port}));s.serve_forever()" + session.send_command(command, "XMLRPC OK") + + # now the server is running on the remote node and we can add functions to it + # first connect to the server from the execution node + import xmlrpc.client + server_url = f"http://{tg_node.config.hostname}:8000" + rpc_server_proxy = xmlrpc.client.ServerProxy(server_url) + + # get the function bytes to send + import marshal + function_bytes = marshal.dumps(hello_world.__code__) + rpc_server_proxy.add_rpc_function(hello_world.__name__, function_bytes) + + # now we can execute the function on the server + xmlrpc_binary_recv: xmlrpc.client.Binary = rpc_server_proxy.hello_world() + print(str(xmlrpc_binary_recv)) """ def __init__(self, *args, **kwargs):