From patchwork Thu Apr 4 15:31:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Pratte X-Patchwork-Id: 139100 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 13F0643DFA; Thu, 4 Apr 2024 17:31:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 000CA40696; Thu, 4 Apr 2024 17:31:16 +0200 (CEST) Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) by mails.dpdk.org (Postfix) with ESMTP id 7881140685 for ; Thu, 4 Apr 2024 17:31:15 +0200 (CEST) Received: by mail-ot1-f41.google.com with SMTP id 46e09a7af769-6e2b466d213so167466a34.0 for ; Thu, 04 Apr 2024 08:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1712244674; x=1712849474; darn=dpdk.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=s8z6i2ENV0FdYzz4H106R2oQr8jC87cFA8ShoL9L6ug=; b=Cw+MJRIDX8Im6NyRZOzp9punocpn7EGyERoDlgX+h6I+NsBZMU7ImrqlUwV/rMXtDd k9JQd2okwbSmoePvW1HqZqk6//idVIyq6EQjySPCOPo6vu7AtpfqY8lVnCEErgAG4bXA qshwkiadehEVsqUICpZZ2ISbShKfW3Xql/06w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712244674; x=1712849474; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=s8z6i2ENV0FdYzz4H106R2oQr8jC87cFA8ShoL9L6ug=; b=nbCjdxdvmiKbMIfOZB/nnRt6l1MAOYsqVhmiDcLubTiw0t48OVjPe3PEHtVmLe+uWH HEpyF0rQPbuDMBY0jYC4f9IMZuuQ0f0XGbiAwApz6Erq8HCoruHknmoIGRnvtCYlhEVe FT+B3HQvNDiUdXkyOHCwbEatcZ9JyslMLu3hrC3GQnjXnFzYVSj+wPcge2we7E1V748O jHpQF9TBR4do2EhjumMTrLkdh72mmuUDRT1j/rk+Zx9y9T/g/LHO6h6C/PM3e3ZS0iVT uzs9n9qH8rItlD6reZSxIqUXYxwR9DxTaEe+k16XNAl8oEuPGs6PE1YMdjSQE8BgyK3B vPZQ== X-Gm-Message-State: AOJu0Yxuz/qtQufz66pqAVLszzzbA7PbFJF++2wHynKpaIZtW8iEqyjZ 83cpqqBWqy8hOfYk2aRyBBA/2ZqC/FZc8TE26/JH6XPCrxd186YlT0VSO50+B3k= X-Google-Smtp-Source: AGHT+IFOuCatyYT0MJksvqhyrNI1KAIIx7LpY0lcrMbwA1Ecs1W2Rs5YlEz4oCCbMjziauuBPJBTow== X-Received: by 2002:a05:6830:1150:b0:6e9:e6b4:1e43 with SMTP id x16-20020a056830115000b006e9e6b41e43mr2879334otq.3.1712244674677; Thu, 04 Apr 2024 08:31:14 -0700 (PDT) Received: from d121028.iol.unh.edu ([2606:4100:3880:1271:e2f8:4ec3:8bf3:864c]) by smtp.gmail.com with ESMTPSA id mn2-20020a0562145ec200b006990bcf5b8asm152809qvb.56.2024.04.04.08.31.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Apr 2024 08:31:14 -0700 (PDT) From: Nicholas Pratte To: paul.szczepanek@arm.com, juraj.linkes@pantheon.tech, yoan.picchi@foss.arm.com, probb@iol.unh.edu, thomas@monjalon.net, npratte@iol.unh.edu, wathsala.vithanage@arm.com, Honnappa.Nagarahalli@arm.com Cc: dev@dpdk.org, Jeremy Spewock Subject: [PATCH] dts: Change hugepage runtime config to 2MB Exclusively Date: Thu, 4 Apr 2024 11:31:06 -0400 Message-ID: <20240404153106.19047-1-npratte@iol.unh.edu> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The previous implementation configures and allocates hugepage sizes based on a system default. This can lead to two problems: overallocation of hugepages (which may crash the remote host), and configuration of hugepage sizes that are not recommended during runtime. This new implementation allows only 2MB hugepage allocation during runtime; any other unique hugepage size must be configured by the end-user for initializing DTS. If the amount of 2MB hugepages requested exceeds the amount of 2MB hugepages already configured on the system, then the system will remount hugepages to cover the difference. If the amount of hugepages requested is either less than or equal to the amount already configured on the system, then nothing is done. Bugzilla ID: 1370 Signed-off-by: Nicholas Pratte Reviewed-by: Jeremy Spewock --- dts/conf.yaml | 8 ++++---- dts/framework/config/__init__.py | 4 ++-- dts/framework/config/conf_yaml_schema.json | 6 +++--- dts/framework/config/types.py | 2 +- dts/framework/testbed_model/linux_session.py | 21 +++++++++----------- 5 files changed, 19 insertions(+), 22 deletions(-) diff --git a/dts/conf.yaml b/dts/conf.yaml index 8068345dd5..3690e77ee5 100644 --- a/dts/conf.yaml +++ b/dts/conf.yaml @@ -35,8 +35,8 @@ nodes: lcores: "" # use all the available logical cores use_first_core: false # tells DPDK to use any physical core memory_channels: 4 # tells DPDK to use 4 memory channels - hugepages: # optional; if removed, will use system hugepage configuration - amount: 256 + hugepages_2mb: # optional; if removed, will use system hugepage configuration + amount: 2560 force_first_numa: false ports: # sets up the physical link between "SUT 1"@000:00:08.0 and "TG 1"@0000:00:08.0 @@ -71,8 +71,8 @@ nodes: os_driver: rdma peer_node: "SUT 1" peer_pci: "0000:00:08.1" - hugepages: # optional; if removed, will use system hugepage configuration - amount: 256 + hugepages_2mb: # optional; if removed, will use system hugepage configuration + amount: 2560 force_first_numa: false traffic_generator: type: SCAPY diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index 4cb5c74059..b6f820e39e 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -255,8 +255,8 @@ def from_dict( Either an SUT or TG configuration instance. """ hugepage_config = None - if "hugepages" in d: - hugepage_config_dict = d["hugepages"] + if "hugepages_2mb" in d: + hugepage_config_dict = d["hugepages_2mb"] if "force_first_numa" not in hugepage_config_dict: hugepage_config_dict["force_first_numa"] = False hugepage_config = HugepageConfiguration(**hugepage_config_dict) diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json index 4731f4511d..f4d7199523 100644 --- a/dts/framework/config/conf_yaml_schema.json +++ b/dts/framework/config/conf_yaml_schema.json @@ -146,7 +146,7 @@ "compiler" ] }, - "hugepages": { + "hugepages_2mb": { "type": "object", "description": "Optional hugepage configuration. If not specified, hugepages won't be configured and DTS will use system configuration.", "properties": { @@ -253,8 +253,8 @@ "type": "integer", "description": "How many memory channels to use. Optional, defaults to 1." }, - "hugepages": { - "$ref": "#/definitions/hugepages" + "hugepages_2mb": { + "$ref": "#/definitions/hugepages_2mb" }, "ports": { "type": "array", diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py index 1927910d88..016e0c3dbd 100644 --- a/dts/framework/config/types.py +++ b/dts/framework/config/types.py @@ -46,7 +46,7 @@ class NodeConfigDict(TypedDict): """Allowed keys and values.""" #: - hugepages: HugepageConfigurationDict + hugepages_2mb: HugepageConfigurationDict #: name: str #: diff --git a/dts/framework/testbed_model/linux_session.py b/dts/framework/testbed_model/linux_session.py index 5d24030c3d..ad3c811301 100644 --- a/dts/framework/testbed_model/linux_session.py +++ b/dts/framework/testbed_model/linux_session.py @@ -15,7 +15,7 @@ from typing_extensions import NotRequired -from framework.exception import RemoteCommandExecutionError +from framework.exception import ConfigurationError, RemoteCommandExecutionError from framework.utils import expand_range from .cpu import LogicalCore @@ -87,25 +87,22 @@ def get_dpdk_file_prefix(self, dpdk_prefix: str) -> str: def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None: """Overrides :meth:`~.os_session.OSSession.setup_hugepages`.""" self._logger.info("Getting Hugepage information.") - hugepage_size = self._get_hugepage_size() + if "hugepages-2048kB" not in self.send_command("ls /sys/kernel/mm/hugepages").stdout: + raise ConfigurationError("2MB hugepages not supported by operating system") hugepages_total = self._get_hugepages_total() self._numa_nodes = self._get_numa_nodes() - if force_first_numa or hugepages_total != hugepage_count: + if force_first_numa or hugepages_total < hugepage_count: # when forcing numa, we need to clear existing hugepages regardless # of size, so they can be moved to the first numa node - self._configure_huge_pages(hugepage_count, hugepage_size, force_first_numa) + self._configure_huge_pages(hugepage_count, force_first_numa) else: self._logger.info("Hugepages already configured.") self._mount_huge_pages() - def _get_hugepage_size(self) -> int: - hugepage_size = self.send_command("awk '/Hugepagesize/ {print $2}' /proc/meminfo").stdout - return int(hugepage_size) - def _get_hugepages_total(self) -> int: hugepages_total = self.send_command( - "awk '/HugePages_Total/ { print $2 }' /proc/meminfo" + "cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages" ).stdout return int(hugepages_total) @@ -136,15 +133,15 @@ def _supports_numa(self) -> bool: # there's no reason to do any numa specific configuration) return len(self._numa_nodes) > 1 - def _configure_huge_pages(self, amount: int, size: int, force_first_numa: bool) -> None: + def _configure_huge_pages(self, amount: int, force_first_numa: bool) -> None: self._logger.info("Configuring Hugepages.") - hugepage_config_path = f"/sys/kernel/mm/hugepages/hugepages-{size}kB/nr_hugepages" + hugepage_config_path = "/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages" if force_first_numa and self._supports_numa(): # clear non-numa hugepages self.send_command(f"echo 0 | tee {hugepage_config_path}", privileged=True) hugepage_config_path = ( f"/sys/devices/system/node/node{self._numa_nodes[0]}/hugepages" - f"/hugepages-{size}kB/nr_hugepages" + f"/hugepages-2048kB/nr_hugepages" ) self.send_command(f"echo {amount} | tee {hugepage_config_path}", privileged=True)