From patchwork Tue Sep 6 16:13:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Mattias_R=C3=B6nnblom?= X-Patchwork-Id: 115983 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 77854A0542; Tue, 6 Sep 2022 18:16:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 99EB1427F3; Tue, 6 Sep 2022 18:16:36 +0200 (CEST) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150041.outbound.protection.outlook.com [40.107.15.41]) by mails.dpdk.org (Postfix) with ESMTP id 577B140143 for ; Tue, 6 Sep 2022 18:16:34 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dJnlbCQnSizNZHh7asiAvQkOuCDWWtyBR8rv5j2NoM2L+NivXxdN+JnknHzPaSzAcvIt6KNXehNzo4QUZYruAvV1K2Rjp/Nmn7jVDBOurP6GGuMmBZrv54bsbvNIJC1tbZcGcMLqYhle5l4NDhtw/In/rEeaehd/p9MJQX8jGRzao7evw17mCpeGtD8DVp5o5Xih8IRzTs2qQM2k0EIVylXQ0etQArPD+B7cUfxUsRzF8ccsjV2rVoGyrnjuv2xbb61mfjscke9+tyAeIfdMrCWBh77M0aBfUC/k5RcAlFCWHf2v+CHn9XkOBclsn5OzblbzKkq3cS+Hm7QkWXYGKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G7g0WGO32pOJMpygsNXlSSEBJ3wWSj0Wa9wtNmIMu/Y=; b=OgwxlWH/bCHpG9dOITVFCEUuQ8n7z3BYNTFYcOmnLkN8/Ops+zFwW5tnzO7v9+E8p3dvx5Zt0OG3o1y7iUXLoT64vbM5wO4O3eJx3dqAjXyRTMUptG4sYNMYvPsxF3cv+CrCnHZaRoqo8fOSWh+AX0LnbTqznV3wDagT/r544EKVO4yXOGmZ0pNYtvHMvEd9Zq3sLHcBHuY60h1/iCB8H34jD5azrPqrGtZIfHdpvKKrs7XbLv5Px2174U9RwO9Yx5BILsH7WPx6EvSBc7lRJ6rVa7fBW9dD8sYqXs49KC0AYDgMeFEqrfoBRjOosXK52uMlyXtpAgciJscWggOmFw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=arm.com smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G7g0WGO32pOJMpygsNXlSSEBJ3wWSj0Wa9wtNmIMu/Y=; b=Ro0sr23SjoyXSJX3JDCq4FLtJ5czwF5rrZ3uSyMIodJS0IMMOMvdMcDKFiQTa3+DB9KjpwZwrli6bwlWT0g7PhfNx/pkDBDMwlUWvgDyzFhQf0Z6mPTydckEX70OlXvPTXS3Z05KjHFstDviQqZv7MICxqiJG4kgHVLfY90+LEw= Received: from AS9PR06CA0214.eurprd06.prod.outlook.com (2603:10a6:20b:45e::19) by AM8PR07MB7601.eurprd07.prod.outlook.com (2603:10a6:20b:240::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.11; Tue, 6 Sep 2022 16:16:32 +0000 Received: from AM5EUR02FT052.eop-EUR02.prod.protection.outlook.com (2603:10a6:20b:45e:cafe::db) by AS9PR06CA0214.outlook.office365.com (2603:10a6:20b:45e::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5588.12 via Frontend Transport; Tue, 6 Sep 2022 16:16:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AM5EUR02FT052.mail.protection.outlook.com (10.152.9.13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.5588.10 via Frontend Transport; Tue, 6 Sep 2022 16:16:32 +0000 Received: from ESESBMB502.ericsson.se (153.88.183.169) by ESESSMR505.ericsson.se (153.88.183.127) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2375.31; Tue, 6 Sep 2022 18:16:31 +0200 Received: from seliicinfr00049.seli.gic.ericsson.se (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.185) with Microsoft SMTP Server id 15.1.2375.31 via Frontend Transport; Tue, 6 Sep 2022 18:16:31 +0200 Received: from localhost.localdomain (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00049.seli.gic.ericsson.se (Postfix) with ESMTP id 545F6380061; Tue, 6 Sep 2022 18:16:31 +0200 (CEST) From: =?utf-8?q?Mattias_R=C3=B6nnblom?= To: Van@dpdk.org, Haaren@dpdk.org, Harry CC: , Honnappa Nagarahalli , =?utf-8?q?Morten_Br=C3=B8rup?= , nd , =?utf-8?q?Mattias_R=C3=B6nnblom?= Subject: [PATCH 4/6] service: tweak cycle statistics semantics Date: Tue, 6 Sep 2022 18:13:50 +0200 Message-ID: <20220906161352.296110-4-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220906161352.296110-1-mattias.ronnblom@ericsson.com> References: <20220708125645.3141464-2-harry.van.haaren@intel.com> <20220906161352.296110-1-mattias.ronnblom@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 481e4f57-65d7-4e34-703c-08da902329f8 X-MS-TrafficTypeDiagnostic: AM8PR07MB7601:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: F6NzZBK5YhdWdxFwKQA+/vV7fe8GGJ6iOekazLO0VWGYOXkz3N08vbFKtbgMihbYQWBOlmRk6ZrEtnB0HYIZWrUXdJjkqC/S2IPUBTX0RWoeO3iZ1uKtTbYYPCZx5hMtqZ70RnZV0FFNjAZNcaAz4pAaHb0ezyxLRnI2GRY5e+xR3nB2Zwmjauoyhb7pdxdzMvMxhXiqJj1O7vxGWr7BufjLr3LhaeQBVVQ7GAL7wqqH7LpGv8fATR/oYCLdESndyfGgoAbmTxCtRDpBmsQc0K6NWfOTF1Jw89cnRkAPNdAyoSb1Av08lJPd99aFRoYk4Lou9/BgWrbsHlh/UUtpvUiPNY7Dksvq+e3tIBf5h0E1HkB9g2Jcsaa3M1lY2UeB2G4QDFGrolqAo48edQfmBUAKzylcoRnJoiiNKvAqTFm6GG3TWiX6JfNC3kgkcOZMbufSMM/TWxpA2kebP0KMM/LUVVbIYvlHoYWhpCevNxjvtO1+5HSypnQMhBVwbEaqQbeeai00gXYb2ucZiZgljzyxyxSgPpEvxL/n/Tv/w2evYN+lk8SUKUE6F+SFszjIX8EujDzdQvtWYhS03cQDm7UEb7Znk1+LBuPooJUim78Cg/QxQnCKUFwdoA2anxTx2h6lTw31QU+6MDPs33T+wh/H3sD0npq7aGApMNuAaBgTaii91lWP62DDc3QCLCRpomZkDCuBxrcFq3npaMyBkqm8At2/lx3LLem8UisT2780YgP+9TWPIgNj2FtGK67iIdxZMZ2s+RU/93aDwz+8cA== X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230016)(4636009)(376002)(396003)(39860400002)(346002)(136003)(46966006)(36840700001)(40470700004)(26005)(6666004)(107886003)(41300700001)(478600001)(66574015)(6266002)(47076005)(83380400001)(336012)(2906002)(2616005)(5660300002)(8936002)(40480700001)(82310400005)(6916009)(40460700003)(186003)(54906003)(70206006)(4326008)(8676002)(36756003)(1076003)(82740400003)(86362001)(70586007)(316002)(7636003)(82960400001)(356005)(36860700001); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR07MB7601 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org As a part of its service function, a service usually polls some kind of source (e.g., an RX queue, a ring, an eventdev port, or a timer wheel) to retrieve one or more items of work. In low-load situations, the service framework reports a significant amount of cycles spent for all running services, despite the fact they have performed little or no actual work. The per-call cycle expenditure for an idle service (i.e., a service currently without pending jobs) is typically very low. Polling an empty ring or RX queue is inexpensive. However, since the service function call frequency on an idle or lightly loaded lcore is going to be very high indeed, the service function calls' cycles adds up to a significant amount. The only thing preventing the idle services' cycles counters to make up 100% of the available CPU cycles is the overhead of the service framework itself. If the RTE_SERVICE_ATTR_CYCLES or RTE_SERVICE_LCORE_ATTR_CYCLES are used to estimate service core load, the cores may look very busy when the system is mostly doing nothing useful at all. This patch allows for an idle service to indicate that no actual work was performed during a particular service function call (by returning -EAGAIN). In such cases the RTE_SERVICE_ATTR_CYCLES and RTE_SERVICE_LCORE_ATTR_CYCLES values are not incremented. The convention of returning -EAGAIN for idle services may in the future also be used to have the lcore enter a short sleep, or reduce its operating frequency, in case all services are currently idle. This change is backward-compatible. Signed-off-by: Mattias Rönnblom --- lib/eal/common/rte_service.c | 22 ++++++++++++++-------- lib/eal/include/rte_service_component.h | 5 +++++ 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/lib/eal/common/rte_service.c b/lib/eal/common/rte_service.c index 4cac866792..123610688c 100644 --- a/lib/eal/common/rte_service.c +++ b/lib/eal/common/rte_service.c @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -364,24 +365,29 @@ service_runner_do_callback(struct rte_service_spec_impl *s, if (service_stats_enabled(s)) { uint64_t start = rte_rdtsc(); - s->spec.callback(userdata); - uint64_t end = rte_rdtsc(); - uint64_t cycles = end - start; + int rc = s->spec.callback(userdata); /* The lcore service worker thread is the only writer, * and thus only a non-atomic load and an atomic store * is needed, and not the more expensive atomic * add. */ - __atomic_store_n(&cs->cycles, cs->cycles + cycles, - __ATOMIC_RELAXED); + + if (likely(rc != -EAGAIN)) { + uint64_t end = rte_rdtsc(); + uint64_t cycles = end - start; + + __atomic_store_n(&cs->cycles, cs->cycles + cycles, + __ATOMIC_RELAXED); + __atomic_store_n(&cs->cycles_per_service[service_idx], + cs->cycles_per_service[service_idx] + + cycles, __ATOMIC_RELAXED); + } + __atomic_store_n(&cs->calls_per_service[service_idx], cs->calls_per_service[service_idx] + 1, __ATOMIC_RELAXED); - __atomic_store_n(&cs->cycles_per_service[service_idx], - cs->cycles_per_service[service_idx] + cycles, - __ATOMIC_RELAXED); } else s->spec.callback(userdata); } diff --git a/lib/eal/include/rte_service_component.h b/lib/eal/include/rte_service_component.h index 9e66ee7e29..9be49d698a 100644 --- a/lib/eal/include/rte_service_component.h +++ b/lib/eal/include/rte_service_component.h @@ -19,6 +19,11 @@ extern "C" { /** * Signature of callback function to run a service. + * + * A service function call resulting in no actual work being + * performed, should return -EAGAIN. In that case, the (presumbly few) + * cycles spent will not be counted toward the lcore or service-level + * cycles attributes. */ typedef int32_t (*rte_service_func)(void *args);