From patchwork Thu May 28 09:13:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 70681 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 860F5A00BE; Thu, 28 May 2020 11:13:57 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 50E901D99E; Thu, 28 May 2020 11:13:50 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 56F411D540 for ; Thu, 28 May 2020 11:13:47 +0200 (CEST) IronPort-SDR: oN1IRgkED8Zb0/e8h+empIJiFEv4o3ay30jihh5fS3/1cuH1WC+qhy+CwxZtHPpeJEoDQKugn5 Shcx4fJgVy2A== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2020 02:13:46 -0700 IronPort-SDR: nD+TO//ftKJWyKcA2GB92H27ekX/suLR45Gif8UM1L9NPUR86oCvYNatsgoZZXFJwtl0YdrImC 0FaehHgGis2A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,444,1583222400"; d="scan'208";a="285106507" Received: from silpixa00399498.ir.intel.com (HELO silpixa00399498.ger.corp.intel.com) ([10.237.222.52]) by orsmga002.jf.intel.com with ESMTP; 28 May 2020 02:13:45 -0700 From: Anatoly Burakov To: dev@dpdk.org Cc: David Hunt , liang.j.ma@intel.com, reshma.pattan@intel.com Date: Thu, 28 May 2020 10:13:49 +0100 Message-Id: <38add582d2ecbbfee61624029e752d94d0574f6b.1590656906.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: Subject: [dpdk-dev] [PATCH 1/3] l3fwd-power: disable interrupts by default X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currently, interrupts are enabled in telemetry and empty poll modes, but they are not used. Switch to disabling interrupts by default, and only enable interrupts for modes that require them. Signed-off-by: Anatoly Burakov --- examples/l3fwd-power/main.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c index 9db94ce044..2cc5d7b121 100644 --- a/examples/l3fwd-power/main.c +++ b/examples/l3fwd-power/main.c @@ -256,10 +256,7 @@ static struct rte_eth_conf port_conf = { }, .txmode = { .mq_mode = ETH_MQ_TX_NONE, - }, - .intr_conf = { - .rxq = 1, - }, + } }; static struct rte_mempool * pktmbuf_pool[NB_SOCKETS]; @@ -2270,6 +2267,8 @@ main(int argc, char **argv) /* initialize all ports */ RTE_ETH_FOREACH_DEV(portid) { struct rte_eth_conf local_port_conf = port_conf; + /* not all app modes need interrupts */ + bool need_intr = app_mode == APP_MODE_LEGACY; /* skip ports that are not enabled */ if ((enabled_port_mask & (1 << portid)) == 0) { @@ -2303,7 +2302,10 @@ main(int argc, char **argv) nb_rx_queue, (unsigned)n_tx_queue ); /* If number of Rx queue is 0, no need to enable Rx interrupt */ if (nb_rx_queue == 0) - local_port_conf.intr_conf.rxq = 0; + need_intr = false; + + if (need_intr) + local_port_conf.intr_conf.rxq = 1; ret = rte_eth_dev_info_get(portid, &dev_info); if (ret != 0) From patchwork Thu May 28 09:13:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 70682 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 29827A00BE; Thu, 28 May 2020 11:14:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D3BA21DB56; Thu, 28 May 2020 11:13:51 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 0DC331D992 for ; Thu, 28 May 2020 11:13:48 +0200 (CEST) IronPort-SDR: 0Yiw5iN2VtC1SiKwsLDhFFPL6gZx82hXdmfkRjjnAF1ikrsZpZsRTrbnjfYq1nKPz7KsEXR4di CyOvZYDJaYlA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2020 02:13:48 -0700 IronPort-SDR: yR3iJ390IEQRaTcXsnutJhFLnYDPSCptGCqiv7ez8TGYTozUVZFXzKEbGolIRUXmMgQBVpzXyX SQzjX0RVcnAA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,444,1583222400"; d="scan'208";a="285106513" Received: from silpixa00399498.ir.intel.com (HELO silpixa00399498.ger.corp.intel.com) ([10.237.222.52]) by orsmga002.jf.intel.com with ESMTP; 28 May 2020 02:13:46 -0700 From: Anatoly Burakov To: dev@dpdk.org Cc: David Hunt , liang.j.ma@intel.com, reshma.pattan@intel.com Date: Thu, 28 May 2020 10:13:50 +0100 Message-Id: <9bd570760c40138fc3a4f6b2ade8f972feca7302.1590656906.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: Subject: [dpdk-dev] [PATCH 2/3] l3fwd-power: only allow supported power library envs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currently, l3fwd-power will attempt to run even if the power env is set to KVM, which is not supported. Fix this by preventing the app from initializing unless the env is set to one of the supported modes. Signed-off-by: Anatoly Burakov --- examples/l3fwd-power/main.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c index 2cc5d7b121..5cee9d5387 100644 --- a/examples/l3fwd-power/main.c +++ b/examples/l3fwd-power/main.c @@ -2043,6 +2043,7 @@ static int check_ptype(uint16_t portid) static int init_power_library(void) { + enum power_management_env env; unsigned int lcore_id; int ret = 0; @@ -2055,6 +2056,14 @@ init_power_library(void) lcore_id); return ret; } + /* we're not supporting the VM channel mode */ + env = rte_power_get_env(); + if (env != PM_ENV_ACPI_CPUFREQ && + env != PM_ENV_PSTATE_CPUFREQ) { + RTE_LOG(ERR, POWER, + "Only ACPI and PSTATE mode are supported\n"); + return -1; + } } return ret; } From patchwork Thu May 28 09:13:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 70683 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 71D28A00BE; Thu, 28 May 2020 11:14:12 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 21E071DB6D; Thu, 28 May 2020 11:13:55 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 6BD3D1D9C8 for ; Thu, 28 May 2020 11:13:50 +0200 (CEST) IronPort-SDR: pTnQCRkHlgK72LQ2StudDpuvSozZO7P/mfKP3eLGMsMG8gPwNrch/qiLxhYoqkGvWLy3qNpqIi ThBxoD7tHu4g== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2020 02:13:49 -0700 IronPort-SDR: sMUiEnl96vEmN0k3aMFYyUF0ibO4zHnSWDT0K/KDalSemt62lFNXr+r4yjEXxogHeB4JoqaCLE fS/9Oa8QI/UQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,444,1583222400"; d="scan'208";a="285106521" Received: from silpixa00399498.ir.intel.com (HELO silpixa00399498.ger.corp.intel.com) ([10.237.222.52]) by orsmga002.jf.intel.com with ESMTP; 28 May 2020 02:13:48 -0700 From: Anatoly Burakov To: dev@dpdk.org Cc: David Hunt , liang.j.ma@intel.com, reshma.pattan@intel.com Date: Thu, 28 May 2020 10:13:51 +0100 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: Subject: [dpdk-dev] [PATCH 3/3] l3fwd-power: add interrupt-only mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In addition to existing modes, add a mode which is very similar to legacy mode, but does not do frequency scaling, and thus does not depend on the power library. Signed-off-by: Anatoly Burakov --- examples/l3fwd-power/main.c | 215 +++++++++++++++++++++++++++++++++--- 1 file changed, 202 insertions(+), 13 deletions(-) diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c index 5cee9d5387..4161e01974 100644 --- a/examples/l3fwd-power/main.c +++ b/examples/l3fwd-power/main.c @@ -195,9 +195,11 @@ static int parse_ptype; /**< Parse packet type using rx callback, and */ /**< disabled by default */ enum appmode { - APP_MODE_LEGACY = 0, + APP_MODE_DEFAULT = 0, + APP_MODE_LEGACY, APP_MODE_EMPTY_POLL, - APP_MODE_TELEMETRY + APP_MODE_TELEMETRY, + APP_MODE_INTERRUPT }; enum appmode app_mode; @@ -900,6 +902,170 @@ static int event_register(struct lcore_conf *qconf) return 0; } + +/* main processing loop */ +static int main_intr_loop(__rte_unused void *dummy) +{ + struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; + unsigned int lcore_id; + uint64_t prev_tsc, diff_tsc, cur_tsc; + int i, j, nb_rx; + uint8_t queueid; + uint16_t portid; + struct lcore_conf *qconf; + struct lcore_rx_queue *rx_queue; + uint32_t lcore_rx_idle_count = 0; + uint32_t lcore_idle_hint = 0; + int intr_en = 0; + + const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) / + US_PER_S * BURST_TX_DRAIN_US; + + prev_tsc = 0; + + lcore_id = rte_lcore_id(); + qconf = &lcore_conf[lcore_id]; + + if (qconf->n_rx_queue == 0) { + RTE_LOG(INFO, L3FWD_POWER, "lcore %u has nothing to do\n", + lcore_id); + return 0; + } + + RTE_LOG(INFO, L3FWD_POWER, "entering main interrupt loop on lcore %u\n", + lcore_id); + + for (i = 0; i < qconf->n_rx_queue; i++) { + portid = qconf->rx_queue_list[i].port_id; + queueid = qconf->rx_queue_list[i].queue_id; + RTE_LOG(INFO, L3FWD_POWER, + " -- lcoreid=%u portid=%u rxqueueid=%hhu\n", + lcore_id, portid, queueid); + } + + /* add into event wait list */ + if (event_register(qconf) == 0) + intr_en = 1; + else + RTE_LOG(INFO, L3FWD_POWER, "RX interrupt won't enable.\n"); + + while (!is_done()) { + stats[lcore_id].nb_iteration_looped++; + + cur_tsc = rte_rdtsc(); + + /* + * TX burst queue drain + */ + diff_tsc = cur_tsc - prev_tsc; + if (unlikely(diff_tsc > drain_tsc)) { + for (i = 0; i < qconf->n_tx_port; ++i) { + portid = qconf->tx_port_id[i]; + rte_eth_tx_buffer_flush(portid, + qconf->tx_queue_id[portid], + qconf->tx_buffer[portid]); + } + prev_tsc = cur_tsc; + } + +start_rx: + /* + * Read packet from RX queues + */ + lcore_rx_idle_count = 0; + for (i = 0; i < qconf->n_rx_queue; ++i) { + rx_queue = &(qconf->rx_queue_list[i]); + rx_queue->idle_hint = 0; + portid = rx_queue->port_id; + queueid = rx_queue->queue_id; + + nb_rx = rte_eth_rx_burst(portid, queueid, pkts_burst, + MAX_PKT_BURST); + + stats[lcore_id].nb_rx_processed += nb_rx; + if (unlikely(nb_rx == 0)) { + /** + * no packet received from rx queue, try to + * sleep for a while forcing CPU enter deeper + * C states. + */ + rx_queue->zero_rx_packet_count++; + + if (rx_queue->zero_rx_packet_count <= + MIN_ZERO_POLL_COUNT) + continue; + + rx_queue->idle_hint = power_idle_heuristic( + rx_queue->zero_rx_packet_count); + lcore_rx_idle_count++; + } else { + rx_queue->zero_rx_packet_count = 0; + } + + /* Prefetch first packets */ + for (j = 0; j < PREFETCH_OFFSET && j < nb_rx; j++) { + rte_prefetch0(rte_pktmbuf_mtod( + pkts_burst[j], void *)); + } + + /* Prefetch and forward already prefetched packets */ + for (j = 0; j < (nb_rx - PREFETCH_OFFSET); j++) { + rte_prefetch0(rte_pktmbuf_mtod( + pkts_burst[j + PREFETCH_OFFSET], + void *)); + l3fwd_simple_forward( + pkts_burst[j], portid, qconf); + } + + /* Forward remaining prefetched packets */ + for (; j < nb_rx; j++) { + l3fwd_simple_forward( + pkts_burst[j], portid, qconf); + } + } + + if (unlikely(lcore_rx_idle_count == qconf->n_rx_queue)) { + /** + * All Rx queues empty in recent consecutive polls, + * sleep in a conservative manner, meaning sleep as + * less as possible. + */ + for (i = 1, + lcore_idle_hint = qconf->rx_queue_list[0].idle_hint; + i < qconf->n_rx_queue; ++i) { + rx_queue = &(qconf->rx_queue_list[i]); + if (rx_queue->idle_hint < lcore_idle_hint) + lcore_idle_hint = rx_queue->idle_hint; + } + + if (lcore_idle_hint < SUSPEND_THRESHOLD) + /** + * execute "pause" instruction to avoid context + * switch which generally take hundred of + * microseconds for short sleep. + */ + rte_delay_us(lcore_idle_hint); + else { + /* suspend until rx interrupt triggers */ + if (intr_en) { + turn_on_off_intr(qconf, 1); + sleep_until_rx_interrupt( + qconf->n_rx_queue); + turn_on_off_intr(qconf, 0); + /** + * start receiving packets immediately + */ + if (likely(!is_done())) + goto start_rx; + } + } + stats[lcore_id].sleep_time += lcore_idle_hint; + } + } + + return 0; +} + /* main processing loop */ static int main_telemetry_loop(__rte_unused void *dummy) @@ -1126,7 +1292,7 @@ main_empty_poll_loop(__rte_unused void *dummy) } /* main processing loop */ static int -main_loop(__rte_unused void *dummy) +main_legacy_loop(__rte_unused void *dummy) { struct rte_mbuf *pkts_burst[MAX_PKT_BURST]; unsigned lcore_id; @@ -1438,7 +1604,8 @@ print_usage(const char *prgname) " --empty-poll: enable empty poll detection" " follow (training_flag, high_threshold, med_threshold)\n" " --telemetry: enable telemetry mode, to update" - " empty polls, full polls, and core busyness to telemetry\n", + " empty polls, full polls, and core busyness to telemetry\n" + " --interrupt-only: enable interrupt-only mode\n", prgname); } @@ -1582,6 +1749,7 @@ parse_ep_config(const char *q_arg) } #define CMD_LINE_OPT_PARSE_PTYPE "parse-ptype" #define CMD_LINE_OPT_TELEMETRY "telemetry" +#define CMD_LINE_OPT_INTERRUPT_ONLY "interrupt-only" /* Parse the argument given in the command line of the application */ static int @@ -1601,6 +1769,7 @@ parse_args(int argc, char **argv) {"empty-poll", 1, 0, 0}, {CMD_LINE_OPT_PARSE_PTYPE, 0, 0, 0}, {CMD_LINE_OPT_TELEMETRY, 0, 0, 0}, + {CMD_LINE_OPT_INTERRUPT_ONLY, 0, 0, 0}, {NULL, 0, 0, 0} }; @@ -1674,8 +1843,8 @@ parse_args(int argc, char **argv) if (!strncmp(lgopts[option_index].name, "empty-poll", 10)) { - if (app_mode == APP_MODE_TELEMETRY) { - printf(" empty-poll cannot be enabled as telemetry mode is enabled\n"); + if (app_mode != APP_MODE_DEFAULT) { + printf(" empty-poll mode is mutually exclusive with other modes\n"); return -1; } app_mode = APP_MODE_EMPTY_POLL; @@ -1692,14 +1861,25 @@ parse_args(int argc, char **argv) if (!strncmp(lgopts[option_index].name, CMD_LINE_OPT_TELEMETRY, sizeof(CMD_LINE_OPT_TELEMETRY))) { - if (app_mode == APP_MODE_EMPTY_POLL) { - printf("telemetry mode cannot be enabled as empty poll mode is enabled\n"); + if (app_mode != APP_MODE_DEFAULT) { + printf(" telemetry mode is mutually exclusive with other modes\n"); return -1; } app_mode = APP_MODE_TELEMETRY; printf("telemetry mode is enabled\n"); } + if (!strncmp(lgopts[option_index].name, + CMD_LINE_OPT_INTERRUPT_ONLY, + sizeof(CMD_LINE_OPT_INTERRUPT_ONLY))) { + if (app_mode != APP_MODE_DEFAULT) { + printf(" interrupt-only mode is mutually exclusive with other modes\n"); + return -1; + } + app_mode = APP_MODE_INTERRUPT; + printf("interrupt-only mode is enabled\n"); + } + if (!strncmp(lgopts[option_index].name, "enable-jumbo", 12)) { struct option lenopts = @@ -2253,7 +2433,12 @@ main(int argc, char **argv) if (ret < 0) rte_exit(EXIT_FAILURE, "Invalid L3FWD parameters\n"); - if (app_mode != APP_MODE_TELEMETRY && init_power_library()) + if (app_mode == APP_MODE_DEFAULT) + app_mode = APP_MODE_LEGACY; + + /* only legacy and empty poll mode rely on power library */ + if ((app_mode == APP_MODE_LEGACY || app_mode == APP_MODE_EMPTY_POLL) && + init_power_library()) rte_exit(EXIT_FAILURE, "init_power_library failed\n"); if (update_lcore_params() < 0) @@ -2277,7 +2462,8 @@ main(int argc, char **argv) RTE_ETH_FOREACH_DEV(portid) { struct rte_eth_conf local_port_conf = port_conf; /* not all app modes need interrupts */ - bool need_intr = app_mode == APP_MODE_LEGACY; + bool need_intr = app_mode == APP_MODE_LEGACY || + app_mode == APP_MODE_INTERRUPT; /* skip ports that are not enabled */ if ((enabled_port_mask & (1 << portid)) == 0) { @@ -2526,12 +2712,12 @@ main(int argc, char **argv) /* launch per-lcore init on every lcore */ if (app_mode == APP_MODE_LEGACY) { - rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER); + rte_eal_mp_remote_launch(main_legacy_loop, NULL, CALL_MASTER); } else if (app_mode == APP_MODE_EMPTY_POLL) { empty_poll_stop = false; rte_eal_mp_remote_launch(main_empty_poll_loop, NULL, SKIP_MASTER); - } else { + } else if (app_mode == APP_MODE_TELEMETRY) { unsigned int i; /* Init metrics library */ @@ -2555,6 +2741,8 @@ main(int argc, char **argv) "Returns global power stats. Parameters: None"); rte_eal_mp_remote_launch(main_telemetry_loop, NULL, SKIP_MASTER); + } else if (app_mode == APP_MODE_INTERRUPT) { + rte_eal_mp_remote_launch(main_intr_loop, NULL, CALL_MASTER); } if (app_mode == APP_MODE_EMPTY_POLL || app_mode == APP_MODE_TELEMETRY) @@ -2577,7 +2765,8 @@ main(int argc, char **argv) if (app_mode == APP_MODE_EMPTY_POLL) rte_power_empty_poll_stat_free(); - if (app_mode != APP_MODE_TELEMETRY && deinit_power_library()) + if ((app_mode == APP_MODE_LEGACY || app_mode == APP_MODE_EMPTY_POLL) && + deinit_power_library()) rte_exit(EXIT_FAILURE, "deinit_power_library failed\n"); if (rte_eal_cleanup() < 0)