From patchwork Fri May 26 03:41:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fengnan Chang X-Patchwork-Id: 127532 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2B17442BA3; Fri, 26 May 2023 05:41:37 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1F04541156; Fri, 26 May 2023 05:41:37 +0200 (CEST) Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by mails.dpdk.org (Postfix) with ESMTP id 9C0DF40A87 for ; Fri, 26 May 2023 05:41:35 +0200 (CEST) Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-25377d67da9so29909a91.0 for ; Thu, 25 May 2023 20:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1685072494; x=1687664494; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=8CNeh1O41iuIuShEUskJmJ3jiDd8iw2K4V8VGt0btEc=; b=ZalYX0hvhr14dfTLH2s86fn6HpxgjPllM58npsE4zBoambxDirUxI76VYxAl7YYD+a c6xEirAMGXdITTera00LQ/xnpA1OqXuQq6WDtJuLV8mU6cA2NmHfTlu1bS5LO+6uwioJ 2I9/COrQDRTs1vOm4nGBZljwHhrKo4LU9m2f1FP5PH0DuMe5qa5XuHHr1Q6XcpT5PoDF HKsj6jv0Zi2Tw529SsXT4/HTXXU7h3y8RWV9Z/+diEUO0SMMiK0iryDc0g1APTro7Kvl btnXoBH4fRWWVOgJwFSGrlcrSFiSxf/AL4Cwu4IUNxU2d9SjOAwoNQ9nkfT6FNek/GIS WtoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685072494; x=1687664494; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8CNeh1O41iuIuShEUskJmJ3jiDd8iw2K4V8VGt0btEc=; b=dR+ezRkBoqfQUh9Q+0xgbvvfGKPLbzSsESOZUXcbJV0A5YQ9YqKb45ujtV59Pdhd6d LLjESC3pvZzLzH+T6CAuyDgrzrRimqx0W7AthnKo7RqgsuofpVLXtXEy4T1Xkco0xpDH 4vkQu3pyjuJyYliqHuX9eQ8tX/1ZqrknHzlFSkkuP9pj7qEDsm82rRJPjqdUfZaqH8UU 8Foq0bE4G2IGQJ+vBO8MeRNr230jNyRK/qPTbhakQoEII6+WMuRePNRnHnKUoeKEJol8 03kkF8KNqduzzqt+DKGKaqyHVfvWY3PqjsTX1Yriy/pAZ3z8CQTFLkQTo1zc824RcXx5 95yw== X-Gm-Message-State: AC+VfDyZtfwi+E+LkeAxWCLpLaLdioytbj4Z9/DHjKLDOA3ICgyA46Iw XXVu0EZ5cD2vnL9e5vt+hl12FI+uXv3rjOgvpR0= X-Google-Smtp-Source: ACHHUZ50Og5FJSu3aecz9U45WZ6DHyTFBAYfZQIAy2gEAZTK56vt57uj1/chrbXny3omlq+lbBhyVA== X-Received: by 2002:a17:90b:3ec9:b0:252:b150:6ce with SMTP id rm9-20020a17090b3ec900b00252b15006cemr1137486pjb.1.1685072494234; Thu, 25 May 2023 20:41:34 -0700 (PDT) Received: from HTW5T2C6VL.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id l1-20020a17090aaa8100b002529f2e570esm1891392pjq.28.2023.05.25.20.41.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 25 May 2023 20:41:33 -0700 (PDT) From: Fengnan Chang To: dev@dpdk.org, anatoly.burakov@intel.com Cc: Fengnan Chang , Lin Li Subject: [PATCH v3] eal: fix eal init may failed when too much continuous memsegs under legacy mode Date: Fri, 26 May 2023 11:41:27 +0800 Message-Id: <20230526034127.18231-1-changfengnan@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Under legacy mode, if the number of continuous memsegs greater than RTE_MAX_MEMSEG_PER_LIST, eal init will failed even though another memseg list is empty, because only one memseg list used to check in remap_needed_hugepages. Fix this by make remap_segment return how many segments mapped, remap_segment try to map most contiguous segments it can, if exceed it's capbility, remap_needed_hugepages will continue to map other left pages. For example: hugepage configure: cat /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages 10241 10239 startup log: EAL: Detected memory type: socket_id:0 hugepage_sz:2097152 EAL: Detected memory type: socket_id:1 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152 EAL: Requesting 13370 pages of size 2MB from socket 0 EAL: Requesting 7110 pages of size 2MB from socket 1 EAL: Attempting to map 14220M on socket 1 EAL: Allocated 14220M on socket 1 EAL: Attempting to map 26740M on socket 0 EAL: Could not find space for memseg. Please increase 32768 and/or 65536 in configuration. EAL: Couldn't remap hugepage files into memseg lists EAL: FATAL: Cannot init memory EAL: Cannot init memory Signed-off-by: Fengnan Chang Signed-off-by: Lin Li Signed-off-by: Burakov Anatoly Reviewed-by: Anatoly Burakov --- lib/eal/linux/eal_memory.c | 55 +++++++++++++++++++++++++------------- 1 file changed, 36 insertions(+), 19 deletions(-) diff --git a/lib/eal/linux/eal_memory.c b/lib/eal/linux/eal_memory.c index 60fc8cc6ca..085defdee5 100644 --- a/lib/eal/linux/eal_memory.c +++ b/lib/eal/linux/eal_memory.c @@ -681,6 +681,7 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) /* find free space in memseg lists */ for (msl_idx = 0; msl_idx < RTE_MAX_MEMSEG_LISTS; msl_idx++) { + int free_len; bool empty; msl = &mcfg->memsegs[msl_idx]; arr = &msl->memseg_arr; @@ -692,24 +693,28 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) /* leave space for a hole if array is not empty */ empty = arr->count == 0; - ms_idx = rte_fbarray_find_next_n_free(arr, 0, - seg_len + (empty ? 0 : 1)); - - /* memseg list is full? */ - if (ms_idx < 0) + /* find start of the biggest contiguous block and its size */ + ms_idx = rte_fbarray_find_biggest_free(arr, 0); + free_len = rte_fbarray_find_contig_free(arr, ms_idx); + if (free_len < 0) continue; - /* leave some space between memsegs, they are not IOVA * contiguous, so they shouldn't be VA contiguous either. */ - if (!empty) + if (!empty) { ms_idx++; + free_len--; + } + printf("free len %d seg_len %d ms_idx %d\n", free_len, seg_len, ms_idx); + /* we might not get all of the space we wanted */ + free_len = RTE_MIN(seg_len, free_len); + seg_end = seg_start + free_len; + seg_len = seg_end - seg_start; break; } if (msl_idx == RTE_MAX_MEMSEG_LISTS) { - RTE_LOG(ERR, EAL, "Could not find space for memseg. Please increase %s and/or %s in configuration.\n", - RTE_STR(RTE_MAX_MEMSEG_PER_TYPE), - RTE_STR(RTE_MAX_MEM_MB_PER_TYPE)); + RTE_LOG(ERR, EAL, "Could not find space for memseg. Please increase RTE_MAX_MEMSEG_PER_LIST " + "RTE_MAX_MEMSEG_PER_TYPE and/or RTE_MAX_MEM_MB_PER_TYPE in configuration.\n"); return -1; } @@ -787,7 +792,7 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) } RTE_LOG(DEBUG, EAL, "Allocated %" PRIu64 "M on socket %i\n", (seg_len * page_sz) >> 20, socket_id); - return 0; + return seg_len; } static uint64_t @@ -1022,10 +1027,16 @@ remap_needed_hugepages(struct hugepage_file *hugepages, int n_pages) if (new_memseg) { /* if this isn't the first time, remap segment */ if (cur_page != 0) { - ret = remap_segment(hugepages, seg_start_page, - cur_page); - if (ret != 0) - return -1; + int n_remapped = 0; + int n_needed = cur_page - seg_start_page; + while (n_remapped < n_needed) { + ret = remap_segment(hugepages, seg_start_page, + cur_page); + if (ret < 0) + return -1; + n_remapped += ret; + seg_start_page += ret; + } } /* remember where we started */ seg_start_page = cur_page; @@ -1034,10 +1045,16 @@ remap_needed_hugepages(struct hugepage_file *hugepages, int n_pages) } /* we were stopped, but we didn't remap the last segment, do it now */ if (cur_page != 0) { - ret = remap_segment(hugepages, seg_start_page, - cur_page); - if (ret != 0) - return -1; + int n_remapped = 0; + int n_needed = cur_page - seg_start_page; + while (n_remapped < n_needed) { + ret = remap_segment(hugepages, seg_start_page, + cur_page); + if (ret < 0) + return -1; + n_remapped += ret; + seg_start_page += ret; + } } return 0; }