From patchwork Mon May 29 11:21:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fengnan Chang X-Patchwork-Id: 127664 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 40E9742BD2; Mon, 29 May 2023 13:21:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 314814111C; Mon, 29 May 2023 13:21:44 +0200 (CEST) Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by mails.dpdk.org (Postfix) with ESMTP id F0AEC410D7 for ; Mon, 29 May 2023 13:21:42 +0200 (CEST) Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-52c84543902so318081a12.0 for ; Mon, 29 May 2023 04:21:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1685359302; x=1687951302; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Vt+oTvv2a0H871mrh7/u5ANIk6vFQStm8aii31bNKD8=; b=MDlKTyPtm+IEuBEh3lI9rUVxC+B2qfZUXI/TjkU7nJOvKxH3kZHAhF4xy7+aHMxAEW igQt+wMsChaea25G5XxM52QdJco+CdJ5IP40n9TRXHhiM5B/l1r9R34YcwB3gtttu+iJ j+0NCs03v6ZOyf9Oh8uXSg+lj20AzYqgKCqR7s0WMW4rdIXMeD/R+WDPtonAbHw//vL4 sstpNhwS58MJRuuKHnHpxgJitWqy32G28mQY/OpTLGaOO7MaD1s0zfR9Mc1xWIhvPojK lMnK2vdovOVt4x/fDFX1jrDweh9Y7pJRaseH35JpVfY6sdA1/UCml04WCc5MAT2yeBqw RYOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685359302; x=1687951302; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Vt+oTvv2a0H871mrh7/u5ANIk6vFQStm8aii31bNKD8=; b=kOHmo1L7vwfdf38l7JlKM2/Yo5YuixpD6hwYeE3qtZjt5Em2qt3dkc3ns6Of1IhuPp yz2awBabGM/KbvBbcsbIJ0m6ecvYdPMfd8phEbsuF2mtatkZJ0zEhbRQuFCI6ck15PGI xZIqeOomnIKhAWzw5H7/J5ITEYXSWnqKlpa7IEunD7gxMF4hjbtfKW+7EDE7RY+dK4bt 8fSCxZGWvANCEnFP5VHFM3evFMyT5dZtEv5MQyzhaybCx+jE+Xyd1E6SVlLQTeJ330el r5T5tBhPUBBYd4++iJacWZ9tKjrZp2EOcqI/wr8i1rwgYqcbYW8w60ubLMXZ+0hayYpY QkMQ== X-Gm-Message-State: AC+VfDws9PAHowIXXHpIhj15gRZUcRJ/ehtdUTdvOy9R3MUEC1ZB3mMC +ESWPLnXTZrTmnOe5fa+eBu2hw== X-Google-Smtp-Source: ACHHUZ5am2UopRyqkTIGwJRpRmGDvae+9iVDXGJCpxa+BXisyc8ckxjl5vprHjUzzTln4ztWes8mBg== X-Received: by 2002:a17:903:41d0:b0:1ac:775b:3e0a with SMTP id u16-20020a17090341d000b001ac775b3e0amr12034509ple.5.1685359302180; Mon, 29 May 2023 04:21:42 -0700 (PDT) Received: from HTW5T2C6VL.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id g23-20020a1709029f9700b001aaf2e8b1eesm7967074plq.248.2023.05.29.04.21.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 May 2023 04:21:41 -0700 (PDT) From: Fengnan Chang To: anatoly.burakov@intel.com, dev@dpdk.org Cc: Fengnan Chang , Lin Li Subject: [PATCH v4] eal: fix eal init may failed when too much continuous memsegs under legacy mode Date: Mon, 29 May 2023 19:21:30 +0800 Message-Id: <20230529112130.11198-1-changfengnan@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Under legacy mode, if the number of continuous memsegs greater than RTE_MAX_MEMSEG_PER_LIST, eal init will failed even though another memseg list is empty, because only one memseg list used to check in remap_needed_hugepages. Fix this by make remap_segment return how many segments mapped, remap_segment try to map most contiguous segments it can, if exceed it's capbility, remap_needed_hugepages will continue to map other left pages. For example: hugepage configure: cat /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages 10241 10239 startup log: EAL: Detected memory type: socket_id:0 hugepage_sz:2097152 EAL: Detected memory type: socket_id:1 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152 EAL: Requesting 13370 pages of size 2MB from socket 0 EAL: Requesting 7110 pages of size 2MB from socket 1 EAL: Attempting to map 14220M on socket 1 EAL: Allocated 14220M on socket 1 EAL: Attempting to map 26740M on socket 0 EAL: Could not find space for memseg. Please increase 32768 and/or 65536 in configuration. EAL: Couldn't remap hugepage files into memseg lists EAL: FATAL: Cannot init memory EAL: Cannot init memory Signed-off-by: Fengnan Chang Signed-off-by: Lin Li Signed-off-by: Burakov Anatoly Reviewed-by: Anatoly Burakov --- lib/eal/linux/eal_memory.c | 51 +++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 15 deletions(-) diff --git a/lib/eal/linux/eal_memory.c b/lib/eal/linux/eal_memory.c index 60fc8cc6ca..0876974631 100644 --- a/lib/eal/linux/eal_memory.c +++ b/lib/eal/linux/eal_memory.c @@ -681,6 +681,7 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) /* find free space in memseg lists */ for (msl_idx = 0; msl_idx < RTE_MAX_MEMSEG_LISTS; msl_idx++) { + int free_len; bool empty; msl = &mcfg->memsegs[msl_idx]; arr = &msl->memseg_arr; @@ -692,18 +693,26 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) /* leave space for a hole if array is not empty */ empty = arr->count == 0; - ms_idx = rte_fbarray_find_next_n_free(arr, 0, - seg_len + (empty ? 0 : 1)); - - /* memseg list is full? */ + /* find start of the biggest contiguous block and its size */ + ms_idx = rte_fbarray_find_biggest_free(arr, 0); if (ms_idx < 0) continue; - + /* hole is 1 segment long, so at least two segments long. */ + free_len = rte_fbarray_find_contig_free(arr, ms_idx); + if (free_len < 2) + continue; /* leave some space between memsegs, they are not IOVA * contiguous, so they shouldn't be VA contiguous either. */ - if (!empty) + if (!empty) { ms_idx++; + free_len--; + } + + /* we might not get all of the space we wanted */ + free_len = RTE_MIN(seg_len, free_len); + seg_end = seg_start + free_len; + seg_len = seg_end - seg_start; break; } if (msl_idx == RTE_MAX_MEMSEG_LISTS) { @@ -787,7 +796,7 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) } RTE_LOG(DEBUG, EAL, "Allocated %" PRIu64 "M on socket %i\n", (seg_len * page_sz) >> 20, socket_id); - return 0; + return seg_len; } static uint64_t @@ -1022,10 +1031,16 @@ remap_needed_hugepages(struct hugepage_file *hugepages, int n_pages) if (new_memseg) { /* if this isn't the first time, remap segment */ if (cur_page != 0) { - ret = remap_segment(hugepages, seg_start_page, - cur_page); - if (ret != 0) - return -1; + int n_remapped = 0; + int n_needed = cur_page - seg_start_page; + while (n_remapped < n_needed) { + ret = remap_segment(hugepages, seg_start_page, + cur_page); + if (ret < 0) + return -1; + n_remapped += ret; + seg_start_page += ret; + } } /* remember where we started */ seg_start_page = cur_page; @@ -1034,10 +1049,16 @@ remap_needed_hugepages(struct hugepage_file *hugepages, int n_pages) } /* we were stopped, but we didn't remap the last segment, do it now */ if (cur_page != 0) { - ret = remap_segment(hugepages, seg_start_page, - cur_page); - if (ret != 0) - return -1; + int n_remapped = 0; + int n_needed = cur_page - seg_start_page; + while (n_remapped < n_needed) { + ret = remap_segment(hugepages, seg_start_page, + cur_page); + if (ret < 0) + return -1; + n_remapped += ret; + seg_start_page += ret; + } } return 0; }