| Message ID | 1594993865-396296-1-git-send-email-suanmingm@mellanox.com (mailing list archive) |
|---|---|
| Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 94BF9A053B; Fri, 17 Jul 2020 15:51:44 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EA96C1C132; Fri, 17 Jul 2020 15:51:11 +0200 (CEST) Received: from git-send-mailer.rdmz.labs.mlnx (unknown [37.142.13.130]) by dpdk.org (Postfix) with ESMTP id E6F041C12B for <dev@dpdk.org>; Fri, 17 Jul 2020 15:51:10 +0200 (CEST) From: Suanming Mou <suanmingm@mellanox.com> To: viacheslavo@mellanox.com, matan@mellanox.com Cc: orika@mellanox.com, rasland@mellanox.com, dev@dpdk.org Date: Fri, 17 Jul 2020 21:50:58 +0800 Message-Id: <1594993865-396296-1-git-send-email-suanmingm@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1594785603-152773-1-git-send-email-suanmingm@mellanox.com> References: <1594785603-152773-1-git-send-email-suanmingm@mellanox.com> Subject: [dpdk-dev] [PATCH v3 0/7] net/mlx5: add sys_mem_en devarg X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
| Series |
net/mlx5: add sys_mem_en devarg
|
|
Message
Suanming Mou
July 17, 2020, 1:50 p.m. UTC
Currently, for MLX5 PMD, once millions of flows created, the memory consumption of the flows are also very huge. For the system with limited memory, it means the system need to reserve most of the memory as huge page memory to serve the flows in advance. And other normal applications will have no chance to use this reserved memory any more. While most of the time, the system will not have lots of flows, the reserved huge page memory becomes a bit waste of memory at most of the time. By the new sys_mem_en devarg, once set it to be true, it allows the PMD allocate the memory from system by default with the new add mlx5 memory management functions. Only once the MLX5_MEM_RTE flag is set, the memory will be allocate from rte, otherwise, it allocates memory from system. So in this case, the system with limited memory no need to reserve most of the memory for hugepage. Only some needed memory for datapath objects will be enough to allocated with explicitly flag. Other memory will be allocated from system. For system with enough memory, no need to care about the devarg, the memory will always be from rte hugepage. One restriction is that for DPDK application with multiple PCI devices, if the sys_mem_en devargs are different between the devices, the sys_mem_en only gets the value from the first device devargs, and print out a message to warn that. --- v3: - Rebase on top of latest code. v2: - Add memory function call statistic. - Change msl to aotmic. --- Suanming Mou (7): common/mlx5: add mlx5 memory management functions net/mlx5: add allocate memory from system devarg net/mlx5: convert control path memory to unified malloc common/mlx5: convert control path memory to unified malloc common/mlx5: convert data path objects to unified malloc net/mlx5: convert configuration objects to unified malloc net/mlx5: convert Rx/Tx queue objects to unified malloc doc/guides/nics/mlx5.rst | 7 + drivers/common/mlx5/Makefile | 1 + drivers/common/mlx5/linux/mlx5_glue.c | 13 +- drivers/common/mlx5/linux/mlx5_nl.c | 5 +- drivers/common/mlx5/meson.build | 1 + drivers/common/mlx5/mlx5_common.c | 10 +- drivers/common/mlx5/mlx5_common_mp.c | 7 +- drivers/common/mlx5/mlx5_common_mr.c | 31 ++- drivers/common/mlx5/mlx5_devx_cmds.c | 82 ++++--- drivers/common/mlx5/mlx5_malloc.c | 306 ++++++++++++++++++++++++ drivers/common/mlx5/mlx5_malloc.h | 99 ++++++++ drivers/common/mlx5/rte_common_mlx5_version.map | 6 + drivers/net/mlx5/linux/mlx5_ethdev_os.c | 8 +- drivers/net/mlx5/linux/mlx5_os.c | 28 ++- drivers/net/mlx5/mlx5.c | 108 +++++---- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_ethdev.c | 15 +- drivers/net/mlx5/mlx5_flow.c | 45 ++-- drivers/net/mlx5/mlx5_flow_dv.c | 46 ++-- drivers/net/mlx5/mlx5_flow_meter.c | 11 +- drivers/net/mlx5/mlx5_flow_verbs.c | 8 +- drivers/net/mlx5/mlx5_mp.c | 3 +- drivers/net/mlx5/mlx5_rss.c | 13 +- drivers/net/mlx5/mlx5_rxq.c | 74 +++--- drivers/net/mlx5/mlx5_txpp.c | 30 +-- drivers/net/mlx5/mlx5_txq.c | 82 +++---- drivers/net/mlx5/mlx5_utils.c | 60 +++-- drivers/net/mlx5/mlx5_utils.h | 2 +- drivers/net/mlx5/mlx5_vlan.c | 8 +- 29 files changed, 797 insertions(+), 313 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_malloc.c create mode 100644 drivers/common/mlx5/mlx5_malloc.h
Comments
Hi > -----Original Message----- > From: Suanming Mou <suanmingm@mellanox.com> > Sent: Friday, July 17, 2020 4:51 PM > To: Slava Ovsiienko <viacheslavo@mellanox.com>; Matan Azrad > <matan@mellanox.com> > Cc: Ori Kam <orika@mellanox.com>; Raslan Darawsheh > <rasland@mellanox.com>; dev@dpdk.org > Subject: [PATCH v3 0/7] net/mlx5: add sys_mem_en devarg > > Currently, for MLX5 PMD, once millions of flows created, the memory > consumption of the flows are also very huge. For the system with limited > memory, it means the system need to reserve most of the memory as huge > page memory to serve the flows in advance. And other normal applications > will have no chance to use this reserved memory any more. While most of > the time, the system will not have lots of flows, the reserved huge page > memory becomes a bit waste of memory at most of the time. > > By the new sys_mem_en devarg, once set it to be true, it allows the PMD > allocate the memory from system by default with the new add mlx5 memory > management functions. Only once the MLX5_MEM_RTE flag is set, the > memory > will be allocate from rte, otherwise, it allocates memory from system. > > So in this case, the system with limited memory no need to reserve most > of the memory for hugepage. Only some needed memory for datapath > objects > will be enough to allocated with explicitly flag. Other memory will be > allocated from system. For system with enough memory, no need to care > about the devarg, the memory will always be from rte hugepage. > > One restriction is that for DPDK application with multiple PCI devices, > if the sys_mem_en devargs are different between the devices, the > sys_mem_en only gets the value from the first device devargs, and print > out a message to warn that. > > --- > > v3: > - Rebase on top of latest code. > > v2: > - Add memory function call statistic. > - Change msl to aotmic. > > --- > > Suanming Mou (7): > common/mlx5: add mlx5 memory management functions > net/mlx5: add allocate memory from system devarg > net/mlx5: convert control path memory to unified malloc > common/mlx5: convert control path memory to unified malloc > common/mlx5: convert data path objects to unified malloc > net/mlx5: convert configuration objects to unified malloc > net/mlx5: convert Rx/Tx queue objects to unified malloc > > doc/guides/nics/mlx5.rst | 7 + > drivers/common/mlx5/Makefile | 1 + > drivers/common/mlx5/linux/mlx5_glue.c | 13 +- > drivers/common/mlx5/linux/mlx5_nl.c | 5 +- > drivers/common/mlx5/meson.build | 1 + > drivers/common/mlx5/mlx5_common.c | 10 +- > drivers/common/mlx5/mlx5_common_mp.c | 7 +- > drivers/common/mlx5/mlx5_common_mr.c | 31 ++- > drivers/common/mlx5/mlx5_devx_cmds.c | 82 ++++--- > drivers/common/mlx5/mlx5_malloc.c | 306 > ++++++++++++++++++++++++ > drivers/common/mlx5/mlx5_malloc.h | 99 ++++++++ > drivers/common/mlx5/rte_common_mlx5_version.map | 6 + > drivers/net/mlx5/linux/mlx5_ethdev_os.c | 8 +- > drivers/net/mlx5/linux/mlx5_os.c | 28 ++- > drivers/net/mlx5/mlx5.c | 108 +++++---- > drivers/net/mlx5/mlx5.h | 1 + > drivers/net/mlx5/mlx5_ethdev.c | 15 +- > drivers/net/mlx5/mlx5_flow.c | 45 ++-- > drivers/net/mlx5/mlx5_flow_dv.c | 46 ++-- > drivers/net/mlx5/mlx5_flow_meter.c | 11 +- > drivers/net/mlx5/mlx5_flow_verbs.c | 8 +- > drivers/net/mlx5/mlx5_mp.c | 3 +- > drivers/net/mlx5/mlx5_rss.c | 13 +- > drivers/net/mlx5/mlx5_rxq.c | 74 +++--- > drivers/net/mlx5/mlx5_txpp.c | 30 +-- > drivers/net/mlx5/mlx5_txq.c | 82 +++---- > drivers/net/mlx5/mlx5_utils.c | 60 +++-- > drivers/net/mlx5/mlx5_utils.h | 2 +- > drivers/net/mlx5/mlx5_vlan.c | 8 +- > 29 files changed, 797 insertions(+), 313 deletions(-) > create mode 100644 drivers/common/mlx5/mlx5_malloc.c > create mode 100644 drivers/common/mlx5/mlx5_malloc.h > > -- > 1.8.3.1 Series applied to next-net-mlx, Kindest regards, Raslan Darawsheh