From patchwork Tue Jul 10 16:04:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42729 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2CF607EC7; Tue, 10 Jul 2018 18:05:01 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id 1BBAD5F44 for ; Tue, 10 Jul 2018 18:04:58 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id z6-v6so13958665wma.0 for ; Tue, 10 Jul 2018 09:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=QLWEKxuqhF2sx7AXNTonfLHxyzB3mWcORADPDgYW6JM=; b=Q51q8xNSW1luCQe+wC3oEhb3fb3m/xKmIb4IrwCPVFedzhG5mfprTjZ1/NuwArf9Gi T/h44OjHBYcDgC5htyJTx1z1X7LvpmRh6Afm4Y9aXWEgrqN2XbjYwX9phbxhMUFXM3GU NMiVX4GiA9UeAumgfREVzOXWIXnu1wiQp+bt5oY2RwrFfaNmwLkPYpQ65qT4R77n++g0 FEuW5gMtxM0SsRCgQOuxi9Y7gZlCWimMyJ+3Gd4XcaU/eYWKNSsePwOsLmrigH64D7ei Mv8ZjEtyQONTspJiclz+vplg9PPnOryl0stbhyb8gUut5B2+SwyaiI4XbQ8oZ5i4AGnR vx2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=QLWEKxuqhF2sx7AXNTonfLHxyzB3mWcORADPDgYW6JM=; b=T0V9pNnrIn13TI1+eFxid3pOL5osyG6tYDeoJ7wBywSsCFx7QTCxg6pwXNTWuh6f88 EHOem5iKGCDlgd3e6rlZ/54D1dtHAor4WP8hw99g9JV3AFe1irrpTEipiUmNOEv8WYSd lREyzulHeRfHOOjAfJlv3P1anYT8Im8O0vnpxy42RzkTXDMHpL3wIlqajs5WswowjZD4 By+zcdzNPbftepnoBBDAfCiuNyEy31Dberj59ceKalz665i1AGJG/ICKgpWph6lc6JtO voi/v+aONh3U1vkyuflbQI8BBCTKcgALRYnWHjP6ymKK/Je+4STMvdcYXNIrwjcpE+Ct 4epA== X-Gm-Message-State: AOUpUlEri8m1fDMO4mSND9KAjrlydua0YZpkJGGd7Hx4bR4btaoifMns pspPjcA/f4DKn7L4Q44ZhVersuX/ X-Google-Smtp-Source: AAOMgpdwarEuLx/YVP355apoLiCApS81jh9vedX2pCcnr730EXJ+HOEaiIPbZEDyTcexu4YqZJC6VQ== X-Received: by 2002:a1c:b143:: with SMTP id a64-v6mr8246023wmf.114.1531238698700; Tue, 10 Jul 2018 09:04:58 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 38-v6sm9879437wrm.15.2018.07.10.09.04.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:04:57 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:42 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-2-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 01/10] net/mlx5: rename confusing object in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" There are several attribute objects in this function: - IB device attributes (struct ibv_device_attr_ex device_attr). - Direct Verbs attributes (struct mlx5dv_context attrs_out). - Port attributes (struct ibv_port_attr). - IB device attributes again (struct ibv_device_attr_ex device_attr_ex). "attrs_out" is both odd and initialized using a nonstandard syntax. Rename it "dv_attr" for consistency. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Fixed ctx -> attr_ctx in mlx5_pci_probe(). --- drivers/net/mlx5/mlx5.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index d081bdd05..22cbce8d5 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,6 +654,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; struct ibv_context *attr_ctx = NULL; struct ibv_device_attr_ex device_attr; @@ -670,7 +671,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; int i; - struct mlx5dv_context attrs_out = {0}; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif @@ -736,21 +736,21 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ibv_dev = list[i]; DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif /* * Multi-packet send is supported by ConnectX-4 Lx PF as well * as all ConnectX-5 devices. */ #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &attrs_out); - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { + mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); mps = MLX5_MPW_ENHANCED; } else { @@ -762,14 +762,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mps = MLX5_MPW_DISABLED; } #ifdef HAVE_IBV_MLX5_MOD_SWP - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_SWP) - swp = attrs_out.sw_parsing_caps.sw_parsing_offloads; + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) + swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = - attrs_out.striding_rq_caps; + dv_attr.striding_rq_caps; DRV_LOG(DEBUG, "\tmin_single_stride_log_num_of_bytes: %d", mprq_caps.min_single_stride_log_num_of_bytes); @@ -794,15 +794,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } #endif if (RTE_CACHE_LINE_SIZE == 128 && - !(attrs_out.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) + !(dv_attr.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) cqe_comp = 0; else cqe_comp = 1; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { - tunnel_en = ((attrs_out.tunnel_offloads_caps & + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { + tunnel_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_VXLAN) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GRE)); } DRV_LOG(DEBUG, "tunnel offloading is %ssupported", @@ -812,9 +812,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT - mpls_en = ((attrs_out.tunnel_offloads_caps & + mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP)); DRV_LOG(DEBUG, "MPLS over GRE/UDP tunnel offloading is %ssupported", mpls_en ? "" : "not "); From patchwork Tue Jul 10 16:04:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42730 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id F1B9E1B3A2; Tue, 10 Jul 2018 18:05:03 +0200 (CEST) Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by dpdk.org (Postfix) with ESMTP id 3F2ED7EE3 for ; Tue, 10 Jul 2018 18:05:01 +0200 (CEST) Received: by mail-wm0-f41.google.com with SMTP id w16-v6so25075482wmc.2 for ; Tue, 10 Jul 2018 09:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=LZ7BxY/4owWBpMHTwS+IFzwvWnorKYfJ62epOi5lr6Y=; b=vdKqf/UEAnomKlAXi/zsTKxsojUMzP4zBX3BoAZJano0FWaEVeOCynfQEux/nR6HGz B2PduXEmlbn3qUtcZwO+EpcKMZgAX/BYx+M/g0MNbfmy1wAbVu+JzjrdeoUinMrO5Gun n4kMgkhxnbMP6+dptxPyVOIN0ODper6CdIsnNs6upMoJsH/sdijUtdUGYX+wvJxcI+kz zZfA7EVpKxGVLiaxmfoBQSAZLzWtQNQkH8kV1QLOGJ9HvGFAF1hSxjj8DdFp+2AAv6xv C4SMUFPlpHi5o41ZikzJcjsMuJH2ctvWfFMFysswP+71klpgBGvL3+q/bZkRRWmBnI/D iUSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=LZ7BxY/4owWBpMHTwS+IFzwvWnorKYfJ62epOi5lr6Y=; b=bZg7o8kM1xuSa/FIzV66s7GNiq6uz6zb1KfOg6g0od5ej+pIWqgqicnhDjffoAkl1s CLmQB/Cm4ajEf4vYB0O33s2of5ei9nY57PU1bTNZCNd3nFbd4AjV8v9R8htsKHKg/b/F jLpvl1eapgPEnQqo6wSaTrrfawXGK6k7gVxf+CUYKHbWyVw8No9ns0f8JRNvN8tf5syf NpBKoWKqTxJjpChi5UgLCwOKtvcJuLkhITZbNA01mJW8I/aNCQJBf9LPejugQdEUikzR b+3GOi0Jvdl19AHipLY5VYtNTUTerNLG39YxoPh94CuXXXvyMrPDcLQdl1HrIZVBvBKo SZaA== X-Gm-Message-State: APt69E33T1+Y9KIQvihWIVlo7uz958+2qdB5zG6k5bkqR0jFpXAIKEx9 tdrwfjpSk1calbnh+YDvA4GIyFfc X-Google-Smtp-Source: AAOMgpfqome6AfgFpUR3Qjc2r6jfl/PffQPx94zxkDud74vRVS6s+ly43gfhipAbkXnD9yXVb82KZg== X-Received: by 2002:a1c:ea9b:: with SMTP id g27-v6mr14403497wmi.152.1531238700693; Tue, 10 Jul 2018 09:05:00 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id k188-v6sm27771484wmd.47.2018.07.10.09.04.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:04:59 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:44 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-3-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 02/10] net/mlx5: remove redundant objects in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch gets rid of redundant calls to open the device and query its attributes in order to simplify the code. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Minor indent fix on existing code. --- drivers/net/mlx5/mlx5.c | 64 +++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 22cbce8d5..4e7f29f5b 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,10 +654,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct ibv_context *ctx = NULL; + struct ibv_device_attr_ex attr; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; - struct ibv_context *attr_ctx = NULL; - struct ibv_device_attr_ex device_attr; unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; @@ -714,12 +714,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || (pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - attr_ctx = mlx5_glue->open_device(list[i]); + ctx = mlx5_glue->open_device(list[i]); rte_errno = errno; err = rte_errno; break; } - if (attr_ctx == NULL) { + if (ctx == NULL) { switch (err) { case 0: DRV_LOG(ERR, @@ -748,7 +748,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + mlx5_glue->dv_query_device(ctx, &dv_attr); if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); @@ -822,23 +822,20 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif - err = mlx5_glue->query_device_ex(attr_ctx, NULL, &device_attr); + err = mlx5_glue->query_device_ex(ctx, NULL, &attr); if (err) { DEBUG("ibv_query_device_ex() failed"); goto error; } - DRV_LOG(INFO, "%u port(s) detected", - device_attr.orig_attr.phys_port_cnt); - for (i = 0; i < device_attr.orig_attr.phys_port_cnt; i++) { + DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); + for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { char name[RTE_ETH_NAME_MAX_LEN]; int len; uint32_t port = i + 1; /* ports are indexed from one */ - struct ibv_context *ctx = NULL; struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct priv *priv = NULL; struct rte_eth_dev *eth_dev = NULL; - struct ibv_device_attr_ex device_attr_ex; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -865,7 +862,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, len = snprintf(name, sizeof(name), PCI_PRI_FMT, pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - if (device_attr.orig_attr.phys_port_cnt > 1) + if (attr.orig_attr.phys_port_cnt > 1) snprintf(name + len, sizeof(name), " port %u", i); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); @@ -907,7 +904,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, continue; } DRV_LOG(DEBUG, "using port %u", port); - ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) + ctx = mlx5_glue->open_device(ibv_dev); if (ctx == NULL) { err = ENODEV; goto port_error; @@ -949,7 +947,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); - priv->device_attr = device_attr; + priv->device_attr = attr; priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; @@ -960,17 +958,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); goto port_error; } - err = mlx5_glue->query_device_ex(ctx, NULL, &device_attr_ex); - if (err) { - DRV_LOG(ERR, "ibv_query_device_ex() failed"); - goto port_error; - } - config.hw_csum = !!(device_attr_ex.device_cap_flags_ex & + config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); DRV_LOG(DEBUG, "checksum offloading is %ssupported", (config.hw_csum ? "" : "not ")); #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!(device_attr.max_counter_sets); + config.flow_counter_en = !!attr.max_counter_sets; mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", @@ -978,7 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, cs_desc.attributes); #endif config.ind_table_max_size = - device_attr_ex.rss_caps.max_rwq_indirection_table_size; + attr.rss_caps.max_rwq_indirection_table_size; /* Remove this check once DPDK supports larger/variable * indirection tables. */ if (config.ind_table_max_size > @@ -986,29 +979,28 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", config.ind_table_max_size); - config.hw_vlan_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_vlan_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); DRV_LOG(DEBUG, "VLAN stripping is %ssupported", (config.hw_vlan_strip ? "" : "not ")); - config.hw_fcs_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_fcs_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_SCATTER_FCS); DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", (config.hw_fcs_strip ? "" : "not ")); #ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!device_attr_ex.rx_pad_end_addr_align; + config.hw_padding = !!attr.rx_pad_end_addr_align; #endif DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", (config.hw_padding ? "" : "not ")); config.vf = vf; - config.tso = ((device_attr_ex.tso_caps.max_tso > 0) && - (device_attr_ex.tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); + config.tso = (attr.tso_caps.max_tso > 0 && + (attr.tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); if (config.tso) - config.tso_max_payload_sz = - device_attr_ex.tso_caps.max_tso; + config.tso_max_payload_sz = attr.tso_caps.max_tso; if (config.mps && !mps) { DRV_LOG(ERR, "multi-packet send not supported on this device" @@ -1170,14 +1162,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); + /* + * Each eth_dev instance is assigned its own Verbs context, + * since this one is consumed, let the next iteration open + * another. + */ + ctx = NULL; continue; port_error: if (priv) rte_free(priv); if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); - if (ctx) - claim_zero(mlx5_glue->close_device(ctx)); if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) rte_eth_dev_release_port(eth_dev); break; @@ -1189,8 +1185,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * way to enumerate the registered ethdevs to free the previous ones. */ error: - if (attr_ctx) - claim_zero(mlx5_glue->close_device(attr_ctx)); + if (ctx) + claim_zero(mlx5_glue->close_device(ctx)); if (list) mlx5_glue->free_device_list(list); if (err) { From patchwork Tue Jul 10 16:04:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42731 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3DB961B439; Tue, 10 Jul 2018 18:05:07 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id 268B31B1D3 for ; Tue, 10 Jul 2018 18:05:03 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id v25-v6so25240046wmc.0 for ; Tue, 10 Jul 2018 09:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=9hrYhb06M81OLEec/FAJOkvBG9oAKlUR/iA+Ybmj35w=; b=qW+ESj4d5L3L9vy0tFB0kv2j0FCQLIcKfd7DwW+ylvESvqMfNo/NFzsUBu+7KNPYp9 CwzUxMtCxzmlfnatDecbcNiqxL5e9xZtmpsUXloXhWDWZa1cMdIscAzyg5FE/yHtYoSs YHA6ZGNcA3sKfGPnWz6acdmcAl+7CH7PbcLqE/PYh/7NTBZ0JfM36RuTsbGN4slRjwfr iwRKH0Hba7Q+vYPRlp60xP09MYJiCLC/ku84kSdqUPal1GstDLaXKRXICVy9850CUHyK dqj+vXBSuThaBlfQ8UCdVPYPwv0i5wWUNRsl8nenp0UTfPq0HQaAPLSA1a7xmjXfnTYl VPSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=9hrYhb06M81OLEec/FAJOkvBG9oAKlUR/iA+Ybmj35w=; b=Jyw47h7NjyAdbbrzsUQJzgYFuYVGnFIFaCjNz0ZNSnxhpmljGuHefkQXGcnHPFR6bI 6TAVO7C6odv/teJyEPFjjQPnjrAH4xpXTEh33JZw1T1OIpKYC+fIgDc0c4ziKZOFo+kr 2tWxlGogGUuKgf/hVPqfXV/KN3Z+szAplJAdi9jefhULAu1ovD5/Wr0MZNq7xrZxs6Hh luWghu5sjptTC4Q272OaTjRLd/oUU6vG1v1Bjj+b8J+uOKNfe3UsZyoHyainQ6jHgS9k GfPOZnN20Jw66Oaiu5vzmJk29rzP92Pwbh3pJKTPIW5ZkRXoA4mDorT1M6TReQQULh/F AGpg== X-Gm-Message-State: APt69E3hNgRdTkZv4tknMIZMHa8TiM1yC/1wQ5jM5/83shRCrdU1SSJG t6NPqxpIRcjSS9PmzkjrWtxZMg== X-Google-Smtp-Source: AAOMgpf6TdcDE/dBGNmIxHg2Tkz3k79K3Ut+Uyb9BRfgZdz7Cn5gaH4our1qZKM7MDhKY2J8g3yQgA== X-Received: by 2002:a1c:b801:: with SMTP id i1-v6mr14599425wmf.30.1531238702816; Tue, 10 Jul 2018 09:05:02 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 125-v6sm35478652wmw.9.2018.07.10.09.05.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:01 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:46 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Xueming Li Message-ID: <20180710155743.14448-4-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 03/10] net/mlx5: drop useless support for several Verbs ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Unlike mlx4 from which this capability was inherited, mlx5 devices expose exactly one Verbs port per PCI bus address. Each physical port gets assigned its own bus address with a single Verbs port. While harmless, this code requires an extra loop that would get in the way of subsequent refactoring. No functional impact. Signed-off-by: Adrien Mazarguil Cc: Shahaf Shuler Cc: Xueming Li --- v3 changes: This patch was not present in prior revisions. As discussed [1], it was added after finally deciding to remove this support. [1] https://mails.dpdk.org/archives/dev/2018-June/105661.html --- drivers/net/mlx5/mlx5.c | 96 +++++++++++++------------------------ drivers/net/mlx5/mlx5.h | 1 - drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_txq.c | 2 +- 4 files changed, 34 insertions(+), 67 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 4e7f29f5b..717d8b268 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -652,11 +652,13 @@ static int mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) { - struct ibv_device **list = NULL; - struct ibv_device *ibv_dev; + struct ibv_device **list; struct ibv_context *ctx = NULL; struct ibv_device_attr_ex attr; + struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct rte_eth_dev *eth_dev = NULL; + struct priv *priv = NULL; int err = 0; unsigned int vf = 0; unsigned int mps; @@ -719,6 +721,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; break; } + mlx5_glue->free_device_list(list); if (ctx == NULL) { switch (err) { case 0: @@ -733,7 +736,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } goto error; } - ibv_dev = list[i]; DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; @@ -827,15 +829,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DEBUG("ibv_query_device_ex() failed"); goto error; } - DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); - for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { + { char name[RTE_ETH_NAME_MAX_LEN]; - int len; - uint32_t port = i + 1; /* ports are indexed from one */ struct ibv_port_attr port_attr; - struct ibv_pd *pd = NULL; - struct priv *priv = NULL; - struct rte_eth_dev *eth_dev = NULL; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -859,11 +855,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - len = snprintf(name, sizeof(name), PCI_PRI_FMT, + snprintf(name, sizeof(name), PCI_PRI_FMT, pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - if (attr.orig_attr.phys_port_cnt > 1) - snprintf(name + len, sizeof(name), " port %u", i); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -901,31 +895,22 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); rte_eth_dev_probing_finish(eth_dev); - continue; - } - DRV_LOG(DEBUG, "using port %u", port); - if (!ctx) - ctx = mlx5_glue->open_device(ibv_dev); - if (ctx == NULL) { - err = ENODEV; - goto port_error; + claim_zero(mlx5_glue->close_device(ctx)); + return 0; } /* Check port status. */ - err = mlx5_glue->query_port(ctx, port, &port_attr); + err = mlx5_glue->query_port(ctx, 1, &port_attr); if (err) { DRV_LOG(ERR, "port query failed: %s", strerror(err)); - goto port_error; + goto error; } if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { - DRV_LOG(ERR, - "port %d is not configured in Ethernet mode", - port); + DRV_LOG(ERR, "port is not configured in Ethernet mode"); err = EINVAL; - goto port_error; + goto error; } if (port_attr.state != IBV_PORT_ACTIVE) - DRV_LOG(DEBUG, "port %d is not active: \"%s\" (%d)", - port, + DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", mlx5_glue->port_state_str(port_attr.state), port_attr.state); /* Allocate protection domain. */ @@ -933,7 +918,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (pd == NULL) { DRV_LOG(ERR, "PD allocation failure"); err = ENOMEM; - goto port_error; + goto error; } /* from rte_ethdev.c */ priv = rte_zmalloc("ethdev private structure", @@ -942,13 +927,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (priv == NULL) { DRV_LOG(ERR, "priv allocation failure"); err = ENOMEM; - goto port_error; + goto error; } priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); priv->device_attr = attr; - priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; err = mlx5_args(&config, pci_dev->device.devargs); @@ -956,7 +940,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", strerror(rte_errno)); - goto port_error; + goto error; } config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); @@ -1006,7 +990,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, "multi-packet send not supported on this device" " (" MLX5_TXQ_MPW_EN ")"); err = ENOTSUP; - goto port_error; + goto error; } DRV_LOG(INFO, "%s MPS is %s", config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", @@ -1038,7 +1022,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (eth_dev == NULL) { DRV_LOG(ERR, "can not allocate rte ethdev"); err = ENOMEM; - goto port_error; + goto error; } eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; @@ -1049,7 +1033,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = mlx5_uar_init_primary(eth_dev); if (err) { err = rte_errno; - goto port_error; + goto error; } /* Configure the first MAC address by default. */ if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { @@ -1058,7 +1042,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, " loaded? (errno: %s)", eth_dev->data->port_id, strerror(rte_errno)); err = ENODEV; - goto port_error; + goto error; } DRV_LOG(INFO, "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", @@ -1082,7 +1066,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = mlx5_get_mtu(eth_dev, &priv->mtu); if (err) { err = rte_errno; - goto port_error; + goto error; } DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, priv->mtu); @@ -1131,7 +1115,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "port %u drop queue allocation failed: %s", eth_dev->data->port_id, strerror(rte_errno)); err = rte_errno; - goto port_error; + goto error; } /* Supported Verbs flow priority number detection. */ if (verb_priorities == 0) @@ -1140,7 +1124,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", eth_dev->data->port_id, verb_priorities); err = ENOTSUP; - goto port_error; + goto error; } priv->config.max_verbs_prio = verb_priorities; /* @@ -1154,7 +1138,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->device->numa_node); if (err) { err = rte_errno; - goto port_error; + goto error; } /* Add device to memory callback list. */ rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); @@ -1162,33 +1146,17 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - /* - * Each eth_dev instance is assigned its own Verbs context, - * since this one is consumed, let the next iteration open - * another. - */ - ctx = NULL; - continue; -port_error: - if (priv) - rte_free(priv); - if (pd) - claim_zero(mlx5_glue->dealloc_pd(pd)); - if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_eth_dev_release_port(eth_dev); - break; + return 0; } - /* - * XXX if something went wrong in the loop above, there is a resource - * leak (ctx, pd, priv, dpdk ethdev) but we can do nothing about it as - * long as the dpdk does not provide a way to deallocate a ethdev and a - * way to enumerate the registered ethdevs to free the previous ones. - */ error: + if (priv) + rte_free(priv); + if (pd) + claim_zero(mlx5_glue->dealloc_pd(pd)); + if (eth_dev) + rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (list) - mlx5_glue->free_device_list(list); if (err) { rte_errno = err; return -rte_errno; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 997b04a33..f55ff4a21 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -159,7 +159,6 @@ struct priv { unsigned int vlan_filter_n; /* Number of configured VLAN filters. */ /* Device properties. */ uint16_t mtu; /* Configured MTU. */ - uint8_t port; /* Physical port number. */ unsigned int isolated:1; /* Whether isolated mode is enabled. */ /* RX/TX queues. */ unsigned int rxqs_n; /* RX queues array size. */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index ebe5cb6e3..819f5baad 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -166,7 +166,7 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) if (dev_port == dev_port_prev) goto try_dev_id; dev_port_prev = dev_port; - if (dev_port == (priv->port - 1u)) + if (dev_port == 0) strlcpy(match, name, sizeof(match)); } closedir(dir); diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 669b91319..5057561ae 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -434,7 +434,7 @@ mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t idx) /* Move the QP to this state. */ .qp_state = IBV_QPS_INIT, /* Primary port number. */ - .port_num = priv->port + .port_num = 1, }; ret = mlx5_glue->modify_qp(tmpl.qp, &attr.mod, (IBV_QP_STATE | IBV_QP_PORT)); From patchwork Tue Jul 10 16:04:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42732 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1E25C1B44B; Tue, 10 Jul 2018 18:05:09 +0200 (CEST) Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by dpdk.org (Postfix) with ESMTP id 2A2371B3EE for ; Tue, 10 Jul 2018 18:05:05 +0200 (CEST) Received: by mail-wr1-f66.google.com with SMTP id p1-v6so15193937wrs.9 for ; Tue, 10 Jul 2018 09:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=xtbOJwUnMjHjHmttYfc3zRnrMIIvVv6B5bGkuqy3SOBrAuXW7jwGKuP2imLXxwZfuO +kDYZ8avOJwJp4QrltpPC3XlPapXfbkjqmq1GXlN9ThzYc43R8CLqD5Loo1LkcHdyIJ3 MFG7ukwiBQeCdQcBysjvXgzVRrC44NVzrgBQn2m5o9wJBZReh/fQTJ1Tf6as5wZ9eCAo sAzgE01HwU3Zy2lqHF8RK7r+SeNacZQBC4OKhy/CNKKoyHN9wIK5BQNMp5BKFGqAumm5 Z5DFGPv4DDt8rgLrHuefuPkWTbshFs31ajrgDckb0/ef9aGslhrp521PLKmklbn5U7us LCmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=OvKB40EdOcEWYCCSXLrgnrpZ91YU/KC0LRMZd4144G0y7foSZ9ywfYg8K7ptkAR8yS IYDl/XKPBHZn6DRd5C6157GJXy4tzS334rrsRiWNpqkHWV/U/a/qZij0DLaTRu4AiX7p ErtGe89lenatAShP0OJQoNWEadd5Vp+dyaabJ1NDGNfYAtfSyHPn7p7pTB0EDbUhY4HQ Z381To5X7dL07t291tdO5gaPeNYtaDNjz1VehBtuhkY1fu6INwoXzqNRY1Mu3Q5W94Xi No4JC4nVmyb8RouEz3iOQNktr2UiKtWJFNC88lqtygdI7k6JgYnZnSK+2Z/zufdpT48g 6M7g== X-Gm-Message-State: APt69E2biA/NFlaKevnKPNXmv7WqgIdjaD0sqEM44wOr6E0k+IZ9H4pv q8P6QSFIDF71a23kK/OeYJr3CA== X-Google-Smtp-Source: AAOMgpesiMA6Desqa8RZu+/q7qHowv8ey52n7LtLO2rIV3t3iw2ihsvoxy8XHjEH9Az2cvf9gY2ibg== X-Received: by 2002:adf:b2a7:: with SMTP id g36-v6mr17497921wrd.218.1531238704816; Tue, 10 Jul 2018 09:05:04 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id q70-v6sm31135208wmd.45.2018.07.10.09.05.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:04 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:48 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-5-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 04/10] net/mlx5: split PCI from generic probing code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" All the generic probing code needs is an IB device. While this device is currently supplied by a PCI lookup, other methods will be added soon. This patch divides the original function, which has become huge over time, as follows: 1. PCI-specific (mlx5_pci_probe()). 2. Verbs device (mlx5_dev_spawn()). (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li Cc: Shahaf Shuler --- v3 changes: - Moved VF device check within mlx5_pci_probe() after identifying the device instead of before that. - Merged mlx5_dev_spawn_one() with mlx5_dev_spawn() since there is no need anymore for an intermediate function to iterate over Verbs ports. v2 changes: - Fixed device naming. A port suffix is now appended only if several IB ports happen to be detected. - Added separate message to distinguish missing kernel drivers from other initialization errors, as it was confusing. --- drivers/net/mlx5/mlx5.c | 195 ++++++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 86 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 717d8b268..8916d4684 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -36,6 +36,7 @@ #include #include #include +#include #include "mlx5.h" #include "mlx5_utils.h" @@ -635,32 +636,31 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) } /** - * DPDK callback to register a PCI device. - * - * This function creates an Ethernet device for each port of a given - * PCI device. + * Spawn an Ethernet device from Verbs information. * - * @param[in] pci_drv - * PCI driver structure (mlx5_driver). - * @param[in] pci_dev - * PCI device information. + * @param dpdk_dev + * Backing DPDK device. + * @param ibv_dev + * Verbs device. + * @param vf + * If nonzero, enable VF-specific features. * * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. + * A valid Ethernet device object on success, NULL otherwise and rte_errno + * is set. */ -static int -mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +static struct rte_eth_dev * +mlx5_dev_spawn(struct rte_device *dpdk_dev, + struct ibv_device *ibv_dev, + int vf) { - struct ibv_device **list; - struct ibv_context *ctx = NULL; + struct ibv_context *ctx; struct ibv_device_attr_ex attr; struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; - unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; unsigned int tunnel_en = 0; @@ -672,71 +672,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_max_stride_size_n = 0; unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; - int i; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); - assert(pci_drv == &mlx5_driver); - list = mlx5_glue->get_device_list(&i); - if (list == NULL) { - assert(errno); - err = errno; - if (errno == ENOSYS) - DRV_LOG(ERR, - "cannot list devices, is ib_uverbs loaded?"); - goto error; - } - assert(i >= 0); - /* - * For each listed device, check related sysfs entry against - * the provided PCI ID. - */ - while (i != 0) { - struct rte_pci_addr pci_addr; - - --i; - DRV_LOG(DEBUG, "checking device \"%s\"", list[i]->name); - if (mlx5_ibv_device_to_pci_addr(list[i], &pci_addr)) - continue; - if ((pci_dev->addr.domain != pci_addr.domain) || - (pci_dev->addr.bus != pci_addr.bus) || - (pci_dev->addr.devid != pci_addr.devid) || - (pci_dev->addr.function != pci_addr.function)) - continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", - list[i]->name); - vf = ((pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - ctx = mlx5_glue->open_device(list[i]); - rte_errno = errno; - err = rte_errno; - break; - } - mlx5_glue->free_device_list(list); - if (ctx == NULL) { - switch (err) { - case 0: - DRV_LOG(ERR, - "cannot access device, is mlx5_ib loaded?"); - err = ENODEV; - break; - case EINVAL: - DRV_LOG(ERR, - "cannot use device, are drivers up to date?"); - break; - } - goto error; + errno = 0; + ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) { + rte_errno = errno ? errno : ENODEV; + return NULL; } - DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif @@ -855,9 +802,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - snprintf(name, sizeof(name), PCI_PRI_FMT, - pci_dev->addr.domain, pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -866,7 +811,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; goto error; } - eth_dev->device = &pci_dev->device; + eth_dev->device = dpdk_dev; eth_dev->dev_ops = &mlx5_dev_sec_ops; err = mlx5_uar_init_secondary(eth_dev); if (err) { @@ -894,9 +839,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mlx5_select_rx_function(eth_dev); eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); - rte_eth_dev_probing_finish(eth_dev); claim_zero(mlx5_glue->close_device(ctx)); - return 0; + return eth_dev; } /* Check port status. */ err = mlx5_glue->query_port(ctx, 1, &port_attr); @@ -935,7 +879,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->device_attr = attr; priv->pd = pd; priv->mtu = ETHER_MTU; - err = mlx5_args(&config, pci_dev->device.devargs); + err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", @@ -1027,8 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; - eth_dev->device = &pci_dev->device; - rte_eth_copy_pci_info(eth_dev, pci_dev); + eth_dev->device = dpdk_dev; eth_dev->device->driver = &mlx5_driver.driver; err = mlx5_uar_init_primary(eth_dev); if (err) { @@ -1146,7 +1089,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - return 0; + return eth_dev; } error: if (priv) @@ -1157,11 +1100,91 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (err) { - rte_errno = err; + assert(err > 0); + rte_errno = err; + return NULL; +} + +/** + * DPDK callback to register a PCI device. + * + * This function spawns an Ethernet device out of a given PCI device. + * + * @param[in] pci_drv + * PCI driver structure (mlx5_driver). + * @param[in] pci_dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct ibv_device **ibv_list; + struct rte_eth_dev *eth_dev = NULL; + int vf; + int ret; + + assert(pci_drv == &mlx5_driver); + errno = 0; + ibv_list = mlx5_glue->get_device_list(&ret); + if (!ibv_list) { + rte_errno = errno ? errno : ENOSYS; + DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } - return 0; + while (ret-- > 0) { + struct rte_pci_addr pci_addr; + + DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name); + if (mlx5_ibv_device_to_pci_addr(ibv_list[ret], &pci_addr)) + continue; + if (pci_dev->addr.domain != pci_addr.domain || + pci_dev->addr.bus != pci_addr.bus || + pci_dev->addr.devid != pci_addr.devid || + pci_dev->addr.function != pci_addr.function) + continue; + DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + ibv_list[ret]->name); + break; + } + switch (pci_dev->id.device_id) { + case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF: + vf = 1; + break; + default: + vf = 0; + } + if (ret >= 0) + eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + mlx5_glue->free_device_list(ibv_list); + if (!ret) { + DRV_LOG(WARNING, + "no Verbs device matches PCI device " PCI_PRI_FMT "," + " are kernel drivers loaded?", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function); + rte_errno = ENOENT; + ret = -rte_errno; + } else if (!eth_dev) { + DRV_LOG(ERR, + "probe of PCI device " PCI_PRI_FMT " aborted after" + " encountering an error: %s", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function, + strerror(rte_errno)); + ret = -rte_errno; + } else { + rte_eth_copy_pci_info(eth_dev, pci_dev); + rte_eth_dev_probing_finish(eth_dev); + ret = 0; + } + return ret; } static const struct rte_pci_id mlx5_pci_id_map[] = { From patchwork Tue Jul 10 16:04:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42733 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6098C1B455; Tue, 10 Jul 2018 18:05:12 +0200 (CEST) Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by dpdk.org (Postfix) with ESMTP id 726351B440 for ; Tue, 10 Jul 2018 18:05:07 +0200 (CEST) Received: by mail-wm0-f54.google.com with SMTP id s12-v6so25241610wmc.1 for ; Tue, 10 Jul 2018 09:05:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=DmREd05I1V0ABxeZRbzLBJH+y7fOYJDLHEFO/UKDYhU=; b=ad6siReMzF3PvvGMa9RA/oAGKFfCo5qH72QRdfnaNcl08oW8BmccyHvNxgX39sjz92 /g2ILSeDo2C+dXa3z6400pDo1MuRaoPMZcBEqYR9gbDbPcqiTuXiyFUkjMbrZ8He2/Rm 8Z+gREguElP8+Kq1wZSXTXXl+2kPCfyKaRlOKZwgxl39RYhwqQV4SNf7RxtS4j1Uer9S +mNw8pTI7xL3kPBEG2tDos0HbEzExTm/OwS3d1tl0A4u2C1ey0lr+YgZM5PUKQDKLuNu 9/tu8+RGFUWEaA6MR7zWpi1Z2VQQtdTXEOkdPCy7zS8MSe/pOgTwrOa1AmG91n4DIHJP 1W5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=DmREd05I1V0ABxeZRbzLBJH+y7fOYJDLHEFO/UKDYhU=; b=Ukl+rciET5dZnxa1hE19yksGgROPWfzdeEDDY4dteoWwPTlVFCeBEpbBU7vrfw04tb /irHSwv5xwDPNPbTtYWSlWdRNxhGIwLu56mR3P6Gj03R3r2b0fSNurVU0hk/RyLV0xjQ tjX82tEkK70K9+YdAU8XR7T1+BAAFQP+pAMp6hnzI4WL3o0scwWjgzazGoIQFwzEJnHu 48Rn3qyJFBPHud2C2zrT5Mt7BI2I+H8H52DePJCZO/7o6U20kQv874CSEBpirrqqLvmy x/8DMdEZohIFSmNkuiRwTUOl+RB7FM9Ze917fZ9h78gOhuk057M9j1coE1WOIPyQjNZ9 19kA== X-Gm-Message-State: APt69E1MxpyXeIVk0kaU8PuhMCGBeYIbDh91d/CtgEWV8BzF+lurPgCr lFWXq0PFoFswTBPPvtDI8+cd8J5+ X-Google-Smtp-Source: AAOMgpf7ALRgFo84rSspsYrJAhAFSpsnSThIUt/ApGCxllqMLoQql/vDAVXXIYI7YEMZu2DBV3wEiQ== X-Received: by 2002:a1c:c289:: with SMTP id s131-v6mr17302735wmf.65.1531238706885; Tue, 10 Jul 2018 09:05:06 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 185-v6sm33928483wmw.23.2018.07.10.09.05.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:06 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:50 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-6-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 05/10] net/mlx5: re-indent generic probing function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since commit "net/mlx5: drop useless support for several Verbs ports" removed an inner loop, mlx5_dev_spawn() is left with an unnecessary indent level. This patch eliminates a block, moves its local variables to function scope, and re-indents its contents (diff best viewed with --ignore-all-space). No functional impact. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming(Steven) Li --- v3 changes: - Reworded commit log since original patch was modified. This patch is also much shorter as a consequence. --- drivers/net/mlx5/mlx5.c | 578 +++++++++++++++++++++---------------------- 1 file changed, 282 insertions(+), 296 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 8916d4684..1054bf6d0 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -656,8 +656,25 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, { struct ibv_context *ctx; struct ibv_device_attr_ex attr; + struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct mlx5_dev_config config = { + .vf = !!vf, + .tx_vec_en = 1, + .rx_vec_en = 1, + .mpw_hdr_dseg = 0, + .txq_inline = MLX5_ARG_UNSET, + .txqs_inline = MLX5_ARG_UNSET, + .inline_max_packet_sz = MLX5_ARG_UNSET, + .vf_nl_en = 1, + .mprq = { + .enabled = 0, + .stride_num_n = MLX5_MPRQ_STRIDE_NUM_N, + .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, + .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, + }, + }; struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; @@ -675,6 +692,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif + struct ether_addr mac; + char name[RTE_ETH_NAME_MAX_LEN]; /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); @@ -710,11 +729,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(DEBUG, "MPW isn't supported"); mps = MLX5_MPW_DISABLED; } + config.mps = mps; #ifdef HAVE_IBV_MLX5_MOD_SWP if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif + config.swp = !!swp; #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = @@ -740,6 +761,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, mprq_caps.min_single_wqe_log_num_of_strides; mprq_max_stride_num_n = mprq_caps.max_single_wqe_log_num_of_strides; + config.mprq.stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); } #endif if (RTE_CACHE_LINE_SIZE == 128 && @@ -747,6 +770,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, cqe_comp = 0; else cqe_comp = 1; + config.cqe_comp = cqe_comp; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { tunnel_en = ((dv_attr.tunnel_offloads_caps & @@ -760,6 +784,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif + config.tunnel_en = tunnel_en; #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && @@ -771,326 +796,287 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif + config.mpls_en = mpls_en; err = mlx5_glue->query_device_ex(ctx, NULL, &attr); if (err) { DEBUG("ibv_query_device_ex() failed"); goto error; } - { - char name[RTE_ETH_NAME_MAX_LEN]; - struct ibv_port_attr port_attr; - struct ether_addr mac; - struct mlx5_dev_config config = { - .cqe_comp = cqe_comp, - .mps = mps, - .tunnel_en = tunnel_en, - .mpls_en = mpls_en, - .tx_vec_en = 1, - .rx_vec_en = 1, - .mpw_hdr_dseg = 0, - .txq_inline = MLX5_ARG_UNSET, - .txqs_inline = MLX5_ARG_UNSET, - .inline_max_packet_sz = MLX5_ARG_UNSET, - .vf_nl_en = 1, - .swp = !!swp, - .mprq = { - .enabled = 0, /* Disabled by default. */ - .stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n), - .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, - .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, - }, - }; - - rte_strlcpy(name, dpdk_dev->name, sizeof(name)); - if (rte_eal_process_type() == RTE_PROC_SECONDARY) { - eth_dev = rte_eth_dev_attach_secondary(name); - if (eth_dev == NULL) { - DRV_LOG(ERR, "can not attach rte ethdev"); - rte_errno = ENOMEM; - err = rte_errno; - goto error; - } - eth_dev->device = dpdk_dev; - eth_dev->dev_ops = &mlx5_dev_sec_ops; - err = mlx5_uar_init_secondary(eth_dev); - if (err) { - err = rte_errno; - goto error; - } - /* Receive command fd from primary process */ - err = mlx5_socket_connect(eth_dev); - if (err < 0) { - err = rte_errno; - goto error; - } - /* Remap UAR for Tx queues. */ - err = mlx5_tx_uar_remap(eth_dev, err); - if (err) { - err = rte_errno; - goto error; - } - /* - * Ethdev pointer is still required as input since - * the primary device is not accessible from the - * secondary process. - */ - eth_dev->rx_pkt_burst = - mlx5_select_rx_function(eth_dev); - eth_dev->tx_pkt_burst = - mlx5_select_tx_function(eth_dev); - claim_zero(mlx5_glue->close_device(ctx)); - return eth_dev; - } - /* Check port status. */ - err = mlx5_glue->query_port(ctx, 1, &port_attr); - if (err) { - DRV_LOG(ERR, "port query failed: %s", strerror(err)); - goto error; - } - if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { - DRV_LOG(ERR, "port is not configured in Ethernet mode"); - err = EINVAL; - goto error; - } - if (port_attr.state != IBV_PORT_ACTIVE) - DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", - mlx5_glue->port_state_str(port_attr.state), - port_attr.state); - /* Allocate protection domain. */ - pd = mlx5_glue->alloc_pd(ctx); - if (pd == NULL) { - DRV_LOG(ERR, "PD allocation failure"); - err = ENOMEM; - goto error; - } - /* from rte_ethdev.c */ - priv = rte_zmalloc("ethdev private structure", - sizeof(*priv), - RTE_CACHE_LINE_SIZE); - if (priv == NULL) { - DRV_LOG(ERR, "priv allocation failure"); - err = ENOMEM; - goto error; - } - priv->ctx = ctx; - strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, - sizeof(priv->ibdev_path)); - priv->device_attr = attr; - priv->pd = pd; - priv->mtu = ETHER_MTU; - err = mlx5_args(&config, dpdk_dev->devargs); - if (err) { - err = rte_errno; - DRV_LOG(ERR, "failed to process device arguments: %s", - strerror(rte_errno)); - goto error; - } - config.hw_csum = !!(attr.device_cap_flags_ex & - IBV_DEVICE_RAW_IP_CSUM); - DRV_LOG(DEBUG, "checksum offloading is %ssupported", - (config.hw_csum ? "" : "not ")); -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!attr.max_counter_sets; - mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); - DRV_LOG(DEBUG, - "counter type = %d, num of cs = %ld, attributes = %d", - cs_desc.counter_type, cs_desc.num_of_cs, - cs_desc.attributes); -#endif - config.ind_table_max_size = - attr.rss_caps.max_rwq_indirection_table_size; - /* Remove this check once DPDK supports larger/variable - * indirection tables. */ - if (config.ind_table_max_size > - (unsigned int)ETH_RSS_RETA_SIZE_512) - config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; - DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", - config.ind_table_max_size); - config.hw_vlan_strip = !!(attr.raw_packet_caps & - IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); - DRV_LOG(DEBUG, "VLAN stripping is %ssupported", - (config.hw_vlan_strip ? "" : "not ")); - - config.hw_fcs_strip = !!(attr.raw_packet_caps & - IBV_RAW_PACKET_CAP_SCATTER_FCS); - DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", - (config.hw_fcs_strip ? "" : "not ")); - -#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!attr.rx_pad_end_addr_align; -#endif - DRV_LOG(DEBUG, - "hardware Rx end alignment padding is %ssupported", - (config.hw_padding ? "" : "not ")); - config.vf = vf; - config.tso = (attr.tso_caps.max_tso > 0 && - (attr.tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); - if (config.tso) - config.tso_max_payload_sz = attr.tso_caps.max_tso; - if (config.mps && !mps) { - DRV_LOG(ERR, - "multi-packet send not supported on this device" - " (" MLX5_TXQ_MPW_EN ")"); - err = ENOTSUP; - goto error; - } - DRV_LOG(INFO, "%s MPS is %s", - config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", - config.mps != MLX5_MPW_DISABLED ? "enabled" : - "disabled"); - if (config.cqe_comp && !cqe_comp) { - DRV_LOG(WARNING, "Rx CQE compression isn't supported"); - config.cqe_comp = 0; - } - if (config.mprq.enabled && mprq) { - if (config.mprq.stride_num_n > mprq_max_stride_num_n || - config.mprq.stride_num_n < mprq_min_stride_num_n) { - config.mprq.stride_num_n = - RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n); - DRV_LOG(WARNING, - "the number of strides" - " for Multi-Packet RQ is out of range," - " setting default value (%u)", - 1 << config.mprq.stride_num_n); - } - config.mprq.min_stride_size_n = mprq_min_stride_size_n; - config.mprq.max_stride_size_n = mprq_max_stride_size_n; - } else if (config.mprq.enabled && !mprq) { - DRV_LOG(WARNING, "Multi-Packet RQ isn't supported"); - config.mprq.enabled = 0; - } - eth_dev = rte_eth_dev_allocate(name); + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + if (rte_eal_process_type() == RTE_PROC_SECONDARY) { + eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { - DRV_LOG(ERR, "can not allocate rte ethdev"); - err = ENOMEM; + DRV_LOG(ERR, "can not attach rte ethdev"); + rte_errno = ENOMEM; + err = rte_errno; goto error; } - eth_dev->data->dev_private = priv; - priv->dev_data = eth_dev->data; - eth_dev->data->mac_addrs = priv->mac; eth_dev->device = dpdk_dev; - eth_dev->device->driver = &mlx5_driver.driver; - err = mlx5_uar_init_primary(eth_dev); + eth_dev->dev_ops = &mlx5_dev_sec_ops; + err = mlx5_uar_init_secondary(eth_dev); if (err) { err = rte_errno; goto error; } - /* Configure the first MAC address by default. */ - if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { - DRV_LOG(ERR, - "port %u cannot get MAC address, is mlx5_en" - " loaded? (errno: %s)", - eth_dev->data->port_id, strerror(rte_errno)); - err = ENODEV; - goto error; - } - DRV_LOG(INFO, - "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", - eth_dev->data->port_id, - mac.addr_bytes[0], mac.addr_bytes[1], - mac.addr_bytes[2], mac.addr_bytes[3], - mac.addr_bytes[4], mac.addr_bytes[5]); -#ifndef NDEBUG - { - char ifname[IF_NAMESIZE]; - - if (mlx5_get_ifname(eth_dev, &ifname) == 0) - DRV_LOG(DEBUG, "port %u ifname is \"%s\"", - eth_dev->data->port_id, ifname); - else - DRV_LOG(DEBUG, "port %u ifname is unknown", - eth_dev->data->port_id); - } -#endif - /* Get actual MTU if possible. */ - err = mlx5_get_mtu(eth_dev, &priv->mtu); - if (err) { + /* Receive command fd from primary process */ + err = mlx5_socket_connect(eth_dev); + if (err < 0) { err = rte_errno; goto error; } - DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, - priv->mtu); - /* - * Initialize burst functions to prevent crashes before link-up. - */ - eth_dev->rx_pkt_burst = removed_rx_burst; - eth_dev->tx_pkt_burst = removed_tx_burst; - eth_dev->dev_ops = &mlx5_dev_ops; - /* Register MAC address. */ - claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); - priv->nl_socket = -1; - priv->nl_sn = 0; - if (vf && config.vf_nl_en) { - priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); - if (priv->nl_socket < 0) - priv->nl_socket = -1; - mlx5_nl_mac_addr_sync(eth_dev); - } - TAILQ_INIT(&priv->flows); - TAILQ_INIT(&priv->ctrl_flows); - /* Hint libmlx5 to use PMD allocator for data plane resources */ - struct mlx5dv_ctx_allocators alctr = { - .alloc = &mlx5_alloc_verbs_buf, - .free = &mlx5_free_verbs_buf, - .data = priv, - }; - mlx5_glue->dv_set_context_attr(ctx, - MLX5DV_CTX_ATTR_BUF_ALLOCATORS, - (void *)((uintptr_t)&alctr)); - /* Bring Ethernet device up. */ - DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", - eth_dev->data->port_id); - mlx5_set_link_up(eth_dev); - /* - * Even though the interrupt handler is not installed yet, - * interrupts will still trigger on the asyn_fd from - * Verbs context returned by ibv_open_device(). - */ - mlx5_link_update(eth_dev, 0); - /* Store device configuration on private structure. */ - priv->config = config; - /* Create drop queue. */ - err = mlx5_flow_create_drop_queue(eth_dev); + /* Remap UAR for Tx queues. */ + err = mlx5_tx_uar_remap(eth_dev, err); if (err) { - DRV_LOG(ERR, "port %u drop queue allocation failed: %s", - eth_dev->data->port_id, strerror(rte_errno)); err = rte_errno; goto error; } - /* Supported Verbs flow priority number detection. */ - if (verb_priorities == 0) - verb_priorities = mlx5_get_max_verbs_prio(eth_dev); - if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { - DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", - eth_dev->data->port_id, verb_priorities); - err = ENOTSUP; - goto error; - } - priv->config.max_verbs_prio = verb_priorities; /* - * Once the device is added to the list of memory event - * callback, its global MR cache table cannot be expanded - * on the fly because of deadlock. If it overflows, lookup - * should be done by searching MR list linearly, which is slow. + * Ethdev pointer is still required as input since + * the primary device is not accessible from the + * secondary process. */ - err = mlx5_mr_btree_init(&priv->mr.cache, - MLX5_MR_BTREE_CACHE_N * 2, - eth_dev->device->numa_node); - if (err) { - err = rte_errno; - goto error; - } - /* Add device to memory callback list. */ - rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); - LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, - priv, mem_event_cb); - rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); - rte_eth_dev_probing_finish(eth_dev); + eth_dev->rx_pkt_burst = mlx5_select_rx_function(eth_dev); + eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); + claim_zero(mlx5_glue->close_device(ctx)); return eth_dev; } + /* Check port status. */ + err = mlx5_glue->query_port(ctx, 1, &port_attr); + if (err) { + DRV_LOG(ERR, "port query failed: %s", strerror(err)); + goto error; + } + if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { + DRV_LOG(ERR, "port is not configured in Ethernet mode"); + err = EINVAL; + goto error; + } + if (port_attr.state != IBV_PORT_ACTIVE) + DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", + mlx5_glue->port_state_str(port_attr.state), + port_attr.state); + /* Allocate protection domain. */ + pd = mlx5_glue->alloc_pd(ctx); + if (pd == NULL) { + DRV_LOG(ERR, "PD allocation failure"); + err = ENOMEM; + goto error; + } + priv = rte_zmalloc("ethdev private structure", + sizeof(*priv), + RTE_CACHE_LINE_SIZE); + if (priv == NULL) { + DRV_LOG(ERR, "priv allocation failure"); + err = ENOMEM; + goto error; + } + priv->ctx = ctx; + strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, + sizeof(priv->ibdev_path)); + priv->device_attr = attr; + priv->pd = pd; + priv->mtu = ETHER_MTU; + err = mlx5_args(&config, dpdk_dev->devargs); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + goto error; + } + config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); + DRV_LOG(DEBUG, "checksum offloading is %ssupported", + (config.hw_csum ? "" : "not ")); +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + config.flow_counter_en = !!attr.max_counter_sets; + mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); + DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", + cs_desc.counter_type, cs_desc.num_of_cs, + cs_desc.attributes); +#endif + config.ind_table_max_size = + attr.rss_caps.max_rwq_indirection_table_size; + /* + * Remove this check once DPDK supports larger/variable + * indirection tables. + */ + if (config.ind_table_max_size > (unsigned int)ETH_RSS_RETA_SIZE_512) + config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; + DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", + config.ind_table_max_size); + config.hw_vlan_strip = !!(attr.raw_packet_caps & + IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); + DRV_LOG(DEBUG, "VLAN stripping is %ssupported", + (config.hw_vlan_strip ? "" : "not ")); + config.hw_fcs_strip = !!(attr.raw_packet_caps & + IBV_RAW_PACKET_CAP_SCATTER_FCS); + DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", + (config.hw_fcs_strip ? "" : "not ")); +#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING + config.hw_padding = !!attr.rx_pad_end_addr_align; +#endif + DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", + (config.hw_padding ? "" : "not ")); + config.tso = (attr.tso_caps.max_tso > 0 && + (attr.tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); + if (config.tso) + config.tso_max_payload_sz = attr.tso_caps.max_tso; + if (config.mps && !mps) { + DRV_LOG(ERR, + "multi-packet send not supported on this device" + " (" MLX5_TXQ_MPW_EN ")"); + err = ENOTSUP; + goto error; + } + DRV_LOG(INFO, "%sMPS is %s", + config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", + config.mps != MLX5_MPW_DISABLED ? "enabled" : "disabled"); + if (config.cqe_comp && !cqe_comp) { + DRV_LOG(WARNING, "Rx CQE compression isn't supported"); + config.cqe_comp = 0; + } + if (config.mprq.enabled && mprq) { + if (config.mprq.stride_num_n > mprq_max_stride_num_n || + config.mprq.stride_num_n < mprq_min_stride_num_n) { + config.mprq.stride_num_n = + RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); + DRV_LOG(WARNING, + "the number of strides" + " for Multi-Packet RQ is out of range," + " setting default value (%u)", + 1 << config.mprq.stride_num_n); + } + config.mprq.min_stride_size_n = mprq_min_stride_size_n; + config.mprq.max_stride_size_n = mprq_max_stride_size_n; + } else if (config.mprq.enabled && !mprq) { + DRV_LOG(WARNING, "Multi-Packet RQ isn't supported"); + config.mprq.enabled = 0; + } + eth_dev = rte_eth_dev_allocate(name); + if (eth_dev == NULL) { + DRV_LOG(ERR, "can not allocate rte ethdev"); + err = ENOMEM; + goto error; + } + eth_dev->data->dev_private = priv; + priv->dev_data = eth_dev->data; + eth_dev->data->mac_addrs = priv->mac; + eth_dev->device = dpdk_dev; + eth_dev->device->driver = &mlx5_driver.driver; + err = mlx5_uar_init_primary(eth_dev); + if (err) { + err = rte_errno; + goto error; + } + /* Configure the first MAC address by default. */ + if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { + DRV_LOG(ERR, + "port %u cannot get MAC address, is mlx5_en" + " loaded? (errno: %s)", + eth_dev->data->port_id, strerror(rte_errno)); + err = ENODEV; + goto error; + } + DRV_LOG(INFO, + "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", + eth_dev->data->port_id, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); +#ifndef NDEBUG + { + char ifname[IF_NAMESIZE]; + + if (mlx5_get_ifname(eth_dev, &ifname) == 0) + DRV_LOG(DEBUG, "port %u ifname is \"%s\"", + eth_dev->data->port_id, ifname); + else + DRV_LOG(DEBUG, "port %u ifname is unknown", + eth_dev->data->port_id); + } +#endif + /* Get actual MTU if possible. */ + err = mlx5_get_mtu(eth_dev, &priv->mtu); + if (err) { + err = rte_errno; + goto error; + } + DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, + priv->mtu); + /* Initialize burst functions to prevent crashes before link-up. */ + eth_dev->rx_pkt_burst = removed_rx_burst; + eth_dev->tx_pkt_burst = removed_tx_burst; + eth_dev->dev_ops = &mlx5_dev_ops; + /* Register MAC address. */ + claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); + priv->nl_socket = -1; + priv->nl_sn = 0; + if (vf && config.vf_nl_en) { + priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); + if (priv->nl_socket < 0) + priv->nl_socket = -1; + mlx5_nl_mac_addr_sync(eth_dev); + } + TAILQ_INIT(&priv->flows); + TAILQ_INIT(&priv->ctrl_flows); + /* Hint libmlx5 to use PMD allocator for data plane resources */ + struct mlx5dv_ctx_allocators alctr = { + .alloc = &mlx5_alloc_verbs_buf, + .free = &mlx5_free_verbs_buf, + .data = priv, + }; + mlx5_glue->dv_set_context_attr(ctx, MLX5DV_CTX_ATTR_BUF_ALLOCATORS, + (void *)((uintptr_t)&alctr)); + /* Bring Ethernet device up. */ + DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", + eth_dev->data->port_id); + mlx5_set_link_up(eth_dev); + /* + * Even though the interrupt handler is not installed yet, + * interrupts will still trigger on the asyn_fd from + * Verbs context returned by ibv_open_device(). + */ + mlx5_link_update(eth_dev, 0); + /* Store device configuration on private structure. */ + priv->config = config; + /* Create drop queue. */ + err = mlx5_flow_create_drop_queue(eth_dev); + if (err) { + DRV_LOG(ERR, "port %u drop queue allocation failed: %s", + eth_dev->data->port_id, strerror(rte_errno)); + err = rte_errno; + goto error; + } + /* Supported Verbs flow priority number detection. */ + if (verb_priorities == 0) + verb_priorities = mlx5_get_max_verbs_prio(eth_dev); + if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { + DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", + eth_dev->data->port_id, verb_priorities); + err = ENOTSUP; + goto error; + } + priv->config.max_verbs_prio = verb_priorities; + /* + * Once the device is added to the list of memory event + * callback, its global MR cache table cannot be expanded + * on the fly because of deadlock. If it overflows, lookup + * should be done by searching MR list linearly, which is slow. + */ + err = mlx5_mr_btree_init(&priv->mr.cache, + MLX5_MR_BTREE_CACHE_N * 2, + eth_dev->device->numa_node); + if (err) { + err = rte_errno; + goto error; + } + /* Add device to memory callback list. */ + rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); + LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, + priv, mem_event_cb); + rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); + return eth_dev; error: if (priv) rte_free(priv); From patchwork Tue Jul 10 16:04:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42734 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 916531B459; Tue, 10 Jul 2018 18:05:14 +0200 (CEST) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 710741B450 for ; Tue, 10 Jul 2018 18:05:09 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id n17-v6so24952435wmh.2 for ; Tue, 10 Jul 2018 09:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=kJ968xsTRYu7vz89QNcs7eJmdg27NpUefHP1YnfREhg=; b=BPdTSn3jtuwkU1zJMykXYZP0p/n38JsT50Ou7mr+se/USMpZX01BFmJkXR6TDddWmW RpxiGmQiEll4bQeprqJJdAvKxO3Ps19oyLi79K9+6MJ2O2qn4FB3us1lX5BxVmLYXG+c xLtfWCzLiombzZoRC9DmZYycTdLzMpeov8uX/0CCJEes9Lc8uWl7uWsD6eJ++AOo8hYR WHjnghuid7IzGgAr73PoIovVSDTBBujPgMYQpYXlkPjy6Gjt/uCuTdY8uHx7h5PUHjV9 7K+QmfLwoztGvjGH4DHkkbALCuAdFHRCsJi6PqiWD9pRwEflNGL8HuFvEeoOXO94B09Q 6zrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=kJ968xsTRYu7vz89QNcs7eJmdg27NpUefHP1YnfREhg=; b=PIx9VGwgL9/RlgvJLi7sEFPVqw2h8A0DkUx3G74yykvL5F+tPJyKKnYlBctCVPkD3y 6fqXnlG0HVuGPozuMH0DmyaA05gQVuHnbeIhq2TcpGy+nwi63CD0ZnWbwXv/oicINzcS CdQBs48t68kML58S2vZ92F0Hpa3Ptvu22XRcyeumYmLsUt20wvakW46T7vm+s8NWVKsw ICWyYZq6Pc22ZZG6oh9idtOxLA7uRD9DI8GTfsFDVnsvpO6rlHtTlWbGEcUEqfSh5IWZ MDpwGP5IkQgQyDbxVaXxI6+Uvssph9609XlWgbjYXGB49OugXLmIZ6GcZIaYbKL72WDR k6kw== X-Gm-Message-State: APt69E2IH2Ct3gWMi7ea6GdcFRR6mh6MY9cEpBNVnf72ahQ9ae4VV+M9 8oI4ModHuv5cA1T8O6ao3woutElp X-Google-Smtp-Source: AAOMgpf5mJ0q9xJWqe3fvI964sFESMiTjk77Rm4YwwbDIawN5Ur/L3nVYx9s7ZIsgVZkxaSa239CaQ== X-Received: by 2002:a1c:e5cd:: with SMTP id c196-v6mr15131749wmh.101.1531238708855; Tue, 10 Jul 2018 09:05:08 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id b11-v6sm7507715wrv.82.2018.07.10.09.05.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:08 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:52 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Xueming Li Message-ID: <20180710155743.14448-7-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 06/10] net/mlx5: add port representor awareness X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The current PCI probing method is not aware of Verbs port representors, which appear as standard Verbs devices bound to the same PCI address and cannot be distinguished. Problem is that more often than not, the wrong Verbs device is used, resulting in unexpected traffic. This patch makes the driver discard representors to only use the master device. If unable to identify it (e.g. kernel drivers not recent enough), either: - There is only one matching device which isn't identified as a representor, in that case use it. - Otherwise log an error and do not probe the device. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li Cc: Xueming Li --- v5 changes: - Fixed linux/rdma_netlink.h -> rdma/rdma_netlink.h in Makefile. - Added automatic detection for RDMA_NL_NLDEV, IFLA_PHYS_SWITCH_ID and IFLA_PHYS_PORT_NAME to allow compilation on systems without those. v3 changes: - Replaced all heuristics (including mlx5_cmp_ibv_name()) with Netlink queries to associate IB devices with network interfaces. - Reworded commit log. v2 changes: - Fixed digit detection in mlx5_cmp_ibv_name() so that "foo1" and "foo10" are compared on the integer conversion of "1" against "10" instead of "" and "0". --- drivers/net/mlx5/Makefile | 45 ++++++ drivers/net/mlx5/mlx5.c | 109 ++++++++++++-- drivers/net/mlx5/mlx5.h | 16 ++- drivers/net/mlx5/mlx5_nl.c | 308 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 454 insertions(+), 24 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 955861a41..9e274964b 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -152,6 +152,51 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh infiniband/verbs.h \ enum IBV_FLOW_SPEC_ACTION_COUNT \ $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NL_NLDEV \ + rdma/rdma_netlink.h \ + enum RDMA_NL_NLDEV \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_CMD_GET \ + rdma/rdma_netlink.h \ + enum RDMA_NLDEV_CMD_GET \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_CMD_PORT_GET \ + rdma/rdma_netlink.h \ + enum RDMA_NLDEV_CMD_PORT_GET \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_DEV_INDEX \ + rdma/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_DEV_INDEX \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_DEV_NAME \ + rdma/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_DEV_NAME \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_PORT_INDEX \ + rdma/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_PORT_INDEX \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX \ + rdma/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_NDEV_INDEX \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_IFLA_PHYS_SWITCH_ID \ + linux/if_link.h \ + enum IFLA_PHYS_SWITCH_ID \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_IFLA_PHYS_PORT_NAME \ + linux/if_link.h \ + enum IFLA_PHYS_PORT_NAME \ + $(AUTOCONF_OUTPUT) # Create mlx5_autoconf.h or update it in case it differs from the new one. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 1054bf6d0..d06ba9886 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -13,6 +13,7 @@ #include #include #include +#include #include /* Verbs header. */ @@ -274,8 +275,10 @@ mlx5_dev_close(struct rte_eth_dev *dev) mlx5_socket_uninit(dev); if (priv->config.vf) mlx5_nl_mac_addr_flush(dev); - if (priv->nl_socket >= 0) - close(priv->nl_socket); + if (priv->nl_socket_route >= 0) + close(priv->nl_socket_route); + if (priv->nl_socket_rdma >= 0) + close(priv->nl_socket_rdma); ret = mlx5_hrxq_ibv_verify(dev); if (ret) DRV_LOG(WARNING, "port %u some hash Rx queue still remain", @@ -876,6 +879,10 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->device_attr = attr; priv->pd = pd; priv->mtu = ETHER_MTU; + /* Some internal functions rely on Netlink sockets, open them now. */ + priv->nl_socket_rdma = mlx5_nl_init(0, NETLINK_RDMA); + priv->nl_socket_route = mlx5_nl_init(RTMGRP_LINK, NETLINK_ROUTE); + priv->nl_sn = 0; err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; @@ -1010,14 +1017,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, eth_dev->dev_ops = &mlx5_dev_ops; /* Register MAC address. */ claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); - priv->nl_socket = -1; - priv->nl_sn = 0; - if (vf && config.vf_nl_en) { - priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); - if (priv->nl_socket < 0) - priv->nl_socket = -1; + if (vf && config.vf_nl_en) mlx5_nl_mac_addr_sync(eth_dev); - } TAILQ_INIT(&priv->flows); TAILQ_INIT(&priv->ctrl_flows); /* Hint libmlx5 to use PMD allocator for data plane resources */ @@ -1078,8 +1079,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); return eth_dev; error: - if (priv) + if (priv) { + if (priv->nl_socket_route >= 0) + close(priv->nl_socket_route); + if (priv->nl_socket_rdma >= 0) + close(priv->nl_socket_rdma); rte_free(priv); + } if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); if (eth_dev) @@ -1110,6 +1116,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **ibv_list; struct rte_eth_dev *eth_dev = NULL; + unsigned int n = 0; int vf; int ret; @@ -1121,6 +1128,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } + + struct ibv_device *ibv_match[ret + 1]; + while (ret-- > 0) { struct rte_pci_addr pci_addr; @@ -1132,10 +1142,81 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid != pci_addr.devid || pci_dev->addr.function != pci_addr.function) continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + DRV_LOG(INFO, "PCI information matches for device \"%s\"", ibv_list[ret]->name); + ibv_match[n++] = ibv_list[ret]; + } + ibv_match[n] = NULL; + + unsigned int ifindex[n]; + struct mlx5_switch_info info[n]; + int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; + int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; + unsigned int i; + + /* + * The existence of several matching entries (n > 1) means port + * representors have been instantiated. No existing Verbs call nor + * /sys entries can tell them apart, this can only be done through + * Netlink calls assuming kernel drivers are recent enough to + * support them. + * + * In the event of identification failure through Netlink, either: + * + * 1. No device matches (n == 0), complain and bail out. + * 2. A single IB device matches (n == 1) and is not a representor, + * assume no switch support. + * 3. Otherwise no safe assumptions can be made; complain louder and + * bail out. + */ + for (i = 0; i != n; ++i) { + if (nl_rdma < 0) + ifindex[i] = 0; + else + ifindex[i] = mlx5_nl_ifindex(nl_rdma, + ibv_match[i]->name); + if (nl_route < 0 || + !ifindex[i] || + mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { + ifindex[i] = 0; + memset(&info[i], 0, sizeof(info[i])); + continue; + } + } + if (nl_rdma >= 0) + close(nl_rdma); + if (nl_route >= 0) + close(nl_route); + /* Look for master device. */ + for (i = 0; i != n; ++i) { + if (!info[i].master) + continue; + /* Make it the first entry. */ + if (i == 0) + break; + ibv_match[n] = ibv_match[0]; + ibv_match[0] = ibv_match[i]; + ibv_match[n] = NULL; break; } + if (n && i == n) { + if (n == 1 && !info[0].representor) { + /* Case #2. */ + DRV_LOG(INFO, "no switch support detected"); + } else if (n == 1) { + /* Case #3. */ + DRV_LOG(ERR, + "device looks like a port representor, this is" + " not supported yet"); + n = 0; + } else { + /* Case #3. */ + DRV_LOG(ERR, + "unable to tell which of the matching devices" + " is the master (lack of kernel support?)"); + n = 0; + } + } switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1146,10 +1227,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, default: vf = 0; } - if (ret >= 0) - eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + if (n) + eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); mlx5_glue->free_device_list(ibv_list); - if (!ret) { + if (!n) { DRV_LOG(WARNING, "no Verbs device matches PCI device " PCI_PRI_FMT "," " are kernel drivers loaded?", diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index f55ff4a21..704046270 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -53,6 +53,14 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX5BF = 0xa2d2, }; +/** Switch information returned by mlx5_nl_switch_info(). */ +struct mlx5_switch_info { + uint32_t master:1; /**< Master device. */ + uint32_t representor:1; /**< Representor device. */ + int32_t port_name; /**< Representor port name. */ + uint64_t switch_id; /**< Switch identifier. */ +}; + LIST_HEAD(mlx5_dev_list, priv); /* Shared memory between primary and secondary processes. */ @@ -195,7 +203,8 @@ struct priv { struct mlx5_dev_config config; /* Device configuration. */ struct mlx5_verbs_alloc_ctx verbs_alloc_ctx; /* Context for Verbs allocator. */ - int nl_socket; /* Netlink socket. */ + int nl_socket_rdma; /* Netlink socket (NETLINK_RDMA). */ + int nl_socket_route; /* Netlink socket (NETLINK_ROUTE). */ uint32_t nl_sn; /* Netlink message sequence number. */ }; @@ -342,7 +351,7 @@ int mlx5_socket_connect(struct rte_eth_dev *priv); /* mlx5_nl.c */ -int mlx5_nl_init(uint32_t nlgroups); +int mlx5_nl_init(uint32_t nlgroups, int protocol); int mlx5_nl_mac_addr_add(struct rte_eth_dev *dev, struct ether_addr *mac, uint32_t index); int mlx5_nl_mac_addr_remove(struct rte_eth_dev *dev, struct ether_addr *mac, @@ -351,5 +360,8 @@ void mlx5_nl_mac_addr_sync(struct rte_eth_dev *dev); void mlx5_nl_mac_addr_flush(struct rte_eth_dev *dev); int mlx5_nl_promisc(struct rte_eth_dev *dev, int enable); int mlx5_nl_allmulti(struct rte_eth_dev *dev, int enable); +unsigned int mlx5_nl_ifindex(int nl, const char *name); +int mlx5_nl_switch_info(int nl, unsigned int ifindex, + struct mlx5_switch_info *info); #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_nl.c b/drivers/net/mlx5/mlx5_nl.c index dca85835a..008cd2c31 100644 --- a/drivers/net/mlx5/mlx5_nl.c +++ b/drivers/net/mlx5/mlx5_nl.c @@ -3,10 +3,21 @@ * Copyright 2018 Mellanox Technologies, Ltd */ +#include +#include #include #include +#include +#include +#include +#include +#include +#include +#include #include +#include + #include "mlx5.h" #include "mlx5_utils.h" @@ -27,6 +38,40 @@ ((struct rtattr *)(((char *)(r)) + NLMSG_ALIGN(sizeof(struct ndmsg)))) #endif +/* + * The following definitions are normally found in rdma/rdma_netlink.h, + * however they are so recent that most systems do not expose them yet. + */ +#ifndef HAVE_RDMA_NL_NLDEV +#define RDMA_NL_NLDEV 5 +#endif +#ifndef HAVE_RDMA_NLDEV_CMD_GET +#define RDMA_NLDEV_CMD_GET 1 +#endif +#ifndef HAVE_RDMA_NLDEV_CMD_PORT_GET +#define RDMA_NLDEV_CMD_PORT_GET 5 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_DEV_INDEX +#define RDMA_NLDEV_ATTR_DEV_INDEX 1 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_DEV_NAME +#define RDMA_NLDEV_ATTR_DEV_NAME 2 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_PORT_INDEX +#define RDMA_NLDEV_ATTR_PORT_INDEX 3 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX +#define RDMA_NLDEV_ATTR_NDEV_INDEX 50 +#endif + +/* These are normally found in linux/if_link.h. */ +#ifndef HAVE_IFLA_PHYS_SWITCH_ID +#define IFLA_PHYS_SWITCH_ID 36 +#endif +#ifndef HAVE_IFLA_PHYS_PORT_NAME +#define IFLA_PHYS_PORT_NAME 38 +#endif + /* Add/remove MAC address through Netlink */ struct mlx5_nl_mac_addr { struct ether_addr (*mac)[]; @@ -34,18 +79,27 @@ struct mlx5_nl_mac_addr { int mac_n; /**< Number of addresses in the array. */ }; +/** Data structure used by mlx5_nl_ifindex_cb(). */ +struct mlx5_nl_ifindex_data { + const char *name; /**< IB device name (in). */ + uint32_t ibindex; /**< IB device index (out). */ + uint32_t ifindex; /**< Network interface index (out). */ +}; + /** * Opens a Netlink socket. * * @param nl_groups * Netlink group value (e.g. RTMGRP_LINK). + * @param protocol + * Netlink protocol (e.g. NETLINK_ROUTE, NETLINK_RDMA). * * @return * A file descriptor on success, a negative errno value otherwise and * rte_errno is set. */ int -mlx5_nl_init(uint32_t nl_groups) +mlx5_nl_init(uint32_t nl_groups, int protocol) { int fd; int sndbuf_size = MLX5_SEND_BUF_SIZE; @@ -56,7 +110,7 @@ mlx5_nl_init(uint32_t nl_groups) }; int ret; - fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE); + fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, protocol); if (fd == -1) { rte_errno = errno; return -rte_errno; @@ -334,9 +388,9 @@ mlx5_nl_mac_addr_list(struct rte_eth_dev *dev, struct ether_addr (*mac)[], int ret; uint32_t sn = priv->nl_sn++; - if (priv->nl_socket == -1) + if (priv->nl_socket_route == -1) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; ret = mlx5_nl_request(fd, &req.hdr, sn, &req.ifm, sizeof(struct ifinfomsg)); if (ret < 0) @@ -398,9 +452,9 @@ mlx5_nl_mac_addr_modify(struct rte_eth_dev *dev, struct ether_addr *mac, int ret; uint32_t sn = priv->nl_sn++; - if (priv->nl_socket == -1) + if (priv->nl_socket_route == -1) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; memcpy(RTA_DATA(&req.rta), mac, ETHER_ADDR_LEN); req.hdr.nlmsg_len = NLMSG_ALIGN(req.hdr.nlmsg_len) + RTA_ALIGN(req.rta.rta_len); @@ -569,9 +623,9 @@ mlx5_nl_device_flags(struct rte_eth_dev *dev, uint32_t flags, int enable) int ret; assert(!(flags & ~(IFF_PROMISC | IFF_ALLMULTI))); - if (priv->nl_socket < 0) + if (priv->nl_socket_route < 0) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; ret = mlx5_nl_send(fd, &req.hdr, priv->nl_sn++); if (ret < 0) return ret; @@ -625,3 +679,241 @@ mlx5_nl_allmulti(struct rte_eth_dev *dev, int enable) strerror(rte_errno)); return ret; } + +/** + * Process network interface information from Netlink message. + * + * @param nh + * Pointer to Netlink message header. + * @param arg + * Opaque data pointer for this callback. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_nl_ifindex_cb(struct nlmsghdr *nh, void *arg) +{ + struct mlx5_nl_ifindex_data *data = arg; + size_t off = NLMSG_HDRLEN; + uint32_t ibindex = 0; + uint32_t ifindex = 0; + int found = 0; + + if (nh->nlmsg_type != + RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_GET) && + nh->nlmsg_type != + RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_PORT_GET)) + goto error; + while (off < nh->nlmsg_len) { + struct nlattr *na = (void *)((uintptr_t)nh + off); + void *payload = (void *)((uintptr_t)na + NLA_HDRLEN); + + if (na->nla_len > nh->nlmsg_len - off) + goto error; + switch (na->nla_type) { + case RDMA_NLDEV_ATTR_DEV_INDEX: + ibindex = *(uint32_t *)payload; + break; + case RDMA_NLDEV_ATTR_DEV_NAME: + if (!strcmp(payload, data->name)) + found = 1; + break; + case RDMA_NLDEV_ATTR_NDEV_INDEX: + ifindex = *(uint32_t *)payload; + break; + default: + break; + } + off += NLA_ALIGN(na->nla_len); + } + if (found) { + data->ibindex = ibindex; + data->ifindex = ifindex; + } + return 0; +error: + rte_errno = EINVAL; + return -rte_errno; +} + +/** + * Get index of network interface associated with some IB device. + * + * This is the only somewhat safe method to avoid resorting to heuristics + * when faced with port representors. Unfortunately it requires at least + * Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * + * @return + * A valid (nonzero) interface index on success, 0 otherwise and rte_errno + * is set. + */ +unsigned int +mlx5_nl_ifindex(int nl, const char *name) +{ + static const uint32_t pindex = 1; + uint32_t seq = random(); + struct mlx5_nl_ifindex_data data = { + .name = name, + .ibindex = 0, /* Determined during first pass. */ + .ifindex = 0, /* Determined during second pass. */ + }; + union { + struct nlmsghdr nh; + uint8_t buf[NLMSG_HDRLEN + + NLA_HDRLEN + NLA_ALIGN(sizeof(data.ibindex)) + + NLA_HDRLEN + NLA_ALIGN(sizeof(pindex))]; + } req = { + .nh = { + .nlmsg_len = NLMSG_LENGTH(0), + .nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, + RDMA_NLDEV_CMD_GET), + .nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK | NLM_F_DUMP, + }, + }; + struct nlattr *na; + int ret; + + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret < 0) + return 0; + ret = mlx5_nl_recv(nl, seq, mlx5_nl_ifindex_cb, &data); + if (ret < 0) + return 0; + if (!data.ibindex) + goto error; + ++seq; + req.nh.nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, + RDMA_NLDEV_CMD_PORT_GET); + req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK; + req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(req.buf) - NLMSG_HDRLEN); + na = (void *)((uintptr_t)req.buf + NLMSG_HDRLEN); + na->nla_len = NLA_HDRLEN + sizeof(data.ibindex); + na->nla_type = RDMA_NLDEV_ATTR_DEV_INDEX; + memcpy((void *)((uintptr_t)na + NLA_HDRLEN), + &data.ibindex, sizeof(data.ibindex)); + na = (void *)((uintptr_t)na + NLA_ALIGN(na->nla_len)); + na->nla_len = NLA_HDRLEN + sizeof(pindex); + na->nla_type = RDMA_NLDEV_ATTR_PORT_INDEX; + memcpy((void *)((uintptr_t)na + NLA_HDRLEN), + &pindex, sizeof(pindex)); + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret < 0) + return 0; + ret = mlx5_nl_recv(nl, seq, mlx5_nl_ifindex_cb, &data); + if (ret < 0) + return 0; + if (!data.ifindex) + goto error; + return data.ifindex; +error: + rte_errno = ENODEV; + return 0; +} + +/** + * Process switch information from Netlink message. + * + * @param nh + * Pointer to Netlink message header. + * @param arg + * Opaque data pointer for this callback. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_nl_switch_info_cb(struct nlmsghdr *nh, void *arg) +{ + struct mlx5_switch_info info = { + .master = 0, + .representor = 0, + .port_name = 0, + .switch_id = 0, + }; + size_t off = NLMSG_LENGTH(sizeof(struct ifinfomsg)); + bool port_name_set = false; + bool switch_id_set = false; + + if (nh->nlmsg_type != RTM_NEWLINK) + goto error; + while (off < nh->nlmsg_len) { + struct rtattr *ra = (void *)((uintptr_t)nh + off); + void *payload = RTA_DATA(ra); + char *end; + unsigned int i; + + if (ra->rta_len > nh->nlmsg_len - off) + goto error; + switch (ra->rta_type) { + case IFLA_PHYS_PORT_NAME: + errno = 0; + info.port_name = strtol(payload, &end, 0); + if (errno || + (size_t)(end - (char *)payload) != strlen(payload)) + goto error; + port_name_set = true; + break; + case IFLA_PHYS_SWITCH_ID: + info.switch_id = 0; + for (i = 0; i < RTA_PAYLOAD(ra); ++i) { + info.switch_id <<= 8; + info.switch_id |= ((uint8_t *)payload)[i]; + } + switch_id_set = true; + break; + } + off += RTA_ALIGN(ra->rta_len); + } + info.master = switch_id_set && !port_name_set; + info.representor = switch_id_set && port_name_set; + memcpy(arg, &info, sizeof(info)); + return 0; +error: + rte_errno = EINVAL; + return -rte_errno; +} + +/** + * Get switch information associated with network interface. + * + * @param nl + * Netlink socket of the ROUTE kind (NETLINK_ROUTE). + * @param ifindex + * Network interface index. + * @param[out] info + * Switch information object, populated in case of success. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_nl_switch_info(int nl, unsigned int ifindex, struct mlx5_switch_info *info) +{ + uint32_t seq = random(); + struct { + struct nlmsghdr nh; + struct ifinfomsg info; + } req = { + .nh = { + .nlmsg_len = NLMSG_LENGTH(sizeof(req.info)), + .nlmsg_type = RTM_GETLINK, + .nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK, + }, + .info = { + .ifi_family = AF_UNSPEC, + .ifi_index = ifindex, + }, + }; + int ret; + + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret >= 0) + ret = mlx5_nl_recv(nl, seq, mlx5_nl_switch_info_cb, info); + return ret; +} From patchwork Tue Jul 10 16:04:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42735 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 825711B462; Tue, 10 Jul 2018 18:05:16 +0200 (CEST) Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by dpdk.org (Postfix) with ESMTP id 542611B452 for ; Tue, 10 Jul 2018 18:05:11 +0200 (CEST) Received: by mail-wr1-f65.google.com with SMTP id s11-v6so15192024wra.13 for ; Tue, 10 Jul 2018 09:05:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VhC3Y2HO4ZXiMa/bUy+RgaIjhKfcca8X0XmZin+OFTQ=; b=GaTekbmCBrbDNbVP+nmcisYo/Ry/KOEpzSgjrp6d6kKY3+RvGzUxB4Y+iDmiLo+WtF dVqfTOmo3xXUEg9W2PvuDyZfNI9bTu0m0st8bfBASn66Mc9dCL8f3iL/3VSB50JjOIPO y9YbyaGvHwN6jcUjb70o5csLqfgp+Zf/6NGB9FCF8yribLNMDs6y6du7sjh1I93sO7rL pa394rGSeTGEBMAmX7pblKr6etOrgsXizvsrq9vggyUkiYjRJEXKB9EnZxey4tWw5eWf 8nYPrSyJt1Oz+0A++qeZhaZnyakRCMDo44VbrP3dNs43s+z1c6QtC8xl8kGd9e77zYMt ezPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VhC3Y2HO4ZXiMa/bUy+RgaIjhKfcca8X0XmZin+OFTQ=; b=fgG1TwtsN9H15Dg2tsvAA4t4/aoFiLhSCkGCjOo0vbX5siJfrv3cuUIJ948PMCTNE9 kcNdKsJ7NFuT/GDXwI8QjwghKLjBrVrprTqJTpthdZ2HXO/ItcImKRTLBWzKE9Ihmlv3 JSSp7vIig4gL7Xh9CmxIEZ9VS5DQl9jpXol0rT9jDXzBOCzrDFHdTd7xUsrA09cbCwFK VRPaPaixNmkGzxXilRr5uqKmpc6b1ivTS/5N1U2trLsAne0W1TkEKEtIWwK1Bw5TNXui 5zkb2gr0A24kQwFf4qjyjoAgoekVSlq38g71GCquPyJbBDBRAN37RG4T3H77Xdt1ZcXZ rbSQ== X-Gm-Message-State: APt69E39ZOmBg+IdghwUKzHgM1SUk7XL2nv/obs7Pq5DvUwvHvMutuo9 iqi+zdAiAV9bqtrTYX2YXCu7cQ== X-Google-Smtp-Source: AAOMgpeuVPChlPLjcQf0YtRgZyi6pacBQIX62Cx/fFp6jjBC1t+rpS87ZqYghncXaI17nMF6+A+aAA== X-Received: by 2002:adf:b05c:: with SMTP id g28-v6mr17664983wra.128.1531238710844; Tue, 10 Jul 2018 09:05:10 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id q140-v6sm24473980wmb.35.2018.07.10.09.05.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:10 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:54 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Nelio Laranjeiro , Xueming Li Message-ID: <20180710155743.14448-8-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 07/10] net/mlx5: probe all port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Probe existing port representors in addition to their master device and associate them automatically. To avoid collision between Ethernet devices, they are named as follows: - "{DBDF}" for master/switch devices. - "{DBDF}_representor_{rep}" with "rep" starting from 0 for port representors. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Signed-off-by: Nelio Laranjeiro Reviewed-by: Xueming Li Cc: Xueming Li Cc: Shahaf Shuler --- v4 changes: - Fixed domain ID release once the last port using it is closed. Closed devices are not necessarily detached, their presence is not a good indicator. Code was modified to check if they still use their domain IDs before deciding to release it. v3 changes: - Nelio introduced mlx5_dev_to_port_id() to prevent the master device from releasing a domain ID while representors are still bound. It is now released by the last device closed. - Reverted to original naming convention as requested by Xueming and Shahaf; "net_" prefix and "_0" suffix were dropped. - mlx5_dev_spawn() (previously mlx5_dev_spawn_one()) now decides on its own whether underlying device is a representor. - Devices can now be probed in any order and not necessarily all at once; representors can exist without a master device. - mlx5_pci_probe() iterates on the list of devices directly instead of relying on an intermediate function (previously mlx5_dev_spawn()). - mlx5_get_ifname() was rewritten to rely on mlx5_nl_ifindex() when faced with a representor. - Since it is not necessarily present, master device is now dynamically retrieved in mlx5_dev_infos_get(). v2 changes: - Added representor information to dev_infos_get(). DPDK port ID of master device is now stored in the private structure to retrieve it conveniently. - Master device is assigned dummy representor ID value -1 to better distinguish from the the first actual representor reported by dev_infos_get() as those are indexed from 0. - Added RTE_ETH_DEV_REPRESENTOR device flag. --- drivers/net/mlx5/mlx5.c | 134 ++++++++++++++++++++++++++++-------- drivers/net/mlx5/mlx5.h | 12 +++- drivers/net/mlx5/mlx5_ethdev.c | 133 +++++++++++++++++++++++++++++++---- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_stats.c | 6 +- 5 files changed, 238 insertions(+), 49 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index d06ba9886..c02afbb82 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -307,7 +307,27 @@ mlx5_dev_close(struct rte_eth_dev *dev) if (ret) DRV_LOG(WARNING, "port %u some flows still remain", dev->data->port_id); + if (priv->domain_id != RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) { + unsigned int c = 0; + unsigned int i = mlx5_dev_to_port_id(dev->device, NULL, 0); + uint16_t port_id[i]; + + i = RTE_MIN(mlx5_dev_to_port_id(dev->device, port_id, i), i); + while (i--) { + struct priv *opriv = + rte_eth_devices[port_id[i]].data->dev_private; + + if (!opriv || + opriv->domain_id != priv->domain_id || + &rte_eth_devices[port_id[i]] == dev) + continue; + ++c; + } + if (!c) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); + } memset(priv, 0, sizeof(*priv)); + priv->domain_id = RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID; } const struct eth_dev_ops mlx5_dev_ops = { @@ -647,6 +667,8 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) * Verbs device. * @param vf * If nonzero, enable VF-specific features. + * @param[in] switch_info + * Switch properties of Ethernet device. * * @return * A valid Ethernet device object on success, NULL otherwise and rte_errno @@ -655,7 +677,8 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) static struct rte_eth_dev * mlx5_dev_spawn(struct rte_device *dpdk_dev, struct ibv_device *ibv_dev, - int vf) + int vf, + const struct mlx5_switch_info *switch_info) { struct ibv_context *ctx; struct ibv_device_attr_ex attr; @@ -697,6 +720,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, #endif struct ether_addr mac; char name[RTE_ETH_NAME_MAX_LEN]; + int own_domain_id = 0; + unsigned int i; /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); @@ -805,7 +830,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DEBUG("ibv_query_device_ex() failed"); goto error; } - rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + if (!switch_info->representor) + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + else + snprintf(name, sizeof(name), "%s_representor_%u", + dpdk_dev->name, switch_info->port_name); + DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -874,6 +904,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, goto error; } priv->ctx = ctx; + strncpy(priv->ibdev_name, priv->ctx->device->name, + sizeof(priv->ibdev_name)); strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); priv->device_attr = attr; @@ -883,6 +915,41 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->nl_socket_rdma = mlx5_nl_init(0, NETLINK_RDMA); priv->nl_socket_route = mlx5_nl_init(RTMGRP_LINK, NETLINK_ROUTE); priv->nl_sn = 0; + priv->representor = !!switch_info->representor; + priv->domain_id = RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID; + priv->representor_id = + switch_info->representor ? switch_info->port_name : -1; + /* + * Look for sibling devices in order to reuse their switch domain + * if any, otherwise allocate one. + */ + i = mlx5_dev_to_port_id(dpdk_dev, NULL, 0); + if (i > 0) { + uint16_t port_id[i]; + + i = RTE_MIN(mlx5_dev_to_port_id(dpdk_dev, port_id, i), i); + while (i--) { + const struct priv *opriv = + rte_eth_devices[port_id[i]].data->dev_private; + + if (!opriv || + opriv->domain_id == + RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) + continue; + priv->domain_id = opriv->domain_id; + break; + } + } + if (priv->domain_id == RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) { + err = rte_eth_switch_domain_alloc(&priv->domain_id); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "unable to allocate switch domain: %s", + strerror(rte_errno)); + goto error; + } + own_domain_id = 1; + } err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; @@ -966,6 +1033,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err = ENOMEM; goto error; } + if (priv->representor) + eth_dev->data->dev_flags |= RTE_ETH_DEV_REPRESENTOR; eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; @@ -1084,6 +1153,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, close(priv->nl_socket_route); if (priv->nl_socket_rdma >= 0) close(priv->nl_socket_rdma); + if (own_domain_id) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); rte_free(priv); } if (pd) @@ -1100,7 +1171,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, /** * DPDK callback to register a PCI device. * - * This function spawns an Ethernet device out of a given PCI device. + * This function spawns Ethernet devices out of a given PCI device. * * @param[in] pci_drv * PCI driver structure (mlx5_driver). @@ -1115,7 +1186,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) { struct ibv_device **ibv_list; - struct rte_eth_dev *eth_dev = NULL; unsigned int n = 0; int vf; int ret; @@ -1150,9 +1220,11 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int ifindex[n]; struct mlx5_switch_info info[n]; + struct rte_eth_dev *eth_list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; + unsigned int u; /* * The existence of several matching entries (n > 1) means port @@ -1187,28 +1259,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_rdma); if (nl_route >= 0) close(nl_route); - /* Look for master device. */ - for (i = 0; i != n; ++i) { - if (!info[i].master) - continue; - /* Make it the first entry. */ - if (i == 0) - break; - ibv_match[n] = ibv_match[0]; - ibv_match[0] = ibv_match[i]; - ibv_match[n] = NULL; - break; - } - if (n && i == n) { - if (n == 1 && !info[0].representor) { + /* Count unidentified devices. */ + for (u = 0, i = 0; i != n; ++i) + if (!info[i].master && !info[i].representor) + ++u; + if (u) { + if (n == 1 && u == 1) { /* Case #2. */ DRV_LOG(INFO, "no switch support detected"); - } else if (n == 1) { - /* Case #3. */ - DRV_LOG(ERR, - "device looks like a port representor, this is" - " not supported yet"); - n = 0; } else { /* Case #3. */ DRV_LOG(ERR, @@ -1227,8 +1285,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, default: vf = 0; } - if (n) - eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); + for (i = 0; i != n; ++i) { + uint32_t restore; + + eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], + vf, &info[i]); + if (!eth_list[i]) + break; + restore = eth_list[i]->data->dev_flags; + rte_eth_copy_pci_info(eth_list[i], pci_dev); + /* Restore non-PCI flags cleared by the above call. */ + eth_list[i]->data->dev_flags |= restore; + rte_eth_dev_probing_finish(eth_list[i]); + } mlx5_glue->free_device_list(ibv_list); if (!n) { DRV_LOG(WARNING, @@ -1238,7 +1307,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid, pci_dev->addr.function); rte_errno = ENOENT; ret = -rte_errno; - } else if (!eth_dev) { + } else if (i != n) { DRV_LOG(ERR, "probe of PCI device " PCI_PRI_FMT " aborted after" " encountering an error: %s", @@ -1246,9 +1315,16 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid, pci_dev->addr.function, strerror(rte_errno)); ret = -rte_errno; + /* Roll back. */ + while (i--) { + mlx5_dev_close(eth_list[i]); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_free(eth_list[i]->data->dev_private); + claim_zero(rte_eth_dev_release_port(eth_list[i])); + } + /* Restore original error. */ + rte_errno = -ret; } else { - rte_eth_copy_pci_info(eth_dev, pci_dev); - rte_eth_dev_probing_finish(eth_dev); ret = 0; } return ret; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 704046270..cc01310e0 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -159,6 +159,7 @@ struct priv { struct ibv_context *ctx; /* Verbs context. */ struct ibv_device_attr_ex device_attr; /* Device properties. */ struct ibv_pd *pd; /* Protection Domain. */ + char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */ char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for secondary */ struct ether_addr mac[MLX5_MAX_MAC_ADDRESSES]; /* MAC addresses. */ BITFIELD_DECLARE(mac_own, uint64_t, MLX5_MAX_MAC_ADDRESSES); @@ -168,6 +169,9 @@ struct priv { /* Device properties. */ uint16_t mtu; /* Configured MTU. */ unsigned int isolated:1; /* Whether isolated mode is enabled. */ + unsigned int representor:1; /* Device is a port representor. */ + uint16_t domain_id; /* Switch domain identifier. */ + int32_t representor_id; /* Port representor identifier. */ /* RX/TX queues. */ unsigned int rxqs_n; /* RX queues array size. */ unsigned int txqs_n; /* TX queues array size. */ @@ -217,9 +221,12 @@ int mlx5_getenv_int(const char *); /* mlx5_ethdev.c */ +int mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]); int mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]); int mlx5_ifindex(const struct rte_eth_dev *dev); -int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr); +int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master); int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu); int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags); @@ -244,6 +251,9 @@ int mlx5_set_link_up(struct rte_eth_dev *dev); int mlx5_is_removed(struct rte_eth_dev *dev); eth_tx_burst_t mlx5_select_tx_function(struct rte_eth_dev *dev); eth_rx_burst_t mlx5_select_rx_function(struct rte_eth_dev *dev); +unsigned int mlx5_dev_to_port_id(const struct rte_device *dev, + uint16_t *port_list, + unsigned int port_list_n); /* mlx5_mac.c */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 819f5baad..05f66f7b6 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -27,6 +27,7 @@ #include #include +#include #include #include #include @@ -93,7 +94,7 @@ struct ethtool_link_settings { #endif /** - * Get interface name from private structure. + * Get master interface name from private structure. * * @param[in] dev * Pointer to Ethernet device. @@ -104,7 +105,8 @@ struct ethtool_link_settings { * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]) { struct priv *priv = dev->data->dev_private; DIR *dir; @@ -179,6 +181,39 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) } /** + * Get interface name from private structure. + * + * This is a port representor-aware version of mlx5_get_master_ifname(). + * + * @param[in] dev + * Pointer to Ethernet device. + * @param[out] ifname + * Interface name output buffer. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +{ + struct priv *priv = dev->data->dev_private; + unsigned int ifindex = + priv->nl_socket_rdma >= 0 ? + mlx5_nl_ifindex(priv->nl_socket_rdma, priv->ibdev_name) : 0; + + if (!ifindex) { + if (!priv->representor) + return mlx5_get_master_ifname(dev, ifname); + rte_errno = ENXIO; + return -rte_errno; + } + if (if_indextoname(ifindex, &(*ifname)[0])) + return 0; + rte_errno = errno; + return -rte_errno; +} + +/** * Get the interface index from device name. * * @param[in] dev @@ -214,12 +249,16 @@ mlx5_ifindex(const struct rte_eth_dev *dev) * Request number to pass to ioctl(). * @param[out] ifr * Interface request structure output buffer. + * @param master + * When device is a port representor, perform request on master device + * instead. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) +mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master) { int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP); int ret = 0; @@ -228,7 +267,10 @@ mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) rte_errno = errno; return -rte_errno; } - ret = mlx5_get_ifname(dev, &ifr->ifr_name); + if (master) + ret = mlx5_get_master_ifname(dev, &ifr->ifr_name); + else + ret = mlx5_get_ifname(dev, &ifr->ifr_name); if (ret) goto error; ret = ioctl(sock, req, ifr); @@ -258,7 +300,7 @@ int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request); + int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request, 0); if (ret) return ret; @@ -282,7 +324,7 @@ mlx5_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) { struct ifreq request = { .ifr_mtu = mtu, }; - return mlx5_ifreq(dev, SIOCSIFMTU, &request); + return mlx5_ifreq(dev, SIOCSIFMTU, &request, 0); } /** @@ -302,13 +344,13 @@ int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request); + int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request, 0); if (ret) return ret; request.ifr_flags &= keep; request.ifr_flags |= flags & ~keep; - return mlx5_ifreq(dev, SIOCSIFFLAGS, &request); + return mlx5_ifreq(dev, SIOCSIFFLAGS, &request, 0); } /** @@ -477,6 +519,30 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) info->speed_capa = priv->link_speed_capa; info->flow_type_rss_offloads = ~MLX5_RSS_HF_MASK; mlx5_set_default_params(dev, info); + info->switch_info.name = dev->data->name; + info->switch_info.domain_id = priv->domain_id; + info->switch_info.port_id = priv->representor_id; + if (priv->representor) { + unsigned int i = mlx5_dev_to_port_id(dev->device, NULL, 0); + uint16_t port_id[i]; + + i = RTE_MIN(mlx5_dev_to_port_id(dev->device, port_id, i), i); + while (i--) { + struct priv *opriv = + rte_eth_devices[port_id[i]].data->dev_private; + + if (!opriv || + opriv->representor || + opriv->domain_id != priv->domain_id) + continue; + /* + * Override switch name with that of the master + * device. + */ + info->switch_info.name = opriv->dev_data->name; + break; + } + } } /** @@ -540,7 +606,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, int link_speed = 0; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -550,7 +616,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&edata; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GSET) failed: %s", @@ -611,7 +677,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, uint64_t sc; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -621,7 +687,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&gcmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -638,7 +704,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, *ecmd = gcmd; ifr.ifr_data = (void *)ecmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -801,7 +867,7 @@ mlx5_dev_get_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) int ret; ifr.ifr_data = (void *)ðpause; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GPAUSEPARAM) failed:" @@ -854,7 +920,7 @@ mlx5_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) ethpause.tx_pause = 1; else ethpause.tx_pause = 0; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 0); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_SPAUSEPARAM)" @@ -1193,3 +1259,40 @@ mlx5_is_removed(struct rte_eth_dev *dev) return 1; return 0; } + +/** + * Get port ID list of mlx5 instances sharing a common device. + * + * @param[in] dev + * Device to look for. + * @param[out] port_list + * Result buffer for collected port IDs. + * @param port_list_n + * Maximum number of entries in result buffer. If 0, @p port_list can be + * NULL. + * + * @return + * Number of matching instances regardless of the @p port_list_n + * parameter, 0 if none were found. + */ +unsigned int +mlx5_dev_to_port_id(const struct rte_device *dev, uint16_t *port_list, + unsigned int port_list_n) +{ + uint16_t id; + unsigned int n = 0; + + RTE_ETH_FOREACH_DEV(id) { + struct rte_eth_dev *ldev = &rte_eth_devices[id]; + + if (!ldev->device || + !ldev->device->driver || + strcmp(ldev->device->driver->name, MLX5_DRIVER_NAME) || + ldev->device != dev) + continue; + if (n < port_list_n) + port_list[n] = id; + n++; + } + return n; +} diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index 672a47619..12ee37f55 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -49,7 +49,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[ETHER_ADDR_LEN]) struct ifreq request; int ret; - ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request); + ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request, 0); if (ret) return ret; memcpy(mac, request.ifr_hwaddr.sa_data, ETHER_ADDR_LEN); diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c index 875dd1027..91f3d474a 100644 --- a/drivers/net/mlx5/mlx5_stats.c +++ b/drivers/net/mlx5/mlx5_stats.c @@ -146,7 +146,7 @@ mlx5_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) et_stats->cmd = ETHTOOL_GSTATS; et_stats->n_stats = xstats_ctrl->stats_n; ifr.ifr_data = (caddr_t)et_stats; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to read statistic values from device", @@ -194,7 +194,7 @@ mlx5_ethtool_get_stats_n(struct rte_eth_dev *dev) { drvinfo.cmd = ETHTOOL_GDRVINFO; ifr.ifr_data = (caddr_t)&drvinfo; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to query number of statistics", dev->data->port_id); @@ -244,7 +244,7 @@ mlx5_xstats_init(struct rte_eth_dev *dev) strings->string_set = ETH_SS_STATS; strings->len = dev_stats_n; ifr.ifr_data = (caddr_t)strings; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to get statistic names", dev->data->port_id); From patchwork Tue Jul 10 16:04:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42736 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 369821B492; Tue, 10 Jul 2018 18:05:21 +0200 (CEST) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 3ED8F1B451 for ; Tue, 10 Jul 2018 18:05:13 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id w16-v6so25076292wmc.2 for ; Tue, 10 Jul 2018 09:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=OmHvo6RLnUaNd4Sjyh3IZbSD0DPZ4X0BTUeC/oEEqTCmtsaLdPH755/0hJ0DsUVer0 ZLKW1yeq77i1kFUHkFQ/pQyYnXEu+l0PlEhEmt4qiq5BZuLdeTriGmCRo0ma2OM+dL5z W0xMItesVVe21JyMtrniEFy7HWbpe6T2M6AxoKvVDjFA4Gqm9NRvSKs0RbU37ZkUpyb2 K94kxn8rYeVP7DUm76Xc6ltl6bQTOSka9MFgxDpgeVN43DsnJsYSR7EW+kMTjrDbDjHO avVEVxSzXrYExvuVMTAu3gIUy8aemAkXPLaalyoIDVCZbvKcrsfToW1Z4AIs7feBzhgT gozQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=DYUvg+KvseV1JBBz+0Zr/k/GVpYs06qM8L+3d2+IkJJd2N6NODbAiQlrNsJEPJZFKo HoPuhbVzA4+mbK5RMTybrv/3RDIGLif5enpqJzTI/P80OYuvmifWnZFmFbUFz00qS4zw OLJwjK+kyynIYsTKGvWB03TNtuhkKIs/GkP+s71hLjiSvPxAry0Lfj7QpwWEJt0+dazN V86nSzDWVcOYBVE/cKZLNpASPL5pGHHkEDuOHdepfYmHlDp9fVDI53G445EAq+BOQDKF 2CBE7GHdU6SjCmwWSu8IyuvVuS13FCykJiwSDtUQPqIY6jI+I/ibIoMkXhdplELF8fcX MJzQ== X-Gm-Message-State: APt69E2uk/xpcKMxqAFKaWCoGFoqtr9RS6H4W40ySA8dLcHkRWG9m2Rt rtNv5JxJljPA7vAyOIhrPPKriA== X-Google-Smtp-Source: AAOMgpco+PC6wh315ozjEhpr4Zm02//xY99hHEfPr3+drSZA2F58NqaiH130mI+0QyssUl/rTT6G4Q== X-Received: by 2002:a1c:a341:: with SMTP id m62-v6mr17216787wme.3.1531238712859; Tue, 10 Jul 2018 09:05:12 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id r194-v6sm28424052wmd.36.2018.07.10.09.05.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:12 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:56 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-9-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 08/10] net/mlx5: probe port representors in natural order X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Port representors are probed in whatever unspecified order ibv_get_device_list() returns them. This is counterintuitive to users since DPDK port IDs assignment almost never follows the same sequence as representor IDs. Additionally, the master device does not necessarily inherit the lowest DPDK port ID. Signed-off-by: Adrien Mazarguil --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 95 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c02afbb82..6592480bf 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1168,6 +1168,52 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, return NULL; } +/** Data associated with devices to spawn. */ +struct mlx5_dev_spawn_data { + unsigned int ifindex; /**< Network interface index. */ + struct mlx5_switch_info info; /**< Switch information. */ + struct ibv_device *ibv_dev; /**< Associated IB device. */ + struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ +}; + +/** + * Comparison callback to sort device data. + * + * This is meant to be used with qsort(). + * + * @param a[in] + * Pointer to pointer to first data object. + * @param b[in] + * Pointer to pointer to second data object. + * + * @return + * 0 if both objects are equal, less than 0 if the first argument is less + * than the second, greater than 0 otherwise. + */ +static int +mlx5_dev_spawn_data_cmp(const void *a, const void *b) +{ + const struct mlx5_switch_info *si_a = + &((const struct mlx5_dev_spawn_data *)a)->info; + const struct mlx5_switch_info *si_b = + &((const struct mlx5_dev_spawn_data *)b)->info; + int ret; + + /* Master device first. */ + ret = si_b->master - si_a->master; + if (ret) + return ret; + /* Then representor devices. */ + ret = si_b->representor - si_a->representor; + if (ret) + return ret; + /* Unidentified devices come last in no specific order. */ + if (!si_a->representor) + return 0; + /* Order representors by name. */ + return si_a->port_name - si_b->port_name; +} + /** * DPDK callback to register a PCI device. * @@ -1218,9 +1264,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } ibv_match[n] = NULL; - unsigned int ifindex[n]; - struct mlx5_switch_info info[n]; - struct rte_eth_dev *eth_list[n]; + struct mlx5_dev_spawn_data list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; @@ -1242,16 +1286,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * bail out. */ for (i = 0; i != n; ++i) { + list[i].ibv_dev = ibv_match[i]; + list[i].eth_dev = NULL; if (nl_rdma < 0) - ifindex[i] = 0; + list[i].ifindex = 0; else - ifindex[i] = mlx5_nl_ifindex(nl_rdma, - ibv_match[i]->name); + list[i].ifindex = mlx5_nl_ifindex + (nl_rdma, list[i].ibv_dev->name); if (nl_route < 0 || - !ifindex[i] || - mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { - ifindex[i] = 0; - memset(&info[i], 0, sizeof(info[i])); + !list[i].ifindex || + mlx5_nl_switch_info(nl_route, list[i].ifindex, + &list[i].info)) { + list[i].ifindex = 0; + memset(&list[i].info, 0, sizeof(list[i].info)); continue; } } @@ -1261,7 +1308,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_route); /* Count unidentified devices. */ for (u = 0, i = 0; i != n; ++i) - if (!info[i].master && !info[i].representor) + if (!list[i].info.master && !list[i].info.representor) ++u; if (u) { if (n == 1 && u == 1) { @@ -1275,6 +1322,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, n = 0; } } + /* + * Sort list to probe devices in natural order for users convenience + * (i.e. master first, then representors from lowest to highest ID). + */ + if (n) + qsort(list, n, sizeof(*list), mlx5_dev_spawn_data_cmp); switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1288,15 +1341,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, for (i = 0; i != n; ++i) { uint32_t restore; - eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], - vf, &info[i]); - if (!eth_list[i]) + list[i].eth_dev = mlx5_dev_spawn + (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); + if (!list[i].eth_dev) break; - restore = eth_list[i]->data->dev_flags; - rte_eth_copy_pci_info(eth_list[i], pci_dev); + restore = list[i].eth_dev->data->dev_flags; + rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ - eth_list[i]->data->dev_flags |= restore; - rte_eth_dev_probing_finish(eth_list[i]); + list[i].eth_dev->data->dev_flags |= restore; + rte_eth_dev_probing_finish(list[i].eth_dev); } mlx5_glue->free_device_list(ibv_list); if (!n) { @@ -1317,10 +1370,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { - mlx5_dev_close(eth_list[i]); + mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_free(eth_list[i]->data->dev_private); - claim_zero(rte_eth_dev_release_port(eth_list[i])); + rte_free(list[i].eth_dev->data->dev_private); + claim_zero(rte_eth_dev_release_port(list[i].eth_dev)); } /* Restore original error. */ rte_errno = -ret; From patchwork Tue Jul 10 16:04:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42737 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D11721B4B6; Tue, 10 Jul 2018 18:05:22 +0200 (CEST) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 17BB21B432 for ; Tue, 10 Jul 2018 18:05:15 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id z13-v6so24944486wma.5 for ; Tue, 10 Jul 2018 09:05:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=4qelGfXDR0NYqw3vkex8nX1rMGAhfI/j9iDIJOyh370=; b=vMvyEuUqfyZKKilPgKV2A1kSJ+qk6znFDtL2SmHDnQ/AFNduRbmnT2+gHqejZoSkDV aHUQRAjtjtuOLzPAyiVkxi7txcD12Ht793466wwlf0v2in91DAEqTBQYxy2iBEh44jpl OyM/5Ne3xLfB9tA5b1BOqhZS+ZifNThhO9o/wzu9qQBpr/iOFjgLFO9cvwo2rkY7+BlN Zs+rEGy27OKQUEJsyfjXBVhgcZBgzlv/HO1UKhaK/pYQBOjyrCV/1lcwvRq7D+YWR6H7 cvtp8oWkQV9gANOUhwmHbaPojohUgbkfwz1soy3vc6YO1PbtV3HflXIY+FdGN6i73daL 9xVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=4qelGfXDR0NYqw3vkex8nX1rMGAhfI/j9iDIJOyh370=; b=Yt4rrYSCPjRde919yGJiPeF7wN0nuDg+wIw7GJnnBBN1OuOCoYzY+CAvsIpF18ZxPA slq8V1x6CS2rCWgisSv6PeAwx3N2ST8L2C7fDK3+x7wPq7Y6uS2+kvLkWIDYazGNjnf6 RLZvHAI8dHfDm2Tkkp65G8mssQurWsTcv9H7HjVXg04sunRlpBUrYT9GkCz4lSOlWAVC ja5bS+H+4cgTdmEReq2c+zY81CS9PFV7+Gwa/O9aKDvhdUvJLOzrcSwP7ycGc1Li+/yW j9jnrp1Sfyny8EN3b7hgMQalvlEazr5QY5fBt3Eo1+dp2GU50iGs5qlw6wNFpN5ynpZZ 8eqg== X-Gm-Message-State: AOUpUlH/LLCYrmILks6P4ra8eGpcaE8jHbWvldSKmNYF1ARRrZyJ4Mzm REoB76WGEtgdMUvnCeRnOk5GHQ== X-Google-Smtp-Source: AAOMgpdcPqI58HCCplO803zI/qn1ANWoVJtzesZbZZ65EG6ZU6RsrqvBp3R9B/echQh72+1BegjkUg== X-Received: by 2002:a1c:f415:: with SMTP id z21-v6mr3243811wma.80.1531238714808; Tue, 10 Jul 2018 09:05:14 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id s17-v6sm13747678wmc.34.2018.07.10.09.05.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:14 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:58 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-10-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 09/10] net/mlx5: add parameter for port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Prior to this patch, all port representors detected on a given device were probed and Ethernet devices instantiated for each of them. This patch adds support for the standard "representor" parameter, which implies that port representors are not probed by default anymore, except for the list provided through device arguments. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v3 changes: - Adapted representor detection to the reworked mlx5_dev_spawn(). v2 changes: - Added error message for when rte_eth_devargs_parse() fails. --- doc/guides/nics/mlx5.rst | 12 ++++++++ doc/guides/prog_guide/poll_mode_drv.rst | 2 ++ drivers/net/mlx5/mlx5.c | 41 ++++++++++++++++++++++++++-- 3 files changed, 52 insertions(+), 3 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 7dd9c1c5e..0d0d21727 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -392,6 +392,18 @@ Run-time configuration Disabled by default. +- ``representor`` parameter [list] + + This parameter can be used to instantiate DPDK Ethernet devices from + existing port (or VF) representors configured on the device. + + It is a standard parameter whose format is described in + :ref:`ethernet_device_standard_device_arguments`. + + For instance, to probe port representors 0 through 2:: + + representor=[0-2] + Firmware configuration ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index 4b69f6cbe..b2cf48354 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -360,6 +360,8 @@ Ethernet Device API The Ethernet device API exported by the Ethernet PMDs is described in the *DPDK API Reference*. +.. _ethernet_device_standard_device_arguments: + Ethernet Device Standard Device Arguments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 6592480bf..12a77afa8 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -92,6 +92,9 @@ /* Activate Netlink support in VF mode. */ #define MLX5_VF_NL_EN "vf_nl_en" +/* Select port representors to instantiate. */ +#define MLX5_REPRESENTOR "representor" + #ifndef HAVE_IBV_MLX5_MOD_MPW #define MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED (1 << 2) #define MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW (1 << 3) @@ -443,6 +446,9 @@ mlx5_args_check(const char *key, const char *val, void *opaque) struct mlx5_dev_config *config = opaque; unsigned long tmp; + /* No-op, port representors are processed in mlx5_dev_spawn(). */ + if (!strcmp(MLX5_REPRESENTOR, key)) + return 0; errno = 0; tmp = strtoul(val, NULL, 0); if (errno) { @@ -515,6 +521,7 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_RX_VEC_EN, MLX5_L3_VXLAN_EN, MLX5_VF_NL_EN, + MLX5_REPRESENTOR, NULL, }; struct rte_kvargs *kvlist; @@ -672,7 +679,9 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) * * @return * A valid Ethernet device object on success, NULL otherwise and rte_errno - * is set. + * is set. The following error is defined: + * + * EBUSY: device is not supposed to be spawned. */ static struct rte_eth_dev * mlx5_dev_spawn(struct rte_device *dpdk_dev, @@ -723,6 +732,26 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, int own_domain_id = 0; unsigned int i; + /* Determine if this port representor is supposed to be spawned. */ + if (switch_info->representor && dpdk_dev->devargs) { + struct rte_eth_devargs eth_da; + + err = rte_eth_devargs_parse(dpdk_dev->devargs->args, ð_da); + if (err) { + rte_errno = -err; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + return NULL; + } + for (i = 0; i < eth_da.nb_representor_ports; ++i) + if (eth_da.representor_ports[i] == + (uint16_t)switch_info->port_name) + break; + if (i == eth_da.nb_representor_ports) { + rte_errno = EBUSY; + return NULL; + } + } /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); errno = 0; @@ -1343,8 +1372,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, list[i].eth_dev = mlx5_dev_spawn (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); - if (!list[i].eth_dev) - break; + if (!list[i].eth_dev) { + if (rte_errno != EBUSY) + break; + /* Device is disabled, ignore it. */ + continue; + } restore = list[i].eth_dev->data->dev_flags; rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ @@ -1370,6 +1403,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { + if (!list[i].eth_dev) + continue; mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) rte_free(list[i].eth_dev->data->dev_private); From patchwork Tue Jul 10 16:05:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42738 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E7DB11B4C1; Tue, 10 Jul 2018 18:05:24 +0200 (CEST) Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by dpdk.org (Postfix) with ESMTP id DF0541B3A8 for ; Tue, 10 Jul 2018 18:05:16 +0200 (CEST) Received: by mail-wr1-f65.google.com with SMTP id s11-v6so15192333wra.13 for ; Tue, 10 Jul 2018 09:05:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=AtLQ1t/HSGgaM2vxggtceXSTF01jOlclzYI1rcVg0bU=; b=Tgrrcn+ncb/3SAOox2yW/phSXe5ZD9sXOCAzX2YPwvtLUxHl9csfJNyOvRlkyiEW4I pvoPwDQhSLa/FZacQzKIvlnHP7b8SV4vTJUnDiT/cxPltOuAFNEQutV+XL1vAQM7yrLu 5g+EiRMladQSXyWowaBbJCREUngzBOEHQPOmfqZN1Idz99sV9gJ64JGAgrBLIPJ8UqEH 78pL1zNve/7mJJdCafUesJVep5rz0FTSgAKTzOd8Tl0FZ0jfK/B4+L5rWWSkaFH93F8F CG1xgioKQSPsnVbzdpRvfUDWC3uDpxCR3oNXtC+LXBzPFDH2LTTKr7jbgD4+KLUCgrER 7vRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=AtLQ1t/HSGgaM2vxggtceXSTF01jOlclzYI1rcVg0bU=; b=ez1i5rGzg4/nfNIxLYWV8pxwjH73Lb14EERSI5j+mQNZdZ6kQqNJd8ehrdo3qdlNwJ U3STt1zmNxU14WI5zIPDUhzyQDQo8VwVKCgbQe8TNELTc8hO9XLmAqkRqVOdXPDKi6jP fAQKaJEBaqr41nsK9d1/2xcHfyEMEn34HYa3P76euYXxLEh+mwjsOc0tTME4H+Bv1dhn fCIKNp/HFdmfUVAdB7yWqqRDyNY36Vz0ispl0oGQbuiW/W8n8ED8VZUlkOLwJhe3FCm7 nH02uAmCpkDgVwThMQ8CELHMTLYHaebCzVAP2IimxK1AsU4zHSLmfM7l1AYE0IRAuDNA 92/Q== X-Gm-Message-State: APt69E0gJ2zi2g1YSrmUr0ErZbDIWvbF5Ojlm/F/5HPw34dnVvpTuKrC NUDh8+t3NSaMi5/G9sbrGVBgvw== X-Google-Smtp-Source: AAOMgpccD3HekVq3+M1+a7Cw3oxW9ThAOIejh4T+PtlWvNUWxa8dt7EbZy9XpTsws1GD5GZwbNmGZg== X-Received: by 2002:adf:fdcd:: with SMTP id i13-v6mr17452602wrs.276.1531238716689; Tue, 10 Jul 2018 09:05:16 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id w204-v6sm19120680wmw.17.2018.07.10.09.05.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:16 -0700 (PDT) Date: Tue, 10 Jul 2018 18:05:00 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-11-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 10/10] net/mlx5: support negative identifiers for port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch brings support for BlueField representors. Signed-off-by: Adrien Mazarguil Cc: Shahaf Shuler --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 12a77afa8..df7f39844 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1330,6 +1330,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, memset(&list[i].info, 0, sizeof(list[i].info)); continue; } + /* + * Port representors not associated with any VFs (e.g. on + * BlueField devices) report -1 as a port identifier. + * Quietly set it to zero since DPDK only supports positive + * values. + */ + if (list[i].info.representor && list[i].info.port_name == -1) + list[i].info.port_name = 0; } if (nl_rdma >= 0) close(nl_rdma);