From patchwork Thu Jul 5 08:45:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42302 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 222521C12C; Thu, 5 Jul 2018 10:45:44 +0200 (CEST) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id C69A61C11E for ; Thu, 5 Jul 2018 10:45:42 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id w16-v6so9654749wmc.2 for ; Thu, 05 Jul 2018 01:45:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=QLWEKxuqhF2sx7AXNTonfLHxyzB3mWcORADPDgYW6JM=; b=m3sx7lhfjNt96gHoAUDrQpY+r+5/6zjwzOFv0eS3mh4RKP/9I441ag/AoxXQeYonGV ePRaR7mvezAE6d0w5uBU9VQakQoMo1YJ8QWlrqiFdEKpw3HCM1j/cKtmMX1wgbeiW89f WxJtJTnTdBWxn6xe23LCXXS3QODRfLQLKs/ULORxDKC7QMZLCOvlciSFDebzge/BmzSj 8PI9lSTJ8kSCeUl8IaoKPCeS2yBe1gupBcEOARaKdjskaAIkGqXB8jaCV2E/H7BpOZkc sXEj/IOiVkTUhJCl+BjigKJrRudKmVtcY7pbHxCB5eYoUKEveIRu6yJoynrjgrGeLJ/5 rWxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=QLWEKxuqhF2sx7AXNTonfLHxyzB3mWcORADPDgYW6JM=; b=SixjfKDHxWCARAzj7HpvN2/r3rIhwuW18/dxeZJPQTSmaMriSCh477E+gzstCwy/IA Te6b/jQZeuUDIm2K+EAHCav9NtXHSdlFqaoqh+8zuJ4B9Ekr08lKykKUJSREcjJtDIbt NzLuyefj2OXdmZXTs8ecHa7up6hCAfpBSQBFsiikwijIzuR8oNSOxgv1npdOvt+71iSk PD/84KVjfzSuxIgv9gjAJ1iHkMN7rA5jPtsDZoz6LkOZ9fWg9XUf+mFIBzAq3T4UMcvB qmyodXlkYPx3kvygf3TR5oIsK6JpT/KKVlAgqsCD3WG8EhVtQ7Vv9GlvHxKTOJxSUmP/ rNSQ== X-Gm-Message-State: APt69E189fRAFDXa86BSC0CXcy2Vy63uIc4fnpw8RLZaDSkTdC3JTTz0 U8MTRlYV/OdIvyvqmZ9S721ujg== X-Google-Smtp-Source: AAOMgpdR/YNkqVSlyOrtauCyOjRF9MzzUT4/hVOrYecz2KpVIZo5VZIPGXlsOkwtBas1vRkwWfA2XA== X-Received: by 2002:a5d:4452:: with SMTP id x18-v6mr4124844wrr.227.1530780342581; Thu, 05 Jul 2018 01:45:42 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id s11-v6sm3752301wrm.42.2018.07.05.01.45.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:41 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:25 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-2-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 01/10] net/mlx5: rename confusing object in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" There are several attribute objects in this function: - IB device attributes (struct ibv_device_attr_ex device_attr). - Direct Verbs attributes (struct mlx5dv_context attrs_out). - Port attributes (struct ibv_port_attr). - IB device attributes again (struct ibv_device_attr_ex device_attr_ex). "attrs_out" is both odd and initialized using a nonstandard syntax. Rename it "dv_attr" for consistency. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Fixed ctx -> attr_ctx in mlx5_pci_probe(). --- drivers/net/mlx5/mlx5.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index d081bdd05..22cbce8d5 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,6 +654,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; struct ibv_context *attr_ctx = NULL; struct ibv_device_attr_ex device_attr; @@ -670,7 +671,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; int i; - struct mlx5dv_context attrs_out = {0}; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif @@ -736,21 +736,21 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ibv_dev = list[i]; DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif /* * Multi-packet send is supported by ConnectX-4 Lx PF as well * as all ConnectX-5 devices. */ #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &attrs_out); - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { + mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); mps = MLX5_MPW_ENHANCED; } else { @@ -762,14 +762,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mps = MLX5_MPW_DISABLED; } #ifdef HAVE_IBV_MLX5_MOD_SWP - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_SWP) - swp = attrs_out.sw_parsing_caps.sw_parsing_offloads; + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) + swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = - attrs_out.striding_rq_caps; + dv_attr.striding_rq_caps; DRV_LOG(DEBUG, "\tmin_single_stride_log_num_of_bytes: %d", mprq_caps.min_single_stride_log_num_of_bytes); @@ -794,15 +794,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } #endif if (RTE_CACHE_LINE_SIZE == 128 && - !(attrs_out.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) + !(dv_attr.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) cqe_comp = 0; else cqe_comp = 1; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { - tunnel_en = ((attrs_out.tunnel_offloads_caps & + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { + tunnel_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_VXLAN) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GRE)); } DRV_LOG(DEBUG, "tunnel offloading is %ssupported", @@ -812,9 +812,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT - mpls_en = ((attrs_out.tunnel_offloads_caps & + mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP)); DRV_LOG(DEBUG, "MPLS over GRE/UDP tunnel offloading is %ssupported", mpls_en ? "" : "not "); From patchwork Thu Jul 5 08:45:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42303 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B18B01C37A; Thu, 5 Jul 2018 10:45:46 +0200 (CEST) Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by dpdk.org (Postfix) with ESMTP id C176F1C134 for ; Thu, 5 Jul 2018 10:45:44 +0200 (CEST) Received: by mail-wm0-f54.google.com with SMTP id 69-v6so9430434wmf.3 for ; Thu, 05 Jul 2018 01:45:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=LZ7BxY/4owWBpMHTwS+IFzwvWnorKYfJ62epOi5lr6Y=; b=MGkgEC1NwahU9VPIhlbiUufm4yecSimKF1yjHwpAdjZUHPHVqjbmnRVqo+E0/jocSB uqGqI4RGhnKXkLKCNbk/S4CiE2ZFxREjkks598owL8Gu1IO0k7DgjQsl+Feyq25f6gYh ckhOIpGA0jcPHk9D3cQzk8m0Lu9Axdnw927UeshmElebkMlC1aU5gs+BakipNhwu98js Bjx88558l4avtoQlET9zVC6eD6FqJJr9ZCbiwZyyNkH4kRt52kehc4RkLNvVR1qY+15B 0709Gi+0I98n+Jq9D8mJl/nziy9zbN+a1B7g3dkvOxLlHrAjYG2YNi5qV/ihubCUxruj arTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=LZ7BxY/4owWBpMHTwS+IFzwvWnorKYfJ62epOi5lr6Y=; b=X1C8jSIcWRTaCKVv8Sy+un6DTapuz57REtQRC8Ksp3GAlPpO+zClJSSLvf+OzI4xpH /G+t1t/OOIZ3nyTTKdCo8jvT3ezuyJeSSg3okMyFLu4N/igtnNP6E725B//7AfWPbgL2 4qoY3+WNy+oEi8nM0BS6pzpbudwJoqZ1XcrsUAOYtedeQoWN0qwOTNqFz3/tYseaU0ud 0fE0rM1M52Wx97GAZKdmu4L8SwemGz4O8zAD3pxwpbFqC0F/XiefRuJvm/ykCgRP73ZZ 5Fncj9Oa+FwZ+z697jWDVjeehfFKrtFo908wXayWdSf0ZtGct3e4OW1vlhCsdzx85dV1 /JqQ== X-Gm-Message-State: APt69E2Aduf/Hm+rAqlYY6LXKv8DE6pEK6QMpZIw+3NRzvW4qqgwwEiG 6TF7DH/Rm6h3JGYRnQ+o1h/zyOKl X-Google-Smtp-Source: AAOMgpf1PX5+utgNijOw93j0WaaUEWemhsBcTV/9vAGkqJhX9StJydg8ZTivCjBiB63snrZ2FpO7XA== X-Received: by 2002:adf:a64d:: with SMTP id k71-v6mr3687408wrc.78.1530780344495; Thu, 05 Jul 2018 01:45:44 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id x15-v6sm9493909wme.24.2018.07.05.01.45.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:43 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:27 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-3-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 02/10] net/mlx5: remove redundant objects in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch gets rid of redundant calls to open the device and query its attributes in order to simplify the code. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Minor indent fix on existing code. --- drivers/net/mlx5/mlx5.c | 64 +++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 22cbce8d5..4e7f29f5b 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,10 +654,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct ibv_context *ctx = NULL; + struct ibv_device_attr_ex attr; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; - struct ibv_context *attr_ctx = NULL; - struct ibv_device_attr_ex device_attr; unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; @@ -714,12 +714,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || (pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - attr_ctx = mlx5_glue->open_device(list[i]); + ctx = mlx5_glue->open_device(list[i]); rte_errno = errno; err = rte_errno; break; } - if (attr_ctx == NULL) { + if (ctx == NULL) { switch (err) { case 0: DRV_LOG(ERR, @@ -748,7 +748,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + mlx5_glue->dv_query_device(ctx, &dv_attr); if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); @@ -822,23 +822,20 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif - err = mlx5_glue->query_device_ex(attr_ctx, NULL, &device_attr); + err = mlx5_glue->query_device_ex(ctx, NULL, &attr); if (err) { DEBUG("ibv_query_device_ex() failed"); goto error; } - DRV_LOG(INFO, "%u port(s) detected", - device_attr.orig_attr.phys_port_cnt); - for (i = 0; i < device_attr.orig_attr.phys_port_cnt; i++) { + DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); + for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { char name[RTE_ETH_NAME_MAX_LEN]; int len; uint32_t port = i + 1; /* ports are indexed from one */ - struct ibv_context *ctx = NULL; struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct priv *priv = NULL; struct rte_eth_dev *eth_dev = NULL; - struct ibv_device_attr_ex device_attr_ex; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -865,7 +862,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, len = snprintf(name, sizeof(name), PCI_PRI_FMT, pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - if (device_attr.orig_attr.phys_port_cnt > 1) + if (attr.orig_attr.phys_port_cnt > 1) snprintf(name + len, sizeof(name), " port %u", i); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); @@ -907,7 +904,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, continue; } DRV_LOG(DEBUG, "using port %u", port); - ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) + ctx = mlx5_glue->open_device(ibv_dev); if (ctx == NULL) { err = ENODEV; goto port_error; @@ -949,7 +947,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); - priv->device_attr = device_attr; + priv->device_attr = attr; priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; @@ -960,17 +958,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); goto port_error; } - err = mlx5_glue->query_device_ex(ctx, NULL, &device_attr_ex); - if (err) { - DRV_LOG(ERR, "ibv_query_device_ex() failed"); - goto port_error; - } - config.hw_csum = !!(device_attr_ex.device_cap_flags_ex & + config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); DRV_LOG(DEBUG, "checksum offloading is %ssupported", (config.hw_csum ? "" : "not ")); #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!(device_attr.max_counter_sets); + config.flow_counter_en = !!attr.max_counter_sets; mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", @@ -978,7 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, cs_desc.attributes); #endif config.ind_table_max_size = - device_attr_ex.rss_caps.max_rwq_indirection_table_size; + attr.rss_caps.max_rwq_indirection_table_size; /* Remove this check once DPDK supports larger/variable * indirection tables. */ if (config.ind_table_max_size > @@ -986,29 +979,28 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", config.ind_table_max_size); - config.hw_vlan_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_vlan_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); DRV_LOG(DEBUG, "VLAN stripping is %ssupported", (config.hw_vlan_strip ? "" : "not ")); - config.hw_fcs_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_fcs_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_SCATTER_FCS); DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", (config.hw_fcs_strip ? "" : "not ")); #ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!device_attr_ex.rx_pad_end_addr_align; + config.hw_padding = !!attr.rx_pad_end_addr_align; #endif DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", (config.hw_padding ? "" : "not ")); config.vf = vf; - config.tso = ((device_attr_ex.tso_caps.max_tso > 0) && - (device_attr_ex.tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); + config.tso = (attr.tso_caps.max_tso > 0 && + (attr.tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); if (config.tso) - config.tso_max_payload_sz = - device_attr_ex.tso_caps.max_tso; + config.tso_max_payload_sz = attr.tso_caps.max_tso; if (config.mps && !mps) { DRV_LOG(ERR, "multi-packet send not supported on this device" @@ -1170,14 +1162,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); + /* + * Each eth_dev instance is assigned its own Verbs context, + * since this one is consumed, let the next iteration open + * another. + */ + ctx = NULL; continue; port_error: if (priv) rte_free(priv); if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); - if (ctx) - claim_zero(mlx5_glue->close_device(ctx)); if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) rte_eth_dev_release_port(eth_dev); break; @@ -1189,8 +1185,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * way to enumerate the registered ethdevs to free the previous ones. */ error: - if (attr_ctx) - claim_zero(mlx5_glue->close_device(attr_ctx)); + if (ctx) + claim_zero(mlx5_glue->close_device(ctx)); if (list) mlx5_glue->free_device_list(list); if (err) { From patchwork Thu Jul 5 08:45:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42304 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4E0251C37F; Thu, 5 Jul 2018 10:45:48 +0200 (CEST) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id CC76A1C37C for ; Thu, 5 Jul 2018 10:45:46 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id s13-v6so987441wmc.1 for ; Thu, 05 Jul 2018 01:45:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=9hrYhb06M81OLEec/FAJOkvBG9oAKlUR/iA+Ybmj35w=; b=AAVqfY7seMkzJ8u3goRgAdwLwFe90Z8ClitWfm7TPSzfB7GF3TG/8ClM6eOKZrf/uW G0meTJrNV+f17zMt0tWwQ1oCKA28y0ZmagG16wEpnxiUAsx/ozHw8NoHemw1Gr/tNl65 v0GdwrR/Qyah7Kcgzkl56kRNYOzJ90gCwXDfeQAa3rG0qcNlroMPvWxBxDG67k1RAHNW T3f6EtCnCDuxddJ/+AooF88+T8/r2DTtur27cnZp/5e4zNCdUGcffCg6GjKDC/35sR4N WyXKyhHqx0Ixa3rw2sp8w6WyOhbA2gaBDDYOHg/odC2Ol82ifGn6d8oDBEyP7n9KDDjD 0bQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=9hrYhb06M81OLEec/FAJOkvBG9oAKlUR/iA+Ybmj35w=; b=TZ2a73PTUW+/sQipH5bZniOtld1/TKOrECjyyitfTjMTWUHshnevmsP99w/sXh2GWW neCkhDUqQs4Huhjzht5MQ/GQU6vGpTACOrhhCLvxxgwXPiC3ddeVo8V0vSupXiiY688J m5ax+A+CLu8xr3F201ifWk2GzRkP2muZVba7Fxg3Cosz6CoTde+/sL7PS8Bk0b/9A98/ gwguNlZAvWdwqhn5neFRRTp55E2hq+cU80BaIM9hbw1d5OnpuNVqnYM46zhXV33W9O73 sFX47DPIVanjp9wraRoRzrCEypE+uyw3zQpnCnLdTkLs2eJp4RFnv+nqyaYFHGX0kTDV 0R/w== X-Gm-Message-State: APt69E2iJ7pGokUhFhRrVn8Dh51oRO2WxWZIO/bqx3qouR5Jpcn8tem2 Vl2crj2+hd3YAomTzISleoAEEg== X-Google-Smtp-Source: AAOMgpdNrry4bsMTK9prQcq5Uo3bmUbGE64uYBnySNVVPWTPV8pK+JtGURW5D59/GXgco+/dIzqgiQ== X-Received: by 2002:a5d:550a:: with SMTP id b10-v6mr4068365wrv.228.1530780346442; Thu, 05 Jul 2018 01:45:46 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 189-v6sm11430008wmy.25.2018.07.05.01.45.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:45 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:29 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Xueming Li Message-ID: <20180705083934.5535-4-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 03/10] net/mlx5: drop useless support for several Verbs ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Unlike mlx4 from which this capability was inherited, mlx5 devices expose exactly one Verbs port per PCI bus address. Each physical port gets assigned its own bus address with a single Verbs port. While harmless, this code requires an extra loop that would get in the way of subsequent refactoring. No functional impact. Signed-off-by: Adrien Mazarguil Cc: Shahaf Shuler Cc: Xueming Li --- v3 changes: This patch was not present in prior revisions. As discussed [1], it was added after finally deciding to remove this support. [1] https://mails.dpdk.org/archives/dev/2018-June/105661.html --- drivers/net/mlx5/mlx5.c | 96 +++++++++++++------------------------ drivers/net/mlx5/mlx5.h | 1 - drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_txq.c | 2 +- 4 files changed, 34 insertions(+), 67 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 4e7f29f5b..717d8b268 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -652,11 +652,13 @@ static int mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) { - struct ibv_device **list = NULL; - struct ibv_device *ibv_dev; + struct ibv_device **list; struct ibv_context *ctx = NULL; struct ibv_device_attr_ex attr; + struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct rte_eth_dev *eth_dev = NULL; + struct priv *priv = NULL; int err = 0; unsigned int vf = 0; unsigned int mps; @@ -719,6 +721,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; break; } + mlx5_glue->free_device_list(list); if (ctx == NULL) { switch (err) { case 0: @@ -733,7 +736,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } goto error; } - ibv_dev = list[i]; DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; @@ -827,15 +829,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DEBUG("ibv_query_device_ex() failed"); goto error; } - DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); - for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { + { char name[RTE_ETH_NAME_MAX_LEN]; - int len; - uint32_t port = i + 1; /* ports are indexed from one */ struct ibv_port_attr port_attr; - struct ibv_pd *pd = NULL; - struct priv *priv = NULL; - struct rte_eth_dev *eth_dev = NULL; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -859,11 +855,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - len = snprintf(name, sizeof(name), PCI_PRI_FMT, + snprintf(name, sizeof(name), PCI_PRI_FMT, pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - if (attr.orig_attr.phys_port_cnt > 1) - snprintf(name + len, sizeof(name), " port %u", i); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -901,31 +895,22 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); rte_eth_dev_probing_finish(eth_dev); - continue; - } - DRV_LOG(DEBUG, "using port %u", port); - if (!ctx) - ctx = mlx5_glue->open_device(ibv_dev); - if (ctx == NULL) { - err = ENODEV; - goto port_error; + claim_zero(mlx5_glue->close_device(ctx)); + return 0; } /* Check port status. */ - err = mlx5_glue->query_port(ctx, port, &port_attr); + err = mlx5_glue->query_port(ctx, 1, &port_attr); if (err) { DRV_LOG(ERR, "port query failed: %s", strerror(err)); - goto port_error; + goto error; } if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { - DRV_LOG(ERR, - "port %d is not configured in Ethernet mode", - port); + DRV_LOG(ERR, "port is not configured in Ethernet mode"); err = EINVAL; - goto port_error; + goto error; } if (port_attr.state != IBV_PORT_ACTIVE) - DRV_LOG(DEBUG, "port %d is not active: \"%s\" (%d)", - port, + DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", mlx5_glue->port_state_str(port_attr.state), port_attr.state); /* Allocate protection domain. */ @@ -933,7 +918,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (pd == NULL) { DRV_LOG(ERR, "PD allocation failure"); err = ENOMEM; - goto port_error; + goto error; } /* from rte_ethdev.c */ priv = rte_zmalloc("ethdev private structure", @@ -942,13 +927,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (priv == NULL) { DRV_LOG(ERR, "priv allocation failure"); err = ENOMEM; - goto port_error; + goto error; } priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); priv->device_attr = attr; - priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; err = mlx5_args(&config, pci_dev->device.devargs); @@ -956,7 +940,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", strerror(rte_errno)); - goto port_error; + goto error; } config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); @@ -1006,7 +990,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, "multi-packet send not supported on this device" " (" MLX5_TXQ_MPW_EN ")"); err = ENOTSUP; - goto port_error; + goto error; } DRV_LOG(INFO, "%s MPS is %s", config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", @@ -1038,7 +1022,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (eth_dev == NULL) { DRV_LOG(ERR, "can not allocate rte ethdev"); err = ENOMEM; - goto port_error; + goto error; } eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; @@ -1049,7 +1033,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = mlx5_uar_init_primary(eth_dev); if (err) { err = rte_errno; - goto port_error; + goto error; } /* Configure the first MAC address by default. */ if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { @@ -1058,7 +1042,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, " loaded? (errno: %s)", eth_dev->data->port_id, strerror(rte_errno)); err = ENODEV; - goto port_error; + goto error; } DRV_LOG(INFO, "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", @@ -1082,7 +1066,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = mlx5_get_mtu(eth_dev, &priv->mtu); if (err) { err = rte_errno; - goto port_error; + goto error; } DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, priv->mtu); @@ -1131,7 +1115,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "port %u drop queue allocation failed: %s", eth_dev->data->port_id, strerror(rte_errno)); err = rte_errno; - goto port_error; + goto error; } /* Supported Verbs flow priority number detection. */ if (verb_priorities == 0) @@ -1140,7 +1124,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", eth_dev->data->port_id, verb_priorities); err = ENOTSUP; - goto port_error; + goto error; } priv->config.max_verbs_prio = verb_priorities; /* @@ -1154,7 +1138,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->device->numa_node); if (err) { err = rte_errno; - goto port_error; + goto error; } /* Add device to memory callback list. */ rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); @@ -1162,33 +1146,17 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - /* - * Each eth_dev instance is assigned its own Verbs context, - * since this one is consumed, let the next iteration open - * another. - */ - ctx = NULL; - continue; -port_error: - if (priv) - rte_free(priv); - if (pd) - claim_zero(mlx5_glue->dealloc_pd(pd)); - if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_eth_dev_release_port(eth_dev); - break; + return 0; } - /* - * XXX if something went wrong in the loop above, there is a resource - * leak (ctx, pd, priv, dpdk ethdev) but we can do nothing about it as - * long as the dpdk does not provide a way to deallocate a ethdev and a - * way to enumerate the registered ethdevs to free the previous ones. - */ error: + if (priv) + rte_free(priv); + if (pd) + claim_zero(mlx5_glue->dealloc_pd(pd)); + if (eth_dev) + rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (list) - mlx5_glue->free_device_list(list); if (err) { rte_errno = err; return -rte_errno; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 997b04a33..f55ff4a21 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -159,7 +159,6 @@ struct priv { unsigned int vlan_filter_n; /* Number of configured VLAN filters. */ /* Device properties. */ uint16_t mtu; /* Configured MTU. */ - uint8_t port; /* Physical port number. */ unsigned int isolated:1; /* Whether isolated mode is enabled. */ /* RX/TX queues. */ unsigned int rxqs_n; /* RX queues array size. */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index ebe5cb6e3..819f5baad 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -166,7 +166,7 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) if (dev_port == dev_port_prev) goto try_dev_id; dev_port_prev = dev_port; - if (dev_port == (priv->port - 1u)) + if (dev_port == 0) strlcpy(match, name, sizeof(match)); } closedir(dir); diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 669b91319..5057561ae 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -434,7 +434,7 @@ mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t idx) /* Move the QP to this state. */ .qp_state = IBV_QPS_INIT, /* Primary port number. */ - .port_num = priv->port + .port_num = 1, }; ret = mlx5_glue->modify_qp(tmpl.qp, &attr.mod, (IBV_QP_STATE | IBV_QP_PORT)); From patchwork Thu Jul 5 08:45:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42305 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 501091C38A; Thu, 5 Jul 2018 10:45:50 +0200 (CEST) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id B63AF1C382 for ; Thu, 5 Jul 2018 10:45:48 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id s12-v6so9716208wmc.1 for ; Thu, 05 Jul 2018 01:45:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=U+2tgo9qEAMbh5Y43XjgWFJ2nazL87NKgcslsyD/Uk8HEEnPowzLQSf5euQPC2L8vj w5ssEZbWSGKyHmZAPHe7AtMCUHug+aEeCC5gN2MFZvQUSeXD2EH7qmdQm950cAoEuyJb XcHdnp/KlIBVD77GPNfz2aNrVH2NOcq5mwvXg47zRKtL3YorrYtlakgGg6gIco8OQLrm pXQ13j6Ma+KOKdzdEBDnj8Bv6T7tESm4HPLvALSsQ5RoHN4OOn1VTu95c9B9Cv18et4I SRseEKWk5SRiObnQP9F2P4Qnv4DMnmtCV/ekVuFa4TF3OH0UMF5Y2D45BZllpXTiEqHQ BgbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=jlKl9rx2oGEAUZAYQ6Q9X3ipg+/OJkklDEDv27QKqwtM4aXP4a/UtqAcJkoi8fRvqI I/YhEqwZuSee6ObBcijxt54aEOixvvhnpBT05jMemDX0Ei0oK8Ql+WZmwIRVjHzqNFFh HCvgGl/QDH0NIZ0lII+vWCABK5Fejo2awfXoxYoFzSeFewfTBOLrD4wUUdpFu/6Uoh60 8a8sUYrcXAzGZvZGquway5axekTWBz+lwYhbkpwcngicoynzlzuBuH/vccGhw8YqKrLr a4u3qONtzG+gX9hE+e1wDkUqhHWSgz+CJwsYaQowL5rRkHe2DLho9n5oz4M+w4cKixdB NeEg== X-Gm-Message-State: APt69E2kQgXfY0FrhlngNhmByFlN73weYy5hlN7pjbObaFzp20fx13e2 2LU7W1einEZO4Aa7/9+2hcgaqA== X-Google-Smtp-Source: AAOMgpfrwWVB0gXB8r6jvpI34rDFM3vow5gpJ4KG8VQEjbf9s9mSDK08j9K+6+l1aVjOnUwx6FPg4g== X-Received: by 2002:a1c:7409:: with SMTP id p9-v6mr3276156wmc.43.1530780348359; Thu, 05 Jul 2018 01:45:48 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id s10-v6sm10165442wmb.12.2018.07.05.01.45.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:47 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:31 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-5-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 04/10] net/mlx5: split PCI from generic probing code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" All the generic probing code needs is an IB device. While this device is currently supplied by a PCI lookup, other methods will be added soon. This patch divides the original function, which has become huge over time, as follows: 1. PCI-specific (mlx5_pci_probe()). 2. Verbs device (mlx5_dev_spawn()). (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li Cc: Shahaf Shuler --- v3 changes: - Moved VF device check within mlx5_pci_probe() after identifying the device instead of before that. - Merged mlx5_dev_spawn_one() with mlx5_dev_spawn() since there is no need anymore for an intermediate function to iterate over Verbs ports. v2 changes: - Fixed device naming. A port suffix is now appended only if several IB ports happen to be detected. - Added separate message to distinguish missing kernel drivers from other initialization errors, as it was confusing. --- drivers/net/mlx5/mlx5.c | 195 ++++++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 86 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 717d8b268..8916d4684 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -36,6 +36,7 @@ #include #include #include +#include #include "mlx5.h" #include "mlx5_utils.h" @@ -635,32 +636,31 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) } /** - * DPDK callback to register a PCI device. - * - * This function creates an Ethernet device for each port of a given - * PCI device. + * Spawn an Ethernet device from Verbs information. * - * @param[in] pci_drv - * PCI driver structure (mlx5_driver). - * @param[in] pci_dev - * PCI device information. + * @param dpdk_dev + * Backing DPDK device. + * @param ibv_dev + * Verbs device. + * @param vf + * If nonzero, enable VF-specific features. * * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. + * A valid Ethernet device object on success, NULL otherwise and rte_errno + * is set. */ -static int -mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +static struct rte_eth_dev * +mlx5_dev_spawn(struct rte_device *dpdk_dev, + struct ibv_device *ibv_dev, + int vf) { - struct ibv_device **list; - struct ibv_context *ctx = NULL; + struct ibv_context *ctx; struct ibv_device_attr_ex attr; struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; - unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; unsigned int tunnel_en = 0; @@ -672,71 +672,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_max_stride_size_n = 0; unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; - int i; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); - assert(pci_drv == &mlx5_driver); - list = mlx5_glue->get_device_list(&i); - if (list == NULL) { - assert(errno); - err = errno; - if (errno == ENOSYS) - DRV_LOG(ERR, - "cannot list devices, is ib_uverbs loaded?"); - goto error; - } - assert(i >= 0); - /* - * For each listed device, check related sysfs entry against - * the provided PCI ID. - */ - while (i != 0) { - struct rte_pci_addr pci_addr; - - --i; - DRV_LOG(DEBUG, "checking device \"%s\"", list[i]->name); - if (mlx5_ibv_device_to_pci_addr(list[i], &pci_addr)) - continue; - if ((pci_dev->addr.domain != pci_addr.domain) || - (pci_dev->addr.bus != pci_addr.bus) || - (pci_dev->addr.devid != pci_addr.devid) || - (pci_dev->addr.function != pci_addr.function)) - continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", - list[i]->name); - vf = ((pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - ctx = mlx5_glue->open_device(list[i]); - rte_errno = errno; - err = rte_errno; - break; - } - mlx5_glue->free_device_list(list); - if (ctx == NULL) { - switch (err) { - case 0: - DRV_LOG(ERR, - "cannot access device, is mlx5_ib loaded?"); - err = ENODEV; - break; - case EINVAL: - DRV_LOG(ERR, - "cannot use device, are drivers up to date?"); - break; - } - goto error; + errno = 0; + ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) { + rte_errno = errno ? errno : ENODEV; + return NULL; } - DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif @@ -855,9 +802,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - snprintf(name, sizeof(name), PCI_PRI_FMT, - pci_dev->addr.domain, pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -866,7 +811,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; goto error; } - eth_dev->device = &pci_dev->device; + eth_dev->device = dpdk_dev; eth_dev->dev_ops = &mlx5_dev_sec_ops; err = mlx5_uar_init_secondary(eth_dev); if (err) { @@ -894,9 +839,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mlx5_select_rx_function(eth_dev); eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); - rte_eth_dev_probing_finish(eth_dev); claim_zero(mlx5_glue->close_device(ctx)); - return 0; + return eth_dev; } /* Check port status. */ err = mlx5_glue->query_port(ctx, 1, &port_attr); @@ -935,7 +879,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->device_attr = attr; priv->pd = pd; priv->mtu = ETHER_MTU; - err = mlx5_args(&config, pci_dev->device.devargs); + err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", @@ -1027,8 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; - eth_dev->device = &pci_dev->device; - rte_eth_copy_pci_info(eth_dev, pci_dev); + eth_dev->device = dpdk_dev; eth_dev->device->driver = &mlx5_driver.driver; err = mlx5_uar_init_primary(eth_dev); if (err) { @@ -1146,7 +1089,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - return 0; + return eth_dev; } error: if (priv) @@ -1157,11 +1100,91 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (err) { - rte_errno = err; + assert(err > 0); + rte_errno = err; + return NULL; +} + +/** + * DPDK callback to register a PCI device. + * + * This function spawns an Ethernet device out of a given PCI device. + * + * @param[in] pci_drv + * PCI driver structure (mlx5_driver). + * @param[in] pci_dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct ibv_device **ibv_list; + struct rte_eth_dev *eth_dev = NULL; + int vf; + int ret; + + assert(pci_drv == &mlx5_driver); + errno = 0; + ibv_list = mlx5_glue->get_device_list(&ret); + if (!ibv_list) { + rte_errno = errno ? errno : ENOSYS; + DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } - return 0; + while (ret-- > 0) { + struct rte_pci_addr pci_addr; + + DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name); + if (mlx5_ibv_device_to_pci_addr(ibv_list[ret], &pci_addr)) + continue; + if (pci_dev->addr.domain != pci_addr.domain || + pci_dev->addr.bus != pci_addr.bus || + pci_dev->addr.devid != pci_addr.devid || + pci_dev->addr.function != pci_addr.function) + continue; + DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + ibv_list[ret]->name); + break; + } + switch (pci_dev->id.device_id) { + case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF: + vf = 1; + break; + default: + vf = 0; + } + if (ret >= 0) + eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + mlx5_glue->free_device_list(ibv_list); + if (!ret) { + DRV_LOG(WARNING, + "no Verbs device matches PCI device " PCI_PRI_FMT "," + " are kernel drivers loaded?", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function); + rte_errno = ENOENT; + ret = -rte_errno; + } else if (!eth_dev) { + DRV_LOG(ERR, + "probe of PCI device " PCI_PRI_FMT " aborted after" + " encountering an error: %s", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function, + strerror(rte_errno)); + ret = -rte_errno; + } else { + rte_eth_copy_pci_info(eth_dev, pci_dev); + rte_eth_dev_probing_finish(eth_dev); + ret = 0; + } + return ret; } static const struct rte_pci_id mlx5_pci_id_map[] = { From patchwork Thu Jul 5 08:45:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42306 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E93901C392; Thu, 5 Jul 2018 10:45:51 +0200 (CEST) Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by dpdk.org (Postfix) with ESMTP id EC7A21C38F for ; Thu, 5 Jul 2018 10:45:50 +0200 (CEST) Received: by mail-wr1-f49.google.com with SMTP id h10-v6so739268wre.6 for ; Thu, 05 Jul 2018 01:45:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=DmREd05I1V0ABxeZRbzLBJH+y7fOYJDLHEFO/UKDYhU=; b=Yg/rXh6DoHf/RW3EslQ9raYt0bUUDz7L0gGbD1eJMyicGH9tcgYJw7mMWh3JGGEyX6 nE7ogNvodsJFOTrIqoO9n439QRNrDZ6Xpa5toY3P8i5l7Cg6OOOkUM3Ejhy/8WwjRCXB yADuRt820UxxQzPAFhFH9y4arVMi2V+xKQ6P/YAd6FZtKIumi9zzf96O4+/UFh/Q7Ok5 5J0cdIaOWD8vfsVxUO9t9xYSLskXfDfdSUK3UhGH3AVUiP4Qr44gW4FWLqBlDK29w0dD hk1kWYp5zL306BkwB9tOEyCs1JoVJbFo+k/vWaRSzdiXwtG8aTTru+Uzu8pNykPJkYO4 k4cA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=DmREd05I1V0ABxeZRbzLBJH+y7fOYJDLHEFO/UKDYhU=; b=uQuey7HnXb5xudoAZTNEBSl5TuJbhsjTrCYYyhD6kAlFrcQQXkQJGNS9+EOWPndPzA 79/xMoOinmOtox/wHglErzlgd4IA7On1MJrbbJJkpFcc8y4vad9OV7AV4w1j7HimEKho jH/P2XuvmrFcnbymLq27kacbNQGkOLgxGVZDI5v8Y7pMivFTQFE8UNOAJnUrAttubaLY z1RP880pzyXqcMhBqcIXEcpeJu8rGtAqokfq6/rfi9SK9k501x9DmZwNOFnlF4sRW+T2 a/X/z04D9hgVNJ2ZQDROYoDvyZM7oxV5Whk9sTyM+WJWkiEtKi0Vf3Rf9AxAYWArGSab KyLw== X-Gm-Message-State: APt69E31A2NyUWSOr6NXiOTbviZ2gDzk7XKHjSVwtWFqcZTQynuedap5 fCxJyYrMnzE4dZFhP0WEj5iHrC9W X-Google-Smtp-Source: AAOMgpexDW65GT0h8TYJWpBV9BaQD1oiqgD6tEU/gHG777rrTnL5jTNahcE9+RewfWnnFZEKaRgdxg== X-Received: by 2002:a5d:6550:: with SMTP id z16-v6mr3701920wrv.194.1530780350363; Thu, 05 Jul 2018 01:45:50 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id z9-v6sm10765524wre.49.2018.07.05.01.45.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:49 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:33 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-6-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 05/10] net/mlx5: re-indent generic probing function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since commit "net/mlx5: drop useless support for several Verbs ports" removed an inner loop, mlx5_dev_spawn() is left with an unnecessary indent level. This patch eliminates a block, moves its local variables to function scope, and re-indents its contents (diff best viewed with --ignore-all-space). No functional impact. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming(Steven) Li --- v3 changes: - Reworded commit log since original patch was modified. This patch is also much shorter as a consequence. --- drivers/net/mlx5/mlx5.c | 578 +++++++++++++++++++++---------------------- 1 file changed, 282 insertions(+), 296 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 8916d4684..1054bf6d0 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -656,8 +656,25 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, { struct ibv_context *ctx; struct ibv_device_attr_ex attr; + struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct mlx5_dev_config config = { + .vf = !!vf, + .tx_vec_en = 1, + .rx_vec_en = 1, + .mpw_hdr_dseg = 0, + .txq_inline = MLX5_ARG_UNSET, + .txqs_inline = MLX5_ARG_UNSET, + .inline_max_packet_sz = MLX5_ARG_UNSET, + .vf_nl_en = 1, + .mprq = { + .enabled = 0, + .stride_num_n = MLX5_MPRQ_STRIDE_NUM_N, + .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, + .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, + }, + }; struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; @@ -675,6 +692,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif + struct ether_addr mac; + char name[RTE_ETH_NAME_MAX_LEN]; /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); @@ -710,11 +729,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(DEBUG, "MPW isn't supported"); mps = MLX5_MPW_DISABLED; } + config.mps = mps; #ifdef HAVE_IBV_MLX5_MOD_SWP if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif + config.swp = !!swp; #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = @@ -740,6 +761,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, mprq_caps.min_single_wqe_log_num_of_strides; mprq_max_stride_num_n = mprq_caps.max_single_wqe_log_num_of_strides; + config.mprq.stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); } #endif if (RTE_CACHE_LINE_SIZE == 128 && @@ -747,6 +770,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, cqe_comp = 0; else cqe_comp = 1; + config.cqe_comp = cqe_comp; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { tunnel_en = ((dv_attr.tunnel_offloads_caps & @@ -760,6 +784,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif + config.tunnel_en = tunnel_en; #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && @@ -771,326 +796,287 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif + config.mpls_en = mpls_en; err = mlx5_glue->query_device_ex(ctx, NULL, &attr); if (err) { DEBUG("ibv_query_device_ex() failed"); goto error; } - { - char name[RTE_ETH_NAME_MAX_LEN]; - struct ibv_port_attr port_attr; - struct ether_addr mac; - struct mlx5_dev_config config = { - .cqe_comp = cqe_comp, - .mps = mps, - .tunnel_en = tunnel_en, - .mpls_en = mpls_en, - .tx_vec_en = 1, - .rx_vec_en = 1, - .mpw_hdr_dseg = 0, - .txq_inline = MLX5_ARG_UNSET, - .txqs_inline = MLX5_ARG_UNSET, - .inline_max_packet_sz = MLX5_ARG_UNSET, - .vf_nl_en = 1, - .swp = !!swp, - .mprq = { - .enabled = 0, /* Disabled by default. */ - .stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n), - .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, - .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, - }, - }; - - rte_strlcpy(name, dpdk_dev->name, sizeof(name)); - if (rte_eal_process_type() == RTE_PROC_SECONDARY) { - eth_dev = rte_eth_dev_attach_secondary(name); - if (eth_dev == NULL) { - DRV_LOG(ERR, "can not attach rte ethdev"); - rte_errno = ENOMEM; - err = rte_errno; - goto error; - } - eth_dev->device = dpdk_dev; - eth_dev->dev_ops = &mlx5_dev_sec_ops; - err = mlx5_uar_init_secondary(eth_dev); - if (err) { - err = rte_errno; - goto error; - } - /* Receive command fd from primary process */ - err = mlx5_socket_connect(eth_dev); - if (err < 0) { - err = rte_errno; - goto error; - } - /* Remap UAR for Tx queues. */ - err = mlx5_tx_uar_remap(eth_dev, err); - if (err) { - err = rte_errno; - goto error; - } - /* - * Ethdev pointer is still required as input since - * the primary device is not accessible from the - * secondary process. - */ - eth_dev->rx_pkt_burst = - mlx5_select_rx_function(eth_dev); - eth_dev->tx_pkt_burst = - mlx5_select_tx_function(eth_dev); - claim_zero(mlx5_glue->close_device(ctx)); - return eth_dev; - } - /* Check port status. */ - err = mlx5_glue->query_port(ctx, 1, &port_attr); - if (err) { - DRV_LOG(ERR, "port query failed: %s", strerror(err)); - goto error; - } - if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { - DRV_LOG(ERR, "port is not configured in Ethernet mode"); - err = EINVAL; - goto error; - } - if (port_attr.state != IBV_PORT_ACTIVE) - DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", - mlx5_glue->port_state_str(port_attr.state), - port_attr.state); - /* Allocate protection domain. */ - pd = mlx5_glue->alloc_pd(ctx); - if (pd == NULL) { - DRV_LOG(ERR, "PD allocation failure"); - err = ENOMEM; - goto error; - } - /* from rte_ethdev.c */ - priv = rte_zmalloc("ethdev private structure", - sizeof(*priv), - RTE_CACHE_LINE_SIZE); - if (priv == NULL) { - DRV_LOG(ERR, "priv allocation failure"); - err = ENOMEM; - goto error; - } - priv->ctx = ctx; - strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, - sizeof(priv->ibdev_path)); - priv->device_attr = attr; - priv->pd = pd; - priv->mtu = ETHER_MTU; - err = mlx5_args(&config, dpdk_dev->devargs); - if (err) { - err = rte_errno; - DRV_LOG(ERR, "failed to process device arguments: %s", - strerror(rte_errno)); - goto error; - } - config.hw_csum = !!(attr.device_cap_flags_ex & - IBV_DEVICE_RAW_IP_CSUM); - DRV_LOG(DEBUG, "checksum offloading is %ssupported", - (config.hw_csum ? "" : "not ")); -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!attr.max_counter_sets; - mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); - DRV_LOG(DEBUG, - "counter type = %d, num of cs = %ld, attributes = %d", - cs_desc.counter_type, cs_desc.num_of_cs, - cs_desc.attributes); -#endif - config.ind_table_max_size = - attr.rss_caps.max_rwq_indirection_table_size; - /* Remove this check once DPDK supports larger/variable - * indirection tables. */ - if (config.ind_table_max_size > - (unsigned int)ETH_RSS_RETA_SIZE_512) - config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; - DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", - config.ind_table_max_size); - config.hw_vlan_strip = !!(attr.raw_packet_caps & - IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); - DRV_LOG(DEBUG, "VLAN stripping is %ssupported", - (config.hw_vlan_strip ? "" : "not ")); - - config.hw_fcs_strip = !!(attr.raw_packet_caps & - IBV_RAW_PACKET_CAP_SCATTER_FCS); - DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", - (config.hw_fcs_strip ? "" : "not ")); - -#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!attr.rx_pad_end_addr_align; -#endif - DRV_LOG(DEBUG, - "hardware Rx end alignment padding is %ssupported", - (config.hw_padding ? "" : "not ")); - config.vf = vf; - config.tso = (attr.tso_caps.max_tso > 0 && - (attr.tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); - if (config.tso) - config.tso_max_payload_sz = attr.tso_caps.max_tso; - if (config.mps && !mps) { - DRV_LOG(ERR, - "multi-packet send not supported on this device" - " (" MLX5_TXQ_MPW_EN ")"); - err = ENOTSUP; - goto error; - } - DRV_LOG(INFO, "%s MPS is %s", - config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", - config.mps != MLX5_MPW_DISABLED ? "enabled" : - "disabled"); - if (config.cqe_comp && !cqe_comp) { - DRV_LOG(WARNING, "Rx CQE compression isn't supported"); - config.cqe_comp = 0; - } - if (config.mprq.enabled && mprq) { - if (config.mprq.stride_num_n > mprq_max_stride_num_n || - config.mprq.stride_num_n < mprq_min_stride_num_n) { - config.mprq.stride_num_n = - RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n); - DRV_LOG(WARNING, - "the number of strides" - " for Multi-Packet RQ is out of range," - " setting default value (%u)", - 1 << config.mprq.stride_num_n); - } - config.mprq.min_stride_size_n = mprq_min_stride_size_n; - config.mprq.max_stride_size_n = mprq_max_stride_size_n; - } else if (config.mprq.enabled && !mprq) { - DRV_LOG(WARNING, "Multi-Packet RQ isn't supported"); - config.mprq.enabled = 0; - } - eth_dev = rte_eth_dev_allocate(name); + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + if (rte_eal_process_type() == RTE_PROC_SECONDARY) { + eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { - DRV_LOG(ERR, "can not allocate rte ethdev"); - err = ENOMEM; + DRV_LOG(ERR, "can not attach rte ethdev"); + rte_errno = ENOMEM; + err = rte_errno; goto error; } - eth_dev->data->dev_private = priv; - priv->dev_data = eth_dev->data; - eth_dev->data->mac_addrs = priv->mac; eth_dev->device = dpdk_dev; - eth_dev->device->driver = &mlx5_driver.driver; - err = mlx5_uar_init_primary(eth_dev); + eth_dev->dev_ops = &mlx5_dev_sec_ops; + err = mlx5_uar_init_secondary(eth_dev); if (err) { err = rte_errno; goto error; } - /* Configure the first MAC address by default. */ - if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { - DRV_LOG(ERR, - "port %u cannot get MAC address, is mlx5_en" - " loaded? (errno: %s)", - eth_dev->data->port_id, strerror(rte_errno)); - err = ENODEV; - goto error; - } - DRV_LOG(INFO, - "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", - eth_dev->data->port_id, - mac.addr_bytes[0], mac.addr_bytes[1], - mac.addr_bytes[2], mac.addr_bytes[3], - mac.addr_bytes[4], mac.addr_bytes[5]); -#ifndef NDEBUG - { - char ifname[IF_NAMESIZE]; - - if (mlx5_get_ifname(eth_dev, &ifname) == 0) - DRV_LOG(DEBUG, "port %u ifname is \"%s\"", - eth_dev->data->port_id, ifname); - else - DRV_LOG(DEBUG, "port %u ifname is unknown", - eth_dev->data->port_id); - } -#endif - /* Get actual MTU if possible. */ - err = mlx5_get_mtu(eth_dev, &priv->mtu); - if (err) { + /* Receive command fd from primary process */ + err = mlx5_socket_connect(eth_dev); + if (err < 0) { err = rte_errno; goto error; } - DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, - priv->mtu); - /* - * Initialize burst functions to prevent crashes before link-up. - */ - eth_dev->rx_pkt_burst = removed_rx_burst; - eth_dev->tx_pkt_burst = removed_tx_burst; - eth_dev->dev_ops = &mlx5_dev_ops; - /* Register MAC address. */ - claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); - priv->nl_socket = -1; - priv->nl_sn = 0; - if (vf && config.vf_nl_en) { - priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); - if (priv->nl_socket < 0) - priv->nl_socket = -1; - mlx5_nl_mac_addr_sync(eth_dev); - } - TAILQ_INIT(&priv->flows); - TAILQ_INIT(&priv->ctrl_flows); - /* Hint libmlx5 to use PMD allocator for data plane resources */ - struct mlx5dv_ctx_allocators alctr = { - .alloc = &mlx5_alloc_verbs_buf, - .free = &mlx5_free_verbs_buf, - .data = priv, - }; - mlx5_glue->dv_set_context_attr(ctx, - MLX5DV_CTX_ATTR_BUF_ALLOCATORS, - (void *)((uintptr_t)&alctr)); - /* Bring Ethernet device up. */ - DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", - eth_dev->data->port_id); - mlx5_set_link_up(eth_dev); - /* - * Even though the interrupt handler is not installed yet, - * interrupts will still trigger on the asyn_fd from - * Verbs context returned by ibv_open_device(). - */ - mlx5_link_update(eth_dev, 0); - /* Store device configuration on private structure. */ - priv->config = config; - /* Create drop queue. */ - err = mlx5_flow_create_drop_queue(eth_dev); + /* Remap UAR for Tx queues. */ + err = mlx5_tx_uar_remap(eth_dev, err); if (err) { - DRV_LOG(ERR, "port %u drop queue allocation failed: %s", - eth_dev->data->port_id, strerror(rte_errno)); err = rte_errno; goto error; } - /* Supported Verbs flow priority number detection. */ - if (verb_priorities == 0) - verb_priorities = mlx5_get_max_verbs_prio(eth_dev); - if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { - DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", - eth_dev->data->port_id, verb_priorities); - err = ENOTSUP; - goto error; - } - priv->config.max_verbs_prio = verb_priorities; /* - * Once the device is added to the list of memory event - * callback, its global MR cache table cannot be expanded - * on the fly because of deadlock. If it overflows, lookup - * should be done by searching MR list linearly, which is slow. + * Ethdev pointer is still required as input since + * the primary device is not accessible from the + * secondary process. */ - err = mlx5_mr_btree_init(&priv->mr.cache, - MLX5_MR_BTREE_CACHE_N * 2, - eth_dev->device->numa_node); - if (err) { - err = rte_errno; - goto error; - } - /* Add device to memory callback list. */ - rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); - LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, - priv, mem_event_cb); - rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); - rte_eth_dev_probing_finish(eth_dev); + eth_dev->rx_pkt_burst = mlx5_select_rx_function(eth_dev); + eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); + claim_zero(mlx5_glue->close_device(ctx)); return eth_dev; } + /* Check port status. */ + err = mlx5_glue->query_port(ctx, 1, &port_attr); + if (err) { + DRV_LOG(ERR, "port query failed: %s", strerror(err)); + goto error; + } + if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { + DRV_LOG(ERR, "port is not configured in Ethernet mode"); + err = EINVAL; + goto error; + } + if (port_attr.state != IBV_PORT_ACTIVE) + DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", + mlx5_glue->port_state_str(port_attr.state), + port_attr.state); + /* Allocate protection domain. */ + pd = mlx5_glue->alloc_pd(ctx); + if (pd == NULL) { + DRV_LOG(ERR, "PD allocation failure"); + err = ENOMEM; + goto error; + } + priv = rte_zmalloc("ethdev private structure", + sizeof(*priv), + RTE_CACHE_LINE_SIZE); + if (priv == NULL) { + DRV_LOG(ERR, "priv allocation failure"); + err = ENOMEM; + goto error; + } + priv->ctx = ctx; + strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, + sizeof(priv->ibdev_path)); + priv->device_attr = attr; + priv->pd = pd; + priv->mtu = ETHER_MTU; + err = mlx5_args(&config, dpdk_dev->devargs); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + goto error; + } + config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); + DRV_LOG(DEBUG, "checksum offloading is %ssupported", + (config.hw_csum ? "" : "not ")); +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + config.flow_counter_en = !!attr.max_counter_sets; + mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); + DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", + cs_desc.counter_type, cs_desc.num_of_cs, + cs_desc.attributes); +#endif + config.ind_table_max_size = + attr.rss_caps.max_rwq_indirection_table_size; + /* + * Remove this check once DPDK supports larger/variable + * indirection tables. + */ + if (config.ind_table_max_size > (unsigned int)ETH_RSS_RETA_SIZE_512) + config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; + DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", + config.ind_table_max_size); + config.hw_vlan_strip = !!(attr.raw_packet_caps & + IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); + DRV_LOG(DEBUG, "VLAN stripping is %ssupported", + (config.hw_vlan_strip ? "" : "not ")); + config.hw_fcs_strip = !!(attr.raw_packet_caps & + IBV_RAW_PACKET_CAP_SCATTER_FCS); + DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", + (config.hw_fcs_strip ? "" : "not ")); +#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING + config.hw_padding = !!attr.rx_pad_end_addr_align; +#endif + DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", + (config.hw_padding ? "" : "not ")); + config.tso = (attr.tso_caps.max_tso > 0 && + (attr.tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); + if (config.tso) + config.tso_max_payload_sz = attr.tso_caps.max_tso; + if (config.mps && !mps) { + DRV_LOG(ERR, + "multi-packet send not supported on this device" + " (" MLX5_TXQ_MPW_EN ")"); + err = ENOTSUP; + goto error; + } + DRV_LOG(INFO, "%sMPS is %s", + config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", + config.mps != MLX5_MPW_DISABLED ? "enabled" : "disabled"); + if (config.cqe_comp && !cqe_comp) { + DRV_LOG(WARNING, "Rx CQE compression isn't supported"); + config.cqe_comp = 0; + } + if (config.mprq.enabled && mprq) { + if (config.mprq.stride_num_n > mprq_max_stride_num_n || + config.mprq.stride_num_n < mprq_min_stride_num_n) { + config.mprq.stride_num_n = + RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); + DRV_LOG(WARNING, + "the number of strides" + " for Multi-Packet RQ is out of range," + " setting default value (%u)", + 1 << config.mprq.stride_num_n); + } + config.mprq.min_stride_size_n = mprq_min_stride_size_n; + config.mprq.max_stride_size_n = mprq_max_stride_size_n; + } else if (config.mprq.enabled && !mprq) { + DRV_LOG(WARNING, "Multi-Packet RQ isn't supported"); + config.mprq.enabled = 0; + } + eth_dev = rte_eth_dev_allocate(name); + if (eth_dev == NULL) { + DRV_LOG(ERR, "can not allocate rte ethdev"); + err = ENOMEM; + goto error; + } + eth_dev->data->dev_private = priv; + priv->dev_data = eth_dev->data; + eth_dev->data->mac_addrs = priv->mac; + eth_dev->device = dpdk_dev; + eth_dev->device->driver = &mlx5_driver.driver; + err = mlx5_uar_init_primary(eth_dev); + if (err) { + err = rte_errno; + goto error; + } + /* Configure the first MAC address by default. */ + if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { + DRV_LOG(ERR, + "port %u cannot get MAC address, is mlx5_en" + " loaded? (errno: %s)", + eth_dev->data->port_id, strerror(rte_errno)); + err = ENODEV; + goto error; + } + DRV_LOG(INFO, + "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", + eth_dev->data->port_id, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); +#ifndef NDEBUG + { + char ifname[IF_NAMESIZE]; + + if (mlx5_get_ifname(eth_dev, &ifname) == 0) + DRV_LOG(DEBUG, "port %u ifname is \"%s\"", + eth_dev->data->port_id, ifname); + else + DRV_LOG(DEBUG, "port %u ifname is unknown", + eth_dev->data->port_id); + } +#endif + /* Get actual MTU if possible. */ + err = mlx5_get_mtu(eth_dev, &priv->mtu); + if (err) { + err = rte_errno; + goto error; + } + DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, + priv->mtu); + /* Initialize burst functions to prevent crashes before link-up. */ + eth_dev->rx_pkt_burst = removed_rx_burst; + eth_dev->tx_pkt_burst = removed_tx_burst; + eth_dev->dev_ops = &mlx5_dev_ops; + /* Register MAC address. */ + claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); + priv->nl_socket = -1; + priv->nl_sn = 0; + if (vf && config.vf_nl_en) { + priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); + if (priv->nl_socket < 0) + priv->nl_socket = -1; + mlx5_nl_mac_addr_sync(eth_dev); + } + TAILQ_INIT(&priv->flows); + TAILQ_INIT(&priv->ctrl_flows); + /* Hint libmlx5 to use PMD allocator for data plane resources */ + struct mlx5dv_ctx_allocators alctr = { + .alloc = &mlx5_alloc_verbs_buf, + .free = &mlx5_free_verbs_buf, + .data = priv, + }; + mlx5_glue->dv_set_context_attr(ctx, MLX5DV_CTX_ATTR_BUF_ALLOCATORS, + (void *)((uintptr_t)&alctr)); + /* Bring Ethernet device up. */ + DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", + eth_dev->data->port_id); + mlx5_set_link_up(eth_dev); + /* + * Even though the interrupt handler is not installed yet, + * interrupts will still trigger on the asyn_fd from + * Verbs context returned by ibv_open_device(). + */ + mlx5_link_update(eth_dev, 0); + /* Store device configuration on private structure. */ + priv->config = config; + /* Create drop queue. */ + err = mlx5_flow_create_drop_queue(eth_dev); + if (err) { + DRV_LOG(ERR, "port %u drop queue allocation failed: %s", + eth_dev->data->port_id, strerror(rte_errno)); + err = rte_errno; + goto error; + } + /* Supported Verbs flow priority number detection. */ + if (verb_priorities == 0) + verb_priorities = mlx5_get_max_verbs_prio(eth_dev); + if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { + DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", + eth_dev->data->port_id, verb_priorities); + err = ENOTSUP; + goto error; + } + priv->config.max_verbs_prio = verb_priorities; + /* + * Once the device is added to the list of memory event + * callback, its global MR cache table cannot be expanded + * on the fly because of deadlock. If it overflows, lookup + * should be done by searching MR list linearly, which is slow. + */ + err = mlx5_mr_btree_init(&priv->mr.cache, + MLX5_MR_BTREE_CACHE_N * 2, + eth_dev->device->numa_node); + if (err) { + err = rte_errno; + goto error; + } + /* Add device to memory callback list. */ + rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); + LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, + priv, mem_event_cb); + rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); + return eth_dev; error: if (priv) rte_free(priv); From patchwork Thu Jul 5 08:45:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42307 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CAC531C39C; Thu, 5 Jul 2018 10:45:54 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id DB0BC1C396 for ; Thu, 5 Jul 2018 10:45:52 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id s12-v6so9716667wmc.1 for ; Thu, 05 Jul 2018 01:45:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=EM3d5ieHBtd5+GEdF7A4rgOgY3EDx79w9cdZs81cSjg=; b=S2spjC0SUosmOAObe5FmgTV6M5XU7LZ7RamFIMpZr13atoqyHTwEGJgsKAhFIoh0dw 9y9hLdNOTC4pU2kUC29O6mzlxbWPPFeIgwy5IjWWHVLrlMEMdVI8JToB0tf/Jid627QR IlnFTcWOMRw63LwEdcJlgLDt76gioQhidown41bo3Cg++cnT+sGJzJmFAYqP81ARk+yX qb30v/luPqUu8NgO1hTEDlUIo3yKGwSC7FChkFi4VGs97ydk93PflACkvkSiLOMeFco7 gPZlieclcnkT2mx187jIuHili5RwZ0X4T6Oc/L7/aVx5q5yF+WeieKFz8Qhf/QGhRCT4 x9YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=EM3d5ieHBtd5+GEdF7A4rgOgY3EDx79w9cdZs81cSjg=; b=Jm6craeOVoHGUXObrm67SGfqjZDGbRgAA47g/ktBDYqqDBMriYWfIVkLBDOdIn6agw rgb2PBCmPub6Z7E+XNNhkqMXV1j7JmFTaWR6py95YYjh97GIpCiopSID5TZuVQ0s6DCY 9UMSJEO3MLjFsTyIkh8+l9i7nEyyW6stmEH9Qz34uy5ASr0P21AJknODPpaFaDsG/pDT axnQ9L/S6hI7YxAJoUnGnOXrrMD05INZLnX5rlPvoduomWqurtZ7yEULOsyMU3y0Lbop rovbkPOFRkRYvfsOJUOewPIMvdlryGk1epwecmdl6OlpUZgWbas4FjA5tCpfeJpOv60C 3WgQ== X-Gm-Message-State: APt69E21LwxoW9OoXhbpx6SRb//WQbmpfX1Gji4CJnRr0bUJYQDfghzY G16Z/gQRc/IPAoSjWLV2Jxkh7Q== X-Google-Smtp-Source: AAOMgpfoPoDXyvA4GcGPyMpOScFGFwlfGWou22AtwKgiHqpdFxUTzWbmdjlOVw6xBmx7wH0/TRCKCw== X-Received: by 2002:adf:d24a:: with SMTP id o10-v6mr3742519wri.229.1530780352384; Thu, 05 Jul 2018 01:45:52 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id o21-v6sm1116950wmg.28.2018.07.05.01.45.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:51 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:35 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Xueming Li Message-ID: <20180705083934.5535-7-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 06/10] net/mlx5: add port representor awareness X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The current PCI probing method is not aware of Verbs port representors, which appear as standard Verbs devices bound to the same PCI address and cannot be distinguished. Problem is that more often than not, the wrong Verbs device is used, resulting in unexpected traffic. This patch makes the driver discard representors to only use the master device. If unable to identify it (e.g. kernel drivers not recent enough), either: - There is only one matching device which isn't identified as a representor, in that case use it. - Otherwise log an error and do not probe the device. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li Cc: Xueming Li --- v3 changes: - Replaced all heuristics (including mlx5_cmp_ibv_name()) with Netlink queries to associate IB devices with network interfaces. - Reworded commit log. v2 changes: - Fixed digit detection in mlx5_cmp_ibv_name() so that "foo1" and "foo10" are compared on the integer conversion of "1" against "10" instead of "" and "0". --- drivers/net/mlx5/Makefile | 30 ++++ drivers/net/mlx5/mlx5.c | 109 +++++++++++++-- drivers/net/mlx5/mlx5.h | 16 ++- drivers/net/mlx5/mlx5_nl.c | 297 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 428 insertions(+), 24 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 955861a41..745752e23 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -152,6 +152,36 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh infiniband/verbs.h \ enum IBV_FLOW_SPEC_ACTION_COUNT \ $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_CMD_GET \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_CMD_GET \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_CMD_PORT_GET \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_CMD_PORT_GET \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_DEV_INDEX \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_DEV_INDEX \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_DEV_NAME \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_DEV_NAME \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_PORT_INDEX \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_PORT_INDEX \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_NDEV_INDEX \ + $(AUTOCONF_OUTPUT) # Create mlx5_autoconf.h or update it in case it differs from the new one. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 1054bf6d0..d06ba9886 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -13,6 +13,7 @@ #include #include #include +#include #include /* Verbs header. */ @@ -274,8 +275,10 @@ mlx5_dev_close(struct rte_eth_dev *dev) mlx5_socket_uninit(dev); if (priv->config.vf) mlx5_nl_mac_addr_flush(dev); - if (priv->nl_socket >= 0) - close(priv->nl_socket); + if (priv->nl_socket_route >= 0) + close(priv->nl_socket_route); + if (priv->nl_socket_rdma >= 0) + close(priv->nl_socket_rdma); ret = mlx5_hrxq_ibv_verify(dev); if (ret) DRV_LOG(WARNING, "port %u some hash Rx queue still remain", @@ -876,6 +879,10 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->device_attr = attr; priv->pd = pd; priv->mtu = ETHER_MTU; + /* Some internal functions rely on Netlink sockets, open them now. */ + priv->nl_socket_rdma = mlx5_nl_init(0, NETLINK_RDMA); + priv->nl_socket_route = mlx5_nl_init(RTMGRP_LINK, NETLINK_ROUTE); + priv->nl_sn = 0; err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; @@ -1010,14 +1017,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, eth_dev->dev_ops = &mlx5_dev_ops; /* Register MAC address. */ claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); - priv->nl_socket = -1; - priv->nl_sn = 0; - if (vf && config.vf_nl_en) { - priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); - if (priv->nl_socket < 0) - priv->nl_socket = -1; + if (vf && config.vf_nl_en) mlx5_nl_mac_addr_sync(eth_dev); - } TAILQ_INIT(&priv->flows); TAILQ_INIT(&priv->ctrl_flows); /* Hint libmlx5 to use PMD allocator for data plane resources */ @@ -1078,8 +1079,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); return eth_dev; error: - if (priv) + if (priv) { + if (priv->nl_socket_route >= 0) + close(priv->nl_socket_route); + if (priv->nl_socket_rdma >= 0) + close(priv->nl_socket_rdma); rte_free(priv); + } if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); if (eth_dev) @@ -1110,6 +1116,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **ibv_list; struct rte_eth_dev *eth_dev = NULL; + unsigned int n = 0; int vf; int ret; @@ -1121,6 +1128,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } + + struct ibv_device *ibv_match[ret + 1]; + while (ret-- > 0) { struct rte_pci_addr pci_addr; @@ -1132,10 +1142,81 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid != pci_addr.devid || pci_dev->addr.function != pci_addr.function) continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + DRV_LOG(INFO, "PCI information matches for device \"%s\"", ibv_list[ret]->name); + ibv_match[n++] = ibv_list[ret]; + } + ibv_match[n] = NULL; + + unsigned int ifindex[n]; + struct mlx5_switch_info info[n]; + int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; + int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; + unsigned int i; + + /* + * The existence of several matching entries (n > 1) means port + * representors have been instantiated. No existing Verbs call nor + * /sys entries can tell them apart, this can only be done through + * Netlink calls assuming kernel drivers are recent enough to + * support them. + * + * In the event of identification failure through Netlink, either: + * + * 1. No device matches (n == 0), complain and bail out. + * 2. A single IB device matches (n == 1) and is not a representor, + * assume no switch support. + * 3. Otherwise no safe assumptions can be made; complain louder and + * bail out. + */ + for (i = 0; i != n; ++i) { + if (nl_rdma < 0) + ifindex[i] = 0; + else + ifindex[i] = mlx5_nl_ifindex(nl_rdma, + ibv_match[i]->name); + if (nl_route < 0 || + !ifindex[i] || + mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { + ifindex[i] = 0; + memset(&info[i], 0, sizeof(info[i])); + continue; + } + } + if (nl_rdma >= 0) + close(nl_rdma); + if (nl_route >= 0) + close(nl_route); + /* Look for master device. */ + for (i = 0; i != n; ++i) { + if (!info[i].master) + continue; + /* Make it the first entry. */ + if (i == 0) + break; + ibv_match[n] = ibv_match[0]; + ibv_match[0] = ibv_match[i]; + ibv_match[n] = NULL; break; } + if (n && i == n) { + if (n == 1 && !info[0].representor) { + /* Case #2. */ + DRV_LOG(INFO, "no switch support detected"); + } else if (n == 1) { + /* Case #3. */ + DRV_LOG(ERR, + "device looks like a port representor, this is" + " not supported yet"); + n = 0; + } else { + /* Case #3. */ + DRV_LOG(ERR, + "unable to tell which of the matching devices" + " is the master (lack of kernel support?)"); + n = 0; + } + } switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1146,10 +1227,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, default: vf = 0; } - if (ret >= 0) - eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + if (n) + eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); mlx5_glue->free_device_list(ibv_list); - if (!ret) { + if (!n) { DRV_LOG(WARNING, "no Verbs device matches PCI device " PCI_PRI_FMT "," " are kernel drivers loaded?", diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index f55ff4a21..704046270 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -53,6 +53,14 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX5BF = 0xa2d2, }; +/** Switch information returned by mlx5_nl_switch_info(). */ +struct mlx5_switch_info { + uint32_t master:1; /**< Master device. */ + uint32_t representor:1; /**< Representor device. */ + int32_t port_name; /**< Representor port name. */ + uint64_t switch_id; /**< Switch identifier. */ +}; + LIST_HEAD(mlx5_dev_list, priv); /* Shared memory between primary and secondary processes. */ @@ -195,7 +203,8 @@ struct priv { struct mlx5_dev_config config; /* Device configuration. */ struct mlx5_verbs_alloc_ctx verbs_alloc_ctx; /* Context for Verbs allocator. */ - int nl_socket; /* Netlink socket. */ + int nl_socket_rdma; /* Netlink socket (NETLINK_RDMA). */ + int nl_socket_route; /* Netlink socket (NETLINK_ROUTE). */ uint32_t nl_sn; /* Netlink message sequence number. */ }; @@ -342,7 +351,7 @@ int mlx5_socket_connect(struct rte_eth_dev *priv); /* mlx5_nl.c */ -int mlx5_nl_init(uint32_t nlgroups); +int mlx5_nl_init(uint32_t nlgroups, int protocol); int mlx5_nl_mac_addr_add(struct rte_eth_dev *dev, struct ether_addr *mac, uint32_t index); int mlx5_nl_mac_addr_remove(struct rte_eth_dev *dev, struct ether_addr *mac, @@ -351,5 +360,8 @@ void mlx5_nl_mac_addr_sync(struct rte_eth_dev *dev); void mlx5_nl_mac_addr_flush(struct rte_eth_dev *dev); int mlx5_nl_promisc(struct rte_eth_dev *dev, int enable); int mlx5_nl_allmulti(struct rte_eth_dev *dev, int enable); +unsigned int mlx5_nl_ifindex(int nl, const char *name); +int mlx5_nl_switch_info(int nl, unsigned int ifindex, + struct mlx5_switch_info *info); #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_nl.c b/drivers/net/mlx5/mlx5_nl.c index dca85835a..58ef2f4f0 100644 --- a/drivers/net/mlx5/mlx5_nl.c +++ b/drivers/net/mlx5/mlx5_nl.c @@ -3,10 +3,21 @@ * Copyright 2018 Mellanox Technologies, Ltd */ +#include +#include #include #include +#include +#include +#include +#include +#include +#include +#include #include +#include + #include "mlx5.h" #include "mlx5_utils.h" @@ -27,6 +38,29 @@ ((struct rtattr *)(((char *)(r)) + NLMSG_ALIGN(sizeof(struct ndmsg)))) #endif +/* + * The following definitions are normally found in rdma/rdma_netlink.h, + * however they are so recent that most systems do not expose them yet. + */ +#ifndef HAVE_RDMA_NLDEV_CMD_GET +#define RDMA_NLDEV_CMD_GET 1 +#endif +#ifndef HAVE_RDMA_NLDEV_CMD_PORT_GET +#define RDMA_NLDEV_CMD_PORT_GET 5 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_DEV_INDEX +#define RDMA_NLDEV_ATTR_DEV_INDEX 1 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_DEV_NAME +#define RDMA_NLDEV_ATTR_DEV_NAME 2 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_PORT_INDEX +#define RDMA_NLDEV_ATTR_PORT_INDEX 3 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX +#define RDMA_NLDEV_ATTR_NDEV_INDEX 50 +#endif + /* Add/remove MAC address through Netlink */ struct mlx5_nl_mac_addr { struct ether_addr (*mac)[]; @@ -34,18 +68,27 @@ struct mlx5_nl_mac_addr { int mac_n; /**< Number of addresses in the array. */ }; +/** Data structure used by mlx5_nl_ifindex_cb(). */ +struct mlx5_nl_ifindex_data { + const char *name; /**< IB device name (in). */ + uint32_t ibindex; /**< IB device index (out). */ + uint32_t ifindex; /**< Network interface index (out). */ +}; + /** * Opens a Netlink socket. * * @param nl_groups * Netlink group value (e.g. RTMGRP_LINK). + * @param protocol + * Netlink protocol (e.g. NETLINK_ROUTE, NETLINK_RDMA). * * @return * A file descriptor on success, a negative errno value otherwise and * rte_errno is set. */ int -mlx5_nl_init(uint32_t nl_groups) +mlx5_nl_init(uint32_t nl_groups, int protocol) { int fd; int sndbuf_size = MLX5_SEND_BUF_SIZE; @@ -56,7 +99,7 @@ mlx5_nl_init(uint32_t nl_groups) }; int ret; - fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE); + fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, protocol); if (fd == -1) { rte_errno = errno; return -rte_errno; @@ -334,9 +377,9 @@ mlx5_nl_mac_addr_list(struct rte_eth_dev *dev, struct ether_addr (*mac)[], int ret; uint32_t sn = priv->nl_sn++; - if (priv->nl_socket == -1) + if (priv->nl_socket_route == -1) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; ret = mlx5_nl_request(fd, &req.hdr, sn, &req.ifm, sizeof(struct ifinfomsg)); if (ret < 0) @@ -398,9 +441,9 @@ mlx5_nl_mac_addr_modify(struct rte_eth_dev *dev, struct ether_addr *mac, int ret; uint32_t sn = priv->nl_sn++; - if (priv->nl_socket == -1) + if (priv->nl_socket_route == -1) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; memcpy(RTA_DATA(&req.rta), mac, ETHER_ADDR_LEN); req.hdr.nlmsg_len = NLMSG_ALIGN(req.hdr.nlmsg_len) + RTA_ALIGN(req.rta.rta_len); @@ -569,9 +612,9 @@ mlx5_nl_device_flags(struct rte_eth_dev *dev, uint32_t flags, int enable) int ret; assert(!(flags & ~(IFF_PROMISC | IFF_ALLMULTI))); - if (priv->nl_socket < 0) + if (priv->nl_socket_route < 0) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; ret = mlx5_nl_send(fd, &req.hdr, priv->nl_sn++); if (ret < 0) return ret; @@ -625,3 +668,241 @@ mlx5_nl_allmulti(struct rte_eth_dev *dev, int enable) strerror(rte_errno)); return ret; } + +/** + * Process network interface information from Netlink message. + * + * @param nh + * Pointer to Netlink message header. + * @param arg + * Opaque data pointer for this callback. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_nl_ifindex_cb(struct nlmsghdr *nh, void *arg) +{ + struct mlx5_nl_ifindex_data *data = arg; + size_t off = NLMSG_HDRLEN; + uint32_t ibindex = 0; + uint32_t ifindex = 0; + int found = 0; + + if (nh->nlmsg_type != + RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_GET) && + nh->nlmsg_type != + RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_PORT_GET)) + goto error; + while (off < nh->nlmsg_len) { + struct nlattr *na = (void *)((uintptr_t)nh + off); + void *payload = (void *)((uintptr_t)na + NLA_HDRLEN); + + if (na->nla_len > nh->nlmsg_len - off) + goto error; + switch (na->nla_type) { + case RDMA_NLDEV_ATTR_DEV_INDEX: + ibindex = *(uint32_t *)payload; + break; + case RDMA_NLDEV_ATTR_DEV_NAME: + if (!strcmp(payload, data->name)) + found = 1; + break; + case RDMA_NLDEV_ATTR_NDEV_INDEX: + ifindex = *(uint32_t *)payload; + break; + default: + break; + } + off += NLA_ALIGN(na->nla_len); + } + if (found) { + data->ibindex = ibindex; + data->ifindex = ifindex; + } + return 0; +error: + rte_errno = EINVAL; + return -rte_errno; +} + +/** + * Get index of network interface associated with some IB device. + * + * This is the only somewhat safe method to avoid resorting to heuristics + * when faced with port representors. Unfortunately it requires at least + * Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * + * @return + * A valid (nonzero) interface index on success, 0 otherwise and rte_errno + * is set. + */ +unsigned int +mlx5_nl_ifindex(int nl, const char *name) +{ + static const uint32_t pindex = 1; + uint32_t seq = random(); + struct mlx5_nl_ifindex_data data = { + .name = name, + .ibindex = 0, /* Determined during first pass. */ + .ifindex = 0, /* Determined during second pass. */ + }; + union { + struct nlmsghdr nh; + uint8_t buf[NLMSG_HDRLEN + + NLA_HDRLEN + NLA_ALIGN(sizeof(data.ibindex)) + + NLA_HDRLEN + NLA_ALIGN(sizeof(pindex))]; + } req = { + .nh = { + .nlmsg_len = NLMSG_LENGTH(0), + .nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, + RDMA_NLDEV_CMD_GET), + .nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK | NLM_F_DUMP, + }, + }; + struct nlattr *na; + int ret; + + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret < 0) + return 0; + ret = mlx5_nl_recv(nl, seq, mlx5_nl_ifindex_cb, &data); + if (ret < 0) + return 0; + if (!data.ibindex) + goto error; + ++seq; + req.nh.nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, + RDMA_NLDEV_CMD_PORT_GET); + req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK; + req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(req.buf) - NLMSG_HDRLEN); + na = (void *)((uintptr_t)req.buf + NLMSG_HDRLEN); + na->nla_len = NLA_HDRLEN + sizeof(data.ibindex); + na->nla_type = RDMA_NLDEV_ATTR_DEV_INDEX; + memcpy((void *)((uintptr_t)na + NLA_HDRLEN), + &data.ibindex, sizeof(data.ibindex)); + na = (void *)((uintptr_t)na + NLA_ALIGN(na->nla_len)); + na->nla_len = NLA_HDRLEN + sizeof(pindex); + na->nla_type = RDMA_NLDEV_ATTR_PORT_INDEX; + memcpy((void *)((uintptr_t)na + NLA_HDRLEN), + &pindex, sizeof(pindex)); + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret < 0) + return 0; + ret = mlx5_nl_recv(nl, seq, mlx5_nl_ifindex_cb, &data); + if (ret < 0) + return 0; + if (!data.ifindex) + goto error; + return data.ifindex; +error: + rte_errno = ENODEV; + return 0; +} + +/** + * Process switch information from Netlink message. + * + * @param nh + * Pointer to Netlink message header. + * @param arg + * Opaque data pointer for this callback. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_nl_switch_info_cb(struct nlmsghdr *nh, void *arg) +{ + struct mlx5_switch_info info = { + .master = 0, + .representor = 0, + .port_name = 0, + .switch_id = 0, + }; + size_t off = NLMSG_LENGTH(sizeof(struct ifinfomsg)); + bool port_name_set = false; + bool switch_id_set = false; + + if (nh->nlmsg_type != RTM_NEWLINK) + goto error; + while (off < nh->nlmsg_len) { + struct rtattr *ra = (void *)((uintptr_t)nh + off); + void *payload = RTA_DATA(ra); + char *end; + unsigned int i; + + if (ra->rta_len > nh->nlmsg_len - off) + goto error; + switch (ra->rta_type) { + case IFLA_PHYS_PORT_NAME: + errno = 0; + info.port_name = strtol(payload, &end, 0); + if (errno || + (size_t)(end - (char *)payload) != strlen(payload)) + goto error; + port_name_set = true; + break; + case IFLA_PHYS_SWITCH_ID: + info.switch_id = 0; + for (i = 0; i < RTA_PAYLOAD(ra); ++i) { + info.switch_id <<= 8; + info.switch_id |= ((uint8_t *)payload)[i]; + } + switch_id_set = true; + break; + } + off += RTA_ALIGN(ra->rta_len); + } + info.master = switch_id_set && !port_name_set; + info.representor = switch_id_set && port_name_set; + memcpy(arg, &info, sizeof(info)); + return 0; +error: + rte_errno = EINVAL; + return -rte_errno; +} + +/** + * Get switch information associated with network interface. + * + * @param nl + * Netlink socket of the ROUTE kind (NETLINK_ROUTE). + * @param ifindex + * Network interface index. + * @param[out] info + * Switch information object, populated in case of success. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_nl_switch_info(int nl, unsigned int ifindex, struct mlx5_switch_info *info) +{ + uint32_t seq = random(); + struct { + struct nlmsghdr nh; + struct ifinfomsg info; + } req = { + .nh = { + .nlmsg_len = NLMSG_LENGTH(sizeof(req.info)), + .nlmsg_type = RTM_GETLINK, + .nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK, + }, + .info = { + .ifi_family = AF_UNSPEC, + .ifi_index = ifindex, + }, + }; + int ret; + + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret >= 0) + ret = mlx5_nl_recv(nl, seq, mlx5_nl_switch_info_cb, info); + return ret; +} From patchwork Thu Jul 5 08:45:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42308 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 27A931C3A0; Thu, 5 Jul 2018 10:45:56 +0200 (CEST) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 181AD1C39F for ; Thu, 5 Jul 2018 10:45:55 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id n17-v6so9502227wmh.2 for ; Thu, 05 Jul 2018 01:45:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VhC3Y2HO4ZXiMa/bUy+RgaIjhKfcca8X0XmZin+OFTQ=; b=HVPkxtmNaUEG51UcyQicwKhs19zFyDsBdYEiw9nbX4iB4/1UaUCj/hdVzJ+qY7x284 Rnr4xdmqKTknK67u91rieFzABELcs74SluLRR+vJ2sYJM6a1b9+T7aWWaIs/NlcH9RGf hQNzJ+5Qq+nmNARVl2UfTZYB2/w545c1gsj6kBEZcpHnfdCxb9hz2mejPn94nKC2RAKk HphPNhqElWXOPG5ftbQUk+nMV34bS67tXzNqg5IdHfS9bEwfxoK3+V9xtAWlkX87piRE Uhgw/LyWZx39tNkMRZMWSuVYYf2ahLI6WRSlfO+e/iSgcbHFGQQDSGCEcBk03FmvcFb5 81bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VhC3Y2HO4ZXiMa/bUy+RgaIjhKfcca8X0XmZin+OFTQ=; b=Ak2NpwA6gSz/CR6ju/DEyagSYvxJlMAlTIJHmYgrZPQQVNDX2+sr0B9lefJb3KuXpK 5sz5lCFUQIl3+1NjRFW3z9pHJiuKw69HEWhdmQrsonDXYgyorxCZAD9ZWYSwkXbDBOAW MIYp47wQ5YDMncQI2Ymz5751SofyMT7EOep2IXIceYzGKDY/i/JpFkl4ojyd2Gd4bXhY YNuOKMwUMs3NCgE4WtZlVH7wNREzDOWE+kTwpI7wbw8cPSniOfuFoC4WKuZtSuWOPUfE 9Z6vtx77GDxB/0wH9OaxJ72GfEMSEkt1JNvbKQP4jy7ITXrIqjU/DlwF71QUEZ05BOfA 0NVg== X-Gm-Message-State: APt69E1wBYro1h+3cD+69ahjvl4BLqSkgb+3fya4G/NdAIYCYkBMpL3F H/Wkmpz6AfHkw08beKDWNQxzng== X-Google-Smtp-Source: AAOMgpf4ptbPfgOKVqXmBVi3SpUA2gt8HxH8gSbQz/HvpLSZ2a4mSvNKwLG0UZBf14BwaB997NIG2Q== X-Received: by 2002:a1c:d8:: with SMTP id 207-v6mr3223131wma.99.1530780354416; Thu, 05 Jul 2018 01:45:54 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id c18-v6sm7541423wrq.17.2018.07.05.01.45.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:53 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:37 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Nelio Laranjeiro , Xueming Li Message-ID: <20180705083934.5535-8-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 07/10] net/mlx5: probe all port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Probe existing port representors in addition to their master device and associate them automatically. To avoid collision between Ethernet devices, they are named as follows: - "{DBDF}" for master/switch devices. - "{DBDF}_representor_{rep}" with "rep" starting from 0 for port representors. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Signed-off-by: Nelio Laranjeiro Reviewed-by: Xueming Li Cc: Xueming Li Cc: Shahaf Shuler --- v4 changes: - Fixed domain ID release once the last port using it is closed. Closed devices are not necessarily detached, their presence is not a good indicator. Code was modified to check if they still use their domain IDs before deciding to release it. v3 changes: - Nelio introduced mlx5_dev_to_port_id() to prevent the master device from releasing a domain ID while representors are still bound. It is now released by the last device closed. - Reverted to original naming convention as requested by Xueming and Shahaf; "net_" prefix and "_0" suffix were dropped. - mlx5_dev_spawn() (previously mlx5_dev_spawn_one()) now decides on its own whether underlying device is a representor. - Devices can now be probed in any order and not necessarily all at once; representors can exist without a master device. - mlx5_pci_probe() iterates on the list of devices directly instead of relying on an intermediate function (previously mlx5_dev_spawn()). - mlx5_get_ifname() was rewritten to rely on mlx5_nl_ifindex() when faced with a representor. - Since it is not necessarily present, master device is now dynamically retrieved in mlx5_dev_infos_get(). v2 changes: - Added representor information to dev_infos_get(). DPDK port ID of master device is now stored in the private structure to retrieve it conveniently. - Master device is assigned dummy representor ID value -1 to better distinguish from the the first actual representor reported by dev_infos_get() as those are indexed from 0. - Added RTE_ETH_DEV_REPRESENTOR device flag. --- drivers/net/mlx5/mlx5.c | 134 ++++++++++++++++++++++++++++-------- drivers/net/mlx5/mlx5.h | 12 +++- drivers/net/mlx5/mlx5_ethdev.c | 133 +++++++++++++++++++++++++++++++---- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_stats.c | 6 +- 5 files changed, 238 insertions(+), 49 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index d06ba9886..c02afbb82 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -307,7 +307,27 @@ mlx5_dev_close(struct rte_eth_dev *dev) if (ret) DRV_LOG(WARNING, "port %u some flows still remain", dev->data->port_id); + if (priv->domain_id != RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) { + unsigned int c = 0; + unsigned int i = mlx5_dev_to_port_id(dev->device, NULL, 0); + uint16_t port_id[i]; + + i = RTE_MIN(mlx5_dev_to_port_id(dev->device, port_id, i), i); + while (i--) { + struct priv *opriv = + rte_eth_devices[port_id[i]].data->dev_private; + + if (!opriv || + opriv->domain_id != priv->domain_id || + &rte_eth_devices[port_id[i]] == dev) + continue; + ++c; + } + if (!c) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); + } memset(priv, 0, sizeof(*priv)); + priv->domain_id = RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID; } const struct eth_dev_ops mlx5_dev_ops = { @@ -647,6 +667,8 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) * Verbs device. * @param vf * If nonzero, enable VF-specific features. + * @param[in] switch_info + * Switch properties of Ethernet device. * * @return * A valid Ethernet device object on success, NULL otherwise and rte_errno @@ -655,7 +677,8 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) static struct rte_eth_dev * mlx5_dev_spawn(struct rte_device *dpdk_dev, struct ibv_device *ibv_dev, - int vf) + int vf, + const struct mlx5_switch_info *switch_info) { struct ibv_context *ctx; struct ibv_device_attr_ex attr; @@ -697,6 +720,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, #endif struct ether_addr mac; char name[RTE_ETH_NAME_MAX_LEN]; + int own_domain_id = 0; + unsigned int i; /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); @@ -805,7 +830,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DEBUG("ibv_query_device_ex() failed"); goto error; } - rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + if (!switch_info->representor) + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + else + snprintf(name, sizeof(name), "%s_representor_%u", + dpdk_dev->name, switch_info->port_name); + DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -874,6 +904,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, goto error; } priv->ctx = ctx; + strncpy(priv->ibdev_name, priv->ctx->device->name, + sizeof(priv->ibdev_name)); strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); priv->device_attr = attr; @@ -883,6 +915,41 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->nl_socket_rdma = mlx5_nl_init(0, NETLINK_RDMA); priv->nl_socket_route = mlx5_nl_init(RTMGRP_LINK, NETLINK_ROUTE); priv->nl_sn = 0; + priv->representor = !!switch_info->representor; + priv->domain_id = RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID; + priv->representor_id = + switch_info->representor ? switch_info->port_name : -1; + /* + * Look for sibling devices in order to reuse their switch domain + * if any, otherwise allocate one. + */ + i = mlx5_dev_to_port_id(dpdk_dev, NULL, 0); + if (i > 0) { + uint16_t port_id[i]; + + i = RTE_MIN(mlx5_dev_to_port_id(dpdk_dev, port_id, i), i); + while (i--) { + const struct priv *opriv = + rte_eth_devices[port_id[i]].data->dev_private; + + if (!opriv || + opriv->domain_id == + RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) + continue; + priv->domain_id = opriv->domain_id; + break; + } + } + if (priv->domain_id == RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) { + err = rte_eth_switch_domain_alloc(&priv->domain_id); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "unable to allocate switch domain: %s", + strerror(rte_errno)); + goto error; + } + own_domain_id = 1; + } err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; @@ -966,6 +1033,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err = ENOMEM; goto error; } + if (priv->representor) + eth_dev->data->dev_flags |= RTE_ETH_DEV_REPRESENTOR; eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; @@ -1084,6 +1153,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, close(priv->nl_socket_route); if (priv->nl_socket_rdma >= 0) close(priv->nl_socket_rdma); + if (own_domain_id) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); rte_free(priv); } if (pd) @@ -1100,7 +1171,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, /** * DPDK callback to register a PCI device. * - * This function spawns an Ethernet device out of a given PCI device. + * This function spawns Ethernet devices out of a given PCI device. * * @param[in] pci_drv * PCI driver structure (mlx5_driver). @@ -1115,7 +1186,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) { struct ibv_device **ibv_list; - struct rte_eth_dev *eth_dev = NULL; unsigned int n = 0; int vf; int ret; @@ -1150,9 +1220,11 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int ifindex[n]; struct mlx5_switch_info info[n]; + struct rte_eth_dev *eth_list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; + unsigned int u; /* * The existence of several matching entries (n > 1) means port @@ -1187,28 +1259,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_rdma); if (nl_route >= 0) close(nl_route); - /* Look for master device. */ - for (i = 0; i != n; ++i) { - if (!info[i].master) - continue; - /* Make it the first entry. */ - if (i == 0) - break; - ibv_match[n] = ibv_match[0]; - ibv_match[0] = ibv_match[i]; - ibv_match[n] = NULL; - break; - } - if (n && i == n) { - if (n == 1 && !info[0].representor) { + /* Count unidentified devices. */ + for (u = 0, i = 0; i != n; ++i) + if (!info[i].master && !info[i].representor) + ++u; + if (u) { + if (n == 1 && u == 1) { /* Case #2. */ DRV_LOG(INFO, "no switch support detected"); - } else if (n == 1) { - /* Case #3. */ - DRV_LOG(ERR, - "device looks like a port representor, this is" - " not supported yet"); - n = 0; } else { /* Case #3. */ DRV_LOG(ERR, @@ -1227,8 +1285,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, default: vf = 0; } - if (n) - eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); + for (i = 0; i != n; ++i) { + uint32_t restore; + + eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], + vf, &info[i]); + if (!eth_list[i]) + break; + restore = eth_list[i]->data->dev_flags; + rte_eth_copy_pci_info(eth_list[i], pci_dev); + /* Restore non-PCI flags cleared by the above call. */ + eth_list[i]->data->dev_flags |= restore; + rte_eth_dev_probing_finish(eth_list[i]); + } mlx5_glue->free_device_list(ibv_list); if (!n) { DRV_LOG(WARNING, @@ -1238,7 +1307,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid, pci_dev->addr.function); rte_errno = ENOENT; ret = -rte_errno; - } else if (!eth_dev) { + } else if (i != n) { DRV_LOG(ERR, "probe of PCI device " PCI_PRI_FMT " aborted after" " encountering an error: %s", @@ -1246,9 +1315,16 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid, pci_dev->addr.function, strerror(rte_errno)); ret = -rte_errno; + /* Roll back. */ + while (i--) { + mlx5_dev_close(eth_list[i]); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_free(eth_list[i]->data->dev_private); + claim_zero(rte_eth_dev_release_port(eth_list[i])); + } + /* Restore original error. */ + rte_errno = -ret; } else { - rte_eth_copy_pci_info(eth_dev, pci_dev); - rte_eth_dev_probing_finish(eth_dev); ret = 0; } return ret; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 704046270..cc01310e0 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -159,6 +159,7 @@ struct priv { struct ibv_context *ctx; /* Verbs context. */ struct ibv_device_attr_ex device_attr; /* Device properties. */ struct ibv_pd *pd; /* Protection Domain. */ + char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */ char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for secondary */ struct ether_addr mac[MLX5_MAX_MAC_ADDRESSES]; /* MAC addresses. */ BITFIELD_DECLARE(mac_own, uint64_t, MLX5_MAX_MAC_ADDRESSES); @@ -168,6 +169,9 @@ struct priv { /* Device properties. */ uint16_t mtu; /* Configured MTU. */ unsigned int isolated:1; /* Whether isolated mode is enabled. */ + unsigned int representor:1; /* Device is a port representor. */ + uint16_t domain_id; /* Switch domain identifier. */ + int32_t representor_id; /* Port representor identifier. */ /* RX/TX queues. */ unsigned int rxqs_n; /* RX queues array size. */ unsigned int txqs_n; /* TX queues array size. */ @@ -217,9 +221,12 @@ int mlx5_getenv_int(const char *); /* mlx5_ethdev.c */ +int mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]); int mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]); int mlx5_ifindex(const struct rte_eth_dev *dev); -int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr); +int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master); int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu); int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags); @@ -244,6 +251,9 @@ int mlx5_set_link_up(struct rte_eth_dev *dev); int mlx5_is_removed(struct rte_eth_dev *dev); eth_tx_burst_t mlx5_select_tx_function(struct rte_eth_dev *dev); eth_rx_burst_t mlx5_select_rx_function(struct rte_eth_dev *dev); +unsigned int mlx5_dev_to_port_id(const struct rte_device *dev, + uint16_t *port_list, + unsigned int port_list_n); /* mlx5_mac.c */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 819f5baad..05f66f7b6 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -27,6 +27,7 @@ #include #include +#include #include #include #include @@ -93,7 +94,7 @@ struct ethtool_link_settings { #endif /** - * Get interface name from private structure. + * Get master interface name from private structure. * * @param[in] dev * Pointer to Ethernet device. @@ -104,7 +105,8 @@ struct ethtool_link_settings { * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]) { struct priv *priv = dev->data->dev_private; DIR *dir; @@ -179,6 +181,39 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) } /** + * Get interface name from private structure. + * + * This is a port representor-aware version of mlx5_get_master_ifname(). + * + * @param[in] dev + * Pointer to Ethernet device. + * @param[out] ifname + * Interface name output buffer. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +{ + struct priv *priv = dev->data->dev_private; + unsigned int ifindex = + priv->nl_socket_rdma >= 0 ? + mlx5_nl_ifindex(priv->nl_socket_rdma, priv->ibdev_name) : 0; + + if (!ifindex) { + if (!priv->representor) + return mlx5_get_master_ifname(dev, ifname); + rte_errno = ENXIO; + return -rte_errno; + } + if (if_indextoname(ifindex, &(*ifname)[0])) + return 0; + rte_errno = errno; + return -rte_errno; +} + +/** * Get the interface index from device name. * * @param[in] dev @@ -214,12 +249,16 @@ mlx5_ifindex(const struct rte_eth_dev *dev) * Request number to pass to ioctl(). * @param[out] ifr * Interface request structure output buffer. + * @param master + * When device is a port representor, perform request on master device + * instead. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) +mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master) { int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP); int ret = 0; @@ -228,7 +267,10 @@ mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) rte_errno = errno; return -rte_errno; } - ret = mlx5_get_ifname(dev, &ifr->ifr_name); + if (master) + ret = mlx5_get_master_ifname(dev, &ifr->ifr_name); + else + ret = mlx5_get_ifname(dev, &ifr->ifr_name); if (ret) goto error; ret = ioctl(sock, req, ifr); @@ -258,7 +300,7 @@ int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request); + int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request, 0); if (ret) return ret; @@ -282,7 +324,7 @@ mlx5_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) { struct ifreq request = { .ifr_mtu = mtu, }; - return mlx5_ifreq(dev, SIOCSIFMTU, &request); + return mlx5_ifreq(dev, SIOCSIFMTU, &request, 0); } /** @@ -302,13 +344,13 @@ int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request); + int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request, 0); if (ret) return ret; request.ifr_flags &= keep; request.ifr_flags |= flags & ~keep; - return mlx5_ifreq(dev, SIOCSIFFLAGS, &request); + return mlx5_ifreq(dev, SIOCSIFFLAGS, &request, 0); } /** @@ -477,6 +519,30 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) info->speed_capa = priv->link_speed_capa; info->flow_type_rss_offloads = ~MLX5_RSS_HF_MASK; mlx5_set_default_params(dev, info); + info->switch_info.name = dev->data->name; + info->switch_info.domain_id = priv->domain_id; + info->switch_info.port_id = priv->representor_id; + if (priv->representor) { + unsigned int i = mlx5_dev_to_port_id(dev->device, NULL, 0); + uint16_t port_id[i]; + + i = RTE_MIN(mlx5_dev_to_port_id(dev->device, port_id, i), i); + while (i--) { + struct priv *opriv = + rte_eth_devices[port_id[i]].data->dev_private; + + if (!opriv || + opriv->representor || + opriv->domain_id != priv->domain_id) + continue; + /* + * Override switch name with that of the master + * device. + */ + info->switch_info.name = opriv->dev_data->name; + break; + } + } } /** @@ -540,7 +606,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, int link_speed = 0; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -550,7 +616,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&edata; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GSET) failed: %s", @@ -611,7 +677,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, uint64_t sc; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -621,7 +687,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&gcmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -638,7 +704,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, *ecmd = gcmd; ifr.ifr_data = (void *)ecmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -801,7 +867,7 @@ mlx5_dev_get_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) int ret; ifr.ifr_data = (void *)ðpause; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GPAUSEPARAM) failed:" @@ -854,7 +920,7 @@ mlx5_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) ethpause.tx_pause = 1; else ethpause.tx_pause = 0; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 0); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_SPAUSEPARAM)" @@ -1193,3 +1259,40 @@ mlx5_is_removed(struct rte_eth_dev *dev) return 1; return 0; } + +/** + * Get port ID list of mlx5 instances sharing a common device. + * + * @param[in] dev + * Device to look for. + * @param[out] port_list + * Result buffer for collected port IDs. + * @param port_list_n + * Maximum number of entries in result buffer. If 0, @p port_list can be + * NULL. + * + * @return + * Number of matching instances regardless of the @p port_list_n + * parameter, 0 if none were found. + */ +unsigned int +mlx5_dev_to_port_id(const struct rte_device *dev, uint16_t *port_list, + unsigned int port_list_n) +{ + uint16_t id; + unsigned int n = 0; + + RTE_ETH_FOREACH_DEV(id) { + struct rte_eth_dev *ldev = &rte_eth_devices[id]; + + if (!ldev->device || + !ldev->device->driver || + strcmp(ldev->device->driver->name, MLX5_DRIVER_NAME) || + ldev->device != dev) + continue; + if (n < port_list_n) + port_list[n] = id; + n++; + } + return n; +} diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index 672a47619..12ee37f55 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -49,7 +49,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[ETHER_ADDR_LEN]) struct ifreq request; int ret; - ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request); + ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request, 0); if (ret) return ret; memcpy(mac, request.ifr_hwaddr.sa_data, ETHER_ADDR_LEN); diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c index 875dd1027..91f3d474a 100644 --- a/drivers/net/mlx5/mlx5_stats.c +++ b/drivers/net/mlx5/mlx5_stats.c @@ -146,7 +146,7 @@ mlx5_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) et_stats->cmd = ETHTOOL_GSTATS; et_stats->n_stats = xstats_ctrl->stats_n; ifr.ifr_data = (caddr_t)et_stats; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to read statistic values from device", @@ -194,7 +194,7 @@ mlx5_ethtool_get_stats_n(struct rte_eth_dev *dev) { drvinfo.cmd = ETHTOOL_GDRVINFO; ifr.ifr_data = (caddr_t)&drvinfo; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to query number of statistics", dev->data->port_id); @@ -244,7 +244,7 @@ mlx5_xstats_init(struct rte_eth_dev *dev) strings->string_set = ETH_SS_STATS; strings->len = dev_stats_n; ifr.ifr_data = (caddr_t)strings; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to get statistic names", dev->data->port_id); From patchwork Thu Jul 5 08:45:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42309 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2FBE31C3A9; Thu, 5 Jul 2018 10:45:59 +0200 (CEST) Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by dpdk.org (Postfix) with ESMTP id 814281C3A3 for ; Thu, 5 Jul 2018 10:45:56 +0200 (CEST) Received: by mail-wm0-f66.google.com with SMTP id b188-v6so9713149wme.3 for ; Thu, 05 Jul 2018 01:45:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=gTHBmySYHBQHoLvZAoF0Xe2V4bN3rZ/UdTVO1zNxJfCjkumCdkOJlaoK2vb0g3FXRv j5NazypJ2yv1uMFhdeAi6RrxJvNBlfkxAylfMF+L6bp/Wc10IXHQZ8cO/+hFKJ4VfnMy a9HcA1xbgIFT04sPnmDk9MRed7WKh3RpGOEmUffexkPtRttoK9nP9l477VAjJzPb12tR NQvAnLGfX6cNh8H8+IYH1lU5ZR2f//vN5l5Clbgm72i6Es6FLy7+qu7rDpqTjidBbCzA f337cIWqR+PWQoPY1+nSDGAmEtjsSkXIa8B0vYGD+d1aMGaaN0LgM/glI+oLAQJ7O+8W qr9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=nuK8owX0ZlL+LVo343hc++FUGFoFZaeuSf1/oo5+T4Oigm578P734tPSJbuK4GBEJV z4xBJIcLkhMSjXzkmiLN/mgBdPebRu5VcRAqjAeu7RkJ+9IO+IMS5fmuyEZNnHebx170 mo2D1Y847bWW6oguAKIMYBrYvobz+mGSyN69R4tZHecTDLEW92vraGB8m5hrzcJffYFu 3UwA8DIZ9d50Doc3it24Ls8uXQ+VsnPh+225YL/40fTd6mclKOXflyVEwEPuojdO28HA KyGx+b+qSRkvYTvHsAAkoUGNshaqfZmCkGjXWOtJG+YJMzJpRqDKAndK5/eJID5Guqsq fZag== X-Gm-Message-State: APt69E15aqX0KN+i+hwmW+EYxyCTopr0HRseO6rCWPRFDiJ88tRWu02k xV1DsGq8itnTBWXuVUlfoHBG8A== X-Google-Smtp-Source: AAOMgpc9v+juxXacGpW533+Jaa6K3EW747X1SDJ04m8spU45EkIdAz3I5xM1Klayj7Oc9UInCg4jMA== X-Received: by 2002:adf:fc47:: with SMTP id e7-v6mr3765255wrs.157.1530780356289; Thu, 05 Jul 2018 01:45:56 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 77-v6sm5031472wmq.22.2018.07.05.01.45.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:55 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:39 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-9-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 08/10] net/mlx5: probe port representors in natural order X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Port representors are probed in whatever unspecified order ibv_get_device_list() returns them. This is counterintuitive to users since DPDK port IDs assignment almost never follows the same sequence as representor IDs. Additionally, the master device does not necessarily inherit the lowest DPDK port ID. Signed-off-by: Adrien Mazarguil --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 95 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c02afbb82..6592480bf 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1168,6 +1168,52 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, return NULL; } +/** Data associated with devices to spawn. */ +struct mlx5_dev_spawn_data { + unsigned int ifindex; /**< Network interface index. */ + struct mlx5_switch_info info; /**< Switch information. */ + struct ibv_device *ibv_dev; /**< Associated IB device. */ + struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ +}; + +/** + * Comparison callback to sort device data. + * + * This is meant to be used with qsort(). + * + * @param a[in] + * Pointer to pointer to first data object. + * @param b[in] + * Pointer to pointer to second data object. + * + * @return + * 0 if both objects are equal, less than 0 if the first argument is less + * than the second, greater than 0 otherwise. + */ +static int +mlx5_dev_spawn_data_cmp(const void *a, const void *b) +{ + const struct mlx5_switch_info *si_a = + &((const struct mlx5_dev_spawn_data *)a)->info; + const struct mlx5_switch_info *si_b = + &((const struct mlx5_dev_spawn_data *)b)->info; + int ret; + + /* Master device first. */ + ret = si_b->master - si_a->master; + if (ret) + return ret; + /* Then representor devices. */ + ret = si_b->representor - si_a->representor; + if (ret) + return ret; + /* Unidentified devices come last in no specific order. */ + if (!si_a->representor) + return 0; + /* Order representors by name. */ + return si_a->port_name - si_b->port_name; +} + /** * DPDK callback to register a PCI device. * @@ -1218,9 +1264,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } ibv_match[n] = NULL; - unsigned int ifindex[n]; - struct mlx5_switch_info info[n]; - struct rte_eth_dev *eth_list[n]; + struct mlx5_dev_spawn_data list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; @@ -1242,16 +1286,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * bail out. */ for (i = 0; i != n; ++i) { + list[i].ibv_dev = ibv_match[i]; + list[i].eth_dev = NULL; if (nl_rdma < 0) - ifindex[i] = 0; + list[i].ifindex = 0; else - ifindex[i] = mlx5_nl_ifindex(nl_rdma, - ibv_match[i]->name); + list[i].ifindex = mlx5_nl_ifindex + (nl_rdma, list[i].ibv_dev->name); if (nl_route < 0 || - !ifindex[i] || - mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { - ifindex[i] = 0; - memset(&info[i], 0, sizeof(info[i])); + !list[i].ifindex || + mlx5_nl_switch_info(nl_route, list[i].ifindex, + &list[i].info)) { + list[i].ifindex = 0; + memset(&list[i].info, 0, sizeof(list[i].info)); continue; } } @@ -1261,7 +1308,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_route); /* Count unidentified devices. */ for (u = 0, i = 0; i != n; ++i) - if (!info[i].master && !info[i].representor) + if (!list[i].info.master && !list[i].info.representor) ++u; if (u) { if (n == 1 && u == 1) { @@ -1275,6 +1322,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, n = 0; } } + /* + * Sort list to probe devices in natural order for users convenience + * (i.e. master first, then representors from lowest to highest ID). + */ + if (n) + qsort(list, n, sizeof(*list), mlx5_dev_spawn_data_cmp); switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1288,15 +1341,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, for (i = 0; i != n; ++i) { uint32_t restore; - eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], - vf, &info[i]); - if (!eth_list[i]) + list[i].eth_dev = mlx5_dev_spawn + (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); + if (!list[i].eth_dev) break; - restore = eth_list[i]->data->dev_flags; - rte_eth_copy_pci_info(eth_list[i], pci_dev); + restore = list[i].eth_dev->data->dev_flags; + rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ - eth_list[i]->data->dev_flags |= restore; - rte_eth_dev_probing_finish(eth_list[i]); + list[i].eth_dev->data->dev_flags |= restore; + rte_eth_dev_probing_finish(list[i].eth_dev); } mlx5_glue->free_device_list(ibv_list); if (!n) { @@ -1317,10 +1370,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { - mlx5_dev_close(eth_list[i]); + mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_free(eth_list[i]->data->dev_private); - claim_zero(rte_eth_dev_release_port(eth_list[i])); + rte_free(list[i].eth_dev->data->dev_private); + claim_zero(rte_eth_dev_release_port(list[i].eth_dev)); } /* Restore original error. */ rte_errno = -ret; From patchwork Thu Jul 5 08:45:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42310 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CB8881C3B1; Thu, 5 Jul 2018 10:46:00 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id 755201C3A3 for ; Thu, 5 Jul 2018 10:45:58 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id s13-v6so988010wmc.1 for ; Thu, 05 Jul 2018 01:45:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=4qelGfXDR0NYqw3vkex8nX1rMGAhfI/j9iDIJOyh370=; b=a5sVCA5aPwK2WiBVREF7uGQopLKM3JQr/2oWFjAJydgmfQjWMS4qlojUz0ruFKP6QH e5BdaCp0spKrbXKzp71UOhHqlaVplbnM1mzK/qpNaezWq7nWWfzMezNrLEHiz5/I+jZF rUAtK5ITQubK6pR1JY9gGbHcbKBpUnaY94FFaVF1eioncIrrVRLaN5TfbQKrLzysy91D P0e5MgZRO6XfbmOmgPwmO+wcAlKA1ejezu+5AZlSeGPzuKXugCgvPYH3jZZlBoJP4l4a ptqbvnVgHoLb2zZ3/o82ge4g0AtYeEbiByGposH/OWpEX9EEiRqRAIxRwGvkj13JK9dI Ec/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=4qelGfXDR0NYqw3vkex8nX1rMGAhfI/j9iDIJOyh370=; b=fhL4ZuVRd6A9hreJRzk0yL2JxSAHfyCvyD7iQkZuxHC4/8wKdUgN2j6tUH/lvA2/+b uqhrHeJ6rpgajXtHdKPxa3TcUrrehCbumq1uQRFg7bjN4+nj+vmgYv0hr8av2wDr3scP sX9y+TdHcMvJGNzwOFO9VOnEoCOP0hkz/WJe344sEU4esWZRO+jeALWaquToH5WKfKb3 xwfSsEdZWBM5iM8t6FPEXnWTgnDMCE1y4YX3yLkbol7TrH6MSC+58JA07AoPgftWse9r 7cqg4gIgEHE9HMPL3ejmKBAlhIWaxeNx6WBpndFAD1xlN4X0nI9qcKFw5nYwXWJDzUsG eSrw== X-Gm-Message-State: APt69E18P6imuWr8iKXro21PEw0v/BlLvaU/R4iDBhE1hTbqpd7zkmdv ohisYmqg2IbHMmSkGL84jzpf6A== X-Google-Smtp-Source: AAOMgpcLMc3k+a5XIuTLdVC8hDKsXO10G8D9T5RW9hbyk7hUPbjeQOzKTN5Itv5p5LvtjOo/3QvusQ== X-Received: by 2002:adf:bc92:: with SMTP id g18-v6mr3820550wrh.266.1530780358202; Thu, 05 Jul 2018 01:45:58 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 24-v6sm14879322wrz.94.2018.07.05.01.45.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:57 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:41 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-10-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 09/10] net/mlx5: add parameter for port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Prior to this patch, all port representors detected on a given device were probed and Ethernet devices instantiated for each of them. This patch adds support for the standard "representor" parameter, which implies that port representors are not probed by default anymore, except for the list provided through device arguments. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v3 changes: - Adapted representor detection to the reworked mlx5_dev_spawn(). v2 changes: - Added error message for when rte_eth_devargs_parse() fails. --- doc/guides/nics/mlx5.rst | 12 ++++++++ doc/guides/prog_guide/poll_mode_drv.rst | 2 ++ drivers/net/mlx5/mlx5.c | 41 ++++++++++++++++++++++++++-- 3 files changed, 52 insertions(+), 3 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 7dd9c1c5e..0d0d21727 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -392,6 +392,18 @@ Run-time configuration Disabled by default. +- ``representor`` parameter [list] + + This parameter can be used to instantiate DPDK Ethernet devices from + existing port (or VF) representors configured on the device. + + It is a standard parameter whose format is described in + :ref:`ethernet_device_standard_device_arguments`. + + For instance, to probe port representors 0 through 2:: + + representor=[0-2] + Firmware configuration ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index 4b69f6cbe..b2cf48354 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -360,6 +360,8 @@ Ethernet Device API The Ethernet device API exported by the Ethernet PMDs is described in the *DPDK API Reference*. +.. _ethernet_device_standard_device_arguments: + Ethernet Device Standard Device Arguments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 6592480bf..12a77afa8 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -92,6 +92,9 @@ /* Activate Netlink support in VF mode. */ #define MLX5_VF_NL_EN "vf_nl_en" +/* Select port representors to instantiate. */ +#define MLX5_REPRESENTOR "representor" + #ifndef HAVE_IBV_MLX5_MOD_MPW #define MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED (1 << 2) #define MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW (1 << 3) @@ -443,6 +446,9 @@ mlx5_args_check(const char *key, const char *val, void *opaque) struct mlx5_dev_config *config = opaque; unsigned long tmp; + /* No-op, port representors are processed in mlx5_dev_spawn(). */ + if (!strcmp(MLX5_REPRESENTOR, key)) + return 0; errno = 0; tmp = strtoul(val, NULL, 0); if (errno) { @@ -515,6 +521,7 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_RX_VEC_EN, MLX5_L3_VXLAN_EN, MLX5_VF_NL_EN, + MLX5_REPRESENTOR, NULL, }; struct rte_kvargs *kvlist; @@ -672,7 +679,9 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) * * @return * A valid Ethernet device object on success, NULL otherwise and rte_errno - * is set. + * is set. The following error is defined: + * + * EBUSY: device is not supposed to be spawned. */ static struct rte_eth_dev * mlx5_dev_spawn(struct rte_device *dpdk_dev, @@ -723,6 +732,26 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, int own_domain_id = 0; unsigned int i; + /* Determine if this port representor is supposed to be spawned. */ + if (switch_info->representor && dpdk_dev->devargs) { + struct rte_eth_devargs eth_da; + + err = rte_eth_devargs_parse(dpdk_dev->devargs->args, ð_da); + if (err) { + rte_errno = -err; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + return NULL; + } + for (i = 0; i < eth_da.nb_representor_ports; ++i) + if (eth_da.representor_ports[i] == + (uint16_t)switch_info->port_name) + break; + if (i == eth_da.nb_representor_ports) { + rte_errno = EBUSY; + return NULL; + } + } /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); errno = 0; @@ -1343,8 +1372,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, list[i].eth_dev = mlx5_dev_spawn (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); - if (!list[i].eth_dev) - break; + if (!list[i].eth_dev) { + if (rte_errno != EBUSY) + break; + /* Device is disabled, ignore it. */ + continue; + } restore = list[i].eth_dev->data->dev_flags; rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ @@ -1370,6 +1403,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { + if (!list[i].eth_dev) + continue; mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) rte_free(list[i].eth_dev->data->dev_private); From patchwork Thu Jul 5 08:45:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42311 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 48C441C3B5; Thu, 5 Jul 2018 10:46:02 +0200 (CEST) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 463361C3AD for ; Thu, 5 Jul 2018 10:46:00 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id p11-v6so9431718wmc.4 for ; Thu, 05 Jul 2018 01:46:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=AtLQ1t/HSGgaM2vxggtceXSTF01jOlclzYI1rcVg0bU=; b=yN+xLqqCMVVLa1WZNpHV/7EyVX4ym/UHz/vRxcL1/FMUVyPvNRIwOD2JxWLuX4Ogoj 5VzIq6U+doclOPCUigHw4CFK+eogQ0hQE70bP/OB31TKcGREa0u+KsxsmTdFLi1qaDHF 34dg7fSwPBMBo9bvWAuzaclxRrbg0WidJAdrlz03G91gBOTVLnUGekRztnxPzc9UNjdR ZFJjCxvt09AWL0Tr6aMzRPHZMME8NzPub3+iIuRPxQArt6BC0Hj7ztdMDEUgYd5cihJy hBP1PtEKik2XTLzojjP7DLJsW1DQ1ZUfqkM20qWPNVbMk5UOHIvQLrowp2nTvYHOyW8b 4dlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=AtLQ1t/HSGgaM2vxggtceXSTF01jOlclzYI1rcVg0bU=; b=IZznycbuFwbDveZKOO8NeluBrlqZoJOEVlw1CaSY54id2y+HyKrZ+yyqvU60i2i50B yLCzqecjiWFK2+2a3pOf6YzNs8FftO2QFXXgzl5nwKMrt6yrYCRy36GsHptbLW3PZgRm BhmuRCj2SgBtNVAXEcXf158NkNkekmwn6pqu0m4aVvjDUygUtAI3S81IF3daLNJvXqmG 86+z7wfbHUG4f6GhFYCDPeybAOgBd7xVWOMf7iwZc8JSfEDh4dGLFX6GjKnn/GKcOuEJ 11+nt1o6qcHWPecRgQu0uQd9IMbSvPg5PBs2iePxxJtxIIkPXZC4ltDhFvf1pCtk5Vax W7kw== X-Gm-Message-State: APt69E2nPX/9gbfPxHWv+hRtBa3tjHctGSuX7XDXhZvmJ/w2r/9+yoNJ UAoxy/vwazMfFEPCSmh2PForZQ== X-Google-Smtp-Source: AAOMgpcdr/FdX6ulxwZBsou/08yiVD8WGN+SYPkVarapA6RGYVK8zxI2ULL07hGAitCK9DVMJwD3GQ== X-Received: by 2002:a1c:c241:: with SMTP id s62-v6mr3713859wmf.146.1530780360109; Thu, 05 Jul 2018 01:46:00 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 24-v6sm14879448wrz.94.2018.07.05.01.45.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:59 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:43 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-11-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 10/10] net/mlx5: support negative identifiers for port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch brings support for BlueField representors. Signed-off-by: Adrien Mazarguil Cc: Shahaf Shuler --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 12a77afa8..df7f39844 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1330,6 +1330,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, memset(&list[i].info, 0, sizeof(list[i].info)); continue; } + /* + * Port representors not associated with any VFs (e.g. on + * BlueField devices) report -1 as a port identifier. + * Quietly set it to zero since DPDK only supports positive + * values. + */ + if (list[i].info.representor && list[i].info.port_name == -1) + list[i].info.port_name = 0; } if (nl_rdma >= 0) close(nl_rdma);