From patchwork Thu Jun 14 08:34:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 41079 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 613B51E416; Thu, 14 Jun 2018 10:35:09 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id 715C11E3B5 for ; Thu, 14 Jun 2018 10:35:06 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id p126-v6so9077079wmb.2 for ; Thu, 14 Jun 2018 01:35:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=H/ZpZT6xCaR2rDmZ27Gj3UrIaboXsXF3Wy3BQR/8tRM=; b=HuXjo8lxJrvXBob46Mq5y952rpKnfgE2n9W2WLI+yDpBfEVV28Q60ya1ivK1mCxrue wh1LR4M3SokYkmGoqdGikbP4xK/SinHy0cAivPNCB0+HhsF4BiuY7UrHiwEX+vkWnNpj ehyViifqyjeBj3mXmuq/BjwsMhKNATa0IE3oVVW6TO1vCDkp3lsX2lEGakre/coD8Ljx r90yZFa/oFMfDyEqdD9ENzbg/oa5Oj4YNYsBaRy4uj4irtsufYcHB7WAA/XFnmFDqxGs WyvDSjSHBx7SM7bTFs6G9WpIsTo2kF17aKdCtiE03YapUxYHdlzBgyoIeRsBlr3jj9ne 2x7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=H/ZpZT6xCaR2rDmZ27Gj3UrIaboXsXF3Wy3BQR/8tRM=; b=CHKJPcH0ZXlnF7fn6tHb87ttrzEL8fs6f235+HcxX2hrXwaB9AOrXBumXaSzOzHAdg qhnxsi2Xn51JNotYaW/YVPoVClb0iM2H73sBJaxNXw/Ga5SW1NmKs1jIgm5gBVrCHHdZ bX8fax6ie+vk7OJzmNqtwKFUvmdslZlmc9MNbC6lYHhEonc6bEHmwlqoIJBhJAzIf9MJ OmKhwSSCe9pvHAVu5KppMEw++EK0zmHh7owAg+GmqenSk44sB7BMVV1tezTN2UTvWYRr 5F/ckj/dZyiwMcXoPbL5e9PmVYNRJ+MhBoeYIilXJwjyLGR45WcZveN4ko0nH38cdCSJ j6AQ== X-Gm-Message-State: APt69E2ntgrwRBcuEAn0NOBdAcEQ+P5YP0mamaC1x3ks6ppqjZ61Yzjt b0CQX9bqT5xeEgzZKmMMQWQBwPHd X-Google-Smtp-Source: ADUXVKLbyP9iNQrjJr8sC4A7ICojAwet0UtlLGodDyoZ/aI/Nsak4Jzrn+zPTZMQEYGyEo0JI5mLiA== X-Received: by 2002:a1c:338c:: with SMTP id z134-v6mr1107132wmz.93.1528965306203; Thu, 14 Jun 2018 01:35:06 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id h7-v6sm3015524wmb.48.2018.06.14.01.35.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:35:04 -0700 (PDT) Date: Thu, 14 Jun 2018 10:34:49 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180614083047.10812-2-adrien.mazarguil@6wind.com> References: <20180525161814.13873-1-adrien.mazarguil@6wind.com> <20180614083047.10812-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180614083047.10812-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 1/7] net/mlx5: rename confusing object in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" There are several attribute objects in this function: - IB device attributes (struct ibv_device_attr_ex device_attr). - Direct Verbs attributes (struct mlx5dv_context attrs_out). - Port attributes (struct ibv_port_attr). - IB device attributes again (struct ibv_device_attr_ex device_attr_ex). "attrs_out" is both odd and initialized using a nonstandard syntax. Rename it "dv_attr" for consistency. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Fixed ctx -> attr_ctx in mlx5_pci_probe(). --- drivers/net/mlx5/mlx5.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 3e0a1b186..3bdcb3970 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,6 +654,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; struct ibv_context *attr_ctx = NULL; struct ibv_device_attr_ex device_attr; @@ -670,7 +671,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; int i; - struct mlx5dv_context attrs_out = {0}; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif @@ -736,21 +736,21 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ibv_dev = list[i]; DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif /* * Multi-packet send is supported by ConnectX-4 Lx PF as well * as all ConnectX-5 devices. */ #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &attrs_out); - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { + mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); mps = MLX5_MPW_ENHANCED; } else { @@ -762,14 +762,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mps = MLX5_MPW_DISABLED; } #ifdef HAVE_IBV_MLX5_MOD_SWP - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_SWP) - swp = attrs_out.sw_parsing_caps.sw_parsing_offloads; + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) + swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = - attrs_out.striding_rq_caps; + dv_attr.striding_rq_caps; DRV_LOG(DEBUG, "\tmin_single_stride_log_num_of_bytes: %d", mprq_caps.min_single_stride_log_num_of_bytes); @@ -794,15 +794,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } #endif if (RTE_CACHE_LINE_SIZE == 128 && - !(attrs_out.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) + !(dv_attr.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) cqe_comp = 0; else cqe_comp = 1; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { - tunnel_en = ((attrs_out.tunnel_offloads_caps & + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { + tunnel_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_VXLAN) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GRE)); } DRV_LOG(DEBUG, "tunnel offloading is %ssupported", @@ -812,9 +812,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT - mpls_en = ((attrs_out.tunnel_offloads_caps & + mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP)); DRV_LOG(DEBUG, "MPLS over GRE/UDP tunnel offloading is %ssupported", mpls_en ? "" : "not "); From patchwork Thu Jun 14 08:34:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 41080 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DD92C1E436; Thu, 14 Jun 2018 10:35:10 +0200 (CEST) Received: from mail-wr0-f170.google.com (mail-wr0-f170.google.com [209.85.128.170]) by dpdk.org (Postfix) with ESMTP id 740601E409 for ; Thu, 14 Jun 2018 10:35:08 +0200 (CEST) Received: by mail-wr0-f170.google.com with SMTP id h10-v6so5419862wrq.8 for ; Thu, 14 Jun 2018 01:35:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=XMoJwcopedEjdu0b76//vItV9waRTNBNH3pEozTruVA=; b=zGeJ98OHWJoTh19FBsaYghpRfQa2ucQJHkmYPlA8NrnsvWOUo9e6jo4s8xnPNANm0F m0G36uB+qdeQoJxaiNmny+Fl2ohh6HfF/N2zFmSuqAynLJ3f2MLSuSAEjBwc4HFroX8J P9yowEe0Ueid2gEqivhbbHeSMbxbas1FJ25M54PlFRrMqXEcKCLDiTV9j7zr+GV+7B5N T3M6bpolvp0AKTiT0p4OwTTN8F26IIO/tH5ftom15xg1EuiUyJdyBn/0vW//s3EVMy6I kjdZKxYi4/8fS3Ym7XIVAycqxoK5WJH87y8gjWnTCNtRb8168YpuDnxcnwlx8YfM5//a GrYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=XMoJwcopedEjdu0b76//vItV9waRTNBNH3pEozTruVA=; b=oRGwmnfpVQHIrb+9gpjn5XSQ5YdKmKOOl2Iq3uv0IF012WfWCSJ3W0GXTzs17FwAm2 JV81A2Bea2e5DiaIDzKrnUipXWOHQ9A9hKZEd1qq4j5zHxzR6/ZNYQIyVAFvq4AryNq1 MY0YNoCtNmnRiylT8ZwYedGRRhlUD6U+43fJdTESxaSxSVQbc6sJkhNyzQBrOosRYAOw V3sgrC/dzm2w4gpiG2Ipt2CBBBjeEtGtBR8PQLOO67r+4FQ+DQ5dpGUQlwwekawrVQX9 OiZ7fKFM4X4aAQ2z/wgM3aupBbf/i2Sx/HhNVkiANBB5MFWKAl1xLJZaFCOwhhG5mRXq eMmQ== X-Gm-Message-State: APt69E1KDAZcINlG/ktbQa+jVOqAqsxkdSlpLV7idjA4PCB1NMpr4F+v 6E+jJZlSxQn1ySjDDwvTzWLRZg== X-Google-Smtp-Source: ADUXVKKmY/vXXbhvOJ4qGwuz1f5+7usTzkQZS4d4BOmQQBUHzG8RZs9CdQcG6keVl8Qf0inW92SGtA== X-Received: by 2002:adf:ef4f:: with SMTP id c15-v6mr1200115wrp.182.1528965308173; Thu, 14 Jun 2018 01:35:08 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id n56-v6sm10705890wrn.72.2018.06.14.01.35.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:35:07 -0700 (PDT) Date: Thu, 14 Jun 2018 10:34:52 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180614083047.10812-3-adrien.mazarguil@6wind.com> References: <20180525161814.13873-1-adrien.mazarguil@6wind.com> <20180614083047.10812-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180614083047.10812-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 2/7] net/mlx5: remove redundant objects in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch gets rid of redundant calls to open the device and query its attributes in order to simplify the code. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Minor indent fix on existing code. --- drivers/net/mlx5/mlx5.c | 64 +++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 3bdcb3970..1a5391e63 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,10 +654,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct ibv_context *ctx = NULL; + struct ibv_device_attr_ex attr; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; - struct ibv_context *attr_ctx = NULL; - struct ibv_device_attr_ex device_attr; unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; @@ -714,12 +714,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || (pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - attr_ctx = mlx5_glue->open_device(list[i]); + ctx = mlx5_glue->open_device(list[i]); rte_errno = errno; err = rte_errno; break; } - if (attr_ctx == NULL) { + if (ctx == NULL) { switch (err) { case 0: DRV_LOG(ERR, @@ -748,7 +748,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + mlx5_glue->dv_query_device(ctx, &dv_attr); if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); @@ -822,23 +822,20 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif - err = mlx5_glue->query_device_ex(attr_ctx, NULL, &device_attr); + err = mlx5_glue->query_device_ex(ctx, NULL, &attr); if (err) { DEBUG("ibv_query_device_ex() failed"); goto error; } - DRV_LOG(INFO, "%u port(s) detected", - device_attr.orig_attr.phys_port_cnt); - for (i = 0; i < device_attr.orig_attr.phys_port_cnt; i++) { + DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); + for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { char name[RTE_ETH_NAME_MAX_LEN]; int len; uint32_t port = i + 1; /* ports are indexed from one */ - struct ibv_context *ctx = NULL; struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct priv *priv = NULL; struct rte_eth_dev *eth_dev = NULL; - struct ibv_device_attr_ex device_attr_ex; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -865,7 +862,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, len = snprintf(name, sizeof(name), PCI_PRI_FMT, pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - if (device_attr.orig_attr.phys_port_cnt > 1) + if (attr.orig_attr.phys_port_cnt > 1) snprintf(name + len, sizeof(name), " port %u", i); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); @@ -907,7 +904,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, continue; } DRV_LOG(DEBUG, "using port %u", port); - ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) + ctx = mlx5_glue->open_device(ibv_dev); if (ctx == NULL) { err = ENODEV; goto port_error; @@ -949,7 +947,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); - priv->device_attr = device_attr; + priv->device_attr = attr; priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; @@ -960,17 +958,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); goto port_error; } - err = mlx5_glue->query_device_ex(ctx, NULL, &device_attr_ex); - if (err) { - DRV_LOG(ERR, "ibv_query_device_ex() failed"); - goto port_error; - } - config.hw_csum = !!(device_attr_ex.device_cap_flags_ex & + config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); DRV_LOG(DEBUG, "checksum offloading is %ssupported", (config.hw_csum ? "" : "not ")); #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!(device_attr.max_counter_sets); + config.flow_counter_en = !!attr.max_counter_sets; mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", @@ -978,7 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, cs_desc.attributes); #endif config.ind_table_max_size = - device_attr_ex.rss_caps.max_rwq_indirection_table_size; + attr.rss_caps.max_rwq_indirection_table_size; /* Remove this check once DPDK supports larger/variable * indirection tables. */ if (config.ind_table_max_size > @@ -986,29 +979,28 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", config.ind_table_max_size); - config.hw_vlan_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_vlan_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); DRV_LOG(DEBUG, "VLAN stripping is %ssupported", (config.hw_vlan_strip ? "" : "not ")); - config.hw_fcs_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_fcs_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_SCATTER_FCS); DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", (config.hw_fcs_strip ? "" : "not ")); #ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!device_attr_ex.rx_pad_end_addr_align; + config.hw_padding = !!attr.rx_pad_end_addr_align; #endif DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", (config.hw_padding ? "" : "not ")); config.vf = vf; - config.tso = ((device_attr_ex.tso_caps.max_tso > 0) && - (device_attr_ex.tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); + config.tso = (attr.tso_caps.max_tso > 0 && + (attr.tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); if (config.tso) - config.tso_max_payload_sz = - device_attr_ex.tso_caps.max_tso; + config.tso_max_payload_sz = attr.tso_caps.max_tso; if (config.mps && !mps) { DRV_LOG(ERR, "multi-packet send not supported on this device" @@ -1168,14 +1160,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); + /* + * Each eth_dev instance is assigned its own Verbs context, + * since this one is consumed, let the next iteration open + * another. + */ + ctx = NULL; continue; port_error: if (priv) rte_free(priv); if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); - if (ctx) - claim_zero(mlx5_glue->close_device(ctx)); if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) rte_eth_dev_release_port(eth_dev); break; @@ -1187,8 +1183,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * way to enumerate the registered ethdevs to free the previous ones. */ error: - if (attr_ctx) - claim_zero(mlx5_glue->close_device(attr_ctx)); + if (ctx) + claim_zero(mlx5_glue->close_device(ctx)); if (list) mlx5_glue->free_device_list(list); if (err) { From patchwork Thu Jun 14 08:34:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 41081 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BBA781E487; Thu, 14 Jun 2018 10:35:12 +0200 (CEST) Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by dpdk.org (Postfix) with ESMTP id E68AD1E439 for ; Thu, 14 Jun 2018 10:35:10 +0200 (CEST) Received: by mail-wm0-f66.google.com with SMTP id r15-v6so9068142wmc.1 for ; Thu, 14 Jun 2018 01:35:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=l1TkjrvUe2fOguoLut4rhGe5WVJ2uLHdovj43ipBVF4=; b=yPpzIxlyDRSbhd+PD0ifKvwEixu48Kvp9/rRRCScP20Jfj5zYuYadLNyNrqbltgFqR +EYd2e8zA0ExgSW7ADpePWjONuxc24Pxr2Idiveeim5OYlGuEaAxmC5H8sVIM5ZWJ9xk d1MrarzGOrqC3jGDKYb4FHBKQRiMnSjj5d52199Y+1OutYNS7/m7pIXcNIlZAZE7YzPQ a4YjWDWee6YaXeEWJyF2EBaIOS6Wfvu2B/wFuKTgyjzRgqF0bhXGWtRG9sUoMX0QnEU6 OKnRULc2Voxm0VNuOi5MolfL6N7Ntwd4vcKj5ZVvdiZlIokPsxs6+oGFJiZzlmOf/J1a tCdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=l1TkjrvUe2fOguoLut4rhGe5WVJ2uLHdovj43ipBVF4=; b=mvtpf2v5IoHBUHMoFCLzy1vXNmW66Bi0mL2cZYeecaellnrs4LPRdDrDjNNZGXG5he Ow1l1n5SvjA6EG9PKjoDTs6r8GCdDeFyQaCFUpdBM44pWmjtaITrXDLmS1ngF+zP3q44 PnOlUpeIK5QiBT9TJNu8tAaUsnizo6eO8hL3cp7rzc2ETniuFyTAxgc+NOTBBNhEkHOr ZPI3wnmxAvexwjlinhe09+bXeupMhDpveOmGjzQUD73QdHL9b+g279XUm0p9jEWVlBZC WLVtQI8/9dy5LmZeUV0zGZysoUrweUEqQEWASy1ispT35cfQcmhOjO1ABZf8pYglwCuP dddQ== X-Gm-Message-State: APt69E1c8OnQEGGYkpRm9ZbzWHe8/ZY0zNwcgl5A4eVgQTn1ey2iT+17 +YCWdPQ570vZvnCzaW9wPnbl4A== X-Google-Smtp-Source: ADUXVKLWBcz+fa/Ytm6/UcK8jS8Ys6odfmMm7pC2yte6DnXowrwNjhNMsJe/OXuN1tczu7MggGIzmQ== X-Received: by 2002:a1c:9e95:: with SMTP id h143-v6mr1102279wme.68.1528965310248; Thu, 14 Jun 2018 01:35:10 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id l84-v6sm7462578wmi.3.2018.06.14.01.35.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:35:09 -0700 (PDT) Date: Thu, 14 Jun 2018 10:34:54 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180614083047.10812-4-adrien.mazarguil@6wind.com> References: <20180525161814.13873-1-adrien.mazarguil@6wind.com> <20180614083047.10812-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180614083047.10812-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 3/7] net/mlx5: split PCI from generic probing code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" All the generic probing code needs is an IB device. While this device is currently supplied by a PCI lookup, other methods will be added soon. This patch divides the original function, which has become huge over time, as follows: 1. PCI-specific (mlx5_pci_probe()). 2. All ports of a Verbs device (mlx5_dev_spawn()). 3. A given port of a Verbs device (mlx5_dev_spawn_one()). (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Fixed device naming. A port suffix is now appended only if several IB ports happen to be detected. - Added separate message to distinguish missing kernel drivers from other initialization errors, as it was confusing. --- drivers/net/mlx5/mlx5.c | 340 ++++++++++++++++++++++++++----------------- 1 file changed, 209 insertions(+), 131 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 1a5391e63..01dcf25b9 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -635,30 +635,34 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) } /** - * DPDK callback to register a PCI device. - * - * This function creates an Ethernet device for each port of a given - * PCI device. + * Spawn an Ethernet device from Verbs information. * - * @param[in] pci_drv - * PCI driver structure (mlx5_driver). - * @param[in] pci_dev - * PCI device information. + * @param dpdk_dev + * Backing DPDK device. + * @param ibv_dev + * Verbs device. + * @param vf + * If nonzero, enable VF-specific features. + * @param[in] attr + * Verbs device attributes. + * @param port + * Verbs port to use (indexed from 1). * * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. + * A valid Ethernet device object on success, NULL otherwise and rte_errno + * is set. */ -static int -mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +static struct rte_eth_dev * +mlx5_dev_spawn_one(struct rte_device *dpdk_dev, + struct ibv_device *ibv_dev, + int vf, + const struct ibv_device_attr_ex *attr, + unsigned int port) { - struct ibv_device **list = NULL; - struct ibv_device *ibv_dev; - struct ibv_context *ctx = NULL; - struct ibv_device_attr_ex attr; + struct ibv_context *ctx; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct rte_eth_dev *eth_dev = NULL; int err = 0; - unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; unsigned int tunnel_en = 0; @@ -670,71 +674,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_max_stride_size_n = 0; unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; - int i; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); - assert(pci_drv == &mlx5_driver); - list = mlx5_glue->get_device_list(&i); - if (list == NULL) { - assert(errno); - err = errno; - if (errno == ENOSYS) - DRV_LOG(ERR, - "cannot list devices, is ib_uverbs loaded?"); - goto error; - } - assert(i >= 0); - /* - * For each listed device, check related sysfs entry against - * the provided PCI ID. - */ - while (i != 0) { - struct rte_pci_addr pci_addr; - - --i; - DRV_LOG(DEBUG, "checking device \"%s\"", list[i]->name); - if (mlx5_ibv_device_to_pci_addr(list[i], &pci_addr)) - continue; - if ((pci_dev->addr.domain != pci_addr.domain) || - (pci_dev->addr.bus != pci_addr.bus) || - (pci_dev->addr.devid != pci_addr.devid) || - (pci_dev->addr.function != pci_addr.function)) - continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", - list[i]->name); - vf = ((pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - ctx = mlx5_glue->open_device(list[i]); - rte_errno = errno; - err = rte_errno; - break; - } - if (ctx == NULL) { - switch (err) { - case 0: - DRV_LOG(ERR, - "cannot access device, is mlx5_ib loaded?"); - err = ENODEV; - break; - case EINVAL: - DRV_LOG(ERR, - "cannot use device, are drivers up to date?"); - break; - } - goto error; + errno = 0; + ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) { + rte_errno = errno ? errno : ENODEV; + return NULL; } - ibv_dev = list[i]; - DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif @@ -822,20 +773,11 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif - err = mlx5_glue->query_device_ex(ctx, NULL, &attr); - if (err) { - DEBUG("ibv_query_device_ex() failed"); - goto error; - } - DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); - for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { + { char name[RTE_ETH_NAME_MAX_LEN]; - int len; - uint32_t port = i + 1; /* ports are indexed from one */ struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct priv *priv = NULL; - struct rte_eth_dev *eth_dev = NULL; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -859,11 +801,11 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - len = snprintf(name, sizeof(name), PCI_PRI_FMT, - pci_dev->addr.domain, pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); - if (attr.orig_attr.phys_port_cnt > 1) - snprintf(name + len, sizeof(name), " port %u", i); + if (attr->orig_attr.phys_port_cnt > 1) + snprintf(name, sizeof(name), "%s port %u", + dpdk_dev->name, port); + else + snprintf(name, sizeof(name), "%s", dpdk_dev->name); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -872,7 +814,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; goto error; } - eth_dev->device = &pci_dev->device; + eth_dev->device = dpdk_dev; eth_dev->dev_ops = &mlx5_dev_sec_ops; err = mlx5_uar_init_secondary(eth_dev); if (err) { @@ -900,16 +842,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mlx5_select_rx_function(eth_dev); eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); - rte_eth_dev_probing_finish(eth_dev); - continue; + mlx5_glue->close_device(ctx); + return eth_dev; } DRV_LOG(DEBUG, "using port %u", port); - if (!ctx) - ctx = mlx5_glue->open_device(ibv_dev); - if (ctx == NULL) { - err = ENODEV; - goto port_error; - } /* Check port status. */ err = mlx5_glue->query_port(ctx, port, &port_attr); if (err) { @@ -947,23 +883,23 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); - priv->device_attr = attr; + priv->device_attr = *attr; priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; - err = mlx5_args(&config, pci_dev->device.devargs); + err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", strerror(rte_errno)); goto port_error; } - config.hw_csum = !!(attr.device_cap_flags_ex & + config.hw_csum = !!(attr->device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); DRV_LOG(DEBUG, "checksum offloading is %ssupported", (config.hw_csum ? "" : "not ")); #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!attr.max_counter_sets; + config.flow_counter_en = !!attr->max_counter_sets; mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", @@ -971,7 +907,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, cs_desc.attributes); #endif config.ind_table_max_size = - attr.rss_caps.max_rwq_indirection_table_size; + attr->rss_caps.max_rwq_indirection_table_size; /* Remove this check once DPDK supports larger/variable * indirection tables. */ if (config.ind_table_max_size > @@ -979,28 +915,28 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", config.ind_table_max_size); - config.hw_vlan_strip = !!(attr.raw_packet_caps & + config.hw_vlan_strip = !!(attr->raw_packet_caps & IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); DRV_LOG(DEBUG, "VLAN stripping is %ssupported", (config.hw_vlan_strip ? "" : "not ")); - config.hw_fcs_strip = !!(attr.raw_packet_caps & + config.hw_fcs_strip = !!(attr->raw_packet_caps & IBV_RAW_PACKET_CAP_SCATTER_FCS); DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", (config.hw_fcs_strip ? "" : "not ")); #ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!attr.rx_pad_end_addr_align; + config.hw_padding = !!attr->rx_pad_end_addr_align; #endif DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", (config.hw_padding ? "" : "not ")); config.vf = vf; - config.tso = (attr.tso_caps.max_tso > 0 && - (attr.tso_caps.supported_qpts & + config.tso = (attr->tso_caps.max_tso > 0 && + (attr->tso_caps.supported_qpts & (1 << IBV_QPT_RAW_PACKET))); if (config.tso) - config.tso_max_payload_sz = attr.tso_caps.max_tso; + config.tso_max_payload_sz = attr->tso_caps.max_tso; if (config.mps && !mps) { DRV_LOG(ERR, "multi-packet send not supported on this device" @@ -1041,8 +977,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; - eth_dev->device = &pci_dev->device; - rte_eth_copy_pci_info(eth_dev, pci_dev); + eth_dev->device = dpdk_dev; eth_dev->device->driver = &mlx5_driver.driver; err = mlx5_uar_init_primary(eth_dev); if (err) { @@ -1160,13 +1095,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - /* - * Each eth_dev instance is assigned its own Verbs context, - * since this one is consumed, let the next iteration open - * another. - */ - ctx = NULL; - continue; + return eth_dev; port_error: if (priv) rte_free(priv); @@ -1174,24 +1103,173 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, claim_zero(mlx5_glue->dealloc_pd(pd)); if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) rte_eth_dev_release_port(eth_dev); - break; } - /* - * XXX if something went wrong in the loop above, there is a resource - * leak (ctx, pd, priv, dpdk ethdev) but we can do nothing about it as - * long as the dpdk does not provide a way to deallocate a ethdev and a - * way to enumerate the registered ethdevs to free the previous ones. - */ error: if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (list) - mlx5_glue->free_device_list(list); - if (err) { - rte_errno = err; + assert(err > 0); + rte_errno = err; + return NULL; +} + +/** + * Spawn Ethernet devices from Verbs information, one per detected port. + * + * @param dpdk_dev + * Backing DPDK device. + * @param ibv_dev + * Verbs device. + * @param vf + * If nonzero, enable VF-specific features. + * + * @return + * A NULL-terminated list of Ethernet device objects on success, NULL + * otherwise and rte_errno is set. Caller is expected to release list + * memory through free(). + */ +static struct rte_eth_dev ** +mlx5_dev_spawn(struct rte_device *dpdk_dev, + struct ibv_device *ibv_dev, + int vf) +{ + struct rte_eth_dev **eth_list = NULL; + struct ibv_context *ctx; + struct ibv_device_attr_ex attr; + unsigned int i; + int ret; + + errno = 0; + ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) { + rte_errno = errno ? errno : ENODEV; + if (rte_errno == ENODEV) + DRV_LOG(ERR, + "cannot access device, is mlx5_ib loaded?"); + else + DRV_LOG(ERR, + "cannot use device, are drivers up to date?"); + return NULL; + } + ret = mlx5_glue->query_device_ex(ctx, NULL, &attr); + mlx5_glue->close_device(ctx); + if (ret) { + rte_errno = ret; + DRV_LOG(ERR, "unable to query device information: %s", + strerror(rte_errno)); + return NULL; + } + DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); + eth_list = malloc(sizeof(*eth_list) * + (attr.orig_attr.phys_port_cnt + 1)); + if (!eth_list) { + rte_errno = errno; + return NULL; + } + for (i = 0; i < attr.orig_attr.phys_port_cnt; ++i) { + eth_list[i] = mlx5_dev_spawn_one(dpdk_dev, ibv_dev, vf, + &attr, i + 1); + if (eth_list[i]) + continue; + /* Save rte_errno and roll back in case of failure. */ + ret = rte_errno; + while (i--) { + mlx5_dev_close(eth_list[i]); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_free(eth_list[i]->data->dev_private); + claim_zero(rte_eth_dev_release_port(eth_list[i])); + } + free(eth_list); + rte_errno = ret; + return NULL; + } + eth_list[i] = NULL; + return eth_list; +} + +/** + * DPDK callback to register a PCI device. + * + * This function creates an Ethernet device for each port of a given + * PCI device. + * + * @param[in] pci_drv + * PCI driver structure (mlx5_driver). + * @param[in] pci_dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct ibv_device **ibv_list; + struct rte_eth_dev **eth_list = NULL; + int vf; + int ret; + + assert(pci_drv == &mlx5_driver); + switch (pci_dev->id.device_id) { + case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF: + vf = 1; + break; + default: + vf = 0; + } + errno = 0; + ibv_list = mlx5_glue->get_device_list(&ret); + if (!ibv_list) { + rte_errno = errno ? errno : ENOSYS; + DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } - return 0; + while (ret-- > 0) { + struct rte_pci_addr pci_addr; + + DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name); + if (mlx5_ibv_device_to_pci_addr(ibv_list[ret], &pci_addr)) + continue; + if (pci_dev->addr.domain != pci_addr.domain || + pci_dev->addr.bus != pci_addr.bus || + pci_dev->addr.devid != pci_addr.devid || + pci_dev->addr.function != pci_addr.function) + continue; + DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + ibv_list[ret]->name); + break; + } + if (ret >= 0) + eth_list = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + mlx5_glue->free_device_list(ibv_list); + if (!ret) { + DRV_LOG(WARNING, + "no Verbs device matches PCI device " PCI_PRI_FMT "," + " are kernel drivers loaded?", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function); + rte_errno = ENOENT; + ret = -rte_errno; + } else if (!eth_list || !*eth_list) { + DRV_LOG(ERR, + "probe of PCI device " PCI_PRI_FMT " aborted after" + " encountering an error: %s", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function, + strerror(rte_errno)); + ret = -rte_errno; + } else { + for (ret = 0; eth_list[ret]; ++ret) { + rte_eth_copy_pci_info(eth_list[ret], pci_dev); + rte_eth_dev_probing_finish(eth_list[ret]); + } + ret = 0; + } + free(eth_list); + return ret; } static const struct rte_pci_id mlx5_pci_id_map[] = { From patchwork Thu Jun 14 08:34:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 41082 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3BEC01E4A3; Thu, 14 Jun 2018 10:35:15 +0200 (CEST) Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by dpdk.org (Postfix) with ESMTP id 14BD91E495 for ; Thu, 14 Jun 2018 10:35:12 +0200 (CEST) Received: by mail-wm0-f45.google.com with SMTP id n5-v6so10251626wmc.5 for ; Thu, 14 Jun 2018 01:35:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=jhml4OMAm2owFgVdqLqUHV2IF6aO4g0eaY6gYbGUtC8=; b=MCS3cdfF1VpGi9m7rwn7WdhQWnkDCraZ9ruP95EpffZcsiN0NqZYsaZzEiOjKeixyV SSNT6GeMm8MN1uzCsHn6oNsWTFM3YODjJgPqDxWg/W054Itac/YFTtbrNF/seFFJFhWa YXD0fgP5cJNw6XfKSVllzGQTj+9V1+j2IRp1CepC161oVzYa/luzRcRR0TVEn96NWAFh mt3CvvZaVu5hNmrNo5zx28WnuK+H2riQ+EJrY728BMCkD3ZSVEzpjsXbw86N0K+RmLty DWQjyisM35Qyx7QoD8cD8tsybb7s86ATiBkQPgpT6361/wWj3aMyKk3tECW40pkIJ2U0 DrKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=jhml4OMAm2owFgVdqLqUHV2IF6aO4g0eaY6gYbGUtC8=; b=t6ah1W0S9Sm0dwdlpxsS5PbfmHPzxK90yo7a59iNiAPgRBwusjAtTSDPTgspjyuG36 fNJTS7w6Xb+2nOvd9dOeu2YRyzW1VHdoCiky3phaKVG9bYUFB6q3Cl4QLbH40M+oepqg fpHZ/3ACyJQfmKmQHf8A7egrsGJtt2QTSJ/RvzDBSjkfdoyjpQFJCWdv5pTpNDFj82vc RkXu59qwOX3/iez6kx0RqsUh8RWcKPcq+4g8t/U2XWST6nuBtmCxTlYhAU2kYPV7UvnU TrRNRmIzgv+LOlMp+VuxQpnPMM9yRChthwjmFELPbikldEZllbuY+ozdmoXbVIGTKhL9 smag== X-Gm-Message-State: APt69E3KXQ969wvXvpeXSfy+ltjhiiJjW2dMXuqaA+7AaIBvsTYgWuae mPMGoh5gPIg4OLJreAIPs/oLxQ== X-Google-Smtp-Source: ADUXVKJblu9ASwsX7ZvKe+HGjuIS8fcNz0ic/eJTh0JK1fub3iPseASe+87qVolZ/J6aKr2mtqDuFw== X-Received: by 2002:a1c:852:: with SMTP id 79-v6mr1020822wmi.115.1528965312257; Thu, 14 Jun 2018 01:35:12 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id g129-v6sm4797623wmf.5.2018.06.14.01.35.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:35:11 -0700 (PDT) Date: Thu, 14 Jun 2018 10:34:56 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180614083047.10812-5-adrien.mazarguil@6wind.com> References: <20180525161814.13873-1-adrien.mazarguil@6wind.com> <20180614083047.10812-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180614083047.10812-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 4/7] net/mlx5: re-indent generic probing function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since commit "net/mlx5: split PCI from generic probing code" extracted the inner loop to a separate function, mlx5_dev_spawn_one() is left with an unnecessary indent level. This patch eliminates a block, moves its local variables to function scope, and re-indents its contents. No functional impact. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming(Steven) Li --- drivers/net/mlx5/mlx5.c | 615 +++++++++++++++++++++---------------------- 1 file changed, 299 insertions(+), 316 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 01dcf25b9..c9815d721 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -660,8 +660,27 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, unsigned int port) { struct ibv_context *ctx; + struct ibv_port_attr port_attr; + struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct mlx5_dev_config config = { + .vf = !!vf, + .tx_vec_en = 1, + .rx_vec_en = 1, + .mpw_hdr_dseg = 0, + .txq_inline = MLX5_ARG_UNSET, + .txqs_inline = MLX5_ARG_UNSET, + .inline_max_packet_sz = MLX5_ARG_UNSET, + .vf_nl_en = 1, + .mprq = { + .enabled = 0, + .stride_num_n = MLX5_MPRQ_STRIDE_NUM_N, + .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, + .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, + }, + }; struct rte_eth_dev *eth_dev = NULL; + struct priv *priv = NULL; int err = 0; unsigned int mps; unsigned int cqe_comp; @@ -677,6 +696,8 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif + struct ether_addr mac; + char name[RTE_ETH_NAME_MAX_LEN]; /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); @@ -712,11 +733,13 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, DRV_LOG(DEBUG, "MPW isn't supported"); mps = MLX5_MPW_DISABLED; } + config.mps = mps; #ifdef HAVE_IBV_MLX5_MOD_SWP if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif + config.swp = !!swp; #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = @@ -742,6 +765,8 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, mprq_caps.min_single_wqe_log_num_of_strides; mprq_max_stride_num_n = mprq_caps.max_single_wqe_log_num_of_strides; + config.mprq.stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); } #endif if (RTE_CACHE_LINE_SIZE == 128 && @@ -749,6 +774,7 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, cqe_comp = 0; else cqe_comp = 1; + config.cqe_comp = cqe_comp; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { tunnel_en = ((dv_attr.tunnel_offloads_caps & @@ -762,6 +788,7 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif + config.tunnel_en = tunnel_en; #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && @@ -773,338 +800,294 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif - { - char name[RTE_ETH_NAME_MAX_LEN]; - struct ibv_port_attr port_attr; - struct ibv_pd *pd = NULL; - struct priv *priv = NULL; - struct ether_addr mac; - struct mlx5_dev_config config = { - .cqe_comp = cqe_comp, - .mps = mps, - .tunnel_en = tunnel_en, - .mpls_en = mpls_en, - .tx_vec_en = 1, - .rx_vec_en = 1, - .mpw_hdr_dseg = 0, - .txq_inline = MLX5_ARG_UNSET, - .txqs_inline = MLX5_ARG_UNSET, - .inline_max_packet_sz = MLX5_ARG_UNSET, - .vf_nl_en = 1, - .swp = !!swp, - .mprq = { - .enabled = 0, /* Disabled by default. */ - .stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n), - .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, - .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, - }, - }; - - if (attr->orig_attr.phys_port_cnt > 1) - snprintf(name, sizeof(name), "%s port %u", - dpdk_dev->name, port); - else - snprintf(name, sizeof(name), "%s", dpdk_dev->name); - if (rte_eal_process_type() == RTE_PROC_SECONDARY) { - eth_dev = rte_eth_dev_attach_secondary(name); - if (eth_dev == NULL) { - DRV_LOG(ERR, "can not attach rte ethdev"); - rte_errno = ENOMEM; - err = rte_errno; - goto error; - } - eth_dev->device = dpdk_dev; - eth_dev->dev_ops = &mlx5_dev_sec_ops; - err = mlx5_uar_init_secondary(eth_dev); - if (err) { - err = rte_errno; - goto error; - } - /* Receive command fd from primary process */ - err = mlx5_socket_connect(eth_dev); - if (err < 0) { - err = rte_errno; - goto error; - } - /* Remap UAR for Tx queues. */ - err = mlx5_tx_uar_remap(eth_dev, err); - if (err) { - err = rte_errno; - goto error; - } - /* - * Ethdev pointer is still required as input since - * the primary device is not accessible from the - * secondary process. - */ - eth_dev->rx_pkt_burst = - mlx5_select_rx_function(eth_dev); - eth_dev->tx_pkt_burst = - mlx5_select_tx_function(eth_dev); - mlx5_glue->close_device(ctx); - return eth_dev; - } - DRV_LOG(DEBUG, "using port %u", port); - /* Check port status. */ - err = mlx5_glue->query_port(ctx, port, &port_attr); - if (err) { - DRV_LOG(ERR, "port query failed: %s", strerror(err)); - goto port_error; - } - if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { - DRV_LOG(ERR, - "port %d is not configured in Ethernet mode", - port); - err = EINVAL; - goto port_error; - } - if (port_attr.state != IBV_PORT_ACTIVE) - DRV_LOG(DEBUG, "port %d is not active: \"%s\" (%d)", - port, - mlx5_glue->port_state_str(port_attr.state), - port_attr.state); - /* Allocate protection domain. */ - pd = mlx5_glue->alloc_pd(ctx); - if (pd == NULL) { - DRV_LOG(ERR, "PD allocation failure"); - err = ENOMEM; - goto port_error; - } - /* from rte_ethdev.c */ - priv = rte_zmalloc("ethdev private structure", - sizeof(*priv), - RTE_CACHE_LINE_SIZE); - if (priv == NULL) { - DRV_LOG(ERR, "priv allocation failure"); - err = ENOMEM; - goto port_error; - } - priv->ctx = ctx; - strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, - sizeof(priv->ibdev_path)); - priv->device_attr = *attr; - priv->port = port; - priv->pd = pd; - priv->mtu = ETHER_MTU; - err = mlx5_args(&config, dpdk_dev->devargs); - if (err) { - err = rte_errno; - DRV_LOG(ERR, "failed to process device arguments: %s", - strerror(rte_errno)); - goto port_error; - } - config.hw_csum = !!(attr->device_cap_flags_ex & - IBV_DEVICE_RAW_IP_CSUM); - DRV_LOG(DEBUG, "checksum offloading is %ssupported", - (config.hw_csum ? "" : "not ")); -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!attr->max_counter_sets; - mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); - DRV_LOG(DEBUG, - "counter type = %d, num of cs = %ld, attributes = %d", - cs_desc.counter_type, cs_desc.num_of_cs, - cs_desc.attributes); -#endif - config.ind_table_max_size = - attr->rss_caps.max_rwq_indirection_table_size; - /* Remove this check once DPDK supports larger/variable - * indirection tables. */ - if (config.ind_table_max_size > - (unsigned int)ETH_RSS_RETA_SIZE_512) - config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; - DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", - config.ind_table_max_size); - config.hw_vlan_strip = !!(attr->raw_packet_caps & - IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); - DRV_LOG(DEBUG, "VLAN stripping is %ssupported", - (config.hw_vlan_strip ? "" : "not ")); - - config.hw_fcs_strip = !!(attr->raw_packet_caps & - IBV_RAW_PACKET_CAP_SCATTER_FCS); - DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", - (config.hw_fcs_strip ? "" : "not ")); - -#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!attr->rx_pad_end_addr_align; -#endif - DRV_LOG(DEBUG, - "hardware Rx end alignment padding is %ssupported", - (config.hw_padding ? "" : "not ")); - config.vf = vf; - config.tso = (attr->tso_caps.max_tso > 0 && - (attr->tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); - if (config.tso) - config.tso_max_payload_sz = attr->tso_caps.max_tso; - if (config.mps && !mps) { - DRV_LOG(ERR, - "multi-packet send not supported on this device" - " (" MLX5_TXQ_MPW_EN ")"); - err = ENOTSUP; - goto port_error; - } - DRV_LOG(INFO, "%s MPS is %s", - config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", - config.mps != MLX5_MPW_DISABLED ? "enabled" : - "disabled"); - if (config.cqe_comp && !cqe_comp) { - DRV_LOG(WARNING, "Rx CQE compression isn't supported"); - config.cqe_comp = 0; - } - config.mprq.enabled = config.mprq.enabled && mprq; - if (config.mprq.enabled) { - if (config.mprq.stride_num_n > mprq_max_stride_num_n || - config.mprq.stride_num_n < mprq_min_stride_num_n) { - config.mprq.stride_num_n = - RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n); - DRV_LOG(WARNING, - "the number of strides" - " for Multi-Packet RQ is out of range," - " setting default value (%u)", - 1 << config.mprq.stride_num_n); - } - config.mprq.min_stride_size_n = mprq_min_stride_size_n; - config.mprq.max_stride_size_n = mprq_max_stride_size_n; - } - eth_dev = rte_eth_dev_allocate(name); + config.mpls_en = mpls_en; + if (attr->orig_attr.phys_port_cnt > 1) + snprintf(name, sizeof(name), "%s port %u", + dpdk_dev->name, port); + else + snprintf(name, sizeof(name), "%s", dpdk_dev->name); + if (rte_eal_process_type() == RTE_PROC_SECONDARY) { + eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { - DRV_LOG(ERR, "can not allocate rte ethdev"); - err = ENOMEM; - goto port_error; + DRV_LOG(ERR, "can not attach rte ethdev"); + rte_errno = ENOMEM; + err = rte_errno; + goto error; } - eth_dev->data->dev_private = priv; - priv->dev_data = eth_dev->data; - eth_dev->data->mac_addrs = priv->mac; eth_dev->device = dpdk_dev; - eth_dev->device->driver = &mlx5_driver.driver; - err = mlx5_uar_init_primary(eth_dev); + eth_dev->dev_ops = &mlx5_dev_sec_ops; + err = mlx5_uar_init_secondary(eth_dev); if (err) { err = rte_errno; - goto port_error; - } - /* Configure the first MAC address by default. */ - if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { - DRV_LOG(ERR, - "port %u cannot get MAC address, is mlx5_en" - " loaded? (errno: %s)", - eth_dev->data->port_id, strerror(rte_errno)); - err = ENODEV; - goto port_error; - } - DRV_LOG(INFO, - "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", - eth_dev->data->port_id, - mac.addr_bytes[0], mac.addr_bytes[1], - mac.addr_bytes[2], mac.addr_bytes[3], - mac.addr_bytes[4], mac.addr_bytes[5]); -#ifndef NDEBUG - { - char ifname[IF_NAMESIZE]; - - if (mlx5_get_ifname(eth_dev, &ifname) == 0) - DRV_LOG(DEBUG, "port %u ifname is \"%s\"", - eth_dev->data->port_id, ifname); - else - DRV_LOG(DEBUG, "port %u ifname is unknown", - eth_dev->data->port_id); + goto error; } -#endif - /* Get actual MTU if possible. */ - err = mlx5_get_mtu(eth_dev, &priv->mtu); - if (err) { + /* Receive command fd from primary process */ + err = mlx5_socket_connect(eth_dev); + if (err < 0) { err = rte_errno; - goto port_error; - } - DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, - priv->mtu); - /* - * Initialize burst functions to prevent crashes before link-up. - */ - eth_dev->rx_pkt_burst = removed_rx_burst; - eth_dev->tx_pkt_burst = removed_tx_burst; - eth_dev->dev_ops = &mlx5_dev_ops; - /* Register MAC address. */ - claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); - priv->nl_socket = -1; - priv->nl_sn = 0; - if (vf && config.vf_nl_en) { - priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); - if (priv->nl_socket < 0) - priv->nl_socket = -1; - mlx5_nl_mac_addr_sync(eth_dev); + goto error; } - TAILQ_INIT(&priv->flows); - TAILQ_INIT(&priv->ctrl_flows); - /* Hint libmlx5 to use PMD allocator for data plane resources */ - struct mlx5dv_ctx_allocators alctr = { - .alloc = &mlx5_alloc_verbs_buf, - .free = &mlx5_free_verbs_buf, - .data = priv, - }; - mlx5_glue->dv_set_context_attr(ctx, - MLX5DV_CTX_ATTR_BUF_ALLOCATORS, - (void *)((uintptr_t)&alctr)); - /* Bring Ethernet device up. */ - DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", - eth_dev->data->port_id); - mlx5_set_link_up(eth_dev); - /* - * Even though the interrupt handler is not installed yet, - * interrupts will still trigger on the asyn_fd from - * Verbs context returned by ibv_open_device(). - */ - mlx5_link_update(eth_dev, 0); - /* Store device configuration on private structure. */ - priv->config = config; - /* Create drop queue. */ - err = mlx5_flow_create_drop_queue(eth_dev); + /* Remap UAR for Tx queues. */ + err = mlx5_tx_uar_remap(eth_dev, err); if (err) { - DRV_LOG(ERR, "port %u drop queue allocation failed: %s", - eth_dev->data->port_id, strerror(rte_errno)); err = rte_errno; - goto port_error; - } - /* Supported Verbs flow priority number detection. */ - if (verb_priorities == 0) - verb_priorities = mlx5_get_max_verbs_prio(eth_dev); - if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { - DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", - eth_dev->data->port_id, verb_priorities); - err = ENOTSUP; - goto port_error; + goto error; } - priv->config.max_verbs_prio = verb_priorities; /* - * Once the device is added to the list of memory event - * callback, its global MR cache table cannot be expanded - * on the fly because of deadlock. If it overflows, lookup - * should be done by searching MR list linearly, which is slow. + * Ethdev pointer is still required as input since + * the primary device is not accessible from the + * secondary process. */ - err = mlx5_mr_btree_init(&priv->mr.cache, - MLX5_MR_BTREE_CACHE_N * 2, - eth_dev->device->numa_node); - if (err) { - err = rte_errno; - goto port_error; - } - /* Add device to memory callback list. */ - rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); - LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, - priv, mem_event_cb); - rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); - rte_eth_dev_probing_finish(eth_dev); + eth_dev->rx_pkt_burst = mlx5_select_rx_function(eth_dev); + eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); + mlx5_glue->close_device(ctx); return eth_dev; -port_error: - if (priv) - rte_free(priv); - if (pd) - claim_zero(mlx5_glue->dealloc_pd(pd)); - if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_eth_dev_release_port(eth_dev); } + DRV_LOG(DEBUG, "using port %u", port); + /* Check port status. */ + err = mlx5_glue->query_port(ctx, port, &port_attr); + if (err) { + DRV_LOG(ERR, "port query failed: %s", strerror(err)); + goto error; + } + if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { + DRV_LOG(ERR, "port %d is not configured in Ethernet mode", + port); + err = EINVAL; + goto error; + } + if (port_attr.state != IBV_PORT_ACTIVE) + DRV_LOG(DEBUG, "port %d is not active: \"%s\" (%d)", + port, mlx5_glue->port_state_str(port_attr.state), + port_attr.state); + /* Allocate protection domain. */ + pd = mlx5_glue->alloc_pd(ctx); + if (pd == NULL) { + DRV_LOG(ERR, "PD allocation failure"); + err = ENOMEM; + goto error; + } + priv = rte_zmalloc("ethdev private structure", + sizeof(*priv), + RTE_CACHE_LINE_SIZE); + if (priv == NULL) { + DRV_LOG(ERR, "priv allocation failure"); + err = ENOMEM; + goto error; + } + priv->ctx = ctx; + strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, + sizeof(priv->ibdev_path)); + priv->device_attr = *attr; + priv->port = port; + priv->pd = pd; + priv->mtu = ETHER_MTU; + err = mlx5_args(&config, dpdk_dev->devargs); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + goto error; + } + config.hw_csum = !!(attr->device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); + DRV_LOG(DEBUG, "checksum offloading is %ssupported", + (config.hw_csum ? "" : "not ")); +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + config.flow_counter_en = !!attr->max_counter_sets; + mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); + DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", + cs_desc.counter_type, cs_desc.num_of_cs, + cs_desc.attributes); +#endif + config.ind_table_max_size = + attr->rss_caps.max_rwq_indirection_table_size; + /* + * Remove this check once DPDK supports larger/variable + * indirection tables. + */ + if (config.ind_table_max_size > (unsigned int)ETH_RSS_RETA_SIZE_512) + config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; + DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", + config.ind_table_max_size); + config.hw_vlan_strip = !!(attr->raw_packet_caps & + IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); + DRV_LOG(DEBUG, "VLAN stripping is %ssupported", + (config.hw_vlan_strip ? "" : "not ")); + config.hw_fcs_strip = !!(attr->raw_packet_caps & + IBV_RAW_PACKET_CAP_SCATTER_FCS); + DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", + (config.hw_fcs_strip ? "" : "not ")); +#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING + config.hw_padding = !!attr->rx_pad_end_addr_align; +#endif + DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", + (config.hw_padding ? "" : "not ")); + config.tso = (attr->tso_caps.max_tso > 0 && + (attr->tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); + if (config.tso) + config.tso_max_payload_sz = attr->tso_caps.max_tso; + if (config.mps && !mps) { + DRV_LOG(ERR, + "multi-packet send not supported on this device" + " (" MLX5_TXQ_MPW_EN ")"); + err = ENOTSUP; + goto error; + } + DRV_LOG(INFO, "%sMPS is %s", + config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", + config.mps != MLX5_MPW_DISABLED ? "enabled" : "disabled"); + if (config.cqe_comp && !cqe_comp) { + DRV_LOG(WARNING, "Rx CQE compression isn't supported"); + config.cqe_comp = 0; + } + config.mprq.enabled = config.mprq.enabled && mprq; + if (config.mprq.enabled) { + if (config.mprq.stride_num_n > mprq_max_stride_num_n || + config.mprq.stride_num_n < mprq_min_stride_num_n) { + config.mprq.stride_num_n = + RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); + DRV_LOG(WARNING, + "the number of strides" + " for Multi-Packet RQ is out of range," + " setting default value (%u)", + 1 << config.mprq.stride_num_n); + } + config.mprq.min_stride_size_n = mprq_min_stride_size_n; + config.mprq.max_stride_size_n = mprq_max_stride_size_n; + } + eth_dev = rte_eth_dev_allocate(name); + if (eth_dev == NULL) { + DRV_LOG(ERR, "can not allocate rte ethdev"); + err = ENOMEM; + goto error; + } + eth_dev->data->dev_private = priv; + priv->dev_data = eth_dev->data; + eth_dev->data->mac_addrs = priv->mac; + eth_dev->device = dpdk_dev; + eth_dev->device->driver = &mlx5_driver.driver; + err = mlx5_uar_init_primary(eth_dev); + if (err) { + err = rte_errno; + goto error; + } + /* Configure the first MAC address by default. */ + if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { + DRV_LOG(ERR, + "port %u cannot get MAC address, is mlx5_en" + " loaded? (errno: %s)", + eth_dev->data->port_id, strerror(rte_errno)); + err = ENODEV; + goto error; + } + DRV_LOG(INFO, + "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", + eth_dev->data->port_id, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); +#ifndef NDEBUG + { + char ifname[IF_NAMESIZE]; + + if (mlx5_get_ifname(eth_dev, &ifname) == 0) + DRV_LOG(DEBUG, "port %u ifname is \"%s\"", + eth_dev->data->port_id, ifname); + else + DRV_LOG(DEBUG, "port %u ifname is unknown", + eth_dev->data->port_id); + } +#endif + /* Get actual MTU if possible. */ + err = mlx5_get_mtu(eth_dev, &priv->mtu); + if (err) { + err = rte_errno; + goto error; + } + DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, + priv->mtu); + /* Initialize burst functions to prevent crashes before link-up. */ + eth_dev->rx_pkt_burst = removed_rx_burst; + eth_dev->tx_pkt_burst = removed_tx_burst; + eth_dev->dev_ops = &mlx5_dev_ops; + /* Register MAC address. */ + claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); + priv->nl_socket = -1; + priv->nl_sn = 0; + if (vf && config.vf_nl_en) { + priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); + if (priv->nl_socket < 0) + priv->nl_socket = -1; + mlx5_nl_mac_addr_sync(eth_dev); + } + TAILQ_INIT(&priv->flows); + TAILQ_INIT(&priv->ctrl_flows); + /* Hint libmlx5 to use PMD allocator for data plane resources */ + struct mlx5dv_ctx_allocators alctr = { + .alloc = &mlx5_alloc_verbs_buf, + .free = &mlx5_free_verbs_buf, + .data = priv, + }; + mlx5_glue->dv_set_context_attr(ctx, MLX5DV_CTX_ATTR_BUF_ALLOCATORS, + (void *)((uintptr_t)&alctr)); + /* Bring Ethernet device up. */ + DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", + eth_dev->data->port_id); + mlx5_set_link_up(eth_dev); + /* + * Even though the interrupt handler is not installed yet, + * interrupts will still trigger on the asyn_fd from + * Verbs context returned by ibv_open_device(). + */ + mlx5_link_update(eth_dev, 0); + /* Store device configuration on private structure. */ + priv->config = config; + /* Create drop queue. */ + err = mlx5_flow_create_drop_queue(eth_dev); + if (err) { + DRV_LOG(ERR, "port %u drop queue allocation failed: %s", + eth_dev->data->port_id, strerror(rte_errno)); + err = rte_errno; + goto error; + } + /* Supported Verbs flow priority number detection. */ + if (verb_priorities == 0) + verb_priorities = mlx5_get_max_verbs_prio(eth_dev); + if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { + DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", + eth_dev->data->port_id, verb_priorities); + err = ENOTSUP; + goto error; + } + priv->config.max_verbs_prio = verb_priorities; + /* + * Once the device is added to the list of memory event + * callback, its global MR cache table cannot be expanded + * on the fly because of deadlock. If it overflows, lookup + * should be done by searching MR list linearly, which is slow. + */ + err = mlx5_mr_btree_init(&priv->mr.cache, + MLX5_MR_BTREE_CACHE_N * 2, + eth_dev->device->numa_node); + if (err) { + err = rte_errno; + goto error; + } + /* Add device to memory callback list. */ + rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); + LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, + priv, mem_event_cb); + rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); + return eth_dev; error: + if (priv) + rte_free(priv); + if (pd) + claim_zero(mlx5_glue->dealloc_pd(pd)); + if (eth_dev) + rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); assert(err > 0); From patchwork Thu Jun 14 08:34:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 41083 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6F5D21E4BB; Thu, 14 Jun 2018 10:35:17 +0200 (CEST) Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com [209.85.128.193]) by dpdk.org (Postfix) with ESMTP id 898931E49A for ; Thu, 14 Jun 2018 10:35:14 +0200 (CEST) Received: by mail-wr0-f193.google.com with SMTP id l41-v6so5437638wre.7 for ; Thu, 14 Jun 2018 01:35:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=yNUUjGItogqUZ+HSpP8/oe62lbl8J/hjOY7zS+xO7ZE=; b=fjPfuWHkVjpLymbcQVbgnWehh3GFCjekgbibyghx/BTeaYNcDQrdFY49w9J+WThUZB c34Ahvzhz2vQbCLnUKSATmM0kiWc/jIHuyP73mtYSEsp2yotiiLzeY2Ym2+lQdfp1N9l /1rFsYljBUFuwvx2fAgaIRZCWpbU3HmduA6qNbi6vs16k908JuTEwsxbG05vVW+DvLZP wRcMt44+wJI5JIFNCJMC/sWZYPCkUYsTkVLprJMC0I7pI0OXQvtQ3i7cfnMG3y2oyOq8 xR8q0eHZmdwGD5w9CcRtNMc1LXq84c+ZWJ3HhBfJILM85QIhFLefjBf4AH3KB7pgz87x 4HXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=yNUUjGItogqUZ+HSpP8/oe62lbl8J/hjOY7zS+xO7ZE=; b=ZIEPUEVG5Whl3zpNiNjXwtewBe4xJQ00CvWyqXCl4V1gELV2Blo2I4jBIFPOQr/VMd cZjSLD8NYj9GXcTZR86nZIB8hEuWtqj/x4hWVO/Re08O+/xz5PVhPz3IjlnmFh2XDg8r SHL/XJkmIAAkYzzJ3/itrS/zMm6pAJmF5TKwcJK/rGq9zEjKdJwN2DOhF0If7P7QKCV9 qvYDlhPJaqQAYnLPcApiJ7e8qUFpfAvGN2glnNFeXBEGpHElwFvljrTdiMvQV30Hyf89 zkBfwC99LW7a56fTkUTHhqYJNYN1MWVkqwyA8G4NP2n8uiMuMzUrpXtGZWYCp71uSuzd OfgQ== X-Gm-Message-State: APt69E2GKEkOkcODEyFN/SQb6GxTaP804Hpmlwu1FKVWIedlmXOfTrYH qJ8yTaXnweS/Ew/C7IspTmViKg== X-Google-Smtp-Source: ADUXVKIzjuXBNJRsHjBv4olGUeMxRVY6xtG3Yo2GrgMjj7gprHkLMebVK/ei6K28l1cvsKERfSNmXA== X-Received: by 2002:adf:979c:: with SMTP id s28-v6mr1410129wrb.28.1528965314245; Thu, 14 Jun 2018 01:35:14 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id a9-v6sm4338695wmh.38.2018.06.14.01.35.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:35:13 -0700 (PDT) Date: Thu, 14 Jun 2018 10:34:58 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180614083047.10812-6-adrien.mazarguil@6wind.com> References: <20180525161814.13873-1-adrien.mazarguil@6wind.com> <20180614083047.10812-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180614083047.10812-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 5/7] net/mlx5: add port representor awareness X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The current PCI probing method is not aware of Verbs port representors, which appear as standard Verbs devices bound to the same PCI address and cannot be distinguished. Problem is that more often than not, the wrong Verbs device is used, resulting in unexpected traffic. This patch adds necessary heuristics to bind affected driver instances to the intended (i.e. non-representor) device. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Fixed digit detection in mlx5_cmp_ibv_name() so that "foo1" and "foo10" are compared on the integer conversion of "1" against "10" instead of "" and "0". --- drivers/net/mlx5/mlx5.c | 66 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 5 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c9815d721..498f80c89 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -3,6 +3,7 @@ * Copyright 2015 Mellanox Technologies, Ltd */ +#include #include #include #include @@ -1170,6 +1171,34 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, } /** + * Comparison callback to sort Verbs device names. + * + * This is meant to be used with qsort(). + * + * @param a[in] + * Pointer to pointer to first Verbs device. + * @param b[in] + * Pointer to pointer to second Verbs device. + * + * @return + * 0 if both names are equal, less than 0 if the first argument is less + * than the second, greater than 0 otherwise. + */ +static int +mlx5_cmp_ibv_name(const void *a, const void *b) +{ + const char *name_a = (*(const struct ibv_device *const *)a)->name; + const char *name_b = (*(const struct ibv_device *const *)b)->name; + size_t i = 0; + + while (name_a[i] && name_a[i] == name_b[i]) + ++i; + while (i && isdigit(name_a[i - 1]) && isdigit(name_b[i - 1])) + --i; + return atoi(name_a + i) - atoi(name_b + i); +} + +/** * DPDK callback to register a PCI device. * * This function creates an Ethernet device for each port of a given @@ -1189,6 +1218,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **ibv_list; struct rte_eth_dev **eth_list = NULL; + int n = 0; int vf; int ret; @@ -1210,6 +1240,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } + + struct ibv_device *ibv_match[ret + 1]; + while (ret-- > 0) { struct rte_pci_addr pci_addr; @@ -1221,14 +1254,37 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid != pci_addr.devid || pci_dev->addr.function != pci_addr.function) continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + DRV_LOG(INFO, "PCI information matches for device \"%s\"", ibv_list[ret]->name); - break; + ibv_match[n++] = ibv_list[ret]; + } + ibv_match[n] = NULL; + if (n > 1) { + /* + * The existence of several matching entries means port + * representors have been instantiated. No existing Verbs + * call nor /sys entries can tell them apart at this point. + * + * While definitely hackish, assume their names are numbered + * based on order of creation with master device first, + * followed by first port representor, followed by the + * second one and so on. + */ + DRV_LOG(WARNING, + "probing device with port representors involves" + " heuristics with uncertain outcome"); + qsort(ibv_match, n, sizeof(*ibv_match), mlx5_cmp_ibv_name); + DRV_LOG(WARNING, "assuming \"%s\" is the master device", + ibv_match[0]->name); + for (ret = 1; ret < n; ++ret) + DRV_LOG(WARNING, + "assuming \"%s\" is port representor #%d", + ibv_match[ret]->name, ret - 1); } - if (ret >= 0) - eth_list = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + if (n) + eth_list = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); mlx5_glue->free_device_list(ibv_list); - if (!ret) { + if (!n) { DRV_LOG(WARNING, "no Verbs device matches PCI device " PCI_PRI_FMT "," " are kernel drivers loaded?", From patchwork Thu Jun 14 08:35:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 41084 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D568E1E40F; Thu, 14 Jun 2018 10:35:18 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id 184A11E4BB for ; Thu, 14 Jun 2018 10:35:16 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id v16-v6so9044074wmh.5 for ; Thu, 14 Jun 2018 01:35:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ZtBAoQ1EF3KjkOzULMCfu4AfjlyWij9k6c/UDyXWHmI=; b=wGgEwV+D91L2HARzedZTZIHYxh/nFbjgRWusYb2ukqI9nDusFjBHnbE7f1AtEbvFmm 4W1wQUz1W6oIiF3A5WvNm4R7hzJhqV2x56Qn+H9BsQ0D3bWkQluwptK97H/QOSxdLleu osoS5rGWcnTYs2a/JA9hkUWG0vi90dVfWeJOBP5OfK4MdVpny1DdGBpJn9B5sOm//RF7 vOkKpG20ds5DNsPCV9c1s5a84swGtuy7ioNeAcqjg+fWpiE+Y/QYO+axa2JLVxcfFZja 2HddUayyGRZ2weqIohJeEvuGHQLVWOtUMVZTMlydSmdWNYGRfrB58mlpDrma+p11zfTO hLYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ZtBAoQ1EF3KjkOzULMCfu4AfjlyWij9k6c/UDyXWHmI=; b=GZZY3Yy+yeeTKGoLf5bAZ/+pZ1g4/SqW4lQSMzVpqT0kooE8+opl6TXZ7XVDqfy2xQ bmI5LmWQ9B4QkcQaO0F20lmCSC/yvqfObesKE9bdI9ELNYPRa8cANpczzkSztRhgmEiO Ri07NDa7qV/78VJWNWvP4dCzHIWwZNmrwW3XUxujXaJh6hrfo+iMMHVNdr21HMINzN7p SOsXxCx7otO8lE+3IaXzX68QXay4wsek9DYqXfPqZ0K+6ZrqagIj7ws7sbItsCkPH6Zu fmPGPeEqT3k/+6Mtikyr1axxiOdVgLsMpL1nDoSdGB7h1XVv3O2TKEUoid07wrkgG1S0 iwHA== X-Gm-Message-State: APt69E2XiWkp2fyJ2yKzmbKSElp5tUWhX2t1tdH+4S7QlzEH4qOWa31H VTYpXYYPXgy/GmHnPfZ1aBg/Gg== X-Google-Smtp-Source: ADUXVKJ9ug0aBgNiWjVSg5Sz8qmoMlkx+pe3+WBu4OOoHOnQjR9MqbnVm93Rg0g4gBYq2BVGDhq3/A== X-Received: by 2002:a1c:f513:: with SMTP id t19-v6mr1149987wmh.50.1528965316304; Thu, 14 Jun 2018 01:35:16 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id c53-v6sm12025494wrg.12.2018.06.14.01.35.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:35:15 -0700 (PDT) Date: Thu, 14 Jun 2018 10:35:00 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180614083047.10812-7-adrien.mazarguil@6wind.com> References: <20180525161814.13873-1-adrien.mazarguil@6wind.com> <20180614083047.10812-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180614083047.10812-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 6/7] net/mlx5: probe all port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Probe existing port representors in addition to their master device and associate them automatically. To avoid name collision between Ethernet devices, their names use the same convention as ixgbe and i40e PMDs, that is, instead of only a PCI address in DBDF notation: - "net_{DBDF}_0" for master/switch devices. - "net_{DBDF}_representor_{rep}" with "rep" starting from 0 for port representors. Both optionally suffixed with "_port_{num}" instead of " port {num}" for devices that expose several Verbs ports (note this is never the case on mlx5, but kept for historical reasons for the time being). (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Added representor information to dev_infos_get(). DPDK port ID of master device is now stored in the private structure to retrieve it conveniently. - Master device is assigned dummy representor ID value -1 to better distinguish from the the first actual representor reported by dev_infos_get() as those are indexed from 0. - Added RTE_ETH_DEV_REPRESENTOR device flag. --- drivers/net/mlx5/mlx5.c | 138 ++++++++++++++++++++++++-------- drivers/net/mlx5/mlx5.h | 9 ++- drivers/net/mlx5/mlx5_ethdev.c | 151 ++++++++++++++++++++++++++++++++---- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_stats.c | 6 +- 5 files changed, 252 insertions(+), 54 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 498f80c89..716c9d9a5 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -304,6 +304,9 @@ mlx5_dev_close(struct rte_eth_dev *dev) if (ret) DRV_LOG(WARNING, "port %u some flows still remain", dev->data->port_id); + if (!priv->representor && + priv->domain_id != RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); memset(priv, 0, sizeof(*priv)); } @@ -648,6 +651,10 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) * Verbs device attributes. * @param port * Verbs port to use (indexed from 1). + * @param master + * Master device in case @p ibv_dev is a port representor. + * @param rep_id + * Representor identifier when @p master is non-NULL. * * @return * A valid Ethernet device object on success, NULL otherwise and rte_errno @@ -658,7 +665,9 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, struct ibv_device *ibv_dev, int vf, const struct ibv_device_attr_ex *attr, - unsigned int port) + unsigned int port, + struct rte_eth_dev *master, + unsigned int rep_id) { struct ibv_context *ctx; struct ibv_port_attr port_attr; @@ -802,11 +811,14 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, " old OFED/rdma-core version or firmware configuration"); #endif config.mpls_en = mpls_en; - if (attr->orig_attr.phys_port_cnt > 1) - snprintf(name, sizeof(name), "%s port %u", - dpdk_dev->name, port); + if (!master) + snprintf(name, sizeof(name), "net_%s_0", dpdk_dev->name); else - snprintf(name, sizeof(name), "%s", dpdk_dev->name); + snprintf(name, sizeof(name), "net_%s_representor_%u", + dpdk_dev->name, rep_id); + if (attr->orig_attr.phys_port_cnt > 1) + snprintf(name, sizeof(name), "%s_port_%u", name, port); + DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -883,6 +895,30 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; + /* + * Allocate a switch domain for master devices and share it with + * port representors. + */ + if (!master) { + priv->representor = 0; + priv->master_id = -1; /* Updated once known. */ + priv->domain_id = RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID; + priv->rep_id = -1; /* Dummy unique value. */ + err = rte_eth_switch_domain_alloc(&priv->domain_id); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "unable to allocate switch domain: %s", + strerror(rte_errno)); + goto error; + } + } else { + priv->representor = 1; + priv->master_id = + ((struct priv *)master->data->dev_private)->master_id; + priv->domain_id = + ((struct priv *)master->data->dev_private)->domain_id; + priv->rep_id = rep_id; + } err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; @@ -964,6 +1000,18 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, err = ENOMEM; goto error; } + /* + * Now that eth_dev is allocated and its port ID is known, make + * non-representor ports target their own port ID as master for + * convenience. + * + * Master port ID is already set for actual representors. Those only + * need the right device flag. + */ + if (!master) + priv->master_id = eth_dev->data->port_id; + else + eth_dev->data->dev_flags |= RTE_ETH_DEV_REPRESENTOR; eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; @@ -1083,8 +1131,12 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); return eth_dev; error: - if (priv) + if (priv) { + if (!priv->representor && + priv->domain_id != RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); rte_free(priv); + } if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); if (eth_dev) @@ -1097,12 +1149,14 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, } /** - * Spawn Ethernet devices from Verbs information, one per detected port. + * Spawn Ethernet devices from Verbs information, one per detected port and + * port representor. * * @param dpdk_dev * Backing DPDK device. * @param ibv_dev - * Verbs device. + * NULL-terminated list of Verbs devices. First entry is the master device + * (mandatory), followed by optional representors. * @param vf * If nonzero, enable VF-specific features. * @@ -1113,17 +1167,21 @@ mlx5_dev_spawn_one(struct rte_device *dpdk_dev, */ static struct rte_eth_dev ** mlx5_dev_spawn(struct rte_device *dpdk_dev, - struct ibv_device *ibv_dev, + struct ibv_device **ibv_dev, int vf) { struct rte_eth_dev **eth_list = NULL; struct ibv_context *ctx; struct ibv_device_attr_ex attr; + void *tmp; unsigned int i; + unsigned int j = 0; + unsigned int n = 0; int ret; +next: errno = 0; - ctx = mlx5_glue->open_device(ibv_dev); + ctx = mlx5_glue->open_device(ibv_dev[j]); if (!ctx) { rte_errno = errno ? errno : ENODEV; if (rte_errno == ENODEV) @@ -1132,7 +1190,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, else DRV_LOG(ERR, "cannot use device, are drivers up to date?"); - return NULL; + goto error; } ret = mlx5_glue->query_device_ex(ctx, NULL, &attr); mlx5_glue->close_device(ctx); @@ -1140,34 +1198,42 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, rte_errno = ret; DRV_LOG(ERR, "unable to query device information: %s", strerror(rte_errno)); - return NULL; + goto error; } - DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); - eth_list = malloc(sizeof(*eth_list) * - (attr.orig_attr.phys_port_cnt + 1)); - if (!eth_list) { + DRV_LOG(INFO, "%u port(s) detected on \"%s\"", + attr.orig_attr.phys_port_cnt, ibv_dev[j]->name); + tmp = realloc(eth_list, sizeof(*eth_list) * + (n + attr.orig_attr.phys_port_cnt + 1)); + if (!tmp) { rte_errno = errno; - return NULL; + goto error; } + eth_list = tmp; for (i = 0; i < attr.orig_attr.phys_port_cnt; ++i) { - eth_list[i] = mlx5_dev_spawn_one(dpdk_dev, ibv_dev, vf, - &attr, i + 1); - if (eth_list[i]) - continue; - /* Save rte_errno and roll back in case of failure. */ - ret = rte_errno; - while (i--) { - mlx5_dev_close(eth_list[i]); - if (rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_free(eth_list[i]->data->dev_private); - claim_zero(rte_eth_dev_release_port(eth_list[i])); - } - free(eth_list); - rte_errno = ret; - return NULL; + eth_list[n] = mlx5_dev_spawn_one(dpdk_dev, ibv_dev[j], vf, + &attr, i + 1, + j ? eth_list[0] : NULL, + j - 1); + if (!eth_list[n]) + goto error; + ++n; } - eth_list[i] = NULL; + if (ibv_dev[++j]) + goto next; + eth_list[n] = NULL; return eth_list; +error: + /* Save rte_errno and roll back in case of failure. */ + ret = rte_errno; + while (n--) { + mlx5_dev_close(eth_list[n]); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_free(eth_list[n]->data->dev_private); + claim_zero(rte_eth_dev_release_port(eth_list[n])); + } + free(eth_list); + rte_errno = ret; + return NULL; } /** @@ -1282,7 +1348,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ibv_match[ret]->name, ret - 1); } if (n) - eth_list = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); + eth_list = mlx5_dev_spawn(&pci_dev->device, ibv_match, vf); mlx5_glue->free_device_list(ibv_list); if (!n) { DRV_LOG(WARNING, @@ -1302,7 +1368,11 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; } else { for (ret = 0; eth_list[ret]; ++ret) { + uint32_t restore = eth_list[ret]->data->dev_flags; + rte_eth_copy_pci_info(eth_list[ret], pci_dev); + /* Restore non-PCI flags cleared by the above call. */ + eth_list[ret]->data->dev_flags |= restore; rte_eth_dev_probing_finish(eth_list[ret]); } ret = 0; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 997b04a33..0fe467140 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -161,6 +161,10 @@ struct priv { uint16_t mtu; /* Configured MTU. */ uint8_t port; /* Physical port number. */ unsigned int isolated:1; /* Whether isolated mode is enabled. */ + unsigned int representor:1; /* Device is a port representor. */ + uint16_t master_id; /* DPDK port ID of switch domain master. */ + uint16_t domain_id; /* Switch domain identifier. */ + unsigned int rep_id; /* Port representor identifier. */ /* RX/TX queues. */ unsigned int rxqs_n; /* RX queues array size. */ unsigned int txqs_n; /* TX queues array size. */ @@ -209,9 +213,12 @@ int mlx5_getenv_int(const char *); /* mlx5_ethdev.c */ +int mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]); int mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]); int mlx5_ifindex(const struct rte_eth_dev *dev); -int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr); +int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master); int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu); int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 90488af33..9d579659e 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -93,7 +93,7 @@ struct ethtool_link_settings { #endif /** - * Get interface name from private structure. + * Get master interface name from private structure. * * @param[in] dev * Pointer to Ethernet device. @@ -104,7 +104,8 @@ struct ethtool_link_settings { * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]) { struct priv *priv = dev->data->dev_private; DIR *dir; @@ -179,6 +180,113 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) } /** + * Get interface name from private structure. + * + * This is a port representor-aware version of mlx5_get_master_ifname(). + * + * @param[in] dev + * Pointer to Ethernet device. + * @param[out] ifname + * Interface name output buffer. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +{ + struct priv *priv = dev->data->dev_private; + int ret; + char master[IF_NAMESIZE]; + FILE *file; + DIR *dir; + uint64_t phys_switch_id; + + if (!priv->representor) + return mlx5_get_master_ifname(dev, ifname); + ret = mlx5_get_master_ifname(dev, &master); + if (ret) + return ret; + { + MKSTR(path, "%s/device/net/%s/phys_switch_id", + priv->ibdev_path, master); + + file = fopen(path, "rb"); + } + if (!file) { + rte_errno = errno; + return -rte_errno; + } + ret = fscanf(file, "%" SCNx64, &phys_switch_id); + fclose(file); + if (ret != 1) { + rte_errno = EINVAL; + return -rte_errno; + } + { + MKSTR(path, "%s/device/net/%s/subsystem", + priv->ibdev_path, master); + + dir = opendir(path); + } + if (!dir) { + rte_errno = errno; + return -rte_errno; + } + /* + * Scan network interfaces to find one with matching phys_switch_id + * and phys_switch_name. + */ + do { + struct dirent *dent; + uint64_t phys_switch_id_rep; + int rep_id; + + ret = -ENOENT; + dent = readdir(dir); + if (!dent) + break; + { + MKSTR(path, + "%s/device/net/%s/subsystem/%s/phys_switch_id", + priv->ibdev_path, master, dent->d_name); + + file = fopen(path, "rb"); + } + if (!file) + continue; + ret = fscanf(file, "%" SCNx64, &phys_switch_id_rep); + fclose(file); + if (ret != 1) + continue; + if (phys_switch_id_rep != phys_switch_id) + continue; + { + MKSTR(path, + "%s/device/net/%s/subsystem/%s/phys_port_name", + priv->ibdev_path, master, dent->d_name); + + file = fopen(path, "rb"); + } + if (!file) + continue; + ret = fscanf(file, "%d", &rep_id); + fclose(file); + if (ret != 1) + continue; + if (rep_id < 0 || (unsigned int)rep_id != priv->rep_id) + continue; + strlcpy(*ifname, dent->d_name, sizeof(*ifname)); + ret = 0; + break; + } while (1); + closedir(dir); + if (ret) + rte_errno = -ret; + return ret; +} + +/** * Get the interface index from device name. * * @param[in] dev @@ -214,12 +322,16 @@ mlx5_ifindex(const struct rte_eth_dev *dev) * Request number to pass to ioctl(). * @param[out] ifr * Interface request structure output buffer. + * @param master + * When device is a port representor, perform request on master device + * instead. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) +mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master) { int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP); int ret = 0; @@ -228,7 +340,10 @@ mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) rte_errno = errno; return -rte_errno; } - ret = mlx5_get_ifname(dev, &ifr->ifr_name); + if (master) + ret = mlx5_get_master_ifname(dev, &ifr->ifr_name); + else + ret = mlx5_get_ifname(dev, &ifr->ifr_name); if (ret) goto error; ret = ioctl(sock, req, ifr); @@ -258,7 +373,7 @@ int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request); + int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request, 0); if (ret) return ret; @@ -282,7 +397,7 @@ mlx5_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) { struct ifreq request = { .ifr_mtu = mtu, }; - return mlx5_ifreq(dev, SIOCSIFMTU, &request); + return mlx5_ifreq(dev, SIOCSIFMTU, &request, 0); } /** @@ -302,13 +417,13 @@ int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request); + int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request, 0); if (ret) return ret; request.ifr_flags &= keep; request.ifr_flags |= flags & ~keep; - return mlx5_ifreq(dev, SIOCSIFFLAGS, &request); + return mlx5_ifreq(dev, SIOCSIFFLAGS, &request, 0); } /** @@ -477,6 +592,12 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) info->speed_capa = priv->link_speed_capa; info->flow_type_rss_offloads = ~MLX5_RSS_HF_MASK; mlx5_set_default_params(dev, info); + if (rte_eth_dev_is_valid_port(priv->master_id)) { + info->switch_info.name = + rte_eth_devices[priv->master_id].data->name; + info->switch_info.domain_id = priv->domain_id; + info->switch_info.port_id = priv->rep_id; + } } /** @@ -540,7 +661,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, int link_speed = 0; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -550,7 +671,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&edata; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GSET) failed: %s", @@ -611,7 +732,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, uint64_t sc; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -621,7 +742,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&gcmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -638,7 +759,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, *ecmd = gcmd; ifr.ifr_data = (void *)ecmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -801,7 +922,7 @@ mlx5_dev_get_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) int ret; ifr.ifr_data = (void *)ðpause; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GPAUSEPARAM) failed:" @@ -854,7 +975,7 @@ mlx5_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) ethpause.tx_pause = 1; else ethpause.tx_pause = 0; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 0); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_SPAUSEPARAM)" diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index 672a47619..12ee37f55 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -49,7 +49,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[ETHER_ADDR_LEN]) struct ifreq request; int ret; - ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request); + ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request, 0); if (ret) return ret; memcpy(mac, request.ifr_hwaddr.sa_data, ETHER_ADDR_LEN); diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c index 875dd1027..91f3d474a 100644 --- a/drivers/net/mlx5/mlx5_stats.c +++ b/drivers/net/mlx5/mlx5_stats.c @@ -146,7 +146,7 @@ mlx5_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) et_stats->cmd = ETHTOOL_GSTATS; et_stats->n_stats = xstats_ctrl->stats_n; ifr.ifr_data = (caddr_t)et_stats; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to read statistic values from device", @@ -194,7 +194,7 @@ mlx5_ethtool_get_stats_n(struct rte_eth_dev *dev) { drvinfo.cmd = ETHTOOL_GDRVINFO; ifr.ifr_data = (caddr_t)&drvinfo; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to query number of statistics", dev->data->port_id); @@ -244,7 +244,7 @@ mlx5_xstats_init(struct rte_eth_dev *dev) strings->string_set = ETH_SS_STATS; strings->len = dev_stats_n; ifr.ifr_data = (caddr_t)strings; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to get statistic names", dev->data->port_id); From patchwork Thu Jun 14 08:35:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 41085 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7E8701E553; Thu, 14 Jun 2018 10:35:20 +0200 (CEST) Received: from mail-wr0-f194.google.com (mail-wr0-f194.google.com [209.85.128.194]) by dpdk.org (Postfix) with ESMTP id 934931E40F for ; Thu, 14 Jun 2018 10:35:18 +0200 (CEST) Received: by mail-wr0-f194.google.com with SMTP id e18-v6so5456246wrs.5 for ; Thu, 14 Jun 2018 01:35:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=xLu3gnQ5zQUkUpZFgcgJizivfiIyQhz5ETjbX1E10og=; b=xT0PnnUepM0apcy6e5lJxukiJKSq0xPTM0Eo2xarGvRT1bMoV51mLijubBtOJwQjY6 BEAynqJInAtACX2JW4WDRd7CV2nXVDzOAbqi8e3t7sBBnfQHf0g1mWyRj6P5Ph/B1GHE nVEsMu5Zh7br2pMVyAMU2UHwoEhinzuLa40TCTsn84Mw4iiX34TBigE7Bpd3GC23LcYT DP8GLqhQIjwx5xXwFcc2ItnRx2Ze/zthHbT0oVozp+5odT9kU5FjCCIKRG/4bXGF0ZL8 ZVkBal6QO0Na7N5sBiY9SW/PBK78QzMGsx0h73kkt1D3JAwtzQJdn3wR7+TZsMOAkXk0 DZ/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=xLu3gnQ5zQUkUpZFgcgJizivfiIyQhz5ETjbX1E10og=; b=YiuMTr9VBfF4v0h97DihY+DW2lXRGtGrCctmLpB+46EMngiCKxUbOyaUMEXgP1afwb Q9VU52tey/kX2omC7Tp28cIVYzfkL+6k/MWJyIMBBFG9OMcO7ARcF2cOVpvGwyPF57VT cOgZs0yb4ZpRRswFjKP1iewj0KeLtNGAJ92+RzS5r7XP0QdGH7qE4QizuVDM7YXHeB8A zsrM/CzLHSbfAflQESdjHE9d/tQ6HwXwE0JOCdB5k3zHYJTsQdssO5lqDb9Pl4aW8JeM r0EemoUc9oCkHfwee2pnvSeR12AHPhLDYkKtOBmeMqJIlvC8fTivb7te91+koaBAkMeK 4COQ== X-Gm-Message-State: APt69E2Ye2ZKBpASNvCX/VHjdzaCGCLMqnei/lWUVcFuKNCYmn3paB69 ULkGduZYLExeLkEMmB8tolI8O8yZ X-Google-Smtp-Source: ADUXVKLMKtVuus7TUVpotvXjx28YSDHYBQFF+k/F2PbFyF5cxTadAmvpQMI4+GSOsB8zd9ZWa3nn0A== X-Received: by 2002:adf:fb92:: with SMTP id a18-v6mr1247230wrr.278.1528965318361; Thu, 14 Jun 2018 01:35:18 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id c11-v6sm4848506wrm.65.2018.06.14.01.35.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:35:17 -0700 (PDT) Date: Thu, 14 Jun 2018 10:35:02 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180614083047.10812-8-adrien.mazarguil@6wind.com> References: <20180525161814.13873-1-adrien.mazarguil@6wind.com> <20180614083047.10812-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180614083047.10812-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 7/7] net/mlx5: add parameter for port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Prior to this patch, all port representors detected on a given device were probed and Ethernet devices instantiated for each of them. This patch adds support for the standard "representor" parameter, which implies that port representors are not probed by default anymore, except for the list provided through device arguments. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Added error message for when rte_eth_devargs_parse() fails. --- doc/guides/nics/mlx5.rst | 12 ++++++++++++ doc/guides/prog_guide/poll_mode_drv.rst | 2 ++ drivers/net/mlx5/mlx5.c | 29 ++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 79c982e29..5229e546c 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -388,6 +388,18 @@ Run-time configuration Disabled by default. +- ``representor`` parameter [list] + + This parameter can be used to instantiate DPDK Ethernet devices from + existing port (or VF) representors configured on the device. + + It is a standard parameter whose format is described in + :ref:`ethernet_device_standard_device_arguments`. + + For instance, to probe port representors 0 through 2:: + + representor=[0-2] + Firmware configuration ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index af82352a0..58d49ba0f 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -365,6 +365,8 @@ Ethernet Device API The Ethernet device API exported by the Ethernet PMDs is described in the *DPDK API Reference*. +.. _ethernet_device_standard_device_arguments: + Ethernet Device Standard Device Arguments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 716c9d9a5..26e61d99d 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -91,6 +91,9 @@ /* Activate Netlink support in VF mode. */ #define MLX5_VF_NL_EN "vf_nl_en" +/* Select port representors to instantiate. */ +#define MLX5_REPRESENTOR "representor" + #ifndef HAVE_IBV_MLX5_MOD_MPW #define MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED (1 << 2) #define MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW (1 << 3) @@ -423,6 +426,9 @@ mlx5_args_check(const char *key, const char *val, void *opaque) struct mlx5_dev_config *config = opaque; unsigned long tmp; + /* No-op, port representors are processed in mlx5_dev_spawn(). */ + if (!strcmp(MLX5_REPRESENTOR, key)) + return 0; errno = 0; tmp = strtoul(val, NULL, 0); if (errno) { @@ -495,6 +501,7 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_RX_VEC_EN, MLX5_L3_VXLAN_EN, MLX5_VF_NL_EN, + MLX5_REPRESENTOR, NULL, }; struct rte_kvargs *kvlist; @@ -1173,13 +1180,34 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, struct rte_eth_dev **eth_list = NULL; struct ibv_context *ctx; struct ibv_device_attr_ex attr; + struct rte_eth_devargs eth_da; void *tmp; unsigned int i; unsigned int j = 0; unsigned int n = 0; int ret; + if (dpdk_dev->devargs) { + ret = rte_eth_devargs_parse(dpdk_dev->devargs->args, ð_da); + if (ret) { + rte_errno = -ret; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + goto error; + } + } else { + memset(ð_da, 0, sizeof(eth_da)); + } next: + if (j) { + unsigned int k; + + for (k = 0; k < eth_da.nb_representor_ports; ++k) + if (eth_da.representor_ports[k] == j - 1) + break; + if (k == eth_da.nb_representor_ports) + goto skip; + } errno = 0; ctx = mlx5_glue->open_device(ibv_dev[j]); if (!ctx) { @@ -1218,6 +1246,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, goto error; ++n; } +skip: if (ibv_dev[++j]) goto next; eth_list[n] = NULL;