From patchwork Thu Jul 5 08:45:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42305 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 501091C38A; Thu, 5 Jul 2018 10:45:50 +0200 (CEST) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id B63AF1C382 for ; Thu, 5 Jul 2018 10:45:48 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id s12-v6so9716208wmc.1 for ; Thu, 05 Jul 2018 01:45:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=U+2tgo9qEAMbh5Y43XjgWFJ2nazL87NKgcslsyD/Uk8HEEnPowzLQSf5euQPC2L8vj w5ssEZbWSGKyHmZAPHe7AtMCUHug+aEeCC5gN2MFZvQUSeXD2EH7qmdQm950cAoEuyJb XcHdnp/KlIBVD77GPNfz2aNrVH2NOcq5mwvXg47zRKtL3YorrYtlakgGg6gIco8OQLrm pXQ13j6Ma+KOKdzdEBDnj8Bv6T7tESm4HPLvALSsQ5RoHN4OOn1VTu95c9B9Cv18et4I SRseEKWk5SRiObnQP9F2P4Qnv4DMnmtCV/ekVuFa4TF3OH0UMF5Y2D45BZllpXTiEqHQ BgbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=jlKl9rx2oGEAUZAYQ6Q9X3ipg+/OJkklDEDv27QKqwtM4aXP4a/UtqAcJkoi8fRvqI I/YhEqwZuSee6ObBcijxt54aEOixvvhnpBT05jMemDX0Ei0oK8Ql+WZmwIRVjHzqNFFh HCvgGl/QDH0NIZ0lII+vWCABK5Fejo2awfXoxYoFzSeFewfTBOLrD4wUUdpFu/6Uoh60 8a8sUYrcXAzGZvZGquway5axekTWBz+lwYhbkpwcngicoynzlzuBuH/vccGhw8YqKrLr a4u3qONtzG+gX9hE+e1wDkUqhHWSgz+CJwsYaQowL5rRkHe2DLho9n5oz4M+w4cKixdB NeEg== X-Gm-Message-State: APt69E2kQgXfY0FrhlngNhmByFlN73weYy5hlN7pjbObaFzp20fx13e2 2LU7W1einEZO4Aa7/9+2hcgaqA== X-Google-Smtp-Source: AAOMgpfrwWVB0gXB8r6jvpI34rDFM3vow5gpJ4KG8VQEjbf9s9mSDK08j9K+6+l1aVjOnUwx6FPg4g== X-Received: by 2002:a1c:7409:: with SMTP id p9-v6mr3276156wmc.43.1530780348359; Thu, 05 Jul 2018 01:45:48 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id s10-v6sm10165442wmb.12.2018.07.05.01.45.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:47 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:31 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-5-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 04/10] net/mlx5: split PCI from generic probing code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" All the generic probing code needs is an IB device. While this device is currently supplied by a PCI lookup, other methods will be added soon. This patch divides the original function, which has become huge over time, as follows: 1. PCI-specific (mlx5_pci_probe()). 2. Verbs device (mlx5_dev_spawn()). (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li Cc: Shahaf Shuler --- v3 changes: - Moved VF device check within mlx5_pci_probe() after identifying the device instead of before that. - Merged mlx5_dev_spawn_one() with mlx5_dev_spawn() since there is no need anymore for an intermediate function to iterate over Verbs ports. v2 changes: - Fixed device naming. A port suffix is now appended only if several IB ports happen to be detected. - Added separate message to distinguish missing kernel drivers from other initialization errors, as it was confusing. --- drivers/net/mlx5/mlx5.c | 195 ++++++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 86 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 717d8b268..8916d4684 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -36,6 +36,7 @@ #include #include #include +#include #include "mlx5.h" #include "mlx5_utils.h" @@ -635,32 +636,31 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) } /** - * DPDK callback to register a PCI device. - * - * This function creates an Ethernet device for each port of a given - * PCI device. + * Spawn an Ethernet device from Verbs information. * - * @param[in] pci_drv - * PCI driver structure (mlx5_driver). - * @param[in] pci_dev - * PCI device information. + * @param dpdk_dev + * Backing DPDK device. + * @param ibv_dev + * Verbs device. + * @param vf + * If nonzero, enable VF-specific features. * * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. + * A valid Ethernet device object on success, NULL otherwise and rte_errno + * is set. */ -static int -mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +static struct rte_eth_dev * +mlx5_dev_spawn(struct rte_device *dpdk_dev, + struct ibv_device *ibv_dev, + int vf) { - struct ibv_device **list; - struct ibv_context *ctx = NULL; + struct ibv_context *ctx; struct ibv_device_attr_ex attr; struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; - unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; unsigned int tunnel_en = 0; @@ -672,71 +672,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_max_stride_size_n = 0; unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; - int i; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); - assert(pci_drv == &mlx5_driver); - list = mlx5_glue->get_device_list(&i); - if (list == NULL) { - assert(errno); - err = errno; - if (errno == ENOSYS) - DRV_LOG(ERR, - "cannot list devices, is ib_uverbs loaded?"); - goto error; - } - assert(i >= 0); - /* - * For each listed device, check related sysfs entry against - * the provided PCI ID. - */ - while (i != 0) { - struct rte_pci_addr pci_addr; - - --i; - DRV_LOG(DEBUG, "checking device \"%s\"", list[i]->name); - if (mlx5_ibv_device_to_pci_addr(list[i], &pci_addr)) - continue; - if ((pci_dev->addr.domain != pci_addr.domain) || - (pci_dev->addr.bus != pci_addr.bus) || - (pci_dev->addr.devid != pci_addr.devid) || - (pci_dev->addr.function != pci_addr.function)) - continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", - list[i]->name); - vf = ((pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - ctx = mlx5_glue->open_device(list[i]); - rte_errno = errno; - err = rte_errno; - break; - } - mlx5_glue->free_device_list(list); - if (ctx == NULL) { - switch (err) { - case 0: - DRV_LOG(ERR, - "cannot access device, is mlx5_ib loaded?"); - err = ENODEV; - break; - case EINVAL: - DRV_LOG(ERR, - "cannot use device, are drivers up to date?"); - break; - } - goto error; + errno = 0; + ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) { + rte_errno = errno ? errno : ENODEV; + return NULL; } - DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif @@ -855,9 +802,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - snprintf(name, sizeof(name), PCI_PRI_FMT, - pci_dev->addr.domain, pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -866,7 +811,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; goto error; } - eth_dev->device = &pci_dev->device; + eth_dev->device = dpdk_dev; eth_dev->dev_ops = &mlx5_dev_sec_ops; err = mlx5_uar_init_secondary(eth_dev); if (err) { @@ -894,9 +839,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mlx5_select_rx_function(eth_dev); eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); - rte_eth_dev_probing_finish(eth_dev); claim_zero(mlx5_glue->close_device(ctx)); - return 0; + return eth_dev; } /* Check port status. */ err = mlx5_glue->query_port(ctx, 1, &port_attr); @@ -935,7 +879,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->device_attr = attr; priv->pd = pd; priv->mtu = ETHER_MTU; - err = mlx5_args(&config, pci_dev->device.devargs); + err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", @@ -1027,8 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; - eth_dev->device = &pci_dev->device; - rte_eth_copy_pci_info(eth_dev, pci_dev); + eth_dev->device = dpdk_dev; eth_dev->device->driver = &mlx5_driver.driver; err = mlx5_uar_init_primary(eth_dev); if (err) { @@ -1146,7 +1089,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - return 0; + return eth_dev; } error: if (priv) @@ -1157,11 +1100,91 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (err) { - rte_errno = err; + assert(err > 0); + rte_errno = err; + return NULL; +} + +/** + * DPDK callback to register a PCI device. + * + * This function spawns an Ethernet device out of a given PCI device. + * + * @param[in] pci_drv + * PCI driver structure (mlx5_driver). + * @param[in] pci_dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct ibv_device **ibv_list; + struct rte_eth_dev *eth_dev = NULL; + int vf; + int ret; + + assert(pci_drv == &mlx5_driver); + errno = 0; + ibv_list = mlx5_glue->get_device_list(&ret); + if (!ibv_list) { + rte_errno = errno ? errno : ENOSYS; + DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } - return 0; + while (ret-- > 0) { + struct rte_pci_addr pci_addr; + + DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name); + if (mlx5_ibv_device_to_pci_addr(ibv_list[ret], &pci_addr)) + continue; + if (pci_dev->addr.domain != pci_addr.domain || + pci_dev->addr.bus != pci_addr.bus || + pci_dev->addr.devid != pci_addr.devid || + pci_dev->addr.function != pci_addr.function) + continue; + DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + ibv_list[ret]->name); + break; + } + switch (pci_dev->id.device_id) { + case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF: + vf = 1; + break; + default: + vf = 0; + } + if (ret >= 0) + eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + mlx5_glue->free_device_list(ibv_list); + if (!ret) { + DRV_LOG(WARNING, + "no Verbs device matches PCI device " PCI_PRI_FMT "," + " are kernel drivers loaded?", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function); + rte_errno = ENOENT; + ret = -rte_errno; + } else if (!eth_dev) { + DRV_LOG(ERR, + "probe of PCI device " PCI_PRI_FMT " aborted after" + " encountering an error: %s", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function, + strerror(rte_errno)); + ret = -rte_errno; + } else { + rte_eth_copy_pci_info(eth_dev, pci_dev); + rte_eth_dev_probing_finish(eth_dev); + ret = 0; + } + return ret; } static const struct rte_pci_id mlx5_pci_id_map[] = {