From patchwork Tue Jul 10 16:04:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42736 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 369821B492; Tue, 10 Jul 2018 18:05:21 +0200 (CEST) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 3ED8F1B451 for ; Tue, 10 Jul 2018 18:05:13 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id w16-v6so25076292wmc.2 for ; Tue, 10 Jul 2018 09:05:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=OmHvo6RLnUaNd4Sjyh3IZbSD0DPZ4X0BTUeC/oEEqTCmtsaLdPH755/0hJ0DsUVer0 ZLKW1yeq77i1kFUHkFQ/pQyYnXEu+l0PlEhEmt4qiq5BZuLdeTriGmCRo0ma2OM+dL5z W0xMItesVVe21JyMtrniEFy7HWbpe6T2M6AxoKvVDjFA4Gqm9NRvSKs0RbU37ZkUpyb2 K94kxn8rYeVP7DUm76Xc6ltl6bQTOSka9MFgxDpgeVN43DsnJsYSR7EW+kMTjrDbDjHO avVEVxSzXrYExvuVMTAu3gIUy8aemAkXPLaalyoIDVCZbvKcrsfToW1Z4AIs7feBzhgT gozQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=DYUvg+KvseV1JBBz+0Zr/k/GVpYs06qM8L+3d2+IkJJd2N6NODbAiQlrNsJEPJZFKo HoPuhbVzA4+mbK5RMTybrv/3RDIGLif5enpqJzTI/P80OYuvmifWnZFmFbUFz00qS4zw OLJwjK+kyynIYsTKGvWB03TNtuhkKIs/GkP+s71hLjiSvPxAry0Lfj7QpwWEJt0+dazN V86nSzDWVcOYBVE/cKZLNpASPL5pGHHkEDuOHdepfYmHlDp9fVDI53G445EAq+BOQDKF 2CBE7GHdU6SjCmwWSu8IyuvVuS13FCykJiwSDtUQPqIY6jI+I/ibIoMkXhdplELF8fcX MJzQ== X-Gm-Message-State: APt69E2uk/xpcKMxqAFKaWCoGFoqtr9RS6H4W40ySA8dLcHkRWG9m2Rt rtNv5JxJljPA7vAyOIhrPPKriA== X-Google-Smtp-Source: AAOMgpco+PC6wh315ozjEhpr4Zm02//xY99hHEfPr3+drSZA2F58NqaiH130mI+0QyssUl/rTT6G4Q== X-Received: by 2002:a1c:a341:: with SMTP id m62-v6mr17216787wme.3.1531238712859; Tue, 10 Jul 2018 09:05:12 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id r194-v6sm28424052wmd.36.2018.07.10.09.05.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Jul 2018 09:05:12 -0700 (PDT) Date: Tue, 10 Jul 2018 18:04:56 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180710155743.14448-9-adrien.mazarguil@6wind.com> References: <20180705083934.5535-1-adrien.mazarguil@6wind.com> <20180710155743.14448-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180710155743.14448-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v5 08/10] net/mlx5: probe port representors in natural order X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Port representors are probed in whatever unspecified order ibv_get_device_list() returns them. This is counterintuitive to users since DPDK port IDs assignment almost never follows the same sequence as representor IDs. Additionally, the master device does not necessarily inherit the lowest DPDK port ID. Signed-off-by: Adrien Mazarguil --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 95 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c02afbb82..6592480bf 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1168,6 +1168,52 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, return NULL; } +/** Data associated with devices to spawn. */ +struct mlx5_dev_spawn_data { + unsigned int ifindex; /**< Network interface index. */ + struct mlx5_switch_info info; /**< Switch information. */ + struct ibv_device *ibv_dev; /**< Associated IB device. */ + struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ +}; + +/** + * Comparison callback to sort device data. + * + * This is meant to be used with qsort(). + * + * @param a[in] + * Pointer to pointer to first data object. + * @param b[in] + * Pointer to pointer to second data object. + * + * @return + * 0 if both objects are equal, less than 0 if the first argument is less + * than the second, greater than 0 otherwise. + */ +static int +mlx5_dev_spawn_data_cmp(const void *a, const void *b) +{ + const struct mlx5_switch_info *si_a = + &((const struct mlx5_dev_spawn_data *)a)->info; + const struct mlx5_switch_info *si_b = + &((const struct mlx5_dev_spawn_data *)b)->info; + int ret; + + /* Master device first. */ + ret = si_b->master - si_a->master; + if (ret) + return ret; + /* Then representor devices. */ + ret = si_b->representor - si_a->representor; + if (ret) + return ret; + /* Unidentified devices come last in no specific order. */ + if (!si_a->representor) + return 0; + /* Order representors by name. */ + return si_a->port_name - si_b->port_name; +} + /** * DPDK callback to register a PCI device. * @@ -1218,9 +1264,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } ibv_match[n] = NULL; - unsigned int ifindex[n]; - struct mlx5_switch_info info[n]; - struct rte_eth_dev *eth_list[n]; + struct mlx5_dev_spawn_data list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; @@ -1242,16 +1286,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * bail out. */ for (i = 0; i != n; ++i) { + list[i].ibv_dev = ibv_match[i]; + list[i].eth_dev = NULL; if (nl_rdma < 0) - ifindex[i] = 0; + list[i].ifindex = 0; else - ifindex[i] = mlx5_nl_ifindex(nl_rdma, - ibv_match[i]->name); + list[i].ifindex = mlx5_nl_ifindex + (nl_rdma, list[i].ibv_dev->name); if (nl_route < 0 || - !ifindex[i] || - mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { - ifindex[i] = 0; - memset(&info[i], 0, sizeof(info[i])); + !list[i].ifindex || + mlx5_nl_switch_info(nl_route, list[i].ifindex, + &list[i].info)) { + list[i].ifindex = 0; + memset(&list[i].info, 0, sizeof(list[i].info)); continue; } } @@ -1261,7 +1308,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_route); /* Count unidentified devices. */ for (u = 0, i = 0; i != n; ++i) - if (!info[i].master && !info[i].representor) + if (!list[i].info.master && !list[i].info.representor) ++u; if (u) { if (n == 1 && u == 1) { @@ -1275,6 +1322,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, n = 0; } } + /* + * Sort list to probe devices in natural order for users convenience + * (i.e. master first, then representors from lowest to highest ID). + */ + if (n) + qsort(list, n, sizeof(*list), mlx5_dev_spawn_data_cmp); switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1288,15 +1341,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, for (i = 0; i != n; ++i) { uint32_t restore; - eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], - vf, &info[i]); - if (!eth_list[i]) + list[i].eth_dev = mlx5_dev_spawn + (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); + if (!list[i].eth_dev) break; - restore = eth_list[i]->data->dev_flags; - rte_eth_copy_pci_info(eth_list[i], pci_dev); + restore = list[i].eth_dev->data->dev_flags; + rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ - eth_list[i]->data->dev_flags |= restore; - rte_eth_dev_probing_finish(eth_list[i]); + list[i].eth_dev->data->dev_flags |= restore; + rte_eth_dev_probing_finish(list[i].eth_dev); } mlx5_glue->free_device_list(ibv_list); if (!n) { @@ -1317,10 +1370,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { - mlx5_dev_close(eth_list[i]); + mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_free(eth_list[i]->data->dev_private); - claim_zero(rte_eth_dev_release_port(eth_list[i])); + rte_free(list[i].eth_dev->data->dev_private); + claim_zero(rte_eth_dev_release_port(list[i].eth_dev)); } /* Restore original error. */ rte_errno = -ret;