From patchwork Thu Jul 5 08:45:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42309 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2FBE31C3A9; Thu, 5 Jul 2018 10:45:59 +0200 (CEST) Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by dpdk.org (Postfix) with ESMTP id 814281C3A3 for ; Thu, 5 Jul 2018 10:45:56 +0200 (CEST) Received: by mail-wm0-f66.google.com with SMTP id b188-v6so9713149wme.3 for ; Thu, 05 Jul 2018 01:45:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=gTHBmySYHBQHoLvZAoF0Xe2V4bN3rZ/UdTVO1zNxJfCjkumCdkOJlaoK2vb0g3FXRv j5NazypJ2yv1uMFhdeAi6RrxJvNBlfkxAylfMF+L6bp/Wc10IXHQZ8cO/+hFKJ4VfnMy a9HcA1xbgIFT04sPnmDk9MRed7WKh3RpGOEmUffexkPtRttoK9nP9l477VAjJzPb12tR NQvAnLGfX6cNh8H8+IYH1lU5ZR2f//vN5l5Clbgm72i6Es6FLy7+qu7rDpqTjidBbCzA f337cIWqR+PWQoPY1+nSDGAmEtjsSkXIa8B0vYGD+d1aMGaaN0LgM/glI+oLAQJ7O+8W qr9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VXghhL2Z7+VjFqQkx0R/BCV4LUBo98GUAsAKz1Dsfx8=; b=nuK8owX0ZlL+LVo343hc++FUGFoFZaeuSf1/oo5+T4Oigm578P734tPSJbuK4GBEJV z4xBJIcLkhMSjXzkmiLN/mgBdPebRu5VcRAqjAeu7RkJ+9IO+IMS5fmuyEZNnHebx170 mo2D1Y847bWW6oguAKIMYBrYvobz+mGSyN69R4tZHecTDLEW92vraGB8m5hrzcJffYFu 3UwA8DIZ9d50Doc3it24Ls8uXQ+VsnPh+225YL/40fTd6mclKOXflyVEwEPuojdO28HA KyGx+b+qSRkvYTvHsAAkoUGNshaqfZmCkGjXWOtJG+YJMzJpRqDKAndK5/eJID5Guqsq fZag== X-Gm-Message-State: APt69E15aqX0KN+i+hwmW+EYxyCTopr0HRseO6rCWPRFDiJ88tRWu02k xV1DsGq8itnTBWXuVUlfoHBG8A== X-Google-Smtp-Source: AAOMgpc9v+juxXacGpW533+Jaa6K3EW747X1SDJ04m8spU45EkIdAz3I5xM1Klayj7Oc9UInCg4jMA== X-Received: by 2002:adf:fc47:: with SMTP id e7-v6mr3765255wrs.157.1530780356289; Thu, 05 Jul 2018 01:45:56 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 77-v6sm5031472wmq.22.2018.07.05.01.45.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Jul 2018 01:45:55 -0700 (PDT) Date: Thu, 5 Jul 2018 10:45:39 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180705083934.5535-9-adrien.mazarguil@6wind.com> References: <20180704172322.22571-1-adrien.mazarguil@6wind.com> <20180705083934.5535-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180705083934.5535-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 08/10] net/mlx5: probe port representors in natural order X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Port representors are probed in whatever unspecified order ibv_get_device_list() returns them. This is counterintuitive to users since DPDK port IDs assignment almost never follows the same sequence as representor IDs. Additionally, the master device does not necessarily inherit the lowest DPDK port ID. Signed-off-by: Adrien Mazarguil --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 95 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c02afbb82..6592480bf 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1168,6 +1168,52 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, return NULL; } +/** Data associated with devices to spawn. */ +struct mlx5_dev_spawn_data { + unsigned int ifindex; /**< Network interface index. */ + struct mlx5_switch_info info; /**< Switch information. */ + struct ibv_device *ibv_dev; /**< Associated IB device. */ + struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ +}; + +/** + * Comparison callback to sort device data. + * + * This is meant to be used with qsort(). + * + * @param a[in] + * Pointer to pointer to first data object. + * @param b[in] + * Pointer to pointer to second data object. + * + * @return + * 0 if both objects are equal, less than 0 if the first argument is less + * than the second, greater than 0 otherwise. + */ +static int +mlx5_dev_spawn_data_cmp(const void *a, const void *b) +{ + const struct mlx5_switch_info *si_a = + &((const struct mlx5_dev_spawn_data *)a)->info; + const struct mlx5_switch_info *si_b = + &((const struct mlx5_dev_spawn_data *)b)->info; + int ret; + + /* Master device first. */ + ret = si_b->master - si_a->master; + if (ret) + return ret; + /* Then representor devices. */ + ret = si_b->representor - si_a->representor; + if (ret) + return ret; + /* Unidentified devices come last in no specific order. */ + if (!si_a->representor) + return 0; + /* Order representors by name. */ + return si_a->port_name - si_b->port_name; +} + /** * DPDK callback to register a PCI device. * @@ -1218,9 +1264,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } ibv_match[n] = NULL; - unsigned int ifindex[n]; - struct mlx5_switch_info info[n]; - struct rte_eth_dev *eth_list[n]; + struct mlx5_dev_spawn_data list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; @@ -1242,16 +1286,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * bail out. */ for (i = 0; i != n; ++i) { + list[i].ibv_dev = ibv_match[i]; + list[i].eth_dev = NULL; if (nl_rdma < 0) - ifindex[i] = 0; + list[i].ifindex = 0; else - ifindex[i] = mlx5_nl_ifindex(nl_rdma, - ibv_match[i]->name); + list[i].ifindex = mlx5_nl_ifindex + (nl_rdma, list[i].ibv_dev->name); if (nl_route < 0 || - !ifindex[i] || - mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { - ifindex[i] = 0; - memset(&info[i], 0, sizeof(info[i])); + !list[i].ifindex || + mlx5_nl_switch_info(nl_route, list[i].ifindex, + &list[i].info)) { + list[i].ifindex = 0; + memset(&list[i].info, 0, sizeof(list[i].info)); continue; } } @@ -1261,7 +1308,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_route); /* Count unidentified devices. */ for (u = 0, i = 0; i != n; ++i) - if (!info[i].master && !info[i].representor) + if (!list[i].info.master && !list[i].info.representor) ++u; if (u) { if (n == 1 && u == 1) { @@ -1275,6 +1322,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, n = 0; } } + /* + * Sort list to probe devices in natural order for users convenience + * (i.e. master first, then representors from lowest to highest ID). + */ + if (n) + qsort(list, n, sizeof(*list), mlx5_dev_spawn_data_cmp); switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1288,15 +1341,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, for (i = 0; i != n; ++i) { uint32_t restore; - eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], - vf, &info[i]); - if (!eth_list[i]) + list[i].eth_dev = mlx5_dev_spawn + (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); + if (!list[i].eth_dev) break; - restore = eth_list[i]->data->dev_flags; - rte_eth_copy_pci_info(eth_list[i], pci_dev); + restore = list[i].eth_dev->data->dev_flags; + rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ - eth_list[i]->data->dev_flags |= restore; - rte_eth_dev_probing_finish(eth_list[i]); + list[i].eth_dev->data->dev_flags |= restore; + rte_eth_dev_probing_finish(list[i].eth_dev); } mlx5_glue->free_device_list(ibv_list); if (!n) { @@ -1317,10 +1370,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { - mlx5_dev_close(eth_list[i]); + mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_free(eth_list[i]->data->dev_private); - claim_zero(rte_eth_dev_release_port(eth_list[i])); + rte_free(list[i].eth_dev->data->dev_private); + claim_zero(rte_eth_dev_release_port(list[i].eth_dev)); } /* Restore original error. */ rte_errno = -ret;