From patchwork Wed Jul 4 17:27:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42263 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DD0DB1BF79; Wed, 4 Jul 2018 19:28:11 +0200 (CEST) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id AA5B91BF50 for ; Wed, 4 Jul 2018 19:28:05 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id n17-v6so7036383wmh.2 for ; Wed, 04 Jul 2018 10:28:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=/jMy1nFWflphL4wCACofjdcj1fA/62A4fxNekaSHXhY=; b=HMMfKepLcpFIC4Wws8Du+pGLdX6xx1LzTj7xerIyEgPhTLFyiqKTGfNQmctHdOP7Ag byMRTJ9OlhMi0wKUG1cXu08MsMp4a7FpE99F3m2LdycVkzOvmzDImHYs+BjQ6HHRe/8H R555lCpV2DRLOSKfp/LLeG9obrqPwc2EXzdKo+YwL+vET12HKFU1q7A0MgDEF6YkRJCX eZAmUrr27yh71t5m9Bpfu7YXnb5w253igTzMGKAi7uNZoTBnE8mUwvNq+Uhd/G6qebjd DsweFRbxypc6AUYHuVLUBlwr8wMwNF1FFX2tQhx/KbkUdhuoVFFYqc0ukR0MYQ+uBhck KyZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=/jMy1nFWflphL4wCACofjdcj1fA/62A4fxNekaSHXhY=; b=E3cAOBmRciZbDjydG4uodb9YPrh8dUV2ebisviCpW6mH5cOT08YkC3vJzsqGj3/ph7 heD9codw+wDhA2KI8VKpRaW+L9gnK6vtECcssjCNhhCfLLjcIg0VBuc1ugpKY2FAeBAH VUoY4M6XWhX5eGHBcDCnmqHo5esbEeAkSsAIiC5AMMWyJwgdZmYMHDPj3AeA5gV+NbCW XjSLAT4apoIlff3PyBh4+8enQIawg5BXMsYZdPWQ6Ypbtcs0IQwCqzK7U+YgQku9hl65 lOhwKOxmw1e4U/QRn9B1/jcT5vmkxsJsc7BoqZ9/SW+mpUzGZ8/ygISPoy9l1xJ28cot UXTg== X-Gm-Message-State: APt69E3rS6O/8js/4uIVmjJnpf0Y/xQYDCKtSPDlywXtLh7vyM1zKzBq AH3cUrG4DXZc+VfrkmJayEPO1A== X-Google-Smtp-Source: AAOMgpcPHVEmj8ytodEAa1mFE78fZmmZkLO5Er5vwyyADojvTUwR4C4EkEP1/vPCw3X/JOdg5rqKVQ== X-Received: by 2002:adf:9aeb:: with SMTP id a98-v6mr2169144wrc.110.1530725285099; Wed, 04 Jul 2018 10:28:05 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id i4-v6sm7013151wmf.4.2018.07.04.10.28.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:28:04 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:48 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-9-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 08/10] net/mlx5: probe port representors in natural order X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Port representors are probed in whatever unspecified order ibv_get_device_list() returns them. This is counterintuitive to users since DPDK port IDs assignment almost never follows the same sequence as representor IDs. Additionally, the master device does not necessarily inherit the lowest DPDK port ID. Signed-off-by: Adrien Mazarguil --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 95 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 52bfc5c63..05dd3bbe7 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1143,6 +1143,52 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, return NULL; } +/** Data associated with devices to spawn. */ +struct mlx5_dev_spawn_data { + unsigned int ifindex; /**< Network interface index. */ + struct mlx5_switch_info info; /**< Switch information. */ + struct ibv_device *ibv_dev; /**< Associated IB device. */ + struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ +}; + +/** + * Comparison callback to sort device data. + * + * This is meant to be used with qsort(). + * + * @param a[in] + * Pointer to pointer to first data object. + * @param b[in] + * Pointer to pointer to second data object. + * + * @return + * 0 if both objects are equal, less than 0 if the first argument is less + * than the second, greater than 0 otherwise. + */ +static int +mlx5_dev_spawn_data_cmp(const void *a, const void *b) +{ + const struct mlx5_switch_info *si_a = + &((const struct mlx5_dev_spawn_data *)a)->info; + const struct mlx5_switch_info *si_b = + &((const struct mlx5_dev_spawn_data *)b)->info; + int ret; + + /* Master device first. */ + ret = si_b->master - si_a->master; + if (ret) + return ret; + /* Then representor devices. */ + ret = si_b->representor - si_a->representor; + if (ret) + return ret; + /* Unidentified devices come last in no specific order. */ + if (!si_a->representor) + return 0; + /* Order representors by name. */ + return si_a->port_name - si_b->port_name; +} + /** * DPDK callback to register a PCI device. * @@ -1193,9 +1239,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } ibv_match[n] = NULL; - unsigned int ifindex[n]; - struct mlx5_switch_info info[n]; - struct rte_eth_dev *eth_list[n]; + struct mlx5_dev_spawn_data list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; @@ -1217,16 +1261,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * bail out. */ for (i = 0; i != n; ++i) { + list[i].ibv_dev = ibv_match[i]; + list[i].eth_dev = NULL; if (nl_rdma < 0) - ifindex[i] = 0; + list[i].ifindex = 0; else - ifindex[i] = mlx5_nl_ifindex(nl_rdma, - ibv_match[i]->name); + list[i].ifindex = mlx5_nl_ifindex + (nl_rdma, list[i].ibv_dev->name); if (nl_route < 0 || - !ifindex[i] || - mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { - ifindex[i] = 0; - memset(&info[i], 0, sizeof(info[i])); + !list[i].ifindex || + mlx5_nl_switch_info(nl_route, list[i].ifindex, + &list[i].info)) { + list[i].ifindex = 0; + memset(&list[i].info, 0, sizeof(list[i].info)); continue; } } @@ -1236,7 +1283,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_route); /* Count unidentified devices. */ for (u = 0, i = 0; i != n; ++i) - if (!info[i].master && !info[i].representor) + if (!list[i].info.master && !list[i].info.representor) ++u; if (u) { if (n == 1 && u == 1) { @@ -1250,6 +1297,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, n = 0; } } + /* + * Sort list to probe devices in natural order for users convenience + * (i.e. master first, then representors from lowest to highest ID). + */ + if (n) + qsort(list, n, sizeof(*list), mlx5_dev_spawn_data_cmp); switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1263,15 +1316,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, for (i = 0; i != n; ++i) { uint32_t restore; - eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], - vf, &info[i]); - if (!eth_list[i]) + list[i].eth_dev = mlx5_dev_spawn + (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); + if (!list[i].eth_dev) break; - restore = eth_list[i]->data->dev_flags; - rte_eth_copy_pci_info(eth_list[i], pci_dev); + restore = list[i].eth_dev->data->dev_flags; + rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ - eth_list[i]->data->dev_flags |= restore; - rte_eth_dev_probing_finish(eth_list[i]); + list[i].eth_dev->data->dev_flags |= restore; + rte_eth_dev_probing_finish(list[i].eth_dev); } mlx5_glue->free_device_list(ibv_list); if (!n) { @@ -1292,10 +1345,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { - mlx5_dev_close(eth_list[i]); + mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_free(eth_list[i]->data->dev_private); - claim_zero(rte_eth_dev_release_port(eth_list[i])); + rte_free(list[i].eth_dev->data->dev_private); + claim_zero(rte_eth_dev_release_port(list[i].eth_dev)); } /* Restore original error. */ rte_errno = -ret;