From patchwork Mon Sep 27 08:32:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueming Li X-Patchwork-Id: 99746 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8AC12A0547; Mon, 27 Sep 2021 10:33:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6CB4D410E1; Mon, 27 Sep 2021 10:33:33 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2073.outbound.protection.outlook.com [40.107.237.73]) by mails.dpdk.org (Postfix) with ESMTP id 6171E410DB for ; Mon, 27 Sep 2021 10:33:32 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cpCHsLsBKaKfkLs+6+KhJhl6Z1CUfxskhjkON40PkTuin/NE2dyST2PqFx8cX9QhgSHL1yTxkfYGEAK0SrQo59u+DCANVK9x3e+xihHxEZK9org6gzkpsuaSU3D57y8buw4wM4LO+jOVtzNYXyMMF69b9CC5DXAhgnGTXHJPGEAxA/yMe6zQ2cfdFbSJJKnDF/aNviLucYYB1MMPMoQkNFfzmhurPmuyMT/qQ58XUyPZpoZzWYUXEF+iD4kzyCWHXON950R7Y5qpjMor5VsUVi8iBlsQUjC3Ibc6FCSBVJj7ntUfJx9jneihphDWjmtqJ0gMbYpWI3MYB9OWyZTNtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=3q/v4aCleq220Zjr7s/w9n57+b4IzlXPqPa7Wwxmu0Y=; b=HvzdyZ4v66owilO8KIeFYB7KCcgAjvFdModQ3OIKSEH8lWbKzg/exjcDQ1GOuR6zckVvXygqFAsJQLWsVcZ14qcM+zP+UhlARd/V+DYcUrF1n4pXEmF/jJOBpK22clBvK3vDN4ThHDf3iVI7jxcUyvouwQQYDV5gdDtjv/ck5fImKyncbBghQWscISJl8yp3MXVlLg6vi5gdWMrjIeQ/DDD8d6it3Fm9AhvF+24zHZPZ3hd+L3NfY+gvxLSKATVSSmoEw2Mu8kKyve2Fl+u5+vn2usppGmBkG3hQEgLizq7uCGvyDDVcwF6vHzF5E7LjWSQzk+zex8le5yNb6cXuQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3q/v4aCleq220Zjr7s/w9n57+b4IzlXPqPa7Wwxmu0Y=; b=gndv6gI6Hkui0fXrcY1xZFvLke8YU6rP8xKGLxGvcdIugGxM/XL3WpHQ+zRy2MSCJQG9QM4PkLCidnoRXEb1r7FsCOcctrxfFTVWO/IxiDiD8D2kSM9SZIyGMnr9tDpjkv4fndA4cD6XOQ234ZKWhY7LrtHTCPFx0H2fPzoB1sHUvzOA8fMvCgSJ6zlpC5Xi4eZ3Y84elHSwLx6Pj9UI6Szq1V0xVmR96oq34zVdj6Oe0/WGdnxVCPRxkd2HvFiH5jqGAukHgSdaLZPUnCNIhQ59dEXrYC1QKOmrQS3YS7YGUf2ojipomDwMkL+kXYRWDzMYYagSD+/sE2LXNU6aFA== Received: from DS7PR03CA0097.namprd03.prod.outlook.com (2603:10b6:5:3b7::12) by DM6PR12MB4299.namprd12.prod.outlook.com (2603:10b6:5:223::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.13; Mon, 27 Sep 2021 08:33:31 +0000 Received: from DM6NAM11FT046.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b7:cafe::44) by DS7PR03CA0097.outlook.office365.com (2603:10b6:5:3b7::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.13 via Frontend Transport; Mon, 27 Sep 2021 08:33:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT046.mail.protection.outlook.com (10.13.172.121) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4544.13 via Frontend Transport; Mon, 27 Sep 2021 08:33:30 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 27 Sep 2021 08:33:30 +0000 Received: from nvidia.com (172.20.187.6) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 27 Sep 2021 08:33:28 +0000 From: Xueming Li To: CC: , Matan Azrad , "Viacheslav Ovsiienko" , Ray Kinsella Date: Mon, 27 Sep 2021 16:32:49 +0800 Message-ID: <20210927083256.337450-2-xuemingl@nvidia.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210927083256.337450-1-xuemingl@nvidia.com> References: <20210927083256.337450-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 61349627-540c-4ed9-6345-08d981917c9b X-MS-TrafficTypeDiagnostic: DM6PR12MB4299: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tkuajfJDwzxLzgSRtdJXJA/i3exkl66tJN5lZsqc0CR5xKSBm+DNUgjCrMQ7eYIqpj9+j1yijb+81DxYdyNhMu1Hv+1lM9sZ/7kgrtA1KWNb+FkOk4vaCnsVoCEHmN32SAM1jqUvs9j6Mge2sawrIvHtsWyrkllG1Y2fIyaKarlMAmRlAvJu7CxVCjfzv5qWG+ttNKBP9ow9/fMddzLzOcAASYHmPKM0wEoHkW3zYun9UowmMZvmBAYD8Z7G/0sMDvo0TDBVz0MdZQil1bJh456k8ocx9SVLZ0wlDmQ5aGntTabXJX6zM40ROtRfXQc9Lw7MXP180Et0dbIECutpmGMek2P53tPZSqzjLC6wxzZOX1VgZfF9RSiyy6Vbc3c5gJHQt0q9pIWhjy2vihgFECVl+mpqCq8w33R8J1ojJsVjwp2tU3e/HWqkM+CEEWh/iQZdOyDozSJBDt7Oy2G4ZJy538GKUYqgymD0IRcolaX1MJL4rr7+PHyyt03kgv0yjvcY+xPUwiD8WX02yQSc7lqUmRblX4mhfacsJC7cfZRQrNXq7qTX8xtnzAB96SPyLXUVjrM4LhzI+FefuGeoplFV2/bTgx37GKsY9mlRNPH6MSiBsV31kwdDMmsvU30L5yihz7+ZLMycJlu/bI8LglPcNwKvpPAfnKwqP5/1dDmLcTWoMiApEb9DdVHQIfcc436WML6EpvqNfXd/0hER3A== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(70206006)(16526019)(47076005)(70586007)(7696005)(36756003)(26005)(5660300002)(186003)(426003)(82310400003)(86362001)(1076003)(336012)(2616005)(4326008)(6666004)(83380400001)(6916009)(54906003)(356005)(2906002)(8936002)(508600001)(316002)(55016002)(6286002)(7636003)(36860700001)(8676002)(36906005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2021 08:33:30.8905 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 61349627-540c-4ed9-6345-08d981917c9b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT046.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4299 Subject: [dpdk-dev] [PATCH 1/8] common/mlx5: add netlink API to get RDMA port state X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Introduce netlink api to get rdma port state. Signed-off-by: Xueming Li --- drivers/common/mlx5/linux/meson.build | 2 + drivers/common/mlx5/linux/mlx5_nl.c | 116 +++++++++++++++++++------- drivers/common/mlx5/linux/mlx5_nl.h | 3 + drivers/common/mlx5/version.map | 1 + 4 files changed, 94 insertions(+), 28 deletions(-) diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index cbea58f557d..2dcd27b7786 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -175,6 +175,8 @@ has_sym_args = [ 'RDMA_NLDEV_ATTR_DEV_NAME' ], [ 'HAVE_RDMA_NLDEV_ATTR_PORT_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_PORT_INDEX' ], + [ 'HAVE_RDMA_NLDEV_ATTR_PORT_STATE', 'rdma/rdma_netlink.h', + 'RDMA_NLDEV_ATTR_PORT_STATE' ], [ 'HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_NDEV_INDEX' ], [ 'HAVE_MLX5_DR_FLOW_DUMP', 'infiniband/mlx5dv.h', diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index 9120a697fd5..3e775e58b14 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -78,6 +78,9 @@ #ifndef HAVE_RDMA_NLDEV_ATTR_PORT_INDEX #define RDMA_NLDEV_ATTR_PORT_INDEX 3 #endif +#ifndef HAVE_RDMA_NLDEV_ATTR_PORT_STATE +#define RDMA_NLDEV_ATTR_PORT_STATE 12 +#endif #ifndef HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX #define RDMA_NLDEV_ATTR_NDEV_INDEX 50 #endif @@ -160,14 +163,16 @@ struct mlx5_nl_mac_addr { #define MLX5_NL_CMD_GET_IB_INDEX (1 << 1) #define MLX5_NL_CMD_GET_NET_INDEX (1 << 2) #define MLX5_NL_CMD_GET_PORT_INDEX (1 << 3) +#define MLX5_NL_CMD_GET_PORT_STATE (1 << 4) /** Data structure used by mlx5_nl_cmdget_cb(). */ -struct mlx5_nl_ifindex_data { +struct mlx5_nl_port_info { const char *name; /**< IB device name (in). */ uint32_t flags; /**< found attribute flags (out). */ uint32_t ibindex; /**< IB device index (out). */ uint32_t ifindex; /**< Network interface index (out). */ uint32_t portnum; /**< IB device max port number (out). */ + uint16_t state; /**< IB device port state (out). */ }; uint32_t atomic_sn; @@ -966,8 +971,8 @@ mlx5_nl_allmulti(int nlsk_fd, unsigned int iface_idx, int enable) static int mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) { - struct mlx5_nl_ifindex_data *data = arg; - struct mlx5_nl_ifindex_data local = { + struct mlx5_nl_port_info *data = arg; + struct mlx5_nl_port_info local = { .flags = 0, }; size_t off = NLMSG_HDRLEN; @@ -1000,6 +1005,10 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) local.portnum = *(uint32_t *)payload; local.flags |= MLX5_NL_CMD_GET_PORT_INDEX; break; + case RDMA_NLDEV_ATTR_PORT_STATE: + local.state = *(uint8_t *)payload; + local.flags |= MLX5_NL_CMD_GET_PORT_STATE; + break; default: break; } @@ -1016,6 +1025,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) data->ibindex = local.ibindex; data->ifindex = local.ifindex; data->portnum = local.portnum; + data->state = local.state; } return 0; error: @@ -1024,7 +1034,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) } /** - * Get index of network interface associated with some IB device. + * Get port info of network interface associated with some IB device. * * This is the only somewhat safe method to avoid resorting to heuristics * when faced with port representors. Unfortunately it requires at least @@ -1036,23 +1046,19 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) * IB device name. * @param[in] pindex * IB device port index, starting from 1 + * @param[out] data + * Pointer to port info. * @return - * A valid (nonzero) interface index on success, 0 otherwise and rte_errno - * is set. + * A nonzero value on success, 0 otherwise and rte_errno is set. */ -unsigned int -mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +static int +mlx5_nl_port_info(int nl, const char *name, uint32_t pindex, + struct mlx5_nl_port_info *data) { - struct mlx5_nl_ifindex_data data = { - .name = name, - .flags = 0, - .ibindex = 0, /* Determined during first pass. */ - .ifindex = 0, /* Determined during second pass. */ - }; union { struct nlmsghdr nh; uint8_t buf[NLMSG_HDRLEN + - NLA_HDRLEN + NLA_ALIGN(sizeof(data.ibindex)) + + NLA_HDRLEN + NLA_ALIGN(sizeof(data->ibindex)) + NLA_HDRLEN + NLA_ALIGN(sizeof(pindex))]; } req = { .nh = { @@ -1066,26 +1072,27 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) uint32_t sn = MLX5_NL_SN_GENERATE; int ret; + data->name = name; ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX)) + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX)) goto error; - data.flags = 0; + data->flags = 0; sn = MLX5_NL_SN_GENERATE; req.nh.nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_PORT_GET); req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK; req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(req.buf) - NLMSG_HDRLEN); na = (void *)((uintptr_t)req.buf + NLMSG_HDRLEN); - na->nla_len = NLA_HDRLEN + sizeof(data.ibindex); + na->nla_len = NLA_HDRLEN + sizeof(data->ibindex); na->nla_type = RDMA_NLDEV_ATTR_DEV_INDEX; memcpy((void *)((uintptr_t)na + NLA_HDRLEN), - &data.ibindex, sizeof(data.ibindex)); + &data->ibindex, sizeof(data->ibindex)); na = (void *)((uintptr_t)na + NLA_ALIGN(na->nla_len)); na->nla_len = NLA_HDRLEN + sizeof(pindex); na->nla_type = RDMA_NLDEV_ATTR_PORT_INDEX; @@ -1094,20 +1101,73 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX) || - !(data.flags & MLX5_NL_CMD_GET_NET_INDEX) || - !data.ifindex) + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX) || + !(data->flags & MLX5_NL_CMD_GET_NET_INDEX) || + !data->ifindex) goto error; - return data.ifindex; + return 1; error: rte_errno = ENODEV; return 0; } +/** + * Get index of network interface associated with some IB device. + * + * This is the only somewhat safe method to avoid resorting to heuristics + * when faced with port representors. Unfortunately it requires at least + * Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * A valid (nonzero) interface index on success, 0 otherwise and rte_errno + * is set. + */ +unsigned int +mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { .ifindex = 0 }; + + if (mlx5_nl_port_info(nl, name, pindex, &data) == 0) + return 0; + return data.ifindex; +} + +/** + * Get IB device port state. + * + * This is the only somewhat safe method to get info for port number >= 255. + * Unfortunately it requires at least Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * Port state (ibv_port_state) on success, 0 otherwise + * and rte_errno is set. + */ +enum ibv_port_state +mlx5_nl_port_state(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { .state = 0 }; + + if (mlx5_nl_port_info(nl, name, pindex, &data) == 0) + return (enum ibv_port_state)0; + return (enum ibv_port_state)data.state; +} + /** * Get the number of physical ports of given IB device. * @@ -1123,7 +1183,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) unsigned int mlx5_nl_portnum(int nl, const char *name) { - struct mlx5_nl_ifindex_data data = { + struct mlx5_nl_port_info data = { .flags = 0, .name = name, .ifindex = 0, diff --git a/drivers/common/mlx5/linux/mlx5_nl.h b/drivers/common/mlx5/linux/mlx5_nl.h index 15129ffdc88..809639947a6 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.h +++ b/drivers/common/mlx5/linux/mlx5_nl.h @@ -54,6 +54,9 @@ unsigned int mlx5_nl_portnum(int nl, const char *name); __rte_internal unsigned int mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex); __rte_internal +enum ibv_port_state mlx5_nl_port_state(int nl, const char *name, + uint32_t pindex); +__rte_internal int mlx5_nl_vf_mac_addr_modify(int nlsk_fd, unsigned int iface_idx, struct rte_ether_addr *mac, int vf_index); __rte_internal diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index e5cb6b70604..d23634eef8a 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -131,6 +131,7 @@ INTERNAL { mlx5_nl_mac_addr_flush; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_remove; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_sync; # WINDOWS_NO_EXPORT + mlx5_nl_port_state; mlx5_nl_portnum; # WINDOWS_NO_EXPORT mlx5_nl_promisc; # WINDOWS_NO_EXPORT mlx5_nl_switch_info; # WINDOWS_NO_EXPORT