From patchwork Wed Sep 19 06:48:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongseok Koh X-Patchwork-Id: 44888 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0CD364F91; Wed, 19 Sep 2018 08:48:40 +0200 (CEST) Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50076.outbound.protection.outlook.com [40.107.5.76]) by dpdk.org (Postfix) with ESMTP id 0F7714CA0 for ; Wed, 19 Sep 2018 08:48:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=udYX4yw8cuLRqG3sDAevxHCKaY1loBaz/rnn0Nbf9nQ=; b=ciFDATazT4MBEFSsW+feeITqHsqS1YA5BRNLoU4Rh7Q7aBppcMQNuNV+Rxrq/uR4WrGbsgV60nXGq+nIMl3s4xtZhlHXQRXHo6dYso8g3F0TbBzAj6cVW6aC/Mrfv1ayuTOzQkxQiOs1QQQV5DB0VYSAF2Ny28Nn4SEtwZa7J5w= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB4010.eurprd05.prod.outlook.com (52.134.72.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1143.18; Wed, 19 Sep 2018 06:48:35 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::452:cfe7:8363:61c1]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::452:cfe7:8363:61c1%2]) with mapi id 15.20.1143.017; Wed, 19 Sep 2018 06:48:35 +0000 From: Yongseok Koh To: Shahaf Shuler CC: "dev@dpdk.org" , Ori Kam Thread-Topic: [PATCH 03/11] net/mlx5: add flow translate function Thread-Index: AQHUT+TJ1hkqsq+n7EGVX+YUrPDFgw== Date: Wed, 19 Sep 2018 06:48:35 +0000 Message-ID: <20180919064814.21645-4-yskoh@mellanox.com> References: <20180919064814.21645-1-yskoh@mellanox.com> In-Reply-To: <20180919064814.21645-1-yskoh@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: CY4PR19CA0042.namprd19.prod.outlook.com (2603:10b6:903:103::28) To DB3PR0502MB3980.eurprd05.prod.outlook.com (2603:10a6:8:10::27) authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [209.116.155.178] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB3PR0502MB4010; 6:ZNTDWWSx+o0myFuWAzwsczgvEuu8cFGjBtDq/RElmaG4lTGm0rsqAd+OoXslCSERnw//ywGxUmVkNEWMx4+5FLxvxu90ojq23GXCucczTkgGldZHCgMzpxNn7NKDhlpok6aY/DEgFQ/4xTW0arTxmSQKK2eoH7FycZ/58ymc8Ldyf7P8x+dzsDkv7QsCWMatvNRbJjyn/KQfv30Em1XS5e3KbEHDtjgzVpjeWlurUYW/EPfuSHvYu2eqK9onhAhImABLHP1GgsrRERR8k6gViY3n8KGhAd6k/FW7YIxk2IJMTR25tyFBraecE7UC2zKk4XI76WOBl6vhu33g0UJrWyr4+nMwfpZQLDTUa/NPL3jFi6+d9SNdWvOcfHX5F9QH6ussSQYNQLSGIfDMhOQFuP4i8/L3gggPNbt5Wmr1Fmh1sVMK2JU63bbUJPMuxMikKJPCdhf4vO/1Cj5sZWXuLQ==; 5:5nCWwbRLMq5gnORfD1WcX8gDWAO3adjE+QWZ8FGJNk7PHrjM+TbRiADEi4RP1llP9oZ2hBbic1v29tS6e8MYJ/TNiiu8JqzFr27chBXhq/ClaM0chg2g7bxbPY2NgrS5J34mM6C2ne3GAlwDTZCoVXZfWvcCoeJn1R35sGt4lWw=; 7:ylI3xRBy+DOAEzipVru+8BEIy1iA0J/0KE/tE4760DA4dqfuqJcU8ci5av7ghBhRC/ZpW3TvMV2M7FXQHG0IKP3ztQo/SInRJBsE7ImJmEKmuusc4EJqxze7FLmzseDADfisSIgriJTb0AYV+zjs1XK204HtnArzkvXfGdsZv0S4ilaEAREOZLk/pVI+NTILrPQTFyU2pTadtece/E8pC6g+NUEQAoip2/Y3/4GZxRY1PvFJwgyXCTv3TtwQpoN4 x-ms-office365-filtering-correlation-id: 0ed233c8-767f-46bb-30bf-08d61dfbeb92 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB3PR0502MB4010; x-ms-traffictypediagnostic: DB3PR0502MB4010: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(788757137089)(269456686620040)(211171220733660); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(93001095)(3231355)(944501410)(52105095)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123558120)(20161123562045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699050); SRVR:DB3PR0502MB4010; BCL:0; PCL:0; RULEID:; SRVR:DB3PR0502MB4010; x-forefront-prvs: 0800C0C167 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(376002)(39860400002)(346002)(366004)(136003)(189003)(199004)(51234002)(5250100002)(6512007)(107886003)(6862004)(54906003)(97736004)(6636002)(478600001)(8936002)(8676002)(76176011)(14454004)(36756003)(4326008)(26005)(37006003)(25786009)(53936002)(5660300001)(68736007)(52116002)(99286004)(6506007)(86362001)(386003)(6116002)(7736002)(81156014)(305945005)(81166006)(106356001)(316002)(446003)(2900100001)(105586002)(102836004)(66066001)(2616005)(6436002)(11346002)(53946003)(6486002)(16200700003)(476003)(256004)(1076002)(14444005)(2906002)(486006)(3846002)(569006); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4010; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: lNH7Cc6MwbRrEzSIFxSxfi6DKkhr1AelqQde1ZaTI6UGivJPQVlKzj71Y2dtlQkNqQvEr6hcYetVrh0TX3Sf3XJuI6vHTdz2Q4NE4Kzx+hbP8j1/MP/shfFGYYVPtBmyumnB07u4HfcKQzJklZp3jnxV5+xfX8zdagLBgz6w+Sjxllh3+55QgooBoNyKQCqt76CsiE9Qb9Qw22W6RafmEsa58NMES0fG0NDhyiFUNd3YhiSZ7Rt2H041ZF/Hsou69D9DBbSS1IrMQs0CQs7r8EkPlKnrpQGTLbDmy2GPaAy/Ikm5QX2kqi7CzpCK+X7aXLUH5XbLAaJ8U21dWRHA8Dj8JoLG5n9OheacqoBLidE= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ed233c8-767f-46bb-30bf-08d61dfbeb92 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Sep 2018 06:48:35.2201 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4010 Subject: [dpdk-dev] [PATCH 03/11] net/mlx5: add flow translate function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Ori Kam This commit modify the conversion of the input parameters into Verbs spec, in order to support all previous changes. Some of those changes are: removing the use of the parser, storing each flow in its own flow structure. Signed-off-by: Ori Kam Acked-by: Yongseok Koh --- drivers/net/mlx5/mlx5_flow.c | 1624 +++++++++++++++--------------------------- 1 file changed, 580 insertions(+), 1044 deletions(-) diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index bcf240df5..abf6d03ba 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -296,6 +296,7 @@ struct mlx5_flow_verbs { struct mlx5_flow { LIST_ENTRY(mlx5_flow) next; struct rte_flow *flow; /**< Pointer to the main flow. */ + uint32_t layers; /**< Bit-fields that holds the detected layers. */ union { struct mlx5_flow_verbs verbs; /**< Holds the verbs dev-flow. */ }; @@ -316,15 +317,8 @@ struct mlx5_flow_counter { struct rte_flow { TAILQ_ENTRY(rte_flow) next; /**< Pointer to the next flow structure. */ struct rte_flow_attr attributes; /**< User flow attribute. */ - uint32_t layers; + uint32_t layers; /**< Bit-fields that holds the detected layers. */ /**< Bit-fields of present layers see MLX5_FLOW_LAYER_*. */ - uint32_t modifier; - /**< Bit-fields of present modifier see MLX5_FLOW_MOD_*. */ - uint32_t fate; - /**< Bit-fields of present fate see MLX5_FLOW_FATE_*. */ - LIST_HEAD(verbs, mlx5_flow_verbs) verbs; /**< Verbs flows list. */ - struct mlx5_flow_verbs *cur_verbs; - /**< Current Verbs flow structure being filled. */ struct mlx5_flow_counter *counter; /**< Holds Verbs flow counter. */ struct rte_flow_action_rss rss;/**< RSS context. */ uint8_t key[MLX5_RSS_HASH_KEY_LEN]; /**< RSS hash key. */ @@ -332,6 +326,7 @@ struct rte_flow { void *nl_flow; /**< Netlink flow buffer if relevant. */ LIST_HEAD(dev_flows, mlx5_flow) dev_flows; /**< Device flows that are part of the flow. */ + uint32_t actions; /**< Bit-fields which mark all detected actions. */ }; static const struct rte_flow_ops mlx5_flow_ops = { @@ -430,7 +425,7 @@ static struct mlx5_flow_tunnel_info tunnels_info[] = { * Discover the maximum number of priority available. * * @param[in] dev - * Pointer to Ethernet device. + * Pointer to the Ethernet device structure. * * @return * number of supported flow priority on success, a negative errno @@ -497,34 +492,40 @@ mlx5_flow_discover_priorities(struct rte_eth_dev *dev) /** * Adjust flow priority. * - * @param dev - * Pointer to Ethernet device. - * @param flow - * Pointer to an rte flow. + * @param[in] dev + * Pointer to the Ethernet device structure. + * @param[in] priority + * The rule base priority. + * @param[in] subpriority + * The priority based on the items. + * + * @return + * The new priority. */ -static void -mlx5_flow_adjust_priority(struct rte_eth_dev *dev, struct rte_flow *flow) +static uint32_t +mlx5_flow_adjust_priority(struct rte_eth_dev *dev, + int32_t priority, + uint32_t subpriority) { + uint32_t res = 0; struct priv *priv = dev->data->dev_private; - uint32_t priority = flow->attributes.priority; - uint32_t subpriority = flow->cur_verbs->attr->priority; switch (priv->config.flow_prio) { case RTE_DIM(priority_map_3): - priority = priority_map_3[priority][subpriority]; + res = priority_map_3[priority][subpriority]; break; case RTE_DIM(priority_map_5): - priority = priority_map_5[priority][subpriority]; + res = priority_map_5[priority][subpriority]; break; } - flow->cur_verbs->attr->priority = priority; + return res; } /** * Get a flow counter. * * @param[in] dev - * Pointer to Ethernet device. + * Pointer to the Ethernet device structure. * @param[in] shared * Indicate if this counter is shared with other flows. * @param[in] id @@ -595,34 +596,6 @@ mlx5_flow_counter_release(struct mlx5_flow_counter *counter) } /** - * Verify the @p attributes will be correctly understood by the NIC and store - * them in the @p flow if everything is correct. - * - * @param[in] dev - * Pointer to Ethernet device structure. - * @param[in] attributes - * Pointer to flow attributes - * @param[in, out] flow - * Pointer to the rte_flow structure. - * - * @return - * 0 on success. - */ -static int -mlx5_flow_attributes(struct rte_eth_dev *dev, - const struct rte_flow_attr *attributes, - struct rte_flow *flow) -{ - struct priv *priv = dev->data->dev_private; - uint32_t priority_max = priv->config.flow_prio - 1; - - flow->attributes = *attributes; - if (attributes->priority == MLX5_FLOW_PRIO_RSVD) - flow->attributes.priority = priority_max; - return 0; -} - -/** * Verify the @p item specifications (spec, last, mask) are compatible with the * NIC capabilities. * @@ -693,9 +666,9 @@ mlx5_flow_item_acceptable(const struct rte_flow_item *item, * Size in bytes of the specification to copy. */ static void -mlx5_flow_spec_verbs_add(struct rte_flow *flow, void *src, unsigned int size) +mlx5_flow_spec_verbs_add(struct mlx5_flow *flow, void *src, unsigned int size) { - struct mlx5_flow_verbs *verbs = flow->cur_verbs; + struct mlx5_flow_verbs *verbs = &flow->verbs; if (verbs->specs) { void *dst; @@ -710,8 +683,8 @@ mlx5_flow_spec_verbs_add(struct rte_flow *flow, void *src, unsigned int size) /** * Adjust verbs hash fields according to the @p flow information. * - * @param[in, out] flow. - * Pointer to flow structure. + * @param[in] dev_flow. + * Pointer to dev flow structure. * @param[in] tunnel * 1 when the hash field is for a tunnel item. * @param[in] layer_types @@ -720,49 +693,44 @@ mlx5_flow_spec_verbs_add(struct rte_flow *flow, void *src, unsigned int size) * Item hash fields. */ static void -mlx5_flow_verbs_hashfields_adjust(struct rte_flow *flow, +mlx5_flow_verbs_hashfields_adjust(struct mlx5_flow *dev_flow, int tunnel __rte_unused, uint32_t layer_types, uint64_t hash_fields) { #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT + int rss_request_inner = dev_flow->flow->rss.level >= 2; + hash_fields |= (tunnel ? IBV_RX_HASH_INNER : 0); - if (flow->rss.level == 2 && !tunnel) + if (rss_request_inner && !tunnel) hash_fields = 0; - else if (flow->rss.level < 2 && tunnel) + else if (!rss_request_inner && tunnel) hash_fields = 0; #endif - if (!(flow->rss.types & layer_types)) + if (!(dev_flow->flow->rss.types & layer_types)) hash_fields = 0; - flow->cur_verbs->hash_fields |= hash_fields; + dev_flow->verbs.hash_fields |= hash_fields; } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in] item_flags + * Bit field with all detected items. + * @param[in, out] dev_flow + * Pointer to dev_flow structure. */ -static int -mlx5_flow_item_eth(const struct rte_flow_item *item, struct rte_flow *flow, - const size_t flow_size) +static void +flow_verbs_translate_item_eth(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_eth *spec = item->spec; const struct rte_flow_item_eth *mask = item->mask; - const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel = !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); const unsigned int size = sizeof(struct ibv_flow_spec_eth); struct ibv_flow_spec_eth eth = { .type = IBV_FLOW_SPEC_ETH | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -771,10 +739,6 @@ mlx5_flow_item_eth(const struct rte_flow_item *item, struct rte_flow *flow, if (!mask) mask = &rte_flow_item_eth_mask; - flow->layers |= tunnel ? MLX5_FLOW_LAYER_INNER_L2 : - MLX5_FLOW_LAYER_OUTER_L2; - if (size > flow_size) - return size; if (spec) { unsigned int i; @@ -790,14 +754,18 @@ mlx5_flow_item_eth(const struct rte_flow_item *item, struct rte_flow *flow, eth.val.src_mac[i] &= eth.mask.src_mac[i]; } eth.val.ether_type &= eth.mask.ether_type; + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L2; } - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L2; - mlx5_flow_spec_verbs_add(flow, ð, size); - return size; + mlx5_flow_spec_verbs_add(dev_flow, ð, size); + *item_flags |= tunnel ? + MLX5_FLOW_LAYER_INNER_L2 : + MLX5_FLOW_LAYER_OUTER_L2; } /** * Update the VLAN tag in the Verbs Ethernet specification. + * This function assumes that the input is valid and there is space to add + * the requested item. * * @param[in, out] attr * Pointer to Verbs attributes structure. @@ -829,34 +797,26 @@ mlx5_flow_item_vlan_update(struct ibv_flow_attr *attr, } /** - * Convert the @p item into @p flow (or by updating the already present - * Ethernet Verbs) specification after ensuring the NIC will understand and - * process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in, out] item_flags + * Bit mask that holds all detected items. + * @param[in, out] dev_flow + * Pointer to dev_flow structure. */ -static int -mlx5_flow_item_vlan(const struct rte_flow_item *item, struct rte_flow *flow, - const size_t flow_size) +static void +flow_verbs_translate_item_vlan(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_vlan *spec = item->spec; const struct rte_flow_item_vlan *mask = item->mask; unsigned int size = sizeof(struct ibv_flow_spec_eth); - const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel = !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); struct ibv_flow_spec_eth eth = { .type = IBV_FLOW_SPEC_ETH | (tunnel ? IBV_FLOW_SPEC_INNER : 0), .size = size, @@ -874,49 +834,40 @@ mlx5_flow_item_vlan(const struct rte_flow_item *item, struct rte_flow *flow, eth.mask.ether_type = mask->inner_type; eth.val.ether_type &= eth.mask.ether_type; } - if (!(flow->layers & l2m)) { - if (size <= flow_size) { - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L2; - mlx5_flow_spec_verbs_add(flow, ð, size); - } + if (!(*item_flags & l2m)) { + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L2; + mlx5_flow_spec_verbs_add(dev_flow, ð, size); } else { - if (flow->cur_verbs) - mlx5_flow_item_vlan_update(flow->cur_verbs->attr, + mlx5_flow_item_vlan_update(dev_flow->verbs.attr, ð); size = 0; /* Only an update is done in eth specification. */ } - flow->layers |= tunnel ? - (MLX5_FLOW_LAYER_INNER_L2 | MLX5_FLOW_LAYER_INNER_VLAN) : - (MLX5_FLOW_LAYER_OUTER_L2 | MLX5_FLOW_LAYER_OUTER_VLAN); - return size; + *item_flags |= tunnel ? + (MLX5_FLOW_LAYER_INNER_L2 | + MLX5_FLOW_LAYER_INNER_VLAN) : + (MLX5_FLOW_LAYER_OUTER_L2 | MLX5_FLOW_LAYER_OUTER_VLAN); } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_ipv4(const struct rte_flow_item *item, struct rte_flow *flow, - const size_t flow_size) +static void +flow_verbs_translate_item_ipv4(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_ipv4 *spec = item->spec; const struct rte_flow_item_ipv4 *mask = item->mask; - const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel = !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); unsigned int size = sizeof(struct ibv_flow_spec_ipv4_ext); struct ibv_flow_spec_ipv4_ext ipv4 = { .type = IBV_FLOW_SPEC_IPV4_EXT | @@ -926,7 +877,7 @@ mlx5_flow_item_ipv4(const struct rte_flow_item *item, struct rte_flow *flow, if (!mask) mask = &rte_flow_item_ipv4_mask; - flow->layers |= tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 : + *item_flags |= tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV4 : MLX5_FLOW_LAYER_OUTER_L3_IPV4; if (spec) { ipv4.val = (struct ibv_flow_ipv4_ext_filter){ @@ -947,46 +898,37 @@ mlx5_flow_item_ipv4(const struct rte_flow_item *item, struct rte_flow *flow, ipv4.val.proto &= ipv4.mask.proto; ipv4.val.tos &= ipv4.mask.tos; } - if (size <= flow_size) { - mlx5_flow_verbs_hashfields_adjust - (flow, tunnel, - (ETH_RSS_IPV4 | ETH_RSS_FRAG_IPV4 | - ETH_RSS_NONFRAG_IPV4_TCP | - ETH_RSS_NONFRAG_IPV4_UDP | - ETH_RSS_NONFRAG_IPV4_OTHER), - (IBV_RX_HASH_SRC_IPV4 | IBV_RX_HASH_DST_IPV4)); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L3; - mlx5_flow_spec_verbs_add(flow, &ipv4, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, tunnel, + (ETH_RSS_IPV4 | ETH_RSS_FRAG_IPV4 | + ETH_RSS_NONFRAG_IPV4_TCP | + ETH_RSS_NONFRAG_IPV4_UDP | + ETH_RSS_NONFRAG_IPV4_OTHER), + (IBV_RX_HASH_SRC_IPV4 | + IBV_RX_HASH_DST_IPV4)); + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L3; + mlx5_flow_spec_verbs_add(dev_flow, &ipv4, size); } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_ipv6(const struct rte_flow_item *item, struct rte_flow *flow, - const size_t flow_size) +static void +flow_verbs_translate_item_ipv6(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_ipv6 *spec = item->spec; const struct rte_flow_item_ipv6 *mask = item->mask; - const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel = !!(dev_flow->flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int size = sizeof(struct ibv_flow_spec_ipv6); struct ibv_flow_spec_ipv6 ipv6 = { .type = IBV_FLOW_SPEC_IPV6 | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -995,8 +937,8 @@ mlx5_flow_item_ipv6(const struct rte_flow_item *item, struct rte_flow *flow, if (!mask) mask = &rte_flow_item_ipv6_mask; - flow->layers |= tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 : - MLX5_FLOW_LAYER_OUTER_L3_IPV6; + *item_flags |= tunnel ? MLX5_FLOW_LAYER_INNER_L3_IPV6 : + MLX5_FLOW_LAYER_OUTER_L3_IPV6; if (spec) { unsigned int i; uint32_t vtc_flow_val; @@ -1036,46 +978,40 @@ mlx5_flow_item_ipv6(const struct rte_flow_item *item, struct rte_flow *flow, ipv6.val.next_hdr &= ipv6.mask.next_hdr; ipv6.val.hop_limit &= ipv6.mask.hop_limit; } - if (size <= flow_size) { - mlx5_flow_verbs_hashfields_adjust - (flow, tunnel, - (ETH_RSS_IPV6 | ETH_RSS_FRAG_IPV6 | - ETH_RSS_NONFRAG_IPV6_TCP | ETH_RSS_NONFRAG_IPV6_UDP | - ETH_RSS_NONFRAG_IPV6_OTHER | ETH_RSS_IPV6_EX | - ETH_RSS_IPV6_TCP_EX | ETH_RSS_IPV6_UDP_EX), - (IBV_RX_HASH_SRC_IPV6 | IBV_RX_HASH_DST_IPV6)); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L3; - mlx5_flow_spec_verbs_add(flow, &ipv6, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, tunnel, + (ETH_RSS_IPV6 | ETH_RSS_FRAG_IPV6 | + ETH_RSS_NONFRAG_IPV6_TCP | + ETH_RSS_NONFRAG_IPV6_UDP | + ETH_RSS_IPV6_EX | + ETH_RSS_IPV6_TCP_EX | + ETH_RSS_IPV6_UDP_EX | + ETH_RSS_NONFRAG_IPV6_OTHER), + (IBV_RX_HASH_SRC_IPV6 | + IBV_RX_HASH_DST_IPV6)); + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L3; + mlx5_flow_spec_verbs_add(dev_flow, &ipv6, size); } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_udp(const struct rte_flow_item *item, struct rte_flow *flow, - const size_t flow_size) +static void +flow_verbs_translate_item_udp(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_udp *spec = item->spec; const struct rte_flow_item_udp *mask = item->mask; - const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel = !!(*item_flags & MLX5_FLOW_LAYER_TUNNEL); unsigned int size = sizeof(struct ibv_flow_spec_tcp_udp); struct ibv_flow_spec_tcp_udp udp = { .type = IBV_FLOW_SPEC_UDP | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -1084,8 +1020,8 @@ mlx5_flow_item_udp(const struct rte_flow_item *item, struct rte_flow *flow, if (!mask) mask = &rte_flow_item_udp_mask; - flow->layers |= tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP : - MLX5_FLOW_LAYER_OUTER_L4_UDP; + *item_flags |= tunnel ? MLX5_FLOW_LAYER_INNER_L4_UDP : + MLX5_FLOW_LAYER_OUTER_L4_UDP; if (spec) { udp.val.dst_port = spec->hdr.dst_port; udp.val.src_port = spec->hdr.src_port; @@ -1095,44 +1031,34 @@ mlx5_flow_item_udp(const struct rte_flow_item *item, struct rte_flow *flow, udp.val.src_port &= udp.mask.src_port; udp.val.dst_port &= udp.mask.dst_port; } - if (size <= flow_size) { - mlx5_flow_verbs_hashfields_adjust(flow, tunnel, ETH_RSS_UDP, - (IBV_RX_HASH_SRC_PORT_UDP | - IBV_RX_HASH_DST_PORT_UDP)); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L4; - mlx5_flow_spec_verbs_add(flow, &udp, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, + tunnel, ETH_RSS_UDP, + (IBV_RX_HASH_SRC_PORT_UDP | + IBV_RX_HASH_DST_PORT_UDP)); + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L4; + mlx5_flow_spec_verbs_add(dev_flow, &udp, size); } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_tcp(const struct rte_flow_item *item, struct rte_flow *flow, - const size_t flow_size) +static void +flow_verbs_translate_item_tcp(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_tcp *spec = item->spec; const struct rte_flow_item_tcp *mask = item->mask; - const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); + const int tunnel = !!(dev_flow->flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int size = sizeof(struct ibv_flow_spec_tcp_udp); struct ibv_flow_spec_tcp_udp tcp = { .type = IBV_FLOW_SPEC_TCP | (tunnel ? IBV_FLOW_SPEC_INNER : 0), @@ -1141,8 +1067,8 @@ mlx5_flow_item_tcp(const struct rte_flow_item *item, struct rte_flow *flow, if (!mask) mask = &rte_flow_item_tcp_mask; - flow->layers |= tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP : - MLX5_FLOW_LAYER_OUTER_L4_TCP; + *item_flags |= tunnel ? MLX5_FLOW_LAYER_INNER_L4_TCP : + MLX5_FLOW_LAYER_OUTER_L4_TCP; if (spec) { tcp.val.dst_port = spec->hdr.dst_port; tcp.val.src_port = spec->hdr.src_port; @@ -1152,40 +1078,30 @@ mlx5_flow_item_tcp(const struct rte_flow_item *item, struct rte_flow *flow, tcp.val.src_port &= tcp.mask.src_port; tcp.val.dst_port &= tcp.mask.dst_port; } - if (size <= flow_size) { - mlx5_flow_verbs_hashfields_adjust(flow, tunnel, ETH_RSS_TCP, - (IBV_RX_HASH_SRC_PORT_TCP | - IBV_RX_HASH_DST_PORT_TCP)); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L4; - mlx5_flow_spec_verbs_add(flow, &tcp, size); - } - return size; + mlx5_flow_verbs_hashfields_adjust(dev_flow, + tunnel, ETH_RSS_TCP, + (IBV_RX_HASH_SRC_PORT_TCP | + IBV_RX_HASH_DST_PORT_TCP)); + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L4; + mlx5_flow_spec_verbs_add(dev_flow, &tcp, size); } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_vxlan(const struct rte_flow_item *item, struct rte_flow *flow, - const size_t flow_size) +static void +flow_verbs_translate_item_vxlan(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_vxlan *spec = item->spec; const struct rte_flow_item_vxlan *mask = item->mask; @@ -1209,38 +1125,27 @@ mlx5_flow_item_vxlan(const struct rte_flow_item *item, struct rte_flow *flow, /* Remove unwanted bits from values. */ vxlan.val.tunnel_id &= vxlan.mask.tunnel_id; } - if (size <= flow_size) { - mlx5_flow_spec_verbs_add(flow, &vxlan, size); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L2; - } - flow->layers |= MLX5_FLOW_LAYER_VXLAN; - return size; + mlx5_flow_spec_verbs_add(dev_flow, &vxlan, size); + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L2; + *item_flags |= MLX5_FLOW_LAYER_VXLAN; } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_vxlan_gpe(const struct rte_flow_item *item, - struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_item_vxlan_gpe(const struct rte_flow_item *item, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_item_vxlan_gpe *spec = item->spec; const struct rte_flow_item_vxlan_gpe *mask = item->mask; @@ -1264,12 +1169,9 @@ mlx5_flow_item_vxlan_gpe(const struct rte_flow_item *item, /* Remove unwanted bits from values. */ vxlan_gpe.val.tunnel_id &= vxlan_gpe.mask.tunnel_id; } - if (size <= flow_size) { - mlx5_flow_spec_verbs_add(flow, &vxlan_gpe, size); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L2; - } - flow->layers |= MLX5_FLOW_LAYER_VXLAN_GPE; - return size; + mlx5_flow_spec_verbs_add(dev_flow, &vxlan_gpe, size); + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L2; + *item_flags |= MLX5_FLOW_LAYER_VXLAN_GPE; } /** @@ -1325,34 +1227,30 @@ mlx5_flow_item_gre_ip_protocol_update(struct ibv_flow_attr *attr, } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * It will also update the previous L3 layer with the protocol value matching - * the GRE. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p item into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested item + * into the flow. * - * @param dev - * Pointer to Ethernet device. * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_gre(const struct rte_flow_item *item __rte_unused, - struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_item_gre(const struct rte_flow_item *item __rte_unused, + uint64_t *item_flags, + struct mlx5_flow *dev_flow) { - struct mlx5_flow_verbs *verbs = flow->cur_verbs; -#ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT + struct mlx5_flow_verbs *verbs = &dev_flow->verbs; +#ifndef HAVE_IBV_DEVICE_MPLS_SUPPORT + unsigned int size = sizeof(struct ibv_flow_spec_tunnel); + struct ibv_flow_spec_tunnel tunnel = { + .type = IBV_FLOW_SPEC_VXLAN_TUNNEL, + .size = size, + }; +#else const struct rte_flow_item_gre *spec = item->spec; const struct rte_flow_item_gre *mask = item->mask; unsigned int size = sizeof(struct ibv_flow_spec_gre); @@ -1360,15 +1258,7 @@ mlx5_flow_item_gre(const struct rte_flow_item *item __rte_unused, .type = IBV_FLOW_SPEC_GRE, .size = size, }; -#else - unsigned int size = sizeof(struct ibv_flow_spec_tunnel); - struct ibv_flow_spec_tunnel tunnel = { - .type = IBV_FLOW_SPEC_VXLAN_TUNNEL, - .size = size, - }; -#endif -#ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT if (!mask) mask = &rte_flow_item_gre_mask; if (spec) { @@ -1381,51 +1271,36 @@ mlx5_flow_item_gre(const struct rte_flow_item *item __rte_unused, tunnel.val.protocol &= tunnel.mask.protocol; tunnel.val.key &= tunnel.mask.key; } -#else -#endif /* !HAVE_IBV_DEVICE_MPLS_SUPPORT */ - if (size <= flow_size) { - if (flow->layers & MLX5_FLOW_LAYER_OUTER_L3_IPV4) - mlx5_flow_item_gre_ip_protocol_update - (verbs->attr, IBV_FLOW_SPEC_IPV4_EXT, - MLX5_IP_PROTOCOL_GRE); - else - mlx5_flow_item_gre_ip_protocol_update - (verbs->attr, IBV_FLOW_SPEC_IPV6, - MLX5_IP_PROTOCOL_GRE); - mlx5_flow_spec_verbs_add(flow, &tunnel, size); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L2; - } - flow->layers |= MLX5_FLOW_LAYER_GRE; - return size; +#endif + if (*item_flags & MLX5_FLOW_LAYER_OUTER_L3_IPV4) + mlx5_flow_item_gre_ip_protocol_update + (verbs->attr, IBV_FLOW_SPEC_IPV4_EXT, + MLX5_IP_PROTOCOL_GRE); + else + mlx5_flow_item_gre_ip_protocol_update + (verbs->attr, IBV_FLOW_SPEC_IPV6, + MLX5_IP_PROTOCOL_GRE); + mlx5_flow_spec_verbs_add(dev_flow, &tunnel, size); + verbs->attr->priority = MLX5_PRIORITY_MAP_L2; + *item_flags |= MLX5_FLOW_LAYER_GRE; } /** - * Convert the @p item into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p action into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested action + * into the flow. This function also return the action that was added. * * @param[in] item * Item specification. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p item has fully been converted, - * otherwise another call with this returned memory size should be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in, out] item_flags + * Bit mask that marks all detected items. + * @param[in, out] dev_flow + * Pointer to sepacific flow structure. */ -static int -mlx5_flow_item_mpls(const struct rte_flow_item *item __rte_unused, - struct rte_flow *flow __rte_unused, - const size_t flow_size __rte_unused, - struct rte_flow_error *error) +static void +flow_verbs_translate_item_mpls(const struct rte_flow_item *item __rte_unused, + uint64_t *action_flags __rte_unused, + struct mlx5_flow *dev_flow __rte_unused) { #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT const struct rte_flow_item_mpls *spec = item->spec; @@ -1444,135 +1319,25 @@ mlx5_flow_item_mpls(const struct rte_flow_item *item __rte_unused, /* Remove unwanted bits from values. */ mpls.val.label &= mpls.mask.label; } - if (size <= flow_size) { - mlx5_flow_spec_verbs_add(flow, &mpls, size); - flow->cur_verbs->attr->priority = MLX5_PRIORITY_MAP_L2; - } - flow->layers |= MLX5_FLOW_LAYER_MPLS; - return size; -#endif /* !HAVE_IBV_DEVICE_MPLS_SUPPORT */ - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ITEM, - item, - "MPLS is not supported by Verbs, please" - " update."); -} - -/** - * Convert the @p pattern into a Verbs specifications after ensuring the NIC - * will understand and process it correctly. - * The conversion is performed item per item, each of them is written into - * the @p flow if its size is lesser or equal to @p flow_size. - * Validation and memory consumption computation are still performed until the - * end of @p pattern, unless an error is encountered. - * - * @param[in] pattern - * Flow pattern. - * @param[in, out] flow - * Pointer to the rte_flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small some - * garbage may be present. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @pattern has fully been - * converted, otherwise another call with this returned memory size should - * be done. - * On error, a negative errno value is returned and rte_errno is set. - */ -static int -mlx5_flow_items(const struct rte_flow_item pattern[], - struct rte_flow *flow, const size_t flow_size, - struct rte_flow_error *error) -{ - int remain = flow_size; - size_t size = 0; - - for (; pattern->type != RTE_FLOW_ITEM_TYPE_END; pattern++) { - int ret = 0; - - switch (pattern->type) { - case RTE_FLOW_ITEM_TYPE_VOID: - break; - case RTE_FLOW_ITEM_TYPE_ETH: - ret = mlx5_flow_item_eth(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_VLAN: - ret = mlx5_flow_item_vlan(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_IPV4: - ret = mlx5_flow_item_ipv4(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_IPV6: - ret = mlx5_flow_item_ipv6(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_UDP: - ret = mlx5_flow_item_udp(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_TCP: - ret = mlx5_flow_item_tcp(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_VXLAN: - ret = mlx5_flow_item_vxlan(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_VXLAN_GPE: - ret = mlx5_flow_item_vxlan_gpe(pattern, flow, - remain); - break; - case RTE_FLOW_ITEM_TYPE_GRE: - ret = mlx5_flow_item_gre(pattern, flow, remain); - break; - case RTE_FLOW_ITEM_TYPE_MPLS: - ret = mlx5_flow_item_mpls(pattern, flow, remain, error); - break; - default: - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ITEM, - pattern, - "item not supported"); - } - if (ret < 0) - return ret; - if (remain > ret) - remain -= ret; - else - remain = 0; - size += ret; - } - if (!flow->layers) { - const struct rte_flow_item item = { - .type = RTE_FLOW_ITEM_TYPE_ETH, - }; - - return mlx5_flow_item_eth(&item, flow, flow_size); - } - return size; + mlx5_flow_spec_verbs_add(dev_flow, &mpls, size); + dev_flow->verbs.attr->priority = MLX5_PRIORITY_MAP_L2; + *action_flags |= MLX5_FLOW_LAYER_MPLS; +#endif } /** - * Convert the @p action into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. - * - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. + * Convert the @p action into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested action + * into the flow. This function also return the action that was added. * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size should - * be done. - * On error, a negative errno value is returned and rte_errno is set. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_drop(struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_action_drop(uint64_t *action_flags, + struct mlx5_flow *dev_flow) { unsigned int size = sizeof(struct ibv_flow_spec_action_drop); struct ibv_flow_spec_action_drop drop = { @@ -1580,53 +1345,55 @@ mlx5_flow_action_drop(struct rte_flow *flow, const size_t flow_size) .size = size, }; - if (size < flow_size) - mlx5_flow_spec_verbs_add(flow, &drop, size); - flow->fate |= MLX5_FLOW_FATE_DROP; - return size; + mlx5_flow_spec_verbs_add(dev_flow, &drop, size); + *action_flags |= MLX5_ACTION_DROP; } /** - * Convert the @p action into @p flow after ensuring the NIC will understand - * and process it correctly. + * Convert the @p action into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested action + * into the flow. This function also return the action that was added. * * @param[in] action * Action configuration. - * @param[in, out] flow - * Pointer to flow structure. - * - * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_queue(const struct rte_flow_action *action, - struct rte_flow *flow) +static void +flow_verbs_translate_action_queue(const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_action_queue *queue = action->conf; + struct rte_flow *flow = dev_flow->flow; if (flow->queue) (*flow->queue)[0] = queue->index; flow->rss.queue_num = 1; - flow->fate |= MLX5_FLOW_FATE_QUEUE; - return 0; + *action_flags |= MLX5_ACTION_QUEUE; } /** - * Ensure the @p action will be understood and used correctly by the NIC. + * Convert the @p action into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested action + * into the flow. This function also return the action that was added. * * @param[in] action * Action configuration. - * @param flow[in, out] - * Pointer to the rte_flow structure. - * - * @return - * 0 On success. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_rss(const struct rte_flow_action *action, - struct rte_flow *flow) +static void +flow_verbs_translate_action_rss(const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { const struct rte_flow_action_rss *rss = action->conf; + struct rte_flow *flow = dev_flow->flow; if (flow->queue) memcpy((*flow->queue), rss->queue, @@ -1635,30 +1402,26 @@ mlx5_flow_action_rss(const struct rte_flow_action *action, memcpy(flow->key, rss->key, MLX5_RSS_HASH_KEY_LEN); flow->rss.types = rss->types; flow->rss.level = rss->level; - flow->fate |= MLX5_FLOW_FATE_RSS; - return 0; + *action_flags |= MLX5_ACTION_RSS; } /** - * Convert the @p action into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. + * Convert the @p action into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested action + * into the flow. This function also return the action that was added. * - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size should - * be done. + * @param[in] action + * Action configuration. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. */ -static int -mlx5_flow_action_flag(struct rte_flow *flow, const size_t flow_size) +static void +flow_verbs_translate_action_flag + (const struct rte_flow_action *action __rte_unused, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { unsigned int size = sizeof(struct ibv_flow_spec_action_tag); struct ibv_flow_spec_action_tag tag = { @@ -1666,14 +1429,8 @@ mlx5_flow_action_flag(struct rte_flow *flow, const size_t flow_size) .size = size, .tag_id = mlx5_flow_mark_set(MLX5_FLOW_MARK_DEFAULT), }; - struct mlx5_flow_verbs *verbs = flow->cur_verbs; - - if (flow->modifier & MLX5_FLOW_MOD_MARK) - size = 0; - else if (size <= flow_size && verbs) - mlx5_flow_spec_verbs_add(flow, &tag, size); - flow->modifier |= MLX5_FLOW_MOD_FLAG; - return size; + *action_flags |= MLX5_ACTION_MARK; + mlx5_flow_spec_verbs_add(dev_flow, &tag, size); } /** @@ -1685,499 +1442,115 @@ mlx5_flow_action_flag(struct rte_flow *flow, const size_t flow_size) * Mark identifier to replace the flag. */ static void -mlx5_flow_verbs_mark_update(struct mlx5_flow_verbs *verbs, uint32_t mark_id) +flow_verbs_mark_update(struct mlx5_flow_verbs *verbs, uint32_t mark_id) { struct ibv_spec_header *hdr; int i; - if (!verbs) - return; - /* Update Verbs specification. */ - hdr = (struct ibv_spec_header *)verbs->specs; - if (!hdr) - return; - for (i = 0; i != verbs->attr->num_of_specs; ++i) { - if (hdr->type == IBV_FLOW_SPEC_ACTION_TAG) { - struct ibv_flow_spec_action_tag *t = - (struct ibv_flow_spec_action_tag *)hdr; - - t->tag_id = mlx5_flow_mark_set(mark_id); - } - hdr = (struct ibv_spec_header *)((uintptr_t)hdr + hdr->size); - } -} - -/** - * Convert the @p action into @p flow (or by updating the already present - * Flag Verbs specification) after ensuring the NIC will understand and - * process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. - * - * @param[in] action - * Action configuration. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size should - * be done. - */ -static int -mlx5_flow_action_mark(const struct rte_flow_action *action, - struct rte_flow *flow, const size_t flow_size) -{ - const struct rte_flow_action_mark *mark = action->conf; - unsigned int size = sizeof(struct ibv_flow_spec_action_tag); - struct ibv_flow_spec_action_tag tag = { - .type = IBV_FLOW_SPEC_ACTION_TAG, - .size = size, - }; - struct mlx5_flow_verbs *verbs = flow->cur_verbs; - - if (flow->modifier & MLX5_FLOW_MOD_FLAG) { - mlx5_flow_verbs_mark_update(verbs, mark->id); - size = 0; - } else if (size <= flow_size) { - tag.tag_id = mlx5_flow_mark_set(mark->id); - mlx5_flow_spec_verbs_add(flow, &tag, size); - } - flow->modifier |= MLX5_FLOW_MOD_MARK; - return size; -} - -/** - * Convert the @p action into a Verbs specification after ensuring the NIC - * will understand and process it correctly. - * If the necessary size for the conversion is greater than the @p flow_size, - * nothing is written in @p flow, the validation is still performed. - * - * @param action[in] - * Action configuration. - * @param flow[in, out] - * Pointer to flow structure. - * @param flow_size[in] - * Size in bytes of the available space in @p flow, if too small, nothing is - * written. - * @param error[int, out] - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p action has fully been - * converted, otherwise another call with this returned memory size should - * be done. - * On error, a negative errno value is returned and rte_errno is set. - */ -static int -mlx5_flow_action_count(struct rte_eth_dev *dev, - const struct rte_flow_action *action, - struct rte_flow *flow, - const size_t flow_size __rte_unused, - struct rte_flow_error *error) -{ - const struct rte_flow_action_count *count = action->conf; -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - unsigned int size = sizeof(struct ibv_flow_spec_counter_action); - struct ibv_flow_spec_counter_action counter = { - .type = IBV_FLOW_SPEC_ACTION_COUNT, - .size = size, - }; -#endif - - if (!flow->counter) { - flow->counter = mlx5_flow_counter_new(dev, count->shared, - count->id); - if (!flow->counter) - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ACTION, - action, - "cannot get counter" - " context."); - } - flow->modifier |= MLX5_FLOW_MOD_COUNT; -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - counter.counter_set_handle = flow->counter->cs->handle; - if (size <= flow_size) - mlx5_flow_spec_verbs_add(flow, &counter, size); - return size; -#endif - return 0; -} - -/** - * Convert the @p action into @p flow after ensuring the NIC will understand - * and process it correctly. - * The conversion is performed action per action, each of them is written into - * the @p flow if its size is lesser or equal to @p flow_size. - * Validation and memory consumption computation are still performed until the - * end of @p action, unless an error is encountered. - * - * @param[in] dev - * Pointer to Ethernet device structure. - * @param[in] actions - * Pointer to flow actions array. - * @param[in, out] flow - * Pointer to the rte_flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small some - * garbage may be present. - * @param[out] error - * Pointer to error structure. - * - * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the @p actions has fully been - * converted, otherwise another call with this returned memory size should - * be done. - * On error, a negative errno value is returned and rte_errno is set. - */ -static int -mlx5_flow_actions(struct rte_eth_dev *dev, - const struct rte_flow_action actions[], - struct rte_flow *flow, const size_t flow_size, - struct rte_flow_error *error) -{ - size_t size = 0; - int remain = flow_size; - int ret = 0; - - for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { - switch (actions->type) { - case RTE_FLOW_ACTION_TYPE_VOID: - break; - case RTE_FLOW_ACTION_TYPE_FLAG: - ret = mlx5_flow_action_flag(flow, remain); - break; - case RTE_FLOW_ACTION_TYPE_MARK: - ret = mlx5_flow_action_mark(actions, flow, remain); - break; - case RTE_FLOW_ACTION_TYPE_DROP: - ret = mlx5_flow_action_drop(flow, remain); - break; - case RTE_FLOW_ACTION_TYPE_QUEUE: - ret = mlx5_flow_action_queue(actions, flow); - break; - case RTE_FLOW_ACTION_TYPE_RSS: - ret = mlx5_flow_action_rss(actions, flow); - break; - case RTE_FLOW_ACTION_TYPE_COUNT: - ret = mlx5_flow_action_count(dev, actions, flow, remain, - error); - break; - default: - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_ACTION, - actions, - "action not supported"); - } - if (ret < 0) - return ret; - if (remain > ret) - remain -= ret; - else - remain = 0; - size += ret; - } - if (!flow->fate) - return rte_flow_error_set(error, ENOTSUP, - RTE_FLOW_ERROR_TYPE_UNSPECIFIED, - NULL, - "no fate action found"); - return size; -} - -/** - * Validate flow rule and fill flow structure accordingly. - * - * @param dev - * Pointer to Ethernet device. - * @param[out] flow - * Pointer to flow structure. - * @param flow_size - * Size of allocated space for @p flow. - * @param[in] attr - * Flow rule attributes. - * @param[in] pattern - * Pattern specification (list terminated by the END pattern item). - * @param[in] actions - * Associated actions (list terminated by the END action). - * @param[out] error - * Perform verbose error reporting if not NULL. - * - * @return - * A positive value representing the size of the flow object in bytes - * regardless of @p flow_size on success, a negative errno value otherwise - * and rte_errno is set. - */ -static int -mlx5_flow_merge_switch(struct rte_eth_dev *dev, - struct rte_flow *flow, - size_t flow_size, - const struct rte_flow_attr *attr, - const struct rte_flow_item pattern[], - const struct rte_flow_action actions[], - struct rte_flow_error *error) -{ - unsigned int n = mlx5_dev_to_port_id(dev->device, NULL, 0); - uint16_t port_id[!n + n]; - struct mlx5_nl_flow_ptoi ptoi[!n + n + 1]; - size_t off = RTE_ALIGN_CEIL(sizeof(*flow), alignof(max_align_t)); - unsigned int i; - unsigned int own = 0; - int ret; - - /* At least one port is needed when no switch domain is present. */ - if (!n) { - n = 1; - port_id[0] = dev->data->port_id; - } else { - n = RTE_MIN(mlx5_dev_to_port_id(dev->device, port_id, n), n); - } - for (i = 0; i != n; ++i) { - struct rte_eth_dev_info dev_info; - - rte_eth_dev_info_get(port_id[i], &dev_info); - if (port_id[i] == dev->data->port_id) - own = i; - ptoi[i].port_id = port_id[i]; - ptoi[i].ifindex = dev_info.if_index; - } - /* Ensure first entry of ptoi[] is the current device. */ - if (own) { - ptoi[n] = ptoi[0]; - ptoi[0] = ptoi[own]; - ptoi[own] = ptoi[n]; - } - /* An entry with zero ifindex terminates ptoi[]. */ - ptoi[n].port_id = 0; - ptoi[n].ifindex = 0; - if (flow_size < off) - flow_size = 0; - ret = mlx5_nl_flow_transpose((uint8_t *)flow + off, - flow_size ? flow_size - off : 0, - ptoi, attr, pattern, actions, error); - if (ret < 0) - return ret; - if (flow_size) { - *flow = (struct rte_flow){ - .attributes = *attr, - .nl_flow = (uint8_t *)flow + off, - }; - /* - * Generate a reasonably unique handle based on the address - * of the target buffer. - * - * This is straightforward on 32-bit systems where the flow - * pointer can be used directly. Otherwise, its least - * significant part is taken after shifting it by the - * previous power of two of the pointed buffer size. - */ - if (sizeof(flow) <= 4) - mlx5_nl_flow_brand(flow->nl_flow, (uintptr_t)flow); - else - mlx5_nl_flow_brand - (flow->nl_flow, - (uintptr_t)flow >> - rte_log2_u32(rte_align32prevpow2(flow_size))); + if (!verbs) + return; + /* Update Verbs specification. */ + hdr = (struct ibv_spec_header *)verbs->specs; + if (!hdr) + return; + for (i = 0; i != verbs->attr->num_of_specs; ++i) { + if (hdr->type == IBV_FLOW_SPEC_ACTION_TAG) { + struct ibv_flow_spec_action_tag *t = + (struct ibv_flow_spec_action_tag *)hdr; + + t->tag_id = mlx5_flow_mark_set(mark_id); + } + hdr = (struct ibv_spec_header *)((uintptr_t)hdr + hdr->size); } - return off + ret; } -static unsigned int -mlx5_find_graph_root(const struct rte_flow_item pattern[], uint32_t rss_level) +/** + * Convert the @p action into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested action + * into the flow. This function also return the action that was added. + * + * @param[in] action + * Action configuration. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. + */ +static void +flow_verbs_translate_action_mark(const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow) { - const struct rte_flow_item *item; - unsigned int has_vlan = 0; + const struct rte_flow_action_mark *mark = action->conf; + unsigned int size = sizeof(struct ibv_flow_spec_action_tag); + struct ibv_flow_spec_action_tag tag = { + .type = IBV_FLOW_SPEC_ACTION_TAG, + .size = size, + }; + struct mlx5_flow_verbs *verbs = &dev_flow->verbs; - for (item = pattern; item->type != RTE_FLOW_ITEM_TYPE_END; item++) { - if (item->type == RTE_FLOW_ITEM_TYPE_VLAN) { - has_vlan = 1; - break; - } + if (*action_flags & MLX5_ACTION_FLAG) { + flow_verbs_mark_update(verbs, mark->id); + size = 0; + } else { + tag.tag_id = mlx5_flow_mark_set(mark->id); + mlx5_flow_spec_verbs_add(dev_flow, &tag, size); } - if (has_vlan) - return rss_level < 2 ? MLX5_EXPANSION_ROOT_ETH_VLAN : - MLX5_EXPANSION_ROOT_OUTER_ETH_VLAN; - return rss_level < 2 ? MLX5_EXPANSION_ROOT : - MLX5_EXPANSION_ROOT_OUTER; + *action_flags |= MLX5_ACTION_MARK; } /** - * Convert the @p attributes, @p pattern, @p action, into an flow for the NIC - * after ensuring the NIC will understand and process it correctly. - * The conversion is only performed item/action per item/action, each of - * them is written into the @p flow if its size is lesser or equal to @p - * flow_size. - * Validation and memory consumption computation are still performed until the - * end, unless an error is encountered. + * Convert the @p action into a Verbs specification. This function assumes that + * the input is valid and that there is space to insert the requested action + * into the flow. This function also return the action that was added. * * @param[in] dev - * Pointer to Ethernet device. - * @param[in, out] flow - * Pointer to flow structure. - * @param[in] flow_size - * Size in bytes of the available space in @p flow, if too small some - * garbage may be present. - * @param[in] attributes - * Flow rule attributes. - * @param[in] pattern - * Pattern specification (list terminated by the END pattern item). - * @param[in] actions - * Associated actions (list terminated by the END action). + * Pointer to the Ethernet device structure. + * @param[in] action + * Action configuration. + * @param[in, out] action_flags + * Pointer to the detected actions. + * @param[in] dev_flow + * Pointer to mlx5_flow. * @param[out] error - * Perform verbose error reporting if not NULL. + * Pointer to error structure. * * @return - * On success the number of bytes consumed/necessary, if the returned value - * is lesser or equal to @p flow_size, the flow has fully been converted and - * can be applied, otherwise another call with this returned memory size - * should be done. - * On error, a negative errno value is returned and rte_errno is set. + * 0 On success else a negative errno value is returned and rte_errno is set. */ static int -mlx5_flow_merge(struct rte_eth_dev *dev, struct rte_flow *flow, - const size_t flow_size, - const struct rte_flow_attr *attributes, - const struct rte_flow_item pattern[], - const struct rte_flow_action actions[], - struct rte_flow_error *error) +flow_verbs_translate_action_count(struct rte_eth_dev *dev, + const struct rte_flow_action *action, + uint64_t *action_flags, + struct mlx5_flow *dev_flow, + struct rte_flow_error *error) { - struct rte_flow local_flow = { .layers = 0, }; - size_t size = sizeof(*flow); - union { - struct rte_flow_expand_rss buf; - uint8_t buffer[2048]; - } expand_buffer; - struct rte_flow_expand_rss *buf = &expand_buffer.buf; - struct mlx5_flow_verbs *original_verbs = NULL; - size_t original_verbs_size = 0; - uint32_t original_layers = 0; - int expanded_pattern_idx = 0; - int ret = 0; - uint32_t i; - - if (attributes->transfer) - return mlx5_flow_merge_switch(dev, flow, flow_size, - attributes, pattern, - actions, error); - if (size > flow_size) - flow = &local_flow; - ret = mlx5_flow_attributes(dev->data->dev_private, attributes, flow); - if (ret < 0) - return ret; - ret = mlx5_flow_actions(dev, actions, &local_flow, 0, error); - if (ret < 0) - return ret; - if (local_flow.rss.types) { - unsigned int graph_root; + const struct rte_flow_action_count *count = action->conf; + struct rte_flow *flow = dev_flow->flow; +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + unsigned int size = sizeof(struct ibv_flow_spec_counter_action); + struct ibv_flow_spec_counter_action counter = { + .type = IBV_FLOW_SPEC_ACTION_COUNT, + .size = size, + }; +#endif - graph_root = mlx5_find_graph_root(pattern, - local_flow.rss.level); - ret = rte_flow_expand_rss(buf, sizeof(expand_buffer.buffer), - pattern, local_flow.rss.types, - mlx5_support_expansion, - graph_root); - assert(ret > 0 && - (unsigned int)ret < sizeof(expand_buffer.buffer)); - } else { - buf->entries = 1; - buf->entry[0].pattern = (void *)(uintptr_t)pattern; - } - size += RTE_ALIGN_CEIL(local_flow.rss.queue_num * sizeof(uint16_t), - sizeof(void *)); - if (size <= flow_size) - flow->queue = (void *)(flow + 1); - LIST_INIT(&flow->verbs); - flow->layers = 0; - flow->modifier = 0; - flow->fate = 0; - for (i = 0; i != buf->entries; ++i) { - size_t off = size; - size_t off2; - - flow->layers = original_layers; - size += sizeof(struct ibv_flow_attr) + - sizeof(struct mlx5_flow_verbs); - off2 = size; - if (size < flow_size) { - flow->cur_verbs = (void *)((uintptr_t)flow + off); - flow->cur_verbs->attr = (void *)(flow->cur_verbs + 1); - flow->cur_verbs->specs = - (void *)(flow->cur_verbs->attr + 1); - } - /* First iteration convert the pattern into Verbs. */ - if (i == 0) { - /* Actions don't need to be converted several time. */ - ret = mlx5_flow_actions(dev, actions, flow, - (size < flow_size) ? - flow_size - size : 0, - error); - if (ret < 0) - return ret; - size += ret; - } else { - /* - * Next iteration means the pattern has already been - * converted and an expansion is necessary to match - * the user RSS request. For that only the expanded - * items will be converted, the common part with the - * user pattern are just copied into the next buffer - * zone. - */ - size += original_verbs_size; - if (size < flow_size) { - rte_memcpy(flow->cur_verbs->attr, - original_verbs->attr, - original_verbs_size + - sizeof(struct ibv_flow_attr)); - flow->cur_verbs->size = original_verbs_size; - } - } - ret = mlx5_flow_items - ((const struct rte_flow_item *) - &buf->entry[i].pattern[expanded_pattern_idx], - flow, - (size < flow_size) ? flow_size - size : 0, error); - if (ret < 0) - return ret; - size += ret; - if (size <= flow_size) { - mlx5_flow_adjust_priority(dev, flow); - LIST_INSERT_HEAD(&flow->verbs, flow->cur_verbs, next); - } - /* - * Keep a pointer of the first verbs conversion and the layers - * it has encountered. - */ - if (i == 0) { - original_verbs = flow->cur_verbs; - original_verbs_size = size - off2; - original_layers = flow->layers; - /* - * move the index of the expanded pattern to the - * first item not addressed yet. - */ - if (pattern->type == RTE_FLOW_ITEM_TYPE_END) { - expanded_pattern_idx++; - } else { - const struct rte_flow_item *item = pattern; - - for (item = pattern; - item->type != RTE_FLOW_ITEM_TYPE_END; - ++item) - expanded_pattern_idx++; - } - } + if (!flow->counter) { + flow->counter = mlx5_flow_counter_new(dev, count->shared, + count->id); + if (!flow->counter) + return rte_flow_error_set(error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ACTION, + action, + "cannot get counter" + " context."); } - /* Restore the origin layers in the flow. */ - flow->layers = original_layers; - return size; + *action_flags |= MLX5_ACTION_COUNT; +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + counter.counter_set_handle = flow->counter->cs->handle; + mlx5_flow_spec_verbs_add(dev_flow, &counter, size); +#endif + return 0; } /** @@ -2212,7 +1585,7 @@ mlx5_flow_rxq_tunnel_ptype_update(struct mlx5_rxq_ctrl *rxq_ctrl) * Set the Rx queue flags (Mark/Flag and Tunnel Ptypes) according to the flow. * * @param[in] dev - * Pointer to Ethernet device. + * Pointer to the Ethernet device structure. * @param[in] flow * Pointer to flow structure. */ @@ -2220,8 +1593,8 @@ static void mlx5_flow_rxq_flags_set(struct rte_eth_dev *dev, struct rte_flow *flow) { struct priv *priv = dev->data->dev_private; - const int mark = !!(flow->modifier & - (MLX5_FLOW_MOD_FLAG | MLX5_FLOW_MOD_MARK)); + const int mark = !!(flow->actions & + (MLX5_ACTION_FLAG | MLX5_ACTION_MARK)); const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; @@ -2264,8 +1637,8 @@ static void mlx5_flow_rxq_flags_trim(struct rte_eth_dev *dev, struct rte_flow *flow) { struct priv *priv = dev->data->dev_private; - const int mark = !!(flow->modifier & - (MLX5_FLOW_MOD_FLAG | MLX5_FLOW_MOD_MARK)); + const int mark = !!(flow->actions & + (MLX5_ACTION_FLAG | MLX5_ACTION_MARK)); const int tunnel = !!(flow->layers & MLX5_FLOW_LAYER_TUNNEL); unsigned int i; @@ -3559,21 +2932,21 @@ mlx5_flow_verbs_get_items_and_size(const struct rte_flow_item items[], size += sizeof(struct ibv_flow_spec_tunnel); detected_items |= MLX5_FLOW_LAYER_VXLAN_GPE; break; - case RTE_FLOW_ITEM_TYPE_GRE: #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT + case RTE_FLOW_ITEM_TYPE_GRE: size += sizeof(struct ibv_flow_spec_gre); detected_items |= MLX5_FLOW_LAYER_GRE; -#else - size += sizeof(struct ibv_flow_spec_tunnel); - detected_items |= MLX5_FLOW_LAYER_TUNNEL; -#endif break; case RTE_FLOW_ITEM_TYPE_MPLS: -#ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT size += sizeof(struct ibv_flow_spec_mpls); detected_items |= MLX5_FLOW_LAYER_MPLS; -#endif break; +#else + case RTE_FLOW_ITEM_TYPE_GRE: + size += sizeof(struct ibv_flow_spec_tunnel); + detected_items |= MLX5_FLOW_LAYER_TUNNEL; + break; +#endif default: break; } @@ -3636,7 +3009,7 @@ mlx5_flow_verbs_prepare(const struct rte_flow_attr *attr __rte_unused, uint64_t *action_flags, struct rte_flow_error *error) { - uint32_t size = sizeof(struct ibv_flow_attr); + uint32_t size = sizeof(struct mlx5_flow) + sizeof(struct ibv_flow_attr); struct mlx5_flow *flow; size += mlx5_flow_verbs_get_actions_and_size(actions, action_flags); @@ -3649,10 +3022,150 @@ mlx5_flow_verbs_prepare(const struct rte_flow_attr *attr __rte_unused, "not enough memory to create flow"); return NULL; } + flow->verbs.attr = (void *)(flow + 1); + flow->verbs.specs = (uint8_t *)(flow + 1) + + sizeof(struct ibv_flow_attr); return flow; } /** + * + * Fill the flow with verb spec. + * + * @param[in] dev + * Pointer to Ethernet device. + * @param[in, out] dev_flow + * Pointer to the mlx5 flow. + * @param[in] attr + * Pointer to the flow attributes. + * @param[in] items + * Pointer to the list of items. + * @param[in] actions + * Pointer to the list of actions. + * @param[out] error + * Pointer to the error structure. + * + * @return + * 0 on success, else a negative errno value otherwise and rte_ernno is set. + */ +static int mlx5_flow_verbs_translate(struct rte_eth_dev *dev, + struct mlx5_flow *dev_flow, + const struct rte_flow_attr *attr, + const struct rte_flow_item items[], + const struct rte_flow_action actions[], + struct rte_flow_error *error) +{ + uint64_t action_flags = 0; + uint64_t item_flags = 0; + uint64_t priority = attr->priority; + struct priv *priv = dev->data->dev_private; + + if (priority == MLX5_FLOW_PRIO_RSVD) + priority = priv->config.flow_prio - 1; + for (; actions->type != RTE_FLOW_ACTION_TYPE_END; actions++) { + int ret; + switch (actions->type) { + case RTE_FLOW_ACTION_TYPE_VOID: + break; + case RTE_FLOW_ACTION_TYPE_FLAG: + flow_verbs_translate_action_flag(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_MARK: + flow_verbs_translate_action_mark(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_DROP: + flow_verbs_translate_action_drop(&action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_QUEUE: + flow_verbs_translate_action_queue(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_RSS: + flow_verbs_translate_action_rss(actions, + &action_flags, + dev_flow); + break; + case RTE_FLOW_ACTION_TYPE_COUNT: + ret = flow_verbs_translate_action_count(dev, + actions, + &action_flags, + dev_flow, + error); + if (ret < 0) + return ret; + break; + default: + return rte_flow_error_set(error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ACTION, + actions, + "action not supported"); + } + } + dev_flow->flow->actions |= action_flags; + for (; items->type != RTE_FLOW_ITEM_TYPE_END; items++) { + switch (items->type) { + case RTE_FLOW_ITEM_TYPE_VOID: + break; + case RTE_FLOW_ITEM_TYPE_ETH: + flow_verbs_translate_item_eth(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_VLAN: + flow_verbs_translate_item_vlan(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_IPV4: + flow_verbs_translate_item_ipv4(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_IPV6: + flow_verbs_translate_item_ipv6(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_UDP: + flow_verbs_translate_item_udp(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_TCP: + flow_verbs_translate_item_tcp(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_VXLAN: + flow_verbs_translate_item_vxlan(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_VXLAN_GPE: + flow_verbs_translate_item_vxlan_gpe(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_GRE: + flow_verbs_translate_item_gre(items, &item_flags, + dev_flow); + break; + case RTE_FLOW_ITEM_TYPE_MPLS: + flow_verbs_translate_item_mpls(items, &item_flags, + dev_flow); + break; + default: + return rte_flow_error_set(error, ENOTSUP, + RTE_FLOW_ERROR_TYPE_ITEM, + NULL, + "item not supported"); + } + } + dev_flow->verbs.attr->priority = mlx5_flow_adjust_priority(dev, + priority, + dev_flow->verbs.attr->priority); + return 0; +} + +/** * Remove the flow. * * @param[in] dev @@ -3665,16 +3178,18 @@ mlx5_flow_remove(struct rte_eth_dev *dev, struct rte_flow *flow) { struct priv *priv = dev->data->dev_private; struct mlx5_flow_verbs *verbs; + struct mlx5_flow *dev_flow; if (flow->nl_flow && priv->mnl_socket) mlx5_nl_flow_destroy(priv->mnl_socket, flow->nl_flow, NULL); - LIST_FOREACH(verbs, &flow->verbs, next) { + LIST_FOREACH(dev_flow, &flow->dev_flows, next) { + verbs = &dev_flow->verbs; if (verbs->flow) { claim_zero(mlx5_glue->destroy_flow(verbs->flow)); verbs->flow = NULL; } if (verbs->hrxq) { - if (flow->fate & MLX5_FLOW_FATE_DROP) + if (flow->actions & MLX5_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); @@ -3706,10 +3221,12 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte_flow *flow, { struct priv *priv = dev->data->dev_private; struct mlx5_flow_verbs *verbs; + struct mlx5_flow *dev_flow; int err; - LIST_FOREACH(verbs, &flow->verbs, next) { - if (flow->fate & MLX5_FLOW_FATE_DROP) { + LIST_FOREACH(dev_flow, &flow->dev_flows, next) { + verbs = &dev_flow->verbs; + if (flow->actions & MLX5_ACTION_DROP) { verbs->hrxq = mlx5_hrxq_drop_new(dev); if (!verbs->hrxq) { rte_flow_error_set @@ -3745,8 +3262,8 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte_flow *flow, } verbs->hrxq = hrxq; } - verbs->flow = - mlx5_glue->create_flow(verbs->hrxq->qp, verbs->attr); + verbs->flow = mlx5_glue->create_flow(verbs->hrxq->qp, + verbs->attr); if (!verbs->flow) { rte_flow_error_set(error, errno, RTE_FLOW_ERROR_TYPE_UNSPECIFIED, @@ -3762,9 +3279,10 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte_flow *flow, return 0; error: err = rte_errno; /* Save rte_errno before cleanup. */ - LIST_FOREACH(verbs, &flow->verbs, next) { + LIST_FOREACH(dev_flow, &flow->dev_flows, next) { + verbs = &dev_flow->verbs; if (verbs->hrxq) { - if (flow->fate & MLX5_FLOW_FATE_DROP) + if (flow->actions & MLX5_ACTION_DROP) mlx5_hrxq_drop_release(dev); else mlx5_hrxq_release(dev, verbs->hrxq); @@ -3775,6 +3293,25 @@ mlx5_flow_apply(struct rte_eth_dev *dev, struct rte_flow *flow, return -rte_errno; } +static unsigned int +mlx5_find_graph_root(const struct rte_flow_item pattern[], uint32_t rss_level) +{ + const struct rte_flow_item *item; + unsigned int has_vlan = 0; + + for (item = pattern; item->type != RTE_FLOW_ITEM_TYPE_END; item++) { + if (item->type == RTE_FLOW_ITEM_TYPE_VLAN) { + has_vlan = 1; + break; + } + } + if (has_vlan) + return rss_level < 2 ? MLX5_EXPANSION_ROOT_ETH_VLAN : + MLX5_EXPANSION_ROOT_OUTER_ETH_VLAN; + return rss_level < 2 ? MLX5_EXPANSION_ROOT : + MLX5_EXPANSION_ROOT_OUTER; +} + /** * Create a flow and add it to @p list. * @@ -3804,7 +3341,6 @@ mlx5_flow_list_create(struct rte_eth_dev *dev, { struct rte_flow *flow = NULL; struct mlx5_flow *dev_flow; - size_t size = 0; uint64_t action_flags = 0; uint64_t item_flags = 0; const struct rte_flow_action_rss *rss; @@ -3815,13 +3351,21 @@ mlx5_flow_list_create(struct rte_eth_dev *dev, struct rte_flow_expand_rss *buf = &expand_buffer.buf; int ret; uint32_t i; + uint32_t flow_size; ret = mlx5_flow_validate(dev, attr, items, actions, error); if (ret < 0) return NULL; - flow = rte_calloc(__func__, 1, sizeof(*flow), 0); - LIST_INIT(&flow->dev_flows); + flow_size = sizeof(struct rte_flow); rss = mlx5_flow_get_rss_action(actions); + if (rss) + flow_size += RTE_ALIGN_CEIL(rss->queue_num * sizeof(uint16_t), + sizeof(void *)); + else + flow_size += RTE_ALIGN_CEIL(sizeof(uint16_t), sizeof(void *)); + flow = rte_calloc(__func__, 1, flow_size, 0); + flow->queue = (void *)(flow + 1); + LIST_INIT(&flow->dev_flows); if (rss && rss->types) { unsigned int graph_root; @@ -3842,25 +3386,11 @@ mlx5_flow_list_create(struct rte_eth_dev *dev, &action_flags, error); dev_flow->flow = flow; LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); + mlx5_flow_verbs_translate(dev, dev_flow, attr, + buf->entry[i].pattern, + actions, + error); } - ret = mlx5_flow_merge(dev, flow, size, attr, items, actions, error); - if (ret < 0) - return NULL; - size = ret; - flow = rte_calloc(__func__, 1, size, 0); - if (!flow) { - rte_flow_error_set(error, ENOMEM, - RTE_FLOW_ERROR_TYPE_UNSPECIFIED, - NULL, - "not enough memory to create flow"); - return NULL; - } - ret = mlx5_flow_merge(dev, flow, size, attr, items, actions, error); - if (ret < 0) { - rte_free(flow); - return NULL; - } - assert((size_t)ret == size); if (dev->data->dev_started) { ret = mlx5_flow_apply(dev, flow, error); if (ret < 0) { @@ -3918,6 +3448,12 @@ mlx5_flow_list_destroy(struct rte_eth_dev *dev, struct mlx5_flows *list, */ if (dev->data->dev_started) mlx5_flow_rxq_flags_trim(dev, flow); + while (!LIST_EMPTY(&flow->dev_flows)) { + struct mlx5_flow *dev_flow; + dev_flow = LIST_FIRST(&flow->dev_flows); + LIST_REMOVE(dev_flow, next); + rte_free(dev_flow); + } rte_free(flow); } @@ -4193,7 +3729,7 @@ mlx5_flow_query_count(struct rte_flow *flow __rte_unused, struct rte_flow_error *error) { #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - if (flow->modifier & MLX5_FLOW_MOD_COUNT) { + if (flow->actions & MLX5_ACTION_COUNT) { struct rte_flow_query_count *qc = data; uint64_t counters[2] = {0, 0}; struct ibv_query_counter_set_attr query_cs_attr = {