From patchwork Wed Aug 23 07:33:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shahaf Shuler X-Patchwork-Id: 27741 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 0474A7D6A; Wed, 23 Aug 2017 09:34:12 +0200 (CEST) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0050.outbound.protection.outlook.com [104.47.2.50]) by dpdk.org (Postfix) with ESMTP id 7ADD87D52; Wed, 23 Aug 2017 09:34:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=hr379UWEaX0tPwSabYdN6lRHVUyWvbVqRzznOL6VcQ0=; b=plg4rUCbQJNlRNRU+uCrSfdT0O+eYTeLLlushoofzAORY7MrCLljaKv+A++cuUUNDadfOheFH7yq42L2WacT5P0zst01EMmF6OFQ2W9IQkKk8J3L5TMri6N+SVZvMsr84bGQhmPaqAywRfYctEUzFzhocjZehfOPIzPdanJTJp4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; Received: from mellanox.com (82.166.227.17) by HE1PR05MB3145.eurprd05.prod.outlook.com (2603:10a6:7:36::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1362.18; Wed, 23 Aug 2017 07:34:07 +0000 From: Shahaf Shuler To: nelio.laranjeiro@6wind.com, adrien.mazarguil@6wind.com Cc: dev@dpdk.org, yskoh@mellanox.com, stable@dpdk.org Date: Wed, 23 Aug 2017 10:33:58 +0300 Message-Id: <20170823073358.116786-2-shahafs@mellanox.com> X-Mailer: git-send-email 2.12.0 In-Reply-To: <20170823073358.116786-1-shahafs@mellanox.com> References: <20170823073358.116786-1-shahafs@mellanox.com> MIME-Version: 1.0 X-Originating-IP: [82.166.227.17] X-ClientProxiedBy: HE1PR05CA0158.eurprd05.prod.outlook.com (2603:10a6:7:28::45) To HE1PR05MB3145.eurprd05.prod.outlook.com (2603:10a6:7:36::15) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 38402a46-0075-41ca-f474-08d4e9f9570f X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(48565401081)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:HE1PR05MB3145; X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB3145; 3:fGZ30RZ9uxkUohopBW1qAcghGL+ahZnMbA4Yy7YbkJIgvCo9wB6lpFOBRHoZL+OknyqX5l5IXfjmNEWGHFgYejBAaXMqU+XqfIPCHwl839we4Honu7BXPL106RDlCBb9ZX2n4oyCpQzDaLcHoux8kuAc0rEs2GCIppAa0sWOcX5DDM6TeaHuq0zQRd9as4MzICvm/GXPNCthxRt69HUUCnyHkauZiF78CAHY7J67/IOBobaGOShrpw0jGSRGy99E; 25:EL2h6MOx6lO4jaYWc+bIn7gfE+63RPcQYXFmFBjk0Rb1KEkJNH3DVXu+Bz2NLPWsbNkPz0O5EZafxlBGFuRBBr1JGm7RJzNZfa0KlKaUoDJlxiwy/4ltyIVVhZ5dY66OkaRSjTGdNJuCW94xb1J5Lc10qtT8VtgvQyW0yaYAaqJs5LtAQ5PsTPNs9LhvAtpWX0E4clXANqqP5L7dHadu41IGTKxhMSbLCXx7uyddFupHfOeeP7BUkAb8zuTNlsLn8hXJkH3taT7m93Ov+HdHnloUrbNu6Bcy8ACWmfp8U9R7jV0N1wOej37RLyCJ35Ui4KxiDGnoWv31BgqcD6Lsqg==; 31:lGiN7iuj2rmLpm+ilH3qDfXtQyA51R6cRZBXrBpLtt9ZgkWUohl8yYPIwxbOEl3SZi18pN25BsUkG15PV24DCSgghKHyytXXC3xGgC4/semg8/1DV0+/yB2UVxpknj3yHoxZexY87WxQ/9H+NguhIVNVB0JLh+zcXX2Lj9ilVBST64k6ZZLibzNAcPv4lXsw0cGA+FwbeF0geCBkxVuNSfD6c67ZoUr+3u9arLChc5A= X-MS-TrafficTypeDiagnostic: HE1PR05MB3145: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB3145; 20:y0zGNHPJPbRbHCA7zGMUMQM3LpVpNOdI5h5Kxmvq+Wdu7HFIrJcbIzvvmy0fYKK7dG+VJCyrO0G1C4gQLdpBycMCXp5SeUkJ1OemwyqJPqx7ZFvY1+bf70TTV6ak+O7WidXxVP0wRIfUdJwwimKMokEdAx0lnsRtppneH8Fg42/LQEqoZFReUpXSx7lLvA9hSFD00Q6iPzPxFXeFy1UeVn+/iqUhtLNDBKMJUQ1RQS/fABSlBEmtcnL6bnowYgenNklX85En+TjX5aa2jkALArrVw+dYvff/AB8GTMyINuhdu3Pj4e/NmpPYBPN2CUi+WuNrVLB4NcFZiJqkxG/QrCrsjkFxKBK4KcC3M3RCocdKueXl+8O4YXOqxNy6B1yg39xVv3ix4BrefkFBbodoVd0QWv+u2+yKjhLbxv+FACLnaPJ0f7dPAUagI1QSHrESbM+bSqvBQqZBO6ce2qHzi6QfM2jrM7QXmfbIU9T+JJBrEtl++yHtU6LgubcZ2iTZ; 4:LJ1pmZBV2nX7KG39wa1NnDSMvliov99Yu2cm9bKONYO3KqDDaqVJF1SF0XILTtFdhos8ZY5KvcNmx6jRT4+tRQLEfpzzsqC4ZEohlLMltvHL+GP7f5K7CpqW0Z6CbU1pEuN7GFCFLoWzOhLqgXRjbQs/3qmshgHi1YZuoyCMw5Fts7WpLrs6bDkeEyWYn0uvtxlsbQyxF+1KysNQvn9Ul+QA/jlbEF5EuNz6vZv/SisZy72ShuX2q+ZpbW1hhX6B X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(93006095)(93001095)(10201501046)(3002001)(6055026)(6041248)(20161123564025)(20161123558100)(20161123562025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:HE1PR05MB3145; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:HE1PR05MB3145; X-Forefront-PRVS: 040866B734 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(7370300001)(6009001)(199003)(189002)(7736002)(21086003)(6666003)(7350300001)(42186005)(47776003)(33026002)(6306002)(4326008)(55016002)(33646002)(97736004)(53936002)(305945005)(1076002)(86362001)(3846002)(6116002)(68736007)(8676002)(50986999)(48376002)(25786009)(5003940100001)(53376002)(81166006)(110136004)(69596002)(5660300001)(50466002)(478600001)(2950100002)(36756003)(189998001)(101416001)(50226002)(66066001)(105586002)(76176999)(966005)(2906002)(81156014)(106356001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR05MB3145; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; HE1PR05MB3145; 23:/Rc5nG6GQ1R6unoS+EGiDfd9t7S7Na238lz+uQBcM?= tjzhEwcqjVZF+g0Jfw5n74PijygmjVJGOzd8uwyb8pqcN1cQa7TbqWsEXdTPv1QYO7S5VBSF0ZCtbWP5AWuBEyMdM7zRhBUfmY0VnMvoMRz/mWVUDrDn6iLNjpz6fdz3MWdmebRmgDEEU2DOihDR0aSsoE32yoaztuufmbEDrvoNH2kMJtTiZdxP9STU7+gnCAaxiXObo+N/wKTYPh8XDQ8/gtw/ChycdXLlh8JrXJ7T5TXJDXy6jpdH5s8C3H3lNoxnYqZaIQY7q9nWizz/jv6yDUk0bynGrYfaUbvUgavefXpovfdA6caItHU/kXQizSo1fMfs8uUR1JoNOyRxDjaQVNQRVDINV935RtcmQgKvy1ysSqWyJM9ISQoh6RLHW4QAGkyRtsZiw0URuDwQu6VJtVNhH+NT5hCqYIGSsw6O08w5g0+0iE5ISQXDZeVr4nSDglr03s32QdWWyLWdoN5smq/eo1ivjMuA2nPvnYkPtn7ap1C21XWiofctHsQCgzAODoC5Fjwns48VmRbQE+9t2uodqSQeIMzd6kXkx7LUJZBkIQ8zAUyIcVINNhsFO0JnE6Wq1lrGef4rYetvDTIe2oLmi7EadvDSfXhAM21etlvGC9u0PZMQCgnLeTpQcHb0LFm2MTBBsjwIXfphWMO42nZj8mpJCI6+t+naIqW2H6hOIh5FGoajgKecsl43vqTBpWXIhmzIgyYt3oETV3OZx5W8GxGGO8YRIvfrXMUq5IEDcOJD8iZcuCnD4/xZ5qzJPEG2YZtHrW1+CO4yujU748CV/o/FoLlEopJG3TP5aEU5/2IkttbFLqh6O4ueynYoo7YY23hgKKLN7yqnXxSz4jDdGmxLT7BafdZi4DJHaVus6rlk0uFTOQw7I2wcIP6WD0KsQ6IArOfQaMCraUzUcwcb7ehRYFkzQr30dnLoZwQNjf3XDGVPeTmm8ZfTzjvEiNBJZTGVnr8YlggA3M7qbTMMR/yIW7k13TwI/uSK4nO+XjbxZmcvUzo0LyepQ9UkgGwqjlOFqrxZPigP9edPR2p5mFBoSdSpnjb9tqwpoTkRkgxsImIA3Ubm+Zzs8xau9MMMmUUm6NzJxtmdg4ylWwhgXGWB5c55QZSWS6xag== X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB3145; 6:FgryrXG9oYf0dSzDaSjrnmJB3XCD0GnbFaprsVRnRDgVokyVG+xUYaKubX5WXHOC+h9X7h2WHHHmgSeS8K+zVH/RjcdvNlqZodTFNJWLt2s8N2d4iGZnX77q7uI7SCjcVlaTz4HxCS1DiPVh/TUX6Tss85XQQUDIdMb/qCYBPqCmikD7ETon5KP0n3EQ7PZTLBpydKkgmApLTYYcspXFmkppCXbGhn7p/x6wp0c1USvikZHj7W7f2kCLCJqiAhmIZc2iMHAn5i5GlZVEDbvw1rKzRdwJQ3tjaqiNHyQ0P93IxobS4jJXa5BwzjIkNS0zIzOyX5iyU+WrsbVtB7tg1w==; 5:AtrkDw0Tvp7svmof8MellTIeZtk0fUe3PxN+Mop96acrpxc8yRhN5gPFxqaT1X2zJymSGyXwmIlimINT8akIPHkFpsGI1rXp3PRo6MU0S/HH2tLxYN58pTcYdk5LtfVFFs7aPzCXp9tvsmjxCd4czQ==; 24:4NoTBQJcvdOifyhj6FbN8JRIwERs5wjfUrzinLS0PfokR3cvqHSBtjyqPmDY2UhnzgXWwIpqzxERxXDFF6DXuDut0+4faesMccmQl0p70YY=; 7:FoiMnrmWm7ybUH3OI1Up8Ydd0ScXMPiRVqebCcS1e7fc1yKgVQZtw8167hd1iZPsFOhX49QDRNHYJynV1jnO2JGfzLBxgMyhWqk+d3L9lxco0i1+rdIHw0IlRE/YzDjGS1VcXyJjl1SP3TzI1EWNri8dS236w/jR1IJZ7tPLLXR9kI2dzYUlIa2DIeUdUQvEJoZwQeV34sJVD8Pmo0LodKdajE5PosjZ9pwq365vZaE= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Aug 2017 07:34:07.4280 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR05MB3145 Subject: [dpdk-dev] [PATCH 2/2] net/mlx5: enforce Tx num of segments limitation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Mellanox NICs has a limitation on the number of mbuf segments a multi segment mbuf can have. The max number depends on the Tx offloads requested. The current code not enforce such limitation, which might cause malformed WQEs to be written to the device. This commit adds verification for the number of mbuf segments posted to the device. In case of overflow the packet will not be sent. Debug prints were added to help application identify the cause for such case. Cc: stable@dpdk.org Signed-off-by: Shahaf Shuler --- This patch should be applied only after the series: http://dpdk.org/dev/patchwork/patch/27367/ --- drivers/net/mlx5/mlx5_defs.h | 3 ++- drivers/net/mlx5/mlx5_prm.h | 3 +++ drivers/net/mlx5/mlx5_rxtx.c | 30 +++++++++++++++++++++++++++--- drivers/net/mlx5/mlx5_rxtx_vec_sse.c | 8 ++++++++ drivers/net/mlx5/mlx5_txq.c | 27 +++++++++++++++++++++++++++ 5 files changed, 67 insertions(+), 4 deletions(-) diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h index a76bc6f65..7d2a65099 100644 --- a/drivers/net/mlx5/mlx5_defs.h +++ b/drivers/net/mlx5/mlx5_defs.h @@ -100,7 +100,8 @@ /* * Maximum size of burst for vectorized Tx. This is related to the maximum size - * of Enhaned MPW (eMPW) WQE as vectorized Tx is supported with eMPW. + * of Enhacned MPW (eMPW) WQE as vectorized Tx is supported with eMPW. + * Careful when changing, large value can cause wqe DS to overlap. */ #define MLX5_VPMD_TX_MAX_BURST 32U diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h index 608072f7e..87244e7db 100644 --- a/drivers/net/mlx5/mlx5_prm.h +++ b/drivers/net/mlx5/mlx5_prm.h @@ -154,6 +154,9 @@ /* Default mark value used when none is provided. */ #define MLX5_FLOW_MARK_DEFAULT 0xffffff +/* Maximum number of DS in WQE. */ +#define MLX5_MAX_DS (63) + /* Subset of struct mlx5_wqe_eth_seg. */ struct mlx5_wqe_eth_seg_small { uint32_t rsvd0; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index b07bcd118..77a789c2c 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -664,6 +664,15 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) else j += sg; next_pkt: + if (ds > MLX5_MAX_DS) { +#ifndef NDEBUG + WARN("Cannot send packet %p with %d segments " + "wqe.ds = %d, wqe.inline_sz = %d", + (void *)pkts, (*pkts)->nb_segs, ds, + pkt_inline_sz); +#endif + break; + } ++elts_head; ++pkts; ++i; @@ -850,8 +859,13 @@ mlx5_tx_burst_mpw(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) if (max_elts < segs_n) break; /* Do not bother with large packets MPW cannot handle. */ - if (segs_n > MLX5_MPW_DSEG_MAX) + if (segs_n > MLX5_MPW_DSEG_MAX) { +#ifndef NDEBUG + WARN("Cannot send packet %p with %d segments", + (void *)buf, segs_n); +#endif break; + } max_elts -= segs_n; --pkts_n; /* Should we enable HW CKSUM offload */ @@ -1071,8 +1085,13 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts, if (max_elts < segs_n) break; /* Do not bother with large packets MPW cannot handle. */ - if (segs_n > MLX5_MPW_DSEG_MAX) + if (segs_n > MLX5_MPW_DSEG_MAX) { +#ifndef NDEBUG + WARN("Cannot send packet %p with %d segments", + (void *)buf, segs_n); +#endif break; + } max_elts -= segs_n; --pkts_n; /* @@ -1360,8 +1379,13 @@ mlx5_tx_burst_empw(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) if (max_elts - j < segs_n) break; /* Do not bother with large packets MPW cannot handle. */ - if (segs_n > MLX5_MPW_DSEG_MAX) + if (segs_n > MLX5_MPW_DSEG_MAX) { +#ifndef NDEBUG + WARN("Cannot send packet %p with %d segments", + (void *)buf, segs_n); +#endif break; + } /* Should we enable HW CKSUM offload. */ if (buf->ol_flags & (PKT_TX_IP_CKSUM | PKT_TX_TCP_CKSUM | PKT_TX_UDP_CKSUM)) diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c index 30727e6dd..6f66517df 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c @@ -257,6 +257,13 @@ txq_scatter_v(struct txq *txq, struct rte_mbuf **pkts, uint16_t pkts_n) if (segs_n == 1 || max_elts < segs_n || max_wqe < 2) break; + if (segs_n > MLX5_MPW_DSEG_MAX) { +#ifndef NDEBUG + WARN("Cannot send packet %p with %d segments", + (void *)buf, segs_n); +#endif + break; + } wqe = &((volatile struct mlx5_wqe64 *) txq->wqes)[wqe_ci & wq_mask].hdr; if (buf->ol_flags & @@ -374,6 +381,7 @@ txq_burst_v(struct txq *txq, struct rte_mbuf **pkts, uint16_t pkts_n, max_elts = (elts_n - (elts_head - txq->elts_tail)); max_wqe = (1u << txq->wqe_n) - (txq->wqe_ci - txq->wqe_pi); pkts_n = RTE_MIN((unsigned int)RTE_MIN(pkts_n, max_wqe), max_elts); + assert(pkts_n <= MLX5_MAX_DS - nb_dword_in_hdr); if (unlikely(!pkts_n)) return 0; elts = &(*txq->elts)[elts_head & elts_m]; diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 98aaa7ca0..6ea02485f 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -295,6 +295,8 @@ txq_ctrl_setup(struct rte_eth_dev *dev, struct txq_ctrl *txq_ctrl, .comp_mask = IBV_EXP_QP_INIT_ATTR_PD, }; if (priv->txq_inline && (priv->txqs_n >= priv->txqs_inline)) { + unsigned int ds_cnt; + tmpl.txq.max_inline = ((priv->txq_inline + (RTE_CACHE_LINE_SIZE - 1)) / RTE_CACHE_LINE_SIZE); @@ -327,6 +329,31 @@ txq_ctrl_setup(struct rte_eth_dev *dev, struct txq_ctrl *txq_ctrl, attr.init.cap.max_inline_data = tmpl.txq.max_inline * RTE_CACHE_LINE_SIZE; } + /* + * Check if the inline size is too large in a way which + * can make the wqe DS to overflow. + * Considering in calculation: + * WQE CTRL (1 DS) + * WQE ETH (1 DS) + * inline part (N DS) + */ + ds_cnt = 2 + + (attr.init.cap.max_inline_data / MLX5_WQE_DWORD_SIZE); + if (ds_cnt > MLX5_MAX_DS) { + unsigned int max_inline = (MLX5_MAX_DS - 2) * + MLX5_WQE_DWORD_SIZE; + + /* Ceil down*/ + max_inline = max_inline - (max_inline % + RTE_CACHE_LINE_SIZE); + WARN("txq inline is too large (%d) setting it to " + "the maximum possible: %d\n", + priv->txq_inline, max_inline); + tmpl.txq.max_inline = max_inline / RTE_CACHE_LINE_SIZE; + attr.init.cap.max_inline_data = max_inline; + if (priv->mps == MLX5_MPW_ENHANCED) + tmpl.txq.inline_max_packet_sz = max_inline; + } } if (priv->tso) { attr.init.max_tso_header =