From patchwork Mon Oct 30 10:07:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matan Azrad X-Patchwork-Id: 31031 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4F2991B33E; Mon, 30 Oct 2017 11:08:14 +0100 (CET) Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10045.outbound.protection.outlook.com [40.107.1.45]) by dpdk.org (Postfix) with ESMTP id E10CE1B2A0 for ; Mon, 30 Oct 2017 11:08:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=PyMfALOarSbKCqtYyW+KgrLZvBGPDoutLJV6v31J4eo=; b=XHL5KEg0VTkY9FSN36tsq2x3fgpg958Jnakm5Ic2UG3RqQSSpHFTXTGRsk40mmoVICFzCFw17AnDCXUBATJ+NJz1KnMYUfMJqR50RwKHqjnlCZHMjy7zeLwj6IpRBVvPMFaDZMum+ecSdRIyLDPF3dB4M46juSDWI06JgwEwXV0= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; Received: from mellanox.com (37.142.13.130) by VI1PR0502MB3662.eurprd05.prod.outlook.com (2603:10a6:803:f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.178.6; Mon, 30 Oct 2017 10:08:01 +0000 From: Matan Azrad To: Adrien Mazarguil Cc: dev@dpdk.org, Ophir Munk Date: Mon, 30 Oct 2017 10:07:27 +0000 Message-Id: <1509358049-18854-6-git-send-email-matan@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1509358049-18854-1-git-send-email-matan@mellanox.com> References: <1508768520-4810-1-git-send-email-ophirmu@mellanox.com> <1509358049-18854-1-git-send-email-matan@mellanox.com> MIME-Version: 1.0 X-Originating-IP: [37.142.13.130] X-ClientProxiedBy: VI1PR08CA0217.eurprd08.prod.outlook.com (2603:10a6:802:15::26) To VI1PR0502MB3662.eurprd05.prod.outlook.com (2603:10a6:803:f::17) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2a431332-c1c3-4903-6e92-08d51f7e1b2a X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081)(4534020)(4602075)(2017052603199); SRVR:VI1PR0502MB3662; X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3662; 3:QTJvME+8Yf9MBeO89C/4E8MmxsNxV1ORbPV2+EQkJ0jjtqjyzUpcjpAlZbNAgwD3iSapGdlstpcVH25VEEyz37Q5MeZZeiPxW+YCfL2Tu1pMG6xeIicufXWc3fbqRG+hNN08lFQgQXsNXxfymHMsjiBca72UWroQIWWfHkLZqbyD9pj2XVtL/iYLIuff4gOMJeFTViWU5WUhtXUdbHg6KHWN6BNvILeyfpp/+5TN6mlahPZ4qF+Y9WCDBJflcvlK; 25:LYYV4YzX70MgflZiPD06IIeo3KiAwrjclYtuEmlMw/sM4zt0GlatV0hlxv2pi1SkzOhfgXdbn40XJwPgq6TltvptP9cluOn9aSMScuHSfOz85kQICz38zzMxkDorGDvgaj5stm75L0JqiYJ/taSfqZB+17JIqHLdVmT3jRcdtGoPs1XXc2SBFbRfgLcB7O5y26TqVcLmsD7UWVqhrbHCIKjrknNKPxz9w8qJT4Ud2EZc0xVULhZob8HVYElsaiMbUAYgZ0CUWgBY3nCoAKk+O0raFdE4ZyfTwTueYpyJxm+KlmrHsYd+Yz3FqmdTwgHG1vBqk1a3GNNgtAJoQA8bSg==; 31:tFqO9J1KvZSAXt+ZJ4xzcfQi0asJU3hQ/w2h7W0HRjTdv5x1aZ28HDm5BBR6c9/AgsPGfOIKxIBe4ZD9DrSFEhVF59dB34s6tGK9CH4McmlAS1tmhWL6Dn2PUa1+kmG0R5vhj7DMVHykJjel1qOwLXVTWmLivoSBemkpOQNXY4IB25FcwMrAB1I12ECzIgG9IE+V5UNeC7nhVhkhSnyvMJfM0foeabDOPMamdegWF0A= X-MS-TrafficTypeDiagnostic: VI1PR0502MB3662: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3662; 20:xD1og9d+YnTJtjFhyCh9JUn3PZgBB5pELvixBSd/jwU6ws7PC4a1FThv5Nhed2affTvV+DULmXpThuum/ql2UdY5elHwzdQSHWyfjUV81EgqiYzvFoQfhvp//kyzuyxT+sCdvInCNbF1xEpHeu4DSTXq/40XkWEWEvGofvCSmizAZG/LHIIOmAL0Bm6boDENrv24dVHlFQAKYO89xMFeVBsWzGJi6MDn/Q4FP++nZILaUGv989lbbP6ARwqLXhIrcWDL3SEzQQVnw73efGwscShO3rbrRW7Y43M0ryLfbe91C1GDKwMrawGq+q3mp6Wpy4PLW2ntdk1a9s/pl5owQiG7mSNH6qyUHO2cbG1254lZzFdirHVuojABhHzNm+WZJEyZcBifOvmG21dMNnTX7ePsggI7mzzDqXObt8qharpXhLnGqraFMef46x/4E+6O+hUeJlzYMikMmiGzMswIzSX0kRJaT5Ir0CSk6VwhnvIDAQst7ryk/o0N03N1Hj/m; 4:IMiRs6rzq6tJei1snfTwzpb6YP+LgZR5gdSgKLfR6uddCxJd6eaDwbiAXyauz2GHPCkoOD2D12pNa670M3No2dkChKvhr9yJIbB3RZglzNVL6DuUzxphBcaz37DmnC2hqD5okmU0I+oohPf51D0bCU0xMlvdQcM19u1UL2I4cuj1rAt92tYT4dGUj9vrXb+5VbxriwiqSqnu+12r7se4YeFQRbTgMOhMUn9aRVkpqnL2o71LQ3J0LjZRLNb2XlSOe04lNyhehg1PDN9qoiNm0K0d3d99VQMNn1lnWVwu/Jkil+Dz2Sql1ZO4JIq//lIn X-Exchange-Antispam-Report-Test: UriScan:(60795455431006); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(3231020)(10201501046)(100000703101)(100105400095)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123555025)(20161123562025)(20161123560025)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:VI1PR0502MB3662; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:VI1PR0502MB3662; X-Forefront-PRVS: 0476D4AB88 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(39860400002)(376002)(346002)(199003)(189002)(16526018)(97736004)(66066001)(47776003)(21086003)(5660300001)(4326008)(50226002)(316002)(53936002)(55016002)(4720700003)(6666003)(16586007)(6916009)(48376002)(36756003)(50466002)(2906002)(7736002)(305945005)(2950100002)(106356001)(101416001)(6116002)(3846002)(69596002)(68736007)(33026002)(81156014)(81166006)(8676002)(105586002)(189998001)(33646002)(107886003)(478600001)(8936002)(5003940100001)(25786009)(86362001)(76176999)(50986999); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0502MB3662; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; VI1PR0502MB3662; 23:mdoUcDmXw8dSCTE/YH83BdVx73OxxVRuheh5WWx?= yYKgwBxccFjeT+JO32aqegdNFUNn/5mgTO1tZfmggflGDadUqP2HwDHD4X+n5pE1SNXjGe4T1CjAqXg47payh1aqoYOvmgCfySv2O5Z53Yhza0Gza3hNxADkblrbzVe7sI9Y81kdkBM6VLhxbGd3OTsZVxO8xatxed2IqHZn+cmicanQOjkusetScS2dPFtXvcnU3kKaUFblvVqyr1/8PSihBcXmYdQrbHVFssEJQ1Q4fewhWFsUhS56rpOqVR+MmA3ggwZ+0H56jrXv8M0y8K8VpZcqF+lh7ouRSRb2HmZNdX+ZNK7SCOKAIjZ4BcrqvlMYeRW3ZYg3hZsad4qIeW5y287qQPmMrS0sSLcUaVdMMHbTTjXtn9EEZlcgn29+1TrNS6CRFr0HoSF6r+YW7NR+Iw/w2Nkbd1iAN7meC0xYOnQusjZyVG/90VOZUVXkMN223MZuVDsX0s9o3t4H1+IrJkDGZjo4Z31W0aHhTNWHvOxtJGuYqaZjTCZyqOjQxKbCzKA5XukfmesTcCWNPe/XNMcQy4hDPbHr1AF+ST+YwRs/+oswWj/TPQwlr5rDrG1RAuOWW8hC0br5Cqf1kXOGVK9rcBPkBLxWHROIrrJp4ndbaw8UQFseaou5nbHNqAmmuJAJsbKfePSkZncnaA/nbFxGJ4wBl9qfweEjPFRK7+VKxgJuwvsLXPzIWUrxJ0I8xUVYaShU0SzRL/7E0WrD+buxcJVSn+B9mSIhiPeqNzcDkvUbq9AoecpGqDB1JbObEJyOfT9xknyN1Or/L3NJyJ22uew0OOgKIEHpH3CdjuF5xzKv3UOHhHPpT58EC6UoY7yR26O8URFIh3HXMtkH9b7VfwY/oaG18MZu4JR71o3PUpXfJ4cNuNnO0O6FKa77nxIElNlZAog2tBMKMwJcTVidfdjWq5OzkMlPptz768GjRiTTi8LVWZx6813Mao3nA8RFVtbVsNnDJpeNQSKMndIaGEUa5l0sp+sWAwIbc3X6fbDAtiEhwU9mQW9LotkOyozimcwE8Cf4u0pzEELedUPYAs1qOSPN4+0HzywC6sABTzH11TpKOw+kzEs2oDNz5EU7X3f0QCK/BFKzObZYIK3+7ArrJxCibP0KuircF9Q== X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3662; 6:6lD+QQPW+5EM987mGDpxI7YpZVoKUjZof2uRwA4LgoFmQCTK0kntG469HPKoO6JrVQ6CCBGvFtCuX/M2MgX9qQhHP1gDPDhCQPdTbuNVq7xEv1qSALqpW3qMu0dDwdO54K8wqbhkoCIsL5Bg1Vxn1k/27/SjAEOztAEVGEoZ/sYCC7wS02E8YChYge+93HzvyTzm2xqJhjwXVfWf9Js0b5mowyIh1HKQ1eMJqcP4+jDbxI1Svltzr0AvZi/ZyXA2HbkixpZM6dASfhD5RBcoazQ37sXs8KE4hNdysbhxQIFNbr1BQiSl+VKOSy1++24Hdg9/TL3s0jGfNekyJyefzFkPMJohwWIvC4VmfKPeTlE=; 5:5PldJdGkMF4NxAccYDYK5vut9PyG3iywFoQFioxynuZXwaTLSMqk2aUK/lRz8azzk5lLbhMdFm8XAKAYpGUwdtlIuWx2MeOBuipsL7BcZLR13sQ8QP/obDU+tPzRLu73pqo2LuAXIeey+pfAD0DGa6kO3cmOQeUtIdB/hw/J+9c=; 24:s0E8/OOpPRtA2EkPDs/K9GV5XlhCRK9Js9NpnUf03WMfqnJqqseovcpKNioHjZ7VDDAlEFknA57OlLLobbVIxb2U4S//NE3UDcp7TzgNeYQ=; 7:kwfCWvMZyY2Gd+UQua4Kj+lLZiczT7AXY+474oliNoHqsjZdO6mUgmAp6XC5xTF+t6SrRCjWHRSryVrc+RXFSRrQ+KNc/Bzin3zsva/IbpPNG8AKxIRoxx4ZtvkPdnV9B8RTtSKwXR5onMTz+aTAnpSdd6M0KRMKrw6GYuXGUZpbE2L4Wje7dCT0L5aIFC/m2Z3Sbyw1y7F36X2YOxAWn+sUp28nO5yxUMhD+ZvdlLt8+hQeMaCZi8obazUQA6mA SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Oct 2017 10:08:01.9117 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2a431332-c1c3-4903-6e92-08d51f7e1b2a X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0502MB3662 Subject: [dpdk-dev] [PATCH v3 5/7] net/mlx4: separate Tx segment cases X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since single segment packets shouldn't use additional memory to save segments byte count we can prevent additional memory unnecessary usage in this case; Prevent loop management. Call a dedicated function for handling multi segments case. Signed-off-by: Matan Azrad Signed-off-by: Ophir Munk --- drivers/net/mlx4/mlx4_rxtx.c | 247 +++++++++++++++++++++++++++---------------- 1 file changed, 158 insertions(+), 89 deletions(-) diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c index 8ce70d6..8ea8851 100644 --- a/drivers/net/mlx4/mlx4_rxtx.c +++ b/drivers/net/mlx4/mlx4_rxtx.c @@ -62,6 +62,9 @@ #include "mlx4_rxtx.h" #include "mlx4_utils.h" +#define WQE_ONE_DATA_SEG_SIZE \ + (sizeof(struct mlx4_wqe_ctrl_seg) + sizeof(struct mlx4_wqe_data_seg)) + /** * Pointer-value pair structure used in tx_post_send for saving the first * DWORD (32 byte) of a TXBB. @@ -140,22 +143,19 @@ struct pv { * @return * 0 on success, -1 on failure. */ -static int -mlx4_txq_complete(struct txq *txq) +static inline int +mlx4_txq_complete(struct txq *txq, const unsigned int elts_n, + struct mlx4_sq *sq) { unsigned int elts_comp = txq->elts_comp; unsigned int elts_tail = txq->elts_tail; - const unsigned int elts_n = txq->elts_n; struct mlx4_cq *cq = &txq->mcq; - struct mlx4_sq *sq = &txq->msq; struct mlx4_cqe *cqe; uint32_t cons_index = cq->cons_index; uint16_t new_index; uint16_t nr_txbbs = 0; int pkts = 0; - if (unlikely(elts_comp == 0)) - return 0; /* * Traverse over all CQ entries reported and handle each WQ entry * reported by them. @@ -238,6 +238,122 @@ struct pv { return buf->pool; } +static int handle_multi_segs(struct rte_mbuf *buf, + struct txq *txq, + struct mlx4_wqe_ctrl_seg **pctrl) +{ + int wqe_real_size; + int nr_txbbs; + struct pv *pv = (struct pv *)txq->bounce_buf; + struct mlx4_sq *sq = &txq->msq; + uint32_t head_idx = sq->head & sq->txbb_cnt_mask; + struct mlx4_wqe_ctrl_seg *ctrl; + struct mlx4_wqe_data_seg *dseg; + uint32_t lkey; + uintptr_t addr; + uint32_t byte_count; + int pv_counter = 0; + + /* Calculate the needed work queue entry size for this packet. */ + wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) + + buf->nb_segs * sizeof(struct mlx4_wqe_data_seg); + nr_txbbs = MLX4_SIZE_TO_TXBBS(wqe_real_size); + /* + * Check that there is room for this WQE in the send queue and that + * the WQE size is legal. + */ + if (((sq->head - sq->tail) + nr_txbbs + + sq->headroom_txbbs) >= sq->txbb_cnt || + nr_txbbs > MLX4_MAX_WQE_TXBBS) { + return -1; + } + + /* Get the control and data entries of the WQE. */ + ctrl = (struct mlx4_wqe_ctrl_seg *)mlx4_get_send_wqe(sq, head_idx); + dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl + + sizeof(struct mlx4_wqe_ctrl_seg)); + *pctrl = ctrl; + /* Fill the data segments with buffer information. */ + struct rte_mbuf *sbuf; + + for (sbuf = buf; sbuf != NULL; sbuf = sbuf->next, dseg++) { + addr = rte_pktmbuf_mtod(sbuf, uintptr_t); + rte_prefetch0((volatile void *)addr); + /* Handle WQE wraparound. */ + if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob) + dseg = (struct mlx4_wqe_data_seg *)sq->buf; + dseg->addr = rte_cpu_to_be_64(addr); + /* Memory region key (big endian) for this memory pool. */ + lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(sbuf)); + dseg->lkey = rte_cpu_to_be_32(lkey); +#ifndef NDEBUG + /* Calculate the needed work queue entry size for this packet */ + if (unlikely(dseg->lkey == rte_cpu_to_be_32((uint32_t)-1))) { + /* MR does not exist. */ + DEBUG("%p: unable to get MP <-> MR association", + (void *)txq); + /* + * Restamp entry in case of failure. + * Make sure that size is written correctly + * Note that we give ownership to the SW, not the HW. + */ + wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) + + buf->nb_segs * sizeof(struct mlx4_wqe_data_seg); + ctrl->fence_size = (wqe_real_size >> 4) & 0x3f; + mlx4_txq_stamp_freed_wqe(sq, head_idx, + (sq->head & sq->txbb_cnt) ? 0 : 1); + return -1; + } +#endif /* NDEBUG */ + if (likely(sbuf->data_len)) { + byte_count = rte_cpu_to_be_32(sbuf->data_len); + } else { + /* + * Zero length segment is treated as inline segment + * with zero data. + */ + byte_count = RTE_BE32(0x80000000); + } + /* + * If the data segment is not at the beginning of a + * Tx basic block (TXBB) then write the byte count, + * else postpone the writing to just before updating the + * control segment. + */ + if ((uintptr_t)dseg & (uintptr_t)(MLX4_TXBB_SIZE - 1)) { + /* + * Need a barrier here before writing the byte_count + * fields to make sure that all the data is visible + * before the byte_count field is set. + * Otherwise, if the segment begins a new cacheline, + * the HCA prefetcher could grab the 64-byte chunk and + * get a valid (!= 0xffffffff) byte count but stale + * data, and end up sending the wrong data. + */ + rte_io_wmb(); + dseg->byte_count = byte_count; + } else { + /* + * This data segment starts at the beginning of a new + * TXBB, so we need to postpone its byte_count writing + * for later. + */ + pv[pv_counter].dseg = dseg; + pv[pv_counter++].val = byte_count; + } + } + /* Write the first DWORD of each TXBB save earlier. */ + if (pv_counter) { + /* Need a barrier here before writing the byte_count. */ + rte_io_wmb(); + for (--pv_counter; pv_counter >= 0; pv_counter--) + pv[pv_counter].dseg->byte_count = pv[pv_counter].val; + } + /* Fill the control parameters for this packet. */ + ctrl->fence_size = (wqe_real_size >> 4) & 0x3f; + + return nr_txbbs; +} /** * DPDK callback for Tx. * @@ -261,10 +377,11 @@ struct pv { unsigned int i; unsigned int max; struct mlx4_sq *sq = &txq->msq; - struct pv *pv = (struct pv *)txq->bounce_buf; + int nr_txbbs; assert(txq->elts_comp_cd != 0); - mlx4_txq_complete(txq); + if (likely(txq->elts_comp != 0)) + mlx4_txq_complete(txq, elts_n, sq); max = (elts_n - (elts_head - txq->elts_tail)); if (max > elts_n) max -= elts_n; @@ -283,7 +400,6 @@ struct pv { uint32_t owner_opcode = MLX4_OPCODE_SEND; struct mlx4_wqe_ctrl_seg *ctrl; struct mlx4_wqe_data_seg *dseg; - struct rte_mbuf *sbuf; union { uint32_t flags; uint16_t flags16[2]; @@ -291,10 +407,6 @@ struct pv { uint32_t head_idx = sq->head & sq->txbb_cnt_mask; uint32_t lkey; uintptr_t addr; - uint32_t byte_count; - int wqe_real_size; - int nr_txbbs; - int pv_counter = 0; /* Clean up old buffer. */ if (likely(elt->buf != NULL)) { @@ -313,40 +425,29 @@ struct pv { } while (tmp != NULL); } RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); - - /* - * Calculate the needed work queue entry size - * for this packet. - */ - wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) + - buf->nb_segs * sizeof(struct mlx4_wqe_data_seg); - nr_txbbs = MLX4_SIZE_TO_TXBBS(wqe_real_size); - /* - * Check that there is room for this WQE in the send - * queue and that the WQE size is legal. - */ - if (((sq->head - sq->tail) + nr_txbbs + - sq->headroom_txbbs) >= sq->txbb_cnt || - nr_txbbs > MLX4_MAX_WQE_TXBBS) { - elt->buf = NULL; - break; - } - /* Get the control and data entries of the WQE. */ - ctrl = (struct mlx4_wqe_ctrl_seg *) - mlx4_get_send_wqe(sq, head_idx); - dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl + - sizeof(struct mlx4_wqe_ctrl_seg)); - /* Fill the data segments with buffer information. */ - for (sbuf = buf; sbuf != NULL; sbuf = sbuf->next, dseg++) { - addr = rte_pktmbuf_mtod(sbuf, uintptr_t); + if (buf->nb_segs == 1) { + /* + * Check that there is room for this WQE in the send + * queue and that the WQE size is legal + */ + if (((sq->head - sq->tail) + 1 + sq->headroom_txbbs) >= + sq->txbb_cnt || 1 > MLX4_MAX_WQE_TXBBS) { + elt->buf = NULL; + break; + } + /* Get the control and data entries of the WQE. */ + ctrl = (struct mlx4_wqe_ctrl_seg *) + mlx4_get_send_wqe(sq, head_idx); + dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl + + sizeof(struct mlx4_wqe_ctrl_seg)); + addr = rte_pktmbuf_mtod(buf, uintptr_t); rte_prefetch0((volatile void *)addr); /* Handle WQE wraparound. */ - if (unlikely(dseg >= - (struct mlx4_wqe_data_seg *)sq->eob)) + if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob) dseg = (struct mlx4_wqe_data_seg *)sq->buf; dseg->addr = rte_cpu_to_be_64(addr); /* Memory region key (big endian). */ - lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(sbuf)); + lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(buf)); dseg->lkey = rte_cpu_to_be_32(lkey); #ifndef NDEBUG if (unlikely(dseg->lkey == @@ -360,61 +461,28 @@ struct pv { * Note that we give ownership to the SW, * not the HW. */ - ctrl->fence_size = (wqe_real_size >> 4) & 0x3f; + ctrl->fence_size = + (WQE_ONE_DATA_SEG_SIZE >> 4) & 0x3f; mlx4_txq_stamp_freed_wqe(sq, head_idx, (sq->head & sq->txbb_cnt) ? 0 : 1); elt->buf = NULL; break; } #endif /* NDEBUG */ - if (likely(sbuf->data_len)) { - byte_count = rte_cpu_to_be_32(sbuf->data_len); - } else { - /* - * Zero length segment is treated as inline - * segment with zero data. - */ - byte_count = RTE_BE32(0x80000000); - } - /* - * If the data segment is not at the beginning - * of a Tx basic block (TXBB) then write the - * byte count, else postpone the writing to - * just before updating the control segment. - */ - if ((uintptr_t)dseg & (uintptr_t)(MLX4_TXBB_SIZE - 1)) { - /* - * Need a barrier here before writing the - * byte_count fields to make sure that all the - * data is visible before the byte_count field - * is set. otherwise, if the segment begins a - * new cacheline, the HCA prefetcher could grab - * the 64-byte chunk and get a valid - * (!= 0xffffffff) byte count but stale data, - * and end up sending the wrong data. - */ - rte_io_wmb(); - dseg->byte_count = byte_count; - } else { - /* - * This data segment starts at the beginning of - * a new TXBB, so we need to postpone its - * byte_count writing for later. - */ - pv[pv_counter].dseg = dseg; - pv[pv_counter++].val = byte_count; - } - } - /* Write the first DWORD of each TXBB save earlier. */ - if (pv_counter) { - /* Need a barrier before writing the byte_count. */ + /* Need a barrier here before byte count store. */ rte_io_wmb(); - for (--pv_counter; pv_counter >= 0; pv_counter--) - pv[pv_counter].dseg->byte_count = - pv[pv_counter].val; + dseg->byte_count = rte_cpu_to_be_32(buf->data_len); + + /* Fill the control parameters for this packet. */ + ctrl->fence_size = (WQE_ONE_DATA_SEG_SIZE >> 4) & 0x3f; + nr_txbbs = 1; + } else { + nr_txbbs = handle_multi_segs(buf, txq, &ctrl); + if (nr_txbbs < 0) { + elt->buf = NULL; + break; + } } - /* Fill the control parameters for this packet. */ - ctrl->fence_size = (wqe_real_size >> 4) & 0x3f; /* * For raw Ethernet, the SOLICIT flag is used to indicate * that no ICRC should be calculated. @@ -469,6 +537,7 @@ struct pv { ctrl->owner_opcode = rte_cpu_to_be_32(owner_opcode | ((sq->head & sq->txbb_cnt) ? MLX4_BIT_WQE_OWN : 0)); + sq->head += nr_txbbs; elt->buf = buf; bytes_sent += buf->pkt_len;