From patchwork Tue Nov 28 12:19:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matan Azrad X-Patchwork-Id: 31720 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A6DC97CBA; Tue, 28 Nov 2017 13:20:19 +0100 (CET) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0071.outbound.protection.outlook.com [104.47.1.71]) by dpdk.org (Postfix) with ESMTP id 67151377A for ; Tue, 28 Nov 2017 13:20:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=F7gsgjOqVEK438Del5wjeLr9H/QwbfTeJyWbnfsMzMU=; b=ZARg0q8v2H/3WeXiR/br5zgqranNgHMBpNaubVoSsSIDcO0ctYpoMy66Q3TwcvO8ZeS48RU+VNZvlY8V+I2LiB8QXuJ2rv+KDchVy77iLfRnDtEiY1Q4YYutXfk2hC9OiIomL1BMTdqybX34Nm9LGNo8yZ4J/uH3D87OF2GJjjw= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; Received: from mellanox.com (37.142.13.130) by AM6PR0502MB3653.eurprd05.prod.outlook.com (2603:10a6:209:9::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.260.4; Tue, 28 Nov 2017 12:20:07 +0000 From: Matan Azrad To: Adrien Mazarguil Cc: dev@dpdk.org Date: Tue, 28 Nov 2017 12:19:28 +0000 Message-Id: <1511871570-16826-7-git-send-email-matan@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1511871570-16826-1-git-send-email-matan@mellanox.com> References: <1511871570-16826-1-git-send-email-matan@mellanox.com> MIME-Version: 1.0 X-Originating-IP: [37.142.13.130] X-ClientProxiedBy: DB6P193CA0012.EURP193.PROD.OUTLOOK.COM (2603:10a6:6:29::22) To AM6PR0502MB3653.eurprd05.prod.outlook.com (2603:10a6:209:9::14) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: a7e50842-cb83-49d8-158d-08d5365a5d5a X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(48565401081)(5600026)(4604075)(2017052603199); SRVR:AM6PR0502MB3653; X-Microsoft-Exchange-Diagnostics: 1; AM6PR0502MB3653; 3:0i+CUWIpiBehioBlQDzAZDpO/sbARnZf9MgjNKxEOOYLhzfhK0wVkli9rpoeHi7Ut1kofwQJfmH01Z84TCkq6BzDppGRSTmFLMXWzIy+0OkCNKFneoCuv9R5ZEZGRrrlfFCWaCVEqUqP9QiLl2PBZnoZ0NkezEaM9ITov3CY0gS5lrxUPwY01p2lwo4TK5aNF16s6te9eNsYrm2V/Xob8Kf+B+irx3GH/wcyF4E0jmA8tuYPepxxzxqkaCJTQ2mB; 25:K+NQb4DhDrLTQJ/qW4IS6MNVf9GmARTa3j9dUci1JHTgG6sjqfFV3kPg6ufHquExdQsSF70s4XzA278nfDnA3dN8lzbXWbYzIJ2EYbpoV1F24kkXWywVScBAx4iUqNEXwk7cGO7bNZ1PXznOSqNYdXZpnctmvuamOw7ZXuqM1Ri9iDWGZssAPdihqLrWTApC0OHpJSNzd+eekH7veGpxzq0fhj9Jpy04GZeknG6Bj3lSfNi+o7vhqGVA47r8Z73QcRJWsNOQdDRaETgF87NNhK57eNqvDMIsDoi1ql/aCJxNo+jXfk9o9/C8grbpWSlvuTWahqEDfn5uNp2iA9EU+J+ekd27eMKHCuxTbGtGcGk=; 31:ZLpuisLOl9BvaHOA6qtfrBgnYxJLxpPn2qKt02dOpc2TbrVsE587MZkegIh8xVPxwcf+jU9F+RojnyEFIuZN3Wd9zFMZdheFAUWcIJxBuNioXksD5BWYaUqUBsGy1QQtxTx14Dzn3sOjXpRTVgWTyKvz1RrkKCXwNTS0WF6zFP7l/0kPoCH7rD/r2Xmu+Bi2GdpSjs0GDWsiWrL4htN4S1RWtYp5UPZZB/sUygVmZmc= X-MS-TrafficTypeDiagnostic: AM6PR0502MB3653: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM6PR0502MB3653; 20:NI5brUH/S6AIasStwNx7Oo7PwqDnj2XuGr2lNCIE1Ac3Gp7bvxJc/Y9i/+YijeAA+Hbq8Mho9s2E19eTjBmJRmE4GRNrGiucxKWSWoyhCZt3rqx2eX6VEoa3xuzpeqLSU129X0tRClPZsK+BjrqGgAuXExbcz+mJ1ktg2pR22WjEbIfCIIPxR7NXo8kux6SSkpm178lYHI9NOpypeHiS9tpXl2bp7YcPMtmKneeitjz7N/XKwZPvqzFDGsrJfJttjfJIO9NNtRNFsoNfL9qFolNn9O6ZvXhpoMzQBqp0M8qYUu35LyR5exkM+dvzZCoEmBmPWAUPJ1W4HdY6Y2w0PWW9CEoMnjnDqyQxQ2ruFPbFqylIcchnfPnrMZ6iXGKnD1I5Rw5D8VsyGWYSzLLW7Y7sGGYk9+fif+mwLjfj6d4nXHL22F+dis9ilNVxbqUTB9AgtAzOqjp4//9ZnWvM3cN8fXmpotcX2e36Iw5w5vOpmwpVKfVPT2pPJCw/zW0H; 4:/GDit+FChQN9J+5AMsp2tjXOPmC1lh/RUHHd+ajEzaU6uFDX7sKrtXen0iZRTREaRxjYszIUhWtDlhbQldx5Au/HFKhiyLV1OfdcSFuCHmuWoGJTiwugBAUXhMQQ37P9DGejDZKnSaJptEs40MCNXMNGamVhBGgHYcRWS9w6Qv4f3aB6BbEbY3x73eCdqdZ0h2KJ1N/aN7eoFuT03a0zTId93VT67Tv7UaqJBJPPiaffDUdFFNGDZzxgPAyORJQL+iDNpjaEe/mw5DX3sGCaiA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(8121501046)(5005006)(3231022)(3002001)(10201501046)(93006095)(93001095)(6055026)(6041248)(20161123562025)(20161123558100)(20161123555025)(20161123564025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:AM6PR0502MB3653; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:AM6PR0502MB3653; X-Forefront-PRVS: 0505147DDB X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(346002)(366004)(39860400002)(199003)(189002)(101416001)(76176999)(86362001)(50986999)(4720700003)(8676002)(81166006)(81156014)(6666003)(36756003)(55016002)(33646002)(106356001)(105586002)(2950100002)(6916009)(4326008)(478600001)(189998001)(48376002)(25786009)(68736007)(2906002)(50466002)(52116002)(21086003)(53936002)(8936002)(47776003)(33026002)(66066001)(5660300001)(50226002)(6116002)(3846002)(316002)(7736002)(305945005)(97736004)(51416003)(69596002)(7696005)(16586007)(16526018); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR0502MB3653; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM6PR0502MB3653; 23:dNECOJ1/wB0g3q/bECbdHu48aiZyDdkBfquFciQ?= NZ49kvNjqVa1yNmWqD588moSOpiZtFzCyt2zzTxVQbnLvkbQo84GywV/zEOG4JNtmZzeUKh+EEPkacY0B0xHD00n9gGQl/YvkpwM2vKPQ/S6kl2NM6PZ6boY+WVnik1iETlmedzdi+WQUS9V3J24XDS8mOWD5UFZqh8cg8ziaZXEEP8Kv0fEUONwm5Qgqk/BUQ+aARy5qnBMOVeejHaCVoOcKinVySqbm6UcZIfrAVyTZ8/kxZCw4hzEEowoeL6OxMSInb6jlGX786++xKLnfVVCvclOCfRelyZkpkRdlhGJHp91CRsA2c1ASbi2XlPdI+9DlNW69aBaBXWjge5+REsF+fCJvoDoS32aHdeYF8Gg88OcIuFJB+6E5HzYTMuSVtpIG0AXM28RVvABxh7yZCtw96UyVLfPWdmVz0hOj3U7ditmk4gKmjAckf5gbyU9GIExxMXfMVgBeckRxWRsd60S7cYSbNnoWCbsFiCXS8PB8D93cHUiCKc+AXCKRHuM7OPIqUwEWBoFMEjDsoRhcCU0DIkJgXf318I1Uq/wvYi1ka/0J6hVqVQbpuS9oNekhdie0uwAlmHwaqz1IDMSIsaTpPLuqsxI59gBi/0rx3LbFyrOZY6prnUXRFb/z3o0omAp2HTyoT4tdKsLVoh39ge84Zndw6tBpVM52jJwuZVWT0fbukczVH10LSvK+x9SE6wZGqMbLO+oNNmePYHiGXib42CeUlAIw+rkjtEKnNrr7FCcjPVN41fMejwMZYhXMtoc9lNEfx1VIdAyoBcrh6nmigGM/4L1jEDvbbFAJnGq4q/S8w7BghtQjCvpjh1PFqVjjnO/U6fAK4eckOiA5wkyI1oQf4OwdRZu1CzJWdiEzV5r3ZRQt4+/EOovitIfwLOp7eXcd8U6+5653HNxocZUg0Kfgnn8Bvlyp5ai05R3fnQyZldK2odS2EBdeAlKCRplGrTwnHLFzpSVoybnAlrS9TNmKlvsyzGteO5EhmGVaXCHHwHUkIe7/cP8tiagtL2zMD3U3pvMI9bw5g3Ne51jyezienv1GyTdMtkR9CJpNpvN/sFJ53WIWDNqw+jX/YzNZ9wBjft4Am5bbmRMpzodeYaV2bPfUdWr1rFc8lOFDc/8pwyjhfib0rEs4VIfHM5k= X-Microsoft-Exchange-Diagnostics: 1; AM6PR0502MB3653; 6:Bz3oCJmXI9KulqDebXIGZz3K3Y2BNMS0nz/oGgdZj0gCny4EI8E2Wq7Dc1XOhwaskXDE+D9JuifEzMLSd2GSRYni3QlcuqAvpQWEelSdsPNiN8eBPqWAn+s4NnCDaBT3vkez7WExMVOQ4gMvhz125gKbxuiMTGIj7n3Gcc2jc+QhEgvuKNdsQqD9ja96nfphnaoSui8EvYcnBigpfrKF2xJM9WK6+sZJxDBuJ0+d9AH3Y8d7iQvECZq2//CDIhB4JyrKOLZrV8l7MgumMmmKcXG9twi5S6+Xx3/X6gomiwSXHpikuVI5mEThw9ESvd2V5Z4DVFfWdwiWw9jEjjzF9lLfyqATZ5jr3vAseWP2BqY=; 5:nLZNcrTnMD8pHZdZNVdqbDXXEvon9Jg0y9iU01fNxU6O/Nxb+j3/mw9BsVi2ljeA0w/VlPe8OdK3BHYxtKJzAieSjzWCzqsbIuqd5YjaGoejiPV0/00Xx6kGFsTnA1IP7tu994kdreletFQf2Wym8u+f4pqRfXHw2gS72jwhqJo=; 24:zX1pG4eYJBYwOCs3IZWMUoFDxwnwQ37nI3w5qEIR8Zi8aFWvrmP4BH8APsW3qei/y11IHaHoNdsQ3c8sL4gNUDOBbNF9/s75FWbZWWWFGRs=; 7:j00EJn7fzTbpa6LZdR3PuVPfoMwv1clBg6lUeKqFk8wC2WkzB3NraUHswba01pa8cMrQO6M65m01JWIi1coi1lvaGfNL3TmlUfmiAC+u+kqT5eFHYwYn3r5s9/iP2tNeoxbycSh66CPeBD0yU+Yh3a9tboh88jpWZ9zSxAZpBstgzWR639w7UCDHkPo+S3vOqI+dvLLM4ShGwL8nwEFHGcUOOCa2EYVP+EW9HMDkGzsmTS8roxG9AL5pnL+6MxSi SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2017 12:20:07.8466 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a7e50842-cb83-49d8-158d-08d5365a5d5a X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR0502MB3653 Subject: [dpdk-dev] [PATCH 6/8] net/mlx4: mitigate Tx send entry size calculations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The previuse code took a send queue entry size for stamping from the send queue entry pointed by completion queue entry; This 2 reads were done per packet in completion stage. The completion burst packets number is managed by fixed size stored in Tx queue, so we can infer that each valid completion entry actually frees the next fixed number packets. The descriptors ring holds the send queue entry, so we just can infer all the completion burst packet entries size by simple calculation and prevent calculations per packet. Adjust completion functions to free full completion bursts packets by one time and prevent per packet work queue entry reads and calculations. Save only start of completion burst or Tx burst send queue entry pointers in the appropriate descriptor element. Signed-off-by: Matan Azrad Acked-by: Adrien Mazarguil --- drivers/net/mlx4/mlx4_rxtx.c | 105 +++++++++++++++++++------------------------ drivers/net/mlx4/mlx4_rxtx.h | 5 ++- 2 files changed, 50 insertions(+), 60 deletions(-) diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c index 0a8ef93..30f2930 100644 --- a/drivers/net/mlx4/mlx4_rxtx.c +++ b/drivers/net/mlx4/mlx4_rxtx.c @@ -258,54 +258,47 @@ struct pv { }; /** - * Stamp a WQE so it won't be reused by the HW. + * Stamp TXBB burst so it won't be reused by the HW. * * Routine is used when freeing WQE used by the chip or when failing * building an WQ entry has failed leaving partial information on the queue. * * @param sq * Pointer to the SQ structure. - * @param wqe - * Pointer of WQE to stamp. + * @param start + * Pointer to the first TXBB to stamp. + * @param end + * Pointer to the followed end TXBB to stamp. * * @return - * WQE size. + * Stamping burst size in byte units. */ -static uint32_t -mlx4_txq_stamp_freed_wqe(struct mlx4_sq *sq, volatile uint32_t **wqe) +static int32_t +mlx4_txq_stamp_freed_wqe(struct mlx4_sq *sq, volatile uint32_t *start, + volatile uint32_t *end) { uint32_t stamp = sq->stamp; - volatile uint32_t *next_txbb = *wqe; - /* Extract the size from the control segment of the WQE. */ - uint32_t size = RTE_ALIGN((uint32_t) - ((((volatile struct mlx4_wqe_ctrl_seg *) - next_txbb)->fence_size & 0x3f) << 4), - MLX4_TXBB_SIZE); - uint32_t size_cd = size; + int32_t size = (intptr_t)end - (intptr_t)start; - /* Optimize the common case when there is no wrap-around. */ - if ((uintptr_t)next_txbb + size < (uintptr_t)sq->eob) { - /* Stamp the freed descriptor. */ + assert(start != end); + /* Hold SQ ring wrap around. */ + if (size < 0) { + size = (int32_t)sq->size + size; do { - *next_txbb = stamp; - next_txbb += MLX4_SQ_STAMP_DWORDS; - size_cd -= MLX4_TXBB_SIZE; - } while (size_cd); - } else { - /* Stamp the freed descriptor. */ - do { - *next_txbb = stamp; - next_txbb += MLX4_SQ_STAMP_DWORDS; - if ((volatile uint8_t *)next_txbb >= sq->eob) { - next_txbb = (volatile uint32_t *)sq->buf; - /* Flip invalid stamping ownership. */ - stamp ^= RTE_BE32(0x1 << MLX4_SQ_OWNER_BIT); - sq->stamp = stamp; - } - size_cd -= MLX4_TXBB_SIZE; - } while (size_cd); + *start = stamp; + start += MLX4_SQ_STAMP_DWORDS; + } while (start != (volatile uint32_t *)sq->eob); + start = (volatile uint32_t *)sq->buf; + /* Flip invalid stamping ownership. */ + stamp ^= RTE_BE32(0x1 << MLX4_SQ_OWNER_BIT); + sq->stamp = stamp; + if (start == end) + return size; } - *wqe = next_txbb; + do { + *start = stamp; + start += MLX4_SQ_STAMP_DWORDS; + } while (start != end); return size; } @@ -327,14 +320,10 @@ struct pv { unsigned int elts_tail = txq->elts_tail; struct mlx4_cq *cq = &txq->mcq; volatile struct mlx4_cqe *cqe; + uint32_t completed; uint32_t cons_index = cq->cons_index; - volatile uint32_t *first_wqe; - volatile uint32_t *next_wqe = (volatile uint32_t *) - ((&(*txq->elts)[elts_tail])->wqe); - volatile uint32_t *last_wqe; - uint16_t mask = (((uintptr_t)sq->eob - (uintptr_t)sq->buf) >> - MLX4_TXBB_SHIFT) - 1; - uint32_t pkts = 0; + volatile uint32_t *first_txbb; + /* * Traverse over all CQ entries reported and handle each WQ entry * reported by them. @@ -360,28 +349,23 @@ struct pv { break; } #endif /* NDEBUG */ - /* Get WQE address buy index from the CQE. */ - last_wqe = (volatile uint32_t *)((uintptr_t)sq->buf + - ((rte_be_to_cpu_16(cqe->wqe_index) & mask) << - MLX4_TXBB_SHIFT)); - do { - /* Free next descriptor. */ - first_wqe = next_wqe; - sq->remain_size += - mlx4_txq_stamp_freed_wqe(sq, &next_wqe); - pkts++; - } while (first_wqe != last_wqe); cons_index++; } while (1); - if (unlikely(pkts == 0)) + completed = (cons_index - cq->cons_index) * txq->elts_comp_cd_init; + if (unlikely(!completed)) return; + /* First stamping address is the end of the last one. */ + first_txbb = (&(*txq->elts)[elts_tail])->eocb; + elts_tail += completed; + if (elts_tail >= elts_n) + elts_tail -= elts_n; + /* The new tail element holds the end address. */ + sq->remain_size += mlx4_txq_stamp_freed_wqe(sq, first_txbb, + (&(*txq->elts)[elts_tail])->eocb); /* Update CQ consumer index. */ cq->cons_index = cons_index; *cq->set_ci_db = rte_cpu_to_be_32(cons_index & MLX4_CQ_DB_CI_MASK); - txq->elts_comp -= pkts; - elts_tail += pkts; - if (elts_tail >= elts_n) - elts_tail -= elts_n; + txq->elts_comp -= completed; txq->elts_tail = elts_tail; } @@ -570,7 +554,7 @@ struct pv { if (max > pkts_n) max = pkts_n; elt = &(*txq->elts)[elts_head]; - /* Each element saves its appropriate work queue. */ + /* First Tx burst element saves the next WQE control segment. */ ctrl = elt->wqe; for (i = 0; (i != max); ++i) { struct rte_mbuf *buf = pkts[i]; @@ -644,6 +628,8 @@ struct pv { * that no ICRC should be calculated. */ if (--txq->elts_comp_cd == 0) { + /* Save the completion burst end address. */ + elt_next->eocb = (volatile uint32_t *)ctrl_next; txq->elts_comp_cd = txq->elts_comp_cd_init; srcrb.flags = RTE_BE32(MLX4_WQE_CTRL_SOLICIT | MLX4_WQE_CTRL_CQ_UPDATE); @@ -693,13 +679,14 @@ struct pv { elt->buf = buf; bytes_sent += buf->pkt_len; elts_head = elts_head_next; - elt_next->wqe = ctrl_next; ctrl = ctrl_next; elt = elt_next; } /* Take a shortcut if nothing must be sent. */ if (unlikely(i == 0)) return 0; + /* Save WQE address of the next Tx burst element. */ + elt->wqe = ctrl; /* Increment send statistics counters. */ txq->stats.opackets += i; txq->stats.obytes += bytes_sent; diff --git a/drivers/net/mlx4/mlx4_rxtx.h b/drivers/net/mlx4/mlx4_rxtx.h index c092afa..9d83aeb 100644 --- a/drivers/net/mlx4/mlx4_rxtx.h +++ b/drivers/net/mlx4/mlx4_rxtx.h @@ -105,7 +105,10 @@ struct mlx4_rss { /** Tx element. */ struct txq_elt { struct rte_mbuf *buf; /**< Buffer. */ - volatile struct mlx4_wqe_ctrl_seg *wqe; /**< SQ WQE. */ + union { + volatile struct mlx4_wqe_ctrl_seg *wqe; /**< SQ WQE. */ + volatile uint32_t *eocb; /**< End of completion burst. */ + }; }; /** Rx queue counters. */