From patchwork Tue Oct 31 18:21:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matan Azrad X-Patchwork-Id: 31069 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5A89E1B2D2; Tue, 31 Oct 2017 19:22:10 +0100 (CET) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0061.outbound.protection.outlook.com [104.47.0.61]) by dpdk.org (Postfix) with ESMTP id EFB291B267 for ; Tue, 31 Oct 2017 19:22:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=2gk/InMV9LDB139wqFoWR+jON9p0PWg6QiCFKHkKxWw=; b=CUj8A9TjYN/T6Ekhw9GFSR7ZtAHK7vpeSiEmitrBnfVf6k6P29H92gy8dAIGrX0jhWQ6CjANdp7Gjr4vrXUAZoQZNujxJb8fKX/Zc7mI7yUb35sYD2ufSaJGdhwONAvfMs6WMbjG0Py1IrBiNWaXI9SKq5BZ1HUoieRui9OMoqY= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; Received: from mellanox.com (37.142.13.130) by VI1PR0502MB3664.eurprd05.prod.outlook.com (2603:10a6:803:f::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.178.6; Tue, 31 Oct 2017 18:21:59 +0000 From: Matan Azrad To: Adrien Mazarguil Cc: dev@dpdk.org, Ophir Munk Date: Tue, 31 Oct 2017 18:21:32 +0000 Message-Id: <1509474093-31388-8-git-send-email-matan@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1509474093-31388-1-git-send-email-matan@mellanox.com> References: <1509358049-18854-1-git-send-email-matan@mellanox.com> <1509474093-31388-1-git-send-email-matan@mellanox.com> MIME-Version: 1.0 X-Originating-IP: [37.142.13.130] X-ClientProxiedBy: VI1PR0801CA0074.eurprd08.prod.outlook.com (2603:10a6:800:7d::18) To VI1PR0502MB3664.eurprd05.prod.outlook.com (2603:10a6:803:f::23) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9300e669-ad3c-4b2e-b036-08d5208c4727 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081)(4534020)(4602075)(2017052603199); SRVR:VI1PR0502MB3664; X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3664; 3:NDqqbLZuIAwyqLAzENjWM+bMAW9DVaXfR6G7kkxNsSL2BIBjNodu84zdUrxHKJ+gi6AUj7/TWeaRtZ+RBr7/a10WwQ4LrRMit61t3ccxGh9AbFJaw0XU51UpaLjczVfvFxqIBTnGY3B1V+D8rBqat7TnNBTGp3yrN3Xb5vDcA4SW7qXEq/n9vu156kJZzEgLtjHXk0qm7Jz1TFE1RGHq/IRe7HawGH2/RWF7X9hPHG/KzumJ54MwOToBhbYPvq+9; 25:4QBRQyBFVC3JBqCcAuqW7+8OmXD8RVHcZFi2x+6usgPu2zxtw8o39zzMubGtL/bf/1VUPK9PG/TfYxoNfIQQok2NtE4k1MW1/wp75QAtj6J27wIAvoqN5BTxXLQNllxdvAVcms3zjjJWriQlBFkoUCroY65mLVHo+3O+mQ44AG4czfkPomWx5B5e2G2not8awkUkY0PGmzt10veBDAoEqtJe+z0HhZS6lcblpGX85uuLi8tq+VuRdJJ6YhRMi80MokX10NZXCRySyP1CRGbhd94ayHI5P8w/1YzyGMj+PSdjbgvLueYiAp+QjpQHJPDwuzz1WNn0LSM6AdWmN0Zf6A==; 31:mmzSBTKgZwjjJfgbdOq5/u7fXtE83FK8EzqD4vIU8CqFKyyXCpxz838vKQxCLBKI7qmg5YoGTdaIfzfi55/r1xQ+/ymy0rPwFbEC9C+7khyXfwXhO7UVA2OJw5x+w8+f2+djPQMIuFSnrmZ+BbswkWx8P+r+/cfwh6LHQH8uPobo93W6WY8lH17BGPrAfHS6HmYZL2RzgeIRRqSH2GJboMoo93/8Jewve0ryLBtxhuU= X-MS-TrafficTypeDiagnostic: VI1PR0502MB3664: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3664; 20:5USIZgZPt8x5ukKDVnD6bfTieHZVB2BEJn82G6d2qaJ282pkOm87oshPVPIdptPWz+1iymMkAthCVhcJe/OydO7ndl2gebPCTLuMGyy95R5HuRwCHfXfoV/bHwk2nRoY3d9nAjdPbV1sOwGGoD3I/XhFGv9s5QuMrY1GQyCRHckxMqrkUUOL2PEHByZ29iCf87doYa9zs3pSHCboYZGxRCxmNar6MyjHraZXxdGbBupcYr2CbWKowb4y97VYuMb2CzaWQCgYD0nHSyZU0KUln/nS74qxEsjeK+novFQ5VLrgPGeJWn/NdpEkLgVX+BB81DVNVF1WzJsMnGXx63U3XQWwLgMZZ9rm3qb1+ZvGIETQCDK5WGkuFOc6xoOQpg6dWDGEvCkQQXiJitM7XvMNuc1nF39y07NRdHhXS5VFdaw2ElMMWkrDQCz9OeAbIvVH3YTKihn29Tm8ZC5jE9voY9eBGCpdpOkR31vUCdECvyf56SzLhEUS2NtS2MsLjPVk; 4:5MGCZt5TFj5O+d0huNZOxtRGMvv3H2PTfz0RH3M4Ek3SrayM7Ft/0aZ7stwThVCvv1UdqRHzdIfy+6HrJBpANJZ2XEQA9AXU0a8YfJPo2z33TUVU16amAfH2WPj38LOpDOxUEE8Tg35CCs5pYwNzcuK1ryLlorpPLMUOvS+KP82suvNScJ8Uh6qNnNN17gtHJPgwwQDXzj56TrdupnFZcgdSV2iJ2dMsTMtO7YCDKSJDaui4tF63C/jE3BWDv8EBl9051ev19DqLgX+1bb4x6Q== X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(3231020)(10201501046)(100000703101)(100105400095)(6055026)(6041248)(20161123555025)(20161123558100)(20161123564025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:VI1PR0502MB3664; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:VI1PR0502MB3664; X-Forefront-PRVS: 04772EA191 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(39860400002)(376002)(189002)(199003)(50466002)(97736004)(66066001)(478600001)(189998001)(36756003)(33026002)(25786009)(7736002)(316002)(107886003)(16586007)(4326008)(575784001)(86362001)(50226002)(53936002)(16526018)(101416001)(106356001)(5660300001)(2906002)(5003940100001)(69596002)(47776003)(6116002)(3846002)(68736007)(105586002)(81156014)(8936002)(81166006)(21086003)(305945005)(55016002)(4720700003)(2950100002)(76176999)(50986999)(6916009)(6666003)(48376002)(8676002)(33646002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0502MB3664; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; VI1PR0502MB3664; 23:uOBkaupxB/un6JjjqUhP4i/DRregwLf9RJ8eZLd?= uuva5GBlGQ0bO7P7lbzuHkck7gUbewWfY6iQ0fQCRXLf51bPfO31b0UfXDcwg7nIBneDQFeIqgsc0kyEhnauU3JkU/rnocY9Sr1PwvhUQagP3QwfCx2xHBplWsVAap9PJWww2Duo5lMSEP1t71IrJ75SP08Qz7QhkkA5wpw1t3kwnlsH8aQJMZUoSRzXZ4GAT4DegNAqa4A8lnzkaJ18/veRHRYk2rSbB5oYS4ro/RnyEViiskrUkwow5SeiU+HbGcEAYRfgQh85QcOAB2TwjaVK01KtOSqi+cXsw8y0E8Vep6s/f5xdlaatvCzKf3AJ+mVnUM+c5gwKCwFy7c5xdqbMtt0GbcgnMBeDTBRqG6twO10Wv2bVqt0U9VhYiI7nVeX8S45iBe3GmFH+DXMslERUAeBKi82ur91eosj9R92hK8gCySBtCYXkXRq0xp7e35UUSQXQHlZ/FI/CRC1VYlH90z8JnS57l0XqOI6v2ZDBDE4IffsCunlVhJXbmTnVvrHJBtdx7HNaCAUsrl4JcD7UJsu1tbYhClfsmKvCdxrS9q1XK3/jtyCH1wvAJn66auS1yzpnkbKzCyJSrtdFjoZoTqgebkwz89WWs2Ye+x3UW2JLb0o8IGPC4OdHoWCX7s61FZzk8hHqwbZbULolr/NTCz2luZJulEyH00reSWTpNKEEgCBeVeIgjJ0YPCmrDDwIU62Fthl5/KQ/YBUkKgnmMFs7DnSFUbqMk/MXx65lrxnBGH/dJXg9T3YsnF/wyLs6uPB9w/RQjrpVubQinbKr7LCw3/wes4GTaXwlY7oEqNcSDzDG0kxabjLuTbjEbq2eVEU1Tl/hcX8bJjEc8VMFYcXg/E8co+vnU7+HCXHIQBMURnagj/1fUeM+rz1pPvSEtTq9svcxwdaOQ9qLHikReeZxeciN+I8EWRFz6QPgABjg2LL+mYcgFV/dfeyQbABP+9ZECd14Y9NUMlVV6fGRGevlEI3byv3lGLJRDc2T4D3Tfqm1vfmk/vAxKFU4RCgIcXo/LtHMb21XT6Q0q1OW1ZvrOtNpJbFLZ+cy7bqVE7gN9aHwCzsdY8B7KCOVrTDMradFNHvfyq5sZdRmxlZ9mggkF+dYpyyFOD0g6BfL3ZLypPvBCrkFZODjo+a6sub8= X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3664; 6:quRWHrMZkFVwpk+DXrcbr8GeA3tnB9/s/jp7AqDoWFZfotf77ZmTcmTy/WPdL60z/hTGFj/AWk0IkG9QYgjSCWFkN94kUoFII6DRDEUZUzoqQn50YACExeFvcmztXPweztjfEoX2keOweFjHLg390NXFhZm/H6BczoDFS0Zz/R5m2ycEZv8SYRDaU36wuKoUV+e2Mj+PrmGHaaiZq6cKdx6vqVPcQmnCUP4+K1Y18X1VFY9SJcXCboCFcxkLbf9mKSeQagQtlf81wnD+HwdQS7AwBTqA7P693r67D69uwRZaxwgR2DJGSeGU0Y08KNTJxZeSmyKB5IMpp1M1fVVTg2UPtaXMpqvylxisZhNlcq8=; 5:/uLPU2xTPIwfdWIeSBMH9BISzQOWEiZYXUQN04pugZHByF2PQ2HWYeOajtI02ZjEwneAbLv+rWoPm67cOJk1J0ZlvEXvWD8sr9at4it8pg719FGwHFxkWbAqOPGuCuVV77RYsZEZoaWhY7jU1c62ZAU1DfUlsPuy/seiskt8lGs=; 24:wUPKeTI360N212ZDy0A735ecbdfzGTptOLdQEZ4RndPnt5bPCSogPX3A0x6uvyZfER9fAyvi3JJJSoJI2LVBQoFW03uMV/I7azOgyWO6084=; 7:nies04ZlR8vXxc0N8MJsP4gQbaaYDlhE+j53JANZSE9bVzLSnxh54pPC105yNdPM84FMhTZzMzd0YeLnU36h/2n1bSA1ihgpuU6WHdg7dy6UIqU95oiX/UplQOJDGVuQZHipOtg3/8xDSetpA6aYuTEtFM8j66/nBf41ZSRSL2mu8A0VKaGKSAWPU+lvvSGp8F5060/ewnwd0BdEAFNhcbUQQI9p207A1AxmwGjjHLYuS3bZ3o5pXco4m+iaoSyS SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Oct 2017 18:21:59.8149 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9300e669-ad3c-4b2e-b036-08d5208c4727 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0502MB3664 Subject: [dpdk-dev] [PATCH v4 7/8] net/mlx4: fix HW memory optimizations careless X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Volatilize all Rx/Tx HW negotiation memories to be sure no compiler optimization prevents either load or store commands. Fixes: c3c977bbecbd ("net/mlx4: add Tx bypassing Verbs") Fixes: 9f57340a8087 ("net/mlx4: restore Rx offloads") Fixes: 6681b845034c ("net/mlx4: add Rx bypassing Verbs") Fixes: 62e96ffb93ad ("net/mlx4: fix no Rx interrupts") Signed-off-by: Matan Azrad Acked-by: Adrien Mazarguil --- drivers/net/mlx4/mlx4_prm.h | 18 ++++++------ drivers/net/mlx4/mlx4_rxtx.c | 67 ++++++++++++++++++++++++-------------------- 2 files changed, 45 insertions(+), 40 deletions(-) diff --git a/drivers/net/mlx4/mlx4_prm.h b/drivers/net/mlx4/mlx4_prm.h index b0fd982..6d10b4e 100644 --- a/drivers/net/mlx4/mlx4_prm.h +++ b/drivers/net/mlx4/mlx4_prm.h @@ -80,14 +80,14 @@ enum { /* Send queue information. */ struct mlx4_sq { - uint8_t *buf; /**< SQ buffer. */ - uint8_t *eob; /**< End of SQ buffer */ + volatile uint8_t *buf; /**< SQ buffer. */ + volatile uint8_t *eob; /**< End of SQ buffer */ uint32_t head; /**< SQ head counter in units of TXBBS. */ uint32_t tail; /**< SQ tail counter in units of TXBBS. */ uint32_t txbb_cnt; /**< Num of WQEBB in the Q (should be ^2). */ uint32_t txbb_cnt_mask; /**< txbbs_cnt mask (txbb_cnt is ^2). */ uint32_t headroom_txbbs; /**< Num of txbbs that should be kept free. */ - uint32_t *db; /**< Pointer to the doorbell. */ + volatile uint32_t *db; /**< Pointer to the doorbell. */ uint32_t doorbell_qpn; /**< qp number to write to the doorbell. */ }; @@ -101,10 +101,10 @@ struct mlx4_sq { /* Completion queue information. */ struct mlx4_cq { void *cq_uar; /**< CQ user access region. */ - void *cq_db_reg; /**< CQ doorbell register. */ - uint32_t *set_ci_db; /**< Pointer to the completion queue doorbell. */ - uint32_t *arm_db; /**< Pointer to doorbell for arming Rx events. */ - uint8_t *buf; /**< Pointer to the completion queue buffer. */ + volatile void *cq_db_reg; /**< CQ doorbell register. */ + volatile uint32_t *set_ci_db; /**< Pointer to the CQ doorbell. */ + volatile uint32_t *arm_db; /**< Arming Rx events doorbell. */ + volatile uint8_t *buf; /**< Pointer to the completion queue buffer. */ uint32_t cqe_cnt; /**< Number of entries in the queue. */ uint32_t cqe_64:1; /**< CQ entry size is 64 bytes. */ uint32_t cons_index; /**< Last queue entry that was handled. */ @@ -128,10 +128,10 @@ struct mlx4_cq { * @return * Pointer to CQE entry. */ -static inline struct mlx4_cqe * +static inline volatile struct mlx4_cqe * mlx4_get_cqe(struct mlx4_cq *cq, uint32_t index) { - return (struct mlx4_cqe *)(cq->buf + + return (volatile struct mlx4_cqe *)(cq->buf + ((index & (cq->cqe_cnt - 1)) << (5 + cq->cqe_64)) + (cq->cqe_64 << 5)); diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c index 176000f..bd6d888 100644 --- a/drivers/net/mlx4/mlx4_rxtx.c +++ b/drivers/net/mlx4/mlx4_rxtx.c @@ -70,7 +70,7 @@ * DWORD (32 byte) of a TXBB. */ struct pv { - struct mlx4_wqe_data_seg *dseg; + volatile struct mlx4_wqe_data_seg *dseg; uint32_t val; }; @@ -98,14 +98,15 @@ struct pv { { uint32_t stamp = rte_cpu_to_be_32(MLX4_SQ_STAMP_VAL | (!!owner << MLX4_SQ_STAMP_SHIFT)); - uint8_t *wqe = mlx4_get_send_wqe(sq, (index & sq->txbb_cnt_mask)); - uint32_t *ptr = (uint32_t *)wqe; + volatile uint8_t *wqe = mlx4_get_send_wqe(sq, + (index & sq->txbb_cnt_mask)); + volatile uint32_t *ptr = (volatile uint32_t *)wqe; int i; int txbbs_size; int num_txbbs; /* Extract the size from the control segment of the WQE. */ - num_txbbs = MLX4_SIZE_TO_TXBBS((((struct mlx4_wqe_ctrl_seg *) + num_txbbs = MLX4_SIZE_TO_TXBBS((((volatile struct mlx4_wqe_ctrl_seg *) wqe)->fence_size & 0x3f) << 4); txbbs_size = num_txbbs * MLX4_TXBB_SIZE; /* Optimize the common case when there is no wrap-around. */ @@ -120,8 +121,8 @@ struct pv { for (i = 0; i < txbbs_size; i += MLX4_SQ_STAMP_STRIDE) { *ptr = stamp; ptr += MLX4_SQ_STAMP_DWORDS; - if ((uint8_t *)ptr >= sq->eob) { - ptr = (uint32_t *)sq->buf; + if ((volatile uint8_t *)ptr >= sq->eob) { + ptr = (volatile uint32_t *)sq->buf; stamp ^= RTE_BE32(0x80000000); } } @@ -150,7 +151,7 @@ struct pv { unsigned int elts_comp = txq->elts_comp; unsigned int elts_tail = txq->elts_tail; struct mlx4_cq *cq = &txq->mcq; - struct mlx4_cqe *cqe; + volatile struct mlx4_cqe *cqe; uint32_t cons_index = cq->cons_index; uint16_t new_index; uint16_t nr_txbbs = 0; @@ -161,7 +162,7 @@ struct pv { * reported by them. */ do { - cqe = (struct mlx4_cqe *)mlx4_get_cqe(cq, cons_index); + cqe = (volatile struct mlx4_cqe *)mlx4_get_cqe(cq, cons_index); if (unlikely(!!(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK) ^ !!(cons_index & cq->cqe_cnt))) break; @@ -172,8 +173,8 @@ struct pv { #ifndef NDEBUG if (unlikely((cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) == MLX4_CQE_OPCODE_ERROR)) { - struct mlx4_err_cqe *cqe_err = - (struct mlx4_err_cqe *)cqe; + volatile struct mlx4_err_cqe *cqe_err = + (volatile struct mlx4_err_cqe *)cqe; ERROR("%p CQE error - vendor syndrome: 0x%x" " syndrome: 0x%x\n", (void *)txq, cqe_err->vendor_err, @@ -240,15 +241,15 @@ struct pv { static int mlx4_tx_burst_segs(struct rte_mbuf *buf, struct txq *txq, - struct mlx4_wqe_ctrl_seg **pctrl) + volatile struct mlx4_wqe_ctrl_seg **pctrl) { int wqe_real_size; int nr_txbbs; struct pv *pv = (struct pv *)txq->bounce_buf; struct mlx4_sq *sq = &txq->msq; uint32_t head_idx = sq->head & sq->txbb_cnt_mask; - struct mlx4_wqe_ctrl_seg *ctrl; - struct mlx4_wqe_data_seg *dseg; + volatile struct mlx4_wqe_ctrl_seg *ctrl; + volatile struct mlx4_wqe_data_seg *dseg; struct rte_mbuf *sbuf; uint32_t lkey; uintptr_t addr; @@ -256,8 +257,8 @@ struct pv { int pv_counter = 0; /* Calculate the needed work queue entry size for this packet. */ - wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) + - buf->nb_segs * sizeof(struct mlx4_wqe_data_seg); + wqe_real_size = sizeof(volatile struct mlx4_wqe_ctrl_seg) + + buf->nb_segs * sizeof(volatile struct mlx4_wqe_data_seg); nr_txbbs = MLX4_SIZE_TO_TXBBS(wqe_real_size); /* * Check that there is room for this WQE in the send queue and that @@ -269,17 +270,18 @@ struct pv { return -1; } /* Get the control and data entries of the WQE. */ - ctrl = (struct mlx4_wqe_ctrl_seg *)mlx4_get_send_wqe(sq, head_idx); - dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl + - sizeof(struct mlx4_wqe_ctrl_seg)); + ctrl = (volatile struct mlx4_wqe_ctrl_seg *) + mlx4_get_send_wqe(sq, head_idx); + dseg = (volatile struct mlx4_wqe_data_seg *) + ((uintptr_t)ctrl + sizeof(struct mlx4_wqe_ctrl_seg)); *pctrl = ctrl; /* Fill the data segments with buffer information. */ for (sbuf = buf; sbuf != NULL; sbuf = sbuf->next, dseg++) { addr = rte_pktmbuf_mtod(sbuf, uintptr_t); rte_prefetch0((volatile void *)addr); /* Handle WQE wraparound. */ - if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob) - dseg = (struct mlx4_wqe_data_seg *)sq->buf; + if (dseg >= (volatile struct mlx4_wqe_data_seg *)sq->eob) + dseg = (volatile struct mlx4_wqe_data_seg *)sq->buf; dseg->addr = rte_cpu_to_be_64(addr); /* Memory region key (big endian) for this memory pool. */ lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(sbuf)); @@ -395,8 +397,8 @@ struct pv { struct txq_elt *elt_next = &(*txq->elts)[elts_head_next]; struct txq_elt *elt = &(*txq->elts)[elts_head]; uint32_t owner_opcode = MLX4_OPCODE_SEND; - struct mlx4_wqe_ctrl_seg *ctrl; - struct mlx4_wqe_data_seg *dseg; + volatile struct mlx4_wqe_ctrl_seg *ctrl; + volatile struct mlx4_wqe_data_seg *dseg; union { uint32_t flags; uint16_t flags16[2]; @@ -433,15 +435,18 @@ struct pv { break; } /* Get the control and data entries of the WQE. */ - ctrl = (struct mlx4_wqe_ctrl_seg *) + ctrl = (volatile struct mlx4_wqe_ctrl_seg *) mlx4_get_send_wqe(sq, head_idx); - dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl + + dseg = (volatile struct mlx4_wqe_data_seg *) + ((uintptr_t)ctrl + sizeof(struct mlx4_wqe_ctrl_seg)); addr = rte_pktmbuf_mtod(buf, uintptr_t); rte_prefetch0((volatile void *)addr); /* Handle WQE wraparound. */ - if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob) - dseg = (struct mlx4_wqe_data_seg *)sq->buf; + if (dseg >= + (volatile struct mlx4_wqe_data_seg *)sq->eob) + dseg = (volatile struct mlx4_wqe_data_seg *) + sq->buf; dseg->addr = rte_cpu_to_be_64(addr); /* Memory region key (big endian). */ lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(buf)); @@ -633,7 +638,7 @@ struct pv { * CQE checksum information. */ static inline uint32_t -mlx4_cqe_flags(struct mlx4_cqe *cqe, int csum, int csum_l2tun) +mlx4_cqe_flags(volatile struct mlx4_cqe *cqe, int csum, int csum_l2tun) { uint32_t flags = 0; @@ -666,13 +671,13 @@ struct pv { * Number of bytes of the CQE, 0 in case there is no completion. */ static unsigned int -mlx4_cq_poll_one(struct rxq *rxq, struct mlx4_cqe **out) +mlx4_cq_poll_one(struct rxq *rxq, volatile struct mlx4_cqe **out) { int ret = 0; - struct mlx4_cqe *cqe = NULL; + volatile struct mlx4_cqe *cqe = NULL; struct mlx4_cq *cq = &rxq->mcq; - cqe = (struct mlx4_cqe *)mlx4_get_cqe(cq, cq->cons_index); + cqe = (volatile struct mlx4_cqe *)mlx4_get_cqe(cq, cq->cons_index); if (!!(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK) ^ !!(cq->cons_index & cq->cqe_cnt)) goto out; @@ -717,7 +722,7 @@ struct pv { int len = 0; while (pkts_n) { - struct mlx4_cqe *cqe; + volatile struct mlx4_cqe *cqe; uint32_t idx = rq_ci & wr_cnt; struct rte_mbuf *rep = (*rxq->elts)[idx]; volatile struct mlx4_wqe_data_seg *scat = &(*rxq->wqes)[idx];