From patchwork Sun Jun 3 15:29:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Moti Haimovsky X-Patchwork-Id: 40596 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id F0E4F374E; Sun, 3 Jun 2018 17:29:33 +0200 (CEST) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0054.outbound.protection.outlook.com [104.47.0.54]) by dpdk.org (Postfix) with ESMTP id 6FE712C16 for ; Sun, 3 Jun 2018 17:29:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gct7AYBR/vZK8ZEH90amH87PCCJXssLnVkmkjwMkBVE=; b=KQtO0SrHkx/9vGwU4Q84b7JV4G5CEUW/zO/h+m+0wU9U25dEMFy/RbFF+fFgIlO3+eEklR1loKp+/HeM476TAKkGr8szqczNZIewxhfoQ70SBXGTBautno12NuK1MtX+Wl2fXxbnxVF2PYRCsWPSwkEQ5K5e4X7kjpY0IRqE4BE= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=motih@mellanox.com; Received: from localhost.localdomain (37.142.13.130) by AM6PR05MB4440.eurprd05.prod.outlook.com (2603:10a6:209:43::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.820.11; Sun, 3 Jun 2018 15:29:29 +0000 From: Moti Haimovsky To: adrien.mazarguil@6wind.com, nelio.laranjeiro@6wind.com, yskoh@mellanox.com Cc: dev@dpdk.org, Moti Haimovsky Date: Sun, 3 Jun 2018 18:29:13 +0300 Message-Id: <1528039753-25889-1-git-send-email-motih@mellanox.com> X-Mailer: git-send-email 1.7.1 MIME-Version: 1.0 X-Originating-IP: [37.142.13.130] X-ClientProxiedBy: VI1PR0501CA0029.eurprd05.prod.outlook.com (2603:10a6:800:60::15) To AM6PR05MB4440.eurprd05.prod.outlook.com (2603:10a6:209:43::21) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:AM6PR05MB4440; X-Microsoft-Exchange-Diagnostics: 1; AM6PR05MB4440; 3:qGR91CwMLgUHny6cJBYAGbncAzF8d436JGw6yZdJ4uaigUKsvFZ5frvuay2oe0q9YHID1oshv045DemLQY8/W8QeS7+0xeJwlGgGpMZqJEtLX8fPeWjFIB/Drwwk5aZ6nweem+qIDaEILwTbLfjxlXoigMO9n732kUeiEd4HbZ4VLgqpfB26LbB+Tz/Su3MLw5GftKiR4EPvP87E/3GLH40MgVFS2wlv6xa5NxI1qFf2tqHI4A675EaibiNUQfCO; 25:Sa+CMPUEXeJ4/IKkeY00zyEjnyzCj2vJLI67IoALT+9rDFGyqJDxRupiGwF8WSzFuab0P0XcWZbWa+IfPLO0bb2p9gATVZ5qpMFBPWpZD18zhq77N1TFSzG4wv92OePxUidYgcwwfdbYVVhHh+4q3Y8NIsUO3yaotA1LLtfxb+G9g+wBhg+jBn/0a9ewP4A71jcG2kQ9Xi20tKbUPIQEtkaNflVzHd22o8M5pxfg0E2zGS+gFZfSQ1JtxDd05ilSFuGWM244Z/WpEEmuFkuWjeeF74c2IwPab5j+oi9vB3gWiH5+nAJph3qrkjoGYTJjDJ35BfY0TbLGNYzPCz8g0Q==; 31:1Y80+O8zH9dm6vfOAX/iSuLM06Cr+9kRRad7x3SY1mnzrimu4oLjVQ67pPhaxxZDhPhbYG3CN6/6+JbKdmymas/osG0RjZ3UYbbIcG1xW8I89Mmqr+ZT2AD5L2Jpb7NHvAdSdEpZsLhlrPY/WcSzRDn8lK8gAkfybgPt8fFQXuXe6eTGdExPhSsqBE9rLsMVWIMszojkhG9DQn4odd7upPrcA1BdcsjXPLESuKJ75V8= X-MS-TrafficTypeDiagnostic: AM6PR05MB4440: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM6PR05MB4440; 20:U1U1gTncD924TDD4W2wn4EAutGuZCPL9JoZKJAOogwu7m//kWOOMC48pGs3DjbWDVTkqMDHMXLstzHJyY50ygVLvxdBake7uPuFGd/yu1UlfZ0EnmUVoIy+volKijullSbcHXyxykRGL4z5jPREQLIc2ufeNsGg4xTtykDIVhItqp8RnU54DZUeS9yoamWY1y6zGGBmYmeypdh9Zz/b/ZW2g8hX9CQF3GZ7SgKMnONP19fZZX45pgoQkWmnEMmHzqOii4rQGUeN6ASaT/CLvLQgU04myeU1zH+vDwQxuhWt873KuqxiyTITpKGCGYnBwDZ+GyZ/chkU+8xsGbTNLAr/05FuvTldE2CYUKgKGlpI21IASPysrkXpm3PvtdRtLImz7wwdNxtQkQmpXNpvN4HhSuYorsfJO3KXZPwoY1mOiG5ztnxE/mtpelEw49ysZigShZfgNlB5BqI2qmxEZPy4wCYoNoh1xcIZv4sVaek+wL0T2IIKd4Zg6sqw9Leo7; 4:qRvDJZ7qtapUs8yXQlfCiWt+VwiZJeZAvPEFmgSgM0y3pvNOT6q121sB4MjUG2NBWzmi7+zPzMD12OeNx2pzl9r4ID50UiKFH1/JNHuwCjTidAXV3vTyuGe40C1C9zTghlHF5wPfqUpcGHhkBreQXRpNC5gB1EYNs0pNHiAdEUOaIlg9WReumTIL2cuO3gBp5eL5fliy7x8xCKIBs4EwA59WyTcs4VLHm9ogcMydtLvGKlCE7m/Nn+buWl7P+T5TC7uTOvLNesVaI8LcumszcA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(3231254)(944501410)(52105095)(6055026)(149027)(150027)(6041310)(20161123562045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(6072148)(201708071742011)(7699016); SRVR:AM6PR05MB4440; BCL:0; PCL:0; RULEID:; SRVR:AM6PR05MB4440; X-Forefront-PRVS: 069255B8B8 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(376002)(346002)(396003)(366004)(39380400002)(39860400002)(199004)(189003)(8676002)(52116002)(6512007)(81166006)(3846002)(6116002)(26005)(386003)(6506007)(68736007)(59450400001)(51416003)(107886003)(66066001)(47776003)(16526019)(50226002)(478600001)(7736002)(305945005)(86362001)(5660300001)(53936002)(4326008)(48376002)(8936002)(316002)(6486002)(6636002)(5890100001)(16586007)(106356001)(6666003)(105586002)(97736004)(186003)(50466002)(2906002)(36756003)(81156014)(956004)(486006)(476003)(25786009)(2616005); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR05MB4440; H:localhost.localdomain; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM6PR05MB4440; 23:yHfP6tA7OD/tDqpoKQyUrMEaVuRl+3tiSJJ/LNAFZ?= Q9G8QxlTkWTdKOmON+714KJSS2uAJt9br/I3zb5FFOrmGGI/NmI5AgxkE845sxBFT+uHXq7/PV3d+hGsdCRdWxFd75mvN5M8UEdPISF9MC6p1tyHH2p439HppmFmFFwe11Sy6V+BNjIlXITH5mM+8bIFGIfXZBYVPzQzOMgYWF2pO14fqvOEi+57FRK4qU9THetbJqkotxWLseFdRlXDeccry9qpH8Dr+RUlC5K/C+cH6F4p3sbj9HUQUUEAtI+9VQdG6Qgrw6hjbXJRNs08nhXPH5oNhxnbh0kcliEdvuQZ/2YWSoSd5hIzdegq7RtsGu6X1nPCt7TIqReNT818CQxFxKwklDRoVD4YFNr3fvVDxEP7A5XQ0Mw8MZs3ieT8feapieXAog3Dx+TVx1Impvi6u3PNZOW9ucXmT1k41GoetsgK3xzrjDG/jZBM1enlsAJKolK4dVUxl6QJmWBokHTvFgInTc8Rdcbi/utxXDq61x68Go2RaI6jgKxslFkTLmsToM3KyVi5T6yJ9voxGVTa4hueiGCUPL1wonohbgp0VlSWR4sd2D33ynDNoYkPM6O72Zj8IzGhqEu1iv29Qu43gVO7PY76C8FmfzEXk2rMmAJc5HJWtw0B8AzArGeKq1YD9cGiYYiKybiDaoROcybn+WGzb1eUdGfJQT9WObLDx8/jKO6CHd1P9KPxv7jrHBfeAX6l7Roghb5+p5hdIeelRRVehdokM6yUEf1kmsvHCYcPcFAcjYRnV1F0yQmHrnUKgkPK4SLitQyNVL/mPxOY/ZzxzgFI9gslnTZ/QUQB6k7mCVcsHQ2zcIaUKuUYhbLuG1p1zrES1Sc3LBT2GSGMyKyq3Ancqi5140DDMqhpmamSJGVNw18sVDnl7on6VhOyyGp0PbV7ytKnQXAwWWg4bhb7tzBC8+bry7eAyIV38+gGwQLyb/U3bVCMrsKoEoZ8VAQ6pDoop1GlY0U7hUlZV4QT8mSAZjJeec3OuAvFeg1AS4SynFzwvj9Y86pBpouFdskxayKBQBSvTpSS2frDrrv3LtWRMqo/E0LOhFCwLtNcFMx3cgFtIp6GbftnvLYvKAWWtp5aVQ73WcClwGFfnd+RAv9udtd3QW9zHYnBsc62rfUeiDDxxtiGhpTqs03aRvkAgf8fiBxHu0UQVN5 X-Microsoft-Antispam-Message-Info: rzyeL1TnJTUSTQ7vycIo02NxDf566FABsCGD6PF+fL4tID/JBqx1mwnD9ycCSEvhH9zHSxalPNgf1+P1bUpqHdSRkOFh7zwX0gGZx5X3NfDxgGXFj4L2eSZ94GYenYUM3OTGvFCmNu6Wmdu+F2sVTbfFbYhm55L/OB/TJtEp9wS37dym9CsuLDN7oR6Aid25 X-Microsoft-Exchange-Diagnostics: 1; AM6PR05MB4440; 6:zCmqNtHYw/WFWAK3iiLSjEpV1M6dLkI5O3M+YaMldgos8Hpo+zvpyZDbxPHn3a9U1F3mRBVqT8vfohzNH0NRX/GyRbit0614FyEwp4REu8HEIlfx9hYJGQmS6/uYMFWaCDERjG8AbiXPvmUKPh37vXEABu8Pz6yBHErCPC6O8Fi9Hsia9L5V2mBe8iABN5HnMw4jLH0WXQ/OJG8RWvuXiq04P1sl/UHPvzpnNiTR6EzG5Q4kK37QzbWI4YJ+2seXbJva3QdJvvzRg7fA3oY6PeOln1mvHMRwAvFh3u+7Bm47crKyGU3NiMTMONxMYOnYwkq8VFeDGA1WhZ8gHx/sW9+YQWD9rxOrMmcFpHUTQex/hAuiW98fXzkBQjtmgiDLJrj3Rj7LNCnxSc0jH7EVxMfbxp+oM303ApsVjv0JdkZf7VpHzLfyG65OU2lHLTB3Pjj7G8tT7vjlkv6bNazRQw==; 5:8N1+SLyXFZayLGsbMtwb/6Ykv8fGoTZxSj983d3F5Erd1h7//0W6+syCbGJFemJjMExxzi8NQRB4pwNXAulGav/fu6OO4gwQWDnssPBuwrjgcKPEfjRT8qiPG9wHi2uiMe846tWZBBleyl0QwjfZC6QeCXI6g9fyaV7yaxro6l4=; 24:I3D3Z1sgSyu0ViRWU8KLxVHy7yGBsoJi4iiPr3lo7Hp+anHNli2hULuX6tdfnlnxckhsypJbBTzrULExBIXcX8yj7nWX/F/Oi1NHSmACJcE= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM6PR05MB4440; 7:wUE1TT2xAGPL9gGkNbsUdkjt2e2F9U0nPxQ+O8wycRRBmcOT+/XNLe3qY1ghUHE2xlmRkWPVfgQDYVCK3bc1yrZDGZCw12wwF0wGvMuQeqf+LZRfH6oTmvFCX/Y1o+j3E4cu/LZBUWNGmAEQBL0g7O/w4IB3wwh0uTxcSgoiHvwJcl3SpjYZ13SIHJo8z99ylUF0z2ggCFlcxB8Dc+gFUv9CzCqzSijxPJKWeWlvbhDOyTgx1do9H/G44pYm9WnN X-MS-Office365-Filtering-Correlation-Id: 9b0e9d4a-86a9-42db-2a2b-08d5c966ccf7 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2018 15:29:29.6474 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9b0e9d4a-86a9-42db-2a2b-08d5c966ccf7 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB4440 Subject: [dpdk-dev] [RFC] net/mlx5: add support 32bit build X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This RFC and patch aim to add support for building mlx5 PMD on 32bit archs such as i686. The attached patch was not tested on 32bit environments and is given here for explanatory purposes. Prerequisites: * RDMA-core with 32bit support libraries. * compile and build tools for 32bit systems according to DPDK documentation. Assumptions: * Kernel and drivers are 64bit. * Only user programs and support libraries are 32Bit. Changes proposed in the PMD for adding 32bit support: The attached patch provides the changes needed in the PMD in order to support 32bit builds with the major changes being: * Modifying the UAR access routine to support non-atomic 64bit writes according to the mlx5 hardware reference manual. * Handling 64bit address fields that are written to the device WQEs. * Modify UAR address and size mapping to fit into the 32bit address space. Concerns: * The major concern of this modification is the UAR mapping. Current PMD assumes 64bit address and therefore feels free to map 4G size UAR to virtual address just above the huge-page addresses which of course cannot work for 32bit address space. As a workaround I reduced the UAR size to half the original size but this of course is not the solution. * Will vectorized support work on 32bit builds ? I am interested in your inputs regarding the concerns listed above, especially the UAR mapping issue mentioned there. Signed-off-by: Moti Haimovsky --- doc/guides/nics/features/mlx5.ini | 1 + drivers/net/mlx5/mlx5.c | 29 ++++++++++++++++++++++++--- drivers/net/mlx5/mlx5_rxq.c | 2 +- drivers/net/mlx5/mlx5_rxtx.c | 12 +++++------ drivers/net/mlx5/mlx5_txq.c | 2 +- drivers/net/mlx5/mlx5_utils.h | 42 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 77 insertions(+), 11 deletions(-) diff --git a/doc/guides/nics/features/mlx5.ini b/doc/guides/nics/features/mlx5.ini index e75b14b..b28b43e 100644 --- a/doc/guides/nics/features/mlx5.ini +++ b/doc/guides/nics/features/mlx5.ini @@ -43,5 +43,6 @@ Multiprocess aware = Y Other kdrv = Y ARMv8 = Y Power8 = Y +x86-32 = Y x86-64 = Y Usage doc = Y diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c933e27..6e9566d 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -110,6 +110,28 @@ /** Driver-specific log messages type. */ int mlx5_logtype; +static rte_spinlock_t mlx5_mmio_spinlock = RTE_SPINLOCK_INITIALIZER; + +/** + * Provide an emulation for 64 bit store on arch that does not have a 64 bit + * atomic store command. This is done using two 32bit store operations in + * address ascending order while holding a global spinlock. + */ +void +mlx5_mmio_write64(uint64_t be_val, volatile void *addr) +{ + uint32_t first_dword = + rte_cpu_to_be_32(rte_be_to_cpu_64(be_val) >> 32); + uint32_t second_dword = + rte_cpu_to_be_32(rte_be_to_cpu_64(be_val)); + + rte_spinlock_lock(&mlx5_mmio_spinlock); + rte_write32(first_dword, addr); + rte_write32(second_dword, + (volatile void *)((volatile char *)addr + 4)); + rte_spinlock_unlock(&mlx5_mmio_spinlock); +} + /** * Prepare shared data between primary and secondary process. */ @@ -600,9 +622,10 @@ rte_memseg_walk(find_lower_va_bound, &addr); /* keep distance to hugepages to minimize potential conflicts. */ - addr = RTE_PTR_SUB(addr, MLX5_UAR_OFFSET + MLX5_UAR_SIZE); + addr = RTE_PTR_SUB(addr, + (uintptr_t)((MLX5_UAR_OFFSET + MLX5_UAR_SIZE) >> 2)); /* anonymous mmap, no real memory consumption. */ - addr = mmap(addr, MLX5_UAR_SIZE, + addr = mmap(addr, MLX5_UAR_SIZE >> 2, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { DRV_LOG(ERR, @@ -644,7 +667,7 @@ return 0; } /* anonymous mmap, no real memory consumption. */ - addr = mmap(priv->uar_base, MLX5_UAR_SIZE, + addr = mmap(priv->uar_base, MLX5_UAR_SIZE >> 2, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { DRV_LOG(ERR, "port %u UAR mmap failed: %p size: %llu", diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index de3f869..6fc5a55 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -643,7 +643,7 @@ doorbell = (uint64_t)doorbell_hi << 32; doorbell |= rxq->cqn; rxq->cq_db[MLX5_CQ_ARM_DB] = rte_cpu_to_be_32(doorbell_hi); - rte_write64(rte_cpu_to_be_64(doorbell), cq_db_reg); + mlx5_write64(rte_cpu_to_be_64(doorbell), cq_db_reg); } /** diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 5278594..1e38841 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -715,8 +715,8 @@ *dseg = (rte_v128u32_t){ rte_cpu_to_be_32(length), mlx5_tx_mb2mr(txq, buf), - addr, - addr >> 32, + lower_32_bits(addr), + upper_32_bits(addr), }; ++ds; if (!segs_n) @@ -754,8 +754,8 @@ *dseg = (rte_v128u32_t){ rte_cpu_to_be_32(length), mlx5_tx_mb2mr(txq, buf), - addr, - addr >> 32, + lower_32_bits(addr), + upper_32_bits(addr), }; (*txq->elts)[++elts_head & elts_m] = buf; if (--segs_n) @@ -1591,8 +1591,8 @@ *dseg = (rte_v128u32_t) { rte_cpu_to_be_32(length), mlx5_tx_mb2mr(txq, buf), - addr, - addr >> 32, + lower_32_bits(addr), + upper_32_bits(addr), }; mpw.data.raw = (volatile void *)(dseg + 1); mpw.total_len += (inl_pad + sizeof(*dseg)); diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 691ea07..6832c82 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -276,7 +276,7 @@ } /* new address in reserved UAR address space. */ addr = RTE_PTR_ADD(priv->uar_base, - uar_va & (MLX5_UAR_SIZE - 1)); + uar_va & (uintptr_t)(MLX5_UAR_SIZE - 1)); if (!already_mapped) { pages[pages_n++] = uar_va; /* fixed mmap to specified address in reserved diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index 886f60e..d3c6bc2 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -13,6 +13,8 @@ #include #include +#include + #include "mlx5_defs.h" /* Bit-field manipulation. */ @@ -160,4 +162,44 @@ return l + r; } +/** + * A basic shift-right of a 64- or 32-bit quantity. Use this to suppress + * the "right shift count >= width of type" warning when that quantity is + * 32-bits. + * + * @param n + * the number we're accessing + */ +#define upper_32_bits(n) ((uint32_t)(((n) >> 16) >> 16)) + +/** + * lower_32_bits - return bits 0-31 of a number + * + * @param n + * the number we're accessing + */ +#define lower_32_bits(n) ((uint32_t)(n)) + +/** + * Provide safe 64bit store operation to mlx5 UAR region for both 32bit and + * 64bit architectures. + * + * @param ve_val + * value to write in big endian format. + * @param addr + * Address to write to. + */ +extern void +mlx5_mmio_write64(uint64_t be_val, volatile void *addr); + +static inline void +mlx5_write64(uint64_t be_val, volatile void *addr) +{ +#ifdef RTE_ARCH_64 + rte_write64(be_val, addr); +#else + mlx5_mmio_write64(be_val, addr); +#endif +} + #endif /* RTE_PMD_MLX5_UTILS_H_ */