From patchwork Thu Nov 5 16:38:16 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jerin Jacob X-Patchwork-Id: 8723 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 02C7691AD; Thu, 5 Nov 2015 17:40:01 +0100 (CET) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0085.outbound.protection.outlook.com [157.56.110.85]) by dpdk.org (Postfix) with ESMTP id D7A329201 for ; Thu, 5 Nov 2015 17:39:58 +0100 (CET) Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@caviumnetworks.com; Received: from jerin.caveonetworks.com (111.93.218.67) by CY1PR0701MB1977.namprd07.prod.outlook.com (10.163.141.19) with Microsoft SMTP Server (TLS) id 15.1.312.18; Thu, 5 Nov 2015 16:39:55 +0000 From: Jerin Jacob To: Date: Thu, 5 Nov 2015 22:08:16 +0530 Message-ID: <1446741498-3096-14-git-send-email-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 1.9.3 In-Reply-To: <1446741498-3096-13-git-send-email-jerin.jacob@caviumnetworks.com> References: <1446741498-3096-1-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-2-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-3-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-4-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-5-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-6-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-7-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-8-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-9-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-10-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-11-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-12-git-send-email-jerin.jacob@caviumnetworks.com> <1446741498-3096-13-git-send-email-jerin.jacob@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: MA1PR01CA0067.INDPRD01.PROD.OUTLOOK.COM (25.164.116.167) To CY1PR0701MB1977.namprd07.prod.outlook.com (25.163.141.19) X-Microsoft-Exchange-Diagnostics: 1; CY1PR0701MB1977; 2:Vbq0Vvx2E6JXXT/3aAU+5yi8k78q3Y/t4MM6y+1J+CzZiP8QD09UYgytw7ZlpNejvb6xh5FOGYA0BLleSk5Kz7Fd7xf+KaE9jlX0Mbrm2C66C72xcEMrZlO3ILsDQ7pcTeCcAFHJM8iz8cuajtjB0b446zSYtq9tHLzvJ5XJzpY=; 3:vff2hz9j7vpsivyVYQU40+Zl8/73He1uQmLq55ymx8I6fzWy7/gY0XFDNUVUbB5LvVEOPWQP3byt6TjlEB80taPAVloD6C/HUuyKuCKhU6s3xjBBQuSJbxjS2xMpx9VsUpxeeAu9aOhvGnlmYfwS0Q==; 25:3uD0ocCEDEPTZ0QF4gVIOfiu9Fh7yker3px41iDAhlWk8yy7VGrEekABHmMpEWk72OA3WETpBm94KHzazXwUHUQ+hJ0swceXFKg8DzG7/TW+MyG04gqOssQff56A7XAD1uAag96MApO8A/iS0xBrsS/uaHhkxNGi529PCUzFchfVarFe0GJOFNVLANCwEwPbgy/xyFGlceDoWRo6h0SgVqAtYE4f5XqS9QVx9inxmJao80bANX6D1i90YeNvK3ccBsk1zsAJrGSJ684GuXj+IA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0701MB1977; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0701MB1977; 20:wDD6ZZ513Q1MkUKCG1EncYlr7IgRjGn+bGFn/90E80R2K9UJftleMA75PZr8e7nBaaR8JUHYJA2UcGKWmDiHQbYe/zrhfpf8fBw9Pfd2H54F+nqLpTrcXJZrLP1uU4/pC9L7x4S+z4hiatCn2R05nTO5Yk0H6z2z+HN9fAU3XOQzilfiAa25gW8S58JzPkabL8MLSeFJW4VDiwsLq8tgG/4jP6dFpnYnZD2x8zlJV9QmiuoIx9tL2PsvEsAEWvhyyHw2CFjmDcJgpBfFpNFz0K7mRPocfX6TnMzQRAfpwTJrBK45hqQV5SlMHv2NU7hXVBLON1DMINAftV+09FpzsPlEZ313sC0cmTttdgzoBvNZlYE3D50TNYM2rQ7f1BsO9BOihHPmnuX4ghl/csYB3xKITge0Hp7/iW9K8ARrFiXGilq7ZfzqPhfI+xU2iPZzPluVyJayL2qvma7wEqpOsggRXoCUunV4rjKSe6sPMcQ7adSz1uTE45hg/6bmHImdgQ6Cu5fR6y2rZWBNQBnEHe9y4GpVQW+4MOCQ2ITDfilDMGxw2TS625mGdZeqpid26JOvVu0YUqbTQ2YxN84hLYMMnBMtSewBanm69ZkhLcc= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(236414709691187)(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001)(10201501046); SRVR:CY1PR0701MB1977; BCL:0; PCL:0; RULEID:; SRVR:CY1PR0701MB1977; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0701MB1977; 4:JsNSV6qVAuSM3LDIrcrL9V1efP+M9nzn8nJ0eldihlbyIC2aEBxSc6hq/njV6ozZdcYiaSWNBsNfjaa2QHE0wKq/fUXnpFK95nKxF9zIRBQrEdzpQPfD1fTNh+NbVw4psJ5q2BMjUsdskB1dmOBq1kBITv0RTVAUkDTvq374pIjJ+kqtOLeWbjJSMFTjFtXeDl4UiWfRHk0dSlchMJczGiC1N1uxpV4ret5F4l+SSovSyoYdbX9ek1+yu9GcHl+Zw7hJRPztvILTR+a55uxWEZ+uYoA6Nhc6hDhTwJOel+RKctYgsVClwMAVw/U9Z0SLNmlRH6XbyVdq/1ozVYdpckaKFjeXPUGFFz6hdAD3w8gF4zEynkglJSK9U0YMnD/g X-Forefront-PRVS: 0751474A44 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(189002)(199003)(5007970100001)(86362001)(5001960100002)(107886002)(36756003)(122386002)(40100003)(189998001)(42186005)(5008740100001)(575784001)(87976001)(110136002)(97736004)(66066001)(101416001)(92566002)(2950100001)(5009440100003)(4001430100002)(19580405001)(76176999)(5003940100001)(50986999)(50226001)(50466002)(106356001)(77096005)(105586002)(5004730100002)(2351001)(81156007)(229853001)(48376002)(33646002)(19580395003)(69596002)(53416004)(93886004)(47776003)(7099028); DIR:OUT; SFP:1101; SCL:1; SRVR:CY1PR0701MB1977; H:jerin.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: caviumnetworks.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY1PR0701MB1977; 23:q+XBZAHDXP0yoxVAavzfOs8Js6P9qW1nHzxWNeA?= =?us-ascii?Q?nDRvR3ATprLABEMQKAsHPQ3PYv+DTx5uzN5VGQTAWVtjdLTz1ryp20vdD1AO?= =?us-ascii?Q?xGmy4lye78MgrVyLSsv/ZJSwQhtOrEZ4R5GfywzXUfSbsznRFUP1y4zJMQNj?= =?us-ascii?Q?ceVelQCyBUu+R5Lwy0cWTUau0ec2EBr2u8APbT12s2wweEpbQnSMulwxuhWF?= =?us-ascii?Q?XH/KoAFjQ9nP/5WNhMc9PfslhY8eExNQ4cjHviFdr/Ne+TtN5sVdjq3covbA?= =?us-ascii?Q?fXcInisKPVFC9gQ2ULU5Dxc1P2+mhyAaL5bbYyBoz4cK9vHUjVZyFAcIjXje?= =?us-ascii?Q?muU30KBW+tzmIw7TfYPo3x5hxxlyqv0AvIPVjbNK1vviWbfUtYuni0Fv+H79?= =?us-ascii?Q?8wGkrOhLo2FHtmVxhjlJthO6GHpkS9smTh+XkQuCwNG+HIxEhtCYdhExCO/a?= =?us-ascii?Q?0Q3Q5sR1fVaR1XpLkFpxPymbvvSiRdbm+KedGDiAH2lNQ4g4g07/O/fFit1S?= =?us-ascii?Q?O+gctTGFkvyCdYgmVjMkuAaOvB+Fo5Ea/1tunberhJLgk+e/v7/WdxWct82U?= =?us-ascii?Q?h1jkNUZVeW4kYXcvC8T7GwKdcX5MzkM5uyME7s95iByXbmZFsx12kd3W3CYD?= =?us-ascii?Q?8Xd1lFZCMR+ZfcOL5Ctc//swsFttqp/Q6O8+QRKMImCLXUgd8ymdoRH8jtlZ?= =?us-ascii?Q?AtsDHORRKkRwby/iNp/kThj02ikpaQp9dHxOV7mC+shoVEFCGGo7eiSff5O/?= =?us-ascii?Q?0uocKnL8COm0fWFE8RgVbbsDl2kmaVB30mvoAfUzXjRmpChQrgc8zRPZZfpQ?= =?us-ascii?Q?8AL8rL+2+At0exP5kxJqd7BDXsOzPmak8RElhSIyMoAcH8rhWS86SrLeGzGP?= =?us-ascii?Q?OY2oZRKbFTJQSYln/usrBQLcu6TU30IngjKt7/DDzVZ0UWaSfilVIFwHxKMl?= =?us-ascii?Q?BCCoeB1N7hfXQ+7N+gXCs1nETX1ht9Pxq5W+7hzyISy6UsVrgUJytCJQl1fN?= =?us-ascii?Q?HYus9weFJxa80zoaObR56arQTfNWHSBngYbNw2tfDMupuAwUE00NcuvZEZAa?= =?us-ascii?Q?tKKPFraUqgsZQqazwSCRdev5NF13FuRPqYwIAQuWw1Y/j+EBXBEFT3ly/Sc1?= =?us-ascii?Q?/NCcfR7NkiemaRI57ynZYH77chgePTp+AlWRIL5sJE8VmrcSt4NJU4enboUY?= =?us-ascii?Q?2LTo8x6WImdXiB38=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY1PR0701MB1977; 5:BJAjHuBMKZZs/GMHYmiWIuyarqypvPYylF2rr2L4BpCip1rUVYox0mOpA1nODegsrIAbUxI7Ajw9VfeEjmUpGA/A8zSNQBil3gEAWEINPBI4ziQaRLHuOv+NPFCOIlEhK4M6ZxLGsrXpnGLVElCWMw==; 24:/3FeYDmoprK6tIw1tvD62numKlKvUIhEuXHDEelPudQFf2ykRXr4Tji1pVxQB2IFcSd0kentt2hoVtmDo+C40dKxrwKmrdleHlFa72cccbE=; 20:sWnJBHF/xO+uROtuwxtIKy+EuKkbHIlAtI49py/Sm29Tio9KIinsfcBb89jjBn7HDAlg0yverm47QlGOYCMkPQ== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Nov 2015 16:39:55.6490 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0701MB1977 Subject: [dpdk-dev] [PATCH 13/15] eal: introduce rte_smp_*mb() for memory barriers to use between lcores X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit introduce rte_smp_mb(), rte_smp_wmb() and rte_smp_rmb(), in order to enable memory barriers between lcores. The patch does not provide any functional change for IA, the goal is to have infrastructure for weakly ordered machines like ARM to work on DPDK. Signed-off-by: Jerin Jacob Acked-by: Konstantin Ananyev --- drivers/net/virtio/virtqueue.h | 8 +++---- drivers/net/xenvirt/rte_eth_xenvirt.c | 4 ++-- drivers/net/xenvirt/virtqueue.h | 2 +- .../common/include/arch/ppc_64/rte_atomic.h | 6 +++++ .../common/include/arch/tile/rte_atomic.h | 6 +++++ .../common/include/arch/x86/rte_atomic.h | 6 +++++ lib/librte_eal/common/include/generic/rte_atomic.h | 27 ++++++++++++++++++++++ lib/librte_ring/rte_ring.h | 8 +++---- 8 files changed, 55 insertions(+), 12 deletions(-) diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h index 7789411..d233be6 100644 --- a/drivers/net/virtio/virtqueue.h +++ b/drivers/net/virtio/virtqueue.h @@ -53,12 +53,10 @@ struct rte_mbuf; * accesses through relaxed memory I/O windows, so smp_mb() et al are * sufficient. * - * This driver is for virtio_pci on SMP and therefore can assume - * weaker (compiler barriers) */ -#define virtio_mb() rte_mb() -#define virtio_rmb() rte_compiler_barrier() -#define virtio_wmb() rte_compiler_barrier() +#define virtio_mb() rte_smp_mb() +#define virtio_rmb() rte_smp_rmb() +#define virtio_wmb() rte_smp_wmb() #ifdef RTE_PMD_PACKET_PREFETCH #define rte_packet_prefetch(p) rte_prefetch1(p) diff --git a/drivers/net/xenvirt/rte_eth_xenvirt.c b/drivers/net/xenvirt/rte_eth_xenvirt.c index 73e8bce..8c33a02 100644 --- a/drivers/net/xenvirt/rte_eth_xenvirt.c +++ b/drivers/net/xenvirt/rte_eth_xenvirt.c @@ -99,7 +99,7 @@ eth_xenvirt_rx(void *q, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) nb_used = VIRTQUEUE_NUSED(rxvq); - rte_compiler_barrier(); /* rmb */ + rte_smp_rmb(); num = (uint16_t)(likely(nb_used <= nb_pkts) ? nb_used : nb_pkts); num = (uint16_t)(likely(num <= VIRTIO_MBUF_BURST_SZ) ? num : VIRTIO_MBUF_BURST_SZ); if (unlikely(num == 0)) return 0; @@ -150,7 +150,7 @@ eth_xenvirt_tx(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts); nb_used = VIRTQUEUE_NUSED(txvq); - rte_compiler_barrier(); /* rmb */ + rte_smp_rmb(); num = (uint16_t)(likely(nb_used <= VIRTIO_MBUF_BURST_SZ) ? nb_used : VIRTIO_MBUF_BURST_SZ); num = virtqueue_dequeue_burst(txvq, snd_pkts, len, num); diff --git a/drivers/net/xenvirt/virtqueue.h b/drivers/net/xenvirt/virtqueue.h index eff6208..6dcb0ef 100644 --- a/drivers/net/xenvirt/virtqueue.h +++ b/drivers/net/xenvirt/virtqueue.h @@ -151,7 +151,7 @@ vq_ring_update_avail(struct virtqueue *vq, uint16_t desc_idx) */ avail_idx = (uint16_t)(vq->vq_ring.avail->idx & (vq->vq_nentries - 1)); vq->vq_ring.avail->ring[avail_idx] = desc_idx; - rte_compiler_barrier(); /* wmb , for IA memory model barrier is enough*/ + rte_smp_wmb(); vq->vq_ring.avail->idx++; } diff --git a/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h b/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h index fb7af2b..b8bc2c0 100644 --- a/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h +++ b/lib/librte_eal/common/include/arch/ppc_64/rte_atomic.h @@ -72,6 +72,12 @@ extern "C" { */ #define rte_rmb() {asm volatile("sync" : : : "memory"); } +#define rte_smp_mb() rte_mb() + +#define rte_smp_wmb() rte_compiler_barrier() + +#define rte_smp_rmb() rte_compiler_barrier() + /*------------------------- 16 bit atomic operations -------------------------*/ /* To be compatible with Power7, use GCC built-in functions for 16 bit * operations */ diff --git a/lib/librte_eal/common/include/arch/tile/rte_atomic.h b/lib/librte_eal/common/include/arch/tile/rte_atomic.h index 3dc8eb8..28825ff 100644 --- a/lib/librte_eal/common/include/arch/tile/rte_atomic.h +++ b/lib/librte_eal/common/include/arch/tile/rte_atomic.h @@ -79,6 +79,12 @@ static inline void rte_rmb(void) __sync_synchronize(); } +#define rte_smp_mb() rte_mb() + +#define rte_smp_wmb() rte_compiler_barrier() + +#define rte_smp_rmb() rte_compiler_barrier() + #ifdef __cplusplus } #endif diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic.h b/lib/librte_eal/common/include/arch/x86/rte_atomic.h index e93e8ee..41178c7 100644 --- a/lib/librte_eal/common/include/arch/x86/rte_atomic.h +++ b/lib/librte_eal/common/include/arch/x86/rte_atomic.h @@ -53,6 +53,12 @@ extern "C" { #define rte_rmb() _mm_lfence() +#define rte_smp_mb() rte_mb() + +#define rte_smp_wmb() rte_compiler_barrier() + +#define rte_smp_rmb() rte_compiler_barrier() + /*------------------------- 16 bit atomic operations -------------------------*/ #ifndef RTE_FORCE_INTRINSICS diff --git a/lib/librte_eal/common/include/generic/rte_atomic.h b/lib/librte_eal/common/include/generic/rte_atomic.h index 6c7581a..26d1f56 100644 --- a/lib/librte_eal/common/include/generic/rte_atomic.h +++ b/lib/librte_eal/common/include/generic/rte_atomic.h @@ -72,6 +72,33 @@ static inline void rte_wmb(void); */ static inline void rte_rmb(void); +/** + * General memory barrier between lcores + * + * Guarantees that the LOAD and STORE operations that precede the + * rte_smp_mb() call are globally visible across the lcores + * before the the LOAD and STORE operations that follows it. + */ +static inline void rte_smp_mb(void); + +/** + * Write memory barrier between lcores + * + * Guarantees that the STORE operations that precede the + * rte_smp_wmb() call are globally visible across the lcores + * before the the STORE operations that follows it. + */ +static inline void rte_smp_wmb(void); + +/** + * Read memory barrier between lcores + * + * Guarantees that the LOAD operations that precede the + * rte_smp_rmb() call are globally visible across the lcores + * before the the LOAD operations that follows it. + */ +static inline void rte_smp_rmb(void); + #endif /* __DOXYGEN__ */ /** diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index af68888..19ea1bb 100644 --- a/lib/librte_ring/rte_ring.h +++ b/lib/librte_ring/rte_ring.h @@ -457,7 +457,7 @@ __rte_ring_mp_do_enqueue(struct rte_ring *r, void * const *obj_table, /* write entries in ring */ ENQUEUE_PTRS(); - rte_compiler_barrier(); + rte_smp_wmb(); /* if we exceed the watermark */ if (unlikely(((mask + 1) - free_entries + n) > r->prod.watermark)) { @@ -552,7 +552,7 @@ __rte_ring_sp_do_enqueue(struct rte_ring *r, void * const *obj_table, /* write entries in ring */ ENQUEUE_PTRS(); - rte_compiler_barrier(); + rte_smp_wmb(); /* if we exceed the watermark */ if (unlikely(((mask + 1) - free_entries + n) > r->prod.watermark)) { @@ -643,7 +643,7 @@ __rte_ring_mc_do_dequeue(struct rte_ring *r, void **obj_table, /* copy in table */ DEQUEUE_PTRS(); - rte_compiler_barrier(); + rte_smp_rmb(); /* * If there are other dequeues in progress that preceded us, @@ -727,7 +727,7 @@ __rte_ring_sc_do_dequeue(struct rte_ring *r, void **obj_table, /* copy in table */ DEQUEUE_PTRS(); - rte_compiler_barrier(); + rte_smp_rmb(); __RING_STAT_ADD(r, deq_success, n); r->cons.tail = cons_next;