[v4,1/1] eal: add 128-bit compare exchange (x86-64 only)
Checks
Commit Message
This operation can be used for non-blocking algorithms, such as a
non-blocking stack or ring.
Signed-off-by: Gage Eads <gage.eads@intel.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
.../common/include/arch/x86/rte_atomic_64.h | 81 ++++++++++++++++++++++
1 file changed, 81 insertions(+)
Comments
03/04/2019 19:34, Gage Eads:
> This operation can be used for non-blocking algorithms, such as a
> non-blocking stack or ring.
>
> Signed-off-by: Gage Eads <gage.eads@intel.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
> --- a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
> +++ b/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
> +/**
> + * An atomic compare and set function used by the mutex functions.
> + * (Atomically) Equivalent to:
> + * if (*dst == *exp)
> + * *dst = *src
> + * else
> + * *exp = *dst
> + *
> + * @note The success and failure arguments must be one of the __ATOMIC_* values
> + * defined in the C++11 standard. For details on their behavior, refer to the
> + * standard.
> + *
> + * @param dst
> + * The destination into which the value will be written.
> + * @param exp
> + * Pointer to the expected value. If the operation fails, this memory is
> + * updated with the actual value.
> + * @param src
> + * Pointer to the new value.
> + * @param weak
> + * A value of true allows the comparison to spuriously fail and allows the
> + * 'exp' update to occur non-atomically (i.e. a torn read may occur).
> + * Implementations may ignore this argument and only implement the strong
> + * variant.
> + * @param success
> + * If successful, the operation's memory behavior conforms to this (or a
> + * stronger) model.
> + * @param failure
> + * If unsuccessful, the operation's memory behavior conforms to this (or a
> + * stronger) model. This argument cannot be __ATOMIC_RELEASE,
> + * __ATOMIC_ACQ_REL, or a stronger model than success.
> + * @return
> + * Non-zero on success; 0 on failure.
> + */
> +static inline int __rte_experimental
> +rte_atomic128_cmp_exchange(rte_int128_t *dst,
> + rte_int128_t *exp,
> + const rte_int128_t *src,
> + unsigned int weak,
> + int success,
> + int failure)
I was thinking about keeping the doxygen in the generic file.
Is it possible?
> 03/04/2019 19:34, Gage Eads:
> > This operation can be used for non-blocking algorithms, such as a
> > non-blocking stack or ring.
> >
> > Signed-off-by: Gage Eads <gage.eads@intel.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > ---
> > --- a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
> > +++ b/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
> > +/**
> > + * An atomic compare and set function used by the mutex functions.
> > + * (Atomically) Equivalent to:
> > + * if (*dst == *exp)
> > + * *dst = *src
> > + * else
> > + * *exp = *dst
> > + *
> > + * @note The success and failure arguments must be one of the
> > +__ATOMIC_* values
> > + * defined in the C++11 standard. For details on their behavior,
> > +refer to the
> > + * standard.
> > + *
> > + * @param dst
> > + * The destination into which the value will be written.
> > + * @param exp
> > + * Pointer to the expected value. If the operation fails, this memory is
> > + * updated with the actual value.
> > + * @param src
> > + * Pointer to the new value.
> > + * @param weak
> > + * A value of true allows the comparison to spuriously fail and allows the
> > + * 'exp' update to occur non-atomically (i.e. a torn read may occur).
> > + * Implementations may ignore this argument and only implement the
> strong
> > + * variant.
> > + * @param success
> > + * If successful, the operation's memory behavior conforms to this (or a
> > + * stronger) model.
> > + * @param failure
> > + * If unsuccessful, the operation's memory behavior conforms to this (or
> a
> > + * stronger) model. This argument cannot be __ATOMIC_RELEASE,
> > + * __ATOMIC_ACQ_REL, or a stronger model than success.
> > + * @return
> > + * Non-zero on success; 0 on failure.
> > + */
> > +static inline int __rte_experimental
> > +rte_atomic128_cmp_exchange(rte_int128_t *dst,
> > + rte_int128_t *exp,
> > + const rte_int128_t *src,
> > + unsigned int weak,
> > + int success,
> > + int failure)
>
> I was thinking about keeping the doxygen in the generic file.
> Is it possible?
>
We'd need to include the definition of rte_int128_t, so we'd also need either an ifdef on RTE_ARCH_64 or RTE_ARCH_X86_64 to protect 32-bit builds. That macro would prevent doxygen from parsing that section, unless we add a workaround like, for example:
#if defined(RTE_ARCH_64) || defined(__DOXYGEN__)
So the patch would look like the v3, with the declaration in generic/rte_atomic.h, but with that preprocessor change. If we change RTE_ARCH_X86_64 to RTE_ARCH_64, I'd add a note clarifying that it's only implemented for x86-64. What do you think?
03/04/2019 21:21, Eads, Gage:
> > 03/04/2019 19:34, Gage Eads:
> > > This operation can be used for non-blocking algorithms, such as a
> > > non-blocking stack or ring.
> > >
> > > Signed-off-by: Gage Eads <gage.eads@intel.com>
> > > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > > ---
> > > --- a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
> > > +++ b/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
> > > +/**
> > > + * An atomic compare and set function used by the mutex functions.
> > > + * (Atomically) Equivalent to:
> > > + * if (*dst == *exp)
> > > + * *dst = *src
> > > + * else
> > > + * *exp = *dst
> > > + *
> > > + * @note The success and failure arguments must be one of the
> > > +__ATOMIC_* values
> > > + * defined in the C++11 standard. For details on their behavior,
> > > +refer to the
> > > + * standard.
> > > + *
> > > + * @param dst
> > > + * The destination into which the value will be written.
> > > + * @param exp
> > > + * Pointer to the expected value. If the operation fails, this memory is
> > > + * updated with the actual value.
> > > + * @param src
> > > + * Pointer to the new value.
> > > + * @param weak
> > > + * A value of true allows the comparison to spuriously fail and allows the
> > > + * 'exp' update to occur non-atomically (i.e. a torn read may occur).
> > > + * Implementations may ignore this argument and only implement the
> > strong
> > > + * variant.
> > > + * @param success
> > > + * If successful, the operation's memory behavior conforms to this (or a
> > > + * stronger) model.
> > > + * @param failure
> > > + * If unsuccessful, the operation's memory behavior conforms to this (or
> > a
> > > + * stronger) model. This argument cannot be __ATOMIC_RELEASE,
> > > + * __ATOMIC_ACQ_REL, or a stronger model than success.
> > > + * @return
> > > + * Non-zero on success; 0 on failure.
> > > + */
> > > +static inline int __rte_experimental
> > > +rte_atomic128_cmp_exchange(rte_int128_t *dst,
> > > + rte_int128_t *exp,
> > > + const rte_int128_t *src,
> > > + unsigned int weak,
> > > + int success,
> > > + int failure)
> >
> > I was thinking about keeping the doxygen in the generic file.
> > Is it possible?
> >
>
> We'd need to include the definition of rte_int128_t, so we'd also need either an ifdef on RTE_ARCH_64 or RTE_ARCH_X86_64 to protect 32-bit builds. That macro would prevent doxygen from parsing that section, unless we add a workaround like, for example:
>
> #if defined(RTE_ARCH_64) || defined(__DOXYGEN__)
>
> So the patch would look like the v3, with the declaration in generic/rte_atomic.h, but with that preprocessor change. If we change RTE_ARCH_X86_64 to RTE_ARCH_64, I'd add a note clarifying that it's only implemented for x86-64. What do you think?
I would like to see the doc in the generic file
and the implementation in the x86 file.
I tried forward declaration of the typedef:
struct rte_int128;
typedef struct rte_int128 rte_int128_t;
I don't why it does not work.
So I'm trying to protect the declaration with
#ifdef __DOXYGEN__
@@ -34,6 +34,7 @@
/*
* Inspired from FreeBSD src/sys/amd64/include/atomic.h
* Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
* All rights reserved.
*/
@@ -46,6 +47,7 @@
#include <stdint.h>
#include <rte_common.h>
+#include <rte_compat.h>
#include <rte_atomic.h>
/*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,83 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
}
#endif
+/*------------------------ 128 bit atomic operations -------------------------*/
+
+/**
+ * 128-bit integer structure.
+ */
+RTE_STD_C11
+typedef struct {
+ RTE_STD_C11
+ union {
+ uint64_t val[2];
+ __int128 int128;
+ };
+} __rte_aligned(16) rte_int128_t;
+
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ * if (*dst == *exp)
+ * *dst = *src
+ * else
+ * *exp = *dst
+ *
+ * @note The success and failure arguments must be one of the __ATOMIC_* values
+ * defined in the C++11 standard. For details on their behavior, refer to the
+ * standard.
+ *
+ * @param dst
+ * The destination into which the value will be written.
+ * @param exp
+ * Pointer to the expected value. If the operation fails, this memory is
+ * updated with the actual value.
+ * @param src
+ * Pointer to the new value.
+ * @param weak
+ * A value of true allows the comparison to spuriously fail and allows the
+ * 'exp' update to occur non-atomically (i.e. a torn read may occur).
+ * Implementations may ignore this argument and only implement the strong
+ * variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure
+ * If unsuccessful, the operation's memory behavior conforms to this (or a
+ * stronger) model. This argument cannot be __ATOMIC_RELEASE,
+ * __ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ * Non-zero on success; 0 on failure.
+ */
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+ rte_int128_t *exp,
+ const rte_int128_t *src,
+ unsigned int weak,
+ int success,
+ int failure)
+{
+ RTE_SET_USED(weak);
+ RTE_SET_USED(success);
+ RTE_SET_USED(failure);
+ uint8_t res;
+
+ asm volatile (
+ MPLOCKED
+ "cmpxchg16b %[dst];"
+ " sete %[res]"
+ : [dst] "=m" (dst->val[0]),
+ "=a" (exp->val[0]),
+ "=d" (exp->val[1]),
+ [res] "=r" (res)
+ : "b" (src->val[0]),
+ "c" (src->val[1]),
+ "a" (exp->val[0]),
+ "d" (exp->val[1]),
+ "m" (dst->val[0])
+ : "memory");
+
+ return res;
+}
+
#endif /* _RTE_ATOMIC_X86_64_H_ */