[v8,3/5] kni: add app specific mempool create & free routine
Checks
Commit Message
From: Vamsi Attunuru <vattunuru@marvell.com>
When KNI operates in IOVA = VA mode, it requires mbuf memory
to be physically contiguous to ensure KNI kernel module could
translate IOVA addresses properly. Patch adds a KNI specific
mempool create routine to populate the KNI packet mbuf pool
with memory objects that are being on a page.
KNI applications need to use this mempool create & free routines
so that mbuf related requirements in IOVA = VA mode are handled
inside those routines based on the enabled mode.
Updated the release notes with these new routine details.
Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
doc/guides/rel_notes/release_19_08.rst | 6 ++++
examples/kni/main.c | 6 +++-
lib/librte_kni/Makefile | 1 +
lib/librte_kni/meson.build | 1 +
lib/librte_kni/rte_kni.c | 59 ++++++++++++++++++++++++++++++++++
lib/librte_kni/rte_kni.h | 49 ++++++++++++++++++++++++++++
lib/librte_kni/rte_kni_version.map | 2 ++
7 files changed, 123 insertions(+), 1 deletion(-)
Comments
On 7/23/19 8:38 AM, vattunuru@marvell.com wrote:
> From: Vamsi Attunuru <vattunuru@marvell.com>
>
> When KNI operates in IOVA = VA mode, it requires mbuf memory
> to be physically contiguous to ensure KNI kernel module could
> translate IOVA addresses properly. Patch adds a KNI specific
> mempool create routine to populate the KNI packet mbuf pool
> with memory objects that are being on a page.
>
> KNI applications need to use this mempool create & free routines
> so that mbuf related requirements in IOVA = VA mode are handled
> inside those routines based on the enabled mode.
>
> Updated the release notes with these new routine details.
>
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
> ---
> doc/guides/rel_notes/release_19_08.rst | 6 ++++
> examples/kni/main.c | 6 +++-
> lib/librte_kni/Makefile | 1 +
> lib/librte_kni/meson.build | 1 +
> lib/librte_kni/rte_kni.c | 59 ++++++++++++++++++++++++++++++++++
> lib/librte_kni/rte_kni.h | 49 ++++++++++++++++++++++++++++
> lib/librte_kni/rte_kni_version.map | 2 ++
> 7 files changed, 123 insertions(+), 1 deletion(-)
>
> diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst
> index 0a3f840..bd01e99 100644
> --- a/doc/guides/rel_notes/release_19_08.rst
> +++ b/doc/guides/rel_notes/release_19_08.rst
> @@ -281,6 +281,12 @@ API Changes
> offload flag from the library. The application must set this flag if it is
> supported by the platform and application wishes to use it.
>
> +* kni: ``rte_kni_pktmbuf_pool_create`` ``rte_kni_pktmbuf_pool_free`` functions
> + were introduced for KNI applications for creating & freeing packet pool.
> + Since IOVA=VA mode was added in KNI, packet pool's mbuf memory should be
> + physically contiguous for the KNI kernel module to work in IOVA=VA mode,
> + this requirment was taken care in the kni packet pool creation fucntions.
> +
>
> ABI Changes
> -----------
> diff --git a/examples/kni/main.c b/examples/kni/main.c
> index 4710d71..3b9c067 100644
> --- a/examples/kni/main.c
> +++ b/examples/kni/main.c
> @@ -975,7 +975,7 @@ main(int argc, char** argv)
> rte_exit(EXIT_FAILURE, "Could not parse input parameters\n");
>
> /* Create the mbuf pool */
> - pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NB_MBUF,
> + pktmbuf_pool = rte_kni_pktmbuf_pool_create("mbuf_pool", NB_MBUF,
> MEMPOOL_CACHE_SZ, 0, MBUF_DATA_SZ, rte_socket_id());
> if (pktmbuf_pool == NULL) {
> rte_exit(EXIT_FAILURE, "Could not initialise mbuf pool\n");
> @@ -1043,6 +1043,10 @@ main(int argc, char** argv)
> continue;
> kni_free_kni(port);
> }
> +
> + if (pktmbuf_pool)
Typically pointer is compared to NULL, but it is not required here
anyway, since rte_mempool_free() handles NULL perfectly itself.
> + rte_kni_pktmbuf_pool_free(pktmbuf_pool);
> +
> for (i = 0; i < RTE_MAX_ETHPORTS; i++)
> if (kni_port_params_array[i]) {
> rte_free(kni_port_params_array[i]);
<...>
> diff --git a/lib/librte_kni/rte_kni.h b/lib/librte_kni/rte_kni.h
> index 5699a64..7f11927 100644
> --- a/lib/librte_kni/rte_kni.h
> +++ b/lib/librte_kni/rte_kni.h
> @@ -20,6 +20,7 @@
> #include <rte_pci.h>
> #include <rte_memory.h>
> #include <rte_mempool.h>
> +#include <rte_mbuf_pool_ops.h>
I don't understand why it is included here.
> #include <rte_ether.h>
>
> #include <rte_kni_common.h>
>
<...>
> -----Original Message-----
> From: Andrew Rybchenko <arybchenko@solarflare.com>
> Sent: Tuesday, July 23, 2019 4:20 PM
> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org
> Cc: thomas@monjalon.net; Jerin Jacob Kollanukkaran <jerinj@marvell.com>;
> olivier.matz@6wind.com; ferruh.yigit@intel.com; anatoly.burakov@intel.com;
> Kiran Kumar Kokkilagadda <kirankumark@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH v8 3/5] kni: add app specific mempool create &
> free routine
>
> On 7/23/19 8:38 AM, vattunuru@marvell.com wrote:
> > From: Vamsi Attunuru <vattunuru@marvell.com>
> >
> > When KNI operates in IOVA = VA mode, it requires mbuf memory to be
> > physically contiguous to ensure KNI kernel module could translate IOVA
> > addresses properly. Patch adds a KNI specific mempool create routine
> > to populate the KNI packet mbuf pool with memory objects that are
> > being on a page.
> >
> > KNI applications need to use this mempool create & free routines so
> > that mbuf related requirements in IOVA = VA mode are handled inside
> > those routines based on the enabled mode.
> >
> > Updated the release notes with these new routine details.
> >
> > Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> > Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
> > ---
> > doc/guides/rel_notes/release_19_08.rst | 6 ++++
> > examples/kni/main.c | 6 +++-
> > lib/librte_kni/Makefile | 1 +
> > lib/librte_kni/meson.build | 1 +
> > lib/librte_kni/rte_kni.c | 59 ++++++++++++++++++++++++++++++++++
> > lib/librte_kni/rte_kni.h | 49 ++++++++++++++++++++++++++++
> > lib/librte_kni/rte_kni_version.map | 2 ++
> > 7 files changed, 123 insertions(+), 1 deletion(-)
> >
> > diff --git a/doc/guides/rel_notes/release_19_08.rst
> > b/doc/guides/rel_notes/release_19_08.rst
> > index 0a3f840..bd01e99 100644
> > --- a/doc/guides/rel_notes/release_19_08.rst
> > +++ b/doc/guides/rel_notes/release_19_08.rst
> > @@ -281,6 +281,12 @@ API Changes
> > offload flag from the library. The application must set this flag if it is
> > supported by the platform and application wishes to use it.
> >
> > +* kni: ``rte_kni_pktmbuf_pool_create`` ``rte_kni_pktmbuf_pool_free``
> > +functions
> > + were introduced for KNI applications for creating & freeing packet pool.
> > + Since IOVA=VA mode was added in KNI, packet pool's mbuf memory
> > +should be
> > + physically contiguous for the KNI kernel module to work in IOVA=VA
> > +mode,
> > + this requirment was taken care in the kni packet pool creation fucntions.
> > +
> >
> > ABI Changes
> > -----------
> > diff --git a/examples/kni/main.c b/examples/kni/main.c index
> > 4710d71..3b9c067 100644
> > --- a/examples/kni/main.c
> > +++ b/examples/kni/main.c
> > @@ -975,7 +975,7 @@ main(int argc, char** argv)
> > rte_exit(EXIT_FAILURE, "Could not parse input parameters\n");
> >
> > /* Create the mbuf pool */
> > - pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NB_MBUF,
> > + pktmbuf_pool = rte_kni_pktmbuf_pool_create("mbuf_pool", NB_MBUF,
> > MEMPOOL_CACHE_SZ, 0, MBUF_DATA_SZ, rte_socket_id());
> > if (pktmbuf_pool == NULL) {
> > rte_exit(EXIT_FAILURE, "Could not initialise mbuf pool\n"); @@
> > -1043,6 +1043,10 @@ main(int argc, char** argv)
> > continue;
> > kni_free_kni(port);
> > }
> > +
> > + if (pktmbuf_pool)
>
> Typically pointer is compared to NULL, but it is not required here anyway, since
> rte_mempool_free() handles NULL perfectly itself.
Ack
>
> > + rte_kni_pktmbuf_pool_free(pktmbuf_pool);
> > +
> > for (i = 0; i < RTE_MAX_ETHPORTS; i++)
> > if (kni_port_params_array[i]) {
> > rte_free(kni_port_params_array[i]);
>
> <...>
>
> > diff --git a/lib/librte_kni/rte_kni.h b/lib/librte_kni/rte_kni.h index
> > 5699a64..7f11927 100644
> > --- a/lib/librte_kni/rte_kni.h
> > +++ b/lib/librte_kni/rte_kni.h
> > @@ -20,6 +20,7 @@
> > #include <rte_pci.h>
> > #include <rte_memory.h>
> > #include <rte_mempool.h>
> > +#include <rte_mbuf_pool_ops.h>
>
> I don't understand why it is included here.
>
included to fix compilation of rte_kni.c post this patch changes, could be included in rte_kni.c though.
> > #include <rte_ether.h>
> >
> > #include <rte_kni_common.h>
> >
>
> <...>
@@ -281,6 +281,12 @@ API Changes
offload flag from the library. The application must set this flag if it is
supported by the platform and application wishes to use it.
+* kni: ``rte_kni_pktmbuf_pool_create`` ``rte_kni_pktmbuf_pool_free`` functions
+ were introduced for KNI applications for creating & freeing packet pool.
+ Since IOVA=VA mode was added in KNI, packet pool's mbuf memory should be
+ physically contiguous for the KNI kernel module to work in IOVA=VA mode,
+ this requirment was taken care in the kni packet pool creation fucntions.
+
ABI Changes
-----------
@@ -975,7 +975,7 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Could not parse input parameters\n");
/* Create the mbuf pool */
- pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NB_MBUF,
+ pktmbuf_pool = rte_kni_pktmbuf_pool_create("mbuf_pool", NB_MBUF,
MEMPOOL_CACHE_SZ, 0, MBUF_DATA_SZ, rte_socket_id());
if (pktmbuf_pool == NULL) {
rte_exit(EXIT_FAILURE, "Could not initialise mbuf pool\n");
@@ -1043,6 +1043,10 @@ main(int argc, char** argv)
continue;
kni_free_kni(port);
}
+
+ if (pktmbuf_pool)
+ rte_kni_pktmbuf_pool_free(pktmbuf_pool);
+
for (i = 0; i < RTE_MAX_ETHPORTS; i++)
if (kni_port_params_array[i]) {
rte_free(kni_port_params_array[i]);
@@ -6,6 +6,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
# library name
LIB = librte_kni.a
+CFLAGS += -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 -fno-strict-aliasing
CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
LDLIBS += -lrte_eal -lrte_mempool -lrte_mbuf -lrte_ethdev
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
+allow_experimental_apis = true
if not is_linux or not dpdk_conf.get('RTE_ARCH_64')
build = false
reason = 'only supported on 64-bit linux'
@@ -687,6 +687,65 @@ kni_allocate_mbufs(struct rte_kni *kni)
}
}
+struct rte_mempool *
+rte_kni_pktmbuf_pool_create(const char *name, unsigned int n,
+ unsigned int cache_size, uint16_t priv_size, uint16_t data_room_size,
+ int socket_id)
+{
+ struct rte_pktmbuf_pool_private mbp_priv;
+ const char *mp_ops_name;
+ struct rte_mempool *mp;
+ unsigned int elt_size;
+ int ret;
+
+ if (RTE_ALIGN(priv_size, RTE_MBUF_PRIV_ALIGN) != priv_size) {
+ RTE_LOG(ERR, MBUF, "mbuf priv_size=%u is not aligned\n",
+ priv_size);
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ elt_size = sizeof(struct rte_mbuf) + (unsigned int)priv_size +
+ (unsigned int)data_room_size;
+ mbp_priv.mbuf_data_room_size = data_room_size;
+ mbp_priv.mbuf_priv_size = priv_size;
+
+ mp = rte_mempool_create_empty(name, n, elt_size, cache_size,
+ sizeof(struct rte_pktmbuf_pool_private), socket_id, 0);
+ if (mp == NULL)
+ return NULL;
+
+ mp_ops_name = rte_mbuf_best_mempool_ops();
+ ret = rte_mempool_set_ops_byname(mp, mp_ops_name, NULL);
+ if (ret != 0) {
+ RTE_LOG(ERR, MBUF, "error setting mempool handler\n");
+ rte_mempool_free(mp);
+ rte_errno = -ret;
+ return NULL;
+ }
+ rte_pktmbuf_pool_init(mp, &mbp_priv);
+
+ if (rte_eal_iova_mode() == RTE_IOVA_VA)
+ ret = rte_mempool_populate_from_pg_sz_chunks(mp);
+ else
+ ret = rte_mempool_populate_default(mp);
+
+ if (ret < 0) {
+ rte_mempool_free(mp);
+ rte_errno = -ret;
+ return NULL;
+ }
+
+ rte_mempool_obj_iter(mp, rte_pktmbuf_init, NULL);
+
+ return mp;
+}
+
+void
+rte_kni_pktmbuf_pool_free(struct rte_mempool *mp)
+{
+ rte_mempool_free(mp);
+}
+
struct rte_kni *
rte_kni_get(const char *name)
{
@@ -20,6 +20,7 @@
#include <rte_pci.h>
#include <rte_memory.h>
#include <rte_mempool.h>
+#include <rte_mbuf_pool_ops.h>
#include <rte_ether.h>
#include <rte_kni_common.h>
@@ -184,6 +185,54 @@ unsigned rte_kni_tx_burst(struct rte_kni *kni, struct rte_mbuf **mbufs,
unsigned num);
/**
+ * Create a kni packet mbuf pool.
+ *
+ * This function creates and initializes a packet mbuf pool for KNI applications
+ * It calls the required mempool populate routine based on the IOVA mode.
+ *
+ * @param name
+ * The name of the mbuf pool.
+ * @param n
+ * The number of elements in the mbuf pool. The optimum size (in terms
+ * of memory usage) for a mempool is when n is a power of two minus one:
+ * n = (2^q - 1).
+ * @param cache_size
+ * Size of the per-core object cache. See rte_mempool_create() for
+ * details.
+ * @param priv_size
+ * Size of application private are between the rte_mbuf structure
+ * and the data buffer. This value must be aligned to RTE_MBUF_PRIV_ALIGN.
+ * @param data_room_size
+ * Size of data buffer in each mbuf, including RTE_PKTMBUF_HEADROOM.
+ * @param socket_id
+ * The socket identifier where the memory should be allocated. The
+ * value can be *SOCKET_ID_ANY* if there is no NUMA constraint for the
+ * reserved zone.
+ * @return
+ * The pointer to the new allocated mempool, on success. NULL on error
+ * with rte_errno set appropriately. Possible rte_errno values include:
+ * - E_RTE_NO_CONFIG - function could not get pointer to rte_config structure
+ * - E_RTE_SECONDARY - function was called from a secondary process instance
+ * - EINVAL - cache size provided is too large, or priv_size is not aligned.
+ * - ENOSPC - the maximum number of memzones has already been allocated
+ * - EEXIST - a memzone with the same name already exists
+ * - ENOMEM - no appropriate memory area found in which to create memzone
+ */
+__rte_experimental
+struct rte_mempool *rte_kni_pktmbuf_pool_create(const char *name,
+ unsigned int n, unsigned int cache_size, uint16_t priv_size,
+ uint16_t data_room_size, int socket_id);
+
+/**
+ * Free the given packet mempool.
+ *
+ * @param mp
+ * The mempool pointer.
+ */
+__rte_experimental
+void rte_kni_pktmbuf_pool_free(struct rte_mempool *mp);
+
+/**
* Get the KNI context of its name.
*
* @param name
@@ -20,4 +20,6 @@ EXPERIMENTAL {
global:
rte_kni_update_link;
+ rte_kni_pktmbuf_pool_create;
+ rte_kni_pktmbuf_pool_free;
};