[dpdk-dev,v5,3/3] doc: add eventdev library to programmers guide

Message ID 1498830673-56759-4-git-send-email-david.hunt@intel.com (mailing list archive)
State Superseded, archived
Delegated to: Jerin Jacob
Headers

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK

Commit Message

Hunt, David June 30, 2017, 1:51 p.m. UTC
  From: Harry van Haaren <harry.van.haaren@intel.com>

This commit adds an entry in the programmers guide
explaining the eventdev library.

The rte_event struct, queues and ports are explained.
An API walktrough of a simple two stage atomic pipeline
provides the reader with a step by step overview of the
expected usage of the Eventdev API.

Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
---
 doc/guides/prog_guide/eventdev.rst           | 365 ++++++++++
 doc/guides/prog_guide/img/eventdev_usage.svg | 994 +++++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst              |   1 +
 3 files changed, 1360 insertions(+)
 create mode 100644 doc/guides/prog_guide/eventdev.rst
 create mode 100644 doc/guides/prog_guide/img/eventdev_usage.svg
  

Comments

John McNamara June 30, 2017, 2:38 p.m. UTC | #1
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of David Hunt
> Sent: Friday, June 30, 2017 2:51 PM
> To: dev@dpdk.org
> Cc: jerin.jacob@caviumnetworks.com; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [dpdk-dev] [PATCH v5 3/3] doc: add eventdev library to
> programmers guide
> 
> From: Harry van Haaren <harry.van.haaren@intel.com>
> 
> This commit adds an entry in the programmers guide explaining the eventdev
> library.
> 
> The rte_event struct, queues and ports are explained.
> An API walktrough of a simple two stage atomic pipeline provides the
> reader with a step by step overview of the expected usage of the Eventdev
> API.
> 
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>


Acked-by: John McNamara <john.mcnamara@intel.com>
  
Jerin Jacob July 2, 2017, 12:08 p.m. UTC | #2
-----Original Message-----
> Date: Fri, 30 Jun 2017 14:51:13 +0100
> From: David Hunt <david.hunt@intel.com>
> To: dev@dpdk.org
> CC: jerin.jacob@caviumnetworks.com, harry.van.haaren@intel.com
> Subject: [PATCH v5 3/3] doc: add eventdev library to programmers guide
> X-Mailer: git-send-email 2.7.4
> 
> From: Harry van Haaren <harry.van.haaren@intel.com>
> 
> This commit adds an entry in the programmers guide
> explaining the eventdev library.
> 
> The rte_event struct, queues and ports are explained.
> An API walktrough of a simple two stage atomic pipeline
> provides the reader with a step by step overview of the
> expected usage of the Eventdev API.
> 
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

Thanks for document. Overall it looks good. A few comments below.

> ---

Snip

> + The eventdev framework is provided as a DPDK library, allowing
> +applications to use it if they wish, but not require its usage.
> +
> +The goal of this library is to enable applications to build processing
> +pipelines where the load balancing and scheduling is handled by the eventdev.

Adding the differences between polling and event driver model will be
useful. I propose to change the above two paragraphs and add more
detailed text for the  use case. Something like below,

----------Start--------------------
The eventdev framework introduces the event driven programming model.

In a polling model, lcores poll ethdev ports and associated Rx queues
directly to look for a packet. By contrast in an event driven model,
lcores call the scheduler that selects packets for them based on
programmer-specified criteria. The Eventdev library adds support for an
event driven programming model, which offers applications automatic
multicore scaling, dynamic load balancing, pipelining, packet ingress
order maintenance and synchronization services to simplify application
packet processing.

By introducing an event driven programming model, DPDK can support both
polling and event driven programming models for packet processing, and
applications are free to choose whatever model (or combination of the
two) best suits their needs.
-----------End--------------------


> +Step-by-step instructions of the eventdev design is available in the `API
> +Walk-through`_ section later in this document.
> +
> +Event struct
> +------------
> +
> +The eventdev API represents each event with a generic struct, which contains a
> +payload and metadata required for scheduling by an eventdev.  The
> +``rte_event`` struct is a 16 byte C structure, defined in
> +``libs/librte_eventdev/rte_eventdev.h``.
> +

Snip

> +Event Payload
> +~~~~~~~~~~~~~
> +
> +The rte_event struct contains a union for payload, allowing flexibility in what
> +the actual event being scheduled is. The payload is a union of the following:
> +
> +* ``uint64_t u64``
> +* ``void *event_ptr``
> +* ``struct rte_mbuf *mbuf``
> +
> +These three items in a union occupy the same 64 bits at the end of the rte_event
> +structure. The application can utilize the 64 bits directly by accessing the
> +u64 variable, while the event_ptr and mbuf are provided as convenience
> +variables.  For example the mbuf pointer in the union can used to schedule a
> +DPDK packet.
> +
> +Queues
> +~~~~~~
> +
> +A queue is a logical "stage" of a packet processing graph, where each stage
> +has a specified scheduling type.  The application configures each queue for a
> +specific type of scheduling, and just enqueues all events to the eventdev.
> +The Eventdev API supports the following scheduling types per queue:

IMO, The above definition fits nicely if the queue is NOT with
RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES.

I think, We can take the queue definition from header file.
http://dpdk.org/browse/next/dpdk-next-eventdev/tree/lib/librte_eventdev/rte_eventdev.h#n113

and tell about the RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES and non
RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES capability queues here.


> +
> +*   Atomic
> +*   Ordered
> +*   Parallel
> +
> +Atomic, Ordered and Parallel are load-balanced scheduling types: the output
> +of the queue can be spread out over multiple CPU cores.
> +
> +Atomic scheduling on a queue ensures that a single flow is not present on two
> +different CPU cores at the same time. Ordered allows sending all flows to any
> +core, but the scheduler must ensure that on egress the packets are returned to

We can emphasize of when reordering happens ie "On downstream queue enqueue"

> +ingress order. Parallel allows sending all flows to all CPU cores, without any
> +re-ordering guarantees.
> +
> +Single Link Flag
> +^^^^^^^^^^^^^^^^
> +
> +There is a SINGLE_LINK flag which allows an application to indicate that only
> +one port will be connected to a queue.  Queues configured with the single-link
> +flag follow a FIFO like structure, maintaining ordering but it is only capable
> +of being linked to a single port (see below for port and queue linking details).
> +
> +
> +Ports
> +~~~~~
> +
> +Ports are the points of contact between worker cores and the eventdev. The
> +general use-case will see one CPU core using one port to enqueue and dequeue
> +events from an eventdev. Ports are linked to queues in order to retrieve events
> +from those queues (more details in `Linking Queues and Ports`_ below).
> +
> +
> +API Walk-through
> +----------------
> +
> +This section will introduce the reader to the eventdev API, showing how to
> +create and configure an eventdev and use it for a two-stage atomic pipeline
> +with a single core for TX. The diagram below shows the final state of the
> +application after this walk-through:
> +
> +.. _figure_eventdev-usage1:
> +
> +.. figure:: img/eventdev_usage.*
> +
> +   Sample eventdev usage, with RX, two atomic stages and a single-link to TX.
> +
> +
> +A high level overview of the setup steps are:
> +
> +* rte_event_dev_configure()
> +* rte_event_queue_setup()
> +* rte_event_port_setup()
> +* rte_event_port_link()
> +* rte_event_dev_start()
> +

Good.

> +Ingress of New Events
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +Now that the eventdev is set up, and ready to receive events, the RX core must
> +enqueue some events into the system for it to schedule. The events to be
> +scheduled are ordinary DPDK packets, received from an eth_rx_burst() as normal.
> +The following code shows how those packets can be enqueued into the eventdev:
> +
> +.. code-block:: c
> +
> +        const uint16_t nb_rx = rte_eth_rx_burst(eth_port, 0, mbufs, BATCH_SIZE);
> +
> +        for (i = 0; i < nb_rx; i++) {
> +                ev[i].flow_id = mbufs[i]->hash.rss;
> +                ev[i].op = RTE_EVENT_OP_NEW;
> +                ev[i].sched_type = RTE_EVENT_QUEUE_CFG_ATOMIC_ONLY;
> +                ev[i].queue_id = 0;
> +                ev[i].event_type = RTE_EVENT_TYPE_CPU;

ev[i].event_type = RTE_EVENT_TYPE_ETHDEV;

> +                ev[i].sub_event_type = 0;
> +                ev[i].priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
> +                ev[i].mbuf = mbufs[i];
> +        }
> +
> +        const int nb_tx = rte_event_enqueue_burst(dev_id, port_id, ev, nb_rx);
> +        if (nb_tx != nb_rx) {
> +                for(i = nb_tx; i < nb_rx; i++)
> +                        rte_pktmbuf_free(mbufs[i]);
> +        }
> +
> +Forwarding of Events
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Now that the RX core has injected events, there is work to be done by the
> +workers. Note that each worker will dequeue as many events as it can in a burst,
> +process each one individually, and then burst the packets back into the
> +eventdev.
> +
> +The worker can lookup the events source from ``event.queue_id``, which should
> +indicate to the worker what workload needs to be performed on the event.
> +Once done, the worker can update the ``event.queue_id`` to a new value, to send
> +the event to the next stage in the pipeline.
> +
> +.. code-block:: c
> +
> +        int timeout = 0;
> +        struct rte_event events[BATCH_SIZE];
> +        uint16_t nb_rx = rte_event_dequeue_burst(dev_id, worker_port_id, events, BATCH_SIZE, timeout);
> +
> +        for (i = 0; i < nb_rx; i++) {
> +                /* process mbuf using events[i].queue_id as pipeline stage */
> +                struct rte_mbuf *mbuf = events[i].mbuf;
> +                /* Send event to next stage in pipeline */
> +                events[i].queue_id++;
> +        }
> +
> +        uint16_t nb_tx = rte_event_enqueue_burst(dev_id, port_id, events, nb_rx);
> +
> +
> +Egress of Events
> +~~~~~~~~~~~~~~~~
> +
> +Finally, when the packet is ready for egress or needs to be dropped, we need
> +to inform the eventdev that the packet is no longer being handled by the
> +application. This can be done by calling dequeue() or dequeue_burst(), which
> +indicates that the previous burst of packets is no longer in use by the
> +application.

Perfect.

> +
> +.. code-block:: c
> +
> +        struct rte_event events[BATCH_SIZE];
> +        uint16_t n = rte_event_dequeue_burst(dev_id, port_id, events, BATCH_SIZE, 0);
> +        /* burst #1 : now tx or use the packets */
> +        n = rte_event_dequeue_burst(dev_id, port_id, events, BATCH_SIZE, 0);
> +        /* burst #1 is now no longer valid to use in the application, as
> +           the eventdev has dropped any locks or released re-ordered packets */

This above code snippet is bit confusing(dequeue followed by dequeue).
How about changing it in following theme
http://dpdk.org/browse/next/dpdk-next-eventdev/tree/lib/librte_eventdev/rte_eventdev.h#n226


> +
> +Summary
> +-------
> +
> +The eventdev library allows an application to easily schedule events as it
> +requires, either using a run-to-completion or pipeline processing model.  The
> +queues and ports abstract the logical functionality of an eventdev, providing
> +the application with a generic method to schedule events.  With the flexible
> +PMD infrastructure applications benefit of improvements in existing eventdevs
> +and additions of new ones without modification.


With above changes:
Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
  

Patch

diff --git a/doc/guides/prog_guide/eventdev.rst b/doc/guides/prog_guide/eventdev.rst
new file mode 100644
index 0000000..7c67f8a
--- /dev/null
+++ b/doc/guides/prog_guide/eventdev.rst
@@ -0,0 +1,365 @@ 
+..  BSD LICENSE
+    Copyright(c) 2017 Intel Corporation. All rights reserved.
+
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions
+    are met:
+
+    * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+    * Neither the name of Intel Corporation nor the names of its
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+Event Device Library
+====================
+
+The DPDK Event device library is an abstraction that provides the application
+with features to schedule events. This is achieved using the PMD architecture
+similar to the ethdev or cryptodev APIs, which may already be familiar to the
+reader. The eventdev framework is provided as a DPDK library, allowing
+applications to use it if they wish, but not require its usage.
+
+The goal of this library is to enable applications to build processing
+pipelines where the load balancing and scheduling is handled by the eventdev.
+Step-by-step instructions of the eventdev design is available in the `API
+Walk-through`_ section later in this document.
+
+Event struct
+------------
+
+The eventdev API represents each event with a generic struct, which contains a
+payload and metadata required for scheduling by an eventdev.  The
+``rte_event`` struct is a 16 byte C structure, defined in
+``libs/librte_eventdev/rte_eventdev.h``.
+
+Event Metadata
+~~~~~~~~~~~~~~
+
+The rte_event structure contains the following metadata fields, which the
+application fills in to have the event scheduled as required:
+
+* ``flow_id`` - The targeted flow identifier for the enq/deq operation.
+* ``event_type`` - The source of this event, eg RTE_EVENT_TYPE_ETHDEV or CPU.
+* ``sub_event_type`` - Distinguishes events inside the application, that have
+  the same event_type (see above)
+* ``op`` - This field takes one of the RTE_EVENT_OP_* values, and tells the
+  eventdev about the status of the event - valid values are NEW, FORWARD or
+  RELEASE.
+* ``sched_type`` - Represents the type of scheduling that should be performed
+  on this event, valid values are the RTE_SCHED_TYPE_ORDERED, ATOMIC and
+  PARALLEL.
+* ``queue_id`` - The identifier for the event queue that the event is sent to.
+* ``priority`` - The priority of this event, see RTE_EVENT_DEV_PRIORITY.
+
+Event Payload
+~~~~~~~~~~~~~
+
+The rte_event struct contains a union for payload, allowing flexibility in what
+the actual event being scheduled is. The payload is a union of the following:
+
+* ``uint64_t u64``
+* ``void *event_ptr``
+* ``struct rte_mbuf *mbuf``
+
+These three items in a union occupy the same 64 bits at the end of the rte_event
+structure. The application can utilize the 64 bits directly by accessing the
+u64 variable, while the event_ptr and mbuf are provided as convenience
+variables.  For example the mbuf pointer in the union can used to schedule a
+DPDK packet.
+
+Queues
+~~~~~~
+
+A queue is a logical "stage" of a packet processing graph, where each stage
+has a specified scheduling type.  The application configures each queue for a
+specific type of scheduling, and just enqueues all events to the eventdev.
+The Eventdev API supports the following scheduling types per queue:
+
+*   Atomic
+*   Ordered
+*   Parallel
+
+Atomic, Ordered and Parallel are load-balanced scheduling types: the output
+of the queue can be spread out over multiple CPU cores.
+
+Atomic scheduling on a queue ensures that a single flow is not present on two
+different CPU cores at the same time. Ordered allows sending all flows to any
+core, but the scheduler must ensure that on egress the packets are returned to
+ingress order. Parallel allows sending all flows to all CPU cores, without any
+re-ordering guarantees.
+
+Single Link Flag
+^^^^^^^^^^^^^^^^
+
+There is a SINGLE_LINK flag which allows an application to indicate that only
+one port will be connected to a queue.  Queues configured with the single-link
+flag follow a FIFO like structure, maintaining ordering but it is only capable
+of being linked to a single port (see below for port and queue linking details).
+
+
+Ports
+~~~~~
+
+Ports are the points of contact between worker cores and the eventdev. The
+general use-case will see one CPU core using one port to enqueue and dequeue
+events from an eventdev. Ports are linked to queues in order to retrieve events
+from those queues (more details in `Linking Queues and Ports`_ below).
+
+
+API Walk-through
+----------------
+
+This section will introduce the reader to the eventdev API, showing how to
+create and configure an eventdev and use it for a two-stage atomic pipeline
+with a single core for TX. The diagram below shows the final state of the
+application after this walk-through:
+
+.. _figure_eventdev-usage1:
+
+.. figure:: img/eventdev_usage.*
+
+   Sample eventdev usage, with RX, two atomic stages and a single-link to TX.
+
+
+A high level overview of the setup steps are:
+
+* rte_event_dev_configure()
+* rte_event_queue_setup()
+* rte_event_port_setup()
+* rte_event_port_link()
+* rte_event_dev_start()
+
+
+Init and Config
+~~~~~~~~~~~~~~~
+
+The eventdev library uses vdev options to add devices to the DPDK application.
+The ``--vdev`` EAL option allows adding eventdev instances to your DPDK
+application, using the name of the eventdev PMD as an argument.
+
+For example, to create an instance of the software eventdev scheduler, the
+following vdev arguments should be provided to the application EAL command line:
+
+.. code-block:: console
+
+   ./dpdk_application --vdev="event_sw0"
+
+In the following code, we configure eventdev instance with 3 queues
+and 6 ports as follows. The 3 queues consist of 2 Atomic and 1 Single-Link,
+while the 6 ports consist of 4 workers, 1 RX and 1 TX.
+
+.. code-block:: c
+
+        const struct rte_event_dev_config config = {
+                .nb_event_queues = 3,
+                .nb_event_ports = 6,
+                .nb_events_limit  = 4096,
+                .nb_event_queue_flows = 1024,
+                .nb_event_port_dequeue_depth = 128,
+                .nb_event_port_enqueue_depth = 128,
+        };
+        int err = rte_event_dev_configure(dev_id, &config);
+
+The remainder of this walk-through assumes that dev_id is 0.
+
+Setting up Queues
+~~~~~~~~~~~~~~~~~
+
+Once the eventdev itself is configured, the next step is to configure queues.
+This is done by setting the appropriate values in a queue_conf structure, and
+calling the setup function. Repeat this step for each queue, starting from
+0 and ending at ``nb_event_queues - 1`` from the event_dev config above.
+
+.. code-block:: c
+
+        struct rte_event_queue_conf atomic_conf = {
+                .event_queue_cfg = RTE_EVENT_QUEUE_CFG_ATOMIC_ONLY,
+                .priority = RTE_EVENT_DEV_PRIORITY_NORMAL,
+                .nb_atomic_flows = 1024,
+                .nb_atomic_order_sequences = 1024,
+        };
+        int dev_id = 0;
+        int queue_id = 0;
+        int err = rte_event_queue_setup(dev_id, queue_id, &atomic_conf);
+
+The remainder of this walk-through assumes that the queues are configured as
+follows:
+
+ * id 0, atomic queue #1
+ * id 1, atomic queue #2
+ * id 2, single-link queue
+
+Setting up Ports
+~~~~~~~~~~~~~~~~
+
+Once queues are set up successfully, create the ports as required. Each port
+should be set up with its corresponding port_conf type, worker for worker cores,
+rx and tx for the RX and TX cores:
+
+.. code-block:: c
+
+        struct rte_event_port_conf rx_conf = {
+                .dequeue_depth = 128,
+                .enqueue_depth = 128,
+                .new_event_threshold = 1024,
+        };
+        struct rte_event_port_conf worker_conf = {
+                .dequeue_depth = 16,
+                .enqueue_depth = 64,
+                .new_event_threshold = 4096,
+        };
+        struct rte_event_port_conf tx_conf = {
+                .dequeue_depth = 128,
+                .enqueue_depth = 128,
+                .new_event_threshold = 4096,
+        };
+        int dev_id = 0;
+        int port_id = 0;
+        int err = rte_event_port_setup(dev_id, port_id, &CORE_FUNCTION_conf);
+
+It is now assumed that:
+
+ * port 0: RX core
+ * ports 1,2,3,4: Workers
+ * port 5: TX core
+
+Linking Queues and Ports
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The final step is to "wire up" the ports to the queues. After this, the
+eventdev is capable of scheduling events, and when cores request work to do,
+the correct events are provided to that core. Note that the RX core takes input
+from eg: a NIC so it is not linked to any eventdev queues.
+
+Linking all workers to atomic queues, and the TX core to the single-link queue
+can be achieved like this:
+
+.. code-block:: c
+
+        uint8_t port_id = 0;
+        uint8_t atomic_qs[] = {0, 1};
+        uint8_t single_link_q = 2;
+        uint8_t tx_port_id = 5;
+        uin8t_t priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+
+        for(int i = 0; i < 4; i++) {
+                int worker_port = i + 1;
+                int links_made = rte_event_port_link(dev_id, worker_port, atomic_qs, NULL, 2);
+        }
+        int links_made = rte_event_port_link(dev_id, tx_port_id, &single_link_q, &priority, 1);
+
+Starting the EventDev
+~~~~~~~~~~~~~~~~~~~~~
+
+A single function call tells the eventdev instance to start processing
+events. Note that all queues must be linked to for the instance to start, as
+if any queue is not linked to, enqueuing to that queue will cause the
+application to backpressure and eventually stall due to no space in the
+eventdev.
+
+.. code-block:: c
+
+        int err = rte_event_dev_start(dev_id);
+
+Ingress of New Events
+~~~~~~~~~~~~~~~~~~~~~
+
+Now that the eventdev is set up, and ready to receive events, the RX core must
+enqueue some events into the system for it to schedule. The events to be
+scheduled are ordinary DPDK packets, received from an eth_rx_burst() as normal.
+The following code shows how those packets can be enqueued into the eventdev:
+
+.. code-block:: c
+
+        const uint16_t nb_rx = rte_eth_rx_burst(eth_port, 0, mbufs, BATCH_SIZE);
+
+        for (i = 0; i < nb_rx; i++) {
+                ev[i].flow_id = mbufs[i]->hash.rss;
+                ev[i].op = RTE_EVENT_OP_NEW;
+                ev[i].sched_type = RTE_EVENT_QUEUE_CFG_ATOMIC_ONLY;
+                ev[i].queue_id = 0;
+                ev[i].event_type = RTE_EVENT_TYPE_CPU;
+                ev[i].sub_event_type = 0;
+                ev[i].priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+                ev[i].mbuf = mbufs[i];
+        }
+
+        const int nb_tx = rte_event_enqueue_burst(dev_id, port_id, ev, nb_rx);
+        if (nb_tx != nb_rx) {
+                for(i = nb_tx; i < nb_rx; i++)
+                        rte_pktmbuf_free(mbufs[i]);
+        }
+
+Forwarding of Events
+~~~~~~~~~~~~~~~~~~~~
+
+Now that the RX core has injected events, there is work to be done by the
+workers. Note that each worker will dequeue as many events as it can in a burst,
+process each one individually, and then burst the packets back into the
+eventdev.
+
+The worker can lookup the events source from ``event.queue_id``, which should
+indicate to the worker what workload needs to be performed on the event.
+Once done, the worker can update the ``event.queue_id`` to a new value, to send
+the event to the next stage in the pipeline.
+
+.. code-block:: c
+
+        int timeout = 0;
+        struct rte_event events[BATCH_SIZE];
+        uint16_t nb_rx = rte_event_dequeue_burst(dev_id, worker_port_id, events, BATCH_SIZE, timeout);
+
+        for (i = 0; i < nb_rx; i++) {
+                /* process mbuf using events[i].queue_id as pipeline stage */
+                struct rte_mbuf *mbuf = events[i].mbuf;
+                /* Send event to next stage in pipeline */
+                events[i].queue_id++;
+        }
+
+        uint16_t nb_tx = rte_event_enqueue_burst(dev_id, port_id, events, nb_rx);
+
+
+Egress of Events
+~~~~~~~~~~~~~~~~
+
+Finally, when the packet is ready for egress or needs to be dropped, we need
+to inform the eventdev that the packet is no longer being handled by the
+application. This can be done by calling dequeue() or dequeue_burst(), which
+indicates that the previous burst of packets is no longer in use by the
+application.
+
+.. code-block:: c
+
+        struct rte_event events[BATCH_SIZE];
+        uint16_t n = rte_event_dequeue_burst(dev_id, port_id, events, BATCH_SIZE, 0);
+        /* burst #1 : now tx or use the packets */
+        n = rte_event_dequeue_burst(dev_id, port_id, events, BATCH_SIZE, 0);
+        /* burst #1 is now no longer valid to use in the application, as
+           the eventdev has dropped any locks or released re-ordered packets */
+
+Summary
+-------
+
+The eventdev library allows an application to easily schedule events as it
+requires, either using a run-to-completion or pipeline processing model.  The
+queues and ports abstract the logical functionality of an eventdev, providing
+the application with a generic method to schedule events.  With the flexible
+PMD infrastructure applications benefit of improvements in existing eventdevs
+and additions of new ones without modification.
diff --git a/doc/guides/prog_guide/img/eventdev_usage.svg b/doc/guides/prog_guide/img/eventdev_usage.svg
new file mode 100644
index 0000000..7765649
--- /dev/null
+++ b/doc/guides/prog_guide/img/eventdev_usage.svg
@@ -0,0 +1,994 @@ 
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+
+<svg
+   xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/"
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:xlink="http://www.w3.org/1999/xlink"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="683.12061"
+   height="184.672"
+   viewBox="0 0 546.49648 147.7376"
+   xml:space="preserve"
+   color-interpolation-filters="sRGB"
+   class="st9"
+   id="svg2"
+   version="1.1"
+   inkscape:version="0.48.4 r9939"
+   sodipodi:docname="eventdev_usage.svg"
+   style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible"><metadata
+   id="metadata214"><rdf:RDF><cc:Work
+       rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
+         rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><sodipodi:namedview
+   pagecolor="#ffffff"
+   bordercolor="#666666"
+   borderopacity="1"
+   objecttolerance="10"
+   gridtolerance="10"
+   guidetolerance="10"
+   inkscape:pageopacity="0"
+   inkscape:pageshadow="2"
+   inkscape:window-width="1920"
+   inkscape:window-height="1017"
+   id="namedview212"
+   showgrid="false"
+   fit-margin-top="2"
+   fit-margin-left="2"
+   fit-margin-right="2"
+   fit-margin-bottom="2"
+   inkscape:zoom="1.2339869"
+   inkscape:cx="501.15554"
+   inkscape:cy="164.17693"
+   inkscape:window-x="-8"
+   inkscape:window-y="406"
+   inkscape:window-maximized="1"
+   inkscape:current-layer="g17" />
+	<v:documentProperties
+   v:langID="1033"
+   v:viewMarkup="false">
+		<v:userDefs>
+			<v:ud
+   v:nameU="msvSubprocessMaster"
+   v:prompt=""
+   v:val="VT4(Rectangle)" />
+			<v:ud
+   v:nameU="msvNoAutoConnect"
+   v:val="VT0(1):26" />
+		</v:userDefs>
+	</v:documentProperties>
+
+	<style
+   type="text/css"
+   id="style4">
+
+		.st1 {visibility:visible}
+		.st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
+		.st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25}
+		.st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em}
+		.st5 {font-size:1em}
+		.st6 {fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25}
+		.st7 {marker-end:url(#mrkr4-33);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+		.st8 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+		.st9 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+
+	</style>
+
+	<defs
+   id="Markers">
+		<g
+   id="lend4">
+			<path
+   d="M 2,1 0,0 2,-1 2,1"
+   style="stroke:none"
+   id="path8"
+   inkscape:connector-curvature="0" />
+		</g>
+		<marker
+   id="mrkr4-33"
+   class="st8"
+   v:arrowType="4"
+   v:arrowSize="2"
+   v:setback="7.04"
+   refX="-7.04"
+   orient="auto"
+   markerUnits="strokeWidth"
+   overflow="visible"
+   style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible">
+			<use
+   xlink:href="#lend4"
+   transform="scale(-3.52,-3.52)"
+   id="use11"
+   x="0"
+   y="0"
+   width="3"
+   height="3" />
+		</marker>
+	<filter
+   id="filter_2-7"
+   color-interpolation-filters="sRGB"><feGaussianBlur
+     stdDeviation="2"
+     id="feGaussianBlur15-1" /></filter><marker
+   id="mrkr4-33-2"
+   class="st8"
+   v:arrowType="4"
+   v:arrowSize="2"
+   v:setback="7.04"
+   refX="-7.04"
+   orient="auto"
+   markerUnits="strokeWidth"
+   overflow="visible"
+   style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible"><use
+     xlink:href="#lend4"
+     transform="scale(-3.52,-3.52)"
+     id="use11-3"
+     x="0"
+     y="0"
+     width="3"
+     height="3" /></marker><marker
+   id="mrkr4-33-6"
+   class="st8"
+   v:arrowType="4"
+   v:arrowSize="2"
+   v:setback="7.04"
+   refX="-7.04"
+   orient="auto"
+   markerUnits="strokeWidth"
+   overflow="visible"
+   style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible"><use
+     xlink:href="#lend4"
+     transform="scale(-3.52,-3.52)"
+     id="use11-8"
+     x="0"
+     y="0"
+     width="3"
+     height="3" /></marker></defs>
+	<defs
+   id="Filters">
+		<filter
+   id="filter_2"
+   color-interpolation-filters="sRGB">
+			<feGaussianBlur
+   stdDeviation="2"
+   id="feGaussianBlur15" />
+		</filter>
+	</defs>
+	<g
+   v:mID="0"
+   v:index="1"
+   v:groupContext="foregroundPage"
+   id="g17"
+   transform="translate(-47.323579,-90.784072)">
+		<v:userDefs>
+			<v:ud
+   v:nameU="msvThemeOrder"
+   v:val="VT0(0):26" />
+		</v:userDefs>
+		<title
+   id="title19">Page-1</title>
+		<v:pageProperties
+   v:drawingScale="1"
+   v:pageScale="1"
+   v:drawingUnits="0"
+   v:shadowOffsetX="9"
+   v:shadowOffsetY="-9" />
+		<v:layer
+   v:name="Connector"
+   v:index="0" />
+		<g
+   id="shape1-1"
+   v:mID="1"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,128.62352,-288.18843)">
+			<title
+   id="title22">Square</title>
+			<desc
+   id="desc24">Atomic Queue #1</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="30.75"
+   cy="581.25"
+   width="61.5"
+   height="61.5" />
+			<g
+   id="shadow1-2"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<rect
+   x="0"
+   y="550.5"
+   width="61.5"
+   height="61.5"
+   class="st2"
+   id="rect27"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<rect
+   x="0"
+   y="550.5"
+   width="61.5"
+   height="61.5"
+   class="st3"
+   id="rect29"
+   style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" />
+
+		</g>
+		<g
+   id="shape3-8"
+   v:mID="3"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,297.37175,-288.18843)">
+			<title
+   id="title36">Square.3</title>
+			<desc
+   id="desc38">Atomic Queue #2</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="30.75"
+   cy="581.25"
+   width="61.5"
+   height="61.5" />
+			<g
+   id="shadow3-9"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<rect
+   x="0"
+   y="550.5"
+   width="61.5"
+   height="61.5"
+   class="st2"
+   id="rect41"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<rect
+   x="0"
+   y="550.5"
+   width="61.5"
+   height="61.5"
+   class="st3"
+   id="rect43"
+   style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" />
+
+		</g>
+		<g
+   id="shape4-15"
+   v:mID="4"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,466.1192,-288.18843)">
+			<title
+   id="title50">Square.4</title>
+			<desc
+   id="desc52">Single Link Queue # 1</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="30.75"
+   cy="581.25"
+   width="61.5"
+   height="61.5" />
+			<g
+   id="shadow4-16"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<rect
+   x="0"
+   y="550.5"
+   width="61.5"
+   height="61.5"
+   class="st2"
+   id="rect55"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<rect
+   x="0"
+   y="550.5"
+   width="61.5"
+   height="61.5"
+   class="st3"
+   id="rect57"
+   style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" />
+
+		</g>
+		<g
+   id="shape5-22"
+   v:mID="5"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,52.208527,-296.14701)">
+			<title
+   id="title64">Circle</title>
+			<desc
+   id="desc66">RX</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="20.5"
+   cy="591.5"
+   width="35.88"
+   height="30.75" />
+			<g
+   id="shadow5-23"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st2"
+   id="path69"
+   inkscape:connector-curvature="0"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st6"
+   id="path71"
+   inkscape:connector-curvature="0"
+   style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+			<text
+   x="15.19"
+   y="594.5"
+   class="st4"
+   v:langID="1033"
+   id="text73"
+   style="fill:#feffff;font-family:Calibri"><v:paragraph
+   v:horizAlign="1" /><v:tabList />RX</text>
+
+		</g>
+		<g
+   id="shape6-28"
+   v:mID="6"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,84.042834,-305.07614)">
+			<title
+   id="title76">Dynamic connector</title>
+			<path
+   d="m 0,603 50.38,0"
+   class="st7"
+   id="path78"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape7-34"
+   v:mID="7"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,220.95621,-296.14701)">
+			<title
+   id="title81">Circle.7</title>
+			<desc
+   id="desc83">W ..</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="20.5"
+   cy="591.5"
+   width="35.88"
+   height="30.75" />
+			<g
+   id="shadow7-35"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st2"
+   id="path86"
+   inkscape:connector-curvature="0"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st6"
+   id="path88"
+   inkscape:connector-curvature="0"
+   style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+			<text
+   x="12.4"
+   y="594.5"
+   class="st4"
+   v:langID="1033"
+   id="text90"
+   style="fill:#feffff;font-family:Calibri"><v:paragraph
+   v:horizAlign="1" /><v:tabList />W ..</text>
+
+		</g>
+		<g
+   id="shape9-40"
+   v:mID="9"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,220.95621,-243.34865)">
+			<title
+   id="title93">Circle.9</title>
+			<desc
+   id="desc95">W N</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="20.5"
+   cy="591.5"
+   width="35.88"
+   height="30.75" />
+			<g
+   id="shadow9-41"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st2"
+   id="path98"
+   inkscape:connector-curvature="0"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st6"
+   id="path100"
+   inkscape:connector-curvature="0"
+   style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+			<text
+   x="11.69"
+   y="594.5"
+   class="st4"
+   v:langID="1033"
+   id="text102"
+   style="fill:#feffff;font-family:Calibri"><v:paragraph
+   v:horizAlign="1" /><v:tabList />W N</text>
+
+		</g>
+		<g
+   id="shape10-46"
+   v:mID="10"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,220.95621,-348.94537)">
+			<title
+   id="title105">Circle.10</title>
+			<desc
+   id="desc107">W 1</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="20.5"
+   cy="591.5"
+   width="35.88"
+   height="30.75" />
+			<g
+   id="shadow10-47"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st2"
+   id="path110"
+   inkscape:connector-curvature="0"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st6"
+   id="path112"
+   inkscape:connector-curvature="0"
+   style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+			<text
+   x="12.39"
+   y="594.5"
+   class="st4"
+   v:langID="1033"
+   id="text114"
+   style="fill:#feffff;font-family:Calibri"><v:paragraph
+   v:horizAlign="1" /><v:tabList />W 1</text>
+
+		</g>
+		<g
+   id="shape11-52"
+   v:mID="11"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,195.91581,-312.06416)">
+			<title
+   id="title117">Dynamic connector.11</title>
+			<path
+   d="m 0,612 0,-68 25.21,0"
+   class="st7"
+   id="path119"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape12-57"
+   v:mID="12"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,176.37498,-305.07614)">
+			<title
+   id="title122">Dynamic connector.12</title>
+			<path
+   d="m 0,603 50.38,0"
+   class="st7"
+   id="path124"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape13-62"
+   v:mID="13"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,176.37498,-312.06416)">
+			<title
+   id="title127">Dynamic connector.13</title>
+			<path
+   d="m 0,612 25.17,0 0,68 25.21,0"
+   class="st7"
+   id="path129"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape14-67"
+   v:mID="14"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,252.79052,-259.2658)">
+			<title
+   id="title132">Dynamic connector.14</title>
+			<path
+   d="m 0,612 26.88,0 0,-68 23.5,0"
+   class="st7"
+   id="path134"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape15-72"
+   v:mID="15"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,252.79052,-305.07614)">
+			<title
+   id="title137">Dynamic connector.15</title>
+			<path
+   d="m 0,603 50.38,0"
+   class="st7"
+   id="path139"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape19-77"
+   v:mID="19"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,389.70366,-296.14701)">
+			<title
+   id="title142">Circle.19</title>
+			<desc
+   id="desc144">W ..</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="20.5"
+   cy="591.5"
+   width="35.88"
+   height="30.75" />
+			<g
+   id="shadow19-78"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st2"
+   id="path147"
+   inkscape:connector-curvature="0"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st6"
+   id="path149"
+   inkscape:connector-curvature="0"
+   style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+			<text
+   x="12.4"
+   y="594.5"
+   class="st4"
+   v:langID="1033"
+   id="text151"
+   style="fill:#feffff;font-family:Calibri"><v:paragraph
+   v:horizAlign="1" /><v:tabList />W ..</text>
+
+		</g>
+		<g
+   id="shape20-83"
+   v:mID="20"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,389.70366,-243.34865)">
+			<title
+   id="title154">Circle.20</title>
+			<desc
+   id="desc156">W N</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="20.5"
+   cy="591.5"
+   width="35.88"
+   height="30.75" />
+			<g
+   id="shadow20-84"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st2"
+   id="path159"
+   inkscape:connector-curvature="0"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st6"
+   id="path161"
+   inkscape:connector-curvature="0"
+   style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+			<text
+   x="11.69"
+   y="594.5"
+   class="st4"
+   v:langID="1033"
+   id="text163"
+   style="fill:#feffff;font-family:Calibri"><v:paragraph
+   v:horizAlign="1" /><v:tabList />W N</text>
+
+		</g>
+		<g
+   id="shape21-89"
+   v:mID="21"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,389.70366,-348.94537)">
+			<title
+   id="title166">Circle.21</title>
+			<desc
+   id="desc168">W 1</desc>
+			<v:userDefs>
+				<v:ud
+   v:nameU="visVersion"
+   v:val="VT0(15):26" />
+			</v:userDefs>
+			<v:textBlock
+   v:margins="rect(4,4,4,4)" />
+			<v:textRect
+   cx="20.5"
+   cy="591.5"
+   width="35.88"
+   height="30.75" />
+			<g
+   id="shadow21-90"
+   v:groupContext="shadow"
+   v:shadowOffsetX="0.345598"
+   v:shadowOffsetY="-1.97279"
+   v:shadowType="1"
+   transform="translate(0.345598,1.97279)"
+   class="st1"
+   style="visibility:visible">
+				<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st2"
+   id="path171"
+   inkscape:connector-curvature="0"
+   style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+			</g>
+			<path
+   d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+   class="st6"
+   id="path173"
+   inkscape:connector-curvature="0"
+   style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+			<text
+   x="12.39"
+   y="594.5"
+   class="st4"
+   v:langID="1033"
+   id="text175"
+   style="fill:#feffff;font-family:Calibri"><v:paragraph
+   v:horizAlign="1" /><v:tabList />W 1</text>
+
+		</g>
+		<g
+   id="shape28-95"
+   v:mID="28"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,345.12321,-305.07614)">
+			<title
+   id="title178">Dynamic connector.28</title>
+			<path
+   d="m 0,603 50.38,0"
+   class="st7"
+   id="path180"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape29-100"
+   v:mID="29"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,345.12321,-312.06416)">
+			<title
+   id="title183">Dynamic connector.29</title>
+			<path
+   d="m 0,612 28.33,0 0,-68 22.05,0"
+   class="st7"
+   id="path185"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape30-105"
+   v:mID="30"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,345.12321,-312.06416)">
+			<title
+   id="title188">Dynamic connector.30</title>
+			<path
+   d="m 0,612 28.33,0 0,68 22.05,0"
+   class="st7"
+   id="path190"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape31-110"
+   v:mID="31"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,421.53797,-259.2658)">
+			<title
+   id="title193">Dynamic connector.31</title>
+			<path
+   d="m 0,612 24.42,0 0,-68 25.96,0"
+   class="st7"
+   id="path195"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape32-115"
+   v:mID="32"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,421.53797,-305.07614)">
+			<title
+   id="title198">Dynamic connector.32</title>
+			<path
+   d="m 0,603 50.38,0"
+   class="st7"
+   id="path200"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape33-120"
+   v:mID="33"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,421.53797,-364.86253)">
+			<title
+   id="title203">Dynamic connector.33</title>
+			<path
+   d="m 0,612 24.42,0 0,68 25.96,0"
+   class="st7"
+   id="path205"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+		<g
+   id="shape34-125"
+   v:mID="34"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,252.79052,-364.86253)">
+			<title
+   id="title208">Dynamic connector.34</title>
+			<path
+   d="m 0,612 26.88,0 0,68 23.5,0"
+   class="st7"
+   id="path210"
+   inkscape:connector-curvature="0"
+   style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+		</g>
+	<text
+   xml:space="preserve"
+   style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans"
+   x="153.38116"
+   y="165.90149"
+   id="text3106"
+   sodipodi:linespacing="125%"><tspan
+     sodipodi:role="line"
+     x="153.38116"
+     y="165.90149"
+     id="tspan3110"
+     style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Atomic #1</tspan></text>
+<text
+   xml:space="preserve"
+   style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans"
+   x="322.12939"
+   y="165.90149"
+   id="text3106-1"
+   sodipodi:linespacing="125%"><tspan
+     sodipodi:role="line"
+     x="322.12939"
+     y="165.90149"
+     id="tspan3110-4"
+     style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Atomic #2</tspan></text>
+<text
+   xml:space="preserve"
+   style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans"
+   x="491.82089"
+   y="172.79289"
+   id="text3106-0"
+   sodipodi:linespacing="125%"><tspan
+     sodipodi:role="line"
+     x="491.82089"
+     y="172.79289"
+     style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
+     id="tspan3923" /></text>
+<text
+   xml:space="preserve"
+   style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans"
+   x="491.02899"
+   y="165.03951"
+   id="text3106-8-5"
+   sodipodi:linespacing="125%"><tspan
+     sodipodi:role="line"
+     x="491.02899"
+     y="165.03951"
+     id="tspan3110-2-1"
+     style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Single Link</tspan></text>
+<g
+   style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible"
+   id="shape5-22-1"
+   v:mID="5"
+   v:groupContext="shape"
+   transform="matrix(0.77644652,0,0,0.77644652,556.00223,-296.89447)"><title
+     id="title64-5">Circle</title><desc
+     id="desc66-2">RX</desc><v:userDefs><v:ud
+       v:nameU="visVersion"
+       v:val="VT0(15):26" /></v:userDefs><v:textBlock
+     v:margins="rect(4,4,4,4)" /><v:textRect
+     cx="20.5"
+     cy="591.5"
+     width="35.88"
+     height="30.75" /><g
+     id="shadow5-23-7"
+     v:groupContext="shadow"
+     v:shadowOffsetX="0.345598"
+     v:shadowOffsetY="-1.97279"
+     v:shadowType="1"
+     transform="translate(0.345598,1.97279)"
+     class="st1"
+     style="visibility:visible"><path
+       d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+       class="st2"
+       id="path69-6"
+       inkscape:connector-curvature="0"
+       style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-7)" /></g><path
+     d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+     class="st6"
+     id="path71-1"
+     inkscape:connector-curvature="0"
+     style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /><text
+     x="11.06866"
+     y="596.56067"
+     class="st4"
+     v:langID="1033"
+     id="text73-4"
+     style="fill:#feffff;font-family:Calibri"> TX</text>
+</g><g
+   style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible"
+   id="shape28-95-5"
+   v:mID="28"
+   v:groupContext="shape"
+   v:layerMember="0"
+   transform="matrix(0.77644652,0,0,0.77644652,512.00213,-305.42637)"><title
+     id="title178-7">Dynamic connector.28</title><path
+     d="m 0,603 50.38,0"
+     class="st7"
+     id="path180-6"
+     inkscape:connector-curvature="0"
+     style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /></g></g>
+</svg>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index ef5a02a..7578395 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -57,6 +57,7 @@  Programmer's Guide
     multi_proc_support
     kernel_nic_interface
     thread_safety_dpdk_functions
+    eventdev
     qos_framework
     power_man
     packet_classif_access_ctrl