[dpdk-dev,6/6] doc: introduction to prgdev

diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 77f427e..4a24264 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -60,6 +60,7 @@  Programmer's Guide
     qos_framework
     power_man
     packet_classif_access_ctrl
+    prgdev_lib
     packet_framework
     vhost_lib
     port_hotplug_framework
diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
new file mode 100644
index 0000000..e2d1e2e
--- /dev/null
+++ b/doc/guides/prog_guide/prgdev_lib.rst
@@ -0,0 +1,465 @@ 
+Overview
+========
+
+More and more devices start to support on-die programming, which include
+NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
+introduce a API for application to download/upload image to/from the HW.
+So, it's necessary to provide a generic API to perform image download, upload,
+validation, etc.
+
+This RFC patch intends to introduce a DPDK generic programming device layer,
+called prgdev below, to provide an abstract, generic APIs for applications to
+program hardware without knowing the details of programmable devices. From
+driver's perspective, they'll try to adapt their functions to the abstract
+APIs defined in prgdev.
+
+The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
+for FPGA devices, or upgrade firmware for programmble NICs.
+
+But those devices that have the capability to do on-die programming may
+expose versatile talents to apps. Add/delete/update entries in flow table,
+update/overwrite the firmware/microcode, add a new algorithm, update profiles,
+etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
+include the APIs to perform device-unique actions in current stage until it
+became popular. On the contratry, it only provides the simple capability to
+download a segment of buffer(image) from host to on-die device, or vice versa.
+The image could be an algorithm, a flow table in the on-die SRAM, a firmware
+image running in the device, a function within a pipeline, etc. prgdev won't
+try to inteprete the content and it's transparent to prgdev. Apps and devices
+can communicate with a language they can understand each other, which is not
+provided by prgdev to decide what needs to be programmed.
+
+When the set of APIs is introduced, a general question is why we need it in
+DPDK community? Why we can't use offline tool to perform same thing? The answer
+is the prgdev provide a generic, online API to applications, in the meanwile,
+offers a capability to switch driver dynamically when downloaded image changed
+personality and a new driver is required. Comparing offline tool, it have online
+programmability (see below examples); Comparing online tool, it provides a
+generic API for many HWs; Comparing generic online tool/API for many products,
+it provides a capability to switch driver dynamically.
+
+There are various means to reach same goal, we'll try to find the best/better
+way to approach. All above advantages will help prgdev to be a 'better choice'.
+
+
+Scope
+-----
+
+The target devices include but not limited to programmable NIC, FPGA, GPU.
+Any devices, having the capability to store/load a piece of info to/from
+the deivce then changed hardware behavior, and applicable to prgdev programming
+model, could be registered as a prgdev device.
+
+The device can be programmed (dynamic) within DPDK or have been prior programmed
+(static) prior to a DPDK application being launched.
+
+Static example: In a static example a program, bitstream or other firmware
+would have been loaded prior to the start of a DPDK application. The prgdev
+would recognize the firmware loaded, and instantiate higher level interfaces
+(example : PMD, L3 forwarding, flow, etc.).
+
+Dynamic example: In a dynamic example, the programmability of the program,
+bitstream or other firmware would be loaded by the DPDK application upon start
+, or during execution. The prgdev would offer the interface to allow the DPDK
+application to program the device.
+
+
+Dyanamic bind/unbind
+--------------------
+
+Besides the simple API to upload/download image, the prgdev also introduces a
+mechanism internally to switch drivers and register/unregister device on the
+fly. With this mechanism, apps can use the programmable device, unbind other
+personalities on demand, then program it and bind it back with new personalities.
+Apps can follow below examples to simply complete the whole process.
+
+Note that bind/unbind actions are different concept from that a whole device
+attach/detach. For example, ``rte_eal_dev_detach()``, which will try to detach the
+drivers with device and remove the whole device from specific class of devices
+(ethdev, for example). Then, the whole device is invisible until a new 'probe'
+is activated.
+
+During the whole procedure of image upload/download, prgdev handler is always
+valid and apps can use it operate on programmable device. The reason why unbind
+is necessary is it may break the hardware when programming is in progress while
+other functions are active. Using programmble NIC as an example, it's a little
+impossible to receive/forward packets for hardware while updating firmware. In
+this case, apps need to detach ethdev function before firmware upgrading. Again,
+prgdev handler is still valid, it's function-level detach, different from
+device-level detach. After finishing image download, original function needs to
+attach back, either use same or different drivers, depends on personalities
+changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
+may not, see below examples to get more details.
+
+Another reason to provide bind/unbind action is programmble devices, like FPGA,
+are not identified driver by 'vendor ID' and 'device ID', they might not be
+changed in all the ways, even FPGA is fully programmed. So, it depends on
+internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
+drivers, either a register value or signature, depending on FPGA hardware
+design. In this case, EAL or other bus driver doesn't have the capability to
+identify proper driver for FPGA device. With prgdev introduced, while FPGA is
+always a prgdev, FPGA can use prgdev as primary driver, to find proper function
+driver.
+
+The bind/unbind actions are not trying to break the general initialization
+procedure in DPDK. prgdev still follow current EAL initializatio model to probe
+drivers for devices. See below examples for details.
+
+It's not true that bind/unbind is always necessary for programming. If device
+has a capability to perform specific functions and program in parallel, then
+apps needn't do bind/unbind actions. See 'dynamic program' section below.
+
+Authentication
+--------------
+
+Authentication and permissions for programming the device are outside the scope
+of this API. It is dependent upon a different mechanism not to be supported by
+DPDK. Application can utilize a different mechanism to authenticate user and
+or 'program' as validated for loading.
+
+Misc
+----
+
+PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
+device can be loaded with a personality, and it would be up to this programmable
+device to load the appropriate "personality driver" based upon an identification
+mechanism to work with the loaded personality.
+
+
+prgdev API
+==========
+
+Below is the major APIs exposed to apps.
+
+- ``rte_prgdev_get_info``:
+  Acquire general info of the prgdev.
+- ``rte_prgdev_open``:
+  Open a prgdev for later programming. If failed, needs to call below 'unbind'
+  function to detach other functions except prgdev first.
+- ``rte_prgdev_img_download``:
+  Download a piece of buffer from host to the device after 'open' succeded.
+- ``rte_prgdev_img_upload``:
+  Upload a piece of information from the device to host.
+- ``rte_prgdev_check_status``:
+  Even download process is successful, it's necessary to check the status of the
+  downloaded image if worked in expected way by this function. Harware or driver
+  may not have the capability to check that status, so it's optional to driver.
+- ``rte_prgdev_close``:
+  After upload/download actions, close the prgdev.
+- ``rte_prgdev_bind``:
+  After 'close' action, app can call it to attach the 'new' personality with
+  proper drivers, then expose to app with new functions. prgdev driver will be
+  responsible for finding proper drivers to drive the functions. Desired driver
+  will be responsible for exposing the functions to apps.
+- ``rte_prgdev_unbind``:
+  When app are using the device, they need to call this function to unbind the
+  misc functions except prgdev with drivers prior to call 'open' function.
+
+
+Initialization procedure
+========================
+
+In this section, there 2 examples, FPGA and programmable NIC, used as examples
+to describe the initialization procedure on how programmable devices are probed
+, initialized and registered as different devices. In which, FPGA has 2
+different possible cases, blank and prior programmed FPGAs. Note that prgdev
+depends and follow on EAL initialization process to probe drivers for prgdev.
+
+Blank FPGA
+----------
+
+Below is the diagram on how Blank FPGA are probed and registered as a prgdev
+device that expose to apps.
+
+When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
+ID and device ID, then perform HW initialization if both ID match. After then,
+it registered in prgdev abstract layer as a new device with a set of helper
+function ops.
+
+After that, apps will observe single device: prgdev device. Until now, the FPGA
+device is a simply device without any functionalities from application's
+perspective.
+
+In the following action, app will call ``prgdev bind`` function with the device
+ID of the FPGA device to bind the function within the FPGA to specific driver.
+FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
+and find potential function. Since it's a blank without any prior programming, a
+simply SUCCESS return::
+
+              Abstraction                 FPGA driver             Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +-----eal pci probe----->                       |
+                   |                       |                       | +------------+
+                   |                       |                       | |  FPGA dev  |
+                   |                       +--HW initialization----> |            |
+    +------------+ |                       |                       | |+----------+|
+    | prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id  ||
+    +------------+ |                       |                       | |+----------+|
+                   |                       |                       | +------------+
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       +--Read and parse AFU--->
+                   |                       |                       |
+                   |                       <--No personality found-+
+                   |                       |                       |
+                   <-------SUCCESS---------+                       |
+                   |                       |                       |
+                   +                       +                       +
+
+prior programmed FPGA
+---------------------
+
+For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
+except the action of reading signature, as below flow table indicates. It will
+read valid functions, crypto and ethdev, as an example, then find proper driver
+to initialize the functions, then register as different ports.
+
+After initialization, app will observe 3 kinds of devices, a prgdev device, a
+ethdev device and a cryptodev device, which all comes from a single PCIE device::
+
+
+              Abstraction            FPGA driver               Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +----eal pci probe------>                       |
+                   |                       |                       | +-------------+
+                   |                       |                       | |  FPGA dev   |
+                   |                       +--HW initialization----> |             |
+    +------------+ |                       |                       | +-------------+
+    | prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id   ||
+    +------------+ |                       |                       | |-------------|
+                   |                       |                       | +-------------+
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       +--Read and parse AFU--->
+                   |                       |                       |
+                   |                       <--eth and crypto found-+
+    +------------+ |                       |                       |
+    | ethdev port| <---ethdev attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | crypto port| <---crypto attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+                   +                       +                       +
+
+
+Programmable NIC
+----------------
+
+Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
+initialization procedure is much more simpler than FPGA. Below is a flow table
+to describe this.
+
+The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
+checking that the NIC is programmable. What is different from FPGA is it
+registered itself as a ethdev device in the following action due to the NIC
+nature.
+
+Then, app still follow the common process to call ``prgdev bind``, while the NIC
+driver does nothing and just return a SUCCESS::
+
+              Abstraction            NIC driver                Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +----eal pci probe------>                       |
+                   |                       |                       | +-------------+
+                   |                       +--HW initialization----> |Programmable |
+                   |                       |                       | |     NIC     |
+    +------------+ |                       +---Capability check----> +-------------+
+    | prgdev dev | <--prgdev_ops_register--+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | ethdev port| <---ethdev attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       |                       |
+                   <--------SUCCESS--------+                       |
+                   |                       |                       |
+                   +                       +                       +
+
+
+Program procedure
+=================
+
+This section will describe how apps finish an image downloading procedure with
+the prgdev API. Similar to previous section, it will use several FPGA and NIC
+examples to show the common and different steps.
+
+Needs to point out, ``rte_prgdev_img_download`` function didn't carry the info
+of the image, or where downloading the image to. The interpretion info of the
+image can be hidden into the buffer itself.
+
+Common programming steps
+------------------------
+
+Before reaching details, it needs to introduce the common steps for apps to
+follow in order to download the image to the hardware.
+
+Before programming, apps need to call function ``rte_prgdev_open`` to open the
+programming interface. If failed, either functions residing in PCI-E device is
+in use or the PCI-E device is not ready, the programming steps are different.
+
+Below is the steps that 'open' succeeds:
+
+#. rte_prgdev_open --> SUCCESS
+#. rte_prgdev_img_download
+#. rte_prgdev_check_status
+#. rte_prgdev_close
+#. rte_prgdev_bind
+
+If failed to open the programming interface, apps need to follow below steps:
+
+#. rte_prgdev_open --> FAILURE
+#. Close all related functions
+#. ret_prgdev_unbind
+#. rte_prgdev_open --> SUCCESS
+#. rte_prgdev_img_download
+#. rte_prgdev_check_status
+#. rte_prgdev_close
+#. rte_prgdev_bind
+
+
+Detach functions
+----------------
+
+As discussed before, apps need to follow steps to detach all the other functions
+except prgdev when meeting error for 'open' operation. Below is a FPGA example
+with 'ethdev' and 'cryptodev' function attached.
+
+After meeting error, app needs to close ethdev port and cryptodev port first.
+Then, use 'prgdev unbind' to detach all the functions that registered previously
+. FPGA driver will call each 'detach' function for each registered device class
+respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
+available::
+
+
+              Abstraction             FPGA driver              Hardware
+                 layer
+                   +                       +                       +
+    +------------+ |                       |                       |
+    | prgdev dev | +-------prgdev open----->                       |
+    |            | <-------ERR_IN_USE------+                       | +-------------+
+    +------------+ |                       |                       | |  FPGA dev   |
+                   |                       |                       | | +---------+ |
+    +------------+ |                       |                       | | | AFU id  | |
+    | ethdev port| +---rte_ethdev_close---->                       | | +---------+ |
+    |            |                         +------ethdev close-----> +-------------+
+    +------------+ <---------SUCCESS-------+                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | crypto port| +---crypto port close--->                       |
+    |            | |                       +---crypto func close--->
+    +------------+ <---------SUCCESS-------+                       |
+                   |                       |                       |
+                   +-----prgdev unbind----->                       |
+                   |                       |                       |
+    + - - - - - -+ |                       |                       |
+      ethdev port  <-----ethdev detach-----+                       |
+    +- - - - - - + |                       |                       |
+                   |                       |                       |
+    + - - - - - -+ |                       |                       |
+      crypto port  <----cryptodev detach---+                       |
+    +- - - - - - + |                       |                       |
+                   +                       +                       +
+
+Device programming
+------------------
+
+After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
+'close' sequence respectively. After then, they need to 'bind' new functions
+back with 'prgdev bind' call.
+
+Still re-use previous FPGA example, below is the execution sequence. After image
+downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
+registered and 2 new ports are presented to apps::
+
+
+              Abstraction             FPGA driver              Hardware
+                 layer
+                   +                       +                       + +-------------+
+    +------------+ |                       |                       | |  FPGA dev   |
+    |            | +-----prgdev open------->                       | | +---------+ |
+    |            | <-------SUCCESS---------+                       | | | ethdev  | |
+    |            | |                       |                       | | +---------+ |
+    |            | +--prgdev_img_download-->                       | | +---------+ |
+    |            | |                       +----Download image-----> | | crypto  | |
+    |            | |                       <--------SUCCESS--------+ | +---------+ |
+    |            | <---------SUCCESS-------+                       | +-------------+
+    | prgdev dev | |                       |                       |        |
+    |            | +---prgdev_chk_status--->                       |        |
+    |            | |                       +--Verify image status-->        |
+    |            | |                       <--------SUCCESS--------+        |
+    |            | <--------SUCCESS--------+                       |        |
+    |            | +-------prgdev close---->                       |        v
+    |            | |                       |                       | +-------------+
+    |            | +-------prgdev bind----->                       | |  FPGA dev   |
+    +------------+ |                       |                       | | +---------+ |
+                   |                       <--Virtio and flow dev--+ | | virtio  | |
+    +------------+ |                       |                       | | +---------+ |
+    | ethdev port| <-----ethdev attach-----+                       | | +---------+ |
+    +------------+ |                       |                       | | |Flow dev | |
+                   |                       |                       | | +---------+ |
+    +------------+ |                       |                       | +-------------+
+    | Flow port  | <-----Flow dev attach---+                       |
+    +------------+ |                       |                       |
+                   +                       +                       +
+
+
+Dyanamic program
+----------------
+
+If prgdev device has a capability to be programmed while functions are in use,
+then we say it supports dynamic program. It's possible to update a profile while
+device is used as a ethdev device.
+
+The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
+``rte_prgdev_get_info``.
+
+The dynamic programming procedure is the same as previous section shows, while
+ignore the steps to 'close' and 'unbind' actions when device is in use.
+
+Note that prgdev driver needs to consider if it can support dynamic program to
+avoid any hardware or software error.
+
+
+Source code layout
+==================
+
+Below is the suggested source code layout.
+
+- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
+  doesn't change programmable NIC's directory and remains the same. The PMD
+  driver should include 2 parts, one part is the code to support rte_ethdev API,
+  another part is the code to support rte_prgdev API.
+- 'librte_prgdev' under 'lib' is the new directory store definition and
+  implementation of the generic prgdev API.
+- 'prg' under 'drivers' is the directory to store specfic implementation
+  driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
+  support specific hardware product.
+
+::
+
+                    +---------+                         +---------+
+                    | drivers |                         |   lib   |
+                    +---------+                         +----+----+
+                         |                                   |
+              +----------+-----------+                       |
+              |                      |                       |
+          +---v---+              +---v----+          +-------v-------+
+          |  net  |              |   prg  |          | librte_prgdev |
+          +---+---+              +---+----+          +---------------+
+              |                      |
+        +-----+----+           +-----+----+
+        |          |           |          |
+    +---v----+ +---v----+  +---v----+ +---v----+
+    |  i40e  | | prg NIC|  |FPGA ABC| |GPU XYZ |
+    +--------+ +--------+  +--------+ +--------+