diff mbox

[dpdk-dev,6/6] doc: introduction to prgdev

Message ID 1488427438-13646-7-git-send-email-jing.d.chen@intel.com (mailing list archive)
State Changes Requested, archived
Delegated to: Thomas Monjalon
Headers show

Checks

Context Check Description
ci/checkpatch warning coding style issues

Commit Message

Chen, Jing D March 2, 2017, 4:03 a.m. UTC
From: "Chen, Jing D" <jing.d.chen@intel.com>

This is the documentation to describe what prgdev is, how to use
prgdev API and accomplish an image download.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
Signed-off-by: Gerald Rogers <gerald.rogers@intel.com>
Signed-off-by: John Mcnamara <john.mcnamara@intel.com>
---
 doc/guides/prog_guide/index.rst      |    1 +
 doc/guides/prog_guide/prgdev_lib.rst |  465 ++++++++++++++++++++++++++++++++++
 2 files changed, 466 insertions(+), 0 deletions(-)
 create mode 100644 doc/guides/prog_guide/prgdev_lib.rst

Comments

Thomas Monjalon March 6, 2017, 3:26 p.m. UTC | #1
2017-03-02 12:03, Chen Jing D:
> +Overview
> +========

I think the first review pass of this series must be focused on the
overview you give here (thanks for the detailed explanations).

I'll try to sum up while commenting.

[...]
The target is programmable devices.

> +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
> +for FPGA devices, or upgrade firmware for programmble NICs.

This is a control plane feature.
You want to have it in DPDK to allow dynamic programmation while running
the device, right?

[...]
> +When the set of APIs is introduced, a general question is why we need it in
> +DPDK community?

Good question :)

[...]
> +Any devices, having the capability to store/load a piece of info to/from
> +the deivce then changed hardware behavior, and applicable to prgdev programming
> +model, could be registered as a prgdev device.
> +
> +The device can be programmed (dynamic) within DPDK or have been prior programmed
> +(static) prior to a DPDK application being launched.
[...]
> +Besides the simple API to upload/download image, the prgdev also introduces a
> +mechanism internally to switch drivers and register/unregister device on the
> +fly. With this mechanism, apps can use the programmable device, unbind other
> +personalities on demand, then program it and bind it back with new personalities.
> +Apps can follow below examples to simply complete the whole process.

I disagree about the specific bind/unbind for prgdev.
We must improve binding inside DPDK for every devices.

> +Note that bind/unbind actions are different concept from that a whole device
> +attach/detach. For example, ``rte_eal_dev_detach()``, which will try to detach the
> +drivers with device and remove the whole device from specific class of devices
> +(ethdev, for example). Then, the whole device is invisible until a new 'probe'
> +is activated.

I do not understand.

> +During the whole procedure of image upload/download, prgdev handler is always
> +valid and apps can use it operate on programmable device. The reason why unbind
> +is necessary is it may break the hardware when programming is in progress while
> +other functions are active. Using programmble NIC as an example, it's a little
> +impossible to receive/forward packets for hardware while updating firmware. In
> +this case, apps need to detach ethdev function before firmware upgrading. Again,
> +prgdev handler is still valid, it's function-level detach, different from
> +device-level detach. After finishing image download, original function needs to
> +attach back, either use same or different drivers, depends on personalities
> +changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
> +may not, see below examples to get more details.

If the personality of the device is changed, it must be seen as a new device
from e.g. the ethdev point of view, while keeping the same prgdev device.
In other words, a device can have several interfaces at the same time
(ethdev, cryptodev, eventdev, prgdev, whatever).
I think we can dynamically create/destroy some interfaces while keeping track
of the underlying device.

> +Another reason to provide bind/unbind action is programmble devices, like FPGA,
> +are not identified driver by 'vendor ID' and 'device ID', they might not be
> +changed in all the ways, even FPGA is fully programmed. So, it depends on
> +internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
> +drivers, either a register value or signature, depending on FPGA hardware
> +design. In this case, EAL or other bus driver doesn't have the capability to
> +identify proper driver for FPGA device. With prgdev introduced, while FPGA is
> +always a prgdev, FPGA can use prgdev as primary driver, to find proper function
> +driver.

You mean prgdev should help the bus layer to map the right driver interface?
It looks weird and dangerous. The standard way to identify a PCI device is
to look at its IDs. Other unknown methods must be strongly discussed.
Chen, Jing D March 7, 2017, 10:34 a.m. UTC | #2
Hi, Thomas,

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Monday, March 6, 2017 11:27 PM
> To: Chen, Jing D <jing.d.chen@intel.com>
> Cc: dev@dpdk.org; Liang, Cunming <cunming.liang@intel.com>; Rogers,
> Gerald <gerald.rogers@intel.com>; Wiles, Keith <keith.wiles@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 6/6] doc: introduction to prgdev
> 
> 2017-03-02 12:03, Chen Jing D:
> > +Overview
> > +========
> 
> I think the first review pass of this series must be focused on the overview
> you give here (thanks for the detailed explanations).
> 
> I'll try to sum up while commenting.
> 
> [...]
> The target is programmable devices.
> 
> > +The major purpose of prgdev is to help DPDK users to load/upgrade RTL
> > +images for FPGA devices, or upgrade firmware for programmble NICs.
> 
> This is a control plane feature.
> You want to have it in DPDK to allow dynamic programmation while running
> the device, right?

Exactly right.

> 
> [...]
> > +When the set of APIs is introduced, a general question is why we need
> > +it in DPDK community?
> 
> Good question :)
> 
> [...]
> > +Any devices, having the capability to store/load a piece of info
> > +to/from the deivce then changed hardware behavior, and applicable to
> > +prgdev programming model, could be registered as a prgdev device.
> > +
> > +The device can be programmed (dynamic) within DPDK or have been prior
> > +programmed
> > +(static) prior to a DPDK application being launched.
> [...]
> > +Besides the simple API to upload/download image, the prgdev also
> > +introduces a mechanism internally to switch drivers and
> > +register/unregister device on the fly. With this mechanism, apps can
> > +use the programmable device, unbind other personalities on demand,
> then program it and bind it back with new personalities.
> > +Apps can follow below examples to simply complete the whole process.
> 
> I disagree about the specific bind/unbind for prgdev.
> We must improve binding inside DPDK for every devices.

First of all, let us try to imagine what is the safe status for device before apps
intending to download some image to it - apps wouldn't operate on the device, 
any behaviors like configuring registers, receive/transmit data may impair the
device or make the download failed. 
Following first answer to prevent app accessing device during image
downloading, how can we achieve that? Detach drivers with device is a smart
idea, right? But the problem is how can apps use prgdev API to download image
after all drivers detached with the device?
So, the final question is how can we just detached others driver except prgdev
one? I can't find answer in current DPDK framework, that's why I'd like to introduce
bind/unbind functions to detach other drivers from the device.

I'm open to this problem. If any suggestion or mechanism can help on it, I can
remove these 2.

BTW, not all devices or image download actions need the device to detach the 
device, depending on hardware implementation.

> 
> > +Note that bind/unbind actions are different concept from that a whole
> > +device attach/detach. For example, ``rte_eal_dev_detach()``, which
> > +will try to detach the drivers with device and remove the whole
> > +device from specific class of devices (ethdev, for example). Then, the
> whole device is invisible until a new 'probe'
> > +is activated.
> 
> I do not understand.

See above explanations.

> 
> > +During the whole procedure of image upload/download, prgdev handler
> > +is always valid and apps can use it operate on programmable device.
> > +The reason why unbind is necessary is it may break the hardware when
> > +programming is in progress while other functions are active. Using
> > +programmble NIC as an example, it's a little impossible to
> > +receive/forward packets for hardware while updating firmware. In this
> > +case, apps need to detach ethdev function before firmware upgrading.
> > +Again, prgdev handler is still valid, it's function-level detach,
> > +different from device-level detach. After finishing image download,
> > +original function needs to attach back, either use same or different
> > +drivers, depends on personalities changed or not. For a programmble NIC,
> the driver may be the same. For FPGA, it may not, see below examples to get
> more details.
> 
> If the personality of the device is changed, it must be seen as a new device
> from e.g. the ethdev point of view, while keeping the same prgdev device.
> In other words, a device can have several interfaces at the same time (ethdev,
> cryptodev, eventdev, prgdev, whatever).
> I think we can dynamically create/destroy some interfaces while keeping
> track of the underlying device.

Fully agree. But current PCI device with 'vendor_id' and 'device_id' ony can
be bind to single driver at a time. So I add 'rte_prgdev_allocate/release' to
support register/unregister prgdev dynamically without BDF.

> 
> > +Another reason to provide bind/unbind action is programmble devices,
> > +like FPGA, are not identified driver by 'vendor ID' and 'device ID',
> > +they might not be changed in all the ways, even FPGA is fully
> > +programmed. So, it depends on internal mechanism of FPGA, not 'vendor
> > +ID' and 'device ID' to identify proper drivers, either a register
> > +value or signature, depending on FPGA hardware design. In this case,
> > +EAL or other bus driver doesn't have the capability to identify
> > +proper driver for FPGA device. With prgdev introduced, while FPGA is
> > +always a prgdev, FPGA can use prgdev as primary driver, to find proper
> function driver.
> 
> You mean prgdev should help the bus layer to map the right driver interface?
> It looks weird and dangerous. The standard way to identify a PCI device is to
> look at its IDs. Other unknown methods must be strongly discussed.

For programmable Ethernet device, it's not truce. But for FPGA, it's. When FPGA
is produced, the device ID indicate what model it is and won't be changed
anyway, even being reprogrammed. It used some not-generic mechanism, like
AFU id to distinguish the personalities. So, for FPGA, a prgdev driver can be used
as primary driver to identify personalities and then register to specific devices.
Bruce Richardson March 7, 2017, 11:12 a.m. UTC | #3
On Tue, Mar 07, 2017 at 10:34:06AM +0000, Chen, Jing D wrote:
> Hi, Thomas,
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Sent: Monday, March 6, 2017 11:27 PM
> > To: Chen, Jing D <jing.d.chen@intel.com>
> > Cc: dev@dpdk.org; Liang, Cunming <cunming.liang@intel.com>; Rogers,
> > Gerald <gerald.rogers@intel.com>; Wiles, Keith <keith.wiles@intel.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John
> > <john.mcnamara@intel.com>
> > Subject: Re: [dpdk-dev] [PATCH 6/6] doc: introduction to prgdev
> > 
> > 2017-03-02 12:03, Chen Jing D:
> > > +Overview
> > > +========
> > 
> > I think the first review pass of this series must be focused on the overview
> > you give here (thanks for the detailed explanations).
> > 
> > I'll try to sum up while commenting.
> > 
> > [...]
> > The target is programmable devices.
> > 
> > > +The major purpose of prgdev is to help DPDK users to load/upgrade RTL
> > > +images for FPGA devices, or upgrade firmware for programmble NICs.
> > 
> > This is a control plane feature.
> > You want to have it in DPDK to allow dynamic programmation while running
> > the device, right?
> 
> Exactly right.
> 
> > 
> > [...]
> > > +When the set of APIs is introduced, a general question is why we need
> > > +it in DPDK community?
> > 
> > Good question :)
> > 
> > [...]
> > > +Any devices, having the capability to store/load a piece of info
> > > +to/from the deivce then changed hardware behavior, and applicable to
> > > +prgdev programming model, could be registered as a prgdev device.
> > > +
> > > +The device can be programmed (dynamic) within DPDK or have been prior
> > > +programmed
> > > +(static) prior to a DPDK application being launched.
> > [...]
> > > +Besides the simple API to upload/download image, the prgdev also
> > > +introduces a mechanism internally to switch drivers and
> > > +register/unregister device on the fly. With this mechanism, apps can
> > > +use the programmable device, unbind other personalities on demand,
> > then program it and bind it back with new personalities.
> > > +Apps can follow below examples to simply complete the whole process.
> > 
> > I disagree about the specific bind/unbind for prgdev.
> > We must improve binding inside DPDK for every devices.
> 
> First of all, let us try to imagine what is the safe status for device before apps
> intending to download some image to it - apps wouldn't operate on the device, 
> any behaviors like configuring registers, receive/transmit data may impair the
> device or make the download failed. 
> Following first answer to prevent app accessing device during image
> downloading, how can we achieve that? Detach drivers with device is a smart
> idea, right? But the problem is how can apps use prgdev API to download image
> after all drivers detached with the device?
> So, the final question is how can we just detached others driver except prgdev
> one? I can't find answer in current DPDK framework, that's why I'd like to introduce
> bind/unbind functions to detach other drivers from the device.
> 
> I'm open to this problem. If any suggestion or mechanism can help on it, I can
> remove these 2.
> 
> BTW, not all devices or image download actions need the device to detach the 
> device, depending on hardware implementation.
> 
> > 
> > > +Note that bind/unbind actions are different concept from that a whole
> > > +device attach/detach. For example, ``rte_eal_dev_detach()``, which
> > > +will try to detach the drivers with device and remove the whole
> > > +device from specific class of devices (ethdev, for example). Then, the
> > whole device is invisible until a new 'probe'
> > > +is activated.
> > 
> > I do not understand.
> 
> See above explanations.
> 
> > 
> > > +During the whole procedure of image upload/download, prgdev handler
> > > +is always valid and apps can use it operate on programmable device.
> > > +The reason why unbind is necessary is it may break the hardware when
> > > +programming is in progress while other functions are active. Using
> > > +programmble NIC as an example, it's a little impossible to
> > > +receive/forward packets for hardware while updating firmware. In this
> > > +case, apps need to detach ethdev function before firmware upgrading.
> > > +Again, prgdev handler is still valid, it's function-level detach,
> > > +different from device-level detach. After finishing image download,
> > > +original function needs to attach back, either use same or different
> > > +drivers, depends on personalities changed or not. For a programmble NIC,
> > the driver may be the same. For FPGA, it may not, see below examples to get
> > more details.
> > 
> > If the personality of the device is changed, it must be seen as a new device
> > from e.g. the ethdev point of view, while keeping the same prgdev device.
> > In other words, a device can have several interfaces at the same time (ethdev,
> > cryptodev, eventdev, prgdev, whatever).
> > I think we can dynamically create/destroy some interfaces while keeping
> > track of the underlying device.
> 
> Fully agree. But current PCI device with 'vendor_id' and 'device_id' ony can
> be bind to single driver at a time. So I add 'rte_prgdev_allocate/release' to
> support register/unregister prgdev dynamically without BDF.
> 
> > 
> > > +Another reason to provide bind/unbind action is programmble devices,
> > > +like FPGA, are not identified driver by 'vendor ID' and 'device ID',
> > > +they might not be changed in all the ways, even FPGA is fully
> > > +programmed. So, it depends on internal mechanism of FPGA, not 'vendor
> > > +ID' and 'device ID' to identify proper drivers, either a register
> > > +value or signature, depending on FPGA hardware design. In this case,
> > > +EAL or other bus driver doesn't have the capability to identify
> > > +proper driver for FPGA device. With prgdev introduced, while FPGA is
> > > +always a prgdev, FPGA can use prgdev as primary driver, to find proper
> > function driver.
> > 
> > You mean prgdev should help the bus layer to map the right driver interface?
> > It looks weird and dangerous. The standard way to identify a PCI device is to
> > look at its IDs. Other unknown methods must be strongly discussed.
> 
> For programmable Ethernet device, it's not truce. But for FPGA, it's. When FPGA
> is produced, the device ID indicate what model it is and won't be changed
> anyway, even being reprogrammed. It used some not-generic mechanism, like
> AFU id to distinguish the personalities. So, for FPGA, a prgdev driver can be used
> as primary driver to identify personalities and then register to specific devices.

Sounds like we would need an FPGA bus driver in that case. I think that
would be a better solution than having a specific device driver loading
other drivers.

/Bruce
Thomas Monjalon March 7, 2017, 1:05 p.m. UTC | #4
2017-03-07 11:12, Bruce Richardson:
> On Tue, Mar 07, 2017 at 10:34:06AM +0000, Chen, Jing D wrote:
> > From: Thomas Monjalon
> > > > +Any devices, having the capability to store/load a piece of info
> > > > +to/from the deivce then changed hardware behavior, and applicable to
> > > > +prgdev programming model, could be registered as a prgdev device.
> > > > +
> > > > +The device can be programmed (dynamic) within DPDK or have been prior
> > > > +programmed
> > > > +(static) prior to a DPDK application being launched.
> > > [...]
> > > > +Besides the simple API to upload/download image, the prgdev also
> > > > +introduces a mechanism internally to switch drivers and
> > > > +register/unregister device on the fly. With this mechanism, apps can
> > > > +use the programmable device, unbind other personalities on demand,
> > > then program it and bind it back with new personalities.
> > > > +Apps can follow below examples to simply complete the whole process.
> > > 
> > > I disagree about the specific bind/unbind for prgdev.
> > > We must improve binding inside DPDK for every devices.
> > 
> > First of all, let us try to imagine what is the safe status for device before apps
> > intending to download some image to it - apps wouldn't operate on the device, 
> > any behaviors like configuring registers, receive/transmit data may impair the
> > device or make the download failed. 
> > Following first answer to prevent app accessing device during image
> > downloading, how can we achieve that? Detach drivers with device is a smart
> > idea, right? But the problem is how can apps use prgdev API to download image
> > after all drivers detached with the device?
> > So, the final question is how can we just detached others driver except prgdev
> > one? I can't find answer in current DPDK framework, that's why I'd like to introduce
> > bind/unbind functions to detach other drivers from the device.
> > 
> > I'm open to this problem. If any suggestion or mechanism can help on it, I can
> > remove these 2.
> > 
> > BTW, not all devices or image download actions need the device to detach the 
> > device, depending on hardware implementation.

Thanks for the explanations.
It sounds interesting.

> > > > +Another reason to provide bind/unbind action is programmble devices,
> > > > +like FPGA, are not identified driver by 'vendor ID' and 'device ID',
> > > > +they might not be changed in all the ways, even FPGA is fully
> > > > +programmed. So, it depends on internal mechanism of FPGA, not 'vendor
> > > > +ID' and 'device ID' to identify proper drivers, either a register
> > > > +value or signature, depending on FPGA hardware design. In this case,
> > > > +EAL or other bus driver doesn't have the capability to identify
> > > > +proper driver for FPGA device. With prgdev introduced, while FPGA is
> > > > +always a prgdev, FPGA can use prgdev as primary driver, to find proper
> > > function driver.
> > > 
> > > You mean prgdev should help the bus layer to map the right driver interface?
> > > It looks weird and dangerous. The standard way to identify a PCI device is to
> > > look at its IDs. Other unknown methods must be strongly discussed.
> > 
> > For programmable Ethernet device, it's not truce. But for FPGA, it's. When FPGA
> > is produced, the device ID indicate what model it is and won't be changed
> > anyway, even being reprogrammed. It used some not-generic mechanism, like
> > AFU id to distinguish the personalities. So, for FPGA, a prgdev driver can be used
> > as primary driver to identify personalities and then register to specific devices.
> 
> Sounds like we would need an FPGA bus driver in that case. I think that
> would be a better solution than having a specific device driver loading
> other drivers.

Good suggestion. What do you think of a FPGA bus to consider more parameters
than the PCI BDF when matching a device driver?

How other systems manage the drivers for a FPGA-programmed device?
Chen, Jing D March 7, 2017, 1:45 p.m. UTC | #5
> > >
> > > > +Another reason to provide bind/unbind action is programmble
> > > > +devices, like FPGA, are not identified driver by 'vendor ID' and
> > > > +'device ID', they might not be changed in all the ways, even FPGA
> > > > +is fully programmed. So, it depends on internal mechanism of
> > > > +FPGA, not 'vendor ID' and 'device ID' to identify proper drivers,
> > > > +either a register value or signature, depending on FPGA hardware
> > > > +design. In this case, EAL or other bus driver doesn't have the
> > > > +capability to identify proper driver for FPGA device. With prgdev
> > > > +introduced, while FPGA is always a prgdev, FPGA can use prgdev as
> > > > +primary driver, to find proper
> > > function driver.
> > >
> > > You mean prgdev should help the bus layer to map the right driver
> interface?
> > > It looks weird and dangerous. The standard way to identify a PCI
> > > device is to look at its IDs. Other unknown methods must be strongly
> discussed.
> >
> > For programmable Ethernet device, it's not truce. But for FPGA, it's.
> > When FPGA is produced, the device ID indicate what model it is and
> > won't be changed anyway, even being reprogrammed. It used some
> > not-generic mechanism, like AFU id to distinguish the personalities.
> > So, for FPGA, a prgdev driver can be used as primary driver to identify
> personalities and then register to specific devices.
> 
> Sounds like we would need an FPGA bus driver in that case. I think that would
> be a better solution than having a specific device driver loading other drivers.
>

I don't object to introduce a pseudo bus for FPGA, but it's a matter of work that
FPGA driver needs to consider, not in scope of prgdev.
Besides that, I can see DPDK EAL will do other bus probe first, then PCI bus
probe, which is not friendly to introduce pseudo bus for an actual PCI device.
 
> /Bruce
diff mbox

Patch

diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 77f427e..4a24264 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -60,6 +60,7 @@  Programmer's Guide
     qos_framework
     power_man
     packet_classif_access_ctrl
+    prgdev_lib
     packet_framework
     vhost_lib
     port_hotplug_framework
diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
new file mode 100644
index 0000000..e2d1e2e
--- /dev/null
+++ b/doc/guides/prog_guide/prgdev_lib.rst
@@ -0,0 +1,465 @@ 
+Overview
+========
+
+More and more devices start to support on-die programming, which include
+NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
+introduce a API for application to download/upload image to/from the HW.
+So, it's necessary to provide a generic API to perform image download, upload,
+validation, etc.
+
+This RFC patch intends to introduce a DPDK generic programming device layer,
+called prgdev below, to provide an abstract, generic APIs for applications to
+program hardware without knowing the details of programmable devices. From
+driver's perspective, they'll try to adapt their functions to the abstract
+APIs defined in prgdev.
+
+The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
+for FPGA devices, or upgrade firmware for programmble NICs.
+
+But those devices that have the capability to do on-die programming may
+expose versatile talents to apps. Add/delete/update entries in flow table,
+update/overwrite the firmware/microcode, add a new algorithm, update profiles,
+etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
+include the APIs to perform device-unique actions in current stage until it
+became popular. On the contratry, it only provides the simple capability to
+download a segment of buffer(image) from host to on-die device, or vice versa.
+The image could be an algorithm, a flow table in the on-die SRAM, a firmware
+image running in the device, a function within a pipeline, etc. prgdev won't
+try to inteprete the content and it's transparent to prgdev. Apps and devices
+can communicate with a language they can understand each other, which is not
+provided by prgdev to decide what needs to be programmed.
+
+When the set of APIs is introduced, a general question is why we need it in
+DPDK community? Why we can't use offline tool to perform same thing? The answer
+is the prgdev provide a generic, online API to applications, in the meanwile,
+offers a capability to switch driver dynamically when downloaded image changed
+personality and a new driver is required. Comparing offline tool, it have online
+programmability (see below examples); Comparing online tool, it provides a
+generic API for many HWs; Comparing generic online tool/API for many products,
+it provides a capability to switch driver dynamically.
+
+There are various means to reach same goal, we'll try to find the best/better
+way to approach. All above advantages will help prgdev to be a 'better choice'.
+
+
+Scope
+-----
+
+The target devices include but not limited to programmable NIC, FPGA, GPU.
+Any devices, having the capability to store/load a piece of info to/from
+the deivce then changed hardware behavior, and applicable to prgdev programming
+model, could be registered as a prgdev device.
+
+The device can be programmed (dynamic) within DPDK or have been prior programmed
+(static) prior to a DPDK application being launched.
+
+Static example: In a static example a program, bitstream or other firmware
+would have been loaded prior to the start of a DPDK application. The prgdev
+would recognize the firmware loaded, and instantiate higher level interfaces
+(example : PMD, L3 forwarding, flow, etc.).
+
+Dynamic example: In a dynamic example, the programmability of the program,
+bitstream or other firmware would be loaded by the DPDK application upon start
+, or during execution. The prgdev would offer the interface to allow the DPDK
+application to program the device.
+
+
+Dyanamic bind/unbind
+--------------------
+
+Besides the simple API to upload/download image, the prgdev also introduces a
+mechanism internally to switch drivers and register/unregister device on the
+fly. With this mechanism, apps can use the programmable device, unbind other
+personalities on demand, then program it and bind it back with new personalities.
+Apps can follow below examples to simply complete the whole process.
+
+Note that bind/unbind actions are different concept from that a whole device
+attach/detach. For example, ``rte_eal_dev_detach()``, which will try to detach the
+drivers with device and remove the whole device from specific class of devices
+(ethdev, for example). Then, the whole device is invisible until a new 'probe'
+is activated.
+
+During the whole procedure of image upload/download, prgdev handler is always
+valid and apps can use it operate on programmable device. The reason why unbind
+is necessary is it may break the hardware when programming is in progress while
+other functions are active. Using programmble NIC as an example, it's a little
+impossible to receive/forward packets for hardware while updating firmware. In
+this case, apps need to detach ethdev function before firmware upgrading. Again,
+prgdev handler is still valid, it's function-level detach, different from
+device-level detach. After finishing image download, original function needs to
+attach back, either use same or different drivers, depends on personalities
+changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
+may not, see below examples to get more details.
+
+Another reason to provide bind/unbind action is programmble devices, like FPGA,
+are not identified driver by 'vendor ID' and 'device ID', they might not be
+changed in all the ways, even FPGA is fully programmed. So, it depends on
+internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
+drivers, either a register value or signature, depending on FPGA hardware
+design. In this case, EAL or other bus driver doesn't have the capability to
+identify proper driver for FPGA device. With prgdev introduced, while FPGA is
+always a prgdev, FPGA can use prgdev as primary driver, to find proper function
+driver.
+
+The bind/unbind actions are not trying to break the general initialization
+procedure in DPDK. prgdev still follow current EAL initializatio model to probe
+drivers for devices. See below examples for details.
+
+It's not true that bind/unbind is always necessary for programming. If device
+has a capability to perform specific functions and program in parallel, then
+apps needn't do bind/unbind actions. See 'dynamic program' section below.
+
+Authentication
+--------------
+
+Authentication and permissions for programming the device are outside the scope
+of this API. It is dependent upon a different mechanism not to be supported by
+DPDK. Application can utilize a different mechanism to authenticate user and
+or 'program' as validated for loading.
+
+Misc
+----
+
+PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
+device can be loaded with a personality, and it would be up to this programmable
+device to load the appropriate "personality driver" based upon an identification
+mechanism to work with the loaded personality.
+
+
+prgdev API
+==========
+
+Below is the major APIs exposed to apps.
+
+- ``rte_prgdev_get_info``:
+  Acquire general info of the prgdev.
+- ``rte_prgdev_open``:
+  Open a prgdev for later programming. If failed, needs to call below 'unbind'
+  function to detach other functions except prgdev first.
+- ``rte_prgdev_img_download``:
+  Download a piece of buffer from host to the device after 'open' succeded.
+- ``rte_prgdev_img_upload``:
+  Upload a piece of information from the device to host.
+- ``rte_prgdev_check_status``:
+  Even download process is successful, it's necessary to check the status of the
+  downloaded image if worked in expected way by this function. Harware or driver
+  may not have the capability to check that status, so it's optional to driver.
+- ``rte_prgdev_close``:
+  After upload/download actions, close the prgdev.
+- ``rte_prgdev_bind``:
+  After 'close' action, app can call it to attach the 'new' personality with
+  proper drivers, then expose to app with new functions. prgdev driver will be
+  responsible for finding proper drivers to drive the functions. Desired driver
+  will be responsible for exposing the functions to apps.
+- ``rte_prgdev_unbind``:
+  When app are using the device, they need to call this function to unbind the
+  misc functions except prgdev with drivers prior to call 'open' function.
+
+
+Initialization procedure
+========================
+
+In this section, there 2 examples, FPGA and programmable NIC, used as examples
+to describe the initialization procedure on how programmable devices are probed
+, initialized and registered as different devices. In which, FPGA has 2
+different possible cases, blank and prior programmed FPGAs. Note that prgdev
+depends and follow on EAL initialization process to probe drivers for prgdev.
+
+Blank FPGA
+----------
+
+Below is the diagram on how Blank FPGA are probed and registered as a prgdev
+device that expose to apps.
+
+When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
+ID and device ID, then perform HW initialization if both ID match. After then,
+it registered in prgdev abstract layer as a new device with a set of helper
+function ops.
+
+After that, apps will observe single device: prgdev device. Until now, the FPGA
+device is a simply device without any functionalities from application's
+perspective.
+
+In the following action, app will call ``prgdev bind`` function with the device
+ID of the FPGA device to bind the function within the FPGA to specific driver.
+FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
+and find potential function. Since it's a blank without any prior programming, a
+simply SUCCESS return::
+
+              Abstraction                 FPGA driver             Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +-----eal pci probe----->                       |
+                   |                       |                       | +------------+
+                   |                       |                       | |  FPGA dev  |
+                   |                       +--HW initialization----> |            |
+    +------------+ |                       |                       | |+----------+|
+    | prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id  ||
+    +------------+ |                       |                       | |+----------+|
+                   |                       |                       | +------------+
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       +--Read and parse AFU--->
+                   |                       |                       |
+                   |                       <--No personality found-+
+                   |                       |                       |
+                   <-------SUCCESS---------+                       |
+                   |                       |                       |
+                   +                       +                       +
+
+prior programmed FPGA
+---------------------
+
+For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
+except the action of reading signature, as below flow table indicates. It will
+read valid functions, crypto and ethdev, as an example, then find proper driver
+to initialize the functions, then register as different ports.
+
+After initialization, app will observe 3 kinds of devices, a prgdev device, a
+ethdev device and a cryptodev device, which all comes from a single PCIE device::
+
+
+              Abstraction            FPGA driver               Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +----eal pci probe------>                       |
+                   |                       |                       | +-------------+
+                   |                       |                       | |  FPGA dev   |
+                   |                       +--HW initialization----> |             |
+    +------------+ |                       |                       | +-------------+
+    | prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id   ||
+    +------------+ |                       |                       | |-------------|
+                   |                       |                       | +-------------+
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       +--Read and parse AFU--->
+                   |                       |                       |
+                   |                       <--eth and crypto found-+
+    +------------+ |                       |                       |
+    | ethdev port| <---ethdev attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | crypto port| <---crypto attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+                   +                       +                       +
+
+
+Programmable NIC
+----------------
+
+Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
+initialization procedure is much more simpler than FPGA. Below is a flow table
+to describe this.
+
+The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
+checking that the NIC is programmable. What is different from FPGA is it
+registered itself as a ethdev device in the following action due to the NIC
+nature.
+
+Then, app still follow the common process to call ``prgdev bind``, while the NIC
+driver does nothing and just return a SUCCESS::
+
+              Abstraction            NIC driver                Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +----eal pci probe------>                       |
+                   |                       |                       | +-------------+
+                   |                       +--HW initialization----> |Programmable |
+                   |                       |                       | |     NIC     |
+    +------------+ |                       +---Capability check----> +-------------+
+    | prgdev dev | <--prgdev_ops_register--+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | ethdev port| <---ethdev attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       |                       |
+                   <--------SUCCESS--------+                       |
+                   |                       |                       |
+                   +                       +                       +
+
+
+Program procedure
+=================
+
+This section will describe how apps finish an image downloading procedure with
+the prgdev API. Similar to previous section, it will use several FPGA and NIC
+examples to show the common and different steps.
+
+Needs to point out, ``rte_prgdev_img_download`` function didn't carry the info
+of the image, or where downloading the image to. The interpretion info of the
+image can be hidden into the buffer itself.
+
+Common programming steps
+------------------------
+
+Before reaching details, it needs to introduce the common steps for apps to
+follow in order to download the image to the hardware.
+
+Before programming, apps need to call function ``rte_prgdev_open`` to open the
+programming interface. If failed, either functions residing in PCI-E device is
+in use or the PCI-E device is not ready, the programming steps are different.
+
+Below is the steps that 'open' succeeds:
+
+#. rte_prgdev_open --> SUCCESS
+#. rte_prgdev_img_download
+#. rte_prgdev_check_status
+#. rte_prgdev_close
+#. rte_prgdev_bind
+
+If failed to open the programming interface, apps need to follow below steps:
+
+#. rte_prgdev_open --> FAILURE
+#. Close all related functions
+#. ret_prgdev_unbind
+#. rte_prgdev_open --> SUCCESS
+#. rte_prgdev_img_download
+#. rte_prgdev_check_status
+#. rte_prgdev_close
+#. rte_prgdev_bind
+
+
+Detach functions
+----------------
+
+As discussed before, apps need to follow steps to detach all the other functions
+except prgdev when meeting error for 'open' operation. Below is a FPGA example
+with 'ethdev' and 'cryptodev' function attached.
+
+After meeting error, app needs to close ethdev port and cryptodev port first.
+Then, use 'prgdev unbind' to detach all the functions that registered previously
+. FPGA driver will call each 'detach' function for each registered device class
+respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
+available::
+
+
+              Abstraction             FPGA driver              Hardware
+                 layer
+                   +                       +                       +
+    +------------+ |                       |                       |
+    | prgdev dev | +-------prgdev open----->                       |
+    |            | <-------ERR_IN_USE------+                       | +-------------+
+    +------------+ |                       |                       | |  FPGA dev   |
+                   |                       |                       | | +---------+ |
+    +------------+ |                       |                       | | | AFU id  | |
+    | ethdev port| +---rte_ethdev_close---->                       | | +---------+ |
+    |            |                         +------ethdev close-----> +-------------+
+    +------------+ <---------SUCCESS-------+                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | crypto port| +---crypto port close--->                       |
+    |            | |                       +---crypto func close--->
+    +------------+ <---------SUCCESS-------+                       |
+                   |                       |                       |
+                   +-----prgdev unbind----->                       |
+                   |                       |                       |
+    + - - - - - -+ |                       |                       |
+      ethdev port  <-----ethdev detach-----+                       |
+    +- - - - - - + |                       |                       |
+                   |                       |                       |
+    + - - - - - -+ |                       |                       |
+      crypto port  <----cryptodev detach---+                       |
+    +- - - - - - + |                       |                       |
+                   +                       +                       +
+
+Device programming
+------------------
+
+After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
+'close' sequence respectively. After then, they need to 'bind' new functions
+back with 'prgdev bind' call.
+
+Still re-use previous FPGA example, below is the execution sequence. After image
+downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
+registered and 2 new ports are presented to apps::
+
+
+              Abstraction             FPGA driver              Hardware
+                 layer
+                   +                       +                       + +-------------+
+    +------------+ |                       |                       | |  FPGA dev   |
+    |            | +-----prgdev open------->                       | | +---------+ |
+    |            | <-------SUCCESS---------+                       | | | ethdev  | |
+    |            | |                       |                       | | +---------+ |
+    |            | +--prgdev_img_download-->                       | | +---------+ |
+    |            | |                       +----Download image-----> | | crypto  | |
+    |            | |                       <--------SUCCESS--------+ | +---------+ |
+    |            | <---------SUCCESS-------+                       | +-------------+
+    | prgdev dev | |                       |                       |        |
+    |            | +---prgdev_chk_status--->                       |        |
+    |            | |                       +--Verify image status-->        |
+    |            | |                       <--------SUCCESS--------+        |
+    |            | <--------SUCCESS--------+                       |        |
+    |            | +-------prgdev close---->                       |        v
+    |            | |                       |                       | +-------------+
+    |            | +-------prgdev bind----->                       | |  FPGA dev   |
+    +------------+ |                       |                       | | +---------+ |
+                   |                       <--Virtio and flow dev--+ | | virtio  | |
+    +------------+ |                       |                       | | +---------+ |
+    | ethdev port| <-----ethdev attach-----+                       | | +---------+ |
+    +------------+ |                       |                       | | |Flow dev | |
+                   |                       |                       | | +---------+ |
+    +------------+ |                       |                       | +-------------+
+    | Flow port  | <-----Flow dev attach---+                       |
+    +------------+ |                       |                       |
+                   +                       +                       +
+
+
+Dyanamic program
+----------------
+
+If prgdev device has a capability to be programmed while functions are in use,
+then we say it supports dynamic program. It's possible to update a profile while
+device is used as a ethdev device.
+
+The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
+``rte_prgdev_get_info``.
+
+The dynamic programming procedure is the same as previous section shows, while
+ignore the steps to 'close' and 'unbind' actions when device is in use.
+
+Note that prgdev driver needs to consider if it can support dynamic program to
+avoid any hardware or software error.
+
+
+Source code layout
+==================
+
+Below is the suggested source code layout.
+
+- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
+  doesn't change programmable NIC's directory and remains the same. The PMD
+  driver should include 2 parts, one part is the code to support rte_ethdev API,
+  another part is the code to support rte_prgdev API.
+- 'librte_prgdev' under 'lib' is the new directory store definition and
+  implementation of the generic prgdev API.
+- 'prg' under 'drivers' is the directory to store specfic implementation
+  driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
+  support specific hardware product.
+
+::
+
+                    +---------+                         +---------+
+                    | drivers |                         |   lib   |
+                    +---------+                         +----+----+
+                         |                                   |
+              +----------+-----------+                       |
+              |                      |                       |
+          +---v---+              +---v----+          +-------v-------+
+          |  net  |              |   prg  |          | librte_prgdev |
+          +---+---+              +---+----+          +---------------+
+              |                      |
+        +-----+----+           +-----+----+
+        |          |           |          |
+    +---v----+ +---v----+  +---v----+ +---v----+
+    |  i40e  | | prg NIC|  |FPGA ABC| |GPU XYZ |
+    +--------+ +--------+  +--------+ +--------+