diff mbox

[dpdk-dev,RFC,1/2] doc: introduction to prgdev

Message ID 1484882498-18653-2-git-send-email-jing.d.chen@intel.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers show

Checks

Context Check Description
ci/checkpatch warning coding style issues
ci/Intel compilation success Compilation OK

Commit Message

Chen, Jing D Jan. 20, 2017, 3:21 a.m. UTC
This is the documentation to describe what prgdev is, how to use
prgdev API and accomplish an image download.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
---
 doc/guides/prog_guide/prgdev_lib.rst |  457 ++++++++++++++++++++++++++++++++++
 1 files changed, 457 insertions(+), 0 deletions(-)
 create mode 100644 doc/guides/prog_guide/prgdev_lib.rst

Comments

Jan Blunck Feb. 1, 2017, 11:41 a.m. UTC | #1
On Fri, Jan 20, 2017 at 4:21 AM, Chen Jing D(Mark)
<jing.d.chen@intel.com> wrote:
> This is the documentation to describe what prgdev is, how to use
> prgdev API and accomplish an image download.
>
> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
> ---
>  doc/guides/prog_guide/prgdev_lib.rst |  457 ++++++++++++++++++++++++++++++++++
>  1 files changed, 457 insertions(+), 0 deletions(-)
>  create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
>
> diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
> new file mode 100644
> index 0000000..3917c18
> --- /dev/null
> +++ b/doc/guides/prog_guide/prgdev_lib.rst
> @@ -0,0 +1,457 @@
> +Overview
> +========
> +
> +More and more devices start to support on-die programming, which include
> +NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
> +introduce a API for application to download/upload image to/from the HW.
> +So, it's necessary to provide a generic API to perform image download, upload,
> +validation, etc.
> +
> +This RFC patch intends to introduce a DPDK generic programming device layer,
> +called prgdev below, to provide an abstract, generic APIs for applications to
> +program hardware without knowing the details of programmable devices. From
> +driver's perspective, they'll try to adapt their functions to the abstract
> +APIs defined in prgdev.
> +
> +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
> +for FPGA devices, or upgrade firmware for programmble NICs.
> +
> +But those devices that have the capability to do on-die programming may
> +expose versatile talents to apps. Add/delete/update entries in flow table,
> +update/overwrite the firmware/microcode, add a new algorithm, update profiles
> +, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
> +include the APIs to perform device-unique actions in current stage until it
> +became popular. On the contratry, it only provides the simple capability to
> +download a segment of buffer(image) from host to on-die device, or vice versa.
> +The image could be an algorithm, a flow table in the on-die SRAM, a firmware
> +image running in the device, a function within a pipeline, etc. prgdev won't
> +try to inteprete the content and it's transparent to prgdev. Apps and devices
> +can communicate with a language they can understand each other, which is not
> +provided by prgdev to decide what needs to be programmed.
> +
> +When the set of APIs is introduced, a general question is why we need it in
> +DPDK community? Why we can't use offline tool to perform same thing? The answer
> +is the prgdev provide a generic, online API to applications, in the meanwile,
> +offers a capability to switch driver dynamically when downloaded image changed
> +personality and a new driver is required. Comparing offline tool, it have online
> +programmability (see below examples); Comparing online tool, it provides a
> +generic API for many HWs; Comparing generic online tool/API for many products,
> +it provides a capability to switch driver dynamically.
> +
> +There are various means to reach same goal, we'll try to find the best/better
> +way to approach. All above advantages will help prgdev to be a 'better choice'.
> +

From my point of view this doesn't belong into the DPDK. On Linux this
is traditionally handled by udev and you already have the freedom to
use userspace applications to program a device requiring firmware in
that case. I don't believe that modeling this in the DPDK explicitly
is the right thing to do.

Especially if the device supports changing personality it is required
to unplug the existing personality before reprogramming. You can do
this already today. Also writing OOB firmware data that changes
configuration should be possible today by handling interrupts.

Maybe we can come up with an example application that demonstrates how
the different infrastructure components could get orchestrated. Do you
have a device in mind that supports this?

Regards,
Jan

> +
> +Scope
> +-----
> +
> +The target devices include but not limited to programmable NIC, FPGA, GPU.
> +Any devices, having the capability to store/load a piece of info to/from
> +the deivce then changed hardware behavior, and applicable to prgdev programming
> +model, could be registered as a prgdev device.
> +
> +The device can be programmed (dynamic) within DPDK or have been prior programmed
> +(static) prior to a DPDK application being launched.
> +
> +Static example: In a static example a program, bitstream or other firmware
> +would have been loaded prior to the start of a DPDK application. The prgdev
> +would recognize the firmware loaded, and instantiate higher level interfaces
> +(example : PMD, L3 forwarding, flow, etc.).
> +
> +Dynamic example: In a dynamic example, the programmability of the program,
> +bitstream or other firmware would be loaded by the DPDK application upon start
> +, or during execution. The prgdev would offer the interface to allow the DPDK
> +application to program the device.
> +
> +Dyanamic bind/unbind
> +--------------------
> +
> +Besides the simple API to upload/download image, the prgdev also introduces a
> +mechanism internally to switch drivers and register/unregister device on the
> +fly. With this mechanism, apps can use the programmable device, unbind other
> +personalities on demand, then program it and bind it back with new personalities
> +. Apps can follow below examples to simply complete the whole process.
> +
> +Note that bind/unbind actions are different concept from that a whole device
> +attach/detach. For example, rte_eal_dev_detach(), which will try to detach the
> +drivers with device and remove the whole device from specific class of devices
> +(ethdev, for example). Then, the whole device is invisible until a new 'probe'
> +is activated.
> +
> +During the whole procedure of image upload/download, prgdev handler is always
> +valid and apps can use it operate on programmable device. The reason why unbind
> +is necessary is it may break the hardware when programming is in progress while
> +other functions are active. Using programmble NIC as an example, it's a little
> +impossible to receive/forward packets for hardware while updating firmware. In
> +this case, apps need to detach ethdev function before firmware upgrading. Again,
> +prgdev handler is still valid, it's function-level detach, different from
> +device-level detach. After finishing image download, original function needs to
> +attach back, either use same or different drivers, depends on personalities
> +changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
> +may not, see below examples to get more details.
> +
> +Another reason to provide bind/unbind action is programmble devices, like FPGA,
> +are not identified driver by 'vendor ID' and 'device ID', they might not be
> +changed in all the ways, even FPGA is fully programmed. So, it depends on
> +internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
> +drivers, either a register value or signature, depending on FPGA hardware
> +design. In this case, EAL or other bus driver doesn't have the capability to
> +identify proper driver for FPGA device. With prgdev introduced, while FPGA is
> +always a prgdev, FPGA can use prgdev as primary driver, to find proper function
> +driver.
> +
> +The bind/unbind actions are not trying to break the general initialization
> +procedure in DPDK. prgdev still follow current EAL initializatio model to probe
> +drivers for devices. See below examples for details.
> +
> +It's not true that bind/unbind is always necessary for programming. If device
> +has a capability to perform specific functions and program in parallel, then
> +apps needn't do bind/unbind actions. See 'dynamic program' section below.
> +
> +Authentication
> +--------------
> +
> +Authentication and permissions for programming the device are outside the scope
> +of this API. It is dependent upon a different mechanism not to be supported by
> +DPDK. Application can utilize a different mechanism to authenticate user and
> +or 'program' as validated for loading.
> +
> +Misc
> +----
> +
> +PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
> +device can be loaded with a personality, and it would be up to this programmable
> +device to load the appropriate "personality driver" based upon an identification
> +mechanism to work with the loaded personality.
> +
> +
> +prgdev API
> +==========
> +
> +Below is the major APIs exposed to apps.
> +
> +- rte_prgdev_get_info
> +  Acquire general info of the prgdev.
> +- rte_prgdev_open
> +  Open a prgdev for later programming. If failed, needs to call below 'unbind'
> +  function to detach other functions except prgdev first.
> +- rte_prgdev_img_download
> +  Download a piece of buffer from host to the device after 'open' succeded.
> +- rte_prgdev_img_upload
> +  Upload a piece of information from the device to host.
> +- rte_prgdev_check_status
> +  Even download process is successful, it's necessary to check the status of the
> +  downloaded image if worked in expected way by this function. Harware or driver
> +  may not have the capability to check that status, so it's optional to driver.
> +- rte_prgdev_close
> +  After upload/download actions, close the prgdev.
> +- rte_prgdev_bind
> +  After 'close' action, app can call it to attach the 'new' personality with
> +  proper drivers, then expose to app with new functions. prgdev driver will be
> +  responsible for finding proper drivers to drive the functions. Desired driver
> +  will be responsible for exposing the functions to apps.
> +- rte_prgdev_unbind
> +  When app are using the device, they need to call this function to unbind the
> +  misc functions except prgdev with drivers prior to call 'open' function.
> +
> +Initialization procedure
> +========================
> +In this section, there 2 examples, FPGA and programmable NIC, used as examples
> +to describe the initialization procedure on how programmable devices are probed
> +, initialized and registered as different devices. In which, FPGA has 2
> +different possible cases, blank and prior programmed FPGAs. Note that prgdev
> +depends and follow on EAL initialization process to probe drivers for prgdev.
> +
> +Blank FPGA
> +----------
> +Below is the diagram on how Blank FPGA are probed and registered as a prgdev
> +device that expose to apps.
> +
> +When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
> +ID and device ID, then perform HW initialization if both ID match. After then,
> +it registered in prgdev abstract layer as a new device with a set of helper
> +function ops.
> +
> +After that, apps will observe single device: prgdev device. Until now, the FPGA
> +device is a simply device without any functionalities from application's
> +perspective.
> +
> +In the following action, app will call 'prgdev bind' function with the device
> +ID of the FPGA device to bind the function within the FPGA to specific driver.
> +FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
> +and find potential function. Since it's a blank without any prior programming, a
> +simply SUCCESS return.
> +
> +          Abstraction                 FPGA driver             Hardware
> +             layer
> +               +                       +                       +
> +               |                       |                       |
> +               +-----eal pci probe----->                       |
> +               |                       |                       | +------------+
> +               |                       |                       | |  FPGA dev  |
> +               |                       +--HW initialization----> |            |
> ++------------+ |                       |                       | |+----------+|
> +| prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id  ||
> ++------------+ |                       |                       | |+----------+|
> +               |                       |                       | +------------+
> +               +-----prgdev bind------->                       |
> +               |                       |                       |
> +               |                       +--Read and parse AFU--->
> +               |                       |                       |
> +               |                       <--No personality found-+
> +               |                       |                       |
> +               <-------SUCCESS---------+                       |
> +               |                       |                       |
> +               +                       +                       +
> +
> +prior programmed FPGA
> +---------------------
> +
> +For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
> +except the action of reading signature, as below flow table indicates. It will
> +read valid functions, crypto and ethdev, as an example, then find proper driver
> +to initialize the functions, then register as different ports.
> +
> +After initialization, app will observe 3 kinds of devices, a prgdev device, a
> +ethdev device and a cryptodev device, which all comes from a single PCIE device.
> +
> +
> +          Abstraction            FPGA driver               Hardware
> +             layer
> +               +                       +                       +
> +               |                       |                       |
> +               +----eal pci probe------>                       |
> +               |                       |                       | +-------------+
> +               |                       |                       | |  FPGA dev   |
> +               |                       +--HW initialization----> |             |
> ++------------+ |                       |                       | +-------------+
> +| prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id   ||
> ++------------+ |                       |                       | |-------------|
> +               |                       |                       | +-------------+
> +               +-----prgdev bind------->                       |
> +               |                       |                       |
> +               |                       +--Read and parse AFU--->
> +               |                       |                       |
> +               |                       <--eth and crypto found-+
> ++------------+ |                       |                       |
> +| ethdev port| <---ethdev attach-------+                       |
> ++------------+ |                       |                       |
> +               |                       |                       |
> ++------------+ |                       |                       |
> +| crypto port| <---crypto attach-------+                       |
> ++------------+ |                       |                       |
> +               |                       |                       |
> +               +                       +                       +
> +
> +
> +Programmable NIC
> +----------------
> +
> +Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
> +initialization procedure is much more simpler than FPGA. Below is a flow table
> +to describe this.
> +
> +The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
> +checking that the NIC is programmable. What is different from FPGA is it
> +registered itself as a ethdev device in the following action due to the NIC
> +nature.
> +
> +Then, app still follow the common process to call 'prgdev bind', while the NIC
> +driver does nothing and just return a SUCCESS.
> +
> +          Abstraction            NIC driver                Hardware
> +             layer
> +               +                       +                       +
> +               |                       |                       |
> +               +----eal pci probe------>                       |
> +               |                       |                       | +-------------+
> +               |                       +--HW initialization----> |Programmable |
> +               |                       |                       | |     NIC     |
> ++------------+ |                       +---Capability check----> +-------------+
> +| prgdev dev | <--prgdev_ops_register--+                       |
> ++------------+ |                       |                       |
> +               |                       |                       |
> ++------------+ |                       |                       |
> +| ethdev port| <---ethdev attach-------+                       |
> ++------------+ |                       |                       |
> +               |                       |                       |
> +               +-----prgdev bind------->                       |
> +               |                       |                       |
> +               |                       |                       |
> +               <--------SUCCESS--------+                       |
> +               |                       |                       |
> +               +                       +                       +
> +
> +
> +Program procedure
> +=================
> +
> +This section will describe how apps finish an image downloading procedure with
> +the prgdev API. Similar to previous section, it will use several FPGA and NIC
> +examples to show the common and different steps.
> +
> +Needs to point out, 'rte_prgdev_img_download' function didn't carry the info
> +of the image, or where downloading the image to. The interpretion info of the
> +image can be hidden into the buffer itself.
> +
> +Common programming steps
> +------------------------
> +
> +Before reaching details, it needs to introduce the common steps for apps to
> +follow in order to download the image to the hardware.
> +
> +Before programming, apps need to call function 'rte_prgdev_open' to open the
> +programming interface. If failed, either functions residing in PCI-E device is
> +in use or the PCI-E device is not ready, the programming steps are different.
> +
> +Below is the steps that 'open' succeeds:
> +1. rte_prgdev_open --> SUCCESS
> +2. rte_prgdev_img_download
> +3. rte_prgdev_check_status
> +4. rte_prgdev_close
> +5. rte_prgdev_bind
> +
> +If failed to open the programming interface, apps need to follow below steps:
> +1. rte_prgdev_open --> FAILURE
> +2. Close all related functions
> +3. ret_prgdev_unbind
> +4. rte_prgdev_open --> SUCCESS
> +5. rte_prgdev_img_download
> +6. rte_prgdev_check_status
> +7. rte_prgdev_close
> +8. rte_prgdev_bind
> +
> +Detach functions
> +----------------
> +
> +As discussed before, apps need to follow steps to detach all the other functions
> +except prgdev when meeting error for 'open' operation. Below is a FPGA example
> +with 'ethdev' and 'cryptodev' function attached.
> +
> +After meeting error, app needs to close ethdev port and cryptodev port first.
> +Then, use 'prgdev unbind' to detach all the functions that registered previously
> +. FPGA driver will call each 'detach' function for each registered device class
> +respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
> +available.
> +
> +
> +          Abstraction             FPGA driver              Hardware
> +             layer
> +               +                       +                       +
> ++------------+ |                       |                       |
> +| prgdev dev | +-------prgdev open----->                       |
> +|            | <-------ERR_IN_USE------+                       | +-------------+
> ++------------+ |                       |                       | |  FPGA dev   |
> +               |                       |                       | | +---------+ |
> ++------------+ |                       |                       | | | AFU id  | |
> +| ethdev port| +---rte_ethdev_close---->                       | | +---------+ |
> +|            |                         +------ethdev close-----> +-------------+
> ++------------+ <---------SUCCESS-------+                       |
> +               |                       |                       |
> ++------------+ |                       |                       |
> +| crypto port| +---crypto port close--->                       |
> +|            | |                       +---crypto func close--->
> ++------------+ <---------SUCCESS-------+                       |
> +               |                       |                       |
> +               +-----prgdev unbind----->                       |
> +               |                       |                       |
> ++ - - - - - -+ |                       |                       |
> +  ethdev port  <-----ethdev detach-----+                       |
> ++- - - - - - + |                       |                       |
> +               |                       |                       |
> ++ - - - - - -+ |                       |                       |
> +  crypto port  <----cryptodev detach---+                       |
> ++- - - - - - + |                       |                       |
> +               +                       +                       +
> +
> +Device programming
> +------------------
> +
> +After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
> +'close' sequence respectively. After then, they need to 'bind' new functions
> +back with 'prgdev bind' call.
> +
> +Still re-use previous FPGA example, below is the execution sequence. After image
> +downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
> +registered and 2 new ports are presented to apps.
> +
> +
> +          Abstraction             FPGA driver              Hardware
> +             layer
> +               +                       +                       + +-------------+
> ++------------+ |                       |                       | |  FPGA dev   |
> +|            | +-----prgdev open------->                       | | +---------+ |
> +|            | <-------SUCCESS---------+                       | | | ethdev  | |
> +|            | |                       |                       | | +---------+ |
> +|            | +--prgdev_img_download-->                       | | +---------+ |
> +|            | |                       +----Download image-----> | | crypto  | |
> +|            | |                       <--------SUCCESS--------+ | +---------+ |
> +|            | <---------SUCCESS-------+                       | +-------------+
> +| prgdev dev | |                       |                       |        |
> +|            | +---prgdev_chk_status--->                       |        |
> +|            | |                       +--Verify image status-->        |
> +|            | |                       <--------SUCCESS--------+        |
> +|            | <--------SUCCESS--------+                       |        |
> +|            | +-------prgdev close---->                       |        v
> +|            | |                       |                       | +-------------+
> +|            | +-------prgdev bind----->                       | |  FPGA dev   |
> ++------------+ |                       |                       | | +---------+ |
> +               |                       <--Virtio and flow dev--+ | | virtio  | |
> ++------------+ |                       |                       | | +---------+ |
> +| ethdev port| <-----ethdev attach-----+                       | | +---------+ |
> ++------------+ |                       |                       | | |Flow dev | |
> +               |                       |                       | | +---------+ |
> ++------------+ |                       |                       | +-------------+
> +| Flow port  | <-----Flow dev attach---+                       |
> ++------------+ |                       |                       |
> +               +                       +                       +
> +
> +
> +Dyanamic program
> +----------------
> +
> +If prgdev device has a capability to be programmed while functions are in use,
> +then we say it supports dynamic program. It's possible to update a profile while
> +device is used as a ethdev device.
> +
> +The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
> +'rte_prgdev_get_info'.
> +
> +The dynamic programming procedure is the same as previous section shows, while
> +ignore the steps to 'close' and 'unbind' actions when device is in use.
> +
> +Note that prgdev driver needs to consider if it can support dynamic program to
> +avoid any hardware or software error.
> +
> +
> +Source code layout
> +==================
> +
> +Below is the suggested source code layout.
> +- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
> +  doesn't change programmable NIC's directory and remains the same. The PMD
> +  driver should include 2 parts, one part is the code to support rte_ethdev API,
> +  another part is the code to support rte_prgdev API.
> +- 'librte_prgdev' under 'lib' is the new directory store definition and
> +  implementation of the generic prgdev API.
> +- 'prg' under 'drivers' is the directory to store specfic implementation
> +  driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
> +  support specific hardware product.
> +
> +                +---------+                         +---------+
> +                | drivers |                         |   lib   |
> +                +---------+                         +----+----+
> +                     |                                   |
> +          +----------+-----------+                       |
> +          |                      |                       |
> +      +---v---+              +---v----+          +-------v-------+
> +      |  net  |              |   prg  |          | librte_prgdev |
> +      +---+---+              +---+----+          +---------------+
> +          |                      |
> +    +-----+----+           +-----+----+
> +    |          |           |          |
> ++---v----+ +---v----+  +---v----+ +---v----+
> +|  i40e  | | prg NIC|  |FPGA ABC| |GPU XYZ |
> ++--------+ +--------+  +--------+ +--------+
> +
> +
> --
> 1.7.7.6
>
Cunming Liang Feb. 3, 2017, 9:21 a.m. UTC | #2
Hi,


On 2/1/2017 7:41 PM, Jan Blunck wrote:
> On Fri, Jan 20, 2017 at 4:21 AM, Chen Jing D(Mark)
> <jing.d.chen@intel.com> wrote:
>> This is the documentation to describe what prgdev is, how to use
>> prgdev API and accomplish an image download.
>>
>> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
>> ---
>>   doc/guides/prog_guide/prgdev_lib.rst |  457 ++++++++++++++++++++++++++++++++++
>>   1 files changed, 457 insertions(+), 0 deletions(-)
>>   create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
[...]
>  From my point of view this doesn't belong into the DPDK. On Linux this
> is traditionally handled by udev and you already have the freedom to
> use userspace applications to program a device requiring firmware in
> that case. I don't believe that modeling this in the DPDK explicitly
> is the right thing to do.
Good point, but not sure udev has user space device driver support or not.
>
> Especially if the device supports changing personality it is required
> to unplug the existing personality before reprogramming. You can do
> this already today. Also writing OOB firmware data that changes
> configuration should be possible today by handling interrupts.

It's going to allow changing personality in DPDK user space runtime.

If the personality is not belong to a device but part of the component, 
unplug isn't helpful too much.

>
> Maybe we can come up with an example application that demonstrates how
> the different infrastructure components could get orchestrated. Do you
> have a device in mind that supports this?

The coming Purley platform has SKU for Xeon-FPGA.

The FPGA connecting with Xeon has dedicated pcie device id.

The AFU personality for packet I/O depends on the RTL image.

Changing the personality in runtime could be one of the situation.



Regards,

Cunming

>
> Regards,
> Jan
>
[...]
Chen, Jing D Feb. 6, 2017, 9:16 a.m. UTC | #3
Hi, Jan,


On 2/1/2017 7:41 PM, Jan Blunck wrote:
> On Fri, Jan 20, 2017 at 4:21 AM, Chen Jing D(Mark)
> <jing.d.chen@intel.com> wrote:
>> This is the documentation to describe what prgdev is, how to use
>> prgdev API and accomplish an image download.
>>
>> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
>> ---
>>   doc/guides/prog_guide/prgdev_lib.rst |  457 ++++++++++++++++++++++++++++++++++
>>   1 files changed, 457 insertions(+), 0 deletions(-)
>>   create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
>>
>> diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
>> new file mode 100644
>> index 0000000..3917c18
>> --- /dev/null
>> +++ b/doc/guides/prog_guide/prgdev_lib.rst
>> @@ -0,0 +1,457 @@
>> +Overview
>> +========
>> +
>> +More and more devices start to support on-die programming, which include
>> +NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
>> +introduce a API for application to download/upload image to/from the HW.
>> +So, it's necessary to provide a generic API to perform image download, upload,
>> +validation, etc.
>> +
>> +This RFC patch intends to introduce a DPDK generic programming device layer,
>> +called prgdev below, to provide an abstract, generic APIs for applications to
>> +program hardware without knowing the details of programmable devices. From
>> +driver's perspective, they'll try to adapt their functions to the abstract
>> +APIs defined in prgdev.
>> +
>> +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
>> +for FPGA devices, or upgrade firmware for programmble NICs.
>> +
>> +But those devices that have the capability to do on-die programming may
>> +expose versatile talents to apps. Add/delete/update entries in flow table,
>> +update/overwrite the firmware/microcode, add a new algorithm, update profiles
>> +, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
>> +include the APIs to perform device-unique actions in current stage until it
>> +became popular. On the contratry, it only provides the simple capability to
>> +download a segment of buffer(image) from host to on-die device, or vice versa.
>> +The image could be an algorithm, a flow table in the on-die SRAM, a firmware
>> +image running in the device, a function within a pipeline, etc. prgdev won't
>> +try to inteprete the content and it's transparent to prgdev. Apps and devices
>> +can communicate with a language they can understand each other, which is not
>> +provided by prgdev to decide what needs to be programmed.
>> +
>> +When the set of APIs is introduced, a general question is why we need it in
>> +DPDK community? Why we can't use offline tool to perform same thing? The answer
>> +is the prgdev provide a generic, online API to applications, in the meanwile,
>> +offers a capability to switch driver dynamically when downloaded image changed
>> +personality and a new driver is required. Comparing offline tool, it have online
>> +programmability (see below examples); Comparing online tool, it provides a
>> +generic API for many HWs; Comparing generic online tool/API for many products,
>> +it provides a capability to switch driver dynamically.
>> +
>> +There are various means to reach same goal, we'll try to find the best/better
>> +way to approach. All above advantages will help prgdev to be a 'better choice'.
>> +
>  From my point of view this doesn't belong into the DPDK. On Linux this
> is traditionally handled by udev and you already have the freedom to
> use userspace applications to program a device requiring firmware in
> that case. I don't believe that modeling this in the DPDK explicitly
> is the right thing to do.

Can you kindly educate me what udev can do? It's for NIC firmware upgrade?
This prgdev is not only for NIC, the major use case is FPGA, but I found 
programmable
NIC is also applicable to the API model and can be added into the scope.

>
> Especially if the device supports changing personality it is required
> to unplug the existing personality before reprogramming. You can do
> this already today. Also writing OOB firmware data that changes
> configuration should be possible today by handling interrupts.

What if application is using DPDK, can we do in the same manner without 
leaving
DPDK instance?

> Maybe we can come up with an example application that demonstrates how
> the different infrastructure components could get orchestrated. Do you
> have a device in mind that supports this?

As Cunming suggested, the incoming FPGA is the desired device to use 
this API. OVS
that is using DPDK can benefit from the API and download personalities 
in running
time for blank FPGA, or upgrade for prior-programmed case.

>
> Regards,
> Jan
Cunming Liang Feb. 8, 2017, 3:30 a.m. UTC | #4
On 1/20/2017 11:21 AM, Chen Jing D(Mark) wrote:
> This is the documentation to describe what prgdev is, how to use
> prgdev API and accomplish an image download.
>
> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
> ---
>   doc/guides/prog_guide/prgdev_lib.rst |  457 ++++++++++++++++++++++++++++++++++
>   1 files changed, 457 insertions(+), 0 deletions(-)
>   create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
>
> diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
> new file mode 100644
> index 0000000..3917c18
> --- /dev/null
> +++ b/doc/guides/prog_guide/prgdev_lib.rst
> @@ -0,0 +1,457 @@
> +Overview
> +========
> +
[...]
> +When the set of APIs is introduced, a general question is why we need it in
> +DPDK community? Why we can't use offline tool to perform same thing? The answer
> +is the prgdev provide a generic, online API to applications, in the meanwile,
> +offers a capability to switch driver dynamically when downloaded image changed
> +personality and a new driver is required. Comparing offline tool, it have online
> +programmability (see below examples); Comparing online tool, it provides a
> +generic API for many HWs; Comparing generic online tool/API for many products,
> +it provides a capability to switch driver dynamically.
> +
> +There are various means to reach same goal, we'll try to find the best/better
> +way to approach. All above advantages will help prgdev to be a 'better choice'.
> +
>
One more notes.
DPDK takes over the devices in user space. The legacy tools usually 
download the personality by kernel driver. They runs out of DPDK 
context. When a DPDK process is running on top of the device, operation 
to the device by the solo tool during the time causes resource conflict. 
It's one of the motivations to have the native API allowing programming 
the personality within DPDK context. Otherwise, it has to exit the 
process if any personality change is required. Manually detaching the 
device before using the solo tool may ease the conflict. However it 
still limits the situation if the device allows multiple programmable 
instances(e.g. AFUs in FPGA) which are working independently and 
shouldn't be impact by each other.


[...]
diff mbox

Patch

diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
new file mode 100644
index 0000000..3917c18
--- /dev/null
+++ b/doc/guides/prog_guide/prgdev_lib.rst
@@ -0,0 +1,457 @@ 
+Overview
+========
+
+More and more devices start to support on-die programming, which include
+NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
+introduce a API for application to download/upload image to/from the HW.
+So, it's necessary to provide a generic API to perform image download, upload,
+validation, etc.
+
+This RFC patch intends to introduce a DPDK generic programming device layer,
+called prgdev below, to provide an abstract, generic APIs for applications to
+program hardware without knowing the details of programmable devices. From
+driver's perspective, they'll try to adapt their functions to the abstract
+APIs defined in prgdev.
+
+The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
+for FPGA devices, or upgrade firmware for programmble NICs.
+
+But those devices that have the capability to do on-die programming may
+expose versatile talents to apps. Add/delete/update entries in flow table,
+update/overwrite the firmware/microcode, add a new algorithm, update profiles
+, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
+include the APIs to perform device-unique actions in current stage until it
+became popular. On the contratry, it only provides the simple capability to
+download a segment of buffer(image) from host to on-die device, or vice versa. 
+The image could be an algorithm, a flow table in the on-die SRAM, a firmware
+image running in the device, a function within a pipeline, etc. prgdev won't
+try to inteprete the content and it's transparent to prgdev. Apps and devices
+can communicate with a language they can understand each other, which is not
+provided by prgdev to decide what needs to be programmed.
+
+When the set of APIs is introduced, a general question is why we need it in
+DPDK community? Why we can't use offline tool to perform same thing? The answer
+is the prgdev provide a generic, online API to applications, in the meanwile,
+offers a capability to switch driver dynamically when downloaded image changed
+personality and a new driver is required. Comparing offline tool, it have online
+programmability (see below examples); Comparing online tool, it provides a
+generic API for many HWs; Comparing generic online tool/API for many products,
+it provides a capability to switch driver dynamically.
+
+There are various means to reach same goal, we'll try to find the best/better
+way to approach. All above advantages will help prgdev to be a 'better choice'.
+
+
+Scope
+-----
+
+The target devices include but not limited to programmable NIC, FPGA, GPU.
+Any devices, having the capability to store/load a piece of info to/from
+the deivce then changed hardware behavior, and applicable to prgdev programming
+model, could be registered as a prgdev device.
+
+The device can be programmed (dynamic) within DPDK or have been prior programmed
+(static) prior to a DPDK application being launched.
+
+Static example: In a static example a program, bitstream or other firmware
+would have been loaded prior to the start of a DPDK application. The prgdev
+would recognize the firmware loaded, and instantiate higher level interfaces
+(example : PMD, L3 forwarding, flow, etc.).
+
+Dynamic example: In a dynamic example, the programmability of the program,
+bitstream or other firmware would be loaded by the DPDK application upon start
+, or during execution. The prgdev would offer the interface to allow the DPDK
+application to program the device.
+
+Dyanamic bind/unbind
+--------------------
+
+Besides the simple API to upload/download image, the prgdev also introduces a
+mechanism internally to switch drivers and register/unregister device on the
+fly. With this mechanism, apps can use the programmable device, unbind other
+personalities on demand, then program it and bind it back with new personalities
+. Apps can follow below examples to simply complete the whole process.
+
+Note that bind/unbind actions are different concept from that a whole device
+attach/detach. For example, rte_eal_dev_detach(), which will try to detach the
+drivers with device and remove the whole device from specific class of devices
+(ethdev, for example). Then, the whole device is invisible until a new 'probe'
+is activated.
+
+During the whole procedure of image upload/download, prgdev handler is always
+valid and apps can use it operate on programmable device. The reason why unbind
+is necessary is it may break the hardware when programming is in progress while
+other functions are active. Using programmble NIC as an example, it's a little
+impossible to receive/forward packets for hardware while updating firmware. In
+this case, apps need to detach ethdev function before firmware upgrading. Again,
+prgdev handler is still valid, it's function-level detach, different from
+device-level detach. After finishing image download, original function needs to
+attach back, either use same or different drivers, depends on personalities
+changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
+may not, see below examples to get more details.
+
+Another reason to provide bind/unbind action is programmble devices, like FPGA,
+are not identified driver by 'vendor ID' and 'device ID', they might not be
+changed in all the ways, even FPGA is fully programmed. So, it depends on
+internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
+drivers, either a register value or signature, depending on FPGA hardware
+design. In this case, EAL or other bus driver doesn't have the capability to
+identify proper driver for FPGA device. With prgdev introduced, while FPGA is
+always a prgdev, FPGA can use prgdev as primary driver, to find proper function
+driver.
+
+The bind/unbind actions are not trying to break the general initialization
+procedure in DPDK. prgdev still follow current EAL initializatio model to probe
+drivers for devices. See below examples for details.
+
+It's not true that bind/unbind is always necessary for programming. If device
+has a capability to perform specific functions and program in parallel, then
+apps needn't do bind/unbind actions. See 'dynamic program' section below.
+
+Authentication
+--------------
+
+Authentication and permissions for programming the device are outside the scope
+of this API. It is dependent upon a different mechanism not to be supported by
+DPDK. Application can utilize a different mechanism to authenticate user and
+or 'program' as validated for loading.
+
+Misc
+----
+
+PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
+device can be loaded with a personality, and it would be up to this programmable
+device to load the appropriate "personality driver" based upon an identification
+mechanism to work with the loaded personality.
+
+
+prgdev API
+==========
+
+Below is the major APIs exposed to apps.
+
+- rte_prgdev_get_info
+  Acquire general info of the prgdev.
+- rte_prgdev_open
+  Open a prgdev for later programming. If failed, needs to call below 'unbind'
+  function to detach other functions except prgdev first.
+- rte_prgdev_img_download
+  Download a piece of buffer from host to the device after 'open' succeded.
+- rte_prgdev_img_upload
+  Upload a piece of information from the device to host.
+- rte_prgdev_check_status
+  Even download process is successful, it's necessary to check the status of the
+  downloaded image if worked in expected way by this function. Harware or driver
+  may not have the capability to check that status, so it's optional to driver.
+- rte_prgdev_close
+  After upload/download actions, close the prgdev.
+- rte_prgdev_bind
+  After 'close' action, app can call it to attach the 'new' personality with
+  proper drivers, then expose to app with new functions. prgdev driver will be
+  responsible for finding proper drivers to drive the functions. Desired driver
+  will be responsible for exposing the functions to apps.
+- rte_prgdev_unbind
+  When app are using the device, they need to call this function to unbind the
+  misc functions except prgdev with drivers prior to call 'open' function.
+
+Initialization procedure
+========================
+In this section, there 2 examples, FPGA and programmable NIC, used as examples
+to describe the initialization procedure on how programmable devices are probed
+, initialized and registered as different devices. In which, FPGA has 2
+different possible cases, blank and prior programmed FPGAs. Note that prgdev
+depends and follow on EAL initialization process to probe drivers for prgdev.
+
+Blank FPGA
+----------
+Below is the diagram on how Blank FPGA are probed and registered as a prgdev
+device that expose to apps. 
+
+When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
+ID and device ID, then perform HW initialization if both ID match. After then,
+it registered in prgdev abstract layer as a new device with a set of helper
+function ops.
+
+After that, apps will observe single device: prgdev device. Until now, the FPGA
+device is a simply device without any functionalities from application's
+perspective.
+
+In the following action, app will call 'prgdev bind' function with the device
+ID of the FPGA device to bind the function within the FPGA to specific driver.
+FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
+and find potential function. Since it's a blank without any prior programming, a
+simply SUCCESS return.
+                     
+          Abstraction                 FPGA driver             Hardware
+             layer
+               +                       +                       +
+               |                       |                       |
+               +-----eal pci probe----->                       |
+               |                       |                       | +------------+
+               |                       |                       | |  FPGA dev  |
+               |                       +--HW initialization----> |            |
++------------+ |                       |                       | |+----------+|
+| prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id  ||
++------------+ |                       |                       | |+----------+|
+               |                       |                       | +------------+
+               +-----prgdev bind------->                       |
+               |                       |                       |
+               |                       +--Read and parse AFU--->
+               |                       |                       |
+               |                       <--No personality found-+
+               |                       |                       |
+               <-------SUCCESS---------+                       |
+               |                       |                       |
+               +                       +                       +
+
+prior programmed FPGA
+---------------------
+
+For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
+except the action of reading signature, as below flow table indicates. It will
+read valid functions, crypto and ethdev, as an example, then find proper driver
+to initialize the functions, then register as different ports.
+
+After initialization, app will observe 3 kinds of devices, a prgdev device, a
+ethdev device and a cryptodev device, which all comes from a single PCIE device.
+
+ 
+          Abstraction            FPGA driver               Hardware
+             layer
+               +                       +                       +
+               |                       |                       |
+               +----eal pci probe------>                       |
+               |                       |                       | +-------------+
+               |                       |                       | |  FPGA dev   |
+               |                       +--HW initialization----> |             |
++------------+ |                       |                       | +-------------+
+| prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id   ||
++------------+ |                       |                       | |-------------|
+               |                       |                       | +-------------+
+               +-----prgdev bind------->                       |
+               |                       |                       |
+               |                       +--Read and parse AFU--->
+               |                       |                       |
+               |                       <--eth and crypto found-+
++------------+ |                       |                       |
+| ethdev port| <---ethdev attach-------+                       |
++------------+ |                       |                       |
+               |                       |                       |
++------------+ |                       |                       |
+| crypto port| <---crypto attach-------+                       |
++------------+ |                       |                       |
+               |                       |                       |
+               +                       +                       +
+
+
+Programmable NIC
+----------------
+
+Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
+initialization procedure is much more simpler than FPGA. Below is a flow table
+to describe this.
+
+The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
+checking that the NIC is programmable. What is different from FPGA is it
+registered itself as a ethdev device in the following action due to the NIC
+nature.
+
+Then, app still follow the common process to call 'prgdev bind', while the NIC
+driver does nothing and just return a SUCCESS.
+
+          Abstraction            NIC driver                Hardware
+             layer
+               +                       +                       +
+               |                       |                       |
+               +----eal pci probe------>                       |
+               |                       |                       | +-------------+
+               |                       +--HW initialization----> |Programmable |
+               |                       |                       | |     NIC     |
++------------+ |                       +---Capability check----> +-------------+
+| prgdev dev | <--prgdev_ops_register--+                       |
++------------+ |                       |                       |
+               |                       |                       |
++------------+ |                       |                       |
+| ethdev port| <---ethdev attach-------+                       |
++------------+ |                       |                       |
+               |                       |                       |
+               +-----prgdev bind------->                       |
+               |                       |                       |
+               |                       |                       |
+               <--------SUCCESS--------+                       |
+               |                       |                       |
+               +                       +                       +
+
+
+Program procedure
+=================
+
+This section will describe how apps finish an image downloading procedure with
+the prgdev API. Similar to previous section, it will use several FPGA and NIC
+examples to show the common and different steps.
+
+Needs to point out, 'rte_prgdev_img_download' function didn't carry the info
+of the image, or where downloading the image to. The interpretion info of the
+image can be hidden into the buffer itself.
+
+Common programming steps
+------------------------
+
+Before reaching details, it needs to introduce the common steps for apps to
+follow in order to download the image to the hardware.
+
+Before programming, apps need to call function 'rte_prgdev_open' to open the
+programming interface. If failed, either functions residing in PCI-E device is
+in use or the PCI-E device is not ready, the programming steps are different.
+
+Below is the steps that 'open' succeeds:
+1. rte_prgdev_open --> SUCCESS
+2. rte_prgdev_img_download
+3. rte_prgdev_check_status
+4. rte_prgdev_close
+5. rte_prgdev_bind
+
+If failed to open the programming interface, apps need to follow below steps:
+1. rte_prgdev_open --> FAILURE
+2. Close all related functions
+3. ret_prgdev_unbind
+4. rte_prgdev_open --> SUCCESS
+5. rte_prgdev_img_download
+6. rte_prgdev_check_status
+7. rte_prgdev_close
+8. rte_prgdev_bind
+
+Detach functions
+----------------
+
+As discussed before, apps need to follow steps to detach all the other functions
+except prgdev when meeting error for 'open' operation. Below is a FPGA example
+with 'ethdev' and 'cryptodev' function attached.
+
+After meeting error, app needs to close ethdev port and cryptodev port first.
+Then, use 'prgdev unbind' to detach all the functions that registered previously
+. FPGA driver will call each 'detach' function for each registered device class
+respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
+available.
+
+
+          Abstraction             FPGA driver              Hardware
+             layer
+               +                       +                       +
++------------+ |                       |                       |
+| prgdev dev | +-------prgdev open----->                       |
+|            | <-------ERR_IN_USE------+                       | +-------------+
++------------+ |                       |                       | |  FPGA dev   |
+               |                       |                       | | +---------+ |
++------------+ |                       |                       | | | AFU id  | |
+| ethdev port| +---rte_ethdev_close---->                       | | +---------+ |
+|            |                         +------ethdev close-----> +-------------+
++------------+ <---------SUCCESS-------+                       |
+               |                       |                       |
++------------+ |                       |                       |
+| crypto port| +---crypto port close--->                       |
+|            | |                       +---crypto func close--->
++------------+ <---------SUCCESS-------+                       |
+               |                       |                       |
+               +-----prgdev unbind----->                       |
+               |                       |                       |
++ - - - - - -+ |                       |                       |
+  ethdev port  <-----ethdev detach-----+                       |
++- - - - - - + |                       |                       |
+               |                       |                       |
++ - - - - - -+ |                       |                       |
+  crypto port  <----cryptodev detach---+                       |
++- - - - - - + |                       |                       |
+               +                       +                       +
+
+Device programming
+------------------
+
+After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
+'close' sequence respectively. After then, they need to 'bind' new functions
+back with 'prgdev bind' call.
+
+Still re-use previous FPGA example, below is the execution sequence. After image
+downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
+registered and 2 new ports are presented to apps.
+
+
+          Abstraction             FPGA driver              Hardware
+             layer
+               +                       +                       + +-------------+
++------------+ |                       |                       | |  FPGA dev   |
+|            | +-----prgdev open------->                       | | +---------+ |
+|            | <-------SUCCESS---------+                       | | | ethdev  | |
+|            | |                       |                       | | +---------+ |
+|            | +--prgdev_img_download-->                       | | +---------+ |
+|            | |                       +----Download image-----> | | crypto  | |
+|            | |                       <--------SUCCESS--------+ | +---------+ |
+|            | <---------SUCCESS-------+                       | +-------------+
+| prgdev dev | |                       |                       |        |
+|            | +---prgdev_chk_status--->                       |        |
+|            | |                       +--Verify image status-->        |
+|            | |                       <--------SUCCESS--------+        |
+|            | <--------SUCCESS--------+                       |        |
+|            | +-------prgdev close---->                       |        v
+|            | |                       |                       | +-------------+
+|            | +-------prgdev bind----->                       | |  FPGA dev   |
++------------+ |                       |                       | | +---------+ |
+               |                       <--Virtio and flow dev--+ | | virtio  | |
++------------+ |                       |                       | | +---------+ |
+| ethdev port| <-----ethdev attach-----+                       | | +---------+ |
++------------+ |                       |                       | | |Flow dev | |
+               |                       |                       | | +---------+ |
++------------+ |                       |                       | +-------------+
+| Flow port  | <-----Flow dev attach---+                       |
++------------+ |                       |                       |
+               +                       +                       +
+
+
+Dyanamic program
+----------------
+
+If prgdev device has a capability to be programmed while functions are in use,
+then we say it supports dynamic program. It's possible to update a profile while
+device is used as a ethdev device.
+
+The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
+'rte_prgdev_get_info'.
+
+The dynamic programming procedure is the same as previous section shows, while
+ignore the steps to 'close' and 'unbind' actions when device is in use.
+
+Note that prgdev driver needs to consider if it can support dynamic program to
+avoid any hardware or software error.
+
+
+Source code layout
+==================
+
+Below is the suggested source code layout. 
+- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
+  doesn't change programmable NIC's directory and remains the same. The PMD
+  driver should include 2 parts, one part is the code to support rte_ethdev API,
+  another part is the code to support rte_prgdev API.
+- 'librte_prgdev' under 'lib' is the new directory store definition and 
+  implementation of the generic prgdev API.
+- 'prg' under 'drivers' is the directory to store specfic implementation
+  driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
+  support specific hardware product.
+
+                +---------+                         +---------+
+                | drivers |                         |   lib   |
+                +---------+                         +----+----+
+                     |                                   |
+          +----------+-----------+                       |
+          |                      |                       |
+      +---v---+              +---v----+          +-------v-------+
+      |  net  |              |   prg  |          | librte_prgdev |
+      +---+---+              +---+----+          +---------------+
+          |                      |
+    +-----+----+           +-----+----+
+    |          |           |          |
++---v----+ +---v----+  +---v----+ +---v----+
+|  i40e  | | prg NIC|  |FPGA ABC| |GPU XYZ |
++--------+ +--------+  +--------+ +--------+
+
+