From patchwork Fri Jan 20 03:21:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Chen, Jing D" X-Patchwork-Id: 19834 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 772FA326C; Fri, 20 Jan 2017 11:24:56 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id CAE242C06 for ; Fri, 20 Jan 2017 11:24:48 +0100 (CET) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga104.fm.intel.com with ESMTP; 20 Jan 2017 02:24:48 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,258,1477983600"; d="scan'208";a="924722460" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.239.128.150]) by orsmga003.jf.intel.com with ESMTP; 20 Jan 2017 02:24:47 -0800 From: "Chen Jing D(Mark)" To: dev@dpdk.org Cc: keith.wiles@intel.com, gerald.rogers@intel.com, "Chen Jing D(Mark)" Date: Fri, 20 Jan 2017 11:21:37 +0800 Message-Id: <1484882498-18653-2-git-send-email-jing.d.chen@intel.com> X-Mailer: git-send-email 1.9.3 In-Reply-To: <1484882498-18653-1-git-send-email-jing.d.chen@intel.com> References: <1484882498-18653-1-git-send-email-jing.d.chen@intel.com> Subject: [dpdk-dev] [RFC 1/2] doc: introduction to prgdev X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This is the documentation to describe what prgdev is, how to use prgdev API and accomplish an image download. Signed-off-by: Chen Jing D(Mark) --- doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++ 1 files changed, 457 insertions(+), 0 deletions(-) create mode 100644 doc/guides/prog_guide/prgdev_lib.rst diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst new file mode 100644 index 0000000..3917c18 --- /dev/null +++ b/doc/guides/prog_guide/prgdev_lib.rst @@ -0,0 +1,457 @@ +Overview +======== + +More and more devices start to support on-die programming, which include +NIC, GPU, FPGA, etc. This capability has a requirement to drivers that +introduce a API for application to download/upload image to/from the HW. +So, it's necessary to provide a generic API to perform image download, upload, +validation, etc. + +This RFC patch intends to introduce a DPDK generic programming device layer, +called prgdev below, to provide an abstract, generic APIs for applications to +program hardware without knowing the details of programmable devices. From +driver's perspective, they'll try to adapt their functions to the abstract +APIs defined in prgdev. + +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images +for FPGA devices, or upgrade firmware for programmble NICs. + +But those devices that have the capability to do on-die programming may +expose versatile talents to apps. Add/delete/update entries in flow table, +update/overwrite the firmware/microcode, add a new algorithm, update profiles +, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't +include the APIs to perform device-unique actions in current stage until it +became popular. On the contratry, it only provides the simple capability to +download a segment of buffer(image) from host to on-die device, or vice versa. +The image could be an algorithm, a flow table in the on-die SRAM, a firmware +image running in the device, a function within a pipeline, etc. prgdev won't +try to inteprete the content and it's transparent to prgdev. Apps and devices +can communicate with a language they can understand each other, which is not +provided by prgdev to decide what needs to be programmed. + +When the set of APIs is introduced, a general question is why we need it in +DPDK community? Why we can't use offline tool to perform same thing? The answer +is the prgdev provide a generic, online API to applications, in the meanwile, +offers a capability to switch driver dynamically when downloaded image changed +personality and a new driver is required. Comparing offline tool, it have online +programmability (see below examples); Comparing online tool, it provides a +generic API for many HWs; Comparing generic online tool/API for many products, +it provides a capability to switch driver dynamically. + +There are various means to reach same goal, we'll try to find the best/better +way to approach. All above advantages will help prgdev to be a 'better choice'. + + +Scope +----- + +The target devices include but not limited to programmable NIC, FPGA, GPU. +Any devices, having the capability to store/load a piece of info to/from +the deivce then changed hardware behavior, and applicable to prgdev programming +model, could be registered as a prgdev device. + +The device can be programmed (dynamic) within DPDK or have been prior programmed +(static) prior to a DPDK application being launched. + +Static example: In a static example a program, bitstream or other firmware +would have been loaded prior to the start of a DPDK application. The prgdev +would recognize the firmware loaded, and instantiate higher level interfaces +(example : PMD, L3 forwarding, flow, etc.). + +Dynamic example: In a dynamic example, the programmability of the program, +bitstream or other firmware would be loaded by the DPDK application upon start +, or during execution. The prgdev would offer the interface to allow the DPDK +application to program the device. + +Dyanamic bind/unbind +-------------------- + +Besides the simple API to upload/download image, the prgdev also introduces a +mechanism internally to switch drivers and register/unregister device on the +fly. With this mechanism, apps can use the programmable device, unbind other +personalities on demand, then program it and bind it back with new personalities +. Apps can follow below examples to simply complete the whole process. + +Note that bind/unbind actions are different concept from that a whole device +attach/detach. For example, rte_eal_dev_detach(), which will try to detach the +drivers with device and remove the whole device from specific class of devices +(ethdev, for example). Then, the whole device is invisible until a new 'probe' +is activated. + +During the whole procedure of image upload/download, prgdev handler is always +valid and apps can use it operate on programmable device. The reason why unbind +is necessary is it may break the hardware when programming is in progress while +other functions are active. Using programmble NIC as an example, it's a little +impossible to receive/forward packets for hardware while updating firmware. In +this case, apps need to detach ethdev function before firmware upgrading. Again, +prgdev handler is still valid, it's function-level detach, different from +device-level detach. After finishing image download, original function needs to +attach back, either use same or different drivers, depends on personalities +changed or not. For a programmble NIC, the driver may be the same. For FPGA, it +may not, see below examples to get more details. + +Another reason to provide bind/unbind action is programmble devices, like FPGA, +are not identified driver by 'vendor ID' and 'device ID', they might not be +changed in all the ways, even FPGA is fully programmed. So, it depends on +internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper +drivers, either a register value or signature, depending on FPGA hardware +design. In this case, EAL or other bus driver doesn't have the capability to +identify proper driver for FPGA device. With prgdev introduced, while FPGA is +always a prgdev, FPGA can use prgdev as primary driver, to find proper function +driver. + +The bind/unbind actions are not trying to break the general initialization +procedure in DPDK. prgdev still follow current EAL initializatio model to probe +drivers for devices. See below examples for details. + +It's not true that bind/unbind is always necessary for programming. If device +has a capability to perform specific functions and program in parallel, then +apps needn't do bind/unbind actions. See 'dynamic program' section below. + +Authentication +-------------- + +Authentication and permissions for programming the device are outside the scope +of this API. It is dependent upon a different mechanism not to be supported by +DPDK. Application can utilize a different mechanism to authenticate user and +or 'program' as validated for loading. + +Misc +---- + +PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The +device can be loaded with a personality, and it would be up to this programmable +device to load the appropriate "personality driver" based upon an identification +mechanism to work with the loaded personality. + + +prgdev API +========== + +Below is the major APIs exposed to apps. + +- rte_prgdev_get_info + Acquire general info of the prgdev. +- rte_prgdev_open + Open a prgdev for later programming. If failed, needs to call below 'unbind' + function to detach other functions except prgdev first. +- rte_prgdev_img_download + Download a piece of buffer from host to the device after 'open' succeded. +- rte_prgdev_img_upload + Upload a piece of information from the device to host. +- rte_prgdev_check_status + Even download process is successful, it's necessary to check the status of the + downloaded image if worked in expected way by this function. Harware or driver + may not have the capability to check that status, so it's optional to driver. +- rte_prgdev_close + After upload/download actions, close the prgdev. +- rte_prgdev_bind + After 'close' action, app can call it to attach the 'new' personality with + proper drivers, then expose to app with new functions. prgdev driver will be + responsible for finding proper drivers to drive the functions. Desired driver + will be responsible for exposing the functions to apps. +- rte_prgdev_unbind + When app are using the device, they need to call this function to unbind the + misc functions except prgdev with drivers prior to call 'open' function. + +Initialization procedure +======================== +In this section, there 2 examples, FPGA and programmable NIC, used as examples +to describe the initialization procedure on how programmable devices are probed +, initialized and registered as different devices. In which, FPGA has 2 +different possible cases, blank and prior programmed FPGAs. Note that prgdev +depends and follow on EAL initialization process to probe drivers for prgdev. + +Blank FPGA +---------- +Below is the diagram on how Blank FPGA are probed and registered as a prgdev +device that expose to apps. + +When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor +ID and device ID, then perform HW initialization if both ID match. After then, +it registered in prgdev abstract layer as a new device with a set of helper +function ops. + +After that, apps will observe single device: prgdev device. Until now, the FPGA +device is a simply device without any functionalities from application's +perspective. + +In the following action, app will call 'prgdev bind' function with the device +ID of the FPGA device to bind the function within the FPGA to specific driver. +FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware +and find potential function. Since it's a blank without any prior programming, a +simply SUCCESS return. + + Abstraction FPGA driver Hardware + layer + + + + + | | | + +-----eal pci probe-----> | + | | | +------------+ + | | | | FPGA dev | + | +--HW initialization----> | | ++------------+ | | | |+----------+| +| prgdev dev | <--prgdev_ops_register--+ | || AFU id || ++------------+ | | | |+----------+| + | | | +------------+ + +-----prgdev bind-------> | + | | | + | +--Read and parse AFU---> + | | | + | <--No personality found-+ + | | | + <-------SUCCESS---------+ | + | | | + + + + + +prior programmed FPGA +--------------------- + +For a prior programmed FPGA, all execution flow is similar to blank FPGA case, +except the action of reading signature, as below flow table indicates. It will +read valid functions, crypto and ethdev, as an example, then find proper driver +to initialize the functions, then register as different ports. + +After initialization, app will observe 3 kinds of devices, a prgdev device, a +ethdev device and a cryptodev device, which all comes from a single PCIE device. + + + Abstraction FPGA driver Hardware + layer + + + + + | | | + +----eal pci probe------> | + | | | +-------------+ + | | | | FPGA dev | + | +--HW initialization----> | | ++------------+ | | | +-------------+ +| prgdev dev | <--prgdev_ops_register--+ | || AFU id || ++------------+ | | | |-------------| + | | | +-------------+ + +-----prgdev bind-------> | + | | | + | +--Read and parse AFU---> + | | | + | <--eth and crypto found-+ ++------------+ | | | +| ethdev port| <---ethdev attach-------+ | ++------------+ | | | + | | | ++------------+ | | | +| crypto port| <---crypto attach-------+ | ++------------+ | | | + | | | + + + + + + +Programmable NIC +---------------- + +Comparing FPGA, programmable NIC has a nature of being a ethdev device. The +initialization procedure is much more simpler than FPGA. Below is a flow table +to describe this. + +The 'eal pci probe' procedure is similar, it registered itself as a prgdev after +checking that the NIC is programmable. What is different from FPGA is it +registered itself as a ethdev device in the following action due to the NIC +nature. + +Then, app still follow the common process to call 'prgdev bind', while the NIC +driver does nothing and just return a SUCCESS. + + Abstraction NIC driver Hardware + layer + + + + + | | | + +----eal pci probe------> | + | | | +-------------+ + | +--HW initialization----> |Programmable | + | | | | NIC | ++------------+ | +---Capability check----> +-------------+ +| prgdev dev | <--prgdev_ops_register--+ | ++------------+ | | | + | | | ++------------+ | | | +| ethdev port| <---ethdev attach-------+ | ++------------+ | | | + | | | + +-----prgdev bind-------> | + | | | + | | | + <--------SUCCESS--------+ | + | | | + + + + + + +Program procedure +================= + +This section will describe how apps finish an image downloading procedure with +the prgdev API. Similar to previous section, it will use several FPGA and NIC +examples to show the common and different steps. + +Needs to point out, 'rte_prgdev_img_download' function didn't carry the info +of the image, or where downloading the image to. The interpretion info of the +image can be hidden into the buffer itself. + +Common programming steps +------------------------ + +Before reaching details, it needs to introduce the common steps for apps to +follow in order to download the image to the hardware. + +Before programming, apps need to call function 'rte_prgdev_open' to open the +programming interface. If failed, either functions residing in PCI-E device is +in use or the PCI-E device is not ready, the programming steps are different. + +Below is the steps that 'open' succeeds: +1. rte_prgdev_open --> SUCCESS +2. rte_prgdev_img_download +3. rte_prgdev_check_status +4. rte_prgdev_close +5. rte_prgdev_bind + +If failed to open the programming interface, apps need to follow below steps: +1. rte_prgdev_open --> FAILURE +2. Close all related functions +3. ret_prgdev_unbind +4. rte_prgdev_open --> SUCCESS +5. rte_prgdev_img_download +6. rte_prgdev_check_status +7. rte_prgdev_close +8. rte_prgdev_bind + +Detach functions +---------------- + +As discussed before, apps need to follow steps to detach all the other functions +except prgdev when meeting error for 'open' operation. Below is a FPGA example +with 'ethdev' and 'cryptodev' function attached. + +After meeting error, app needs to close ethdev port and cryptodev port first. +Then, use 'prgdev unbind' to detach all the functions that registered previously +. FPGA driver will call each 'detach' function for each registered device class +respectively. After then, the 'ethdev' and 'cryptodev' ports will not be +available. + + + Abstraction FPGA driver Hardware + layer + + + + ++------------+ | | | +| prgdev dev | +-------prgdev open-----> | +| | <-------ERR_IN_USE------+ | +-------------+ ++------------+ | | | | FPGA dev | + | | | | +---------+ | ++------------+ | | | | | AFU id | | +| ethdev port| +---rte_ethdev_close----> | | +---------+ | +| | +------ethdev close-----> +-------------+ ++------------+ <---------SUCCESS-------+ | + | | | ++------------+ | | | +| crypto port| +---crypto port close---> | +| | | +---crypto func close---> ++------------+ <---------SUCCESS-------+ | + | | | + +-----prgdev unbind-----> | + | | | ++ - - - - - -+ | | | + ethdev port <-----ethdev detach-----+ | ++- - - - - - + | | | + | | | ++ - - - - - -+ | | | + crypto port <----cryptodev detach---+ | ++- - - - - - + | | | + + + + + +Device programming +------------------ + +After 'open' prgdev sucessfully, apps follow 'image download', 'check status', +'close' sequence respectively. After then, they need to 'bind' new functions +back with 'prgdev bind' call. + +Still re-use previous FPGA example, below is the execution sequence. After image +downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are +registered and 2 new ports are presented to apps. + + + Abstraction FPGA driver Hardware + layer + + + + +-------------+ ++------------+ | | | | FPGA dev | +| | +-----prgdev open-------> | | +---------+ | +| | <-------SUCCESS---------+ | | | ethdev | | +| | | | | | +---------+ | +| | +--prgdev_img_download--> | | +---------+ | +| | | +----Download image-----> | | crypto | | +| | | <--------SUCCESS--------+ | +---------+ | +| | <---------SUCCESS-------+ | +-------------+ +| prgdev dev | | | | | +| | +---prgdev_chk_status---> | | +| | | +--Verify image status--> | +| | | <--------SUCCESS--------+ | +| | <--------SUCCESS--------+ | | +| | +-------prgdev close----> | v +| | | | | +-------------+ +| | +-------prgdev bind-----> | | FPGA dev | ++------------+ | | | | +---------+ | + | <--Virtio and flow dev--+ | | virtio | | ++------------+ | | | | +---------+ | +| ethdev port| <-----ethdev attach-----+ | | +---------+ | ++------------+ | | | | |Flow dev | | + | | | | +---------+ | ++------------+ | | | +-------------+ +| Flow port | <-----Flow dev attach---+ | ++------------+ | | | + + + + + + +Dyanamic program +---------------- + +If prgdev device has a capability to be programmed while functions are in use, +then we say it supports dynamic program. It's possible to update a profile while +device is used as a ethdev device. + +The device can claim this capability by set a DYNAMIC_PROGRAM flag in function +'rte_prgdev_get_info'. + +The dynamic programming procedure is the same as previous section shows, while +ignore the steps to 'close' and 'unbind' actions when device is in use. + +Note that prgdev driver needs to consider if it can support dynamic program to +avoid any hardware or software error. + + +Source code layout +================== + +Below is the suggested source code layout. +- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This + doesn't change programmable NIC's directory and remains the same. The PMD + driver should include 2 parts, one part is the code to support rte_ethdev API, + another part is the code to support rte_prgdev API. +- 'librte_prgdev' under 'lib' is the new directory store definition and + implementation of the generic prgdev API. +- 'prg' under 'drivers' is the directory to store specfic implementation + driver to support rte_prgdev. Each sub-directory could be a prgdev driver to + support specific hardware product. + + +---------+ +---------+ + | drivers | | lib | + +---------+ +----+----+ + | | + +----------+-----------+ | + | | | + +---v---+ +---v----+ +-------v-------+ + | net | | prg | | librte_prgdev | + +---+---+ +---+----+ +---------------+ + | | + +-----+----+ +-----+----+ + | | | | ++---v----+ +---v----+ +---v----+ +---v----+ +| i40e | | prg NIC| |FPGA ABC| |GPU XYZ | ++--------+ +--------+ +--------+ +--------+ + +