diff mbox

[dpdk-dev,v4,08/11] eal/pci: Cleanup pci driver initialization code

Message ID 1421664027-17971-9-git-send-email-mukawa@igel.co.jp (mailing list archive)
State Superseded, archived
Headers

Commit Message

Tetsuya Mukawa Jan. 19, 2015, 10:40 a.m. UTC 
  - Add rte_eal_pci_close_one_dirver()
  The function is used for closing the specified driver and device.
- Add pci_invoke_all_drivers()
  The function is based on pci_probe_all_drivers. But it can not only
  probe but also close drivers.
- Add pci_close_all_drivers()
  The function tries to find a driver for the specified device, and
  then close the driver.
- Add rte_eal_pci_probe_one() and rte_eal_pci_close_one()
  The functions are used for probe and close a device.
  First the function tries to find a device that has the specfied
  PCI address. Then, probe or close the device.

v4:
- Fix paramerter checking.
- Fix indent of 'if' statement.

Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
---
 lib/librte_eal/common/eal_common_pci.c  | 90 +++++++++++++++++++++++++++++----
 lib/librte_eal/common/eal_private.h     | 25 +++++++++
 lib/librte_eal/common/include/rte_pci.h | 33 ++++++++++++
 lib/librte_eal/linuxapp/eal/eal_pci.c   | 69 +++++++++++++++++++++++++
 4 files changed, 207 insertions(+), 10 deletions(-)

Comments

Tetsuya Mukawa Jan. 30, 2015, 5:42 a.m. UTC | #1 
This patch series adds a dynamic port hotplug framework to DPDK.
With the patches, DPDK apps can attach or detach ports at runtime.

The basic concept of the port hotplug is like followings.
- DPDK apps must have responsibility to manage ports.
  DPDK apps only know which ports are attached or detached at the moment.
  The port hotplug framework is implemented to allow DPDK apps to manage ports.
  For example, when DPDK apps call port attach function, attached port number
  will be returned. Also DPDK apps can detach port by port number.
- Kernel support is needed for attaching or detaching physical device ports.
  To attach a new physical device port, the device will be recognized by
  userspace directly I/O framework in kernel at first. Then DPDK apps can
  call the port hotplug functions to attach ports.
  For detaching, steps are vice versa.
- Before detach ports, ports must be stopped and closed.
  DPDK application must call rte_eth_dev_stop() and rte_eth_dev_close() before
  detaching ports. These function will call finalization codes of PMDs.
  But so far, no PMD frees all resources allocated by initialization.
  It means PMDs are needed to be fixed to support the port hotplug.
  'RTE_PCI_DRV_DETACHABLE' is a new flag indicating a PMD supports detaching.
  Without this flag, detaching will be failed.
- Mustn't affect legacy DPDK apps.
  No DPDK EAL behavior is changed, if the port hotplug functions are't called.
  So all legacy DPDK apps can still work without modifications.

And a few limitations.
- The port hotplug functions are not thread safe.
  DPDK apps should handle it.
- Only support Linux and igb_uio so far.
  BSD and VFIO is not supported. I will send VFIO patches at least, but I don't
  have a plan to submit BSD patch so far.


Here is port hotplug APIs.
-------------------------------------------------------------------------------
/**
 * Attach a new device.
 *
 * @param devargs
 *   A pointer to a strings array describing the new device
 *   to be attached. The strings should be a pci address like
 *   '0000:01:00.0' or virtual device name like 'eth_pcap0'.
 * @param port_id
 *  A pointer to a port identifier actually attached.
 * @return
 *  0 on success and port_id is filled, negative on error
 */
int rte_eal_dev_attach(const char *devargs, uint8_t *port_id);

/**
 * Detach a device.
 *
 * @param port_id
 *   The port identifier of the device to detach.
 * @param addr
 *  A pointer to a device name actually detached.
 * @return
 *  0 on success and devname is filled, negative on error
 */
int rte_eal_dev_detach(uint8_t port_id, char *devname);
-------------------------------------------------------------------------------

This patch series are for DPDK EAL. To use port hotplug function by DPDK apps,
each PMD should be fixed to support 'RTE_PCI_DRV_DETACHABLE' flag. Please check
a patch for pcap PMD.

Also please check testpmd patch. It will show you how to fix your legacy
applications to support port hotplug feature.

PATCH v5 changes
 - Add runtime check passthrough driver type, like vfio-pci, igb_uio
   and uio_pci_generic.
   This was done by Qiu, Michael. Thanks a lot.
 - Change function names like below.
   - rte_eal_dev_find_and_invoke() to rte_eal_vdev_find_and_invoke().
   - rte_eal_dev_invoke() to rte_eal_vdev_invoke().
 - Add code to handle a return value of rte_eal_devargs_remove().
 - Fix pci address format in rte_eal_dev_detach().
 - Remove RTE_EAL_INVOKE_TYPE_UNKNOWN, because it's unused.
 - Change function definition of rte_eal_devargs_remove().
 - Fix pci_unmap_device() to check pt_driver.
 - Fix return value of below functions.
   - rte_eth_dev_get_changed_port().
   - rte_eth_dev_get_port_by_addr().
 - Change paramters of rte_eth_dev_validate_port() to cleanup code.
 - Fix pci_scan_one to handle pt_driver correctly.
   (Thanks to Qiu, Michael for above sugesstions)

PATCH v4 changes
 - Merge patches to review easier.
 - Fix indent of 'if' statement.
 - Fix calculation method of eal_compare_pci_addr().
 - Fix header file declaration.
 - Add header file to determine if hotplug can be enabled.
   (Thanks to Qiu, Michael)
 - Use braces with 'for' loop.
 - Add paramerter checking.
 - Fix sanity check code
 - Fix comments of rte_eth_dev_type.
 - Change function names.
   (Thanks to Iremonger, Bernard)

PATCH v3 changes:
 - Fix enum definition used in rte_ethdev.c.
   (Thanks to Zhang, Helin)

PATCH v2 changes:
 - Replace rte_eal_dev_attach_pdev(), rte_eal_dev_detach_pdev,
   rte_eal_dev_attach_vdev() and rte_eal_dev_detach_vdev() to
   rte_eal_dev_attach() and rte_eal_dev_detach().
 - Add parameter values checking.
 - Refashion a few functions.
   (Thanks to Iremonger, Bernard)

PATCH v1 Changes:
 - Fix error checking code of librte_eth APIs.
 - Fix issue that port from pcap PMD cannot be detached correctly.
 - Fix issue that testpmd could hang after forwarding, if attaching and detaching
   is repeatedly.
 - Fix if-condition of rte_eth_dev_get_port_by_addr().
   (Thanks to Mark Enright)

RFC PATCH v2 Changes:
- remove 'rte_eth_dev_validate_port()', and cleanup codes.


Michael Qiu (2):
  eal_pci: Add flag to hold kernel driver type
  eal_pci: pci memory map work with driver type

Tetsuya Mukawa (11):
  eal/pci,ethdev: Remove assumption that port will not be detached
  eal/pci: Consolidate pci address comparison APIs
  ethdev: Add rte_eth_dev_free to free specified device
  eal,ethdev: Add a function and function pointers to close ether device
  ethdev: Add functions that will be used by port hotplug functions
  eal/linux/pci: Add functions for unmapping igb_uio resources
  eal/pci: Add a function to remove the entry of devargs list
  eal/pci: Cleanup pci driver initialization code
  ethdev: Add one dev_type paramerter to rte_eth_dev_allocate
  eal/pci: Add rte_eal_dev_attach/detach() functions
  eal: Enable port hotplug framework in Linux

 app/test/virtual_pmd.c                          |   2 +-
 config/common_linuxapp                          |   5 +
 lib/librte_eal/bsdapp/eal/eal_pci.c             |  25 +-
 lib/librte_eal/common/Makefile                  |   1 +
 lib/librte_eal/common/eal_common_dev.c          | 274 +++++++++++
 lib/librte_eal/common/eal_common_devargs.c      |  60 +++
 lib/librte_eal/common/eal_common_pci.c          |  92 +++-
 lib/librte_eal/common/eal_private.h             |  35 ++
 lib/librte_eal/common/include/rte_dev.h         |  33 ++
 lib/librte_eal/common/include/rte_dev_hotplug.h |  44 ++
 lib/librte_eal/common/include/rte_devargs.h     |  21 +
 lib/librte_eal/common/include/rte_pci.h         |  84 ++++
 lib/librte_eal/linuxapp/eal/Makefile            |   1 +
 lib/librte_eal/linuxapp/eal/eal_pci.c           | 227 +++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_init.h      |   8 +
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c       |  67 ++-
 lib/librte_ether/rte_ethdev.c                   | 626 +++++++++++++-----------
 lib/librte_ether/rte_ethdev.h                   | 145 +++++-
 lib/librte_pmd_af_packet/rte_eth_af_packet.c    |   2 +-
 lib/librte_pmd_bond/rte_eth_bond_api.c          |   2 +-
 lib/librte_pmd_pcap/rte_eth_pcap.c              |   2 +-
 lib/librte_pmd_ring/rte_eth_ring.c              |   2 +-
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.c        |   2 +-
 23 files changed, 1413 insertions(+), 347 deletions(-)
 create mode 100644 lib/librte_eal/common/include/rte_dev_hotplug.h
Tetsuya Mukawa Feb. 1, 2015, 4:01 a.m. UTC | #2 
This patch series adds a dynamic port hotplug framework to DPDK.
With the patches, DPDK apps can attach or detach ports at runtime.

The basic concept of the port hotplug is like followings.
- DPDK apps must have responsibility to manage ports.
  DPDK apps only know which ports are attached or detached at the moment.
  The port hotplug framework is implemented to allow DPDK apps to manage ports.
  For example, when DPDK apps call port attach function, attached port number
  will be returned. Also DPDK apps can detach port by port number.
- Kernel support is needed for attaching or detaching physical device ports.
  To attach a new physical device port, the device will be recognized by
  userspace directly I/O framework in kernel at first. Then DPDK apps can
  call the port hotplug functions to attach ports.
  For detaching, steps are vice versa.
- Before detach ports, ports must be stopped and closed.
  DPDK application must call rte_eth_dev_stop() and rte_eth_dev_close() before
  detaching ports. These function will call finalization codes of PMDs.
  But so far, no PMD frees all resources allocated by initialization.
  It means PMDs are needed to be fixed to support the port hotplug.
  'RTE_PCI_DRV_DETACHABLE' is a new flag indicating a PMD supports detaching.
  Without this flag, detaching will be failed.
- Mustn't affect legacy DPDK apps.
  No DPDK EAL behavior is changed, if the port hotplug functions are't called.
  So all legacy DPDK apps can still work without modifications.

And a few limitations.
- The port hotplug functions are not thread safe.
  DPDK apps should handle it.
- Only support Linux and igb_uio so far.
  BSD and VFIO is not supported. I will send VFIO patches at least, but I don't
  have a plan to submit BSD patch so far.


Here is port hotplug APIs.
-------------------------------------------------------------------------------
/**
 * Attach a new device.
 *
 * @param devargs
 *   A pointer to a strings array describing the new device
 *   to be attached. The strings should be a pci address like
 *   '0000:01:00.0' or virtual device name like 'eth_pcap0'.
 * @param port_id
 *  A pointer to a port identifier actually attached.
 * @return
 *  0 on success and port_id is filled, negative on error
 */
int rte_eal_dev_attach(const char *devargs, uint8_t *port_id);

/**
 * Detach a device.
 *
 * @param port_id
 *   The port identifier of the device to detach.
 * @param addr
 *  A pointer to a device name actually detached.
 * @return
 *  0 on success and devname is filled, negative on error
 */
int rte_eal_dev_detach(uint8_t port_id, char *devname);
-------------------------------------------------------------------------------

This patch series are for DPDK EAL. To use port hotplug function by DPDK apps,
each PMD should be fixed to support 'RTE_PCI_DRV_DETACHABLE' flag. Please check
a patch for pcap PMD.

Also please check testpmd patch. It will show you how to fix your legacy
applications to support port hotplug feature.

PATCH v6 changes
 - Fix rte_eth_dev_uninit() to handle a return value of uninit
   function of PMD. To do this, below changes also be applied.
   - Fix a paramter of rte_eth_dev_free().
   - Use rte_eth_dev structure as the paramter of rte_eth_dev_free().

PATCH v5 changes
 - Add runtime check passthrough driver type, like vfio-pci, igb_uio
   and uio_pci_generic.
   This was done by Qiu, Michael. Thanks a lot.
 - Change function names like below.
   - rte_eal_dev_find_and_invoke() to rte_eal_vdev_find_and_invoke().
   - rte_eal_dev_invoke() to rte_eal_vdev_invoke().
 - Add code to handle a return value of rte_eal_devargs_remove().
 - Fix pci address format in rte_eal_dev_detach().
 - Remove RTE_EAL_INVOKE_TYPE_UNKNOWN, because it's unused.
 - Change function definition of rte_eal_devargs_remove().
 - Fix pci_unmap_device() to check pt_driver.
 - Fix return value of below functions.
   - rte_eth_dev_get_changed_port().
   - rte_eth_dev_get_port_by_addr().
 - Change paramters of rte_eth_dev_validate_port() to cleanup code.
 - Fix pci_scan_one to handle pt_driver correctly.
   (Thanks to Qiu, Michael for above sugesstions)

PATCH v4 changes
 - Merge patches to review easier.
 - Fix indent of 'if' statement.
 - Fix calculation method of eal_compare_pci_addr().
 - Fix header file declaration.
 - Add header file to determine if hotplug can be enabled.
   (Thanks to Qiu, Michael)
 - Use braces with 'for' loop.
 - Add paramerter checking.
 - Fix sanity check code
 - Fix comments of rte_eth_dev_type.
 - Change function names.
   (Thanks to Iremonger, Bernard)

PATCH v3 changes:
 - Fix enum definition used in rte_ethdev.c.
   (Thanks to Zhang, Helin)

PATCH v2 changes:
 - Replace rte_eal_dev_attach_pdev(), rte_eal_dev_detach_pdev,
   rte_eal_dev_attach_vdev() and rte_eal_dev_detach_vdev() to
   rte_eal_dev_attach() and rte_eal_dev_detach().
 - Add parameter values checking.
 - Refashion a few functions.
   (Thanks to Iremonger, Bernard)

PATCH v1 Changes:
 - Fix error checking code of librte_eth APIs.
 - Fix issue that port from pcap PMD cannot be detached correctly.
 - Fix issue that testpmd could hang after forwarding, if attaching and detaching
   is repeatedly.
 - Fix if-condition of rte_eth_dev_get_port_by_addr().
   (Thanks to Mark Enright)

RFC PATCH v2 Changes:
- remove 'rte_eth_dev_validate_port()', and cleanup codes.


Michael Qiu (2):
  eal_pci: Add flag to hold kernel driver type
  eal_pci: pci memory map work with driver type

Tetsuya Mukawa (11):
  eal/pci,ethdev: Remove assumption that port will not be detached
  eal/pci: Consolidate pci address comparison APIs
  ethdev: Add rte_eth_dev_free to free specified device
  eal,ethdev: Add a function and function pointers to close ether device
  ethdev: Add functions that will be used by port hotplug functions
  eal/linux/pci: Add functions for unmapping igb_uio resources
  eal/pci: Add a function to remove the entry of devargs list
  eal/pci: Cleanup pci driver initialization code
  ethdev: Add one dev_type paramerter to rte_eth_dev_allocate
  eal/pci: Add rte_eal_dev_attach/detach() functions
  eal: Enable port hotplug framework in Linux

 app/test/virtual_pmd.c                          |   2 +-
 config/common_linuxapp                          |   5 +
 lib/librte_eal/bsdapp/eal/eal_pci.c             |  25 +-
 lib/librte_eal/common/Makefile                  |   1 +
 lib/librte_eal/common/eal_common_dev.c          | 274 +++++++++++
 lib/librte_eal/common/eal_common_devargs.c      |  60 +++
 lib/librte_eal/common/eal_common_pci.c          |  92 +++-
 lib/librte_eal/common/eal_private.h             |  35 ++
 lib/librte_eal/common/include/rte_dev.h         |  33 ++
 lib/librte_eal/common/include/rte_dev_hotplug.h |  44 ++
 lib/librte_eal/common/include/rte_devargs.h     |  21 +
 lib/librte_eal/common/include/rte_pci.h         |  84 ++++
 lib/librte_eal/linuxapp/eal/Makefile            |   1 +
 lib/librte_eal/linuxapp/eal/eal_pci.c           | 227 +++++++--
 lib/librte_eal/linuxapp/eal/eal_pci_init.h      |   8 +
 lib/librte_eal/linuxapp/eal/eal_pci_uio.c       |  67 ++-
 lib/librte_ether/rte_ethdev.c                   | 624 +++++++++++++-----------
 lib/librte_ether/rte_ethdev.h                   | 148 +++++-
 lib/librte_pmd_af_packet/rte_eth_af_packet.c    |   2 +-
 lib/librte_pmd_bond/rte_eth_bond_api.c          |   2 +-
 lib/librte_pmd_pcap/rte_eth_pcap.c              |   2 +-
 lib/librte_pmd_ring/rte_eth_ring.c              |   2 +-
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.c        |   2 +-
 23 files changed, 1414 insertions(+), 347 deletions(-)
 create mode 100644 lib/librte_eal/common/include/rte_dev_hotplug.h
Iremonger, Bernard Feb. 3, 2015, 1:03 p.m. UTC | #3 
> -----Original Message-----
> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
> Sent: Sunday, February 1, 2015 4:02 AM
> To: dev@dpdk.org
> Cc: Qiu, Michael; Iremonger, Bernard; Tetsuya Mukawa
> Subject: [PATCH v6 00/13] Port Hotplug Framework
> 
> This patch series adds a dynamic port hotplug framework to DPDK.
> With the patches, DPDK apps can attach or detach ports at runtime.
> 
> The basic concept of the port hotplug is like followings.
> - DPDK apps must have responsibility to manage ports.
>   DPDK apps only know which ports are attached or detached at the moment.
>   The port hotplug framework is implemented to allow DPDK apps to manage ports.
>   For example, when DPDK apps call port attach function, attached port number
>   will be returned. Also DPDK apps can detach port by port number.
> - Kernel support is needed for attaching or detaching physical device ports.
>   To attach a new physical device port, the device will be recognized by
>   userspace directly I/O framework in kernel at first. Then DPDK apps can
>   call the port hotplug functions to attach ports.
>   For detaching, steps are vice versa.
> - Before detach ports, ports must be stopped and closed.
>   DPDK application must call rte_eth_dev_stop() and rte_eth_dev_close() before
>   detaching ports. These function will call finalization codes of PMDs.
>   But so far, no PMD frees all resources allocated by initialization.
>   It means PMDs are needed to be fixed to support the port hotplug.
>   'RTE_PCI_DRV_DETACHABLE' is a new flag indicating a PMD supports detaching.
>   Without this flag, detaching will be failed.
> - Mustn't affect legacy DPDK apps.
>   No DPDK EAL behavior is changed, if the port hotplug functions are't called.
>   So all legacy DPDK apps can still work without modifications.
> 
> And a few limitations.
> - The port hotplug functions are not thread safe.
>   DPDK apps should handle it.
> - Only support Linux and igb_uio so far.
>   BSD and VFIO is not supported. I will send VFIO patches at least, but I don't
>   have a plan to submit BSD patch so far.
> 
> 
> Here is port hotplug APIs.
> -------------------------------------------------------------------------------
> /**
>  * Attach a new device.
>  *
>  * @param devargs
>  *   A pointer to a strings array describing the new device
>  *   to be attached. The strings should be a pci address like
>  *   '0000:01:00.0' or virtual device name like 'eth_pcap0'.
>  * @param port_id
>  *  A pointer to a port identifier actually attached.
>  * @return
>  *  0 on success and port_id is filled, negative on error  */ int rte_eal_dev_attach(const char *devargs,
> uint8_t *port_id);
> 
> /**
>  * Detach a device.
>  *
>  * @param port_id
>  *   The port identifier of the device to detach.
>  * @param addr
>  *  A pointer to a device name actually detached.
>  * @return
>  *  0 on success and devname is filled, negative on error  */ int rte_eal_dev_detach(uint8_t port_id,
> char *devname);
> -------------------------------------------------------------------------------
> 
> This patch series are for DPDK EAL. To use port hotplug function by DPDK apps, each PMD should be
> fixed to support 'RTE_PCI_DRV_DETACHABLE' flag. Please check a patch for pcap PMD.
> 
> Also please check testpmd patch. It will show you how to fix your legacy applications to support port
> hotplug feature.
> 
> PATCH v6 changes
>  - Fix rte_eth_dev_uninit() to handle a return value of uninit
>    function of PMD. To do this, below changes also be applied.
>    - Fix a paramter of rte_eth_dev_free().
>    - Use rte_eth_dev structure as the paramter of rte_eth_dev_free().
> 
> PATCH v5 changes
>  - Add runtime check passthrough driver type, like vfio-pci, igb_uio
>    and uio_pci_generic.
>    This was done by Qiu, Michael. Thanks a lot.
>  - Change function names like below.
>    - rte_eal_dev_find_and_invoke() to rte_eal_vdev_find_and_invoke().
>    - rte_eal_dev_invoke() to rte_eal_vdev_invoke().
>  - Add code to handle a return value of rte_eal_devargs_remove().
>  - Fix pci address format in rte_eal_dev_detach().
>  - Remove RTE_EAL_INVOKE_TYPE_UNKNOWN, because it's unused.
>  - Change function definition of rte_eal_devargs_remove().
>  - Fix pci_unmap_device() to check pt_driver.
>  - Fix return value of below functions.
>    - rte_eth_dev_get_changed_port().
>    - rte_eth_dev_get_port_by_addr().
>  - Change paramters of rte_eth_dev_validate_port() to cleanup code.
>  - Fix pci_scan_one to handle pt_driver correctly.
>    (Thanks to Qiu, Michael for above sugesstions)
> 
> PATCH v4 changes
>  - Merge patches to review easier.
>  - Fix indent of 'if' statement.
>  - Fix calculation method of eal_compare_pci_addr().
>  - Fix header file declaration.
>  - Add header file to determine if hotplug can be enabled.
>    (Thanks to Qiu, Michael)
>  - Use braces with 'for' loop.
>  - Add paramerter checking.
>  - Fix sanity check code
>  - Fix comments of rte_eth_dev_type.
>  - Change function names.
>    (Thanks to Iremonger, Bernard)
> 
> PATCH v3 changes:
>  - Fix enum definition used in rte_ethdev.c.
>    (Thanks to Zhang, Helin)
> 
> PATCH v2 changes:
>  - Replace rte_eal_dev_attach_pdev(), rte_eal_dev_detach_pdev,
>    rte_eal_dev_attach_vdev() and rte_eal_dev_detach_vdev() to
>    rte_eal_dev_attach() and rte_eal_dev_detach().
>  - Add parameter values checking.
>  - Refashion a few functions.
>    (Thanks to Iremonger, Bernard)
> 
> PATCH v1 Changes:
>  - Fix error checking code of librte_eth APIs.
>  - Fix issue that port from pcap PMD cannot be detached correctly.
>  - Fix issue that testpmd could hang after forwarding, if attaching and detaching
>    is repeatedly.
>  - Fix if-condition of rte_eth_dev_get_port_by_addr().
>    (Thanks to Mark Enright)
> 
> RFC PATCH v2 Changes:
> - remove 'rte_eth_dev_validate_port()', and cleanup codes.
> 
> 
> Michael Qiu (2):
>   eal_pci: Add flag to hold kernel driver type
>   eal_pci: pci memory map work with driver type
> 
> Tetsuya Mukawa (11):
>   eal/pci,ethdev: Remove assumption that port will not be detached
>   eal/pci: Consolidate pci address comparison APIs
>   ethdev: Add rte_eth_dev_free to free specified device
>   eal,ethdev: Add a function and function pointers to close ether device
>   ethdev: Add functions that will be used by port hotplug functions
>   eal/linux/pci: Add functions for unmapping igb_uio resources
>   eal/pci: Add a function to remove the entry of devargs list
>   eal/pci: Cleanup pci driver initialization code
>   ethdev: Add one dev_type paramerter to rte_eth_dev_allocate
>   eal/pci: Add rte_eal_dev_attach/detach() functions
>   eal: Enable port hotplug framework in Linux
> 
>  app/test/virtual_pmd.c                          |   2 +-
>  config/common_linuxapp                          |   5 +
>  lib/librte_eal/bsdapp/eal/eal_pci.c             |  25 +-
>  lib/librte_eal/common/Makefile                  |   1 +
>  lib/librte_eal/common/eal_common_dev.c          | 274 +++++++++++
>  lib/librte_eal/common/eal_common_devargs.c      |  60 +++
>  lib/librte_eal/common/eal_common_pci.c          |  92 +++-
>  lib/librte_eal/common/eal_private.h             |  35 ++
>  lib/librte_eal/common/include/rte_dev.h         |  33 ++
>  lib/librte_eal/common/include/rte_dev_hotplug.h |  44 ++
>  lib/librte_eal/common/include/rte_devargs.h     |  21 +
>  lib/librte_eal/common/include/rte_pci.h         |  84 ++++
>  lib/librte_eal/linuxapp/eal/Makefile            |   1 +
>  lib/librte_eal/linuxapp/eal/eal_pci.c           | 227 +++++++--
>  lib/librte_eal/linuxapp/eal/eal_pci_init.h      |   8 +
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c       |  67 ++-
>  lib/librte_ether/rte_ethdev.c                   | 624 +++++++++++++-----------
>  lib/librte_ether/rte_ethdev.h                   | 148 +++++-
>  lib/librte_pmd_af_packet/rte_eth_af_packet.c    |   2 +-
>  lib/librte_pmd_bond/rte_eth_bond_api.c          |   2 +-
>  lib/librte_pmd_pcap/rte_eth_pcap.c              |   2 +-
>  lib/librte_pmd_ring/rte_eth_ring.c              |   2 +-
>  lib/librte_pmd_xenvirt/rte_eth_xenvirt.c        |   2 +-
>  23 files changed, 1414 insertions(+), 347 deletions(-)  create mode 100644
> lib/librte_eal/common/include/rte_dev_hotplug.h
> 
> --
> 1.9.1
Hi Tetsuya,

I have a few comments as follows:

Lib/librte_ether/rte_ethdev.c
Line 400 :   NO_TRACE would be clearer than NONE_TRACE.
Line 440:    function rte_eth_dev_save()  should pass in a size parameter which should be used in the memcpy function.

Lib/librte_eal/common/include/ret_dev_hotplug.h
 Is this file really necessary as it contains only one macro?
Could it be merged into Lib/librte_eal/common/include/ret_dev.h ?

Regards,

Bernard.
Iremonger, Bernard Feb. 3, 2015, 6:35 p.m. UTC | #4 
> -----Original Message-----
> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
> Sent: Sunday, February 1, 2015 4:02 AM
> To: dev@dpdk.org
> Cc: Qiu, Michael; Iremonger, Bernard; Tetsuya Mukawa
> Subject: [PATCH v6 00/13] Port Hotplug Framework
> 
> This patch series adds a dynamic port hotplug framework to DPDK.
> With the patches, DPDK apps can attach or detach ports at runtime.
> 
> The basic concept of the port hotplug is like followings.
> - DPDK apps must have responsibility to manage ports.
>   DPDK apps only know which ports are attached or detached at the moment.
>   The port hotplug framework is implemented to allow DPDK apps to manage ports.
>   For example, when DPDK apps call port attach function, attached port number
>   will be returned. Also DPDK apps can detach port by port number.
> - Kernel support is needed for attaching or detaching physical device ports.
>   To attach a new physical device port, the device will be recognized by
>   userspace directly I/O framework in kernel at first. Then DPDK apps can
>   call the port hotplug functions to attach ports.
>   For detaching, steps are vice versa.
> - Before detach ports, ports must be stopped and closed.
>   DPDK application must call rte_eth_dev_stop() and rte_eth_dev_close() before
>   detaching ports. These function will call finalization codes of PMDs.
>   But so far, no PMD frees all resources allocated by initialization.
>   It means PMDs are needed to be fixed to support the port hotplug.
>   'RTE_PCI_DRV_DETACHABLE' is a new flag indicating a PMD supports detaching.
>   Without this flag, detaching will be failed.
> - Mustn't affect legacy DPDK apps.
>   No DPDK EAL behavior is changed, if the port hotplug functions are't called.
>   So all legacy DPDK apps can still work without modifications.
> 
> And a few limitations.
> - The port hotplug functions are not thread safe.
>   DPDK apps should handle it.
> - Only support Linux and igb_uio so far.
>   BSD and VFIO is not supported. I will send VFIO patches at least, but I don't
>   have a plan to submit BSD patch so far.
> 
> 
> Here is port hotplug APIs.
> -------------------------------------------------------------------------------
> /**
>  * Attach a new device.
>  *
>  * @param devargs
>  *   A pointer to a strings array describing the new device
>  *   to be attached. The strings should be a pci address like
>  *   '0000:01:00.0' or virtual device name like 'eth_pcap0'.
>  * @param port_id
>  *  A pointer to a port identifier actually attached.
>  * @return
>  *  0 on success and port_id is filled, negative on error  */ int rte_eal_dev_attach(const char *devargs,
> uint8_t *port_id);
> 
> /**
>  * Detach a device.
>  *
>  * @param port_id
>  *   The port identifier of the device to detach.
>  * @param addr
>  *  A pointer to a device name actually detached.
>  * @return
>  *  0 on success and devname is filled, negative on error  */ int rte_eal_dev_detach(uint8_t port_id,
> char *devname);
> -------------------------------------------------------------------------------
> 
> This patch series are for DPDK EAL. To use port hotplug function by DPDK apps, each PMD should be
> fixed to support 'RTE_PCI_DRV_DETACHABLE' flag. Please check a patch for pcap PMD.
> 
> Also please check testpmd patch. It will show you how to fix your legacy applications to support port
> hotplug feature.
> 
> PATCH v6 changes
>  - Fix rte_eth_dev_uninit() to handle a return value of uninit
>    function of PMD. To do this, below changes also be applied.
>    - Fix a paramter of rte_eth_dev_free().
>    - Use rte_eth_dev structure as the paramter of rte_eth_dev_free().
> 
> PATCH v5 changes
>  - Add runtime check passthrough driver type, like vfio-pci, igb_uio
>    and uio_pci_generic.
>    This was done by Qiu, Michael. Thanks a lot.
>  - Change function names like below.
>    - rte_eal_dev_find_and_invoke() to rte_eal_vdev_find_and_invoke().
>    - rte_eal_dev_invoke() to rte_eal_vdev_invoke().
>  - Add code to handle a return value of rte_eal_devargs_remove().
>  - Fix pci address format in rte_eal_dev_detach().
>  - Remove RTE_EAL_INVOKE_TYPE_UNKNOWN, because it's unused.
>  - Change function definition of rte_eal_devargs_remove().
>  - Fix pci_unmap_device() to check pt_driver.
>  - Fix return value of below functions.
>    - rte_eth_dev_get_changed_port().
>    - rte_eth_dev_get_port_by_addr().
>  - Change paramters of rte_eth_dev_validate_port() to cleanup code.
>  - Fix pci_scan_one to handle pt_driver correctly.
>    (Thanks to Qiu, Michael for above sugesstions)
> 
> PATCH v4 changes
>  - Merge patches to review easier.
>  - Fix indent of 'if' statement.
>  - Fix calculation method of eal_compare_pci_addr().
>  - Fix header file declaration.
>  - Add header file to determine if hotplug can be enabled.
>    (Thanks to Qiu, Michael)
>  - Use braces with 'for' loop.
>  - Add paramerter checking.
>  - Fix sanity check code
>  - Fix comments of rte_eth_dev_type.
>  - Change function names.
>    (Thanks to Iremonger, Bernard)
> 
> PATCH v3 changes:
>  - Fix enum definition used in rte_ethdev.c.
>    (Thanks to Zhang, Helin)
> 
> PATCH v2 changes:
>  - Replace rte_eal_dev_attach_pdev(), rte_eal_dev_detach_pdev,
>    rte_eal_dev_attach_vdev() and rte_eal_dev_detach_vdev() to
>    rte_eal_dev_attach() and rte_eal_dev_detach().
>  - Add parameter values checking.
>  - Refashion a few functions.
>    (Thanks to Iremonger, Bernard)
> 
> PATCH v1 Changes:
>  - Fix error checking code of librte_eth APIs.
>  - Fix issue that port from pcap PMD cannot be detached correctly.
>  - Fix issue that testpmd could hang after forwarding, if attaching and detaching
>    is repeatedly.
>  - Fix if-condition of rte_eth_dev_get_port_by_addr().
>    (Thanks to Mark Enright)
> 
> RFC PATCH v2 Changes:
> - remove 'rte_eth_dev_validate_port()', and cleanup codes.
> 
> 
> Michael Qiu (2):
>   eal_pci: Add flag to hold kernel driver type
>   eal_pci: pci memory map work with driver type
> 
> Tetsuya Mukawa (11):
>   eal/pci,ethdev: Remove assumption that port will not be detached
>   eal/pci: Consolidate pci address comparison APIs
>   ethdev: Add rte_eth_dev_free to free specified device
>   eal,ethdev: Add a function and function pointers to close ether device
>   ethdev: Add functions that will be used by port hotplug functions
>   eal/linux/pci: Add functions for unmapping igb_uio resources
>   eal/pci: Add a function to remove the entry of devargs list
>   eal/pci: Cleanup pci driver initialization code
>   ethdev: Add one dev_type paramerter to rte_eth_dev_allocate
>   eal/pci: Add rte_eal_dev_attach/detach() functions
>   eal: Enable port hotplug framework in Linux
> 
>  app/test/virtual_pmd.c                          |   2 +-
>  config/common_linuxapp                          |   5 +
>  lib/librte_eal/bsdapp/eal/eal_pci.c             |  25 +-
>  lib/librte_eal/common/Makefile                  |   1 +
>  lib/librte_eal/common/eal_common_dev.c          | 274 +++++++++++
>  lib/librte_eal/common/eal_common_devargs.c      |  60 +++
>  lib/librte_eal/common/eal_common_pci.c          |  92 +++-
>  lib/librte_eal/common/eal_private.h             |  35 ++
>  lib/librte_eal/common/include/rte_dev.h         |  33 ++
>  lib/librte_eal/common/include/rte_dev_hotplug.h |  44 ++
>  lib/librte_eal/common/include/rte_devargs.h     |  21 +
>  lib/librte_eal/common/include/rte_pci.h         |  84 ++++
>  lib/librte_eal/linuxapp/eal/Makefile            |   1 +
>  lib/librte_eal/linuxapp/eal/eal_pci.c           | 227 +++++++--
>  lib/librte_eal/linuxapp/eal/eal_pci_init.h      |   8 +
>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c       |  67 ++-
>  lib/librte_ether/rte_ethdev.c                   | 624 +++++++++++++-----------
>  lib/librte_ether/rte_ethdev.h                   | 148 +++++-
>  lib/librte_pmd_af_packet/rte_eth_af_packet.c    |   2 +-
>  lib/librte_pmd_bond/rte_eth_bond_api.c          |   2 +-
>  lib/librte_pmd_pcap/rte_eth_pcap.c              |   2 +-
>  lib/librte_pmd_ring/rte_eth_ring.c              |   2 +-
>  lib/librte_pmd_xenvirt/rte_eth_xenvirt.c        |   2 +-
>  23 files changed, 1414 insertions(+), 347 deletions(-)  create mode 100644
> lib/librte_eal/common/include/rte_dev_hotplug.h
> 
> --
> 1.9.1
Hi Tetsuya,

In lib/librte_eal/linuxapp/eal/eal_pci.c
Line 183:   should "mapped" be "unmapped" ?

Regards,

Bernard.
Tetsuya Mukawa Feb. 5, 2015, 1:32 a.m. UTC | #5 
On 2015/02/03 22:03, Iremonger, Bernard wrote:
>> -----Original Message-----
>> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
>> Sent: Sunday, February 1, 2015 4:02 AM
>> To: dev@dpdk.org
>> Cc: Qiu, Michael; Iremonger, Bernard; Tetsuya Mukawa
>> Subject: [PATCH v6 00/13] Port Hotplug Framework
>>
>> This patch series adds a dynamic port hotplug framework to DPDK.
>> With the patches, DPDK apps can attach or detach ports at runtime.
>>
>> The basic concept of the port hotplug is like followings.
>> - DPDK apps must have responsibility to manage ports.
>>   DPDK apps only know which ports are attached or detached at the moment.
>>   The port hotplug framework is implemented to allow DPDK apps to manage ports.
>>   For example, when DPDK apps call port attach function, attached port number
>>   will be returned. Also DPDK apps can detach port by port number.
>> - Kernel support is needed for attaching or detaching physical device ports.
>>   To attach a new physical device port, the device will be recognized by
>>   userspace directly I/O framework in kernel at first. Then DPDK apps can
>>   call the port hotplug functions to attach ports.
>>   For detaching, steps are vice versa.
>> - Before detach ports, ports must be stopped and closed.
>>   DPDK application must call rte_eth_dev_stop() and rte_eth_dev_close() before
>>   detaching ports. These function will call finalization codes of PMDs.
>>   But so far, no PMD frees all resources allocated by initialization.
>>   It means PMDs are needed to be fixed to support the port hotplug.
>>   'RTE_PCI_DRV_DETACHABLE' is a new flag indicating a PMD supports detaching.
>>   Without this flag, detaching will be failed.
>> - Mustn't affect legacy DPDK apps.
>>   No DPDK EAL behavior is changed, if the port hotplug functions are't called.
>>   So all legacy DPDK apps can still work without modifications.
>>
>> And a few limitations.
>> - The port hotplug functions are not thread safe.
>>   DPDK apps should handle it.
>> - Only support Linux and igb_uio so far.
>>   BSD and VFIO is not supported. I will send VFIO patches at least, but I don't
>>   have a plan to submit BSD patch so far.
>>
>>
>> Here is port hotplug APIs.
>> -------------------------------------------------------------------------------
>> /**
>>  * Attach a new device.
>>  *
>>  * @param devargs
>>  *   A pointer to a strings array describing the new device
>>  *   to be attached. The strings should be a pci address like
>>  *   '0000:01:00.0' or virtual device name like 'eth_pcap0'.
>>  * @param port_id
>>  *  A pointer to a port identifier actually attached.
>>  * @return
>>  *  0 on success and port_id is filled, negative on error  */ int rte_eal_dev_attach(const char *devargs,
>> uint8_t *port_id);
>>
>> /**
>>  * Detach a device.
>>  *
>>  * @param port_id
>>  *   The port identifier of the device to detach.
>>  * @param addr
>>  *  A pointer to a device name actually detached.
>>  * @return
>>  *  0 on success and devname is filled, negative on error  */ int rte_eal_dev_detach(uint8_t port_id,
>> char *devname);
>> -------------------------------------------------------------------------------
>>
>> This patch series are for DPDK EAL. To use port hotplug function by DPDK apps, each PMD should be
>> fixed to support 'RTE_PCI_DRV_DETACHABLE' flag. Please check a patch for pcap PMD.
>>
>> Also please check testpmd patch. It will show you how to fix your legacy applications to support port
>> hotplug feature.
>>
>> PATCH v6 changes
>>  - Fix rte_eth_dev_uninit() to handle a return value of uninit
>>    function of PMD. To do this, below changes also be applied.
>>    - Fix a paramter of rte_eth_dev_free().
>>    - Use rte_eth_dev structure as the paramter of rte_eth_dev_free().
>>
>> PATCH v5 changes
>>  - Add runtime check passthrough driver type, like vfio-pci, igb_uio
>>    and uio_pci_generic.
>>    This was done by Qiu, Michael. Thanks a lot.
>>  - Change function names like below.
>>    - rte_eal_dev_find_and_invoke() to rte_eal_vdev_find_and_invoke().
>>    - rte_eal_dev_invoke() to rte_eal_vdev_invoke().
>>  - Add code to handle a return value of rte_eal_devargs_remove().
>>  - Fix pci address format in rte_eal_dev_detach().
>>  - Remove RTE_EAL_INVOKE_TYPE_UNKNOWN, because it's unused.
>>  - Change function definition of rte_eal_devargs_remove().
>>  - Fix pci_unmap_device() to check pt_driver.
>>  - Fix return value of below functions.
>>    - rte_eth_dev_get_changed_port().
>>    - rte_eth_dev_get_port_by_addr().
>>  - Change paramters of rte_eth_dev_validate_port() to cleanup code.
>>  - Fix pci_scan_one to handle pt_driver correctly.
>>    (Thanks to Qiu, Michael for above sugesstions)
>>
>> PATCH v4 changes
>>  - Merge patches to review easier.
>>  - Fix indent of 'if' statement.
>>  - Fix calculation method of eal_compare_pci_addr().
>>  - Fix header file declaration.
>>  - Add header file to determine if hotplug can be enabled.
>>    (Thanks to Qiu, Michael)
>>  - Use braces with 'for' loop.
>>  - Add paramerter checking.
>>  - Fix sanity check code
>>  - Fix comments of rte_eth_dev_type.
>>  - Change function names.
>>    (Thanks to Iremonger, Bernard)
>>
>> PATCH v3 changes:
>>  - Fix enum definition used in rte_ethdev.c.
>>    (Thanks to Zhang, Helin)
>>
>> PATCH v2 changes:
>>  - Replace rte_eal_dev_attach_pdev(), rte_eal_dev_detach_pdev,
>>    rte_eal_dev_attach_vdev() and rte_eal_dev_detach_vdev() to
>>    rte_eal_dev_attach() and rte_eal_dev_detach().
>>  - Add parameter values checking.
>>  - Refashion a few functions.
>>    (Thanks to Iremonger, Bernard)
>>
>> PATCH v1 Changes:
>>  - Fix error checking code of librte_eth APIs.
>>  - Fix issue that port from pcap PMD cannot be detached correctly.
>>  - Fix issue that testpmd could hang after forwarding, if attaching and detaching
>>    is repeatedly.
>>  - Fix if-condition of rte_eth_dev_get_port_by_addr().
>>    (Thanks to Mark Enright)
>>
>> RFC PATCH v2 Changes:
>> - remove 'rte_eth_dev_validate_port()', and cleanup codes.
>>
>>
>> Michael Qiu (2):
>>   eal_pci: Add flag to hold kernel driver type
>>   eal_pci: pci memory map work with driver type
>>
>> Tetsuya Mukawa (11):
>>   eal/pci,ethdev: Remove assumption that port will not be detached
>>   eal/pci: Consolidate pci address comparison APIs
>>   ethdev: Add rte_eth_dev_free to free specified device
>>   eal,ethdev: Add a function and function pointers to close ether device
>>   ethdev: Add functions that will be used by port hotplug functions
>>   eal/linux/pci: Add functions for unmapping igb_uio resources
>>   eal/pci: Add a function to remove the entry of devargs list
>>   eal/pci: Cleanup pci driver initialization code
>>   ethdev: Add one dev_type paramerter to rte_eth_dev_allocate
>>   eal/pci: Add rte_eal_dev_attach/detach() functions
>>   eal: Enable port hotplug framework in Linux
>>
>>  app/test/virtual_pmd.c                          |   2 +-
>>  config/common_linuxapp                          |   5 +
>>  lib/librte_eal/bsdapp/eal/eal_pci.c             |  25 +-
>>  lib/librte_eal/common/Makefile                  |   1 +
>>  lib/librte_eal/common/eal_common_dev.c          | 274 +++++++++++
>>  lib/librte_eal/common/eal_common_devargs.c      |  60 +++
>>  lib/librte_eal/common/eal_common_pci.c          |  92 +++-
>>  lib/librte_eal/common/eal_private.h             |  35 ++
>>  lib/librte_eal/common/include/rte_dev.h         |  33 ++
>>  lib/librte_eal/common/include/rte_dev_hotplug.h |  44 ++
>>  lib/librte_eal/common/include/rte_devargs.h     |  21 +
>>  lib/librte_eal/common/include/rte_pci.h         |  84 ++++
>>  lib/librte_eal/linuxapp/eal/Makefile            |   1 +
>>  lib/librte_eal/linuxapp/eal/eal_pci.c           | 227 +++++++--
>>  lib/librte_eal/linuxapp/eal/eal_pci_init.h      |   8 +
>>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c       |  67 ++-
>>  lib/librte_ether/rte_ethdev.c                   | 624 +++++++++++++-----------
>>  lib/librte_ether/rte_ethdev.h                   | 148 +++++-
>>  lib/librte_pmd_af_packet/rte_eth_af_packet.c    |   2 +-
>>  lib/librte_pmd_bond/rte_eth_bond_api.c          |   2 +-
>>  lib/librte_pmd_pcap/rte_eth_pcap.c              |   2 +-
>>  lib/librte_pmd_ring/rte_eth_ring.c              |   2 +-
>>  lib/librte_pmd_xenvirt/rte_eth_xenvirt.c        |   2 +-
>>  23 files changed, 1414 insertions(+), 347 deletions(-)  create mode 100644
>> lib/librte_eal/common/include/rte_dev_hotplug.h
>>
>> --
>> 1.9.1
> Hi Tetsuya,
>
> I have a few comments as follows:
>
> Lib/librte_ether/rte_ethdev.c
> Line 400 :   NO_TRACE would be clearer than NONE_TRACE.
> Line 440:    function rte_eth_dev_save()  should pass in a size parameter which should be used in the memcpy function.

Hi Bernard,

I appreciate your comment, I will fix like above.

> Lib/librte_eal/common/include/ret_dev_hotplug.h
>  Is this file really necessary as it contains only one macro?
> Could it be merged into Lib/librte_eal/common/include/ret_dev.h ?

I guess it's nice to separate.
As you said,  rte_dev_hotplug.h has only one macro.
This macro is used for determining whether user enables hotplug
function, or not.
 And it is included by below headers.

./lib/librte_eal/common/include/rte_pci.h
./lib/librte_eal/linuxapp/eal/eal_pci_init.h

Above 2 headers doesn't need any rte_dev.h definitions without hotplug
macro.
Also when someone reads DPDK codes, they can easily understand 'pci' and
'dev' layer is conceptually separated.
So I guess it's nice to separate hotplug definition from rte_dev.h.

Thanks,
Tetsuya

> Regards,
>
> Bernard.
>
>
>
Tetsuya Mukawa Feb. 5, 2015, 1:34 a.m. UTC | #6 
On 2015/02/04 3:35, Iremonger, Bernard wrote:
>
>> -----Original Message-----
>> From: Tetsuya Mukawa [mailto:mukawa@igel.co.jp]
>> Sent: Sunday, February 1, 2015 4:02 AM
>> To: dev@dpdk.org
>> Cc: Qiu, Michael; Iremonger, Bernard; Tetsuya Mukawa
>> Subject: [PATCH v6 00/13] Port Hotplug Framework
>>
>> This patch series adds a dynamic port hotplug framework to DPDK.
>> With the patches, DPDK apps can attach or detach ports at runtime.
>>
>> The basic concept of the port hotplug is like followings.
>> - DPDK apps must have responsibility to manage ports.
>>   DPDK apps only know which ports are attached or detached at the moment.
>>   The port hotplug framework is implemented to allow DPDK apps to manage ports.
>>   For example, when DPDK apps call port attach function, attached port number
>>   will be returned. Also DPDK apps can detach port by port number.
>> - Kernel support is needed for attaching or detaching physical device ports.
>>   To attach a new physical device port, the device will be recognized by
>>   userspace directly I/O framework in kernel at first. Then DPDK apps can
>>   call the port hotplug functions to attach ports.
>>   For detaching, steps are vice versa.
>> - Before detach ports, ports must be stopped and closed.
>>   DPDK application must call rte_eth_dev_stop() and rte_eth_dev_close() before
>>   detaching ports. These function will call finalization codes of PMDs.
>>   But so far, no PMD frees all resources allocated by initialization.
>>   It means PMDs are needed to be fixed to support the port hotplug.
>>   'RTE_PCI_DRV_DETACHABLE' is a new flag indicating a PMD supports detaching.
>>   Without this flag, detaching will be failed.
>> - Mustn't affect legacy DPDK apps.
>>   No DPDK EAL behavior is changed, if the port hotplug functions are't called.
>>   So all legacy DPDK apps can still work without modifications.
>>
>> And a few limitations.
>> - The port hotplug functions are not thread safe.
>>   DPDK apps should handle it.
>> - Only support Linux and igb_uio so far.
>>   BSD and VFIO is not supported. I will send VFIO patches at least, but I don't
>>   have a plan to submit BSD patch so far.
>>
>>
>> Here is port hotplug APIs.
>> -------------------------------------------------------------------------------
>> /**
>>  * Attach a new device.
>>  *
>>  * @param devargs
>>  *   A pointer to a strings array describing the new device
>>  *   to be attached. The strings should be a pci address like
>>  *   '0000:01:00.0' or virtual device name like 'eth_pcap0'.
>>  * @param port_id
>>  *  A pointer to a port identifier actually attached.
>>  * @return
>>  *  0 on success and port_id is filled, negative on error  */ int rte_eal_dev_attach(const char *devargs,
>> uint8_t *port_id);
>>
>> /**
>>  * Detach a device.
>>  *
>>  * @param port_id
>>  *   The port identifier of the device to detach.
>>  * @param addr
>>  *  A pointer to a device name actually detached.
>>  * @return
>>  *  0 on success and devname is filled, negative on error  */ int rte_eal_dev_detach(uint8_t port_id,
>> char *devname);
>> -------------------------------------------------------------------------------
>>
>> This patch series are for DPDK EAL. To use port hotplug function by DPDK apps, each PMD should be
>> fixed to support 'RTE_PCI_DRV_DETACHABLE' flag. Please check a patch for pcap PMD.
>>
>> Also please check testpmd patch. It will show you how to fix your legacy applications to support port
>> hotplug feature.
>>
>> PATCH v6 changes
>>  - Fix rte_eth_dev_uninit() to handle a return value of uninit
>>    function of PMD. To do this, below changes also be applied.
>>    - Fix a paramter of rte_eth_dev_free().
>>    - Use rte_eth_dev structure as the paramter of rte_eth_dev_free().
>>
>> PATCH v5 changes
>>  - Add runtime check passthrough driver type, like vfio-pci, igb_uio
>>    and uio_pci_generic.
>>    This was done by Qiu, Michael. Thanks a lot.
>>  - Change function names like below.
>>    - rte_eal_dev_find_and_invoke() to rte_eal_vdev_find_and_invoke().
>>    - rte_eal_dev_invoke() to rte_eal_vdev_invoke().
>>  - Add code to handle a return value of rte_eal_devargs_remove().
>>  - Fix pci address format in rte_eal_dev_detach().
>>  - Remove RTE_EAL_INVOKE_TYPE_UNKNOWN, because it's unused.
>>  - Change function definition of rte_eal_devargs_remove().
>>  - Fix pci_unmap_device() to check pt_driver.
>>  - Fix return value of below functions.
>>    - rte_eth_dev_get_changed_port().
>>    - rte_eth_dev_get_port_by_addr().
>>  - Change paramters of rte_eth_dev_validate_port() to cleanup code.
>>  - Fix pci_scan_one to handle pt_driver correctly.
>>    (Thanks to Qiu, Michael for above sugesstions)
>>
>> PATCH v4 changes
>>  - Merge patches to review easier.
>>  - Fix indent of 'if' statement.
>>  - Fix calculation method of eal_compare_pci_addr().
>>  - Fix header file declaration.
>>  - Add header file to determine if hotplug can be enabled.
>>    (Thanks to Qiu, Michael)
>>  - Use braces with 'for' loop.
>>  - Add paramerter checking.
>>  - Fix sanity check code
>>  - Fix comments of rte_eth_dev_type.
>>  - Change function names.
>>    (Thanks to Iremonger, Bernard)
>>
>> PATCH v3 changes:
>>  - Fix enum definition used in rte_ethdev.c.
>>    (Thanks to Zhang, Helin)
>>
>> PATCH v2 changes:
>>  - Replace rte_eal_dev_attach_pdev(), rte_eal_dev_detach_pdev,
>>    rte_eal_dev_attach_vdev() and rte_eal_dev_detach_vdev() to
>>    rte_eal_dev_attach() and rte_eal_dev_detach().
>>  - Add parameter values checking.
>>  - Refashion a few functions.
>>    (Thanks to Iremonger, Bernard)
>>
>> PATCH v1 Changes:
>>  - Fix error checking code of librte_eth APIs.
>>  - Fix issue that port from pcap PMD cannot be detached correctly.
>>  - Fix issue that testpmd could hang after forwarding, if attaching and detaching
>>    is repeatedly.
>>  - Fix if-condition of rte_eth_dev_get_port_by_addr().
>>    (Thanks to Mark Enright)
>>
>> RFC PATCH v2 Changes:
>> - remove 'rte_eth_dev_validate_port()', and cleanup codes.
>>
>>
>> Michael Qiu (2):
>>   eal_pci: Add flag to hold kernel driver type
>>   eal_pci: pci memory map work with driver type
>>
>> Tetsuya Mukawa (11):
>>   eal/pci,ethdev: Remove assumption that port will not be detached
>>   eal/pci: Consolidate pci address comparison APIs
>>   ethdev: Add rte_eth_dev_free to free specified device
>>   eal,ethdev: Add a function and function pointers to close ether device
>>   ethdev: Add functions that will be used by port hotplug functions
>>   eal/linux/pci: Add functions for unmapping igb_uio resources
>>   eal/pci: Add a function to remove the entry of devargs list
>>   eal/pci: Cleanup pci driver initialization code
>>   ethdev: Add one dev_type paramerter to rte_eth_dev_allocate
>>   eal/pci: Add rte_eal_dev_attach/detach() functions
>>   eal: Enable port hotplug framework in Linux
>>
>>  app/test/virtual_pmd.c                          |   2 +-
>>  config/common_linuxapp                          |   5 +
>>  lib/librte_eal/bsdapp/eal/eal_pci.c             |  25 +-
>>  lib/librte_eal/common/Makefile                  |   1 +
>>  lib/librte_eal/common/eal_common_dev.c          | 274 +++++++++++
>>  lib/librte_eal/common/eal_common_devargs.c      |  60 +++
>>  lib/librte_eal/common/eal_common_pci.c          |  92 +++-
>>  lib/librte_eal/common/eal_private.h             |  35 ++
>>  lib/librte_eal/common/include/rte_dev.h         |  33 ++
>>  lib/librte_eal/common/include/rte_dev_hotplug.h |  44 ++
>>  lib/librte_eal/common/include/rte_devargs.h     |  21 +
>>  lib/librte_eal/common/include/rte_pci.h         |  84 ++++
>>  lib/librte_eal/linuxapp/eal/Makefile            |   1 +
>>  lib/librte_eal/linuxapp/eal/eal_pci.c           | 227 +++++++--
>>  lib/librte_eal/linuxapp/eal/eal_pci_init.h      |   8 +
>>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c       |  67 ++-
>>  lib/librte_ether/rte_ethdev.c                   | 624 +++++++++++++-----------
>>  lib/librte_ether/rte_ethdev.h                   | 148 +++++-
>>  lib/librte_pmd_af_packet/rte_eth_af_packet.c    |   2 +-
>>  lib/librte_pmd_bond/rte_eth_bond_api.c          |   2 +-
>>  lib/librte_pmd_pcap/rte_eth_pcap.c              |   2 +-
>>  lib/librte_pmd_ring/rte_eth_ring.c              |   2 +-
>>  lib/librte_pmd_xenvirt/rte_eth_xenvirt.c        |   2 +-
>>  23 files changed, 1414 insertions(+), 347 deletions(-)  create mode 100644
>> lib/librte_eal/common/include/rte_dev_hotplug.h
>>
>> --
>> 1.9.1
> Hi Tetsuya,
>
> In lib/librte_eal/linuxapp/eal/eal_pci.c
> Line 183:   should "mapped" be "unmapped" ?

Hi Bernard,

Thanks, this error message should be "unmapped".
I will fix it.

Thanks,
Tetsuya

> Regards,
>
> Bernard.
>
>
>
diff mbox

Patch

diff --git a/lib/librte_eal/common/eal_common_pci.c b/lib/librte_eal/common/eal_common_pci.c
index a89f5c3..7c9b8c5 100644
--- a/lib/librte_eal/common/eal_common_pci.c
+++ b/lib/librte_eal/common/eal_common_pci.c
@@ -99,19 +99,27 @@  static struct rte_devargs *pci_devargs_lookup(struct rte_pci_device *dev)
 	return NULL;
 }
 
-/*
- * If vendor/device ID match, call the devinit() function of all
- * registered driver for the given device. Return -1 if initialization
- * failed, return 1 if no driver is found for this device.
- */
 static int
-pci_probe_all_drivers(struct rte_pci_device *dev)
+pci_invoke_all_drivers(struct rte_pci_device *dev,
+		enum rte_eal_invoke_type type)
 {
 	struct rte_pci_driver *dr = NULL;
-	int rc;
+	int rc = 0;
+
+	if ((dev == NULL) || (type >= RTE_EAL_INVOKE_TYPE_MAX))
+		return -1;
 
 	TAILQ_FOREACH(dr, &pci_driver_list, next) {
-		rc = rte_eal_pci_probe_one_driver(dr, dev);
+		switch (type) {
+		case RTE_EAL_INVOKE_TYPE_PROBE:
+			rc = rte_eal_pci_probe_one_driver(dr, dev);
+			break;
+		case RTE_EAL_INVOKE_TYPE_CLOSE:
+			rc = rte_eal_pci_close_one_driver(dr, dev);
+			break;
+		default:
+			return -1;
+		}
 		if (rc < 0)
 			/* negative value is an error */
 			return -1;
@@ -123,6 +131,66 @@  pci_probe_all_drivers(struct rte_pci_device *dev)
 	return 1;
 }
 
+#ifdef ENABLE_HOTPLUG
+static int
+rte_eal_pci_invoke_one(struct rte_pci_addr *addr,
+		enum rte_eal_invoke_type type)
+{
+	struct rte_pci_device *dev = NULL;
+	int ret = 0;
+
+	if ((addr == NULL) || (type >= RTE_EAL_INVOKE_TYPE_MAX))
+		return -1;
+
+	TAILQ_FOREACH(dev, &pci_device_list, next) {
+		if (eal_compare_pci_addr(&dev->addr, addr))
+			continue;
+
+		ret = pci_invoke_all_drivers(dev, type);
+		if (ret < 0)
+			goto invoke_err_return;
+
+		if (type == RTE_EAL_INVOKE_TYPE_CLOSE)
+			goto remove_dev;
+
+		return 0;
+	}
+
+	return -1;
+
+invoke_err_return:
+	RTE_LOG(WARNING, EAL, "Requested device " PCI_PRI_FMT
+			" cannot be used\n", dev->addr.domain, dev->addr.bus,
+			dev->addr.devid, dev->addr.function);
+	return -1;
+
+remove_dev:
+	TAILQ_REMOVE(&pci_device_list, dev, next);
+	return 0;
+}
+
+
+/*
+ * Find the pci device specified by pci address, then invoke probe function of
+ * the driver of the devive.
+ */
+int
+rte_eal_pci_probe_one(struct rte_pci_addr *addr)
+{
+	return rte_eal_pci_invoke_one(addr, RTE_EAL_INVOKE_TYPE_PROBE);
+}
+
+/*
+ * Find the pci device specified by pci address, then invoke close function of
+ * the driver of the devive.
+ */
+int
+rte_eal_pci_close_one(struct rte_pci_addr *addr)
+{
+	return rte_eal_pci_invoke_one(addr, RTE_EAL_INVOKE_TYPE_CLOSE);
+}
+#endif /* ENABLE_HOTPLUG */
+
 /*
  * Scan the content of the PCI bus, and call the devinit() function for
  * all registered drivers that have a matching entry in its id_table
@@ -148,10 +216,12 @@  rte_eal_pci_probe(void)
 
 		/* probe all or only whitelisted devices */
 		if (probe_all)
-			ret = pci_probe_all_drivers(dev);
+			ret = pci_invoke_all_drivers(dev,
+					RTE_EAL_INVOKE_TYPE_PROBE);
 		else if (devargs != NULL &&
 			devargs->type == RTE_DEVTYPE_WHITELISTED_PCI)
-			ret = pci_probe_all_drivers(dev);
+			ret = pci_invoke_all_drivers(dev,
+					RTE_EAL_INVOKE_TYPE_PROBE);
 		if (ret < 0)
 			rte_exit(EXIT_FAILURE, "Requested device " PCI_PRI_FMT
 				 " cannot be used\n", dev->addr.domain, dev->addr.bus,
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 159cd66..a97c5d8 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -154,6 +154,16 @@  struct rte_pci_driver;
 struct rte_pci_device;
 
 /**
+ * The invoke type.
+ */
+enum rte_eal_invoke_type {
+	RTE_EAL_INVOKE_TYPE_UNKNOWN,/**< unknown */
+	RTE_EAL_INVOKE_TYPE_PROBE,  /**< invoke probe function */
+	RTE_EAL_INVOKE_TYPE_CLOSE,  /**< invoke close function */
+	RTE_EAL_INVOKE_TYPE_MAX     /**< max value of this enum */
+};
+
+/**
  * Mmap memory for single PCI device
  *
  * This function is private to EAL.
@@ -165,6 +175,21 @@  int rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr,
 		struct rte_pci_device *dev);
 
 /**
+ * Munmap memory for single PCI device
+ *
+ * This function is private to EAL.
+ *
+ * @param	dr
+ *  The pointer to the pci driver structure
+ * @param	dev
+ *  The pointer to the pci device structure
+ * @return
+ *   0 on success, negative on error
+ */
+int rte_eal_pci_close_one_driver(struct rte_pci_driver *dr,
+		struct rte_pci_device *dev);
+
+/**
  * Init tail queues for non-EAL library structures. This is to allow
  * the rings, mempools, etc. lists to be shared among multiple processes
  *
diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h
index fe57cb6..28462f5 100644
--- a/lib/librte_eal/common/include/rte_pci.h
+++ b/lib/librte_eal/common/include/rte_pci.h
@@ -82,6 +82,7 @@  extern "C" {
 #include <inttypes.h>
 
 #include <rte_interrupts.h>
+#include <rte_dev_hotplug.h>
 
 TAILQ_HEAD(pci_device_list, rte_pci_device); /**< PCI devices in D-linked Q. */
 TAILQ_HEAD(pci_driver_list, rte_pci_driver); /**< PCI drivers in D-linked Q. */
@@ -315,6 +316,38 @@  eal_compare_pci_addr(struct rte_pci_addr *addr, struct rte_pci_addr *addr2)
  */
 int rte_eal_pci_probe(void);
 
+#ifdef ENABLE_HOTPLUG
+/**
+ * Probe the single PCI device.
+ *
+ * Scan the content of the PCI bus, and find the pci device specified by pci
+ * address, then call the probe() function for registered driver that has a
+ * matching entry in its id_table for discovered device.
+ *
+ * @param addr
+ *	The PCI Bus-Device-Function address to probe or close.
+ * @return
+ *   - 0 on success.
+ *   - Negative on error.
+ */
+int rte_eal_pci_probe_one(struct rte_pci_addr *addr);
+
+/**
+ * Close the single PCI device.
+ *
+ * Scan the content of the PCI bus, and find the pci device specified by pci
+ * address, then call the close() function for registered driver that has a
+ * matching entry in its id_table for discovered device.
+ *
+ * @param addr
+ *	The PCI Bus-Device-Function address to probe or close.
+ * @return
+ *   - 0 on success.
+ *   - Negative on error.
+ */
+int rte_eal_pci_close_one(struct rte_pci_addr *addr);
+#endif /* ENABLE_HOTPLUG */
+
 /**
  * Dump the content of the PCI bus.
  *
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 52c464c..a23cc59 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -619,6 +619,75 @@  rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr, struct rte_pci_device *d
 	return 1;
 }
 
+#ifdef ENABLE_HOTPLUG
+/*
+ * If vendor/device ID match, call the devuninit() function of the
+ * driver.
+ */
+int
+rte_eal_pci_close_one_driver(struct rte_pci_driver *dr,
+		struct rte_pci_device *dev)
+{
+	struct rte_pci_id *id_table;
+
+	if ((dr == NULL) || (dev == NULL))
+		return -EINVAL;
+
+	for (id_table = dr->id_table ; id_table->vendor_id != 0; id_table++) {
+
+		/* check if device's identifiers match the driver's ones */
+		if (id_table->vendor_id != dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+
+		struct rte_pci_addr *loc = &dev->addr;
+
+		RTE_LOG(DEBUG, EAL,
+				"PCI device "PCI_PRI_FMT" on NUMA socket %i\n",
+				loc->domain, loc->bus, loc->devid,
+				loc->function, dev->numa_node);
+
+		RTE_LOG(DEBUG, EAL, "  remove driver: %x:%x %s\n",
+				dev->id.vendor_id, dev->id.device_id,
+				dr->name);
+
+		/* call the driver devuninit() function */
+		if (dr->devuninit && (dr->devuninit(dr, dev) < 0))
+			return -1;	/* negative value is an error */
+
+		/* clear driver structure */
+		dev->driver = NULL;
+
+		if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING)
+			/* unmap resources for devices that use igb_uio */
+			pci_unmap_device(dev);
+
+		return 0;
+	}
+	/* return positive value if driver is not found */
+	return 1;
+}
+#else /* ENABLE_HOTPLUG */
+int
+rte_eal_pci_close_one_driver(struct rte_pci_driver *dr __rte_unused,
+		struct rte_pci_device *dev __rte_unused)
+{
+	RTE_LOG(ERR, EAL, "Hotplug support isn't enabled\n");
+	return -1;
+}
+#endif /* ENABLE_HOTPLUG */
+
 /* Init the PCI EAL subsystem */
 int
 rte_eal_pci_init(void)