[dpdk-dev,v9,01/12] eal/linux: add interrupt vectors support in intr_handle

  The patch adds interrupt vectors support in rte_intr_handle.
'vec_en' is set when interrupt vectors are detected and associated event fds are set.
Those event fds are stored in efds[].
'intr_vec' is reserved for device driver to initialize the vector mapping table.
When the event fds add to a specified epoll instance, 'eptrs' will hold the rte_epoll_event object pointer.

Signed-off-by: Danny Zhou <danny.zhou@intel.com>
Signed-off-by: Cunming Liang <cunming.liang@intel.com>
---
v7 changes:
 - add eptrs[], it's used to store the register rte_epoll_event instances.
 - add vec_en, to log the vector capability status.

v6 changes:
 - add mapping table between irq vector number and queue id.

v5 changes:
 - Create this new patch file for changed struct rte_intr_handle that
   other patches depend on, to avoid breaking git bisect.

 lib/librte_eal/linuxapp/eal/include/exec-env/rte_interrupts.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
  

Tested-by: Yong Liu <yong.liu@intel.com>

- Tested Commit: 7c4c66bf666b8059ed0ad2f2478ef349b3272f51
- OS: Fedora20 3.15.5
- GCC: gcc version 4.8.3 20140911
- CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
- NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ [8086:10fb]
- NIC: Intel Corporation I350 Gigabit Network Connection [8086:1521]
- Default x86_64-native-linuxapp-gcc configuration
- Prerequisites: vfio related case request vt-d enable in bios and IOMMU enable in kernel 
- Total 17 cases, 15 passed, 2 failed
- Failed cases: pf_rxq_on_uiopcigeneric, pf_lsc_on_uiopcigeneric

- Case: pf_lsc_igbuio_legacy
  Description: check when pf bound to igb_uio with legacy mode, link status change interrupt can be normally handled
  Command / instruction:
    Insmod igb_uio driver with legacy interrupt mode
      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=legacy
    Change port config to lsc enable and rxq disable in l3fwd-power/main.c
    Build l3fwd-power and start l3fwd-power with 2 ports  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Change tester port0 link down and verify link down detected on dut port0 
      Port 0: link down
    Change tester port0 link up and verify link up detected on dut port0
      Port 0: link up
    Change tester port1 link down and verify link down detected on dut port1 
      Port 1: link down
    Change tester port1 link up and verify link up detected on dut port1
      Port 1: link up
  
    Change port config to lsc enable and rxq enable in l3fwd-power/main.c
    Build l3fwd-power and start l3fwd-power with 2 ports
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify lsc disabled for can't enable lsc and rxq in the same time when pf bound to igb_uio
      lsc won't enable because of no intr multiplex
  
- Case: pf_lsc_igbuio_msix
  Description: check when pf bound to igb_uio with msix mode, link status change interrupt can be normally handled
  Command / instruction:
    Insmod igb_uio driver with msix interrupt mode
      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=msix
    Verify link status can be normally handled like previous case pf_lsc_igbuio_legacy.

- Case: pf_lsc_vfio_legacy
  Description: check when pf bound to vfio with legacy mode, link status change interrupt can be normally handled
  Command / instruction:
    Do prerequisites for vfio driver then bind device to vfio-driver
      modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
    Change port config to lsc enable and rxq disable in l3fwd-power/main.c
    Start l3fwd-power with vfio legacy mode
      l3fwd-power -c 0x6 -n 3 --vfio-intr=legacy -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Check link status change interrupt can be normally handled like previous case.
	
    Change port config to lsc enable and rxq enable in l3fwd-power/main.c
    Start l3fwd-power with vfio legacy mode
      l3fwd-power -c 0x6 -n 3 --vfio-intr=legacy -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify lsc disabled for can't enable lsc and rxq in the same time with legacy mode.
	
- Case: pf_lsc_vfio_msi
  Description: check when pf bound to vfio with msi mode, link status change interrupt can be normally handled
  Command / instruction:
    Do prerequisites for vfio driver then bind device to vfio-driver
      modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
    Change port config to lsc enable and rxq disable in l3fwd-power/main.c
    Start l3fwd-power with vfio msi mode
      l3fwd-power -c 0x6 -n 3 --vfio-intr=msi -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Check link status change interrupt can be normally handled like previous case.
	
    Change port config to lsc enable and rxq enable in l3fwd-power/main.c
    Start l3fwd-power with vfio msi mode
      l3fwd-power -c 0x6 -n 3 --vfio-intr=msi -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify lsc disabled for can't enable lsc and rxq in the same time with legacy mode.

- Case: pf_lsc_vfio_msix
  Description: check when pf bound to vfio with msix mode, link status change interrupt can be normally handled
  Command / instruction:
    Do prerequisites for vfio driver then bind device to vfio-driver
      modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
    Change port config to lsc enable and rxq disable in l3fwd-power/main.c
    Start l3fwd-power with vfio msix mode
      l3fwd-power -c 0x6 -n 3 --vfio-intr=msix -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Check link status change interrupt can be normally handled like previous case.
	
    Change port config to lsc enable and rxq enable in l3fwd-power/main.c
    Start l3fwd-power with vfio msix mode
      l3fwd-power -c 0x6 -n 3 --vfio-intr=msix -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Check link status change interrupt can be normally handled like previous case.
	
- Case: pf_rxq_on_vfio_msix
  Description: check when pf bound to vfio with default msix mode, receive packet interrupt can be normally handled
  Command / instruction:
    Do prerequisites for vfio driver then bind device to vfio-driver
          modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Send packet from tester port0 and verify dut core1 wakeup and then sleep.
      lcore 1 is waked up from rx interrupt on port 0 queue 0
      lcore 1 sleeps until interrupt triggers
    Send packet from tester port1 and verify dut core2 wakeup and then sleep.
      lcore 2 is waked up from rx interrupt on port 1 queue 0
      lcore 2 sleeps until interrupt triggers
	  
- Case: pf_rxq_on_vfio_msi
  Description: check when pf bound to vfio with msi mode, receive packet interrupt can be normally handled
  Command / instruction:
    Do prerequisites for vfio driver then bind device to vfio-driver
          modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 --vfio-intr=msi -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
  
- Case: pf_rxq_on_vfio_legacy
  Description: check when pf bound to vfio with legacy mode, receive packet interrupt can be normally handled
  Command / instruction:
    Do prerequisites for vfio driver then bind device to vfio-driver
      modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 --vfio-intr=legacy -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.

- Case: pf_onecore_on_vfio
  Description: check when all pf devices bound to one core, receive packet interrupt can be normally handled
  Command / instruction:
    Do prerequisites for vfio driver then bind device to vfio-driver
      modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
    Start l3fwd-power with 2 ports and 1 cores.  
      l3fwd-power -c 0x2 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,1)"
    Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
	
- Case: pf_multiqueue_on_vfio
  Description: check when pf device has mulit queues, receive packet interrupt can be normally handled
  Command / instruction:
    Start l3fwd-power with 2 ports and 4 cores.
      l3fwd-power -c 0x100000e -n 3 -- -p 0x3 -P --config="(0,0,1),(0,1,2),(1,0,3),(1,1,24)"
    Send enough packets with different destination ip address.
	  sendp([Ether()/IP(dst="127.0.0.X")/UDP()/Raw('0'*18)], iface="p786p1")
    Verify all cores wakeup and then sleep as expected.

- Case: pf_maxqueue_on_vfio
  Description: check when pf device has maximum queues, receive packet interrupt can be normally handled
  Command / instruction:
    Start l3fwd-power with 2 ports and 32 cores [only for niantic], different nic has different maximum rx queues
      l3fwd-power -c 0x3fdfe3fdfe -n 3 -- -p 0x3 -P --config="(0,0,1),(0,1,21),(0,2,2),(0,3,22),\
          (0,4,3),(0,5,23),(0,6,24),(0,7,4),(0,8,25),(0,9,5),(0,10,26),(0,11,6),(0,12,27),(0,13,7),\
          (0,14,8),(0,15,28),(1,0,10),(1,1,30),(1,2,11),(1,3,31),(1,4,32),(1,5,12),(1,6,33),(1,7,13),\
          (1,8,34),(1,9,14),(1,10,35),(1,11,15),(1,12,16),(1,13,36),(1,14,17),(1,15,37),"
    Send enough packets with different destination ip address.
	  sendp([Ether()/IP(dst="127.0.0.X")/UDP()/Raw('0'*18)], iface="p786p1")
    Verify all cores wakeup and then sleep as expected.	
	
- Case: pf_rxq_on_igbuio_legacy
  Description: check when pf bound to igb_uio with legacy mode, receive packet interrupt can be normally handled
  Command / instruction:
    Insmod igb_uio driver with legacy interrupt mode
      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=legacy
      ./tools/dpdk_nic_bind.py --bind=igb_uio 08:00.0 08:00.1  
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
  
- Case: pf_rxq_on_igbuio_msix
  Description: check when pf bound to igb_uio with msix mode, receive packet interrupt can be normally handled
  Command / instruction:
    Insmod igb_uio driver with msix interrupt mode
      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=msix
      ./tools/dpdk_nic_bind.py --bind=igb_uio 08:00.0 08:00.1  
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.

- Case: pf_rxq_on_uiopcigeneric
  Description: check when pf bound to uio_pci_generic, receive packet interrupt can be normally handled
  Command / instruction:
    Insmod uio_pci_generic driver and bind pf device on it.
      ./tools/dpdk_nic_bind.py --bind=uio_pci_generic 08:00.0 08:00.1  
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.	

- Case: pf_lsc_on_uiopcigeneric
  Description: check when pf bound to uio_pci_generic, link status changed interrupt can be normally handled
  Command / instruction:
    Insmod uio_pci_generic driver and bind pf device on it.
      ./tools/dpdk_nic_bind.py --bind=uio_pci_generic 08:00.0 08:00.1  
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Change tester port0 link down and verify link down detected on dut port0 
      Port 0: link down
    Change tester port0 link up and verify link up detected on dut port0
      Port 0: link up
    Change tester port1 link down and verify link down detected on dut port1 
      Port 1: link down
    Change tester port1 link up and verify link up detected on dut port1
      Port 1: link up
 
    Change port config to lsc enable and rxq enable in l3fwd-power/main.c
    Build l3fwd-power and start l3fwd-power with 2 ports
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Verify lsc disabled for can't enable lsc and rxq in the same time when pf bound to uio_pci_generic
      lsc won't enable because of no intr multiplex	
	
- Case: vf_in_vm_rxq
  Description: check when vf bound to igb_uio in virtual machine, receive packet interrupt can be normally handled
               Only support niantic by now.
  Command / instruction:
    Create vf devices and bound into virtual machine
      echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs
      echo 1 > /sys/bus/pci/devices/0000\:08\:00.1/sriov_numvfs
      virsh
      virsh # nodedev-dettach pci_0000_08_10_0
      virsh # nodedev-dettach pci_0000_08_10_1
    Start virtual machine and bind vf devices to driver igb_uio.
      ./tools/dpdk_nic_bind.py --bind=igb_uio eth1 eth2
    Change port config to lsc disable and rxq enable in l3fwd-power/main.c	
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Send packet from tester port0 with promisc mac and verify vm core1 wakeup and then sleep.
      lcore 1 is waked up from rx interrupt on port 0 queue 0
      lcore 1 sleeps until interrupt triggers
    Send packet from tester port1 with promisc mac and verify vm core2 wakeup and then sleep.
      lcore 2 is waked up from rx interrupt on port 1 queue 0
      lcore 2 sleeps until interrupt triggers
	  
- Case: vf_in_host_rxq
  Description: check when vf bound to vfio with msix mode, receive packet interrupt can be normally handled
               Only support niantic by now.
  Command / instruction:
    Create vf devices and bound to vfio
      echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs
      echo 1 > /sys/bus/pci/devices/0000\:08\:00.1/sriov_numvfs
      modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 08:10.0 08:10.1
    Start l3fwd-power with 2 ports and 2 cores.  
      l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Send packet from tester port0 with promisc mac and verify dut core1 wakeup and then sleep.
      lcore 1 is waked up from rx interrupt on port 0 queue 0
      lcore 1 sleeps until interrupt triggers
    Send packet from tester port1 with promisc mac and verify dut core2 wakeup and then sleep.
      lcore 2 is waked up from rx interrupt on port 1 queue 0
      lcore 2 sleeps until interrupt triggers

> -----Original Message-----
> From: Liang, Cunming
> Sent: Friday, May 29, 2015 4:45 PM
> To: dev@dpdk.org
> Cc: shemming@brocade.com; david.marchand@6wind.com;
> thomas.monjalon@6wind.com; Zhou, Danny; Wang, Liang-min; Richardson, Bruce;
> Liu, Yong; nhorman@tuxdriver.com; Liang, Cunming
> Subject: [PATCH v9 01/12] eal/linux: add interrupt vectors support in
> intr_handle
> 
> The patch adds interrupt vectors support in rte_intr_handle.
> 'vec_en' is set when interrupt vectors are detected and associated event
> fds are set.
> Those event fds are stored in efds[].
> 'intr_vec' is reserved for device driver to initialize the vector mapping
> table.
> When the event fds add to a specified epoll instance, 'eptrs' will hold
> the rte_epoll_event object pointer.
> 
> Signed-off-by: Danny Zhou <danny.zhou@intel.com>
> Signed-off-by: Cunming Liang <cunming.liang@intel.com>
> ---
> v7 changes:
>  - add eptrs[], it's used to store the register rte_epoll_event instances.
>  - add vec_en, to log the vector capability status.
> 
> v6 changes:
>  - add mapping table between irq vector number and queue id.
> 
> v5 changes:
>  - Create this new patch file for changed struct rte_intr_handle that
>    other patches depend on, to avoid breaking git bisect.
> 
>  lib/librte_eal/linuxapp/eal/include/exec-env/rte_interrupts.h | 10
> ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_interrupts.h
> b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_interrupts.h
> index bdeb3fc..9c86a15 100644
> --- a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_interrupts.h
> +++ b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_interrupts.h
> @@ -38,6 +38,8 @@
>  #ifndef _RTE_LINUXAPP_INTERRUPTS_H_
>  #define _RTE_LINUXAPP_INTERRUPTS_H_
> 
> +#define RTE_MAX_RXTX_INTR_VEC_ID     32
> +
>  enum rte_intr_handle_type {
>  	RTE_INTR_HANDLE_UNKNOWN = 0,
>  	RTE_INTR_HANDLE_UIO,          /**< uio device handle */
> @@ -49,6 +51,8 @@ enum rte_intr_handle_type {
>  	RTE_INTR_HANDLE_MAX
>  };
> 
> +struct rte_epoll_event;
> +
>  /** Handle for interrupts. */
>  struct rte_intr_handle {
>  	union {
> @@ -58,6 +62,12 @@ struct rte_intr_handle {
>  	};
>  	int fd;	 /**< interrupt event file descriptor */
>  	enum rte_intr_handle_type type;  /**< handle type */
> +	uint32_t max_intr;               /**< max interrupt requested */
> +	uint32_t nb_efd;                 /**< number of available efds */
> +	int efds[RTE_MAX_RXTX_INTR_VEC_ID];  /**< intr vectors/efds mapping
> */
> +	struct rte_epoll_event *elist[RTE_MAX_RXTX_INTR_VEC_ID];
> +					 /**< intr vector epoll event ptr */
> +	int *intr_vec;                   /**< intr vector number array */
>  };
> 
>  #endif /* _RTE_LINUXAPP_INTERRUPTS_H_ */
> --
> 1.8.1.4