mbox series

[v2,00/15] Add patch set for IPN3KE

Message ID 1553875099-166351-1-git-send-email-rosen.xu@intel.com (mailing list archive)
Headers
Series Add patch set for IPN3KE |

Message

Xu, Rosen March 29, 2019, 3:58 p.m. UTC 
  v2 updates:
==========
 - Fix v1 comments
 - Add support for 10G Base Line Design Bitstream
 - Add support for 25G Base Line Design Bitstream

This patch set adds the support of a new net PMD, Intel® FPGA Programmable
Acceleration Card N3000, also called ipn3ke.

The ipn3ke PMD (librte_pmd_ipn3ke) provides poll mode driver support
for Intel® FPGA PAC(Programmable Acceleration Card) N3000 based on
the Intel Ethernet Controller X710/XXV710 and Intel Arria 10 FPGA.

In this card, FPGA is an acceleration bridge between network interface
and the Intel Ethernet Controller. Although both FPGA and Ethernet
Controllers are connected to CPU with PCIe Gen3x16 Switch, all the
packet RX/TX is handled by Intel Ethernet Controller. So from application
point of view the data path is still the legacy Intel Ethernet Controller
X710/XXV710 PMD. Besides this, users can enable more acceleration
features by FPGA IP.

Rosen Xu (7):
  drivers/bus/ifpga: add AFU shared data
  drivers/bus/ifpga: add function for AFU search by name
  drivers/net/ipn3ke: add IPN3KE ethdev PMD driver
  drivers/net/ipn3ke: add IPN3KE representor of PMD driver
  drivers/net/ipn3ke: add IPN3KE TM of PMD driver
  drivers/net/ipn3ke: add IPN3KE Flow of PMD driver
  drivers/raw/ifpga_rawdev: add IPN3KE support for IFPGA Rawdev

Zhang, Tianfei (8):
  raw/ifpga/base: clean up code for ifpga share code
  raw/ifpga/base: store private features in FME and Port list
  raw/ifpga/base: add SPI and MAX10 device driver
  raw/ifpga/base: add I2C and at24 EEPROM driver
  raw/ifpga/base: add eth group driver
  raw/ifpga/base: add device tree support
  raw/ifpga/base: add version description on README
  raw/ifpga/base: using prefix name "ifpga_" for feature and feature_ops
    data struct

 MAINTAINERS                                        |    7 +
 config/common_base                                 |    4 +
 doc/guides/nics/features/ipn3ke.ini                |   55 +
 doc/guides/nics/index.rst                          |    1 +
 doc/guides/nics/ipn3ke.rst                         |   98 +
 drivers/bus/ifpga/ifpga_bus.c                      |   13 +
 drivers/bus/ifpga/rte_bus_ifpga.h                  |   16 +
 drivers/bus/ifpga/rte_bus_ifpga_version.map        |    1 +
 drivers/net/Makefile                               |    1 +
 drivers/net/ipn3ke/Makefile                        |   39 +
 drivers/net/ipn3ke/ipn3ke_ethdev.c                 |  737 +++++++
 drivers/net/ipn3ke/ipn3ke_ethdev.h                 |  897 +++++++++
 drivers/net/ipn3ke/ipn3ke_flow.c                   | 1400 +++++++++++++
 drivers/net/ipn3ke/ipn3ke_flow.h                   |   86 +
 drivers/net/ipn3ke/ipn3ke_logs.h                   |   30 +
 drivers/net/ipn3ke/ipn3ke_rawdev_api.h             |   62 +
 drivers/net/ipn3ke/ipn3ke_representor.c            |  965 +++++++++
 drivers/net/ipn3ke/ipn3ke_tm.c                     | 2082 ++++++++++++++++++++
 drivers/net/ipn3ke/meson.build                     |    9 +
 drivers/net/ipn3ke/rte_pmd_ipn3ke_version.map      |    4 +
 drivers/net/meson.build                            |    1 +
 drivers/raw/ifpga_rawdev/Makefile                  |    1 +
 drivers/raw/ifpga_rawdev/base/Makefile             |    8 +
 drivers/raw/ifpga_rawdev/base/README               |   15 +
 drivers/raw/ifpga_rawdev/base/ifpga_api.c          |   98 +-
 drivers/raw/ifpga_rawdev/base/ifpga_api.h          |    1 +
 drivers/raw/ifpga_rawdev/base/ifpga_defines.h      |   93 +-
 drivers/raw/ifpga_rawdev/base/ifpga_enumerate.c    |  357 ++--
 drivers/raw/ifpga_rawdev/base/ifpga_feature_dev.c  |  174 +-
 drivers/raw/ifpga_rawdev/base/ifpga_feature_dev.h  |   99 +-
 drivers/raw/ifpga_rawdev/base/ifpga_fme.c          |  602 +++++-
 drivers/raw/ifpga_rawdev/base/ifpga_fme_dperf.c    |   16 +-
 drivers/raw/ifpga_rawdev/base/ifpga_fme_error.c    |   18 +-
 drivers/raw/ifpga_rawdev/base/ifpga_fme_iperf.c    |   28 +-
 drivers/raw/ifpga_rawdev/base/ifpga_fme_pr.c       |    8 +-
 drivers/raw/ifpga_rawdev/base/ifpga_hw.h           |   44 +-
 drivers/raw/ifpga_rawdev/base/ifpga_port.c         |   51 +-
 drivers/raw/ifpga_rawdev/base/ifpga_port_error.c   |   10 +-
 drivers/raw/ifpga_rawdev/base/meson.build          |    8 +-
 drivers/raw/ifpga_rawdev/base/opae_at24_eeprom.c   |   88 +
 drivers/raw/ifpga_rawdev/base/opae_at24_eeprom.h   |   14 +
 drivers/raw/ifpga_rawdev/base/opae_debug.c         |   16 +-
 drivers/raw/ifpga_rawdev/base/opae_eth_group.c     |  145 ++
 drivers/raw/ifpga_rawdev/base/opae_eth_group.h     |   96 +
 drivers/raw/ifpga_rawdev/base/opae_hw_api.c        |  200 +-
 drivers/raw/ifpga_rawdev/base/opae_hw_api.h        |   54 +-
 drivers/raw/ifpga_rawdev/base/opae_i2c.c           |  490 +++++
 drivers/raw/ifpga_rawdev/base/opae_i2c.h           |  130 ++
 drivers/raw/ifpga_rawdev/base/opae_intel_max10.c   |  271 +++
 drivers/raw/ifpga_rawdev/base/opae_intel_max10.h   |  106 +
 drivers/raw/ifpga_rawdev/base/opae_osdep.h         |   21 +-
 drivers/raw/ifpga_rawdev/base/opae_spi.c           |  304 +++
 drivers/raw/ifpga_rawdev/base/opae_spi.h           |  160 ++
 .../raw/ifpga_rawdev/base/opae_spi_transaction.c   |  438 ++++
 .../ifpga_rawdev/base/osdep_raw/osdep_generic.h    |    1 +
 .../ifpga_rawdev/base/osdep_rte/osdep_generic.h    |   16 +
 drivers/raw/ifpga_rawdev/ifpga_rawdev.c            |  205 +-
 drivers/raw/ifpga_rawdev/ifpga_rawdev.h            |   12 +
 drivers/raw/ifpga_rawdev/meson.build               |    6 +-
 mk/rte.app.mk                                      |    3 +-
 usertools/dpdk-devbind.py                          |    4 +-
 61 files changed, 10490 insertions(+), 429 deletions(-)
 create mode 100644 doc/guides/nics/features/ipn3ke.ini
 create mode 100644 doc/guides/nics/ipn3ke.rst
 create mode 100644 drivers/net/ipn3ke/Makefile
 create mode 100644 drivers/net/ipn3ke/ipn3ke_ethdev.c
 create mode 100644 drivers/net/ipn3ke/ipn3ke_ethdev.h
 create mode 100644 drivers/net/ipn3ke/ipn3ke_flow.c
 create mode 100644 drivers/net/ipn3ke/ipn3ke_flow.h
 create mode 100644 drivers/net/ipn3ke/ipn3ke_logs.h
 create mode 100644 drivers/net/ipn3ke/ipn3ke_rawdev_api.h
 create mode 100644 drivers/net/ipn3ke/ipn3ke_representor.c
 create mode 100644 drivers/net/ipn3ke/ipn3ke_tm.c
 create mode 100644 drivers/net/ipn3ke/meson.build
 create mode 100644 drivers/net/ipn3ke/rte_pmd_ipn3ke_version.map
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_at24_eeprom.c
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_at24_eeprom.h
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_eth_group.c
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_eth_group.h
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_i2c.c
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_i2c.h
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_intel_max10.c
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_intel_max10.h
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_spi.c
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_spi.h
 create mode 100644 drivers/raw/ifpga_rawdev/base/opae_spi_transaction.c

Comments

Ferruh Yigit March 29, 2019, 6:59 p.m. UTC | #1 
On 3/29/2019 3:58 PM, Rosen Xu wrote:
> v2 updates:
> ==========
>  - Fix v1 comments
>  - Add support for 10G Base Line Design Bitstream
>  - Add support for 25G Base Line Design Bitstream
> 
> This patch set adds the support of a new net PMD, Intel® FPGA Programmable
> Acceleration Card N3000, also called ipn3ke.
> 
> The ipn3ke PMD (librte_pmd_ipn3ke) provides poll mode driver support
> for Intel® FPGA PAC(Programmable Acceleration Card) N3000 based on
> the Intel Ethernet Controller X710/XXV710 and Intel Arria 10 FPGA.
> 
> In this card, FPGA is an acceleration bridge between network interface
> and the Intel Ethernet Controller. Although both FPGA and Ethernet
> Controllers are connected to CPU with PCIe Gen3x16 Switch, all the
> packet RX/TX is handled by Intel Ethernet Controller. So from application
> point of view the data path is still the legacy Intel Ethernet Controller
> X710/XXV710 PMD. Besides this, users can enable more acceleration
> features by FPGA IP.
> 
> Rosen Xu (7):
>   drivers/bus/ifpga: add AFU shared data
>   drivers/bus/ifpga: add function for AFU search by name
>   drivers/net/ipn3ke: add IPN3KE ethdev PMD driver
>   drivers/net/ipn3ke: add IPN3KE representor of PMD driver
>   drivers/net/ipn3ke: add IPN3KE TM of PMD driver
>   drivers/net/ipn3ke: add IPN3KE Flow of PMD driver
>   drivers/raw/ifpga_rawdev: add IPN3KE support for IFPGA Rawdev
> 
> Zhang, Tianfei (8):
>   raw/ifpga/base: clean up code for ifpga share code
>   raw/ifpga/base: store private features in FME and Port list
>   raw/ifpga/base: add SPI and MAX10 device driver
>   raw/ifpga/base: add I2C and at24 EEPROM driver
>   raw/ifpga/base: add eth group driver
>   raw/ifpga/base: add device tree support
>   raw/ifpga/base: add version description on README
>   raw/ifpga/base: using prefix name "ifpga_" for feature and feature_ops
>     data struct

Hi Rosen,

I am seeing some build errors, for 32bit [1], cross compilation [2] and linkage
errors [3], can you please check them?

Is "libfdt.h" dependency in [2] documented in driver doc?


[1]
Building i686-native-linuxapp-gcc ...
.../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c: In function ‘ipn3ke_indirect_read’:
.../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c:68:19: error: cast to pointer from
integer of different size [-Werror=int-to-pointer-cast]
  indirect_addrs = (volatile void *)(base_addr | 0x10);
                   ^
.../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c:76:19: error: cast to pointer from
integer of different size [-Werror=int-to-pointer-cast]
  indirect_addrs = (volatile void *)(base_addr | 0x18);
                   ^
.../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c: In function ‘ipn3ke_indirect_write’:
.../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c:118:19: error: cast to pointer from
integer of different size [-Werror=int-to-pointer-cast]
  indirect_addrs = (volatile void *)(base_addr | 0x10);
                   ^
cc1: all warnings being treated as errors
make[7]: *** [.../dpdk/mk/internal/rte.compile-pre.mk:116: ipn3ke_ethdev.o] Error 1
make[7]: *** Waiting for unfinished jobs....
In file included from .../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c:25:
.../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c: In function ‘ipn3ke_update_link’:
.../dpdk/drivers/net/ipn3ke/ipn3ke_logs.h:13:49: error: format ‘%lx’ expects
argument of type ‘long unsigned int’, but argument 4 has type ‘uint64_t’ {aka
‘long long unsigned int’} [-Werror=format=]
  rte_log(RTE_LOG_ ## level, ipn3ke_afu_logtype, "ipn3ke_afu: " fmt, \
                                                 ^~~~~~~~~~~~~~
.../dpdk/drivers/net/ipn3ke/ipn3ke_logs.h:19:2: note: in expansion of macro
‘IPN3KE_AFU_PMD_LOG’
  IPN3KE_AFU_PMD_LOG(DEBUG, fmt, ## args)
  ^~~~~~~~~~~~~~~~~~
.../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c:388:2: note: in expansion of
macro ‘IPN3KE_AFU_PMD_DEBUG’
  IPN3KE_AFU_PMD_DEBUG("line_link_bitmap is %lx\n", line_link_bitmap);
  ^~~~~~~~~~~~~~~~~~~~
.../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c:388:46: note: format string is
defined here
  IPN3KE_AFU_PMD_DEBUG("line_link_bitmap is %lx\n", line_link_bitmap);
                                            ~~^
                                            %llx
cc1: all warnings being treated as errors



[2]
.../dpdk/drivers/raw/ifpga_rawdev/base/opae_intel_max10.c:6:10: fatal error:
libfdt.h: No such file or directory
 #include <libfdt.h>
          ^~~~~~~~~~


[3]
/usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_init':
ipn3ke_representor.c:(.text+0xc8b): undefined reference to `pthread_create'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xca8): undefined reference to
`pthread_cancel'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xcbe): undefined reference to
`pthread_join'
/usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_uninit':
ipn3ke_representor.c:(.text+0xd6b): undefined reference to `pthread_create'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xd88): undefined reference to
`pthread_cancel'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xd9e): undefined reference to
`pthread_join'
collect2: error: ld returned 1 exit status
/usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_init':
ipn3ke_representor.c:(.text+0xc8b): undefined reference to `pthread_create'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xca8): undefined reference to
`pthread_cancel'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xcbe): undefined reference to
`pthread_join'
/usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_uninit':
ipn3ke_representor.c:(.text+0xd6b): undefined reference to `pthread_create'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xd88): undefined reference to
`pthread_cancel'
/usr/bin/ld: ipn3ke_representor.c:(.text+0xd9e): undefined reference to
`pthread_join'
collect2: error: ld returned 1 exit status
Xu, Rosen March 31, 2019, 2:19 p.m. UTC | #2 
Hi Ferruh,

My reply is online.

> -----Original Message-----
> From: Yigit, Ferruh
> Sent: Saturday, March 30, 2019 2:59
> To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org
> Cc: Zhang, Tianfei <tianfei.zhang@intel.com>; Wei, Dan
> <dan.wei@intel.com>; Pei, Andy <andy.pei@intel.com>; Yang, Qiming
> <qiming.yang@intel.com>; Wang, Haiyue <haiyue.wang@intel.com>; Chen,
> Santos <santos.chen@intel.com>; Zhang, Zhang <zhang.zhang@intel.com>
> Subject: Re: [PATCH v2 00/15] Add patch set for IPN3KE
> 
> On 3/29/2019 3:58 PM, Rosen Xu wrote:
> > v2 updates:
> > ==========
> >  - Fix v1 comments
> >  - Add support for 10G Base Line Design Bitstream
> >  - Add support for 25G Base Line Design Bitstream
> >
> > This patch set adds the support of a new net PMD, Intel® FPGA
> > Programmable Acceleration Card N3000, also called ipn3ke.
> >
> > The ipn3ke PMD (librte_pmd_ipn3ke) provides poll mode driver support
> > for Intel® FPGA PAC(Programmable Acceleration Card) N3000 based on the
> > Intel Ethernet Controller X710/XXV710 and Intel Arria 10 FPGA.
> >
> > In this card, FPGA is an acceleration bridge between network interface
> > and the Intel Ethernet Controller. Although both FPGA and Ethernet
> > Controllers are connected to CPU with PCIe Gen3x16 Switch, all the
> > packet RX/TX is handled by Intel Ethernet Controller. So from
> > application point of view the data path is still the legacy Intel
> > Ethernet Controller
> > X710/XXV710 PMD. Besides this, users can enable more acceleration
> > features by FPGA IP.
> >
> > Rosen Xu (7):
> >   drivers/bus/ifpga: add AFU shared data
> >   drivers/bus/ifpga: add function for AFU search by name
> >   drivers/net/ipn3ke: add IPN3KE ethdev PMD driver
> >   drivers/net/ipn3ke: add IPN3KE representor of PMD driver
> >   drivers/net/ipn3ke: add IPN3KE TM of PMD driver
> >   drivers/net/ipn3ke: add IPN3KE Flow of PMD driver
> >   drivers/raw/ifpga_rawdev: add IPN3KE support for IFPGA Rawdev
> >
> > Zhang, Tianfei (8):
> >   raw/ifpga/base: clean up code for ifpga share code
> >   raw/ifpga/base: store private features in FME and Port list
> >   raw/ifpga/base: add SPI and MAX10 device driver
> >   raw/ifpga/base: add I2C and at24 EEPROM driver
> >   raw/ifpga/base: add eth group driver
> >   raw/ifpga/base: add device tree support
> >   raw/ifpga/base: add version description on README
> >   raw/ifpga/base: using prefix name "ifpga_" for feature and feature_ops
> >     data struct
> 
> Hi Rosen,
> 
> I am seeing some build errors, for 32bit [1], cross compilation [2] and linkage
> errors [3], can you please check them?

I will fix it ASAP in v3.

> Is "libfdt.h" dependency in [2] documented in driver doc?

OPAE share code has dependency on libfdt, I will add it in v3.
 
> 
> [1]
> Building i686-native-linuxapp-gcc ...
> .../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c: In function
> ‘ipn3ke_indirect_read’:
> .../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c:68:19: error: cast to pointer
> from integer of different size [-Werror=int-to-pointer-cast]
>   indirect_addrs = (volatile void *)(base_addr | 0x10);
>                    ^
> .../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c:76:19: error: cast to pointer
> from integer of different size [-Werror=int-to-pointer-cast]
>   indirect_addrs = (volatile void *)(base_addr | 0x18);
>                    ^
> .../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c: In function
> ‘ipn3ke_indirect_write’:
> .../dpdk/drivers/net/ipn3ke/ipn3ke_ethdev.c:118:19: error: cast to pointer
> from integer of different size [-Werror=int-to-pointer-cast]
>   indirect_addrs = (volatile void *)(base_addr | 0x10);
>                    ^
> cc1: all warnings being treated as errors
> make[7]: *** [.../dpdk/mk/internal/rte.compile-pre.mk:116:
> ipn3ke_ethdev.o] Error 1
> make[7]: *** Waiting for unfinished jobs....
> In file included from .../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c:25:
> .../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c: In function
> ‘ipn3ke_update_link’:
> .../dpdk/drivers/net/ipn3ke/ipn3ke_logs.h:13:49: error: format ‘%lx’ expects
> argument of type ‘long unsigned int’, but argument 4 has type ‘uint64_t’ {aka
> ‘long long unsigned int’} [-Werror=format=]
>   rte_log(RTE_LOG_ ## level, ipn3ke_afu_logtype, "ipn3ke_afu: " fmt, \
>                                                  ^~~~~~~~~~~~~~
> .../dpdk/drivers/net/ipn3ke/ipn3ke_logs.h:19:2: note: in expansion of macro
> ‘IPN3KE_AFU_PMD_LOG’
>   IPN3KE_AFU_PMD_LOG(DEBUG, fmt, ## args)
>   ^~~~~~~~~~~~~~~~~~
> .../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c:388:2: note: in expansion
> of macro ‘IPN3KE_AFU_PMD_DEBUG’
>   IPN3KE_AFU_PMD_DEBUG("line_link_bitmap is %lx\n", line_link_bitmap);
>   ^~~~~~~~~~~~~~~~~~~~
> .../dpdk/drivers/net/ipn3ke/ipn3ke_representor.c:388:46: note: format
> string is defined here
>   IPN3KE_AFU_PMD_DEBUG("line_link_bitmap is %lx\n", line_link_bitmap);
>                                             ~~^
>                                             %llx
> cc1: all warnings being treated as errors
> 
> 
> 
> [2]
> .../dpdk/drivers/raw/ifpga_rawdev/base/opae_intel_max10.c:6:10: fatal
> error:
> libfdt.h: No such file or directory
>  #include <libfdt.h>
>           ^~~~~~~~~~
> 
> 
> [3]
> /usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_init':
> ipn3ke_representor.c:(.text+0xc8b): undefined reference to
> `pthread_create'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xca8): undefined reference to
> `pthread_cancel'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xcbe): undefined reference to
> `pthread_join'
> /usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_uninit':
> ipn3ke_representor.c:(.text+0xd6b): undefined reference to
> `pthread_create'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xd88): undefined reference to
> `pthread_cancel'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xd9e): undefined reference to
> `pthread_join'
> collect2: error: ld returned 1 exit status
> /usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_init':
> ipn3ke_representor.c:(.text+0xc8b): undefined reference to
> `pthread_create'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xca8): undefined reference to
> `pthread_cancel'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xcbe): undefined reference to
> `pthread_join'
> /usr/bin/ld: ipn3ke_representor.o: in function `ipn3ke_rpst_uninit':
> ipn3ke_representor.c:(.text+0xd6b): undefined reference to
> `pthread_create'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xd88): undefined reference to
> `pthread_cancel'
> /usr/bin/ld: ipn3ke_representor.c:(.text+0xd9e): undefined reference to
> `pthread_join'
> collect2: error: ld returned 1 exit status