[v9,00/10] Windows bus/pci support

  From: Tal Shnaiderman <talshn@mellanox.com>

This patchset implements the EAL and PCI functions needed for probing PMDs using RTE_KDRV_NONE on Windows.

---
v9:
	* Split the patch of common OS file to config and options patches (ThomasM).
	* Remove rte_ prefixes from new functions as they are eal internal (ThomasM).
	^ Move common declarations to eal_private.h.
	* Add get_pci_hardware_id and parse_pci_hardware_id in pci.c (DmitryK)
	* Fix issues in pci.c (DmitryK)
v8:
	* Move internal_config to static inside eal_common_config.c and use getters for all external uses.
	* Fix duplicated export line (FadyB)
	* Fix FreeBSD missing function.
v7:
	* Remove Unneeded code from MinGW mapping script (DmitryK)
	* Fix error flow issues in pci.c (DmitryK)
	* Fix Unix build errors following unity of common functions.
	* Move strerror to rte_strerror (DmitryK) 
v6:
	* Fix duplication of exported functions for mingw by modifying the version.map in build (new commit)
	* Fix comments and move additional functions to eal_common_config (DavidM)
	* Fix cross-compilation in mingw (DmitryK)
	* Fix Warnings
	* Move off_t type to pci (ThomasM)
v5:
	* Adjust mem-mapping functions to changes in latest memory management patchset.
	* Fix incorrect implib definition in drivers/meson.build
v4:
	* Fixed various warnings and naming conventions(DmitryK).
	* Fixed broken mingw-64 build(DmitryK).
	* Improved logging(DmitryK).
	* Added patch to fix warnings on rte_pci_addr logging.
	* Fixed broken make on linux/freebsd.
v3:
	* Changes in BDF and hardware ids retrieval(DmitryK).
	* Apply new generic MM to all pci unix callers(DmitryK).
	* Minor corrections in pci.c(DmitryK).
v2:
	* fix style issues.
	* fix error handing flow in pci.c
	* change eal_config.c to eal_common_config.c
---

Tal Shnaiderman (10):
  eal: move OS common config objects
  eal: move OS common options functions
  pci: use OS generic memory mapping functions
  pci: build on Windows
  pci: fix format warning on Windows
  drivers: ignore pmdinfogen generation for Windows
  drivers: fix incorrect meson import folder for Windows
  bus/pci: introduce Windows support with stubs
  bus/pci: support Windows with bifurcated drivers
  build: generate version.map file for MinGW on Windows

 buildtools/{map_to_def.py => map_to_win.py}  |  11 +-
 buildtools/meson.build                       |   4 +-
 drivers/baseband/meson.build                 |   4 +
 drivers/bus/ifpga/meson.build                |   6 +
 drivers/bus/pci/bsd/pci.c                    |   2 +-
 drivers/bus/pci/linux/pci_uio.c              |   2 +-
 drivers/bus/pci/linux/pci_vfio.c             |   9 +-
 drivers/bus/pci/meson.build                  |  14 +-
 drivers/bus/pci/pci_common.c                 |   2 -
 drivers/bus/pci/pci_common_uio.c             |   2 +-
 drivers/bus/pci/windows/pci.c                | 424 +++++++++++++++++++++++++++
 drivers/bus/vdev/meson.build                 |   6 +
 drivers/bus/vmbus/meson.build                |   7 +
 drivers/common/meson.build                   |   4 +
 drivers/compress/meson.build                 |   4 +
 drivers/crypto/meson.build                   |   4 +
 drivers/event/meson.build                    |   4 +
 drivers/mempool/meson.build                  |   4 +
 drivers/meson.build                          |  51 ++--
 drivers/net/meson.build                      |   4 +
 drivers/raw/meson.build                      |   4 +
 drivers/vdpa/meson.build                     |   4 +
 lib/librte_eal/common/eal_common_config.c    |  95 ++++++
 lib/librte_eal/common/eal_common_dynmem.c    |  55 ++--
 lib/librte_eal/common/eal_common_fbarray.c   |   8 +-
 lib/librte_eal/common/eal_common_mcfg.c      |  16 +-
 lib/librte_eal/common/eal_common_memalloc.c  |   4 +-
 lib/librte_eal/common/eal_common_memory.c    |  43 ++-
 lib/librte_eal/common/eal_common_options.c   |  48 ++-
 lib/librte_eal/common/eal_common_proc.c      |  33 ++-
 lib/librte_eal/common/eal_common_thread.c    |  10 +-
 lib/librte_eal/common/eal_internal_cfg.h     |   1 -
 lib/librte_eal/common/eal_private.h          |  39 +++
 lib/librte_eal/common/malloc_elem.c          |  15 +-
 lib/librte_eal/common/malloc_heap.c          |  15 +-
 lib/librte_eal/common/meson.build            |   3 +
 lib/librte_eal/freebsd/Makefile              |   1 +
 lib/librte_eal/freebsd/eal.c                 | 219 ++++++--------
 lib/librte_eal/freebsd/eal_hugepage_info.c   |  32 +-
 lib/librte_eal/freebsd/eal_memory.c          |  34 ++-
 lib/librte_eal/linux/Makefile                |   1 +
 lib/librte_eal/linux/eal.c                   | 254 +++++++---------
 lib/librte_eal/linux/eal_hugepage_info.c     |  50 ++--
 lib/librte_eal/linux/eal_memalloc.c          | 137 ++++++---
 lib/librte_eal/linux/eal_memory.c            | 131 +++++----
 lib/librte_eal/linux/eal_timer.c             |  19 +-
 lib/librte_eal/linux/eal_vfio.c              |  20 +-
 lib/librte_eal/rte_eal_exports.def           |  10 +
 lib/librte_eal/windows/eal.c                 | 118 ++++----
 lib/librte_eal/windows/eal_hugepages.c       |   7 +-
 lib/librte_eal/windows/eal_memalloc.c        |  18 +-
 lib/librte_eal/windows/eal_memory.c          |  11 +-
 lib/librte_eal/windows/eal_mp.c              |  15 +
 lib/librte_eal/windows/include/rte_windows.h |   1 +
 lib/librte_pci/rte_pci.c                     |  19 +-
 lib/librte_pci/rte_pci.h                     |   5 +-
 lib/meson.build                              |  24 +-
 57 files changed, 1456 insertions(+), 631 deletions(-)
 rename buildtools/{map_to_def.py => map_to_win.py} (69%)
 create mode 100644 drivers/bus/pci/windows/pci.c
 create mode 100644 lib/librte_eal/common/eal_common_config.c
  

On Wed, Jun 24, 2020 at 11:28:37AM +0300, talshn@mellanox.com wrote:
> From: Tal Shnaiderman <talshn@mellanox.com>
> 
> This patchset implements the EAL and PCI functions needed for probing PMDs using RTE_KDRV_NONE on Windows.
> 
> ---
> v9:
> 	* Split the patch of common OS file to config and options patches (ThomasM).
> 	* Remove rte_ prefixes from new functions as they are eal internal (ThomasM).
> 	^ Move common declarations to eal_private.h.
> 	* Add get_pci_hardware_id and parse_pci_hardware_id in pci.c (DmitryK)
> 	* Fix issues in pci.c (DmitryK)
> v8:
> 	* Move internal_config to static inside eal_common_config.c and use getters for all external uses.
> 	* Fix duplicated export line (FadyB)
> 	* Fix FreeBSD missing function.
> v7:
> 	* Remove Unneeded code from MinGW mapping script (DmitryK)
> 	* Fix error flow issues in pci.c (DmitryK)
> 	* Fix Unix build errors following unity of common functions.
> 	* Move strerror to rte_strerror (DmitryK) 
> v6:
> 	* Fix duplication of exported functions for mingw by modifying the version.map in build (new commit)
> 	* Fix comments and move additional functions to eal_common_config (DavidM)
> 	* Fix cross-compilation in mingw (DmitryK)
> 	* Fix Warnings
> 	* Move off_t type to pci (ThomasM)
> v5:
> 	* Adjust mem-mapping functions to changes in latest memory management patchset.
> 	* Fix incorrect implib definition in drivers/meson.build
> v4:
> 	* Fixed various warnings and naming conventions(DmitryK).
> 	* Fixed broken mingw-64 build(DmitryK).
> 	* Improved logging(DmitryK).
> 	* Added patch to fix warnings on rte_pci_addr logging.
> 	* Fixed broken make on linux/freebsd.
> v3:
> 	* Changes in BDF and hardware ids retrieval(DmitryK).
> 	* Apply new generic MM to all pci unix callers(DmitryK).
> 	* Minor corrections in pci.c(DmitryK).
> v2:
> 	* fix style issues.
> 	* fix error handing flow in pci.c
> 	* change eal_config.c to eal_common_config.c
> ---
Compiled successfully with clang 9.0.0.
2 warnings are generated, suggesting using safer versions of getenv() and strncpy().  
Do we have any test application to test with? dpdk-helloworld runs successfully.
  

> From: Narcisa Ana Maria Vasile <navasile@linux.microsoft.com>
> Subject: Re: [PATCH v9 00/10] Windows bus/pci support
> 
> On Wed, Jun 24, 2020 at 11:28:37AM +0300, talshn@mellanox.com wrote:
> > From: Tal Shnaiderman <talshn@mellanox.com>
> >
> > This patchset implements the EAL and PCI functions needed for probing
> PMDs using RTE_KDRV_NONE on Windows.
> >
> > ---
> > v9:
> > 	* Split the patch of common OS file to config and options patches
> (ThomasM).
> > 	* Remove rte_ prefixes from new functions as they are eal internal
> (ThomasM).
> > 	^ Move common declarations to eal_private.h.
> > 	* Add get_pci_hardware_id and parse_pci_hardware_id in pci.c
> (DmitryK)
> > 	* Fix issues in pci.c (DmitryK)
> > v8:
> > 	* Move internal_config to static inside eal_common_config.c and use
> getters for all external uses.
> > 	* Fix duplicated export line (FadyB)
> > 	* Fix FreeBSD missing function.
> > v7:
> > 	* Remove Unneeded code from MinGW mapping script (DmitryK)
> > 	* Fix error flow issues in pci.c (DmitryK)
> > 	* Fix Unix build errors following unity of common functions.
> > 	* Move strerror to rte_strerror (DmitryK)
> > v6:
> > 	* Fix duplication of exported functions for mingw by modifying the
> version.map in build (new commit)
> > 	* Fix comments and move additional functions to
> eal_common_config (DavidM)
> > 	* Fix cross-compilation in mingw (DmitryK)
> > 	* Fix Warnings
> > 	* Move off_t type to pci (ThomasM)
> > v5:
> > 	* Adjust mem-mapping functions to changes in latest memory
> management patchset.
> > 	* Fix incorrect implib definition in drivers/meson.build
> > v4:
> > 	* Fixed various warnings and naming conventions(DmitryK).
> > 	* Fixed broken mingw-64 build(DmitryK).
> > 	* Improved logging(DmitryK).
> > 	* Added patch to fix warnings on rte_pci_addr logging.
> > 	* Fixed broken make on linux/freebsd.
> > v3:
> > 	* Changes in BDF and hardware ids retrieval(DmitryK).
> > 	* Apply new generic MM to all pci unix callers(DmitryK).
> > 	* Minor corrections in pci.c(DmitryK).
> > v2:
> > 	* fix style issues.
> > 	* fix error handing flow in pci.c
> > 	* change eal_config.c to eal_common_config.c
> > ---
> Compiled successfully with clang 9.0.0.
> 2 warnings are generated, suggesting using safer versions of getenv() and
> strncpy().

strncpy() warning will go away in v10 after applying the comment from Thomas to modify it to strlcpy.

getenv() is a bit tricky:

a) The function it is being used in (rte_pci_get_sysfs_path) is a UNIX only function, from the usage of the function it seems getenv is safe to use.

b) Changing to code to  getenv_s, _dupenv_s or GetEnvironmentVariable would mean that the user will now need to handle a buffer to store the value of the environment variables.

Do we want to modify it for rte_pci_get_sysfs_path and all of the call sites? Looks like we can ignore this warning in this case.

> Do we have any test application to test with? dpdk-helloworld runs
> successfully.

You can use rte_bus_scan() inside rte_eal_init() to test the scanning process.