[v6,00/11] Add ABI compatibility checks to the meson build

  With the recent changes made to stabilize ABI versioning in DPDK, it will
become increasingly important to check patches for ABI compatibility. We
propose adding the ABI compatibility checking to be done as part of the
build.

The advantages to adding the ABI compatibility checking to the build are
two-fold. Firstly, developers can easily check their patches to make sure
they don’t break the ABI without adding any extra steps. Secondly, it
makes the integration into existing CI seamless since there are no extra
scripts to make the CI run. The build will run as usual and if an
incompatibility is detected in the ABI, the build will fail and show the
incompatibility. As an added bonus, enabling the ABI compatibility checks
does not impact the build speed.

The proposed solution works as follows:
1. Generate the ABI dump of the baseline. This can be done with the new
   script added in this set. This step will only need to be done when the
   ABI version changes (so once a year) and can be added to master so it
   exists by default. This step can be skipped if the dump files for the
   baseline already exist.
2. Build with meson. If there is an ABI incompatibility, the build will
   fail and print the incompatibility information.

The patches in this set include the ABI dump file generating script, the
dump files for drivers and libs, the meson option required to
enable/disable the checks, and the required meson changes to run the
compatibility checks during the build.

Note: This patch set depends on: http://patches.dpdk.org/patch/63765/.  The
generated .dump files in this patch set are based on the changes in the
patch "build: fix soname info for 19.11 compatibility". If a decision is
made to use a different format for the sonames, then a new version of this
patch set will be required as the .dump files will need to be regenerated.

Note: The following driver dump files are not included in these patches:
	common/mvep:    missing dependency, "libmusdk"
        net/mvneta:     missing dependency, "libmusdk"
        net/mvpp2:      missing dependency, "libmusdk"
        net/nfb:        missing dependency, "libnfb"
        crypto/mvsam:   missing dependency, "libmusdk"

They have not been included as I do not have access to these dependencies.
Please feel free to add them if you can! (Maintainers of the above Cc'ed).

---
v2:
  - Rebased on master for 19.11.
  - Moved the experimental syms checks next to the abi checks. This also
    removed the dependency on the experimental checks from the shared
    build.
  - General cleanup.

v3:
  - Fixed typo in meson option name in buildtools/meson.build.

v4:
  - removed commits 1 and 2 from the patch set since they are not needed
    as part of this change
  - squash commits 3,4,5,6 into a single commit
  - changes meson option from type boolean to feature
  - update travis packages
  - move check for windows to earlier in build
  - add abignore file to suppress experimental functions
  - make the directories we store dump files in hidden
  - add .dump files for v20.0 ABI to patch set

v5:
  - added more drivers' dump files
  - removed travis build with abi checks until the issues are resolved.

v6:
  - split up patch with dump files as it was too big for the ML

Kevin Laatz (11):
  lib: add dump files for v20.0 ABI
  drivers/bus: add dump files for v20.0 ABI
  drivers/mempool: add dump files for v20.0 ABI
  drivers/common: add dump files for v20.0 ABI
  drivers/raw: add dump files for v20.0 ABI
  drivers/crypto: add dump files for v20.0 ABI
  drivers/compress: add dump files for v20.0 ABI
  drivers/net: add dump files for v20.0 ABI
  drivers/net/intel: add dump files for v20.0 ABI
  build: add abi checks to meson
  build: clean up experimental syms check

 buildtools/dpdk.abignore                      |    2 +
 buildtools/meson.build                        |    3 +
 config/meson.build                            |   10 +
 devtools/gen-abi-dump.sh                      |   24 +
 drivers/.abi/librte_bus_dpaa.dump             | 5303 ++++++++++++++
 drivers/.abi/librte_bus_fslmc.dump            | 5084 +++++++++++++
 drivers/.abi/librte_bus_ifpga.dump            |  972 +++
 drivers/.abi/librte_bus_pci.dump              |  834 +++
 drivers/.abi/librte_bus_vdev.dump             |  408 ++
 drivers/.abi/librte_bus_vmbus.dump            |  918 +++
 drivers/.abi/librte_common_cpt.dump           |   36 +
 drivers/.abi/librte_common_dpaax.dump         |  177 +
 drivers/.abi/librte_common_octeontx.dump      |   77 +
 drivers/.abi/librte_common_octeontx2.dump     |  988 +++
 drivers/.abi/librte_mempool_bucket.dump       |   15 +
 drivers/.abi/librte_mempool_dpaa.dump         |  323 +
 drivers/.abi/librte_mempool_dpaa2.dump        |  484 ++
 drivers/.abi/librte_mempool_octeontx.dump     |   22 +
 drivers/.abi/librte_mempool_octeontx2.dump    |  643 ++
 drivers/.abi/librte_mempool_ring.dump         |   15 +
 drivers/.abi/librte_mempool_stack.dump        |   16 +
 drivers/.abi/librte_pmd_aesni_gcm.dump        | 1315 ++++
 drivers/.abi/librte_pmd_aesni_mb.dump         | 1315 ++++
 drivers/.abi/librte_pmd_af_packet.dump        |   22 +
 drivers/.abi/librte_pmd_af_xdp.dump           |   23 +
 drivers/.abi/librte_pmd_ark.dump              |   26 +
 drivers/.abi/librte_pmd_atlantic.dump         | 1296 ++++
 drivers/.abi/librte_pmd_avp.dump              |   26 +
 drivers/.abi/librte_pmd_axgbe.dump            |   27 +
 .../.abi/librte_pmd_bbdev_fpga_lte_fec.dump   |   74 +
 drivers/.abi/librte_pmd_bbdev_null.dump       |   17 +
 drivers/.abi/librte_pmd_bbdev_turbo_sw.dump   |   17 +
 drivers/.abi/librte_pmd_bnx2x.dump            |   28 +
 drivers/.abi/librte_pmd_bnxt.dump             | 3499 +++++++++
 drivers/.abi/librte_pmd_bond.dump             | 3586 +++++++++
 drivers/.abi/librte_pmd_caam_jr.dump          |   47 +
 drivers/.abi/librte_pmd_ccp.dump              | 1870 +++++
 drivers/.abi/librte_pmd_crypto_scheduler.dump | 1473 ++++
 drivers/.abi/librte_pmd_cxgbe.dump            |   27 +
 drivers/.abi/librte_pmd_dpaa.dump             | 2929 ++++++++
 drivers/.abi/librte_pmd_dpaa2.dump            | 3319 +++++++++
 drivers/.abi/librte_pmd_dpaa2_event.dump      |   36 +
 drivers/.abi/librte_pmd_dpaa2_sec.dump        | 1511 ++++
 drivers/.abi/librte_pmd_dpaa_event.dump       |   32 +
 drivers/.abi/librte_pmd_dpaa_sec.dump         | 1459 ++++
 drivers/.abi/librte_pmd_dsw_event.dump        |   23 +
 drivers/.abi/librte_pmd_e1000.dump            | 3414 +++++++++
 drivers/.abi/librte_pmd_ena.dump              |   32 +
 drivers/.abi/librte_pmd_enetc.dump            |   27 +
 drivers/.abi/librte_pmd_enic.dump             | 3282 +++++++++
 drivers/.abi/librte_pmd_failsafe.dump         | 3162 ++++++++
 drivers/.abi/librte_pmd_fm10k.dump            |  894 +++
 drivers/.abi/librte_pmd_hinic.dump            | 3110 ++++++++
 drivers/.abi/librte_pmd_hns3.dump             |   28 +
 drivers/.abi/librte_pmd_i40e.dump             | 3553 +++++++++
 drivers/.abi/librte_pmd_iavf.dump             |   71 +
 drivers/.abi/librte_pmd_ice.dump              | 6395 +++++++++++++++++
 drivers/.abi/librte_pmd_ifc.dump              |   24 +
 drivers/.abi/librte_pmd_ipn3ke.dump           | 3581 +++++++++
 drivers/.abi/librte_pmd_isal.dump             | 1038 +++
 drivers/.abi/librte_pmd_ixgbe.dump            | 3369 +++++++++
 drivers/.abi/librte_pmd_kasumi.dump           |   41 +
 drivers/.abi/librte_pmd_kni.dump              |   22 +
 drivers/.abi/librte_pmd_lio.dump              |   11 +
 drivers/.abi/librte_pmd_liquidio.dump         |   27 +
 drivers/.abi/librte_pmd_memif.dump            |   27 +
 drivers/.abi/librte_pmd_mlx4.dump             | 2207 ++++++
 drivers/.abi/librte_pmd_mlx5.dump             | 2155 ++++++
 drivers/.abi/librte_pmd_netvsc.dump           |   28 +
 drivers/.abi/librte_pmd_nfp.dump              |   27 +
 drivers/.abi/librte_pmd_nitrox.dump           |   19 +
 drivers/.abi/librte_pmd_null.dump             |   22 +
 drivers/.abi/librte_pmd_null_crypto.dump      | 1317 ++++
 drivers/.abi/librte_pmd_octeontx.dump         |   49 +
 drivers/.abi/librte_pmd_octeontx2.dump        | 3140 ++++++++
 drivers/.abi/librte_pmd_octeontx2_crypto.dump |   28 +
 drivers/.abi/librte_pmd_octeontx2_event.dump  |   29 +
 .../.abi/librte_pmd_octeontx_compress.dump    |   26 +
 drivers/.abi/librte_pmd_octeontx_crypto.dump  |   28 +
 drivers/.abi/librte_pmd_octeontx_event.dump   |   36 +
 drivers/.abi/librte_pmd_octeontx_ssovf.dump   |   16 +
 drivers/.abi/librte_pmd_octeontx_zip.dump     |   12 +
 drivers/.abi/librte_pmd_opdl_event.dump       |   28 +
 drivers/.abi/librte_pmd_openssl.dump          | 1315 ++++
 drivers/.abi/librte_pmd_pcap.dump             |   23 +
 drivers/.abi/librte_pmd_pfe.dump              |   41 +
 drivers/.abi/librte_pmd_qat.dump              |  119 +
 drivers/.abi/librte_pmd_qede.dump             |   46 +
 drivers/.abi/librte_pmd_ring.dump             |  142 +
 drivers/.abi/librte_pmd_sfc.dump              | 3910 ++++++++++
 drivers/.abi/librte_pmd_sfc_efx.dump          |   12 +
 drivers/.abi/librte_pmd_skeleton_event.dump   |   25 +
 drivers/.abi/librte_pmd_snow3g.dump           | 1424 ++++
 drivers/.abi/librte_pmd_softnic.dump          | 5523 ++++++++++++++
 drivers/.abi/librte_pmd_sw_event.dump         |   28 +
 drivers/.abi/librte_pmd_szedata2.dump         |   28 +
 drivers/.abi/librte_pmd_tap.dump              |   26 +
 drivers/.abi/librte_pmd_thunderx.dump         |   28 +
 drivers/.abi/librte_pmd_thunderx_nicvf.dump   |   12 +
 drivers/.abi/librte_pmd_vdev_netvsc.dump      |   22 +
 drivers/.abi/librte_pmd_vhost.dump            |   57 +
 drivers/.abi/librte_pmd_virtio.dump           | 3808 ++++++++++
 drivers/.abi/librte_pmd_virtio_crypto.dump    | 1419 ++++
 drivers/.abi/librte_pmd_vmxnet3.dump          |   27 +
 drivers/.abi/librte_pmd_vmxnet3_uio.dump      |   11 +
 drivers/.abi/librte_pmd_zlib.dump             | 1035 +++
 drivers/.abi/librte_pmd_zuc.dump              | 1315 ++++
 drivers/.abi/librte_rawdev_dpaa2_cmdif.dump   |   32 +
 drivers/.abi/librte_rawdev_dpaa2_qdma.dump    |  185 +
 drivers/.abi/librte_rawdev_ifpga.dump         |  779 ++
 drivers/.abi/librte_rawdev_ioat.dump          |   19 +
 drivers/.abi/librte_rawdev_ntb.dump           |  771 ++
 drivers/.abi/librte_rawdev_octeontx2_dma.dump |   23 +
 drivers/.abi/librte_rawdev_skeleton.dump      |   21 +
 drivers/meson.build                           |   34 +-
 lib/.abi/librte_acl.dump                      |  329 +
 lib/.abi/librte_bbdev.dump                    | 1789 +++++
 lib/.abi/librte_bitratestats.dump             |   85 +
 lib/.abi/librte_bpf.dump                      |  280 +
 lib/.abi/librte_cfgfile.dump                  |  188 +
 lib/.abi/librte_cmdline.dump                  |  728 ++
 lib/.abi/librte_compressdev.dump              | 1249 ++++
 lib/.abi/librte_cryptodev.dump                | 1806 +++++
 lib/.abi/librte_distributor.dump              |  627 ++
 lib/.abi/librte_eal.dump                      | 3329 +++++++++
 lib/.abi/librte_efd.dump                      |  319 +
 lib/.abi/librte_ethdev.dump                   | 5272 ++++++++++++++
 lib/.abi/librte_eventdev.dump                 | 5519 ++++++++++++++
 lib/.abi/librte_fib.dump                      |  327 +
 lib/.abi/librte_flow_classify.dump            | 1054 +++
 lib/.abi/librte_gro.dump                      |  428 ++
 lib/.abi/librte_gso.dump                      |  402 ++
 lib/.abi/librte_hash.dump                     |  578 ++
 lib/.abi/librte_ip_frag.dump                  |  813 +++
 lib/.abi/librte_ipsec.dump                    | 1269 ++++
 lib/.abi/librte_jobstats.dump                 |  197 +
 lib/.abi/librte_kni.dump                      |  657 ++
 lib/.abi/librte_kvargs.dump                   |   98 +
 lib/.abi/librte_latencystats.dump             |  427 ++
 lib/.abi/librte_lpm.dump                      |  330 +
 lib/.abi/librte_mbuf.dump                     |  745 ++
 lib/.abi/librte_member.dump                   |  216 +
 lib/.abi/librte_mempool.dump                  |  763 ++
 lib/.abi/librte_meter.dump                    |  197 +
 lib/.abi/librte_metrics.dump                  |  102 +
 lib/.abi/librte_net.dump                      |  489 ++
 lib/.abi/librte_pci.dump                      |   85 +
 lib/.abi/librte_pdump.dump                    |  339 +
 lib/.abi/librte_pipeline.dump                 | 1795 +++++
 lib/.abi/librte_port.dump                     |  567 ++
 lib/.abi/librte_power.dump                    |  490 ++
 lib/.abi/librte_rawdev.dump                   |  839 +++
 lib/.abi/librte_rcu.dump                      |  217 +
 lib/.abi/librte_reorder.dump                  |  455 ++
 lib/.abi/librte_rib.dump                      |  556 ++
 lib/.abi/librte_ring.dump                     |  284 +
 lib/.abi/librte_sched.dump                    | 1108 +++
 lib/.abi/librte_security.dump                 | 1125 +++
 lib/.abi/librte_stack.dump                    |  183 +
 lib/.abi/librte_table.dump                    |  544 ++
 lib/.abi/librte_telemetry.dump                |  122 +
 lib/.abi/librte_timer.dump                    |  321 +
 lib/.abi/librte_vhost.dump                    | 1629 +++++
 lib/meson.build                               |   34 +-
 meson.build                                   |    4 +
 meson_options.txt                             |    2 +
 166 files changed, 145177 insertions(+), 20 deletions(-)
 create mode 100644 buildtools/dpdk.abignore
 create mode 100755 devtools/gen-abi-dump.sh
 create mode 100644 drivers/.abi/librte_bus_dpaa.dump
 create mode 100644 drivers/.abi/librte_bus_fslmc.dump
 create mode 100644 drivers/.abi/librte_bus_ifpga.dump
 create mode 100644 drivers/.abi/librte_bus_pci.dump
 create mode 100644 drivers/.abi/librte_bus_vdev.dump
 create mode 100644 drivers/.abi/librte_bus_vmbus.dump
 create mode 100644 drivers/.abi/librte_common_cpt.dump
 create mode 100644 drivers/.abi/librte_common_dpaax.dump
 create mode 100644 drivers/.abi/librte_common_octeontx.dump
 create mode 100644 drivers/.abi/librte_common_octeontx2.dump
 create mode 100644 drivers/.abi/librte_mempool_bucket.dump
 create mode 100644 drivers/.abi/librte_mempool_dpaa.dump
 create mode 100644 drivers/.abi/librte_mempool_dpaa2.dump
 create mode 100644 drivers/.abi/librte_mempool_octeontx.dump
 create mode 100644 drivers/.abi/librte_mempool_octeontx2.dump
 create mode 100644 drivers/.abi/librte_mempool_ring.dump
 create mode 100644 drivers/.abi/librte_mempool_stack.dump
 create mode 100644 drivers/.abi/librte_pmd_aesni_gcm.dump
 create mode 100644 drivers/.abi/librte_pmd_aesni_mb.dump
 create mode 100644 drivers/.abi/librte_pmd_af_packet.dump
 create mode 100644 drivers/.abi/librte_pmd_af_xdp.dump
 create mode 100644 drivers/.abi/librte_pmd_ark.dump
 create mode 100644 drivers/.abi/librte_pmd_atlantic.dump
 create mode 100644 drivers/.abi/librte_pmd_avp.dump
 create mode 100644 drivers/.abi/librte_pmd_axgbe.dump
 create mode 100644 drivers/.abi/librte_pmd_bbdev_fpga_lte_fec.dump
 create mode 100644 drivers/.abi/librte_pmd_bbdev_null.dump
 create mode 100644 drivers/.abi/librte_pmd_bbdev_turbo_sw.dump
 create mode 100644 drivers/.abi/librte_pmd_bnx2x.dump
 create mode 100644 drivers/.abi/librte_pmd_bnxt.dump
 create mode 100644 drivers/.abi/librte_pmd_bond.dump
 create mode 100644 drivers/.abi/librte_pmd_caam_jr.dump
 create mode 100644 drivers/.abi/librte_pmd_ccp.dump
 create mode 100644 drivers/.abi/librte_pmd_crypto_scheduler.dump
 create mode 100644 drivers/.abi/librte_pmd_cxgbe.dump
 create mode 100644 drivers/.abi/librte_pmd_dpaa.dump
 create mode 100644 drivers/.abi/librte_pmd_dpaa2.dump
 create mode 100644 drivers/.abi/librte_pmd_dpaa2_event.dump
 create mode 100644 drivers/.abi/librte_pmd_dpaa2_sec.dump
 create mode 100644 drivers/.abi/librte_pmd_dpaa_event.dump
 create mode 100644 drivers/.abi/librte_pmd_dpaa_sec.dump
 create mode 100644 drivers/.abi/librte_pmd_dsw_event.dump
 create mode 100644 drivers/.abi/librte_pmd_e1000.dump
 create mode 100644 drivers/.abi/librte_pmd_ena.dump
 create mode 100644 drivers/.abi/librte_pmd_enetc.dump
 create mode 100644 drivers/.abi/librte_pmd_enic.dump
 create mode 100644 drivers/.abi/librte_pmd_failsafe.dump
 create mode 100644 drivers/.abi/librte_pmd_fm10k.dump
 create mode 100644 drivers/.abi/librte_pmd_hinic.dump
 create mode 100644 drivers/.abi/librte_pmd_hns3.dump
 create mode 100644 drivers/.abi/librte_pmd_i40e.dump
 create mode 100644 drivers/.abi/librte_pmd_iavf.dump
 create mode 100644 drivers/.abi/librte_pmd_ice.dump
 create mode 100644 drivers/.abi/librte_pmd_ifc.dump
 create mode 100644 drivers/.abi/librte_pmd_ipn3ke.dump
 create mode 100644 drivers/.abi/librte_pmd_isal.dump
 create mode 100644 drivers/.abi/librte_pmd_ixgbe.dump
 create mode 100644 drivers/.abi/librte_pmd_kasumi.dump
 create mode 100644 drivers/.abi/librte_pmd_kni.dump
 create mode 100644 drivers/.abi/librte_pmd_lio.dump
 create mode 100644 drivers/.abi/librte_pmd_liquidio.dump
 create mode 100644 drivers/.abi/librte_pmd_memif.dump
 create mode 100644 drivers/.abi/librte_pmd_mlx4.dump
 create mode 100644 drivers/.abi/librte_pmd_mlx5.dump
 create mode 100644 drivers/.abi/librte_pmd_netvsc.dump
 create mode 100644 drivers/.abi/librte_pmd_nfp.dump
 create mode 100644 drivers/.abi/librte_pmd_nitrox.dump
 create mode 100644 drivers/.abi/librte_pmd_null.dump
 create mode 100644 drivers/.abi/librte_pmd_null_crypto.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx2.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx2_crypto.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx2_event.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx_compress.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx_crypto.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx_event.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx_ssovf.dump
 create mode 100644 drivers/.abi/librte_pmd_octeontx_zip.dump
 create mode 100644 drivers/.abi/librte_pmd_opdl_event.dump
 create mode 100644 drivers/.abi/librte_pmd_openssl.dump
 create mode 100644 drivers/.abi/librte_pmd_pcap.dump
 create mode 100644 drivers/.abi/librte_pmd_pfe.dump
 create mode 100644 drivers/.abi/librte_pmd_qat.dump
 create mode 100644 drivers/.abi/librte_pmd_qede.dump
 create mode 100644 drivers/.abi/librte_pmd_ring.dump
 create mode 100644 drivers/.abi/librte_pmd_sfc.dump
 create mode 100644 drivers/.abi/librte_pmd_sfc_efx.dump
 create mode 100644 drivers/.abi/librte_pmd_skeleton_event.dump
 create mode 100644 drivers/.abi/librte_pmd_snow3g.dump
 create mode 100644 drivers/.abi/librte_pmd_softnic.dump
 create mode 100644 drivers/.abi/librte_pmd_sw_event.dump
 create mode 100644 drivers/.abi/librte_pmd_szedata2.dump
 create mode 100644 drivers/.abi/librte_pmd_tap.dump
 create mode 100644 drivers/.abi/librte_pmd_thunderx.dump
 create mode 100644 drivers/.abi/librte_pmd_thunderx_nicvf.dump
 create mode 100644 drivers/.abi/librte_pmd_vdev_netvsc.dump
 create mode 100644 drivers/.abi/librte_pmd_vhost.dump
 create mode 100644 drivers/.abi/librte_pmd_virtio.dump
 create mode 100644 drivers/.abi/librte_pmd_virtio_crypto.dump
 create mode 100644 drivers/.abi/librte_pmd_vmxnet3.dump
 create mode 100644 drivers/.abi/librte_pmd_vmxnet3_uio.dump
 create mode 100644 drivers/.abi/librte_pmd_zlib.dump
 create mode 100644 drivers/.abi/librte_pmd_zuc.dump
 create mode 100644 drivers/.abi/librte_rawdev_dpaa2_cmdif.dump
 create mode 100644 drivers/.abi/librte_rawdev_dpaa2_qdma.dump
 create mode 100644 drivers/.abi/librte_rawdev_ifpga.dump
 create mode 100644 drivers/.abi/librte_rawdev_ioat.dump
 create mode 100644 drivers/.abi/librte_rawdev_ntb.dump
 create mode 100644 drivers/.abi/librte_rawdev_octeontx2_dma.dump
 create mode 100644 drivers/.abi/librte_rawdev_skeleton.dump
 create mode 100644 lib/.abi/librte_acl.dump
 create mode 100644 lib/.abi/librte_bbdev.dump
 create mode 100644 lib/.abi/librte_bitratestats.dump
 create mode 100644 lib/.abi/librte_bpf.dump
 create mode 100644 lib/.abi/librte_cfgfile.dump
 create mode 100644 lib/.abi/librte_cmdline.dump
 create mode 100644 lib/.abi/librte_compressdev.dump
 create mode 100644 lib/.abi/librte_cryptodev.dump
 create mode 100644 lib/.abi/librte_distributor.dump
 create mode 100644 lib/.abi/librte_eal.dump
 create mode 100644 lib/.abi/librte_efd.dump
 create mode 100644 lib/.abi/librte_ethdev.dump
 create mode 100644 lib/.abi/librte_eventdev.dump
 create mode 100644 lib/.abi/librte_fib.dump
 create mode 100644 lib/.abi/librte_flow_classify.dump
 create mode 100644 lib/.abi/librte_gro.dump
 create mode 100644 lib/.abi/librte_gso.dump
 create mode 100644 lib/.abi/librte_hash.dump
 create mode 100644 lib/.abi/librte_ip_frag.dump
 create mode 100644 lib/.abi/librte_ipsec.dump
 create mode 100644 lib/.abi/librte_jobstats.dump
 create mode 100644 lib/.abi/librte_kni.dump
 create mode 100644 lib/.abi/librte_kvargs.dump
 create mode 100644 lib/.abi/librte_latencystats.dump
 create mode 100644 lib/.abi/librte_lpm.dump
 create mode 100644 lib/.abi/librte_mbuf.dump
 create mode 100644 lib/.abi/librte_member.dump
 create mode 100644 lib/.abi/librte_mempool.dump
 create mode 100644 lib/.abi/librte_meter.dump
 create mode 100644 lib/.abi/librte_metrics.dump
 create mode 100644 lib/.abi/librte_net.dump
 create mode 100644 lib/.abi/librte_pci.dump
 create mode 100644 lib/.abi/librte_pdump.dump
 create mode 100644 lib/.abi/librte_pipeline.dump
 create mode 100644 lib/.abi/librte_port.dump
 create mode 100644 lib/.abi/librte_power.dump
 create mode 100644 lib/.abi/librte_rawdev.dump
 create mode 100644 lib/.abi/librte_rcu.dump
 create mode 100644 lib/.abi/librte_reorder.dump
 create mode 100644 lib/.abi/librte_rib.dump
 create mode 100644 lib/.abi/librte_ring.dump
 create mode 100644 lib/.abi/librte_sched.dump
 create mode 100644 lib/.abi/librte_security.dump
 create mode 100644 lib/.abi/librte_stack.dump
 create mode 100644 lib/.abi/librte_table.dump
 create mode 100644 lib/.abi/librte_telemetry.dump
 create mode 100644 lib/.abi/librte_timer.dump
 create mode 100644 lib/.abi/librte_vhost.dump
  

Hello Kevin,

On Fri, Dec 13, 2019 at 5:41 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
>
> With the recent changes made to stabilize ABI versioning in DPDK, it will
> become increasingly important to check patches for ABI compatibility. We
> propose adding the ABI compatibility checking to be done as part of the
> build.
>
> The advantages to adding the ABI compatibility checking to the build are
> two-fold. Firstly, developers can easily check their patches to make sure
> they don’t break the ABI without adding any extra steps. Secondly, it
> makes the integration into existing CI seamless since there are no extra
> scripts to make the CI run. The build will run as usual and if an
> incompatibility is detected in the ABI, the build will fail and show the
> incompatibility. As an added bonus, enabling the ABI compatibility checks
> does not impact the build speed.
>
> The proposed solution works as follows:
> 1. Generate the ABI dump of the baseline. This can be done with the new
>    script added in this set. This step will only need to be done when the
>    ABI version changes (so once a year) and can be added to master so it
>    exists by default. This step can be skipped if the dump files for the
>    baseline already exist.
> 2. Build with meson. If there is an ABI incompatibility, the build will
>    fail and print the incompatibility information.
>
> The patches in this set include the ABI dump file generating script, the
> dump files for drivers and libs, the meson option required to
> enable/disable the checks, and the required meson changes to run the
> compatibility checks during the build.
>
> Note: This patch set depends on: http://patches.dpdk.org/patch/63765/.  The
> generated .dump files in this patch set are based on the changes in the
> patch "build: fix soname info for 19.11 compatibility". If a decision is
> made to use a different format for the sonames, then a new version of this
> patch set will be required as the .dump files will need to be regenerated.
>
> Note: The following driver dump files are not included in these patches:
>         common/mvep:    missing dependency, "libmusdk"
>         net/mvneta:     missing dependency, "libmusdk"
>         net/mvpp2:      missing dependency, "libmusdk"
>         net/nfb:        missing dependency, "libnfb"
>         crypto/mvsam:   missing dependency, "libmusdk"
>
> They have not been included as I do not have access to these dependencies.
> Please feel free to add them if you can! (Maintainers of the above Cc'ed).

- I asked for the dump files, but I can see that it is impractical.

The dump files are huge. I did not expect that :-).
The dump files are architecture specific and maintaining multi arch
dumps would be even bigger than just what you sent for x86_64.
(not even considering the changes in ABI if some configuration items
have an impact...).

As you pointed out, people who don't have all dependencies won't
create/update those dump files.
Dealing with ABI updates (thinking of bumping the ABI version) is
likely a maintainer job, but it will be a source of issues and we
(maintainers) might miss some updates especially for drivers we can't
build.


- Why do we restrict the checks to meson?
The make build framework is going to disappear ok, but we can't leave
it untested.
People still rely on it.

Checking the ABI is orthogonal to building DPDK.
Dumping the ABI and checking it against objects can be done externally.


- For those reasons, I have been trying an alternate approach [1]: in
Travis, generate a reference dump based on the first ancestor tag,
then build the proposed patches.
All contributions get picked up by Aaron robot and would have to pass
through this check.

As an exercise, I tried to integrate Eelco patch [2], that moves
symbols from EXPERIMENTAL to a stable ABI. The check passes fine.
I also tried to bump the ABI major version. The check fails, as
expected, but I prepared a way to bypass such failures for the
releases where we will explicitely break ABI.

For maintainers that integrate patches or developers that get a CI
failure and want to fix it, we would need to help them to:
* generate dumps on a reference version, so I would tend to write some
documentation since playing with the current sources would be too
dangerous from my pov,
* run the checks, like adding the check in the
devtools/test-*-build.sh scripts that already exist, with a new
configuration item to point at the dumps per target,

Those last two points are still to be done.

WDYT?


1: https://github.com/david-marchand/dpdk/commit/f18de2ec157f3cc1e7b319cb19700e1b5e9cecde
2: https://patchwork.dpdk.org/patch/63970/

--
David Marchand
  

19/12/2019 22:58, David Marchand:
> On Fri, Dec 13, 2019 at 5:41 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
> > The proposed solution works as follows:
> > 1. Generate the ABI dump of the baseline. This can be done with the new
> >    script added in this set. This step will only need to be done when the
> >    ABI version changes (so once a year) and can be added to master so it
> >    exists by default. This step can be skipped if the dump files for the
> >    baseline already exist.
> > 2. Build with meson. If there is an ABI incompatibility, the build will
> >    fail and print the incompatibility information.
[...]
> - I asked for the dump files, but I can see that it is impractical.
> 
> The dump files are huge. I did not expect that :-).
> The dump files are architecture specific and maintaining multi arch
> dumps would be even bigger than just what you sent for x86_64.
> (not even considering the changes in ABI if some configuration items
> have an impact...).

I agree that dump files are better managed outside of git.


> As you pointed out, people who don't have all dependencies won't
> create/update those dump files.
> Dealing with ABI updates (thinking of bumping the ABI version) is
> likely a maintainer job, but it will be a source of issues and we
> (maintainers) might miss some updates especially for drivers we can't
> build.
> 
> 
> - Why do we restrict the checks to meson?
> The make build framework is going to disappear ok, but we can't leave
> it untested.
> People still rely on it.
> 
> Checking the ABI is orthogonal to building DPDK.
> Dumping the ABI and checking it against objects can be done externally.

I agree we should not rely on meson for running the check.


> - For those reasons, I have been trying an alternate approach [1]: in
> Travis, generate a reference dump based on the first ancestor tag,
> then build the proposed patches.
> All contributions get picked up by Aaron robot and would have to pass
> through this check.
> 
> As an exercise, I tried to integrate Eelco patch [2], that moves
> symbols from EXPERIMENTAL to a stable ABI. The check passes fine.
> I also tried to bump the ABI major version. The check fails, as
> expected, but I prepared a way to bypass such failures for the
> releases where we will explicitely break ABI.
> 
> For maintainers that integrate patches or developers that get a CI
> failure and want to fix it, we would need to help them to:
> * generate dumps on a reference version, so I would tend to write some
> documentation since playing with the current sources would be too
> dangerous from my pov,
> * run the checks, like adding the check in the
> devtools/test-*-build.sh scripts that already exist, with a new
> configuration item to point at the dumps per target,

We can start with a documented process,
and write a separate script later if we feel it helps.


> Those last two points are still to be done.
> 
> WDYT?
> 
> 
> 1: https://github.com/david-marchand/dpdk/commit/f18de2ec157f3cc1e7b319cb19700e1b5e9cecde
> 2: https://patchwork.dpdk.org/patch/63970/


Thanks to both of you trying some approaches and making progress.
  

On Thu, Dec 19, 2019 at 10:58:35PM +0100, David Marchand wrote:
> Hello Kevin,
> 
> On Fri, Dec 13, 2019 at 5:41 PM Kevin Laatz <kevin.laatz@intel.com> wrote:
> >
> > With the recent changes made to stabilize ABI versioning in DPDK, it will
> > become increasingly important to check patches for ABI compatibility. We
> > propose adding the ABI compatibility checking to be done as part of the
> > build.
> >
> > The advantages to adding the ABI compatibility checking to the build are
> > two-fold. Firstly, developers can easily check their patches to make sure
> > they don’t break the ABI without adding any extra steps. Secondly, it
> > makes the integration into existing CI seamless since there are no extra
> > scripts to make the CI run. The build will run as usual and if an
> > incompatibility is detected in the ABI, the build will fail and show the
> > incompatibility. As an added bonus, enabling the ABI compatibility checks
> > does not impact the build speed.
> >
> > The proposed solution works as follows:
> > 1. Generate the ABI dump of the baseline. This can be done with the new
> >    script added in this set. This step will only need to be done when the
> >    ABI version changes (so once a year) and can be added to master so it
> >    exists by default. This step can be skipped if the dump files for the
> >    baseline already exist.
> > 2. Build with meson. If there is an ABI incompatibility, the build will
> >    fail and print the incompatibility information.
> >
> > The patches in this set include the ABI dump file generating script, the
> > dump files for drivers and libs, the meson option required to
> > enable/disable the checks, and the required meson changes to run the
> > compatibility checks during the build.
> >
> > Note: This patch set depends on: http://patches.dpdk.org/patch/63765/.  The
> > generated .dump files in this patch set are based on the changes in the
> > patch "build: fix soname info for 19.11 compatibility". If a decision is
> > made to use a different format for the sonames, then a new version of this
> > patch set will be required as the .dump files will need to be regenerated.
> >
> > Note: The following driver dump files are not included in these patches:
> >         common/mvep:    missing dependency, "libmusdk"
> >         net/mvneta:     missing dependency, "libmusdk"
> >         net/mvpp2:      missing dependency, "libmusdk"
> >         net/nfb:        missing dependency, "libnfb"
> >         crypto/mvsam:   missing dependency, "libmusdk"
> >
> > They have not been included as I do not have access to these dependencies.
> > Please feel free to add them if you can! (Maintainers of the above Cc'ed).
> 
> - I asked for the dump files, but I can see that it is impractical.
> 
> The dump files are huge. I did not expect that :-).

Yes, they are big alright, but on the other hand, they also don't change
very much (we hope!)

> The dump files are architecture specific and maintaining multi arch
> dumps would be even bigger than just what you sent for x86_64.
> (not even considering the changes in ABI if some configuration items
> have an impact...).

Good point, we missed that when looking at this.

> 
> As you pointed out, people who don't have all dependencies won't
> create/update those dump files.

Well, the creation should be a once-off, the comparison is what is done
regularly and needs the build tools.

> Dealing with ABI updates (thinking of bumping the ABI version) is
> likely a maintainer job, but it will be a source of issues and we
> (maintainers) might miss some updates especially for drivers we can't
> build.
> 
> 
> - Why do we restrict the checks to meson?
> The make build framework is going to disappear ok, but we can't leave
> it untested.
> People still rely on it.
> 
Because as you point out below, checking the ABI is technically orthogonal
to building the DPDK, so we didn't see the payoff in adding support to two
build systems as being worth the additional effort.

> Checking the ABI is orthogonal to building DPDK.
> Dumping the ABI and checking it against objects can be done externally.
> 

True, but the advantage of doing so as part of each and every build is that
any ABI break is caught by the original developer before he even submits
his patch to the CI. As with so many things, the earlier in the process
that something can be run the better it is.

> 
> - For those reasons, I have been trying an alternate approach [1]: in
> Travis, generate a reference dump based on the first ancestor tag,
> then build the proposed patches.
> All contributions get picked up by Aaron robot and would have to pass
> through this check.

Yes, the alternative to having the checks done at build time is to have
them done as part of the CI, though I'd personally perfer the former.

> 
> As an exercise, I tried to integrate Eelco patch [2], that moves
> symbols from EXPERIMENTAL to a stable ABI. The check passes fine.
> I also tried to bump the ABI major version. The check fails, as
> expected, but I prepared a way to bypass such failures for the
> releases where we will explicitely break ABI.
> 

IMHO: we should not bypass such failures, but instead update our reference
ABI dumps. This is one reason why having the ABI spec in the git repo would
work well, any patches that change ABI would also include the updates to
the dump files.

> For maintainers that integrate patches or developers that get a CI
> failure and want to fix it, we would need to help them to:
> * generate dumps on a reference version, so I would tend to write some
> documentation since playing with the current sources would be too
> dangerous from my pov,

This should be a one-off reference dump archived somewhere. Each maintainer
should not have his own copy, I think.

> * run the checks, like adding the check in the
> devtools/test-*-build.sh scripts that already exist, with a new
> configuration item to point at the dumps per target,
> 

Where do we store the dumps then? Do they get stored in a separate git
repo?

> Those last two points are still to be done.
> 
> WDYT?
> 
Makes sense. I still prefer a solution where we keep the offical ABI in git
alongside the source code, but I realise that doing so for multiple
archtectures is likely to get to be very big. However, since these are
plain text files though, they should compress well when stored in the git
repo itself or when being pushed/pulled. [And again, the deltas for these
should be pretty tiny, even when we do update the ABI].

/Bruce
  

On Fri, Dec 20, 2019 at 12:04 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
> > For maintainers that integrate patches or developers that get a CI
> > failure and want to fix it, we would need to help them to:
> > * generate dumps on a reference version, so I would tend to write some
> > documentation since playing with the current sources would be too
> > dangerous from my pov,
>
> This should be a one-off reference dump archived somewhere. Each maintainer
> should not have his own copy, I think.

This is not a one-off thing.
We maintain ABI for some time (1/2 year(s)), and we expect to bump ABI.
When doing this, in size, the diff will be at least the same than what
we have here.


If you disable libraries, features etc... you get a new ABI.
What would be the reference*s* then?
Builds with default options from meson for each architecture?


> > * run the checks, like adding the check in the
> > devtools/test-*-build.sh scripts that already exist, with a new
> > configuration item to point at the dumps per target,
> >
>
> Where do we store the dumps then? Do they get stored in a separate git
> repo?

Creating a separate git repo is just adding more pain to submitters
(/maintainers): they would have to submit (/apply) patches against two
trees.


--
David Marchand
  

On Fri, Dec 20, 2019 at 02:19:13PM +0100, David Marchand wrote:
> On Fri, Dec 20, 2019 at 12:04 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> > > For maintainers that integrate patches or developers that get a CI
> > > failure and want to fix it, we would need to help them to:
> > > * generate dumps on a reference version, so I would tend to write some
> > > documentation since playing with the current sources would be too
> > > dangerous from my pov,
> >
> > This should be a one-off reference dump archived somewhere. Each maintainer
> > should not have his own copy, I think.
> 
> This is not a one-off thing.
> We maintain ABI for some time (1/2 year(s)), and we expect to bump ABI.
> When doing this, in size, the diff will be at least the same than what
> we have here.
> 

I don't think it will be quite that big, but ok, it may be significant, so
I will concede that these are too big to store in the main git repo.

> 
> If you disable libraries, features etc... you get a new ABI.
> What would be the reference*s* then?
> Builds with default options from meson for each architecture?
> 

On the project level API, yes, removing libraries/drivers does affect
things. However, the specific checks are done on the individual .so level,
so having some drivers off in the build should not be a problem. Even with
features - all public functions need to correspond with map file entries,
so we can't conditionally remove them without providing a stub AFAIK.
Therefore, having one master reference of the DPDK_20 ABI is perfectly
feasible.

> 
> > > * run the checks, like adding the check in the
> > > devtools/test-*-build.sh scripts that already exist, with a new
> > > configuration item to point at the dumps per target,
> > >
> >
> > Where do we store the dumps then? Do they get stored in a separate git
> > repo?
> 
> Creating a separate git repo is just adding more pain to submitters
> (/maintainers): they would have to submit (/apply) patches against two
> trees.
> 

Well, the official ABI dumps need to be stored somewhere, because if it's
an official ABI, then it is exactly that. There cannot be different people
with different versions of the ABI to check against. Everyone should check
against the one common, official reference.

Regards,
/Bruce
  

Bruce Richardson <bruce.richardson@intel.com> writes:

> On Fri, Dec 20, 2019 at 02:19:13PM +0100, David Marchand wrote:
>> On Fri, Dec 20, 2019 at 12:04 PM Bruce Richardson
>> <bruce.richardson@intel.com> wrote:
>> > > For maintainers that integrate patches or developers that get a CI
>> > > failure and want to fix it, we would need to help them to:
>> > > * generate dumps on a reference version, so I would tend to write some
>> > > documentation since playing with the current sources would be too
>> > > dangerous from my pov,
>> >
>> > This should be a one-off reference dump archived somewhere. Each maintainer
>> > should not have his own copy, I think.
>> 
>> This is not a one-off thing.
>> We maintain ABI for some time (1/2 year(s)), and we expect to bump ABI.
>> When doing this, in size, the diff will be at least the same than what
>> we have here.
>> 
>
> I don't think it will be quite that big, but ok, it may be significant, so
> I will concede that these are too big to store in the main git repo.
>
>> 
>> If you disable libraries, features etc... you get a new ABI.
>> What would be the reference*s* then?
>> Builds with default options from meson for each architecture?
>> 
>
> On the project level API, yes, removing libraries/drivers does affect
> things. However, the specific checks are done on the individual .so level,
> so having some drivers off in the build should not be a problem. Even with
> features - all public functions need to correspond with map file entries,
> so we can't conditionally remove them without providing a stub AFAIK.
> Therefore, having one master reference of the DPDK_20 ABI is perfectly
> feasible.

But when is it cut?  What happens during an interrim, where users have
an outdated reference but don't remember what they downloaded?  What
about when we have multiple LTS' that have different ABI versions and
need to ensure that we don't break ABIs for backports?

>> 
>> > > * run the checks, like adding the check in the
>> > > devtools/test-*-build.sh scripts that already exist, with a new
>> > > configuration item to point at the dumps per target,
>> > >
>> >
>> > Where do we store the dumps then? Do they get stored in a separate git
>> > repo?
>> 
>> Creating a separate git repo is just adding more pain to submitters
>> (/maintainers): they would have to submit (/apply) patches against two
>> trees.
>> 
>
> Well, the official ABI dumps need to be stored somewhere, because if it's
> an official ABI, then it is exactly that. There cannot be different people
> with different versions of the ABI to check against. Everyone should check
> against the one common, official reference.

Isn't the reference stored in the git repo anyway?  It should always be
possible to generate it.  And I trust the version I generate from the
source of truth anyway, over something I download.

> Regards,
> /Bruce