mbox series

[v10,0/3] pmdinfogen: rewrite in Python

Message ID 20210124205157.14386-1-dmitry.kozliuk@gmail.com (mailing list archive)
Headers
Series pmdinfogen: rewrite in Python |

Message

Dmitry Kozlyuk Jan. 24, 2021, 8:51 p.m. UTC 
  This patchset implements existing pmdinfogen logic in Python, replaces
and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Identity of generated PMD information is checked by comparing
output of pmdinfo before and after the patch:

    find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

Acked-by: Neil Horman <nhorman@tuxdriver.com>
Tested-by: Jie Zhou <jizh@linux.microsoft.com>

---
Changes in v10:

    * Suppress ABI warnings for generated strings (Thomas).

Dmitry Kozlyuk (3):
  pmdinfogen: add Python implementation
  build: use Python pmdinfogen
  pmdinfogen: remove C implementation

 .github/workflows/build.yml           |   4 +-
 .travis.yml                           |   2 +-
 MAINTAINERS                           |   3 +-
 buildtools/gen-pmdinfo-cfile.sh       |   6 +-
 buildtools/meson.build                |  15 +
 buildtools/pmdinfogen.py              | 189 +++++++++++
 buildtools/pmdinfogen/meson.build     |  14 -
 buildtools/pmdinfogen/pmdinfogen.c    | 456 --------------------------
 buildtools/pmdinfogen/pmdinfogen.h    | 119 -------
 devtools/libabigail.abignore          |   4 +
 doc/guides/freebsd_gsg/build_dpdk.rst |   3 +-
 doc/guides/linux_gsg/sys_reqs.rst     |   6 +
 drivers/meson.build                   |   2 +-
 meson.build                           |   1 -
 14 files changed, 225 insertions(+), 599 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

Comments

Thomas Monjalon Jan. 25, 2021, 1:13 p.m. UTC | #1 
> Dmitry Kozlyuk (3):
>   pmdinfogen: add Python implementation
>   build: use Python pmdinfogen
>   pmdinfogen: remove C implementation

Applied, thanks.

Reminder: it is adding a NEW DEPENDENCY on pyelftools,
as highlighted already in CI and maintainers meetings:
	http://mails.dpdk.org/archives/ci/2021-January/000939.html
	http://mails.dpdk.org/archives/dev/2021-January/197814.html

If not already done, please UPDATE your environment! Thanks
Brandon Lo Jan. 25, 2021, 4:08 p.m. UTC | #2 
Hi Thomas,

We have updated all UNH-IOL environments to include pyelftools.

Thanks,
Brandon

On Mon, Jan 25, 2021 at 8:13 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > Dmitry Kozlyuk (3):
> >   pmdinfogen: add Python implementation
> >   build: use Python pmdinfogen
> >   pmdinfogen: remove C implementation
>
> Applied, thanks.
>
> Reminder: it is adding a NEW DEPENDENCY on pyelftools,
> as highlighted already in CI and maintainers meetings:
>         http://mails.dpdk.org/archives/ci/2021-January/000939.html
>         http://mails.dpdk.org/archives/dev/2021-January/197814.html
>
> If not already done, please UPDATE your environment! Thanks
>
>
Ali Alnubani Jan. 25, 2021, 6:51 p.m. UTC | #3 
Hi,

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Dmitry Kozlyuk
> Sent: Sunday, January 24, 2021 10:52 PM
> To: dev@dpdk.org
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; Bruce Richardson
> <bruce.richardson@intel.com>; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; Neil
> Horman <nhorman@tuxdriver.com>; Jie Zhou <jizh@linux.microsoft.com>
> Subject: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python
> 
> This patchset implements existing pmdinfogen logic in Python, replaces and
> removes the old code. The goals of rewriting are:
> 
> * easier maintenance by using a more high-level language,
> * simpler build process without host application and libelf,
> * foundation for adding Windows support.
> 
> Identity of generated PMD information is checked by comparing output of
> pmdinfo before and after the patch:
> 
>     find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
> 
> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> Tested-by: Jie Zhou <jizh@linux.microsoft.com>
> 
> ---

Please see the following build errors:
https://github.com/DPDK/dpdk/actions/runs/509771118
"""
FAILED: drivers/rte_common_cpt.pmd.c 
/usr/bin/python3 ../buildtools/gen-pmdinfo-cfile.py ar /home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.a drivers/rte_common_cpt.pmd.c /usr/bin/python3 ../buildtools/pmdinfogen.py elf
Traceback (most recent call last):
  File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
    [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
"""

I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).

Thanks,
Ali
Dmitry Kozlyuk Jan. 25, 2021, 10:15 p.m. UTC | #4 
Hi Ali,

> Please see the following build errors:
> https://github.com/DPDK/dpdk/actions/runs/509771118
> """
> FAILED: drivers/rte_common_cpt.pmd.c 
> /usr/bin/python3 ../buildtools/gen-pmdinfo-cfile.py ar /home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.a drivers/rte_common_cpt.pmd.c /usr/bin/python3 ../buildtools/pmdinfogen.py elf
> Traceback (most recent call last):
>   File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
>     [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
>   File "/usr/lib/python3.6/subprocess.py", line 423, in run
>     with Popen(*popenargs, **kwargs) as process:
> TypeError: __init__() got an unexpected keyword argument 'capture_output'
> """
> 
> I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).

If fixed the error you encountered and also discovered a bunch of other
issues, all of them hopefully fixed in the following series:

http://patchwork.dpdk.org/project/dpdk/list/?series=14938
Tal Shnaiderman Feb. 2, 2021, 8:48 a.m. UTC | #5 
> Subject: Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python
> 
> External email: Use caution opening links or attachments
> 
> 
> Hi Thomas,
> 
> We have updated all UNH-IOL environments to include pyelftools.

FreeBSD compilation test is still failing on the missing dependency:

buildtools/meson.build:45:2: ERROR: Problem encountered: missing python module: elftools

> 
> Thanks,
> Brandon
> 
> On Mon, Jan 25, 2021 at 8:13 AM Thomas Monjalon
> <thomas@monjalon.net> wrote:
> >
> > > Dmitry Kozlyuk (3):
> > >   pmdinfogen: add Python implementation
> > >   build: use Python pmdinfogen
> > >   pmdinfogen: remove C implementation
> >
> > Applied, thanks.
> >
> > Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted
> > already in CI and maintainers meetings:
> >
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.
> dpdk.org%2Farchives%2Fci%2F2021-
> January%2F000939.html&amp;data=04%7C01%7Ctalshn%40nvidia.com%7C70
> 5c7ce391d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a
> %7C0%7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> 000&amp;sdata=Lei7o0Mtgsb3WZJ14lGolduVJK3fA7EbYuw8ojVCb3g%3D&a
> mp;reserved=0
> >
> >
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails
> > .dpdk.org%2Farchives%2Fdev%2F2021-
> January%2F197814.html&amp;data=04%7C
> >
> 01%7Ctalshn%40nvidia.com%7C705c7ce391d1429bc94508d8c14b9880%7C430
> 83d15
> >
> 727340c1b7db39efd9ccc17a%7C0%7C0%7C637471877725119466%7CUnknow
> n%7CTWFp
> >
> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
> I6Mn
> >
> 0%3D%7C1000&amp;sdata=74Zp6%2B6YB8OCVfo%2F0Gb7C9C43QWAo1YD4
> AUANneBGdA%
> > 3D&amp;reserved=0
> >
> > If not already done, please UPDATE your environment! Thanks
> >
> >
> 
> 
> --
> 
> Brandon Lo
> 
> UNH InterOperability Laboratory
> 
> 21 Madbury Rd, Suite 100, Durham, NH 03824
> 
> blo@iol.unh.edu
> 
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww
> .iol.unh.edu%2F&amp;data=04%7C01%7Ctalshn%40nvidia.com%7C705c7ce3
> 91d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a%7C0%
> 7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&a
> mp;sdata=2Fc4epSpG%2FhLFdxj9D1kVoafaxlfUhaZvpEm4xwHFd4%3D&amp;
> reserved=0