build: output a dependency log in build directory

As meson processes our DPDK source tree it manages dependencies
specified by each individual driver. To enable future analysis of the
dependency links between components, log the dependencies of each DPDK
component as it gets processed. This could potentially allow other tools
to automatically enable or disable components based on the desired end
components to be built, e.g. if the user requests net/ice, ensure that
common/iavf is also enabled in the drivers.

The output file produced is in "dot" or "graphviz" format, which allows
producing a graphical representation of the DPDK dependency tree if so
desired. For example: "dot -Tpng -O build/deps.dot" to produce the
image file "build/deps.dot.png".

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/meson.build        |  1 +
 buildtools/log-deps.py | 43 ++++++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build |  2 ++
 drivers/meson.build    |  1 +
 lib/meson.build        |  1 +
 5 files changed, 48 insertions(+)
 create mode 100644 buildtools/log-deps.py
  

> As meson processes our DPDK source tree it manages dependencies
> specified by each individual driver. To enable future analysis of the
> dependency links between components, log the dependencies of each DPDK
> component as it gets processed. This could potentially allow other tools
> to automatically enable or disable components based on the desired end
> components to be built, e.g. if the user requests net/ice, ensure that
> common/iavf is also enabled in the drivers.
> 
> The output file produced is in "dot" or "graphviz" format, which allows
> producing a graphical representation of the DPDK dependency tree if so
> desired. For example: "dot -Tpng -O build/deps.dot" to produce the
> image file "build/deps.dot.png".
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

I think it is a great idea.
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>

> ---
>  app/meson.build        |  1 +
>  buildtools/log-deps.py | 43 ++++++++++++++++++++++++++++++++++++++++++
>  buildtools/meson.build |  2 ++
>  drivers/meson.build    |  1 +
>  lib/meson.build        |  1 +
>  5 files changed, 48 insertions(+)
>  create mode 100644 buildtools/log-deps.py
> 
> diff --git a/app/meson.build b/app/meson.build
> index 5b2c80c7a1..6afa457f4c 100644
> --- a/app/meson.build
> +++ b/app/meson.build
> @@ -76,6 +76,7 @@ foreach app:apps
> 
>      if build
>          subdir(name)
> +        run_command([log_deps_cmd, name, deps])
>          if not build and require_apps
>              error('Cannot build explicitly requested app "@0@".\n'.format(name)
>                    + '\tReason: ' + reason)
> diff --git a/buildtools/log-deps.py b/buildtools/log-deps.py
> new file mode 100644
> index 0000000000..a4331fa15b
> --- /dev/null
> +++ b/buildtools/log-deps.py
> @@ -0,0 +1,43 @@
> +#! /usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Intel Corporation
> +
> +"""Utility script to build up a list of dependencies from meson."""
> +
> +import os
> +import sys
> +
> +
> +def file_to_list(filename):
> +    """Read file into a list of strings."""
> +    with open(filename) as f:
> +        return f.readlines()
> +
> +
> +def list_to_file(filename, lines):
> +    """Write a list of strings out to a file."""
> +    with open(filename, 'w') as f:
> +        f.writelines(lines)
> +
> +
> +depsfile = f'{os.environ["MESON_BUILD_ROOT"]}/deps.dot'
> +
> +# to reset the deps file on each build, the script is called without any params
> +if len(sys.argv) == 1:
> +    os.remove(depsfile)
> +    sys.exit(0)
> +
> +try:
> +    contents = file_to_list(depsfile)
> +except FileNotFoundError:
> +    contents = ['digraph {\n', '}\n']
> +
> +component = sys.argv[1]
> +if len(sys.argv) > 2:
> +    contents[-1] = f'"{component}" -> {{ "{"\", \"".join(sys.argv[2:])}" }}\n'
> +else:
> +    contents[-1] = f'"{component}"\n'
> +
> +contents.append('}\n')
> +
> +list_to_file(depsfile, contents)
> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 3adf34e1a8..332f0f3d38 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -24,6 +24,8 @@ get_numa_count_cmd = py3 + files('get-numa-count.py')
>  get_test_suites_cmd = py3 + files('get-test-suites.py')
>  has_hugepages_cmd = py3 + files('has-hugepages.py')
>  cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
> +log_deps_cmd = py3 + files('log-deps.py')
> +run_command(log_deps_cmd)  # call with no parameters to reset the file
> 
>  # install any build tools that end-users might want also
>  install_data([
> diff --git a/drivers/meson.build b/drivers/meson.build
> index 66931d4241..44935e067c 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -154,6 +154,7 @@ foreach subpath:subdirs
>          if build
>              # pull in driver directory which should update all the local variables
>              subdir(drv_path)
> +            run_command([log_deps_cmd, drv_path.underscorify(), deps])
> 
>              if dpdk_conf.get('RTE_IOVA_IN_MBUF') == 0 and require_iova_in_mbuf
>                  build = false
> diff --git a/lib/meson.build b/lib/meson.build
> index 162287753f..da2815465f 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -160,6 +160,7 @@ foreach l:libraries
> 
>      if build
>          subdir(l)
> +        run_command([log_deps_cmd, l, deps])
>          if not build and require_libs
>              error('Cannot build explicitly requested lib "@0@".\n'.format(name)
>                      +'\tReason: ' + reason)
> --
> 2.43.0
  

On 7/30/2024 3:55 PM, Bruce Richardson wrote:
> As meson processes our DPDK source tree it manages dependencies
> specified by each individual driver. To enable future analysis of the
> dependency links between components, log the dependencies of each DPDK
> component as it gets processed. This could potentially allow other tools
> to automatically enable or disable components based on the desired end
> components to be built, e.g. if the user requests net/ice, ensure that
> common/iavf is also enabled in the drivers.
> 
> The output file produced is in "dot" or "graphviz" format, which allows
> producing a graphical representation of the DPDK dependency tree if so
> desired. For example: "dot -Tpng -O build/deps.dot" to produce the
> image file "build/deps.dot.png".
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>

I tested it quickly, good to have this dependency list, at least it
makes some duplicated dependencies obvious from .dot file.

But generated dependency graph is too large to be useful, does it make
sense to have a new meson option that control this dependency generation:
-Dgenerate_deps=apps
-Dgenerate_deps=libs
-Dgenerate_deps=drivers
-Dgenerate_deps=all

Not sure about what should be the default option,
as this is not always required/used, disabling this option by default
saves unnecessary work,
but disabling it by default may cause not noticing when it is broken,
perhaps this can be addressed by adding this option to
'devtools/test-meson-builds.sh'.

btw, it generates following warning:

WARNING: You should add the boolean check kwarg to the run_command call.


         It currently defaults to false,


         but it will default to true in future releases of meson.


         See also: https://github.com/mesonbuild/meson/issues/9300


> ---
>  app/meson.build        |  1 +
>  buildtools/log-deps.py | 43 ++++++++++++++++++++++++++++++++++++++++++
>  buildtools/meson.build |  2 ++
>  drivers/meson.build    |  1 +
>  lib/meson.build        |  1 +
>  5 files changed, 48 insertions(+)
>  create mode 100644 buildtools/log-deps.py
> 
> diff --git a/app/meson.build b/app/meson.build
> index 5b2c80c7a1..6afa457f4c 100644
> --- a/app/meson.build
> +++ b/app/meson.build
> @@ -76,6 +76,7 @@ foreach app:apps
>  
>      if build
>          subdir(name)
> +        run_command([log_deps_cmd, name, deps])
>          if not build and require_apps
>              error('Cannot build explicitly requested app "@0@".\n'.format(name)
>                    + '\tReason: ' + reason)
> diff --git a/buildtools/log-deps.py b/buildtools/log-deps.py
> new file mode 100644
> index 0000000000..a4331fa15b
> --- /dev/null
> +++ b/buildtools/log-deps.py
> @@ -0,0 +1,43 @@
> +#! /usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Intel Corporation
> +
> +"""Utility script to build up a list of dependencies from meson."""
> +
> +import os
> +import sys
> +
> +
> +def file_to_list(filename):
> +    """Read file into a list of strings."""
> +    with open(filename) as f:
> +        return f.readlines()
> +
> +
> +def list_to_file(filename, lines):
> +    """Write a list of strings out to a file."""
> +    with open(filename, 'w') as f:
> +        f.writelines(lines)
> +
> +
> +depsfile = f'{os.environ["MESON_BUILD_ROOT"]}/deps.dot'
> +
> +# to reset the deps file on each build, the script is called without any params
> +if len(sys.argv) == 1:
> +    os.remove(depsfile)
> +    sys.exit(0)
> +
> +try:
> +    contents = file_to_list(depsfile)
> +except FileNotFoundError:
> +    contents = ['digraph {\n', '}\n']
> +
> +component = sys.argv[1]
> +if len(sys.argv) > 2:
> +    contents[-1] = f'"{component}" -> {{ "{"\", \"".join(sys.argv[2:])}" }}\n'
> +else:
> +    contents[-1] = f'"{component}"\n'
> +
> +contents.append('}\n')
> +
> +list_to_file(depsfile, contents)
> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 3adf34e1a8..332f0f3d38 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -24,6 +24,8 @@ get_numa_count_cmd = py3 + files('get-numa-count.py')
>  get_test_suites_cmd = py3 + files('get-test-suites.py')
>  has_hugepages_cmd = py3 + files('has-hugepages.py')
>  cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
> +log_deps_cmd = py3 + files('log-deps.py')
> +run_command(log_deps_cmd)  # call with no parameters to reset the file
>  
>  # install any build tools that end-users might want also
>  install_data([
> diff --git a/drivers/meson.build b/drivers/meson.build
> index 66931d4241..44935e067c 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -154,6 +154,7 @@ foreach subpath:subdirs
>          if build
>              # pull in driver directory which should update all the local variables
>              subdir(drv_path)
> +            run_command([log_deps_cmd, drv_path.underscorify(), deps])
>  
>              if dpdk_conf.get('RTE_IOVA_IN_MBUF') == 0 and require_iova_in_mbuf
>                  build = false
> diff --git a/lib/meson.build b/lib/meson.build
> index 162287753f..da2815465f 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -160,6 +160,7 @@ foreach l:libraries
>  
>      if build
>          subdir(l)
> +        run_command([log_deps_cmd, l, deps])
>          if not build and require_libs
>              error('Cannot build explicitly requested lib "@0@".\n'.format(name)
>                      +'\tReason: ' + reason)
  

On Wed, Jul 31, 2024 at 11:17:54AM +0100, Ferruh Yigit wrote:
> On 7/30/2024 3:55 PM, Bruce Richardson wrote:
> > As meson processes our DPDK source tree it manages dependencies
> > specified by each individual driver. To enable future analysis of the
> > dependency links between components, log the dependencies of each DPDK
> > component as it gets processed. This could potentially allow other tools
> > to automatically enable or disable components based on the desired end
> > components to be built, e.g. if the user requests net/ice, ensure that
> > common/iavf is also enabled in the drivers.
> > 
> > The output file produced is in "dot" or "graphviz" format, which allows
> > producing a graphical representation of the DPDK dependency tree if so
> > desired. For example: "dot -Tpng -O build/deps.dot" to produce the
> > image file "build/deps.dot.png".
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> >
> 
> I tested it quickly, good to have this dependency list, at least it
> makes some duplicated dependencies obvious from .dot file.
> 
> But generated dependency graph is too large to be useful, does it make
> sense to have a new meson option that control this dependency generation:
> -Dgenerate_deps=apps
> -Dgenerate_deps=libs
> -Dgenerate_deps=drivers
> -Dgenerate_deps=all
> 

I had indeed noticed that. Right now I'm focused on cutting down our
dependency lists to make things more manageable - I have a script to
automate the identification of excess dependencies and I already have a
patch for cleaning up the libraries to a minimal dependency set.

I also think having separate drivers and libraries charts could be useful I
wouldn't want to add a build option for it though - it's not something that
should be necessary. Instead, I will look at other options, perhaps even
doing multiple dependency files - they don't seem to slow meson down, so
why not always do them all.

> Not sure about what should be the default option,
> as this is not always required/used, disabling this option by default
> saves unnecessary work,
> but disabling it by default may cause not noticing when it is broken,
> perhaps this can be addressed by adding this option to
> 'devtools/test-meson-builds.sh'.
> 
> btw, it generates following warning:

Oops - will look to fix in V2.

/Bruce
  

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 2 August 2024 14.44
> 
> As part of the meson build, we can record the dependencies for each
> component as we process it, logging them to a file. This file can be
> used as input to a number of other scripts and tools, for example, to
> graph the dependencies, or to allow higher-level build-config tools to
> automatically enable component requirements, etc.
> 
> The first patch of this set generates the basic dependency tree. The
> second patch does some processing of that dependency tree to identify
> cases where dependencies are being unnecessarily specified. Reducing
> these makes it easier to have readable dependency graphs in future,
> without affecting the build.
> 
> The following 4 patches are based on the output of the second patch, and
> greatly cut down the number of direct dependency links between
> components. Even with the cut-down dependencies, the full dependency
> graph is nigh-unreadable, so the final patch adds a new script to
> generate dependency tree subgraphs, creating dot files for e.g. the
> dependencies of a particular component, or a component class such as
> mempool drivers.

Thank you for the work on this, Bruce.

For the series,
Acked-by: Morten Brørup <mb@smartsharesystems.com>
  

I saw there was one fail for this series for the Marvell MVNETA container.

I thought I should also mention that the build did fail when our
automation tried to run DTS on the Octeon CN10K DPU (which is also
Marvell). So, that is why you won't get a report for the CN10K DTS
results.

Let me know if you can't get what you need from the MVNETA build logs
failure, and I can manually get more info about why this won't
compile. Cheers.
  

On Fri, Aug 02, 2024 at 11:05:58AM -0400, Patrick Robb wrote:
> I saw there was one fail for this series for the Marvell MVNETA container.
> 
> I thought I should also mention that the build did fail when our
> automation tried to run DTS on the Octeon CN10K DPU (which is also
> Marvell). So, that is why you won't get a report for the CN10K DTS
> results.
> 
> Let me know if you can't get what you need from the MVNETA build logs
> failure, and I can manually get more info about why this won't
> compile. Cheers.

I have already got multiple emails about failures from this series - though
all look to be about the same issue. I think it should be an easy fix for
v3, but I'll waiting till all reports come back.

/Bruce
  

On 8/2/2024 1:44 PM, Bruce Richardson wrote:
> As part of the meson build, we can record the dependencies for each
> component as we process it, logging them to a file. This file can be
> used as input to a number of other scripts and tools, for example, to
> graph the dependencies, or to allow higher-level build-config tools to
> automatically enable component requirements, etc.
> 
> The first patch of this set generates the basic dependency tree. The
> second patch does some processing of that dependency tree to identify
> cases where dependencies are being unnecessarily specified. Reducing
> these makes it easier to have readable dependency graphs in future,
> without affecting the build.
> 
> The following 4 patches are based on the output of the second patch, and
> greatly cut down the number of direct dependency links between
> components. Even with the cut-down dependencies, the full dependency
> graph is nigh-unreadable, so the final patch adds a new script to
> generate dependency tree subgraphs, creating dot files for e.g. the
> dependencies of a particular component, or a component class such as
> mempool drivers.
> 
> Bruce Richardson (7):
>   build: output a dependency log in build directory
>   devtools: add script to flag unneeded dependencies
>   build: remove kvargs from driver class dependencies
>   build: reduce library dependencies
>   build: reduce driver dependencies
>   build: reduce app dependencies
>   devtools: add script to generate DPDK dependency graphs
>

Tested-by: Ferruh Yigit <ferruh.yigit@amd.com>

Thanks for the update, output is now easier to consume with the help of
the 'devtools/draw-dependency-graphs.py' script.
A detail but script is not actually drawing graph, but parsing .dot
file, name is a little misleading.


Also for the patches that are converting explicit dependency to implicit
dependency, I can see the benefit for the meson and graph. But also
there is a value of keeping explicit dependency, it was an easy way to
document dependencies of component for developers.
So, I am not really sure which one is better.
  

On Fri, Aug 02, 2024 at 06:18:26PM +0100, Ferruh Yigit wrote:
> On 8/2/2024 1:44 PM, Bruce Richardson wrote:
> > As part of the meson build, we can record the dependencies for each
> > component as we process it, logging them to a file. This file can be
> > used as input to a number of other scripts and tools, for example, to
> > graph the dependencies, or to allow higher-level build-config tools to
> > automatically enable component requirements, etc.
> > 
> > The first patch of this set generates the basic dependency tree. The
> > second patch does some processing of that dependency tree to identify
> > cases where dependencies are being unnecessarily specified. Reducing
> > these makes it easier to have readable dependency graphs in future,
> > without affecting the build.
> > 
> > The following 4 patches are based on the output of the second patch, and
> > greatly cut down the number of direct dependency links between
> > components. Even with the cut-down dependencies, the full dependency
> > graph is nigh-unreadable, so the final patch adds a new script to
> > generate dependency tree subgraphs, creating dot files for e.g. the
> > dependencies of a particular component, or a component class such as
> > mempool drivers.
> > 
> > Bruce Richardson (7):
> >   build: output a dependency log in build directory
> >   devtools: add script to flag unneeded dependencies
> >   build: remove kvargs from driver class dependencies
> >   build: reduce library dependencies
> >   build: reduce driver dependencies
> >   build: reduce app dependencies
> >   devtools: add script to generate DPDK dependency graphs
> >
> 
> Tested-by: Ferruh Yigit <ferruh.yigit@amd.com>
> 
> Thanks for the update, output is now easier to consume with the help of
> the 'devtools/draw-dependency-graphs.py' script.
> A detail but script is not actually drawing graph, but parsing .dot
> file, name is a little misleading.
> 
> 
> Also for the patches that are converting explicit dependency to implicit
> dependency, I can see the benefit for the meson and graph. But also
> there is a value of keeping explicit dependency, it was an easy way to
> document dependencies of component for developers.
> So, I am not really sure which one is better.
> 

Up till now we did not keep a fully dependency list for each component
because the configuration time with meson completely exploded due to really
slow deduplication of dependencies there. I haven't checked with recent
versions of meson, though, so the problem may no longer be there now. Right
now, we have "partial dependency lists" - neither full list of dependencies
of a component, nor a minimal set. These patches don't bring us fully to a
minimal set, but bring us closer. I honestly don't think we need to enforce
either case - so long as we have all dependencies either explicilty or via
recursive dependency, things are fine.

/Bruce
  

> From: Anatoly Burakov [mailto:anatoly.burakov@intel.com]
> 
> As part of the meson build, we can record the dependencies for each
> component as we process it, logging them to a file. This file can be
> used as input to a number of other scripts and tools, for example, to
> graph the dependencies, or to allow higher-level build-config tools to
> automatically enable component requirements, etc.
> 
> The first patch of this set separates dependencies inside meson into
> optional or mandatory. The second patch of this set generates the basic
> dependency tree. The third patch does some processing of that dependency
> tree to identify cases where dependencies are being unnecessarily
> specified. Reducing these makes it easier to have readable dependency
> graphs in future, without affecting the build.
> 
> The following 4 patches are based on the output of the second patch, and
> greatly cut down the number of direct dependency links between
> components. Even with the cut-down dependencies, the full dependency
> graph is nigh-unreadable, so the final patch adds a new script to
> generate dependency tree subgraphs, creating dot files for e.g. the
> dependencies of a particular component, or a component class such as
> mempool drivers.
> 

For the series,
Acked-by: Morten Brørup <mb@smartsharesystems.com>