[v4,01/53] devtools: script to remove unused headers includes

Message ID 20220114162409.334437-2-sean.morrissey@intel.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers
Series introduce IWYU |

Checks

Context Check Description
ci/checkpatch success coding style OK

Commit Message

Sean Morrissey Jan. 14, 2022, 4:23 p.m. UTC
This script can be used for removing headers flagged for removal by the
include-what-you-use (IWYU) tool. The script has the ability to remove
headers from specified sub-directories or dpdk as a whole and tests the
build after each removal by calling meson compile.

example usages:

Remove headers flagged by iwyu_tool output file
$ ./devtools/process_iwyu.py iwyu.out -b build

Remove headers flagged by iwyu_tool output file from sub-directory
$ ./devtools/process_iwyu.py iwyu.out -b build -d lib/kvargs

Remove headers directly piped from the iwyu_tool
$ iwyu_tool -p build | ./devtools/process_iwyu.py - -b build

Signed-off-by: Sean Morrissey <sean.morrissey@intel.com>
Signed-off-by: Conor Fogarty <conor.fogarty@intel.com>

Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>
---
 devtools/process_iwyu.py | 109 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 109 insertions(+)
 create mode 100755 devtools/process_iwyu.py
  

Comments

Stephen Hemminger Jan. 14, 2022, 5:02 p.m. UTC | #1
On Fri, 14 Jan 2022 16:23:17 +0000
Sean Morrissey <sean.morrissey@intel.com> wrote:

> This script can be used for removing headers flagged for removal by the
> include-what-you-use (IWYU) tool. The script has the ability to remove
> headers from specified sub-directories or dpdk as a whole and tests the
> build after each removal by calling meson compile.
> 
> example usages:
> 
> Remove headers flagged by iwyu_tool output file
> $ ./devtools/process_iwyu.py iwyu.out -b build
> 
> Remove headers flagged by iwyu_tool output file from sub-directory
> $ ./devtools/process_iwyu.py iwyu.out -b build -d lib/kvargs
> 
> Remove headers directly piped from the iwyu_tool
> $ iwyu_tool -p build | ./devtools/process_iwyu.py - -b build
> 
> Signed-off-by: Sean Morrissey <sean.morrissey@intel.com>
> Signed-off-by: Conor Fogarty <conor.fogarty@intel.com>
> 
> Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>

Could this be a shell script instead of python?
That would prevent having to pull in more python library dependencies.
  
Bruce Richardson Jan. 14, 2022, 5:36 p.m. UTC | #2
On Fri, Jan 14, 2022 at 09:02:40AM -0800, Stephen Hemminger wrote:
> On Fri, 14 Jan 2022 16:23:17 +0000
> Sean Morrissey <sean.morrissey@intel.com> wrote:
> 
> > This script can be used for removing headers flagged for removal by the
> > include-what-you-use (IWYU) tool. The script has the ability to remove
> > headers from specified sub-directories or dpdk as a whole and tests the
> > build after each removal by calling meson compile.
> > 
> > example usages:
> > 
> > Remove headers flagged by iwyu_tool output file
> > $ ./devtools/process_iwyu.py iwyu.out -b build
> > 
> > Remove headers flagged by iwyu_tool output file from sub-directory
> > $ ./devtools/process_iwyu.py iwyu.out -b build -d lib/kvargs
> > 
> > Remove headers directly piped from the iwyu_tool
> > $ iwyu_tool -p build | ./devtools/process_iwyu.py - -b build
> > 
> > Signed-off-by: Sean Morrissey <sean.morrissey@intel.com>
> > Signed-off-by: Conor Fogarty <conor.fogarty@intel.com>
> > 
> > Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> Could this be a shell script instead of python?
> That would prevent having to pull in more python library dependencies.

Can you clarify what python library dependencies are you concerned about?
AFAIK everything in this script is just from the standard built-in python
library - Sean, perhaps you can confirm? Converting to shell would lead to
something far less readable, IMHO, and it would be less portable to windows
too in future.

/Bruce
  

Patch

diff --git a/devtools/process_iwyu.py b/devtools/process_iwyu.py
new file mode 100755
index 0000000000..50f3d4c5c7
--- /dev/null
+++ b/devtools/process_iwyu.py
@@ -0,0 +1,109 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021 Intel Corporation
+#
+
+import argparse
+import fileinput
+import sys
+from os.path import abspath, relpath, join
+from pathlib import Path
+from mesonbuild import mesonmain
+
+
+def args_parse():
+    "parse arguments and return the argument object back to main"
+    parser = argparse.ArgumentParser(description="This script can be used to remove includes which are not in use\n")
+    parser.add_argument('-b', '--build_dir', type=str, default='build',
+                        help="The path to the build directory in which the IWYU tool was used in.")
+    parser.add_argument('-d', '--sub_dir', type=str, default='',
+                        help="The sub-directory to remove headers from.")
+    parser.add_argument('file', type=Path,
+                        help="The path to the IWYU log file or output from stdin.")
+
+    return parser.parse_args()
+
+
+def run_meson(args):
+    "Runs a meson command logging output to process.log"
+    with open('process_iwyu.log', 'a') as sys.stdout:
+        ret = mesonmain.run(args, abspath('meson'))
+    sys.stdout = sys.__stdout__
+    return ret
+
+
+def remove_includes(filepath, include, build_dir):
+    "Attempts to remove include, if it fails then revert to original state"
+    with open(filepath) as f:
+        lines = f.readlines()  # Read lines when file is opened
+
+    with open(filepath, 'w') as f:
+        for ln in lines:  # Removes the include passed in
+            if not ln.startswith(include):
+                f.write(ln)
+
+    # run test build -> call meson on the build folder, meson compile -C build
+    ret = run_meson(['compile', '-C', build_dir])
+    if (ret == 0):  # Include is not needed -> build is successful
+        print('SUCCESS')
+    else:
+        # failed, catch the error
+        # return file to original state
+        with open(filepath, 'w') as f:
+            f.writelines(lines)
+        print('FAILED')
+
+
+def get_build_config(builddir, condition):
+    "returns contents of rte_build_config.h"
+    with open(join(builddir, 'rte_build_config.h')) as f:
+        return [ln for ln in f.readlines() if condition(ln)]
+
+
+def uses_libbsd(builddir):
+    "return whether the build uses libbsd or not"
+    return bool(get_build_config(builddir, lambda ln: 'RTE_USE_LIBBSD' in ln))
+
+
+def process(args):
+    "process the iwyu output on a set of files"
+    filepath = None
+    build_dir = abspath(args.build_dir)
+    directory = args.sub_dir
+
+    print("Warning: The results of this script may include false positives which are required for different systems",
+          file=sys.stderr)
+
+    keep_str_fns = uses_libbsd(build_dir)  # check for libbsd
+    if keep_str_fns:
+        print("Warning: libbsd is present, build will fail to detect incorrect removal of rte_string_fns.h",
+              file=sys.stderr)
+    # turn on werror
+    run_meson(['configure', build_dir, '-Dwerror=true'])
+    # Use stdin if no iwyu_tool out file given
+    for line in fileinput.input(args.file):
+        if 'should remove' in line:
+            # If the file path in the iwyu_tool output is an absolute path it
+            # means the file is outside of the dpdk directory, therefore ignore it.
+            # Also check to see if the file is within the specified sub directory.
+            filename = line.split()[0]
+            if (filename != abspath(filename) and
+                    directory in filename):
+                filepath = relpath(join(build_dir, filename))
+        elif line.startswith('-') and filepath:
+            include = '#include ' + line.split()[2]
+            print(f"Remove {include} from {filepath} ... ", end='', flush=True)
+            if keep_str_fns and '<rte_string_fns.h>' in include:
+                print('skipped')
+                continue
+            remove_includes(filepath, include, build_dir)
+        else:
+            filepath = None
+
+
+def main():
+    process(args_parse())
+
+
+if __name__ == '__main__':
+    main()