diff mbox series

devtools: script to track map symbols

Message ID 20210618163659.85933-1-mdr@ashroe.eu (mailing list archive)
State Superseded
Headers show
Series devtools: script to track map symbols | expand

Checks

Context Check Description
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-mellanox-Functional fail Functional Testing issues
ci/intel-Testing success Testing PASS
ci/Intel-compilation success Compilation OK
ci/iol-testing success Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/github-robot fail github build: failed
ci/checkpatch success coding style OK

Commit Message

Kinsella, Ray June 18, 2021, 4:36 p.m. UTC
Script to track growth of stable and experimental symbols
over releases since v19.11.

Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
---
 devtools/count_symbols.py | 230 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 230 insertions(+)
 create mode 100755 devtools/count_symbols.py

Comments

Stephen Hemminger June 18, 2021, 7:40 p.m. UTC | #1
On Fri, 18 Jun 2021 17:36:59 +0100
Ray Kinsella <mdr@ashroe.eu> wrote:

> Script to track growth of stable and experimental symbols
> over releases since v19.11.
> 
> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>

pylint reports some things that should be fixed. Don't worry about the naming style
and docstring but others should be addressed.


************* Module count_symbols
devtools/count_symbols.py:12:0: W0311: Bad indentation. Found 8 spaces, expected 4 (bad-indentation)
devtools/count_symbols.py:14:0: W0311: Bad indentation. Found 8 spaces, expected 4 (bad-indentation)
devtools/count_symbols.py:16:0: W0311: Bad indentation. Found 8 spaces, expected 4 (bad-indentation)
devtools/count_symbols.py:109:0: C0325: Unnecessary parens after 'if' keyword (superfluous-parens)
devtools/count_symbols.py:230:0: W0311: Bad indentation. Found 8 spaces, expected 4 (bad-indentation)
devtools/count_symbols.py:47:41: W1401: Anomalous backslash in string: '\.'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
devtools/count_symbols.py:47:43: W1401: Anomalous backslash in string: '\d'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
devtools/count_symbols.py:1:0: C0114: Missing module docstring (missing-module-docstring)
devtools/count_symbols.py:5:0: C0410: Multiple imports on one line (sys, os) (multiple-imports)
devtools/count_symbols.py:16:8: R1722: Consider using sys.exit() (consider-using-sys-exit)
devtools/count_symbols.py:18:0: C0103: Constant name "symbolMapGrammar" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:37:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:39:4: C0103: Variable name "s" doesn't conform to snake_case naming style (invalid-name)
devtools/count_symbols.py:40:4: C0103: Variable name "s" doesn't conform to snake_case naming style (invalid-name)
devtools/count_symbols.py:44:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:46:4: C0103: Variable name "s" doesn't conform to snake_case naming style (invalid-name)
devtools/count_symbols.py:50:13: W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
devtools/count_symbols.py:66:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:70:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:78:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:82:13: W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
devtools/count_symbols.py:91:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:94:13: W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
devtools/count_symbols.py:112:17: W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
devtools/count_symbols.py:124:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:133:0: C0103: Constant name "fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:133:6: C0103: Constant name "col_fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:135:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:136:4: W0603: Using the global statement (global-statement)
devtools/count_symbols.py:136:4: C0103: Constant name "fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:136:4: C0103: Constant name "col_fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:140:8: W0612: Unused variable 'rel' (unused-variable)
devtools/count_symbols.py:144:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:145:4: W0603: Using the global statement (global-statement)
devtools/count_symbols.py:145:4: C0103: Constant name "fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:145:4: C0103: Constant name "col_fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:149:8: W0612: Unused variable 'rel' (unused-variable)
devtools/count_symbols.py:156:0: C0103: Constant name "directories" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:158:0: C0116: Missing function or method docstring (missing-function-docstring)
devtools/count_symbols.py:158:0: R0914: Too many local variables (20/15) (too-many-locals)
devtools/count_symbols.py:159:4: W0603: Using the global statement (global-statement)
devtools/count_symbols.py:159:4: C0103: Constant name "fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:159:4: C0103: Constant name "col_fmt" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:159:4: C0103: Constant name "symbolMapGrammar" doesn't conform to UPPER_CASE naming style (invalid-name)
devtools/count_symbols.py:177:4: C0103: Variable name "MAPParser" doesn't conform to snake_case naming style (invalid-name)
devtools/count_symbols.py:192:25: W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
devtools/count_symbols.py:5:0: W0611: Unused import sys (unused-import)

-----------------------------------
Your code has been rated at 6.27/10
Kinsella, Ray June 21, 2021, 9:18 a.m. UTC | #2
> 
> pylint reports some things that should be fixed. Don't worry about the naming style
> and docstring but others should be addressed.

[SNIP]

Ah, rookie mistake,

I ran checkpatch and thought that I was all good. 
I will sort it out thanks. 

Ray K
diff mbox series

Patch

diff --git a/devtools/count_symbols.py b/devtools/count_symbols.py
new file mode 100755
index 0000000000..7b29651044
--- /dev/null
+++ b/devtools/count_symbols.py
@@ -0,0 +1,230 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021 Intel Corporation
+from pathlib import Path
+import sys, os
+import subprocess
+import argparse
+import re
+import datetime
+
+try:
+        from parsley import makeGrammar
+except ImportError:
+        print('This script uses the package Parsley to parse C Mapfiles.\n'
+              'This can be installed with \"pip install parsley".')
+        exit()
+
+symbolMapGrammar = r"""
+
+ws = (' ' | '\r' | '\n' | '\t')*
+
+ABI_VER = ({})
+DPDK_VER = ('DPDK_' ABI_VER)
+ABI_NAME = ('INTERNAL' | 'EXPERIMENTAL' | DPDK_VER)
+comment = '#' (~'\n' anything)+ '\n'
+symbol = (~(';' | '}}' | '#') anything )+:c ';' -> ''.join(c)
+global = 'global:'
+local = 'local: *;'
+symbols = comment* symbol:s ws comment* -> s
+
+abi = (abi_section+):m -> dict(m)
+abi_section = (ws ABI_NAME:e ws '{{' ws global* (~local ws symbols)*:s ws local* ws '}}' ws DPDK_VER* ';' ws) -> (e,s)
+"""
+
+#abi_ver = ['21', '20.0.1', '20.0', '20']
+
+def get_abi_versions():
+    year = datetime.date.today().year - 2000
+    s=" |".join(['\'{}\''.format(i) for i in reversed(range(21, year + 1)) ])
+    s = s + ' | \'20.0.1\' | \'20.0\' | \'20\''
+
+    return s
+
+def get_dpdk_releases():
+    year = datetime.date.today().year - 2000
+    s="|".join("{}".format(i) for i in range(19,year + 1))
+    pattern = re.compile('^\"v(' + s + ')\.\d{2}\"$')
+
+    cmd = ['git', 'for-each-ref', '--sort=taggerdate', '--format', '"%(tag)"']
+    result = subprocess.run(cmd, \
+                            stdout=subprocess.PIPE, \
+                            stderr=subprocess.PIPE)
+    if result.stderr.startswith(b'fatal'):
+        result = None
+
+    tags = result.stdout.decode('utf-8').split('\n')
+
+    # find the non-rcs between now and v19.11
+    tags = [ tag.replace('\"','') \
+             for tag in reversed(tags) \
+             if pattern.match(tag) ][:-3]
+
+    return tags
+
+
+def get_terminal_rows():
+    rows, _ = os.popen('stty size', 'r').read().split()
+    return int(rows)
+
+def fix_directory_name(path):
+    mapfilepath1 = str(path.parent.name)
+    mapfilepath2 = str(path.parents[1])
+    mapfilepath = mapfilepath2 + '/librte_' + mapfilepath1
+
+    return mapfilepath
+
+# fix removal of the librte_ from the directory names
+def directory_renamed(path, rel):
+    mapfilepath = fix_directory_name(path)
+    tagfile = '{}:{}/{}'.format(rel, mapfilepath,  path.name)
+
+    result = subprocess.run(['git', 'show', tagfile], \
+                            stdout=subprocess.PIPE, \
+                            stderr=subprocess.PIPE)
+    if result.stderr.startswith(b'fatal'):
+        result = None
+
+    return result
+
+# fix renaming of map files
+def mapfile_renamed(path, rel):
+    newfile = None
+
+    result = subprocess.run(['git', 'ls-tree', \
+                             rel, str(path.parent) + '/'], \
+                            stdout=subprocess.PIPE, \
+                            stderr=subprocess.PIPE)
+    dentries = result.stdout.decode('utf-8')
+    dentries = dentries.split('\n')
+
+    # filter entries looking for the map file
+    dentries = [dentry for dentry in dentries if dentry.endswith('.map')]
+    if len(dentries) > 1 or len(dentries) == 0:
+        return None
+
+    dparts = dentries[0].split('/')
+    newfile = dparts[len(dparts) - 1]
+
+    if(newfile is not None):
+        tagfile = '{}:{}/{}'.format(rel, path.parent, newfile)
+
+        result = subprocess.run(['git', 'show', tagfile], \
+                                stdout=subprocess.PIPE, \
+                                stderr=subprocess.PIPE)
+        if result.stderr.startswith(b'fatal'):
+            result = None
+
+    else:
+        result = None
+
+    return result
+
+# renaming of the map file & renaming of directory
+def mapfile_and_directory_renamed(path, rel):
+    mapfilepath = Path("{}/{}".format(fix_directory_name(path),path.name))
+
+    return mapfile_renamed(mapfilepath, rel)
+
+fix_strategies = [directory_renamed, \
+                  mapfile_renamed, \
+                  mapfile_and_directory_renamed]
+
+fmt = col_fmt = ""
+
+def set_terminal_output(dpdk_rel):
+    global fmt, col_fmt
+
+    fmt = '{:<50}'
+    col_fmt = fmt
+    for rel in dpdk_rel:
+        fmt += '{:<6}{:<6}'
+        col_fmt += '{:<12}'
+
+def set_csv_output(dpdk_rel):
+    global fmt, col_fmt
+
+    fmt = '{},'
+    col_fmt = fmt
+    for rel in dpdk_rel:
+        fmt += '{},{},'
+        col_fmt += '{},,'
+
+output_formats = { None: set_terminal_output, \
+                   'terminal': set_terminal_output, \
+                   'csv': set_csv_output }
+directories = 'drivers, lib'
+
+def main():
+    global fmt, col_fmt, symbolMapGrammar
+
+    parser = argparse.ArgumentParser(description='Count symbols in DPDK Libs')
+    parser.add_argument('--format-output', choices=['terminal','csv'], \
+                        default='terminal')
+    parser.add_argument('--directory', choices=directories,
+                        default=directories)
+    args = parser.parse_args()
+
+    dpdk_rel = get_dpdk_releases()
+
+    # set the output format
+    output_formats[args.format_output](dpdk_rel)
+
+    column_titles = ['mapfile'] + dpdk_rel
+    print(col_fmt.format(*column_titles))
+
+    symbolMapGrammar = symbolMapGrammar.format(get_abi_versions())
+    MAPParser = makeGrammar(symbolMapGrammar, {})
+
+    terminal_rows = get_terminal_rows()
+    row = 0
+
+    for src_dir in args.directory.split(','):
+        for path in Path(src_dir).rglob('*.map'):
+            csym = [0] * 2
+            relsym = [str(path)]
+
+            for rel in dpdk_rel:
+                i = csym[0] = csym[1] = 0
+                abi_sections = None
+
+                tagfile = '{}:{}'.format(rel,path)
+                result = subprocess.run(['git', 'show', tagfile], \
+                                        stdout=subprocess.PIPE, \
+                                        stderr=subprocess.PIPE)
+
+                if result.stderr.startswith(b'fatal'):
+                    result = None
+
+                while(result is None and i < len(fix_strategies)):
+                    result = fix_strategies[i](path, rel)
+                    i += 1
+
+                if result is not None:
+                    mapfile = result.stdout.decode('utf-8')
+                    abi_sections = MAPParser(mapfile).abi()
+
+                if abi_sections is not None:
+                    # which versions are present, and we care about
+                    ignore = ['EXPERIMENTAL','INTERNAL']
+                    found_ver = [ver \
+                                 for ver in abi_sections \
+                                 if ver not in ignore]
+
+                    for ver in found_ver:
+                        csym[0] += len(abi_sections[ver])
+
+                    # count experimental symbols
+                    if 'EXPERIMENTAL' in abi_sections:
+                        csym[1] = len(abi_sections['EXPERIMENTAL'])
+
+                relsym += csym
+
+            print(fmt.format(*relsym))
+            row += 1
+
+        if((terminal_rows>0) and ((row % terminal_rows) == 0)):
+            print(col_fmt.format(*column_titles))
+
+if __name__ == '__main__':
+        main()