[dpdk-dev] devtools: add script to verify map files

Message ID 20180212161312.31795-1-pbhagavatula@caviumnetworks.com
State Superseded, archived
Delegated to: Thomas Monjalon
Headers show

Checks

Context Check Description
ci/Intel-compilation success Compilation OK
ci/checkpatch success coding style OK

Commit Message

Pavan Nikhilesh Feb. 12, 2018, 4:13 p.m.
This script checks for the symbols specified in the map files against
the symbols present in the map file directory.
Currently, the script checks for map files in drivers and lib directory.

Example:
./devtools/check-map.py

The following symbols are unused :

Map file : /home/pavan/Work/clean/dpdk/drivers/mempool/dpaa/...
['rte_dpaa_pool_table']

Map file : /home/pavan/Work/clean/dpdk/drivers/bus/fslmc/...
['qbman_get_version', 'qbman_swp_send_multiple']

...

Signed-off-by: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
---
 devtools/check-map.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
 create mode 100755 devtools/check-map.py

Comments

Ferruh Yigit Feb. 13, 2018, 10:48 a.m. | #1
On 2/12/2018 4:13 PM, Pavan Nikhilesh wrote:
> This script checks for the symbols specified in the map files against
> the symbols present in the map file directory.
> Currently, the script checks for map files in drivers and lib directory.
> 
> Example:
> ./devtools/check-map.py
> 
> The following symbols are unused :
> 
> Map file : /home/pavan/Work/clean/dpdk/drivers/mempool/dpaa/...
> ['rte_dpaa_pool_table']
> 
> Map file : /home/pavan/Work/clean/dpdk/drivers/bus/fslmc/...
> ['qbman_get_version', 'qbman_swp_send_multiple']
> 
> ...
> 
> Signed-off-by: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>

+1 to improve our tools.

Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
Thomas Monjalon May 27, 2018, 8:50 p.m. | #2
Sorry for having missed this patch during so long.

13/02/2018 11:48, Ferruh Yigit:
> On 2/12/2018 4:13 PM, Pavan Nikhilesh wrote:
> > This script checks for the symbols specified in the map files against
> > the symbols present in the map file directory.
> > Currently, the script checks for map files in drivers and lib directory.
> > 
> > Example:
> > ./devtools/check-map.py
> > 
> > The following symbols are unused :
> > 
> > Map file : /home/pavan/Work/clean/dpdk/drivers/mempool/dpaa/...
> > ['rte_dpaa_pool_table']
> > 
> > Map file : /home/pavan/Work/clean/dpdk/drivers/bus/fslmc/...
> > ['qbman_get_version', 'qbman_swp_send_multiple']
> > 
> > ...
> > 
> > Signed-off-by: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
> 
> +1 to improve our tools.
> 
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>

I agree we must have more tools.

I think this check can be a lot simpler as a shell script,
instead of Python. It does not need the list features of Python.

I've run it and I see it does not catch symbols which are renamed
with a prefix. Example:
	vfio_get_group_fd became rte_vfio_get_group_fd
Bruce Richardson May 28, 2018, 9:21 a.m. | #3
On Sun, May 27, 2018 at 10:50:58PM +0200, Thomas Monjalon wrote:
> Sorry for having missed this patch during so long.
> 
> 13/02/2018 11:48, Ferruh Yigit:
> > On 2/12/2018 4:13 PM, Pavan Nikhilesh wrote:
> > > This script checks for the symbols specified in the map files against
> > > the symbols present in the map file directory.
> > > Currently, the script checks for map files in drivers and lib directory.
> > > 
> > > Example:
> > > ./devtools/check-map.py
> > > 
> > > The following symbols are unused :
> > > 
> > > Map file : /home/pavan/Work/clean/dpdk/drivers/mempool/dpaa/...
> > > ['rte_dpaa_pool_table']
> > > 
> > > Map file : /home/pavan/Work/clean/dpdk/drivers/bus/fslmc/...
> > > ['qbman_get_version', 'qbman_swp_send_multiple']
> > > 
> > > ...
> > > 
> > > Signed-off-by: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
> > 
> > +1 to improve our tools.
> > 
> > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
> 
> I agree we must have more tools.
> 
> I think this check can be a lot simpler as a shell script,
> instead of Python. It does not need the list features of Python.
> 

While the shell version can be shorter, I always find python scripts to be
far easier to read, understand and therefore maintain than shell scripts.
While power-users of shell, like yourself, Thomas, can come up with some
amazing stuff done in shell, it tends to make them very hard to follow for
us mere mortals.

/Bruce
Thomas Monjalon May 28, 2018, 9:31 a.m. | #4
28/05/2018 11:21, Bruce Richardson:
> On Sun, May 27, 2018 at 10:50:58PM +0200, Thomas Monjalon wrote:
> > Sorry for having missed this patch during so long.
> > 
> > 13/02/2018 11:48, Ferruh Yigit:
> > > On 2/12/2018 4:13 PM, Pavan Nikhilesh wrote:
> > > > This script checks for the symbols specified in the map files against
> > > > the symbols present in the map file directory.
> > > > Currently, the script checks for map files in drivers and lib directory.
> > > > 
> > > > Example:
> > > > ./devtools/check-map.py
> > > > 
> > > > The following symbols are unused :
> > > > 
> > > > Map file : /home/pavan/Work/clean/dpdk/drivers/mempool/dpaa/...
> > > > ['rte_dpaa_pool_table']
> > > > 
> > > > Map file : /home/pavan/Work/clean/dpdk/drivers/bus/fslmc/...
> > > > ['qbman_get_version', 'qbman_swp_send_multiple']
> > > > 
> > > > ...
> > > > 
> > > > Signed-off-by: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
> > > 
> > > +1 to improve our tools.
> > > 
> > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > 
> > I agree we must have more tools.
> > 
> > I think this check can be a lot simpler as a shell script,
> > instead of Python. It does not need the list features of Python.
> > 
> 
> While the shell version can be shorter, I always find python scripts to be
> far easier to read, understand and therefore maintain than shell scripts.
> While power-users of shell, like yourself, Thomas, can come up with some
> amazing stuff done in shell, it tends to make them very hard to follow for
> us mere mortals.

Some processing are simpler in Python.
But here, the python script is mostly calling some shell commands and
filtering like grep.

For comparison, python version:
	http://dpdk.org/ml/archives/dev/2018-February/090772.html
shell version:
	http://dpdk.org/ml/archives/dev/2018-May/102993.html
Bruce Richardson May 28, 2018, 1:17 p.m. | #5
On Mon, May 28, 2018 at 11:31:31AM +0200, Thomas Monjalon wrote:
> 28/05/2018 11:21, Bruce Richardson:
> > On Sun, May 27, 2018 at 10:50:58PM +0200, Thomas Monjalon wrote:
> > > Sorry for having missed this patch during so long.
> > > 
> > > 13/02/2018 11:48, Ferruh Yigit:
> > > > On 2/12/2018 4:13 PM, Pavan Nikhilesh wrote:
> > > > > This script checks for the symbols specified in the map files against
> > > > > the symbols present in the map file directory.
> > > > > Currently, the script checks for map files in drivers and lib directory.
> > > > > 
> > > > > Example:
> > > > > ./devtools/check-map.py
> > > > > 
> > > > > The following symbols are unused :
> > > > > 
> > > > > Map file : /home/pavan/Work/clean/dpdk/drivers/mempool/dpaa/...
> > > > > ['rte_dpaa_pool_table']
> > > > > 
> > > > > Map file : /home/pavan/Work/clean/dpdk/drivers/bus/fslmc/...
> > > > > ['qbman_get_version', 'qbman_swp_send_multiple']
> > > > > 
> > > > > ...
> > > > > 
> > > > > Signed-off-by: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
> > > > 
> > > > +1 to improve our tools.
> > > > 
> > > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > > 
> > > I agree we must have more tools.
> > > 
> > > I think this check can be a lot simpler as a shell script,
> > > instead of Python. It does not need the list features of Python.
> > > 
> > 
> > While the shell version can be shorter, I always find python scripts to be
> > far easier to read, understand and therefore maintain than shell scripts.
> > While power-users of shell, like yourself, Thomas, can come up with some
> > amazing stuff done in shell, it tends to make them very hard to follow for
> > us mere mortals.
> 
> Some processing are simpler in Python.
> But here, the python script is mostly calling some shell commands and
> filtering like grep.
> 
> For comparison, python version:
> 	http://dpdk.org/ml/archives/dev/2018-February/090772.html
> shell version:
> 	http://dpdk.org/ml/archives/dev/2018-May/102993.html
> 

Well, the python script does have a call to grep in it, true. I'm not going
to have a problem either way, the python version still reads a little
easier, but the shell version is shorter and looks more natural, I suppose.
I'm happy enough to go along with whatever others (who may care) think.

/Bruce

Patch

diff --git a/devtools/check-map.py b/devtools/check-map.py
new file mode 100755
index 000000000..a508469d6
--- /dev/null
+++ b/devtools/check-map.py
@@ -0,0 +1,56 @@ 
+#! /usr/bin/env python
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017 Cavium, Inc
+#
+
+import subprocess
+import re
+import os
+from glob import glob
+
+paths = ['drivers', 'lib']
+map_files = []
+
+for path in paths:
+    map_files = map_files + [y
+                             for x in os.walk(os.environ['PWD'] + '/' + path)
+                             for y in glob(os.path.join(x[0],
+                                                        '*.map'))]
+
+print("\nThe following symbols are unused : \n")
+for map_file in map_files:
+    path = os.path.dirname(os.path.abspath(map_file))
+    with open(map_file) as f:
+        map_content = [line.strip() for line in f]
+    # Filter empty lines.
+    map_content = list(filter(None, map_content))
+    regex = re.compile(
+            r"""
+            \bDPDK.*\b |
+            \bglobal.*\b |
+            \blocal.*\b | \}.* |
+            \bEXPERIMENTAL\b |
+            \bper_lcore.*\b""",
+            flags=re.I | re.X | re.VERBOSE)
+    filtered_map = filter(lambda i: not regex.search(i), map_content)
+    filtered_map = [i for i in map_content if not regex.search(i)]
+    filtered_map = [s.replace(';', '') for s in filtered_map]
+    # Filter out known lines.
+    if not filtered_map:
+        continue
+    not_found = []
+    # Gerp in the map file directory for each symbol
+    for map_symbol in filtered_map:
+        try:
+            op = subprocess.check_output(['grep', '-nr', '--exclude',
+                                          os.path.basename(map_file),
+                                          map_symbol,
+                                          path])
+        except subprocess.CalledProcessError as ex:
+            not_found.append(map_symbol)
+            rc = ex.returncode
+            if rc != 1:
+                raise
+    if not_found:
+        print("Map file : %s" % map_file)
+        print("%s\n" % not_found)