devtools: enhance the license check
Checks
Commit Message
Reformat the license/exceptions.txt file to make it easier to build
a list of exempted files.
Display all files committed in DPDK that are non compliant
with BSD-3 license.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
devtools/check-spdx-tag.sh | 59 +++++++++++++++++++++++++++-----------
license/exceptions.txt | 11 ++++---
2 files changed, 47 insertions(+), 23 deletions(-)
Comments
On Tue, 10 Dec 2024 10:10:39 +0100
David Marchand <david.marchand@redhat.com> wrote:
> +no_license_list=\
> +':^.git* :^.mailmap :^.ci/* :^README :^MAINTAINERS :^VERSION :^ABI_VERSION :^*/Kbuild '\
> +':^*/README* :^license/ :^config/ :^buildtools/ :^*/poetry.lock '\
> +':^kernel/linux/uapi/.gitignore :^kernel/linux/uapi/version :^*.cocci :^*.abignore '\
> +':^*.map :^*.ini :^*.data :^*.json :^*.cfg :^*.txt :^*.svg :^*.png'
> +
What is poetry.lock?
On Tue, Dec 10, 2024 at 6:00 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Tue, 10 Dec 2024 10:10:39 +0100
> David Marchand <david.marchand@redhat.com> wrote:
>
> > +no_license_list=\
> > +':^.git* :^.mailmap :^.ci/* :^README :^MAINTAINERS :^VERSION :^ABI_VERSION :^*/Kbuild '\
> > +':^*/README* :^license/ :^config/ :^buildtools/ :^*/poetry.lock '\
> > +':^kernel/linux/uapi/.gitignore :^kernel/linux/uapi/version :^*.cocci :^*.abignore '\
> > +':^*.map :^*.ini :^*.data :^*.json :^*.cfg :^*.txt :^*.svg :^*.png'
> > +
>
> What is poetry.lock?
I don't know.
It looks like some python packaging config for dts, and it is a generated file.
# This file is automatically @generated by Poetry 1.8.3 and should not
be changed by hand.
Cc: dts maintainers.
On Wed, 11 Dec 2024 10:00:38 +0100
David Marchand <david.marchand@redhat.com> wrote:
> On Tue, Dec 10, 2024 at 6:00 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Tue, 10 Dec 2024 10:10:39 +0100
> > David Marchand <david.marchand@redhat.com> wrote:
> >
> > > +no_license_list=\
> > > +':^.git* :^.mailmap :^.ci/* :^README :^MAINTAINERS :^VERSION :^ABI_VERSION :^*/Kbuild '\
> > > +':^*/README* :^license/ :^config/ :^buildtools/ :^*/poetry.lock '\
> > > +':^kernel/linux/uapi/.gitignore :^kernel/linux/uapi/version :^*.cocci :^*.abignore '\
> > > +':^*.map :^*.ini :^*.data :^*.json :^*.cfg :^*.txt :^*.svg :^*.png'
> > > +
> >
> > What is poetry.lock?
>
> I don't know.
> It looks like some python packaging config for dts, and it is a generated file.
> # This file is automatically @generated by Poetry 1.8.3 and should not
> be changed by hand.
>
> Cc: dts maintainers.
>
>
Put it in .gitignore then please, and the script will ignore that.
11/12/2024 15:55, Stephen Hemminger:
> On Wed, 11 Dec 2024 10:00:38 +0100
> David Marchand <david.marchand@redhat.com> wrote:
>
> > On Tue, Dec 10, 2024 at 6:00 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > On Tue, 10 Dec 2024 10:10:39 +0100
> > > David Marchand <david.marchand@redhat.com> wrote:
> > >
> > > > +no_license_list=\
> > > > +':^.git* :^.mailmap :^.ci/* :^README :^MAINTAINERS :^VERSION :^ABI_VERSION :^*/Kbuild '\
> > > > +':^*/README* :^license/ :^config/ :^buildtools/ :^*/poetry.lock '\
> > > > +':^kernel/linux/uapi/.gitignore :^kernel/linux/uapi/version :^*.cocci :^*.abignore '\
> > > > +':^*.map :^*.ini :^*.data :^*.json :^*.cfg :^*.txt :^*.svg :^*.png'
> > > > +
> > >
> > > What is poetry.lock?
> >
> > I don't know.
> > It looks like some python packaging config for dts, and it is a generated file.
> > # This file is automatically @generated by Poetry 1.8.3 and should not
> > be changed by hand.
> >
> > Cc: dts maintainers.
> >
> >
>
> Put it in .gitignore then please, and the script will ignore that.
How is it related?
I don't like adding versioned files to .gitignore.
(this poetry file is in the repository)
https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
I agree at first glance it appears like poetry.lock should go in the
.gitignore. However, the official docs state that in the case of developing
an application (like DTS), one should commit the generated poetry.lock file
to version control, as it provides a universal lock on python dependencies,
across all minor versions of python specified in the pyproject.toml, and it
also speeds up the dependency resolution process.
I gather from reading some posts online that some folks do opt to
.gitignore their poetry.lock files for shippable applications, relying
solely on the pyproject.toml as a source of truth for dependency versions.
But, again this approach does not guarantee common dependencies across all
the python minor versions supported by the project.
I will bring this up at the CI call tomorrow morning for feedback though.
On Wed, Dec 11, 2024 at 10:01 AM Thomas Monjalon <thomas@monjalon.net>
wrote:
> 11/12/2024 15:55, Stephen Hemminger:
> > On Wed, 11 Dec 2024 10:00:38 +0100
> > David Marchand <david.marchand@redhat.com> wrote:
> >
> > > On Tue, Dec 10, 2024 at 6:00 PM Stephen Hemminger
> > > <stephen@networkplumber.org> wrote:
> > > >
> > > > On Tue, 10 Dec 2024 10:10:39 +0100
> > > > David Marchand <david.marchand@redhat.com> wrote:
> > > >
> > > > > +no_license_list=\
> > > > > +':^.git* :^.mailmap :^.ci/* :^README :^MAINTAINERS :^VERSION
> :^ABI_VERSION :^*/Kbuild '\
> > > > > +':^*/README* :^license/ :^config/ :^buildtools/ :^*/poetry.lock '\
> > > > > +':^kernel/linux/uapi/.gitignore :^kernel/linux/uapi/version
> :^*.cocci :^*.abignore '\
> > > > > +':^*.map :^*.ini :^*.data :^*.json :^*.cfg :^*.txt :^*.svg
> :^*.png'
> > > > > +
> > > >
> > > > What is poetry.lock?
> > >
> > > I don't know.
> > > It looks like some python packaging config for dts, and it is a
> generated file.
> > > # This file is automatically @generated by Poetry 1.8.3 and should not
> > > be changed by hand.
> > >
> > > Cc: dts maintainers.
> > >
> > >
> >
> > Put it in .gitignore then please, and the script will ignore that.
>
> How is it related?
>
> I don't like adding versioned files to .gitignore.
> (this poetry file is in the repository)
>
>
>
On Wed, Dec 11, 2024 at 4:01 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 11/12/2024 15:55, Stephen Hemminger:
> > On Wed, 11 Dec 2024 10:00:38 +0100
> > David Marchand <david.marchand@redhat.com> wrote:
> >
> > > On Tue, Dec 10, 2024 at 6:00 PM Stephen Hemminger
> > > <stephen@networkplumber.org> wrote:
> > > >
> > > > On Tue, 10 Dec 2024 10:10:39 +0100
> > > > David Marchand <david.marchand@redhat.com> wrote:
> > > >
> > > > > +no_license_list=\
> > > > > +':^.git* :^.mailmap :^.ci/* :^README :^MAINTAINERS :^VERSION :^ABI_VERSION :^*/Kbuild '\
> > > > > +':^*/README* :^license/ :^config/ :^buildtools/ :^*/poetry.lock '\
> > > > > +':^kernel/linux/uapi/.gitignore :^kernel/linux/uapi/version :^*.cocci :^*.abignore '\
> > > > > +':^*.map :^*.ini :^*.data :^*.json :^*.cfg :^*.txt :^*.svg :^*.png'
> > > > > +
> > > >
> > > > What is poetry.lock?
> > >
> > > I don't know.
> > > It looks like some python packaging config for dts, and it is a generated file.
> > > # This file is automatically @generated by Poetry 1.8.3 and should not
> > > be changed by hand.
> > >
> > > Cc: dts maintainers.
> > >
> > >
> >
> > Put it in .gitignore then please, and the script will ignore that.
>
> How is it related?
>
> I don't like adding versioned files to .gitignore.
> (this poetry file is in the repository)
Apart from this file, any other comment?
10/12/2024 10:10, David Marchand:
> Reformat the license/exceptions.txt file to make it easier to build
> a list of exempted files.
> Display all files committed in DPDK that are non compliant
> with BSD-3 license.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Applied, thanks
@@ -4,7 +4,8 @@
#
# Produce a list of files with incorrect license tags
-errors=0
+missing_spdx=0
+wrong_license=0
warnings=0
quiet=false
verbose=false
@@ -14,23 +15,44 @@ print_usage () {
exit 1
}
+no_license_list=\
+':^.git* :^.mailmap :^.ci/* :^README :^MAINTAINERS :^VERSION :^ABI_VERSION :^*/Kbuild '\
+':^*/README* :^license/ :^config/ :^buildtools/ :^*/poetry.lock '\
+':^kernel/linux/uapi/.gitignore :^kernel/linux/uapi/version :^*.cocci :^*.abignore '\
+':^*.map :^*.ini :^*.data :^*.json :^*.cfg :^*.txt :^*.svg :^*.png'
+
check_spdx() {
- if $verbose; then
+ if $verbose ; then
echo "Files without SPDX License"
echo "--------------------------"
fi
- git grep -L SPDX-License-Identifier -- \
- ':^.git*' ':^.mailmap' ':^.ci/*' \
- ':^README' ':^MAINTAINERS' ':^VERSION' ':^ABI_VERSION' \
- ':^*/Kbuild' ':^*/README*' \
- ':^license/' ':^config/' ':^buildtools/' ':^*/poetry.lock' \
- ':^kernel/linux/uapi/.gitignore' ':^kernel/linux/uapi/version' \
- ':^*.cocci' ':^*.abignore' \
- ':^*.map' ':^*.ini' ':^*.data' ':^*.json' ':^*.cfg' ':^*.txt' \
- ':^*.svg' ':^*.png' \
- > $tmpfile
-
- errors=$(wc -l < $tmpfile)
+ git grep -L SPDX-License-Identifier -- $no_license_list > $tmpfile
+
+ missing_spdx=$(wc -l < $tmpfile)
+ $quiet || cat $tmpfile
+}
+
+build_exceptions_list() {
+ grep '.*|.*|.*|.*' license/exceptions.txt | grep -v 'TB Approval Date' |
+ while IFS='|' read license tb_date gb_date pattern ; do
+ unset IFS
+ license=${license## *}
+ license=${license%% *}
+ git grep -l "SPDX-License-Identifier:[[:space:]]*$license" $pattern |
+ sed -e 's/^/:^/'
+ done
+}
+
+check_licenses() {
+ if $verbose ; then
+ echo "Files with wrong license and no exception"
+ echo "-----------------------------------------"
+ fi
+ exceptions=$(build_exceptions_list)
+ git grep -l SPDX-License-Identifier: -- $no_license_list $exceptions |
+ xargs grep -L -E 'SPDX-License-Identifier:[[:space:]]*\(?BSD-3-Clause' > $tmpfile
+
+ wrong_license=$(wc -l < $tmpfile)
$quiet || cat $tmpfile
}
@@ -64,8 +86,11 @@ trap 'rm -f -- "$tmpfile"' INT TERM HUP EXIT
check_spdx
$quiet || echo
-check_boilerplate
+check_licenses
+$quiet || echo
+check_boilerplate
$quiet || echo
-echo "total: $errors errors, $warnings warnings"
-exit $errors
+
+echo "total: $missing_spdx missing SPDX errors, $wrong_license license errors, $warnings warnings"
+exit $((missing_spdx + wrong_license))
@@ -10,11 +10,10 @@ Note that following licenses are not exceptions:-
- GPL-2.0 (*Only for kernel code*)
---------------------------------------------------------------------------------------------------
-SPDX Identifier TB Approval Date GB Approval Date File name
+SPDX Identifier | TB Approval Date | GB Approval Date | File name
---------------------------------------------------------------------------------------------------
-1.MIT 10/23/2019 02/10/2020 lib/eal/windows/include/dirent.h
-2.BSD-2-Clause 10/23/2019 12/18/2019 lib/eal/windows/include/getopt.h
-3.ISC AND
- BSD-2-Clause 10/23/2019 12/18/2019 lib/eal/windows/getopt.c
-4. MIT 10/19/2022 10/18/2022 drivers/net/gve/base/*
+MIT | 10/23/2019 | 02/10/2020 | lib/eal/windows/include/dirent.h
+BSD-2-Clause | 10/23/2019 | 12/18/2019 | lib/eal/windows/include/getopt.h
+ISC AND BSD-2-Clause | 10/23/2019 | 12/18/2019 | lib/eal/windows/getopt.c
+MIT | 10/19/2022 | 10/18/2022 | drivers/net/gve/base/*
---------------------------------------------------------------------------------------------------