[v4,0/3] Split logging functionality out of EAL

  There is a general desire to reduce the size and scope of EAL. To this
end, this patchset makes a (very) small step in that direction by taking
the logging functionality out of EAL and putting it into its own library
that can be built and maintained separately.

As with the first RFC for this, the main obstacle is the "fnmatch"
function which is needed by both EAL and the new log function when
building on windows. While the function cannot stay in EAL - or we would
have a circular dependency, moving it to a new library or just putting
it in the log library have the disadvantages that it then "leaks" into
the public namespace without an rte_prefix, which could cause issues.
Since only a single function is involved, subsequent versions take a
different approach to v1, and just moves the offending function to be a
static function in a header file. This allows use by multiple libs
without conflicting names or making it public.

The other complication, as explained in v1 RFC was that of multiple
implementations for different OS's. This is solved here in the same
way as v1, by including the OS in the name and having meson pick the
correct file for each build. Since only one file is involved, there
seemed little need for replicating EAL's separate subdirectories
per-OS.

v4:
* Fixed windows build error, due to missing strdup (_strdup on windows)
* Added doc updates to programmers guide.

v3:
* Fixed missing log file for BSD
* Removed "eal" from the filenames of files in the log directory
* added prefixes to elements in the fnmatch header to avoid conflicts
* fixed space indentation in new lines in telemetry.c (checkpatch)
* removed "extern int logtype" definition in telemetry.c (checkpatch)
* added log directory to list for doxygen scanning

Bruce Richardson (3):
  eal/windows: move fnmatch function to header file
  log: separate logging functions out of EAL
  telemetry: use standard logging

 doc/api/doxy-api.conf.in                      |   1 +
 .../prog_guide/env_abstraction_layer.rst      |   4 +-
 doc/guides/prog_guide/index.rst               |   1 +
 doc/guides/prog_guide/log_lib.rst             | 115 ++++++++++++
 lib/eal/common/eal_common_options.c           |   2 +-
 lib/eal/common/eal_private.h                  |   7 -
 lib/eal/common/meson.build                    |   1 -
 lib/eal/freebsd/eal.c                         |   6 +-
 lib/eal/include/meson.build                   |   1 -
 lib/eal/linux/eal.c                           |   8 +-
 lib/eal/linux/meson.build                     |   1 -
 lib/eal/meson.build                           |   2 +-
 lib/eal/version.map                           |  17 --
 lib/eal/windows/eal.c                         |   2 +-
 lib/eal/windows/fnmatch.c                     | 172 -----------------
 lib/eal/windows/include/fnmatch.h             | 175 ++++++++++++++++--
 lib/eal/windows/meson.build                   |   2 -
 lib/kvargs/meson.build                        |   3 +-
 .../common/eal_common_log.c => log/log.c}     |   7 +-
 lib/log/log_freebsd.c                         |  12 ++
 .../common/eal_log.h => log/log_internal.h}   |  18 +-
 lib/{eal/linux/eal_log.c => log/log_linux.c}  |   2 +-
 .../windows/eal_log.c => log/log_windows.c}   |   2 +-
 lib/log/meson.build                           |   9 +
 lib/{eal/include => log}/rte_log.h            |   0
 lib/log/version.map                           |  34 ++++
 lib/meson.build                               |   1 +
 lib/telemetry/meson.build                     |   3 +-
 lib/telemetry/telemetry.c                     |  11 +-
 lib/telemetry/telemetry_internal.h            |   3 +-
 30 files changed, 370 insertions(+), 252 deletions(-)
 create mode 100644 doc/guides/prog_guide/log_lib.rst
 delete mode 100644 lib/eal/windows/fnmatch.c
 rename lib/{eal/common/eal_common_log.c => log/log.c} (99%)
 create mode 100644 lib/log/log_freebsd.c
 rename lib/{eal/common/eal_log.h => log/log_internal.h} (69%)
 rename lib/{eal/linux/eal_log.c => log/log_linux.c} (97%)
 rename lib/{eal/windows/eal_log.c => log/log_windows.c} (93%)
 create mode 100644 lib/log/meson.build
 rename lib/{eal/include => log}/rte_log.h (100%)
 create mode 100644 lib/log/version.map

--
2.37.2
  

Hi Bruce,

On Fri, Jan 20, 2023 at 7:22 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> There is a general desire to reduce the size and scope of EAL. To this
> end, this patchset makes a (very) small step in that direction by taking
> the logging functionality out of EAL and putting it into its own library
> that can be built and maintained separately.
>
> As with the first RFC for this, the main obstacle is the "fnmatch"
> function which is needed by both EAL and the new log function when
> building on windows. While the function cannot stay in EAL - or we would
> have a circular dependency, moving it to a new library or just putting
> it in the log library have the disadvantages that it then "leaks" into
> the public namespace without an rte_prefix, which could cause issues.
> Since only a single function is involved, subsequent versions take a
> different approach to v1, and just moves the offending function to be a
> static function in a header file. This allows use by multiple libs
> without conflicting names or making it public.
>
> The other complication, as explained in v1 RFC was that of multiple
> implementations for different OS's. This is solved here in the same
> way as v1, by including the OS in the name and having meson pick the
> correct file for each build. Since only one file is involved, there
> seemed little need for replicating EAL's separate subdirectories
> per-OS.

There is another complication.

The ABI check is not handling properly the case where symbols are
moved to the new log library (even though the dependency to librte_log
is explicit in librte_eal elf).
For now, I don't have a good way to handle this.

A workaround to pass the check is to suppress those symbols wrt the eal dump:
[suppress_function]
        symbol_name_regexp = rte_log
[suppress_function]
        symbol_name = rte_openlog_stream
[suppress_function]
        symbol_name = rte_vlog

But this is not a good solution because we would be losing checks on
them for the rest of the v23 ABI life.
  

On Sun, Jan 22, 2023 at 03:56:12PM +0100, David Marchand wrote:
> Hi Bruce,
> 
> On Fri, Jan 20, 2023 at 7:22 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > There is a general desire to reduce the size and scope of EAL. To this
> > end, this patchset makes a (very) small step in that direction by taking
> > the logging functionality out of EAL and putting it into its own library
> > that can be built and maintained separately.
> >
> > As with the first RFC for this, the main obstacle is the "fnmatch"
> > function which is needed by both EAL and the new log function when
> > building on windows. While the function cannot stay in EAL - or we would
> > have a circular dependency, moving it to a new library or just putting
> > it in the log library have the disadvantages that it then "leaks" into
> > the public namespace without an rte_prefix, which could cause issues.
> > Since only a single function is involved, subsequent versions take a
> > different approach to v1, and just moves the offending function to be a
> > static function in a header file. This allows use by multiple libs
> > without conflicting names or making it public.
> >
> > The other complication, as explained in v1 RFC was that of multiple
> > implementations for different OS's. This is solved here in the same
> > way as v1, by including the OS in the name and having meson pick the
> > correct file for each build. Since only one file is involved, there
> > seemed little need for replicating EAL's separate subdirectories
> > per-OS.
> 
> There is another complication.
> 
> The ABI check is not handling properly the case where symbols are
> moved to the new log library (even though the dependency to librte_log
> is explicit in librte_eal elf).
> For now, I don't have a good way to handle this.
> 
> A workaround to pass the check is to suppress those symbols wrt the eal dump:
> [suppress_function]
>         symbol_name_regexp = rte_log
> [suppress_function]
>         symbol_name = rte_openlog_stream
> [suppress_function]
>         symbol_name = rte_vlog
> 
> But this is not a good solution because we would be losing checks on
> them for the rest of the v23 ABI life.
> 
Right, I got error messages from the CI job for this too, but I also have
no idea how to work around this. Perhaps we only get to move content
between libraries when we do an ABI bump? Seems a bit restrictive, though.

/Bruce
  

On Mon, Jan 23, 2023 at 3:24 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Sun, Jan 22, 2023 at 03:56:12PM +0100, David Marchand wrote:
> > Hi Bruce,
> >
> > On Fri, Jan 20, 2023 at 7:22 PM Bruce Richardson
> > <bruce.richardson@intel.com> wrote:
> > >
> > > There is a general desire to reduce the size and scope of EAL. To this
> > > end, this patchset makes a (very) small step in that direction by taking
> > > the logging functionality out of EAL and putting it into its own library
> > > that can be built and maintained separately.
> > >
> > > As with the first RFC for this, the main obstacle is the "fnmatch"
> > > function which is needed by both EAL and the new log function when
> > > building on windows. While the function cannot stay in EAL - or we would
> > > have a circular dependency, moving it to a new library or just putting
> > > it in the log library have the disadvantages that it then "leaks" into
> > > the public namespace without an rte_prefix, which could cause issues.
> > > Since only a single function is involved, subsequent versions take a
> > > different approach to v1, and just moves the offending function to be a
> > > static function in a header file. This allows use by multiple libs
> > > without conflicting names or making it public.
> > >
> > > The other complication, as explained in v1 RFC was that of multiple
> > > implementations for different OS's. This is solved here in the same
> > > way as v1, by including the OS in the name and having meson pick the
> > > correct file for each build. Since only one file is involved, there
> > > seemed little need for replicating EAL's separate subdirectories
> > > per-OS.
> >
> > There is another complication.
> >
> > The ABI check is not handling properly the case where symbols are
> > moved to the new log library (even though the dependency to librte_log
> > is explicit in librte_eal elf).
> > For now, I don't have a good way to handle this.
> >
> > A workaround to pass the check is to suppress those symbols wrt the eal dump:
> > [suppress_function]
> >         symbol_name_regexp = rte_log
> > [suppress_function]
> >         symbol_name = rte_openlog_stream
> > [suppress_function]
> >         symbol_name = rte_vlog
> >
> > But this is not a good solution because we would be losing checks on
> > them for the rest of the v23 ABI life.
> >
> Right, I got error messages from the CI job for this too, but I also have
> no idea how to work around this. Perhaps we only get to move content
> between libraries when we do an ABI bump? Seems a bit restrictive, though.

Moving symbols as you did does not seem an ABI breakage.
An application that links to eal would see the dt_needed entry for the
new log library, load it accordingly and gets the right symbols.
  

On Mon, Jan 23, 2023 at 03:31:58PM +0100, David Marchand wrote:
> On Mon, Jan 23, 2023 at 3:24 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Sun, Jan 22, 2023 at 03:56:12PM +0100, David Marchand wrote:
> > > Hi Bruce,
> > >
> > > On Fri, Jan 20, 2023 at 7:22 PM Bruce Richardson
> > > <bruce.richardson@intel.com> wrote:
> > > >
> > > > There is a general desire to reduce the size and scope of EAL. To this
> > > > end, this patchset makes a (very) small step in that direction by taking
> > > > the logging functionality out of EAL and putting it into its own library
> > > > that can be built and maintained separately.
> > > >
> > > > As with the first RFC for this, the main obstacle is the "fnmatch"
> > > > function which is needed by both EAL and the new log function when
> > > > building on windows. While the function cannot stay in EAL - or we would
> > > > have a circular dependency, moving it to a new library or just putting
> > > > it in the log library have the disadvantages that it then "leaks" into
> > > > the public namespace without an rte_prefix, which could cause issues.
> > > > Since only a single function is involved, subsequent versions take a
> > > > different approach to v1, and just moves the offending function to be a
> > > > static function in a header file. This allows use by multiple libs
> > > > without conflicting names or making it public.
> > > >
> > > > The other complication, as explained in v1 RFC was that of multiple
> > > > implementations for different OS's. This is solved here in the same
> > > > way as v1, by including the OS in the name and having meson pick the
> > > > correct file for each build. Since only one file is involved, there
> > > > seemed little need for replicating EAL's separate subdirectories
> > > > per-OS.
> > >
> > > There is another complication.
> > >
> > > The ABI check is not handling properly the case where symbols are
> > > moved to the new log library (even though the dependency to librte_log
> > > is explicit in librte_eal elf).
> > > For now, I don't have a good way to handle this.
> > >
> > > A workaround to pass the check is to suppress those symbols wrt the eal dump:
> > > [suppress_function]
> > >         symbol_name_regexp = rte_log
> > > [suppress_function]
> > >         symbol_name = rte_openlog_stream
> > > [suppress_function]
> > >         symbol_name = rte_vlog
> > >
> > > But this is not a good solution because we would be losing checks on
> > > them for the rest of the v23 ABI life.
> > >
> > Right, I got error messages from the CI job for this too, but I also have
> > no idea how to work around this. Perhaps we only get to move content
> > between libraries when we do an ABI bump? Seems a bit restrictive, though.
> 
> Moving symbols as you did does not seem an ABI breakage.
> An application that links to eal would see the dt_needed entry for the
> new log library, load it accordingly and gets the right symbols.
>
Yes, I agree. However, I also agree with you that it is risky to lose
symbol checking for the moved symbols if we need to remove them from
analysis. That said, maybe others have some ideas as to how to work around
this, or perhaps we just take the risk of disabling checking.

/Bruce
  

On Mon, Jan 23, 2023 at 3:37 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Mon, Jan 23, 2023 at 03:31:58PM +0100, David Marchand wrote:
> > On Mon, Jan 23, 2023 at 3:24 PM Bruce Richardson
> > <bruce.richardson@intel.com> wrote:
> > >
> > > On Sun, Jan 22, 2023 at 03:56:12PM +0100, David Marchand wrote:
> > > > Hi Bruce,
> > > >
> > > > On Fri, Jan 20, 2023 at 7:22 PM Bruce Richardson
> > > > <bruce.richardson@intel.com> wrote:
> > > > >
> > > > > There is a general desire to reduce the size and scope of EAL. To this
> > > > > end, this patchset makes a (very) small step in that direction by taking
> > > > > the logging functionality out of EAL and putting it into its own library
> > > > > that can be built and maintained separately.
> > > > >
> > > > > As with the first RFC for this, the main obstacle is the "fnmatch"
> > > > > function which is needed by both EAL and the new log function when
> > > > > building on windows. While the function cannot stay in EAL - or we would
> > > > > have a circular dependency, moving it to a new library or just putting
> > > > > it in the log library have the disadvantages that it then "leaks" into
> > > > > the public namespace without an rte_prefix, which could cause issues.
> > > > > Since only a single function is involved, subsequent versions take a
> > > > > different approach to v1, and just moves the offending function to be a
> > > > > static function in a header file. This allows use by multiple libs
> > > > > without conflicting names or making it public.
> > > > >
> > > > > The other complication, as explained in v1 RFC was that of multiple
> > > > > implementations for different OS's. This is solved here in the same
> > > > > way as v1, by including the OS in the name and having meson pick the
> > > > > correct file for each build. Since only one file is involved, there
> > > > > seemed little need for replicating EAL's separate subdirectories
> > > > > per-OS.
> > > >
> > > > There is another complication.
> > > >
> > > > The ABI check is not handling properly the case where symbols are
> > > > moved to the new log library (even though the dependency to librte_log
> > > > is explicit in librte_eal elf).
> > > > For now, I don't have a good way to handle this.
> > > >
> > > > A workaround to pass the check is to suppress those symbols wrt the eal dump:
> > > > [suppress_function]
> > > >         symbol_name_regexp = rte_log
> > > > [suppress_function]
> > > >         symbol_name = rte_openlog_stream
> > > > [suppress_function]
> > > >         symbol_name = rte_vlog
> > > >
> > > > But this is not a good solution because we would be losing checks on
> > > > them for the rest of the v23 ABI life.
> > > >
> > > Right, I got error messages from the CI job for this too, but I also have
> > > no idea how to work around this. Perhaps we only get to move content
> > > between libraries when we do an ABI bump? Seems a bit restrictive, though.
> >
> > Moving symbols as you did does not seem an ABI breakage.
> > An application that links to eal would see the dt_needed entry for the
> > new log library, load it accordingly and gets the right symbols.
> >
> Yes, I agree. However, I also agree with you that it is risky to lose
> symbol checking for the moved symbols if we need to remove them from
> analysis. That said, maybe others have some ideas as to how to work around
> this, or perhaps we just take the risk of disabling checking.

I opened a bz for libabigail.
https://sourceware.org/bugzilla/show_bug.cgi?id=30034

If it is handled fast enough, we may have a solution by the time 23.03
is released and we will remove this workaround for 23.07 development
(no pressure Dodji :-p).