[RFC,v2,0/4] dts: add dts api docs

  Augment the meson build system with dts api generation. The api docs are
generated from Python docstrings in DTS using Sphinx. The format of
choice is the Google format [0].

The guide html sphinx configuration is used to preserve the same style.

The build requires the same Python version and dependencies as DTS,
because Sphinx imports the Python modules. Dependencies are installed
using Poetry from the dts directory:

poetry install --with docs

After installing, enter the Poetry shell:

poetry shell

And then run the build:
ninja -C <meson_build_dir> dts/doc

There's only one properly documented module that serves as a
demonstration of the style - framework.testbed_model.node.

[0] https://google.github.io/styleguide/pyguide.html#s3.8.4-comments-in-classes

Juraj Linkeš (4):
  dts: code adjustments for sphinx
  dts: add doc generation dependencies
  dts: add doc generation
  dts: format docstrigs to google format

 doc/api/meson.build                           |   1 +
 doc/guides/conf.py                            |  22 +-
 doc/guides/meson.build                        |   1 +
 doc/guides/tools/dts.rst                      |  29 +
 dts/doc/doc-index.rst                         |  20 +
 dts/doc/meson.build                           |  50 ++
 dts/framework/config/__init__.py              |  11 +
 .../{testbed_model/hw => config}/cpu.py       |  13 +
 dts/framework/dts.py                          |   8 +-
 dts/framework/remote_session/__init__.py      |   3 +-
 dts/framework/remote_session/linux_session.py |   2 +-
 dts/framework/remote_session/os_session.py    |  12 +-
 .../remote_session/remote/__init__.py         |  16 -
 .../{remote => }/remote_session.py            |   0
 .../{remote => }/ssh_session.py               |   0
 dts/framework/settings.py                     |  55 +-
 dts/framework/testbed_model/__init__.py       |  10 +-
 dts/framework/testbed_model/hw/__init__.py    |  27 -
 dts/framework/testbed_model/node.py           | 164 ++--
 dts/framework/testbed_model/sut_node.py       |   9 +-
 .../testbed_model/{hw => }/virtual_device.py  |   0
 dts/main.py                                   |   3 +-
 dts/meson.build                               |  16 +
 dts/poetry.lock                               | 770 ++++++++++++++++--
 dts/pyproject.toml                            |   7 +
 dts/tests/TestSuite_hello_world.py            |   6 +-
 meson.build                                   |   1 +
 meson_options.txt                             |   2 +
 28 files changed, 1038 insertions(+), 220 deletions(-)
 create mode 100644 dts/doc/doc-index.rst
 create mode 100644 dts/doc/meson.build
 rename dts/framework/{testbed_model/hw => config}/cpu.py (95%)
 delete mode 100644 dts/framework/remote_session/remote/__init__.py
 rename dts/framework/remote_session/{remote => }/remote_session.py (100%)
 rename dts/framework/remote_session/{remote => }/ssh_session.py (100%)
 delete mode 100644 dts/framework/testbed_model/hw/__init__.py
 rename dts/framework/testbed_model/{hw => }/virtual_device.py (100%)
 create mode 100644 dts/meson.build
  

On Thu, May 04, 2023 at 02:37:45PM +0200, Juraj Linkeš wrote:
> Augment the meson build system with dts api generation. The api docs are
> generated from Python docstrings in DTS using Sphinx. The format of
> choice is the Google format [0].
> 
> The guide html sphinx configuration is used to preserve the same style.
> 
> The build requires the same Python version and dependencies as DTS,
> because Sphinx imports the Python modules. Dependencies are installed
> using Poetry from the dts directory:
> 
> poetry install --with docs
> 
> After installing, enter the Poetry shell:
> 
> poetry shell
> 
> And then run the build:
> ninja -C <meson_build_dir> dts/doc
> 
> There's only one properly documented module that serves as a
> demonstration of the style - framework.testbed_model.node.
> 
> [0] https://google.github.io/styleguide/pyguide.html#s3.8.4-comments-in-classes
> 
> Juraj Linkeš (4):
>   dts: code adjustments for sphinx
>   dts: add doc generation dependencies
>   dts: add doc generation
>   dts: format docstrigs to google format
> 

I find the requirement to use poetry to build the docs, and the need to run
specific commands in specific directories quite awkward. With this patchset
there is no ability to just turn on the build option for the DTS doc and
have the docs built on the next rebuild. [Also, with every build I've tried
I can't get it to build without warnings about missing "warlock" module.]

From what I understand from the patchset, the doc building here using
sphinx is primarily concerned with building the API docs. The rest of DPDK
uses doxygen for this, and since doxygen supports python can the same
tooling be used for the DTS docs?

/Bruce
  

On Fri, May 5, 2023 at 4:07 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Thu, May 04, 2023 at 02:37:45PM +0200, Juraj Linkeš wrote:
> > Augment the meson build system with dts api generation. The api docs are
> > generated from Python docstrings in DTS using Sphinx. The format of
> > choice is the Google format [0].
> >
> > The guide html sphinx configuration is used to preserve the same style.
> >
> > The build requires the same Python version and dependencies as DTS,
> > because Sphinx imports the Python modules. Dependencies are installed
> > using Poetry from the dts directory:
> >
> > poetry install --with docs
> >
> > After installing, enter the Poetry shell:
> >
> > poetry shell
> >
> > And then run the build:
> > ninja -C <meson_build_dir> dts/doc
> >
> > There's only one properly documented module that serves as a
> > demonstration of the style - framework.testbed_model.node.
> >
> > [0] https://google.github.io/styleguide/pyguide.html#s3.8.4-comments-in-classes
> >
> > Juraj Linkeš (4):
> >   dts: code adjustments for sphinx
> >   dts: add doc generation dependencies
> >   dts: add doc generation
> >   dts: format docstrigs to google format
> >
>
> I find the requirement to use poetry to build the docs, and the need to run
> specific commands in specific directories quite awkward. With this patchset
> there is no ability to just turn on the build option for the DTS doc and
> have the docs built on the next rebuild. [Also, with every build I've tried
> I can't get it to build without warnings about missing "warlock" module.]
>
> From what I understand from the patchset, the doc building here using
> sphinx is primarily concerned with building the API docs. The rest of DPDK
> uses doxygen for this, and since doxygen supports python can the same
> tooling be used for the DTS docs?
>

I don't think any tool for python API docs would be able to do it
without the dependencies. The standard way to document python code is
in Python docstrings which are accessible during runtime (which is why
the dependencies are needed). Doxygen says that as well:
For Python there is a standard way of documenting the code using so
called documentation strings ("""). Such strings are stored in __doc__
and can be retrieved at runtime. Doxygen will extract such comments
and assume they have to be represented in a preformatted way.

There may be a tool that doesn't use the __doc__ attribute accessible
during runtime (I don't think anyone would implement something like
that though), but that would likely be much worse than Sphinx.

Juraj

> /Bruce
  

On Fri, May 5, 2023 at 4:07 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Thu, May 04, 2023 at 02:37:45PM +0200, Juraj Linkeš wrote:
> > Augment the meson build system with dts api generation. The api docs are
> > generated from Python docstrings in DTS using Sphinx. The format of
> > choice is the Google format [0].
> >
> > The guide html sphinx configuration is used to preserve the same style.
> >
> > The build requires the same Python version and dependencies as DTS,
> > because Sphinx imports the Python modules. Dependencies are installed
> > using Poetry from the dts directory:
> >
> > poetry install --with docs
> >
> > After installing, enter the Poetry shell:
> >
> > poetry shell
> >
> > And then run the build:
> > ninja -C <meson_build_dir> dts/doc
> >
> > There's only one properly documented module that serves as a
> > demonstration of the style - framework.testbed_model.node.
> >
> > [0] https://google.github.io/styleguide/pyguide.html#s3.8.4-comments-in-classes
> >
> > Juraj Linkeš (4):
> >   dts: code adjustments for sphinx
> >   dts: add doc generation dependencies
> >   dts: add doc generation
> >   dts: format docstrigs to google format
> >
>
> I find the requirement to use poetry to build the docs, and the need to run
> specific commands in specific directories quite awkward. With this patchset
> there is no ability to just turn on the build option for the DTS doc and
> have the docs built on the next rebuild. [Also, with every build I've tried
> I can't get it to build without warnings about missing "warlock" module.]
>

I want to ask about the warnings. This suggests a problem with
dependencies, have you entered the Poetry shell? We may need to add
some steps to docs, which are currently:

poetry install --with docs
poetry shell

And then:
ninja -C build dts/doc

Maybe the problem is with Poetry version (1.4.2 and higher should
work), which is not specified in the docs yet. I need to update
http://patches.dpdk.org/project/dpdk/patch/20230331091355.1224059-1-juraj.linkes@pantheon.tech/
with it.

Which are your exact steps for building the docs?

Juraj

> From what I understand from the patchset, the doc building here using
> sphinx is primarily concerned with building the API docs. The rest of DPDK
> uses doxygen for this, and since doxygen supports python can the same
> tooling be used for the DTS docs?
>
> /Bruce