pycommons.dev.doc package

Tools for making the documentation.

In all of my projects, the documentation is generated in the same way from the same data. Therefore, this process is unified here, which makes it easier to apply changes throughout all projects.

The general idea is that we take all the information from the setup.cfg, README.md, and version.py files and use it to automatically fill the parameters of sphinx and to construct index.rst and a variant of README.md that the myst parser used by sphinx can properly render.

Submodules

pycommons.dev.doc.doc_info module

The documentation information.

The DocInfo holds the basic data needed to generate documentation in a unified way. For now, this data is loaded from the setup.cfg file, in which the process expects to find references to a version.py and README.md file. Then, it loads the information from these files as well. This spares us the trouble of defining the information in several places and keeping it synchronized.

class pycommons.dev.doc.doc_info.DocInfo(readme_md, project, author, title, version, last_major_section_index, doc_url)[source]

Bases: object

A class that represents information about documentation.

>>> di = DocInfo(__file__, "a", "b", "c", "1", 12, "https://example.com")
>>> di.doc_url
'https://example.com'
>>> di.last_major_section_index
12
>>> di.project
'a'
>>> di.author
'b'
>>> di.title
'c'
>>> di.readme_md_file[-11:]
'doc_info.py'

>>> di = DocInfo(__file__, "a", "b", "c", "1", None,
...             "https://example.com")
>>> di.doc_url
'https://example.com'
>>> print(di.last_major_section_index)
None
>>> di.title
'c'
>>> di.readme_md_file[-11:]
'doc_info.py'

>>> try:
...     DocInfo(None, "a", "b", "c", "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor '__len__' requires a 'str' object but received a 'NoneType'

>>> try:
...     DocInfo(1, "a", "b", "c", "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor '__len__' requires a 'str' object but received a 'int'

>>> try:
...     DocInfo(__file__, None, "b", "c", "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'NoneType' object

>>> try:
...     DocInfo(__file__, 1, "b", "c", "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'int' object

>>> try:
...     DocInfo(__file__, " ", "b", "c", "1", 12, "https://example.com")
... except ValueError as ve:
...     print(ve)
Invalid project name ' '.

>>> try:
...     DocInfo(__file__, "a", None, "c", "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'NoneType' object

>>> try:
...     DocInfo(__file__, "a", 1, "c", "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'int' object

>>> try:
...     DocInfo(__file__, "a", " ", "c", "1", 12, "https://example.com")
... except ValueError as ve:
...     print(ve)
Invalid author name ' '.

>>> try:
...     DocInfo(__file__, "a", "b", None, "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'NoneType' object

>>> try:
...     DocInfo(__file__, "a", "b", 1, "1", 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'int' object

>>> try:
...     DocInfo(__file__, "b", "c", " ", "1", 12, "https://example.com")
... except ValueError as ve:
...     print(ve)
Invalid title ' '.

>>> try:
...     DocInfo(__file__, "a", "b", "c", None, 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'NoneType' object

>>> try:
...     DocInfo(__file__, "a", "b", "c", 1, 12, "https://example.com")
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'int' object

>>> try:
...     DocInfo(__file__, "a", "b", "c", "x", 12, "https://example.com")
... except ValueError as ve:
...     print(str(ve)[:64])
Invalid version 'x': Cannot convert version='x' to int, let alon

>>> try:
...     DocInfo(__file__, "a", "b", "c", " ", 12, "https://example.com")
... except ValueError as ve:
...     print(str(ve)[:64])
Invalid version ' ': empty or only white space.

>>> try:
...     DocInfo(__file__, "a", "b", "c", "1.x", 12, "https://example.com")
... except ValueError as ve:
...     print(str(ve)[:64])
Invalid version '1.x': Cannot convert version='x' to int, let al

>>> try:
...     DocInfo(__file__, "a", "b", "c", "-1", 12, "https://example.com")
... except ValueError as ve:
...     print(str(ve)[:64])
Invalid version '-1': version=-1 is invalid, must be in 0..10000

>>> try:
...     DocInfo(__file__, "a", "b", "c", "0.-1", 12, "https://example.com")
... except ValueError as ve:
...     print(str(ve)[:64])
Invalid version '0.-1': version=-1 is invalid, must be in 0..100

>>> try:
...     DocInfo(__file__, "a", "b", "c", "1.2", "x",
...             "https://example.com")
... except TypeError as te:
...     print(str(te)[:60])
last_major_section_index should be an instance of int but is

>>> try:
...     DocInfo(__file__, "a", "b", "c", "1.2", 0, "https://example.com")
... except ValueError as ve:
...     print(ve)
last_major_section_index=0 is invalid, must be in 1..1000000.

>>> try:
...     DocInfo(__file__, "a", "b", "c", "1.2", 12, None)
... except TypeError as te:
...     print(te)
descriptor '__len__' requires a 'str' object but received a 'NoneType'

>>> try:
...     DocInfo(__file__, "a", "b", "c", "1.2", 12, 1)
... except TypeError as te:
...     print(te)
descriptor '__len__' requires a 'str' object but received a 'int'

>>> try:
...     DocInfo(__file__, "a", "b", "c", "1.2", 12, "dfg")
... except ValueError as ve:
...     print(ve)
URL part '' has invalid length 0.
author: str

The author name.

doc_url: URL

the base URL of the documentation

last_major_section_index: int | None

The index of the last major section

project: str

The project name.

readme_md_file: Path

The readme.md file.

title: str

The documentation title.

version: str

The version string.

pycommons.dev.doc.doc_info.extract_md_infos(readme_md_file)[source]

Parse a README.md file and find the title and last section index.

Parameters:

readme_md_file (str) – the path to the README.md

Return type:

tuple[str, int | None]

Returns:

a tuple of the title (headline starting with “# “ (but without the “# “), and the last section index, if any)

>>> from os.path import join, dirname
>>> from contextlib import redirect_stdout
>>> with redirect_stdout(None):
...     t = extract_md_infos(join(dirname(dirname(dirname(dirname(
...                          __file__)))), "README.md"))
>>> print(t)
('*pycommons:* Common Utility Functions for Python Projects.', 5)
pycommons.dev.doc.doc_info.load_doc_info_from_setup_cfg(setup_cfg_file)[source]

Load the documentation information from the setup.cfg file.

Parameters:

setup_cfg_file (str) – the path to the setup.cfg file.

Return type:

DocInfo

Returns:

the documentation information

>>> from os.path import dirname, join
>>> from contextlib import redirect_stdout
>>> with redirect_stdout(None):
...     r = load_doc_info_from_setup_cfg(join(dirname(dirname(dirname(
...         dirname(__file__)))), "setup.cfg"))
>>> r.title
'*pycommons:* Common Utility Functions for Python Projects.'
>>> r.doc_url
'https://thomasweise.github.io/pycommons'
>>> r.project
'pycommons'
>>> r.author
'Thomas Weise'
pycommons.dev.doc.doc_info.parse_version_py(version_file, version_attr='__version__')[source]

Parse a version.py file and return the version string.

Parameters:

  • version_file (str) – the path to the version file

  • version_attr (str, default: '__version__') – the version attribute

Return type:

str

Returns:

the version string

>>> from os.path import join, dirname
>>> from contextlib import redirect_stdout
>>> with redirect_stdout(None):
...     s = parse_version_py(join(dirname(dirname(dirname(__file__))),
...         "version.py"))
>>> print(s[:s.rindex(".")])
0.8

>>> try:
...     parse_version_py(None, "v")
... except TypeError as te:
...     print(te)
descriptor '__len__' requires a 'str' object but received a 'NoneType'

>>> try:
...     parse_version_py(1, "v")
... except TypeError as te:
...     print(te)
descriptor '__len__' requires a 'str' object but received a 'int'

>>> try:
...     parse_version_py(__file__, None)
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'NoneType' object

>>> try:
...     parse_version_py(__file__, 1)
... except TypeError as te:
...     print(te)
descriptor 'strip' for 'str' objects doesn't apply to a 'int' object

>>> try:
...     parse_version_py(__file__, "")
... except ValueError as ve:
...     print(ve)
Invalid version attr ''.

>>> from contextlib import redirect_stdout
>>> from io import StringIO
>>> try:
...     with redirect_stdout(None):
...         parse_version_py(__file__, "xyz")
... except ValueError as ve:
...     print(str(ve)[:36])
Did not find version attr 'xyz' in '

pycommons.dev.doc.index_rst module

Make an index.rst file.

In all of my projects, the index.rst files have the same contents, basically. So here we generate them on the fly based on the documentation information data. Since this data contains the index of the last section in the README.md files, this allows me to properly number the code section.

pycommons.dev.doc.index_rst.make_index_rst(info, collector)[source]

Create the index.rst file contents.

Parameters:

  • info (DocInfo) – The documentation information

  • collector (Callable[[str], Any]) – the collector to receive the information.

Return type:

None

>>> di = DocInfo(__file__, "a", "b", "bla", "1.2", 12,
...              "https://example.com")
>>> from contextlib import redirect_stdout
>>> l = []
>>> with redirect_stdout(None):
...     make_index_rst(di, l.append)
>>> for s in l:
...     print(s)
bla
===

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

.. include:: README.md
   :parser: myst_parser.sphinx_

13. Modules and Code
--------------------

.. toctree::
   :maxdepth: 4

   modules

>>> try:
...     make_index_rst(None, print)
... except TypeError as te:
...     print(str(te)[:70])
info should be an instance of pycommons.dev.doc.doc_info.DocInfo but i

>>> try:
...     make_index_rst(1, print)
... except TypeError as te:
...     print(str(te)[:70])
info should be an instance of pycommons.dev.doc.doc_info.DocInfo but i

>>> try:
...     make_index_rst(di, None)
... except TypeError as te:
...     print(str(te))
collector should be a callable but is None.

>>> try:
...     make_index_rst(di, 1)
... except TypeError as te:
...     print(str(te))
collector should be a callable but is int, namely 1.

pycommons.dev.doc.process_md module

Process a markdown file in order to make it useful for distribution.

In order to let sphinx properly load and insert the README.md file into the project’s documentation, we need to process this file from the GitHub style markdown to a variant suitable for the myst parser used in sphinx. While we are at it, we can also turn absolute URLs from the GitHub-README.md file that point to the documentation URL to relative URLs.

pycommons.dev.doc.process_md.process_markdown_for_sphinx(source, dest, base_urls=None, full_urls=None, discard_until='## 1. Introduction')[source]

Process a markdown file in order to make it useful for distribution.

This process changes the GitHub-style markdown to a format that the myst parser, which is used by sphinx, can render properly. This involves several issues:

  1. We discard the top-level heading.

  2. We need to move all sub-headings one step up.

  3. Furthermore, we can turn all absolute URLs pointing to the documentation website to local references starting with ./.

  4. The myst parser drops the numerical prefixes of links, i.e., it tags ## 1.2. Hello with id hello instead of 12-hello. This means that we need to fix all references following the pattern [xxx](#12-hello) to [xxx](#hello).

Parameters:

  • source (Iterable[str]) – the source line iterable

  • dest (Callable[[str], Any]) – the destination callable receiving the output

  • base_urls (Optional[Mapping[str, str]], default: None) – a mapping of basic urls to shortcuts

  • full_urls (Optional[Mapping[str, str]], default: None) – a mapping of full urls to abbreviations

  • discard_until (str | None, default: '## 1. Introduction') – discard all strings until reaching this line. If this is None, all lines will be used. If this is not None, then this will be the first line to be forwarded to dest

Return type:

None

>>> lp = list()
>>> src = ["![image](https://example.com/1.jp)",
...        "# This is `pycommons!`",
...        "Table of contents",
...        "## 1. Introduction",
...        "blabla bla <https://example.com/A>!",
...        "## 2. Some More Text",
...        "We [also say](https://example.com/z/hello.txt) stuff.",
...        "### 2.4. Code Example",
...        "```",
...        "But [not in code](https://example.com/z/hello.txt).",
...        "```",
...        "See also [here](#24-code-example)."]
>>> process_markdown_for_sphinx(src, print,
...     {"https://example.com/": "./"},
...     {"https://example.com/A": "xyz"})
# 1. Introduction
blabla bla <xyz>!
# 2. Some More Text
We [also say](./z/hello.txt) stuff.
## 2.4. Code Example
```
But [not in code](https://example.com/z/hello.txt).
```
See also [here](#code-example).

>>> try:
...     process_markdown_for_sphinx(None, print)
... except TypeError as te:
...     print(te)
source should be an instance of typing.Iterable but is None.

>>> try:
...     process_markdown_for_sphinx(1, print)
... except TypeError as te:
...     print(te)
source should be an instance of typing.Iterable but is int, namely 1.

>>> try:
...     process_markdown_for_sphinx([None], print)
... except TypeError as te:
...     print(te)
descriptor 'rstrip' for 'str' objects doesn't apply to a 'NoneType' object

>>> try:
...     process_markdown_for_sphinx([1], print)
... except TypeError as te:
...     print(te)
descriptor 'rstrip' for 'str' objects doesn't apply to a 'int' object

>>> try:
...     process_markdown_for_sphinx([""], None)
... except TypeError as te:
...     print(te)
dest should be a callable but is None.

>>> try:
...     process_markdown_for_sphinx([""], 1)
... except TypeError as te:
...     print(te)
dest should be a callable but is int, namely 1.

>>> try:
...     process_markdown_for_sphinx([""], print, 1, None, "bla")
... except TypeError as te:
...     print(te)
base_urls should be an instance of typing.Mapping but is int, namely 1.

>>> try:
...     process_markdown_for_sphinx([""], print, None, 1, "bla")
... except TypeError as te:
...     print(te)
full_urls should be an instance of typing.Mapping but is int, namely 1.

>>> try:
...     process_markdown_for_sphinx([""], print, None, None, 1)
... except TypeError as te:
...     print(te)
descriptor '__len__' requires a 'str' object but received a 'int'

>>> try:
...     process_markdown_for_sphinx([""], print, None, None, "")
... except ValueError as ve:
...     print(ve)
discard_until cannot be ''.

>>> process_markdown_for_sphinx([""], print, None, None, None)

pycommons.dev.doc.setup_doc module

Set up the documentation builder in a unified way.

pycommons.dev.doc.setup_doc.setup_doc(doc_dir, root_dir, copyright_start_year=None, dependencies=None, base_urls=None, full_urls=None, static_paths=None)[source]

Set up the documentation building process in a unified way.

This function must be called directly from the conf.py script. It will configure the sphinx documentation generation engine using my default settings. It can automatically link to several standard dependencies, render the README.md file of the project to a format that the myst parser used by sphinx can understand, fix absolute URLs in the README.md file that point to the documentation URL to relative links, get the documentation URL, the project title, author, and version from the root setup.cfg file from which it traces to the README.md and the version.py file of the project, and also construct an index.rst file. All in all, you will end up with a very unified setup for the documentation generation. Nothing fancy, but it will work and work the same in all of my projects without the need to copy and maintain boilerplate code.

Parameters:

  • doc_dir (str) – the folder where the documentation is to be built.

  • root_dir (str) – the root path of the project.

  • copyright_start_year (int | None, default: None) – the copyright start year

  • dependencies (Optional[Iterable[str | tuple[str, str]]], default: None) – the external libraries to use

  • base_urls (Optional[Mapping[str, str]], default: None) – a mapping of basic urls to shortcuts

  • full_urls (Optional[Mapping[str, str]], default: None) – a mapping of full urls to abbreviations

  • static_paths (Optional[Iterable[str]], default: None) – a list of static paths, if there are any

Return type:

None