Sphinx Testify

sphinx-testify is an extension to Sphinx that allows testifying documentation: ensuring that the documentation is up-to-date an aligned with the library or application capabilities.

Installation

To install, run

pip install sphinx-testify

Or get it from the source at https://github.com/basicwolf/sphinx-testify

Basic configuration

To enable sphinx-testify, add it to the list of extensions in Sphinx project conf.py and add a path to JUnit XML -formatted tests result file. For example:

extensions = [
    'sphinx_testify'
]

testify_from = [
    '/path/to/test_results.xml'
]

Usage

Sphinx-testify provides a Sphinx directive, which verifies whether the given test succeeded and fails the documentation build otherwise.

For example, consider a test which verifies that only registered users can access the system. The test results are available in JUnit XML format:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite name="testsuite">
        <testcase classname="testclass"
                  name="test_only_registered_users_have_access" />
    </testsuite>
</testsuites>

You document and testify this behaviour via Sphinx as follows:

Access to the system is restricted to registered users.

.. testify::
   testsuite.testclass.test_only_registered_users_have_access

The directive is not rendered and does not affect the output HTML in any way. However, documentation build fails if the specified test failed or is missing from the test results XML report.

A full example

Sphinx-testify does what it preaches: this very documentation is testified! You can find out more by reading the source code.

Test report parser

The test report parser module consumes JUnit XML -formatted reports. It aims at parsing the most common output and ignore all the unnecessary data.

The test names used for testifying are constructed from Test Suite, Test Class and Test Case names separated by a dot. For example, we need to use the following long name to testify

.. testify::
   pytest.tests.test_report_parser.test_parse_one_successful_testcase

based on the following <testcase> element in the report:

<testsuite name="pytest">
    <testcase classname="tests.test_report_parser"
              name="test_parse_one_successful_testcase"/>
</testsuite>

The parser can consume nested <testsuite> elements and normalizes testcase names, so that for the following test report

<testsuite name="Tests.Authentication">
    <testcase name="testCase3" classname="Tests.Authentication" />
</testsuite>

it is enough to specify

.. testify::
   Tests.Authentication.testCase3

The parser is also tolerant for missing attributes, such as name attribute in <testsuite> or classname attribute in <testcase>. However, testify raises an error if <testcase> is missing name attribute.

Configuration parameters

testify_from

Type:

list[str]

The list of paths to test report files. The list should contain at least one entry.

testify_skip

Type:

bool

Default:

False

Completely skip verifying testified sources. This is useful in environments where you are building documentation, but running tests is not feasible.

For example, you can run tests and build documentation in CI to testify it, and skip tests and testifying in ReadTheDocs builder.

Example:

testify_skip = os.environ.get('READTHEDOCS') == 'True'