Customizing and Extending py.test

basic test configuration

available command line options

You can see command line options by running:

py.test -h

This will display all available command line options in your specific environment.

conftest.py: project specific hooks and configuration

A unique feature of py.test are its conftest.py files which allow to:

or set particular variables to influence the testing process:

  • pytest_plugins: list of named plugins to load
  • collect_ignore: list of paths to ignore during test collection, relative to the containing conftest.py file
  • rsyncdirs: list of to-be-rsynced directories for distributed testing, relative to the containing conftest.py file.

You may put a conftest.py files in your project root directory or into your package directory if you want to add project-specific test options.

py.test loads all conftest.py files upwards from the command line file arguments. It usually looks up configuration values right-to-left, i.e. the closer conftest files will be checked first. This means you can have a conftest.py in your very home directory to have some global configuration values.

setting persistent option defaults

py.test will lookup option values in this order:

  • command line
  • conftest.py files
  • environment variables

To find out about the particular switches and type:

py.test --help-config

This will print information about all options in your environment, including your local plugins.

Temporary directories

You can create directories by calling one of two methods on the config object:

  • config.mktemp(basename): create and return a new tempdir
  • config.ensuretemp(basename): create or return a new tempdir

temporary directories are created as sub directories of a per-session testdir and will keep around the directories of the last three test runs. You can set the base temporary directory through the command line --basetemp` option. When distributing tests on the same machine, py.test takes care to configure a basetemp directory for the sub processes such that all temporary data lands below below a single per-test run basetemp directory.

Plugin basics

project specific "local" or named "global" plugins

py.test implements much of its functionality by calling well specified hooks. Python modules which contain such hook functions are called plugins. Hook functions are discovered in conftest.py files or in named plugins. conftest.py files are sometimes called "anonymous" or conftest plugins. They are useful for keeping test extensions close to your application. Named plugins are normal python modules or packages that can be distributed separately. Named plugins need to follow a naming pattern; they have an all lowercase pytest_ prefixed name. While conftest plugins are discovered automatically, named plugins must be explicitely specified.

Plugin discovery at tool startup

py.test loads plugin modules at tool startup in the following way:

  • by loading all plugins registered through setuptools entry points.
  • by reading the PYTEST_PLUGINS environment variable and importing the comma-separated list of named plugins.
  • by pre-scanning the command line for the -p name option and loading the specified plugin before actual command line parsing.
  • by loading all conftest.py plugin files as inferred by the command line invocation (test files and all of its parent directories). Note that conftest.py files from sub directories are loaded during test collection and not at tool startup.
  • by recursively loading all plugins specified by the pytest_plugins variable in a conftest.py file

Specifying plugins in a test module or plugin

You can specify plugins in a test module or a plugin like this:

pytest_plugins = "name1", "name2",

When the test module or plugin is loaded the specified plugins will be loaded. If you specify plugins without the pytest_ prefix it will be automatically added. All plugin names must be lowercase.

Writing per-project plugins (conftest.py)

The purpose of conftest.py files is to allow project-specific test configuration. They thus make for a good place to implement project-specific test related features through hooks. For example you may set the collect_ignore variable depending on a command line option by defining the following hook in a conftest.py file:

# ./conftest.py in your root or package dir
collect_ignore = ['hello', 'test_world.py']
def pytest_addoption(parser):
    parser.addoption("--runall", action="store_true", default=False)
def pytest_configure(config):
    if config.getvalue("runall"):
        collect_ignore[:] = []

Writing setuptools-registered plugins

If you want to make your plugin publically available, you can use setuptools or Distribute which both allow to register an entry point. py.test will register all objects with the pytest11 entry point. To make your plugin available you may insert the following lines in your setuptools/distribute-based setup-invocation:

# sample ./setup.py file
from setuptools import setup

setup(
    name="myproject",
    packages = ['myproject']

    # the following makes a plugin available to py.test
    entry_points = {
        'pytest11': [
            'name_of_plugin = myproject.pluginmodule',
        ]
    },
)

If a package is installed with this setup, py.test will load myproject.pluginmodule under the name_of_plugin name and use it as a plugin.

Accessing another plugin by name

If a plugin wants to collaborate with code from another plugin it can obtain a reference through the plugin manager like this:

plugin = config.pluginmanager.getplugin("name_of_plugin")

If you want to look at the names of existing plugins, use the --traceconfig option.

Important py.test hooks

py.test calls hooks functions to implement its test collection, running and reporting process. When py.test loads a plugin it validates that all hook functions conform to the hook definition specification.

The hook function name and its argument names need to match exactly but it is allowed for an implementation to accept less parameters. You'll get useful errors on mistyped hook or argument names. Read on for some introductory information on particular hooks. It's sensible to look at existing plugins so see example usages and start off with your own plugin.

command line parsing and configuration hooks

When the test tool starts up it will invoke all hooks that add command line options in the python standard optparse style.

def pytest_addoption(parser):
    """ add command line options. """"
    parser.addoption("--myopt", dest="myopt", action="store_true")

After all these hooks have been called, the command line is parser and a config object is created and another hook is invoked, for example:

def pytest_configure(config):
    config.getvalue("myopt")

When the test run finishes this corresponding finalizer hook is called:

def pytest_unconfigure(config):
    ...

adding global py.test helpers and functionality

If you want to make global helper functions or objects available to your test code you can implement:

def pytest_namespace():
    """ return dictionary with items to be made available on py.test. namespace """

All such returned items will be made available directly on the py.test namespace.

If you want to provide helpers that are specific to a test function run or need to be setup per test function run, please refer to the funcargs mechanism.

generic "runtest" hooks

Each test item is usually executed by calling the following three hooks:

pytest_runtest_setup(item)
pytest_runtest_call(item)
pytest_runtest_teardown(item)

For each of the three invocations a call object encapsulates information about the outcome of the call and is subsequently used to make a report object:

report = hook.pytest_runtest_makereport(item, call)

For example, the pytest_pdb plugin uses this hook to activate interactive debugging on failures when --pdb is specified on the command line.

Usually three reports will be generated for a single test item for each of the three runtest hooks respectively. If pytest_runtest_setup fails then pytest_runtest_teardown will be called but not pytest_runtest_call.

Each of the up to three reports is eventually fed to the logreport hook:

pytest_runtest_logreport(report)

A report object contains status and reporting information:

report.longrepr = string/lines/object to print
report.when = "setup", "call" or "teardown"
report.shortrepr = letter for progress-report
report.passed = True or False
report.failed = True or False
report.skipped = True or False

The terminal plugin uses this hook to print information about a test run.

The whole protocol described here is implemented via this hook:

pytest_runtest_protocol(item) -> True

The call object contains information about a performed call:

call.excinfo = ExceptionInfo object or None
call.when = "setup", "call" or "teardown"
call.outerr = None or tuple of strings representing captured stdout/stderr

generic collection hooks

py.test calls the following two fundamental hooks for collecting files and directories:

def pytest_collect_directory(path, parent):
    """ return Collection node or None for the given path. """

def pytest_collect_file(path, parent):
    """ return Collection node or None for the given path. """

Both return a collection node for a given path. All returned nodes from all hook implementations will participate in the collection and running protocol. The parent object is the parent node and may be used to access command line options via the parent.config object.

Python test function and module hooks

For influencing the collection of objects in Python modules you can use the following hook:

def pytest_pycollect_makeitem(collector, name, obj):
    """ return custom item/collector for a python object in a module, or None.  """

This hook will be called for each Python object in a collected Python module. The return value is a custom collection node or None.

Gateway initialization (distributed testing)

(alpha) For distributed testing it can be useful to prepare the remote environment. For this you can implement the newgateway hook:

def pytest_gwmanage_newgateway(gateway, platinfo):
    """ called after a gateway is instantiated. """

The gateway object here has a spec attribute which is an execnet.XSpec object, which has attributes that map key/values as specified from a --txspec option. The platinfo object is a dictionary with information about the remote process:

  • version: remote python's sys.version_info
  • platform: remote sys.platform
  • cwd: remote os.getcwd

Test Collection process

the collection tree

The collecting process is iterative so that distribution and execution of tests can start as soon as the first test item is collected. Collection nodes with children are called "Collectors" and terminal nodes are called "Items". Here is an example of such a tree, generated with the command py.test --collectonly py/xmlobj:

<Directory 'xmlobj'>
    <Directory 'testing'>
        <Module 'test_html.py' (py.__.xmlobj.testing.test_html)>
            <Function 'test_html_name_stickyness'>
            <Function 'test_stylenames'>
            <Function 'test_class_None'>
            <Function 'test_alternating_style'>
        <Module 'test_xml.py' (py.__.xmlobj.testing.test_xml)>
            <Function 'test_tag_with_text'>
            <Function 'test_class_identity'>
            <Function 'test_tag_with_text_and_attributes'>
            <Function 'test_tag_with_subclassed_attr_simple'>
            <Function 'test_tag_nested'>
            <Function 'test_tag_xmlname'>

By default all directories not starting with a dot are traversed, looking for test_*.py and *_test.py files. Those Python files are imported under their package name.

The Module collector looks for test functions and test classes and methods. Test functions and methods are prefixed test by default. Test classes must start with a capitalized Test prefix.

constructing the package name for test modules

Test modules are imported under their fully qualified name. Given a filesystem fspath it is constructed as follows:

  • walk the directories up to the last one that contains an __init__.py file.
  • perform sys.path.insert(0, basedir).
  • import the root package as root