Mutatest: Python mutation testing¶
Are you confident in your tests? Try out mutatest
and see if your tests will detect small
modifications (mutations) in the code. Surviving mutations represent subtle changes that are
undetectable by your tests. These mutants are potential modifications in source code that continuous
integration checks would miss.
Features:¶
Simple command line tool with multiple configuration options.
Built on Python’s Abstract Syntax Tree (AST) grammar to ensure mutants are valid.
No source code modification, only the
__pycache__
is changed.Uses
coverage
to create only meaningful mutants.Built for efficiency with multiple running modes and random sampling of mutation targets.
Capable of running parallel mutation trials with multiprocessing on Python 3.8.
Flexible enough to run on a whole package or a single file.
Includes an API for custom mutation controls.
Tested on Linux, Windows, and MacOS with Azure pipelines.
Full strict static type annotations throughout the source code and the API.
Quick Start¶
mutatest
requires Python 3.7 or 3.8.
Install from PyPI:
$ pip install mutatest
Install from conda-forge:
$ conda install -c conda-forge mutatest
Alternatively, clone the repo from GitHub and install from the source code:
$ cd mutatest
$ pip install .
mutatest
is designed to work when your test files are separated from your source directory
and are prefixed with test_
.
See Pytest Test Layout
for more details.
mutatest
is a diagnostic command line tool for your test coverage assessment.
If you have a Python package in with an associated tests/
folder, or internal test_
prefixed files,
that are auto-detected with pytest
, then you can run mutatest
without any arguments.
$ mutatest
See more examples with additional configuration options in Command Line Controls.
Help¶
Run mutatest --help
to see command line arguments and supported operations:
$ mutatest --help
usage: Mutatest [-h] [-b [STR [STR ...]]] [-e PATH] [-m {f,s,d,sd}] [-n INT]
[-o PATH] [-r INT] [-s PATH] [-t STR_CMDS]
[-w [STR [STR ...]]] [-x INT] [--debug] [--nocov] [--parallel]
[--timeout_factor FLOAT > 1]
Python mutation testing. Mutatest will manipulate local __pycache__ files.
optional arguments:
-h, --help show this help message and exit
-k [STR [STR ...]], --skip [STR [STR ...]]
Mutation categories to skip for trials. (default: empty list)
-e PATH, --exclude PATH
Path to .py file to exclude, multiple -e entries supported. (default: None)
-m {f,s,d,sd}, --mode {f,s,d,sd}
Running modes, see the choice option descriptions below. (default: s)
-n INT, --nlocations INT
Number of locations in code to randomly select for mutation from possible targets. (default: 10)
-o PATH, --output PATH
Output RST file location for results. (default: No output written)
-r INT, --rseed INT Random seed to use for sample selection.
-s PATH, --src PATH Source code (file or directory) for mutation testing. (default: auto-detection attempt).
-t STR_CMDS, --testcmds STR_CMDS
Test command string to execute. (default: 'pytest')
-y [STR [STR ...]], --only [STR [STR ...]]
Only mutation categories to use for trials. (default: empty list)
-x INT, --exception INT
Count of survivors to raise Mutation Exception for system exit.
--debug Turn on DEBUG level logging output.
--nocov Ignore coverage files for optimization.
--parallel Run with multiprocessing (Py3.8 only).
--timeout_factor FLOAT > 1
If a mutation trial running time is beyond this factor multiplied by the first
clean trial running time then that mutation trial is aborted and logged as a timeout.
Example Output¶
This is an output example running mutation trials against the API Tutorial example folder.
$ mutatest -s example/ -t "pytest" -r 314
Running clean trial
2 mutation targets found in example/a.py AST.
1 mutation targets found in example/b.py AST.
Setting random.seed to: 314
Total sample space size: 2
10 exceeds sample space, using full sample: 2.
Starting individual mutation trials!
Current target location: a.py, LocIndex(ast_class='BinOp', lineno=6, col_offset=11, op_type=<class '_ast.Add'>)
Detected mutation at example/a.py: (6, 11)
Detected mutation at example/a.py: (6, 11)
Surviving mutation at example/a.py: (6, 11)
Break on survival: stopping further mutations at location.
Current target location: b.py, LocIndex(ast_class='CompareIs', lineno=6, col_offset=11, op_type=<class '_ast.Is'>)
Detected mutation at example/b.py: (6, 11)
Running clean trial
Mutatest diagnostic summary
===========================
- Source location: /home/user/Github/mutatest/docs/api_tutorial/example
- Test commands: ['pytest']
- Mode: s
- Excluded files: []
- N locations input: 10
- Random seed: 314
Random sample details
---------------------
- Total locations mutated: 2
- Total locations identified: 2
- Location sample coverage: 100.00 %
Running time details
--------------------
- Clean trial 1 run time: 0:00:00.348999
- Clean trial 2 run time: 0:00:00.350213
- Mutation trials total run time: 0:00:01.389095
Trial Summary Report:
Overall mutation trial summary
==============================
- DETECTED: 3
- SURVIVED: 1
- TOTAL RUNS: 4
- RUN DATETIME: 2019-10-17 16:57:08.645355
Detected mutations:
DETECTED
--------
- example/a.py: (l: 6, c: 11) - mutation from <class '_ast.Add'> to <class '_ast.Sub'>
- example/a.py: (l: 6, c: 11) - mutation from <class '_ast.Add'> to <class '_ast.Mod'>
- example/b.py: (l: 6, c: 11) - mutation from <class '_ast.Is'> to <class '_ast.IsNot'>
Surviving mutations:
SURVIVED
--------
- example/a.py: (l: 6, c: 11) - mutation from <class '_ast.Add'> to <class '_ast.Mult'>
Contents¶
- Installation
- Mutation Trial Process
- Motivation and FAQs
- Command Line Controls
- Specifying source files and test commands
- Coverage filtering
- Auto-detected package structures
- Selecting a running mode
- Controlling randomization behavior and trial number
- Selecting categories of mutations
- Setting the output location
- Raising exceptions for survivor tolerances
- Controlling trial timeout behavior
- Parallelization
- Putting it all together
- Getting help
- Using a config file
- Mutations
- API Tutorial
- API Reference
- License
- Changelog
- Contributing
- GitHub