Interactive online tutorial: Notebook

Test a Lint Rule

This guide assumes you’ve already started implementing your lint rule. If not, please read Build a Lint Rule.

You’ve written your rule, but now you need a test plan for it. Every lint rule should include some test cases.

Writing Unit Tests

While manually running your lint rule is useful, a unit test makes it easier to validate that your rule isn’t failing to report certain violations, prevents future regressions, and helps your reviewers understand your rule. An ideal lint rule written should warn on every violation (avoiding false-negatives), and never warn where there’s no violation (false-positives). For this reason, we require that every rule includes both VALID and INVALID test cases.

[2]:

from fixit import (
    CstLintRule,
    InvalidTestCase as Invalid,
    ValidTestCase as Valid,
)


class NoInheritFromObjectRule(CstLintRule):
    MESSAGE = "Inheriting from object is a no-op. 'class Foo:' is just fine =)"
    VALID = []
    INVALID = []

Valid test cases are defined using ValidTestCase (imported as a shorter name Valid), and invalid test cases are defined using InvalidTestCase (imported as a shorter name Invalid). You should add test cases until all potential edge-cases are covered.

[3]:

import libcst as cst
import libcst.matchers as m

class NoInheritFromObjectRule(CstLintRule):
    MESSAGE = "Inheriting from object is a no-op. 'class Foo:' is just fine =)"

    VALID = [Valid("class A(something):\n    pass"), Valid("class A:\n    pass")]
    INVALID = []

Adding Autofix Test Case

Testing an auto-fixer is as easy as adding the expected replacement code to your Invalid test case. Function strings can be multi-line, similar to docstrings, the leading whitespace is trimmed. The position of the lint suggestion is the starting position of the reported CSTNode. (In our example, it’s the start of ClassDef node which is the position of c in class ) It can be specified by providing line and column numbers in an Invalid test case.

[4]:

class NoInheritFromObjectRule(CstLintRule):
    """
    In Python 3, a class is inherited from ``object`` by default.
    Explicitly inheriting from ``object`` is redundant, so removing it keeps the code simpler.
    """
    MESSAGE = "Inheriting from object is a no-op. 'class Foo:' is just fine =)"
    VALID = [
        Valid("class A(something):    pass"),
        Valid(
            """
            class A:
                pass"""
        ),
    ]
    INVALID = [
        Invalid(
            """
            class B(object):
                pass""",
            line=1,
            column=1,
            expected_replacement="""
            class B:
                pass""",
        ),
        Invalid(
            """
            class B(object, A):
                pass""",
            line=1,
            column=1,
            expected_replacement="""
            class B(A):
                pass""",
        ),
    ]

    def visit_ClassDef(self, node: cst.ClassDef) -> None:
        new_bases = tuple(
            base for base in node.bases if not m.matches(base.value, m.Name("object"))
        )

        if tuple(node.bases) != new_bases:
            # reconstruct classdef, removing parens if bases and keywords are empty
            new_classdef = node.with_changes(
                bases=new_bases,
                lpar=cst.MaybeSentinel.DEFAULT,
                rpar=cst.MaybeSentinel.DEFAULT,
            )

            # report warning and autofix
            self.report(node, replacement=new_classdef)

Run Tests

Fixit provides add_lint_rule_tests_to_module() to automatically generate unittest test cases in a module. If you’re contributing a new lint rule to Fixit, you can add the rule in fixit/rules/ of fixit repository. The add_lint_rule_tests_to_module() is already configured in fixit/tests/__init__.py. Run the added tests by:

[5]:

! python -m unittest fixit.tests.NoInheritFromObjectRule

....
----------------------------------------------------------------------
Ran 4 tests in 0.056s

OK

If you’re developing a custom lint rule in your codebase, you can configure add_lint_rule_tests_to_module() in your test module by passing globals() as the module_attrs argument and providing a collection of the rules you would like to test as the rules argument. E.g. add_lint_rule_tests_to_module(globals(), {my_package.Rule1, my_package.Rule2}) Then run your test module by:

python -m unittest your_test_module

[6]:

from fixit import add_lint_rule_tests_to_module

add_lint_rule_tests_to_module(globals(), rules=[NoInheritFromObjectRule])
import unittest

unittest.main(argv=["first-arg-is-ignored"], exit=False)

....
----------------------------------------------------------------------
Ran 4 tests in 0.063s

OK

[6]:

<unittest.main.TestProgram at 0x7f0b89bf5050>

Run Your Rule in a Codebase

The easiest way to see the effects of your lint rule is to run it against the entire codebase. This is pretty easy:

python -m fixit.cli.run_rules path --help  # see all the supported flags
python -m fixit.cli.run_rules path --rules RewriteToLiteralRule

This runs the linter in parallel across all the Python files in the path. It ignores # noqa, # lint-fixme, and # lint-ignore comments by default, but that can be overridden with the --use-ignore-comments flag (see --help for more details).

Check out a few of the reported violations to see that generated reports are accurate. If possible, you should consider cleaning up these existing violations before shipping your lint rule. Otherwise, consider that fixing these preexisting violations in future changes may slow other developers down.

Reference

class fixit.ValidTestCase
code : str
filename : str = "not/a/real/file/path.py"
config : common.base.LintConfig = LintConfig(allow_list_rules=[], block_list_patterns=['@generated', '@nolint'], block_list_rules=[], fixture_dir='./fixtures', use_noqa=False, formatter=[], packages=['fixit.rules'], repo_root='/home/docs/checkouts/readthedocs.org/user_builds/fixit/envs/v0.1.4/lib/python3.7/site-packages/fixit-0.1.4-py3.7.egg/fixit', rule_config={})
class fixit.InvalidTestCase
code : str
kind : typing.Union[str, NoneType]
line : typing.Union[int, NoneType]
column : typing.Union[int, NoneType]
expected_replacement : typing.Union[str, NoneType]
filename : str = "not/a/real/file/path.py"
config : common.base.LintConfig = LintConfig(allow_list_rules=[], block_list_patterns=['@generated', '@nolint'], block_list_rules=[], fixture_dir='./fixtures', use_noqa=False, formatter=[], packages=['fixit.rules'], repo_root='/home/docs/checkouts/readthedocs.org/user_builds/fixit/envs/v0.1.4/lib/python3.7/site-packages/fixit-0.1.4-py3.7.egg/fixit', rule_config={})
expected_message : typing.Union[str, NoneType]
property expected_str
fixit.add_lint_rule_tests_to_module(module_attrs: Dict[str, Any], rules: Set[Union[Type[fixit.common.base.CstLintRule], Type[fixit.common.pseudo_rule.PseudoLintRule]]], test_case_type: Type[unittest.case.TestCase] = <class 'fixit.common.testing.LintRuleTestCase'>, custom_test_method_name: str = '_test_method', fixture_dir: pathlib.Path = PosixPath('.'), rules_package: str = '') → None

Generates classes inheriting from unittest.TestCase from the data available in rules and adds these to module_attrs. The goal is to facilitate unit test discovery by Python’s unittest framework. This will provide the capability of testing your lint rules by running commands such as python -m unittest <your testing module name>.

module_attrs: A dictionary of attributes we want to add these test cases to. If adding to a module, you can pass globals() as the argument.

rules: A collection of classes extending CstLintRule to be converted to test cases.

test_case_type: A class extending Python’s unittest.TestCase that implements a custom test method for testing lint rules to serve as a stencil for test cases. New classes will be generated, and named after each lint rule. They will inherit directly from the class passed into test_case_type. If argument is omitted, will default to the LintRuleTestCase class from fixit.common.testing.

custom_test_method_name: A member method of the class passed into test_case_type parameter that contains the logic around asserting success or failure of CstLintRule’s ValidTestCase and InvalidTestCase test cases. The method will be dynamically renamed to test_<VALID/INVALID>_<test case index> for discovery by unittest. If argument is omitted, add_lint_rule_tests_to_module will look for a test method named _test_method member of test_case_type.

fixture_dir: The directory in which fixture files for the passed rules live. Necessary only if any lint rules require fixture data for testing.

rules_package: The name of the rules package. This will be used during the search for fixture files and provides insight into the structure of the fixture directory. The structure of the fixture directory is automatically assumed to mirror the structure of the rules package, eg: <rules_package>.submodule.module.rule_class should have fixture files in <fixture_dir>/submodule/module/rule_class/.