Testing
Learn how to check that your rules still fire where you think they do.
A rule is a small program, and a rule that matches nothing fails silently: it loads, runs, and reports success. vale test runs cases you keep beside your rules, each a short document and the alerts linting it must produce.
$ vale test
SUCCESS 2 files — 5 passed, 0 failedWriting cases
Cases live in one of two places. A rule can carry its own under tests:
# styles/House/Hedging.yml
extends: existence
message: "Consider removing '%s'."
level: warning
ignorecase: true
tokens:
- perhaps
- it seems
tests:
- name: flags a hedge
input: Perhaps we should ship.
want: |
1:1:House.Hedging:Consider removing 'Perhaps'.
- name: leaves code alone
input: |
Run `perhaps` first.
want: ''Or a file ending in .test.yml holds a list of them:
name
Required. Identifies the case in the report, and is unique within its file.
about
A note for the reader, such as an issue number. Never checked.
input
The document to lint.
format
The extension input is read as, such as rst or .adoc. Defaults to md.
rule
A rule file, relative to the test file, to run alone. See Isolated and project cases.
view
A View under config/views to read input through, for a rule scoped to one of its pieces.
want
The exact alerts, one line:column:Check:message per line. An empty want asserts that there are none.
contains
Text, or a list of texts, that the alerts must include.
absent
Text, or a list of texts, that the alerts must not include.
Every case needs at least one of want, contains, or absent. A case that asserts nothing passes whatever the rule does, so Vale refuses to run it. Use want when the exact output matters, contains when only some of it does, such as which rule fired or which text it matched, and absent to pin down what a rule must leave alone. about is for the reader: why the case exists, or the issue it came from.
The input
input is the document itself, written in the case rather than read from a file. A multi-line document goes in a YAML block, |, which keeps its line breaks; the alerts' lines and columns count from its first line:
A one-line input can be written plainly, but quote it if it contains : or starts with a character YAML reads as syntax, such as #, -, or [: input: 'Note: this is perhaps it.'
Formats
A case is read as Markdown unless its format names another extension, and it's parsed the way a file of that format would be. A reStructuredText or AsciiDoc case needs the same tools a lint run of that format does, rst2html or asciidoctor. Any extension works, including one a View is attached to, such as COMMIT_EDITMSG.
Views
A rule scoped to one of a View's pieces, such as subject, sees nothing in an isolated case, since an isolated case loads nothing from the project, including its Views. view names the View to read input through, from config/views on the StylesPath:
Isolated and project cases
An isolated case runs one rule and nothing else: not the rest of its style, and not your .vale.ini. It reports at every level, so it says exactly what the rule matches. A case under a rule's tests is isolated to that rule, and a .test.yml case is isolated when it names a rule.
A project case, a .test.yml case with no rule, is linted by the configuration a vale run in that directory would use: its sections, scopes, MinAlertLevel, and every style it enables. It tells you whether the rule reaches real documents. Its format has to be one your configuration covers, or nothing is linted.
An isolated rule keeps the name a project run gives it, so a case can move between the two without its want changing.
Running them
With no arguments, vale test searches the working directory. Name files or directories to run only those:
A failed case shows what was expected against what came back:
--output=JSON prints the same results as JSON, with each failure's reason, got, and want.
Coverage
A passing run says nothing about the rules no case touches. With --coverage, every rule under the given paths has to produce an alert in at least one case:
Script rules
A script rule is tested like any other, by linting an input with it. The case runs the rule's real script, whether it's written inline or kept in config/scripts on the StylesPath, so there's nothing to extract or load into a separate runner:
Give each edge case its own case as you find it, so a later change to the script that breaks it fails the run. An off-by-one in the paragraph count shows up as a diff:
A script that doesn't compile fails every case of its rule as one that could not run, and the run exits 2.
Packages
A package whose rules extend another package names it in the package's own .vale.ini, the way Packages does for a project. vale test installs what a package names into a temporary StylesPath for the run, so its cases resolve without a synced project. A package already on the StylesPath is used as it is.
A package root is the nearest directory above a rule that holds both a .vale.ini and a styles directory, the layout vale sync installs.
In CI
vale test exits like a lint run:
0
Every case passed.
1
A case failed, or, with --coverage, a rule was uncovered.
2
A case could not run: a rule that failed to load, a View that was not found, or no cases at all. The rest still report.
Last updated