Snapshot testing guide (#3357)

* Snapshot testing guide

* Typo fixes

* Some more typo fixes

* Typo fixes

* Update docs/guide/testing.md

Co-authored-by: Rodrigo Girão Serrão <5621605+rodrigogiraoserrao@users.noreply.github.com>

* Add clarifications, PR feedback

* Add clarifications, PR feedback

---------

Co-authored-by: Rodrigo Girão Serrão <5621605+rodrigogiraoserrao@users.noreply.github.com>
This commit is contained in:
Darren Burns
2023-09-21 11:16:11 +01:00
committed by GitHub
parent bbde62fc57
commit 9c8a8df028
5 changed files with 144 additions and 2 deletions

View File

@@ -134,7 +134,7 @@ await pilot.click(Button, offset(0, -1))
### Modifier keys
You can simulate clicks in combination with modifier keys, by setting the `shift`, `meta`, or `control` parameters.
Here's how you could simulate ctrl-clicking a widget with an id of "slider":
Here's how you could simulate ctrl-clicking a widget with an ID of "slider":
```python
await pilot.click("#slider", control=True)
@@ -162,7 +162,149 @@ You can generally solve this by calling [`pause()`][textual.pilot.Pilot.pause] w
You can also supply a `delay` parameter, which will insert a delay prior to waiting for pending messages.
## Textual's test
## Textual's tests
Textual itself has a large battery of tests.
If you are interested in how we write tests, see the [tests/](https://github.com/Textualize/textual/tree/main/tests) directory in the Textual repository.
## Snapshot testing
A _snapshot_ is a record of what an application looked like at a given point in time.
_Snapshot testing_ is the process of creating a snapshot of an application while a test runs, and comparing it to a historical version.
If there's a mismatch, the snapshot testing framework flags it for review.
This offers a simple, automated way of checking our application displays like we expect.
### pytest-textual-snapshot
You can use [`pytest-textual-snapshot`](https://github.com/Textualize/pytest-textual-snapshot) to snapshot test your Textual app.
This is a plugin for pytest which adds support for snapshot testing Textual apps, and it's maintained by the developers of Textual.
A test using this package saves a snapshot (in this case, an SVG screenshot) of a running Textual app to disk.
The next time the test runs, it takes another snapshot and compares it to the previously saved one.
If the snapshots differ, the test fails, and you can view a side-by-side diff showing the visual change.
#### Installation
You can install `pytest-textual-snapshot` using your favorite package manager (`pip`, `poetry`, etc.).
```
pip install pytest-textual-snapshot
```
#### Creating a snapshot test
With the package installed, you now have access to the `snap_compare` pytest fixture.
Let's look at an example of how we'd create a snapshot test for the [calculator app](https://github.com/Textualize/textual/blob/main/examples/calculator.py) below.
```{.textual path="examples/calculator.py" columns=100 lines=41 press="3,.,1,4,5,9,2,wait:400"}
```
First, we need to create a new test and specify the path to the Python file containing the app.
This path should be relative to the location of the test.
```python
def test_calculator(snap_compare):
assert snap_compare("path/to/calculator.py")
```
Let's run the test as normal using `pytest`.
```
pytest
```
When this test runs for the first time, an SVG screenshot of the calculator app is generated, and the test will fail.
Snapshot tests always fail on the first run, since there's no previous version to compare the snapshot to.
![snapshot_report_console_output.png](../images/testing/snapshot_report_console_output.png)
If you open the snapshot report in your browser, you'll see something like this:
![snapshot_report_example.png](../images/testing/snapshot_report_example.png)
!!! tip
You can usually open the link directly from the terminal, but some terminal emulators may
require you to hold ++ctrl++ or ++command++ while clicking for links to work.
The report explains that there's "No history for this test".
It's our job to validate that the initial snapshot looks correct before proceeding.
Our calculator is rendering as we expect, so we'll save this snapshot:
```
pytest --snapshot-update
```
!!! warning
Only ever run pytest with `--snapshot-update` if you're happy with how the output looks
on the left hand side of the snapshot report. When using `--snapshot-update`, you're saying "I'm happy with all of the
screenshots in the snapshot test report, and they will now represent the ground truth which all future runs will be compared
against". As such, you should only run `pytest --snapshot-update` _after_ running `pytest` and confirming the output looks good.
Now that our snapshot is saved, if we run `pytest` (with no arguments) again, the test will pass.
This is because the screenshot taken during this test run matches the one we saved earlier.
#### Catching a bug
The real power of snapshot testing comes from its ability to catch visual regressions which could otherwise easily be missed.
Imagine a new developer joins your team, and tries to make a few changes to the calculator.
While making this change they accidentally break some styling which removes the orange coloring from the buttons on the right of the app.
When they run `pytest`, they're presented with a report which reveals the damage:
![snapshot_report_diff_before.png](../images/testing/snapshot_report_diff_before.png)
On the right, we can see our "historical" snapshot - this is the one we saved earlier.
On the left is how our app is currently rendering - clearly not how we intended!
We can click the "Show difference" toggle at the top right of the diff to overlay the two versions:
![snapshot_report_diff_after.png](../images/testing/snapshot_report_diff_after.png)
This reveals another problem, which could easily be missed in a quick visual inspection -
our new developer has also deleted the number 4!
!!! tip
Snapshot tests work well in CI on all supported operating systems, and the snapshot
report is just an HTML file which can be exported as a build artifact.
#### Pressing keys
You can simulate pressing keys before the snapshot is captured using the `press` parameter.
```python
def test_calculator_pressing_numbers(snap_compare):
assert snap_compare("path/to/calculator.py", press=["1", "2", "3"])
```
#### Changing the terminal size
To capture the snapshot with a different terminal size, pass a tuple `(width, height)` as the `terminal_size` parameter.
```python
def test_calculator(snap_compare):
assert snap_compare("path/to/calculator.py", terminal_size=(50, 100))
```
#### Running setup code
You can also run arbitrary code before the snapshot is captured using the `run_before` parameter.
In this example, we use `run_before` to hover the mouse cursor over the widget with ID `number-5`
before taking the snapshot.
```python
def test_calculator_hover_number(snap_compare):
async def run_before(pilot) -> None:
await pilot.hover("#number-5")
assert snap_compare("path/to/calculator.py", run_before=run_before)
```
For more information, visit the [`pytest-textual-snapshot` repo on GitHub](https://github.com/Textualize/pytest-textual-snapshot).

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 151 KiB