Snapshot-Testing on Lessons learned

Swift Testing withKnownIssue can hide snapshot failures

Fri, 30 Oct 2026 19:45:00 +0100

Two snapshot tests passed while reporting:

No reference was found on disk. Automatically recorded snapshot:

Their Swift Testing withKnownIssue wrapper accepted every issue inside the snapshot assertion, including missing references. Adding a matching: closure limited the exception to expected dependency reports. A deliberate wrong-state snapshot then failed, even though those reports remained known issues.

Restrict which issues count as known

The view had two states: a progress indicator before a check completed, and content afterward. Rendering the content also touched an unimplemented test dependency. The snapshot helper wrapped rendering and comparison in withKnownIssue(isIntermittent: true) to tolerate that report.

Without a matcher, the wrapper also accepted issues from the assertion it surrounded. The initial recording run exited successfully: two tests passed with three known issues. Two of those issues said references were missing; only one was the dependency report the wrapper was meant to tolerate.

The historical fix added this predicate:

String(describing: issue).contains("Unimplemented")

Here is the same policy as a generalized synchronous helper. Pass an existing, nonthrowing snapshot assertion as its body:

import Testing

func withExpectedDependencyIssue(_ body: () -> Void) {
 withKnownIssue(isIntermittent: true) {
 body()
 } matching: { issue in
 String(describing: issue).contains("Unimplemented")
 }
}

The public withKnownIssue matching overload takes a predicate over Issue. Returning false leaves an issue outside this exception. The synchronous overload is rethrows; this helper needs no try because its body is nonthrowing. A throwing assertion needs corresponding throwing handling.

isIntermittent: true allows the body to record no matching issue. That was useful for the progress state, which did not touch the dependency. It does not restrict which issues match. Without matching:, an intermittent wrapper still accepts everything recorded in its scope.

A mutation must change the rendered output

The first attempted verification changed a prefilled field. Both snapshot tests still passed. That was an ineffective mutation for this check; it did not establish that the assertion could detect the intended visual regression.

The stronger mutation completed the check before constructing the view in the test for the before-check reference. It deliberately compared the content state against the progress-state baseline.

That run exited with code 65. The before-check test failed, while the after-check test passed with its expected known issue. Across the two tests, the runner reported three issues, including two known issues. Restoring the original setup produced a successful run of 239 tests in 24 suites with one known issue.

This shows that the narrowed helper rejected an incorrectly rendered state. The session did not run that same wrong-state mutation against the original broad wrapper, so it does not provide an identical before-and-after mutation comparison. The missing-reference reports already demonstrate the broad wrapper’s excessive scope.

Keep the exception smaller than the assertion

The repair used here matched a description substring. It remains brittle: another report containing Unimplemented would also match, and diagnostic wording can change. Where the reporting library exposes a stable public discriminator, prefer that over a broad text search. Better still, provide the missing test dependency so the assertion needs no exception.

Keep that follow-up separate from what was verified. Other snapshot helpers retained broad wrappers; this change did not repair them. A passing suite using this helper says nothing about their failure detection.

These results came from the original Xcode simulator runs. The generalized helper follows Swift Testing’s public matching overload ; it has not been compiled separately.

Before trusting a snapshot helper with a known-issue scope, force a visibly different state against an existing reference and require a failing run. Restore the state and require green again.

Snapshot precision 0.99 let a wrong text line pass all seven screen tests

Tue, 13 Oct 2026 17:04:00 +0200

A code review claimed that a list screen’s snapshot tests couldn’t fail on the very line they were written to pin: a small gray “Location” line under each search result. To check, I broke that line on purpose. The rows got the container’s short name instead of its full name, so every location line rendered the wrong text. Then I ran the suite:

✔ Test listScreenDeletedItemsShowTheirOrigin() passed after 0.296 seconds.
✔ Test listScreenSearchResultsShowTheirLocation() passed after 0.708 seconds.
...
✔ Test run with 7 tests in 1 suite passed after 0.793 seconds.

All seven passed against the broken code. If your snapshot tests record a whole screen at .device layout with a precision below 1.0, they may be just as blind.

Setup

The project used swift-snapshot-testing 1.19.2, with Swift Testing on the iOS simulator. A shared defaults file sets both tolerances:

public static let perceptualPrecision: Float = 0.90
public static let precision: Float = 0.99

The two gates do different jobs:

perceptualPrecision decides whether a single pixel counts as different. 0.90 absorbs anti-aliasing drift between developer Macs.
precision decides what fraction of pixels must match. 0.99 lets 1% of the image differ by any amount.

The 0.99 was there to unblock CI: one render drifted by about 3,600 pixels between machines. The comment next to it even warned that it was “wide enough to hide a small real regression”. Every screen test used it anyway.

The arithmetic

An iPhone 17 Pro device frame is 1179×2556 pixels.

1179 × 2556 = 3,013,524 px
1% of that = 30,135 px may differ freely
wrong location = ~900 px actually differed

A 1% budget sounds strict. On a full screen it’s about a 58-point square of arbitrary change, more than all the location lines on the screen combined. The test was recorded correctly and passed honestly. It just had no way to fail on the thing in its name.

Measuring the mutation

To size the gate, record the mutated renders, diff them against the committed references, then restore the references. A few lines of Pillow give a rough count (a pixel counts if any channel moves by more than 25):

import sys
from PIL import Image

a = Image.open(sys.argv[1]).convert("RGB")
b = Image.open(sys.argv[2]).convert("RGB")
pa, pb = a.load(), b.load()
w, h = a.size
n = sum(1 for y in range(h) for x in range(w)
 if max(abs(c1 - c2) for c1, c2 in zip(pa[x, y], pb[x, y])) > 25)
print(f"{n / (w * h):.2%} of {w}x{h}")

TEST_RUNNER_SNAPSHOT_TESTING_RECORD=all xcodebuild test ... -only-testing:MyTests/ListScreenSnapshotTests
git show HEAD:path/to/reference.png > /tmp/head.png
python3 pxdiff.py /tmp/head.png path/to/reference.png
git checkout -- path/to/__Snapshots__ # put the real references back

...DarkAppearance...: 0.03% of 1179x2556
...SearchResultsAtTheDynamicTypeCap...: 0.14% of 1179x2556
...SearchResultsShowTheirLocation...: 0.03% of 1179x2556
...DeletedItemsShowTheirOrigin...: 0.03% of 1179x2556

The smallest real defect was 0.03%, about 900 pixels, so the budget had to sit well below that. 0.9999 allows 301.

The fix

Give the screen suite its own precision and write the numbers down where the next person will see them:

as: .image(
 precision: Constants.precision,
 perceptualPrecision: SnapshotDefaults.perceptualPrecision,
 layout: .device(config: config)
)

private enum Constants {
 /// A device frame is 1179×2556 px, so the default 1% budget is 30,135 px: wider than every
 /// Location line on the screen together. The wrong name moves about 900 px; this leaves a
 /// 301 px budget, while `perceptualPrecision` still absorbs anti-aliasing drift per pixel.
 static let precision: Float = 0.9999
}

Then rerun with the mutation still in place. It has to go red:

✘ Test listScreenDeletedItemsShowTheirOrigin() failed after 0.594 seconds with 1 issue.
✔ Test listScreenSortMenuOpen() passed after 0.751 seconds.
✔ Test listScreenBrowsingTheRoot() passed after 0.807 seconds.
✘ Test listScreenSearchResultsAtTheDynamicTypeCap() failed after 1.220 seconds with 1 issue.
✘ Test listScreenSearchResultsShowTheirLocation() failed after 1.618 seconds with 1 issue.
✔ Test listScreenSearchWithNoResults() passed after 1.659 seconds.
✘ Test listScreenDarkAppearance() failed after 2.036 seconds with 1 issue.

The four cases that draw location or origin lines failed. The three cases without them passed, which is also correct. After reverting the mutation, all seven passed again at 0.9999.

A 301-pixel budget is tight, and I verified it on one machine. If cross-machine drift trips it, look at the drift before you loosen the number.

Shrink the frame when you can

For a single component, a tight frame beats a tight precision. The same project measured a toolbar button’s enabled-to-disabled color change at 1,808 pixels, or 0.06% of a device frame. Rendered at 140×60 with layout: .fixed(width:height:), those 1,808 pixels became 2.4% of the image, well past a 1% budget. Keep .device for tests that are really about the whole screen, and size their precision from the smallest change they claim to catch.

Takeaway

precision is a fraction of the image, so a fixed value forgives more pixels as the frame grows. Before you trust a green snapshot, break the thing it’s named after, count the changed pixels, and compare that with (1 - precision) × width × height.

Pick one snapshot test in your suite, break its subject on purpose, and see whether it goes red.