Snapshot precision 0.99 let a wrong text line pass all seven screen tests

13 October 2026·4 mins

A code review claimed that a list screen’s snapshot tests couldn’t fail on the very line they were written to pin: a small gray “Location” line under each search result. To check, I broke that line on purpose. The rows got the container’s short name instead of its full name, so every location line rendered the wrong text. Then I ran the suite:

✔ Test listScreenDeletedItemsShowTheirOrigin() passed after 0.296 seconds.
✔ Test listScreenSearchResultsShowTheirLocation() passed after 0.708 seconds.
...
✔ Test run with 7 tests in 1 suite passed after 0.793 seconds.

All seven passed against the broken code. If your snapshot tests record a whole screen at .device layout with a precision below 1.0, they may be just as blind.

Setup

The project used swift-snapshot-testing 1.19.2, with Swift Testing on the iOS simulator. A shared defaults file sets both tolerances:

public static let perceptualPrecision: Float = 0.90
public static let precision: Float = 0.99

The two gates do different jobs:

perceptualPrecision decides whether a single pixel counts as different. 0.90 absorbs anti-aliasing drift between developer Macs.
precision decides what fraction of pixels must match. 0.99 lets 1% of the image differ by any amount.

The 0.99 was there to unblock CI: one render drifted by about 3,600 pixels between machines. The comment next to it even warned that it was “wide enough to hide a small real regression”. Every screen test used it anyway.

The arithmetic

An iPhone 17 Pro device frame is 1179×2556 pixels.

1179 × 2556       = 3,013,524 px
1% of that        =    30,135 px may differ freely
wrong location    =      ~900 px actually differed

A 1% budget sounds strict. On a full screen it’s about a 58-point square of arbitrary change, more than all the location lines on the screen combined. The test was recorded correctly and passed honestly. It just had no way to fail on the thing in its name.

Measuring the mutation

To size the gate, record the mutated renders, diff them against the committed references, then restore the references. A few lines of Pillow give a rough count (a pixel counts if any channel moves by more than 25):

import sys
from PIL import Image

a = Image.open(sys.argv[1]).convert("RGB")
b = Image.open(sys.argv[2]).convert("RGB")
pa, pb = a.load(), b.load()
w, h = a.size
n = sum(1 for y in range(h) for x in range(w)
        if max(abs(c1 - c2) for c1, c2 in zip(pa[x, y], pb[x, y])) > 25)
print(f"{n / (w * h):.2%} of {w}x{h}")

TEST_RUNNER_SNAPSHOT_TESTING_RECORD=all xcodebuild test ... -only-testing:MyTests/ListScreenSnapshotTests
git show HEAD:path/to/reference.png > /tmp/head.png
python3 pxdiff.py /tmp/head.png path/to/reference.png
git checkout -- path/to/__Snapshots__   # put the real references back

...DarkAppearance...:                   0.03% of 1179x2556
...SearchResultsAtTheDynamicTypeCap...: 0.14% of 1179x2556
...SearchResultsShowTheirLocation...:   0.03% of 1179x2556
...DeletedItemsShowTheirOrigin...:      0.03% of 1179x2556

The smallest real defect was 0.03%, about 900 pixels, so the budget had to sit well below that. 0.9999 allows 301.

The fix

Give the screen suite its own precision and write the numbers down where the next person will see them:

as: .image(
    precision: Constants.precision,
    perceptualPrecision: SnapshotDefaults.perceptualPrecision,
    layout: .device(config: config)
)

private enum Constants {
    /// A device frame is 1179×2556 px, so the default 1% budget is 30,135 px: wider than every
    /// Location line on the screen together. The wrong name moves about 900 px; this leaves a
    /// 301 px budget, while `perceptualPrecision` still absorbs anti-aliasing drift per pixel.
    static let precision: Float = 0.9999
}

Then rerun with the mutation still in place. It has to go red:

✘ Test listScreenDeletedItemsShowTheirOrigin() failed after 0.594 seconds with 1 issue.
✔ Test listScreenSortMenuOpen() passed after 0.751 seconds.
✔ Test listScreenBrowsingTheRoot() passed after 0.807 seconds.
✘ Test listScreenSearchResultsAtTheDynamicTypeCap() failed after 1.220 seconds with 1 issue.
✘ Test listScreenSearchResultsShowTheirLocation() failed after 1.618 seconds with 1 issue.
✔ Test listScreenSearchWithNoResults() passed after 1.659 seconds.
✘ Test listScreenDarkAppearance() failed after 2.036 seconds with 1 issue.

The four cases that draw location or origin lines failed. The three cases without them passed, which is also correct. After reverting the mutation, all seven passed again at 0.9999.

A 301-pixel budget is tight, and I verified it on one machine. If cross-machine drift trips it, look at the drift before you loosen the number.

Shrink the frame when you can

For a single component, a tight frame beats a tight precision. The same project measured a toolbar button’s enabled-to-disabled color change at 1,808 pixels, or 0.06% of a device frame. Rendered at 140×60 with layout: .fixed(width:height:), those 1,808 pixels became 2.4% of the image, well past a 1% budget. Keep .device for tests that are really about the whole screen, and size their precision from the smallest change they claim to catch.

Takeaway

precision is a fraction of the image, so a fixed value forgives more pixels as the frame grows. Before you trust a green snapshot, break the thing it’s named after, count the changed pixels, and compare that with (1 - precision) × width × height.

Pick one snapshot test in your suite, break its subject on purpose, and see whether it goes red.

Setup #

The arithmetic #

Measuring the mutation #

The fix #

Shrink the frame when you can #

Takeaway #