Swift-Testing on Lessons learned

Swift Testing withKnownIssue can hide snapshot failures

Fri, 30 Oct 2026 19:45:00 +0100

Two snapshot tests passed while reporting:

No reference was found on disk. Automatically recorded snapshot:

Their Swift Testing withKnownIssue wrapper accepted every issue inside the snapshot assertion, including missing references. Adding a matching: closure limited the exception to expected dependency reports. A deliberate wrong-state snapshot then failed, even though those reports remained known issues.

Restrict which issues count as known

The view had two states: a progress indicator before a check completed, and content afterward. Rendering the content also touched an unimplemented test dependency. The snapshot helper wrapped rendering and comparison in withKnownIssue(isIntermittent: true) to tolerate that report.

Without a matcher, the wrapper also accepted issues from the assertion it surrounded. The initial recording run exited successfully: two tests passed with three known issues. Two of those issues said references were missing; only one was the dependency report the wrapper was meant to tolerate.

The historical fix added this predicate:

String(describing: issue).contains("Unimplemented")

Here is the same policy as a generalized synchronous helper. Pass an existing, nonthrowing snapshot assertion as its body:

import Testing

func withExpectedDependencyIssue(_ body: () -> Void) {
 withKnownIssue(isIntermittent: true) {
 body()
 } matching: { issue in
 String(describing: issue).contains("Unimplemented")
 }
}

The public withKnownIssue matching overload takes a predicate over Issue. Returning false leaves an issue outside this exception. The synchronous overload is rethrows; this helper needs no try because its body is nonthrowing. A throwing assertion needs corresponding throwing handling.

isIntermittent: true allows the body to record no matching issue. That was useful for the progress state, which did not touch the dependency. It does not restrict which issues match. Without matching:, an intermittent wrapper still accepts everything recorded in its scope.

A mutation must change the rendered output

The first attempted verification changed a prefilled field. Both snapshot tests still passed. That was an ineffective mutation for this check; it did not establish that the assertion could detect the intended visual regression.

The stronger mutation completed the check before constructing the view in the test for the before-check reference. It deliberately compared the content state against the progress-state baseline.

That run exited with code 65. The before-check test failed, while the after-check test passed with its expected known issue. Across the two tests, the runner reported three issues, including two known issues. Restoring the original setup produced a successful run of 239 tests in 24 suites with one known issue.

This shows that the narrowed helper rejected an incorrectly rendered state. The session did not run that same wrong-state mutation against the original broad wrapper, so it does not provide an identical before-and-after mutation comparison. The missing-reference reports already demonstrate the broad wrapper’s excessive scope.

Keep the exception smaller than the assertion

The repair used here matched a description substring. It remains brittle: another report containing Unimplemented would also match, and diagnostic wording can change. Where the reporting library exposes a stable public discriminator, prefer that over a broad text search. Better still, provide the missing test dependency so the assertion needs no exception.

Keep that follow-up separate from what was verified. Other snapshot helpers retained broad wrappers; this change did not repair them. A passing suite using this helper says nothing about their failure detection.

These results came from the original Xcode simulator runs. The generalized helper follows Swift Testing’s public matching overload ; it has not been compiled separately.

Before trusting a snapshot helper with a known-issue scope, force a visibly different state against an existing reference and require a failing run. Restore the state and require green again.

Swift Testing: .serialized did not stop another suite deleting test files

Wed, 28 Oct 2026 14:36:00 +0100

A filesystem test failed while writing its fixture, even though its suite used @Suite(.serialized) and each fixture had a UUID directory. The underlying error was:

NSPOSIXErrorDomain Code=2 "No such file or directory"

Another suite was constructing the application core, whose initializer swept the shared cache directory. Injecting that startup operation, with an inert test value, stopped unrelated tests from deleting each other’s fixtures.

A unique child directory still has a shared parent

The failing suite created a directory, then wrote a file into it. Its own cleanup tests ran serially. Elsewhere in the same Swift Testing process, ordinary tests constructed the core as part of their setup. Every construction triggered a sweep of the common cache parent.

The destructive interleaving was small:

cleanup test: create cache/<UUID>/
unrelated test: construct core -> sweep cache children
cleanup test: write cache/<UUID>/fixture.bin -> missing directory

UUIDs prevented name collisions. They could not protect a child from an operation that deleted every child.

Swift Testing’s parallelization trait documentation describes the scope: .serialized orders a suite’s contained tests and sub-suites. It does not order those tests relative to unrelated suites. Putting the attribute on the failing suite left the other constructors free to run alongside it.

Make ordinary test construction inert

The fix moved the startup sweep behind an injected operation. Production kept the real sweep; the test dependency supplied a no-op. Tests that needed to verify sweeping explicitly selected the real operation.

Here is a generalized closure-based adaptation, independent of a dependency-injection library:

import Foundation

struct StartupSweep {
 var run: () throws -> Void

 static var inert: StartupSweep { StartupSweep(run: {}) }

 static func live(root: URL) -> StartupSweep {
 StartupSweep {
 let manager = FileManager.default
 for child in try manager.contentsOfDirectory(
 at: root, includingPropertiesForKeys: nil
 ) {
 try manager.removeItem(at: child)
 }
 }
 }
}

struct CacheCore {
 init(startupSweep: StartupSweep) throws {
 try startupSweep.run()
 }
}

The production composition root passes .live(root: cacheRoot). The ordinary test factory passes .inert; neither caller relies on an initializer default. This example propagates filesystem errors so a cleanup test cannot pass by silently ignoring a failed sweep.

For a test-owned root containing a prepared fixture file, the two assertions look like this. This is a test-body excerpt using Testing:

// Ordinary construction must preserve the existing fixture.
_ = try CacheCore(startupSweep: .inert)
#expect(FileManager.default.fileExists(atPath: fixture.path))

// Cleanup coverage explicitly opts into the real operation.
_ = try CacheCore(startupSweep: .live(root: root))
#expect(!FileManager.default.fileExists(atPath: fixture.path))

Use a disposable root for this check. Never point the illustrative live operation at a directory containing files you need. Injecting a separate root per test is another way to isolate filesystem work when the design permits it.

Prove the constructor contract, then repeat the broader target

The historical investigation used an isolated-suite control: the cleanup suite repeated 30 times with zero issues. The broader core target failed on its first run. That narrowed the search to interference from outside the serialized suite.

A regression test then created a fixture, built the core-under-test dependencies, and asserted that the file still existed. Before the injection fix, that assertion failed. It converted an intermittent write failure into a direct check of the destructive constructor behavior.

After the fix, the archived broader-target output reported:

exit: 0
** TEST SUCCEEDED **
issues:
0
max repetition:
repetition 30
✔ Test run with 165 tests in 28 suites passed after 2.806 seconds.

Those results verify the original fix under Xcode and iOS Simulator. The closure-based adaptation above has not been compiled separately. The reported test count is the runner’s summary, not a total multiplied by the repetition limit.

When a serialized suite loses files, check every initializer that can sweep their shared parent, including constructors called by unrelated suites.

A five-second Swift Testing wait timed out after its loop had run five times

Thu, 22 Oct 2026 16:12:00 +0200

Five concurrency tests passed on their own in 0.086 seconds. In the full target, they failed together at about 6.8 seconds:

✘ Test fetch_threeSources_asksForEverySourceAtOnce() recorded an issue at FetcherConcurrencyTests.swift:21:30:
 Expectation failed: ContinuousClock.now < deadline
✘ Test fetch_threeSources_asksForEverySourceAtOnce() failed after 6.816 seconds with 1 issue.

It looked like a deadlock in the code under test. It was a test helper measuring the wrong thing.

Setup

Swift Testing, an Xcode test target that mixes @MainActor unit suites with snapshot suites.
Two test plans run the same target. One sets "parallelizable": false for it, and the other doesn’t. Only the parallel plan failed.
A small polling helper that the concurrent-fetch tests use to wait for a gated fake service to see requests:

static func until(
 sourceLocation: SourceLocation = #_sourceLocation,
 _ condition: () -> Bool
) async throws {
 let deadline = ContinuousClock.now.advanced(by: .seconds(5))
 while !condition() {
 try #require(ContinuousClock.now < deadline, "Timed out waiting for the condition to hold",
 sourceLocation: sourceLocation)
 await Task.yield()
 }
}

The helper runs on the main actor. Each Task.yield() lets the queued work, in this case the fetcher opening its requests, run before the condition is checked again.

Reproduce before guessing

Running the target alone under the parallel plan with -test-iterations and -run-tests-until-failure failed 3 out of 3 times, always in the same tests and always at the wall-clock deadline. So it wasn’t a rare flake. It failed every time under that plan.

To check whether the waits were stuck or just slow, I temporarily replaced the deadline with a counter and printed what each wait used:

let start = ContinuousClock.now
var turns = 0
while !condition() {
 turns += 1
 await Task.yield()
}
print("WAITPROBE turns=\(turns) elapsed=\(ContinuousClock.now - start) at \(sourceLocation.line)")

5 5.766 FetcherConcurrencyTests.swift:54
5 5.712 FetcherConcurrencyTests.swift:21
4 6.411 FetcherConcurrencyTests.swift:37
3 5.704 SyncLoggingTests.swift:25
3 5.338 InFlightTests.swift:20

Every wait finished, but the slowest needed only 3 to 5 turns and took more than 5 seconds of wall time. The condition was met on almost the first chance the fetcher got to run. It just didn’t get that chance for seconds.

The reason showed up when I sorted tests by duration. In the parallel plan, snapshot suites share the process and the main actor, and each render holds the actor synchronously, some of them for over 7 seconds. While a render ran, the waiting test’s next turn just sat in the queue, and the clock kept running.

Dead end: a longer timeout

I tried a 60-second deadline as an experiment. Both runs passed, with the concurrency tests taking about 7.9 seconds each.

That shows the cause, but it isn’t a fix. 60 is no more principled than 5. A heavier snapshot suite brings the failure back, and a test that really hangs now takes a minute to report it. I reverted it.

Fix: spend both budgets before failing

The wait should fail only once it has had enough chances to make progress and enough time has passed:

static func until(
 sourceLocation: SourceLocation = #_sourceLocation,
 timeBudget: Duration = .seconds(5),
 _ condition: () -> Bool
) async throws {
 let deadline = ContinuousClock.now.advanced(by: timeBudget)
 var turns = 0
 while !condition() {
 let hasBudgetLeft = turns < Constants.turnBudget || ContinuousClock.now < deadline
 try #require(hasBudgetLeft, "Timed out waiting for the condition to hold",
 sourceLocation: sourceLocation)
 turns += 1
 await Task.yield()
 }
}

enum Constants {
 /// A wait under a fully loaded run of this target took at most 5 turns.
 static let turnBudget = 100
}

On an idle main actor, 100 turns go by in milliseconds, so a condition that never holds still fails just after the time budget. On a starved actor, the wait keeps going until it has had real chances to run.

Pin it with a controlled starvation test

The full target is a slow, noisy reproduction, so I wrote a test that intentionally causes starvation. It makes timeBudget a parameter, holds the main actor for longer than that budget, and makes the condition true only afterward:

@MainActor
@Suite
struct WaitTests {
 @Test
 func until_whenTheMainActorIsHeldPastTheTimeBudget_waitsForTheCondition() async throws {
 let holds = LockIsolated(false)
 Task { @MainActor in
 Thread.sleep(forTimeInterval: 0.3) // stands in for a snapshot render
 Task { @MainActor in holds.setValue(true) }
 }
 try await Wait.until(timeBudget: .milliseconds(100)) { holds.value }
 }

 @Test
 func until_whenTheConditionNeverHolds_fails() async {
 await withKnownIssue {
 try await Wait.until(timeBudget: .zero) { false }
 }
 }
}

Before the fix, it failed after 0.335 seconds with the same ContinuousClock.now < deadline error. After the fix, it passed in 0.303 seconds. The second test makes sure the helper can still fail.

Verify both directions

Check	Result
Parallel plan, before the fix	0 of 3 passed
Parallel plan, after the fix	5 of 5 passed
Serial plan, whole target	690 tests passed
Mutant: fetcher limited to one request at a time	all 5 concurrency tests failed, each at 5.0 s

The last row matters most. A longer timeout would also have made the parallel runs pass. What it wouldn’t keep is fast failure for a real bug. I changed the fetcher’s concurrency limit from 4 to 1, so the “all three requests are open at once” condition could never be true. All five tests caught it, each at the 5-second time budget, and I reverted the mutation.

One caveat: that mutant ran under the serial plan. Under heavy parallel load, a real hang fails after 100 turns, which takes longer than 5 seconds. That’s the trade-off I chose, and it applies to this helper and this workload. It isn’t a general fairness guarantee for Task.yield().

A second guess hiding in the same tests

Three tests waited for the first request and then gave the scheduler “five settling turns” before asserting that all three were open. That turn count was another guess, and it could fail under load the same way. The fix was to wait for the exact condition:

// before
try await Wait.until { service.pendingCount > 0 }
await Wait.untilSettled()
#expect(service.pendingCount == 3)

// after
try await Wait.until { service.pendingCount == 3 }

Takeaway

A wall-clock timeout in a polling helper measures how busy the process is, not whether your code is stuck. When a wait times out only in the big parallel run, count its turns before you suspect a deadlock. If it got only a handful of turns, the actor was starved. Make the helper fail only after both a time budget and a turn budget run out. Then check the fix with a mutation that should hang and make sure it still fails quickly.

Try it on your own polling helper: add a turn counter and run the full target in parallel once. If the worst wait used single-digit turns but took seconds, the timeout is measuring main-actor contention.