Testing on Lessons learned

Swift Testing withKnownIssue can hide snapshot failures

Fri, 30 Oct 2026 19:45:00 +0100

Two snapshot tests passed while reporting:

No reference was found on disk. Automatically recorded snapshot:

Their Swift Testing withKnownIssue wrapper accepted every issue inside the snapshot assertion, including missing references. Adding a matching: closure limited the exception to expected dependency reports. A deliberate wrong-state snapshot then failed, even though those reports remained known issues.

Restrict which issues count as known

The view had two states: a progress indicator before a check completed, and content afterward. Rendering the content also touched an unimplemented test dependency. The snapshot helper wrapped rendering and comparison in withKnownIssue(isIntermittent: true) to tolerate that report.

Without a matcher, the wrapper also accepted issues from the assertion it surrounded. The initial recording run exited successfully: two tests passed with three known issues. Two of those issues said references were missing; only one was the dependency report the wrapper was meant to tolerate.

The historical fix added this predicate:

String(describing: issue).contains("Unimplemented")

Here is the same policy as a generalized synchronous helper. Pass an existing, nonthrowing snapshot assertion as its body:

import Testing

func withExpectedDependencyIssue(_ body: () -> Void) {
 withKnownIssue(isIntermittent: true) {
 body()
 } matching: { issue in
 String(describing: issue).contains("Unimplemented")
 }
}

The public withKnownIssue matching overload takes a predicate over Issue. Returning false leaves an issue outside this exception. The synchronous overload is rethrows; this helper needs no try because its body is nonthrowing. A throwing assertion needs corresponding throwing handling.

isIntermittent: true allows the body to record no matching issue. That was useful for the progress state, which did not touch the dependency. It does not restrict which issues match. Without matching:, an intermittent wrapper still accepts everything recorded in its scope.

A mutation must change the rendered output

The first attempted verification changed a prefilled field. Both snapshot tests still passed. That was an ineffective mutation for this check; it did not establish that the assertion could detect the intended visual regression.

The stronger mutation completed the check before constructing the view in the test for the before-check reference. It deliberately compared the content state against the progress-state baseline.

That run exited with code 65. The before-check test failed, while the after-check test passed with its expected known issue. Across the two tests, the runner reported three issues, including two known issues. Restoring the original setup produced a successful run of 239 tests in 24 suites with one known issue.

This shows that the narrowed helper rejected an incorrectly rendered state. The session did not run that same wrong-state mutation against the original broad wrapper, so it does not provide an identical before-and-after mutation comparison. The missing-reference reports already demonstrate the broad wrapper’s excessive scope.

Keep the exception smaller than the assertion

The repair used here matched a description substring. It remains brittle: another report containing Unimplemented would also match, and diagnostic wording can change. Where the reporting library exposes a stable public discriminator, prefer that over a broad text search. Better still, provide the missing test dependency so the assertion needs no exception.

Keep that follow-up separate from what was verified. Other snapshot helpers retained broad wrappers; this change did not repair them. A passing suite using this helper says nothing about their failure detection.

These results came from the original Xcode simulator runs. The generalized helper follows Swift Testing’s public matching overload ; it has not been compiled separately.

Before trusting a snapshot helper with a known-issue scope, force a visibly different state against an existing reference and require a failing run. Restore the state and require green again.

A shared Swift timeout left an earlier task without a deadline

Fri, 30 Oct 2026 17:37:00 +0100

A Swift service had a timeout, but waiting for its in-flight work could still hang. A held-fetch regression test reached Swift Testing’s outer limit:

Time limit was exceeded: 60.000 seconds

Starting another check replaced the shared timer while the earlier fetch kept running. The fix was to give each tracked operation its own cancellation deadline and cancel that deadline when the operation ended. That still depends on the fetch cooperating with cancellation.

One timer could not cover overlapping fetches

The service could start another asynchronous check before the previous one finished. A separate tracker retained the in-flight tasks so a caller could wait for their completion. Its waiter took a snapshot of those tasks and awaited their results.

The mistaken assumption was that the service’s existing timeout covered everything in that snapshot. Review found that starting a check reset the shared timer without canceling the previous fetch. Finishing a check also reset the timer so that an older completion could interfere with a newer check’s deadline.

An illustrative sequence makes the ownership mismatch visible:

start A tracked: A shared timer: A
start B tracked: A, B shared timer: B; A's timer reset
finish B tracked: A shared timer: reset
wait for A A still running no cancellation deadline for A

The tracker remembered more work than the timer covered. Making that timer longer would leave the same mismatch.

Put the deadline beside the operation

The correction belonged where the tracker created each fetch task. That scope already knew which task it owned and when it finished.

For each operation, create a separate deadline task that sleeps for the configured duration, then calls cancel() on that operation’s task handle. Keep the deadline handle local. On exit, cancel the deadline and remove the operation from tracking. Forward cancellation from the caller to the fetch as well.

The resulting ownership is:

operation A owns deadline A; completion A cancels deadline A
operation B owns deadline B; completion B cancels deadline B

Canceling a deadline’s sleep must end the deadline task. Swallowing the sleep’s cancellation error and continuing to cancel() would turn normal cleanup into an unintended cancellation request.

Keeping the handles local also means A’s cleanup cannot accidentally cancel B’s deadline through a shared slot.

Prove that the waiter needs this deadline

The regression test used a fetch held until cancellation, a short injected operation timeout, and a signal confirming that the fetch had started. Nothing outside the tracker canceled it. The test then awaited completion and checked that the fetch ended before the waiting caller continued.

Before the implementation used the injected timeout, the test failed at the one-minute limit. After adding the operation-local deadline, the archived test output reports the held-fetch test passing and the focused test target exiting successfully.

That establishes the missing-deadline behavior for a cooperative fake. It does not reproduce the entire overlapping-check sequence live. For your own tracker, also hold A open while B starts and finishes, then confirm that A still receives its own cancellation request.

A cancellation deadline is not a hard timeout

Swift’s Task.cancel() documentation explicitly says cancellation is cooperative. Calling it flags the task and runs cancellation handlers; arbitrary work does not automatically stop. A fetch that ignores cancellation can outlive this deadline and keep a completion waiter suspended.

Later work therefore separated the operation’s cancellation deadline from the caller’s user-facing wait budget. It also added recovery after an unsuccessful check. Giving every operation a deadline did not settle how long a caller should wait or whether continuing after failure was safe.

Before relying on a service timeout, start two operations and finish the newer one first. Check that the older operation still has its own cancellation deadline.

SwiftUI large title disappears after tapping the selected tab again

Fri, 30 Oct 2026 14:38:00 +0100

The Settings large title disappeared after tapping the already-selected tab again. Its space stayed empty; the small title still appeared when scrolling, and the accessibility check passed:

titleBefore=true titleAfter=true

An opaque navigation-bar background was painting over the title. The eventual repair removed a screen modifier’s writes to UINavigationBar.appearance(). Hiding the affected screen’s bar background was an intermediate workaround that introduced a font regression.

The title still existed

The prototype varied the tab view, loading state, scroll-view wrapper, and swipe-back helper. Its accessibility existence checks stayed green. Adding the app’s opaque navigation-bar appearance also produced green checks, even though the title vanished visually.

Counting dark pixels in the prototype’s title region exposed the difference:

Appearance	After first tap	After re-tap
Default	12955	12955
Opaque	12955	0

Those are diagnostic counts from one fixed light-background rendering, not a portable visibility threshold. They contradicted the existence assertion directly.

Before-and-after hierarchy dumps from the running app showed the same title view moving from the navigation bar into the scroll view. The bar’s opaque background then covered it. Initially, the screen displayed a loader without a scroll view, so UIKit built the title inside the bar. Re-tapping the selected tab triggered the transition that exposed the problem on the investigated simulator.

This establishes a rendering mechanism in that environment. It does not establish identical title placement on every supported runtime.

The offending modifier intended to remove a divider and set one screen’s background. Its essential mistake looked like this generalized fragment inside a SwiftUI .onAppear closure:

let appearance = UINavigationBarAppearance()
appearance.configureWithOpaqueBackground()
appearance.backgroundColor = .secondarySystemBackground
appearance.shadowColor = .clear

UINavigationBar.appearance().standardAppearance = appearance
UINavigationBar.appearance().scrollEdgeAppearance = appearance

The closure belongs to a screen; the appearance proxy configures all instances of that class. UIKit applies appearance changes when a view enters a window. A different tab’s later navigation bar can inherit those settings, so screen order becomes part of the bug’s setup.

The repair deleted this global writer. The calling screens already declared their backgrounds through SwiftUI’s per-screen toolbar styling; the investigated runtimes did not need a replacement divider-hiding implementation. Their existing snapshots passed without re-recording. Appearance save-and-restore workarounds around a system-presented screen went away with the leak.

A focused test supplied stronger evidence than the original title lookup: show a styled screen, then create another navigation bar and check its background. It failed before the deletion because the next bar inherited a background, then passed afterward. Separate checks guarded the calling screen’s color and the next large title’s font.

Why the one-line hotfix was reversed

The initial workaround used .toolbarBackground(.hidden, for: .navigationBar) on Settings. It kept the prototype’s title visible under the opaque appearance.

The attempted app snapshot never reproduced the original disappearance. The real screen reloaded into a spinner first, unlike the prototype’s scroll-view-first case. Capturing the loaded window still showed the title without the fix. We dropped that test; it couldn’t serve as a regression guard.

Later checks found that hiding the background also made the large title use the system font instead of the intended custom font. A snapshot cropped to the bar failed with that modifier present and passed after its removal. Once the global opaque write was gone, the workaround no longer had a reason to stay.

Verify the interaction, not just the label

Builds and broad test plans passed after the cleanup. Final manual checks in the running app remained open, and older supported runtimes were untested. The font snapshot and next-bar test do not close that gap.

For release verification:

Visit the screens that previously wrote global appearance, then open Settings. Repeat with a different screen order.
Tap the selected Settings tab again. Capture the title before and after, then scroll through its large-to-inline transition.
Check actual title pixels and contrast against the background. Use the hierarchy to explain occlusion; accessibility existence, or isHittable alone, is not universal proof that text is painted visibly.
Repeat on supported runtimes, checking pushed screens’ bar styling too.

Audit screen-level UINavigationBar.appearance() writes before adding another visibility workaround.

Xcode Archive can ship the wrong display name despite correct source settings

Thu, 29 Oct 2026 14:22:00 +0100

A production iOS archive had the production bundle ID and backend environment, but its CFBundleDisplayName was the beta name. The tagged source said otherwise, and App Store Connect never shows that name, so only an installed build revealed it. Add a Run Script phase that checks the processed app Info.plist, so a manual Product > Archive also rejects the wrong identity.

Using synthetic names, the mismatch was:

CFBundleIdentifier = com.example.notebook
CFBundleDisplayName = NotebookBeta

Source checks missed the artifact

The investigation initially ruled out the mismatch from project settings and runtime configuration checks. That conclusion was too strong. The runtime check validated the product variant against the backend environment; it did not validate the display name. A production bundle with a beta display name could satisfy it.

Reading the archived app’s plist settled the question. Its name disagreed with the committed production settings. An uncommitted local edit was the likely explanation, but the archive alone could not establish exactly how the value changed.

The release flow had no step that showed the name either. App Store Connect metadata covers the store listing’s app name, not the bundle’s CFBundleDisplayName, which only appears under the home screen icon. The only check is to install the production build from TestFlight, or from the App Store after release, and read that label by hand. It is easy to overlook when the bundle ID and backend already look right.

Source-level regression tests still help, but manual Archive does not run the scheme’s Test action. A check confined to a deployment lane also misses someone archiving directly in Xcode.

Check the processed plist during the build

Add a Run Script phase named Verify App Identity to the app target, after its existing phases. Use /bin/sh. Leave Run script only when installing unchecked.

Declare this Input File:

$(TARGET_BUILD_DIR)/$(INFOPLIST_PATH)

Declare this Output File:

$(DERIVED_FILE_DIR)/verified-app-identity.stamp

Uncheck Based on dependency analysis for this small guard. It should validate every build, including repeated archives. The input declaration still establishes the dependency on plist processing; visual phase order alone is insufficient. Apple’s Run Script documentation explains how declared inputs and outputs determine task ordering.

Paste this script, adapting the explicit configuration allowlist:

set -eu
: "${SCRIPT_OUTPUT_FILE_0:?Declare the stamp Output File}"
stamp="$SCRIPT_OUTPUT_FILE_0"
rm -f "$stamp"
: "${TARGET_BUILD_DIR:?}" "${INFOPLIST_PATH:?}" "${CONFIGURATION:?}"
plist="$TARGET_BUILD_DIR/$INFOPLIST_PATH"
fail() {
 printf 'error: %s\n' "$*" >&2
 exit 1
}
case "$CONFIGURATION" in
 Release)
 expected_id=com.example.notebook
 expected_name=Notebook
 ;;
 Debug)
 expected_id=com.example.notebook.beta
 expected_name=NotebookBeta
 ;;
 *) fail "No identity policy for configuration $CONFIGURATION" ;;
esac
[ -f "$plist" ] || fail "Missing processed Info.plist: $plist"
id=$(/usr/libexec/PlistBuddy -c 'Print :CFBundleIdentifier' "$plist") ||
 fail 'Cannot read CFBundleIdentifier'
name=$(/usr/libexec/PlistBuddy -c 'Print :CFBundleDisplayName' "$plist") ||
 fail 'Cannot read CFBundleDisplayName'
[ "$id" = "$expected_id" ] && [ "$name" = "$expected_name" ] ||
 fail "$CONFIGURATION requires $expected_id / $expected_name; got $id / $name"
printf 'Verified app identity: %s / %s\n' "$id" "$name"
: > "$stamp"

Do not substitute the source INFOPLIST_FILE: generated keys and build-setting expansion can change the packaged values. Keep the expected identity independent of the settings under test.

The declared paths also describe the script’s file access for user-script sandboxing. Keep sandboxing enabled where possible and test these declarations with your Xcode version. The original target had it disabled. The stamp lives outside the signed app and is created only after validation. Removing it first, failing on missing values, and running unconditionally prevent an old stamp from becoming a false success after a failed check.

Prove that wrong builds fail

The historical implementation passed normal builds across all four original configurations. Four deliberate mutations failed: the production ID, production name, beta ID, and beta name, changed separately. Each failure reported the mismatched identity; the project file was restored afterward. These were build checks, not evidence of a fresh Archive after the fix.

For your project, repeat those negative controls, then restore the settings and archive twice without cleaning. Confirm the guard runs both times and inspect the resulting archive’s app plist. Also test a missing key and an unrecognized configuration.

This checks the app’s base plist only. Localized InfoPlist.strings can affect the displayed name, and extensions need separate policies. Check the Archive action’s selected configuration in your scheme, then deliberately break its display name and confirm Archive fails before trusting the guard.

Swift Testing: .serialized did not stop another suite deleting test files

Wed, 28 Oct 2026 14:36:00 +0100

A filesystem test failed while writing its fixture, even though its suite used @Suite(.serialized) and each fixture had a UUID directory. The underlying error was:

NSPOSIXErrorDomain Code=2 "No such file or directory"

Another suite was constructing the application core, whose initializer swept the shared cache directory. Injecting that startup operation, with an inert test value, stopped unrelated tests from deleting each other’s fixtures.

A unique child directory still has a shared parent

The failing suite created a directory, then wrote a file into it. Its own cleanup tests ran serially. Elsewhere in the same Swift Testing process, ordinary tests constructed the core as part of their setup. Every construction triggered a sweep of the common cache parent.

The destructive interleaving was small:

cleanup test: create cache/<UUID>/
unrelated test: construct core -> sweep cache children
cleanup test: write cache/<UUID>/fixture.bin -> missing directory

UUIDs prevented name collisions. They could not protect a child from an operation that deleted every child.

Swift Testing’s parallelization trait documentation describes the scope: .serialized orders a suite’s contained tests and sub-suites. It does not order those tests relative to unrelated suites. Putting the attribute on the failing suite left the other constructors free to run alongside it.

Make ordinary test construction inert

The fix moved the startup sweep behind an injected operation. Production kept the real sweep; the test dependency supplied a no-op. Tests that needed to verify sweeping explicitly selected the real operation.

Here is a generalized closure-based adaptation, independent of a dependency-injection library:

import Foundation

struct StartupSweep {
 var run: () throws -> Void

 static var inert: StartupSweep { StartupSweep(run: {}) }

 static func live(root: URL) -> StartupSweep {
 StartupSweep {
 let manager = FileManager.default
 for child in try manager.contentsOfDirectory(
 at: root, includingPropertiesForKeys: nil
 ) {
 try manager.removeItem(at: child)
 }
 }
 }
}

struct CacheCore {
 init(startupSweep: StartupSweep) throws {
 try startupSweep.run()
 }
}

The production composition root passes .live(root: cacheRoot). The ordinary test factory passes .inert; neither caller relies on an initializer default. This example propagates filesystem errors so a cleanup test cannot pass by silently ignoring a failed sweep.

For a test-owned root containing a prepared fixture file, the two assertions look like this. This is a test-body excerpt using Testing:

// Ordinary construction must preserve the existing fixture.
_ = try CacheCore(startupSweep: .inert)
#expect(FileManager.default.fileExists(atPath: fixture.path))

// Cleanup coverage explicitly opts into the real operation.
_ = try CacheCore(startupSweep: .live(root: root))
#expect(!FileManager.default.fileExists(atPath: fixture.path))

Use a disposable root for this check. Never point the illustrative live operation at a directory containing files you need. Injecting a separate root per test is another way to isolate filesystem work when the design permits it.

Prove the constructor contract, then repeat the broader target

The historical investigation used an isolated-suite control: the cleanup suite repeated 30 times with zero issues. The broader core target failed on its first run. That narrowed the search to interference from outside the serialized suite.

A regression test then created a fixture, built the core-under-test dependencies, and asserted that the file still existed. Before the injection fix, that assertion failed. It converted an intermittent write failure into a direct check of the destructive constructor behavior.

After the fix, the archived broader-target output reported:

exit: 0
** TEST SUCCEEDED **
issues:
0
max repetition:
repetition 30
✔ Test run with 165 tests in 28 suites passed after 2.806 seconds.

Those results verify the original fix under Xcode and iOS Simulator. The closure-based adaptation above has not been compiled separately. The reported test count is the runner’s summary, not a total multiplied by the repetition limit.

When a serialized suite loses files, check every initializer that can sweep their shared parent, including constructors called by unrelated suites.

A CVE cross-reference is not an advisory entry

Wed, 28 Oct 2026 00:00:00 +0000

A vulnerability index can have the correct set of CVE identifiers and still assign them to the wrong advisories. A regex that searches every paragraph also captures narrative references such as “related to CVE-…”. Extract identifiers from declaration blocks, and verify the resulting relationships before using the index to make patch or exposure claims.

The trap is easy to miss during deduplication: the referenced identifier may already have a legitimate entry elsewhere. Removing its false assignment changes the relationships without changing the global distinct-ID count.

How the wrong relationship gets created

Consider two synthetic advisories. Every identifier below is fictional:

Advisory A, component Reader, declares CVE-2099-10001.
Its impact paragraph mentions CVE-2099-10002 as a related issue.
Advisory B, component Writer, declares CVE-2099-10002.

An extractor that carries the current advisory and component into every regex match produces three rows. The false row associates Advisory A and Reader with the second identifier. Both the broken and corrected indexes contain two distinct identifiers.

This error can contaminate a patch matrix: an analyst might attach the wrong affected component or release boundary to a real vulnerability. It requires no malicious input. A crafted narrative mention could produce the same misassociation in a pipeline that ingests untrusted documents, but that is a possible abuse of the parser, not evidence of an exploited advisory feed.

Filter by document role before matching identifiers

For one fixed document layout, a practical repair is to collect complete paragraph text and process only blocks beginning with the declaration marker CVE-. Collecting the whole paragraph matters: identifiers can be split across inline elements, and indentation should not change classification.

Here is a runnable Python example using a deliberately small HTML schema. Save it as check_advisories.py and run python3 check_advisories.py:

import re
from html.parser import HTMLParser

ID = re.compile(r"\bCVE-\d{4}-\d{4,}\b")
FIXTURE = """
<article data-advisory="A" data-component="Reader">
 <p> <strong>CVE-2099-10001</strong> </p>
 <p>Related to <em>CVE-2099-10002</em>.</p>
</article>
<article data-advisory="B" data-component="Writer">
 <p>\n CVE-2099-<span>10002</span> </p>
</article>
"""

class Index(HTMLParser):
 def __init__(self, declarations_only):
 super().__init__()
 self.declarations_only = declarations_only
 self.context = None
 self.parts = None
 self.rows = set()

 def handle_starttag(self, tag, attrs):
 if tag == "article":
 attrs = dict(attrs)
 self.context = (
 attrs["data-advisory"], attrs["data-component"]
 )
 elif tag == "p":
 self.parts = []

 def handle_data(self, data):
 if self.parts is not None:
 self.parts.append(data)

 def handle_endtag(self, tag):
 if tag == "p" and self.parts is not None:
 text = "".join(self.parts).strip()
 if not self.declarations_only or text.startswith("CVE-"):
 for identifier in ID.findall(text):
 self.rows.add((*self.context, identifier))
 self.parts = None
 elif tag == "article":
 self.context = None

def extract(declarations_only):
 parser = Index(declarations_only)
 parser.feed(FIXTURE)
 parser.close()
 return parser.rows

broken = extract(False)
fixed = extract(True)
assert fixed == {
 ("A", "Reader", "CVE-2099-10001"),
 ("B", "Writer", "CVE-2099-10002"),
}
assert broken - fixed == {("A", "Reader", "CVE-2099-10002")}
assert {row[2] for row in broken} == {row[2] for row in fixed}
print(f"relationships: {len(broken)} -> {len(fixed)}; distinct IDs unchanged")

Expected output:

relationships: 3 -> 2; distinct IDs unchanged

A prefix guard has a document contract

This parser is a fixture, not a general advisory importer. A narrative paragraph could itself begin with a CVE identifier; another publisher might declare identifiers in a table or list. For production ingestion, select the publisher’s actual declaration element or structured field, record the source location, and reject or flag layouts the adapter does not recognize. The standard library’s HTMLParser documentation describes its event interface; it cannot infer a publisher’s semantic roles for you.

Keep regression fixtures for narrative references, whitespace, and inline markup. Review added and removed (advisory, component, CVE) tuples during a parser change, including declarations the new selector might accidentally discard.

Before trusting your next vulnerability-index refresh, check one known cross-reference against its owning advisory. A stable distinct-ID total cannot validate that assignment.

An offline cache TTL needs a clock-rollback policy

Tue, 27 Oct 2026 15:44:00 +0100

AAn offline cache can pass its age check after the device clock moves backward. A timestamp later than “now” produces a negative age, and a check that only asks whether age is below the TTL accepts it. Reject future timestamps and remember the latest wall-clock time observed, while recognizing that neither check provides trusted elapsed time.

A negative age passed the upper-bound check

The feature reused a biometric-protected Keychain cache shared with a companion extension. Its initial eligibility rule was effectively:

now - lastWritten <= maxAge

That has two problems. Moving the clock before the write makes the age negative. Moving it backward after a previous access can also make an expired cache look young again without crossing the write timestamp.

The cache contained independently written items. Using the newest item’s date would make an older item appear refreshed, so eligibility used the oldest modification date. A separate check rejected any item dated in the future. Checking only the oldest date would miss a newer, future-dated item.

Retain the latest observed time

The repair added a clock mark in a separate Keychain item. Each eligibility read retained max(previousMark, now). If now preceded the previous mark, access was refused. Writing the rolled-back time into the mark would let the next read pass, so repeated reads were part of the regression test.

This Python model isolates the arithmetic from Keychain storage. Inputs are timestamps in the same unit and a nonnegative maximum age. None represents initial mark creation; it does not distinguish first use from a failed storage read.

def cache_status(written_at, now, mark, max_age):
 if not written_at:
 return "unavailable", mark

 previous = now if mark is None else mark
 updated = max(previous, now)
 fresh = (
 now >= previous
 and max(written_at) <= now
 and now - min(written_at) <= max_age
 )
 return ("fresh" if fresh else "expired"), updated

Persist the returned mark reliably. Define separate behavior for a missing mark, failed reads, and failed writes. Serialize the read/advance/write operation across its writers: two callers storing independently calculated maxima can otherwise overwrite a newer mark with an older one. This function does not implement those storage guarantees.

A successful online sign-in reset the mark to the current time, allowing recovery after a fast clock was legitimately corrected. That reset belonged to the authenticated online path, not to an offline retry.

Refusing access did not mean deleting the cache

The main app’s expiry policy could not delete data the companion extension still needed. An expired result blocked that entry point and requested an online sign-in; it left the shared cache intact.

Offline use also had to stay separate from an ordinary signed-in state. Otherwise, the normal cache writer could rewrite cached data, advance its modification dates, and renew its apparent freshness without server contact. A timestamp is useful evidence of synchronization only while that write invariant holds.

Eligibility was checked again when opening the cache. A login screen left visible across the deadline could not reuse the earlier “fresh” result.

Test the rollback twice

The original implementation’s red run reported four failed clock-mark assertions, including rollback refusal, repeated refusal, mark advancement, and online-sign-in reset. After the repair, the recorded core-target run passed 184 tests in 31 suites. Those are historical Xcode results, not a new Keychain integration run.

A separate run of the function above passed 15 synthetic-timestamp tests, covering expiry boundaries, future-dated items, repeated rollback refusal, and rechecking at open without mutating the cache. It did not exercise Keychain persistence. For example, two reads with these synthetic inputs must preserve the higher mark:

result, mark = cache_status([100], 190, 200, 1000)
assert (result, mark) == ("expired", 200)
assert cache_status([100], 190, mark, 1000) == ("expired", 200)

A high-water mark cannot establish time that was never observed

A device can spend real time offline while its wall clock never exceeds the saved mark. These checks cannot prove how much time passed, guarantee remote revocation, or withstand a compromised device that can alter the protected state. Biometric access control does not authenticate the clock.

Test your cache by expiring it, moving the injected clock backward, and reading eligibility twice. Both reads should refuse access, and the second must not inherit a lowered mark.

iOS jailbreak protection in 2026: detection is a risk signal, not a security boundary

Mon, 26 Oct 2026 23:20:00 +0100

An iOS app had jailbreak detection, passed an external pentest, and still spent more than a year accepting every jailbroken device because an asynchronous result was discarded. After that bug was fixed, the checks were still a 2025 list of Cydia-era paths that missed current rootless and hidden jailbreaks.

If the requirement says “the app must not run on a jailbroken phone,” the useful answer in 2026 is not a longer path list. Treat jailbreak detection as a bypassable risk signal, decide what sensitive operation the signal protects, enforce that decision at the server where possible, and test the whole policy against current jailbreak and bypass tools.

The short version

A defensible implementation today has five parts:

Write a threat model that explains why a compromised device materially changes this app’s risk.
Combine independent local signals and run them near sensitive operations, not only at launch.
Apply a proportional policy: observe, step up, restrict sensitive data, or refuse service according to confidence and impact.
Put the final authorization decision on a server and use App Attest to protect app-to-server requests. Do not call App Attest to prove that a device is not jailbroken.
Test the release binary statically, dynamically, and by trying to bypass it on representative clean and jailbroken devices.

This is also the difference between “the code contains a jailbreak check” and an engineering conclusion that can survive review.

How a working check silently stopped working

The first implementation was conventional:

sysctl checked P_TRACED and terminated a release build under a debugger;
known paths such as /Applications/Cydia.app, /bin/bash, /usr/sbin/sshd, and /etc/apt were checked;
the app tried to write /private/jailbreak.txt outside its sandbox;
cydia:// was passed to canOpenURL;
loaded Mach-O images were searched for MobileSubstrate.

It ran only in non-Debug builds on physical devices. That was sensible, and the pentest team verified it.

One probe had never worked in the affected app targets: canOpenURL returns false for schemes that are not declared under LSApplicationQueriesSchemes, and cydia was not declared. An app extension could not use UIApplication.shared at all. Current IOSSecuritySuite also removed the cydia:// check because an App Store app registered the scheme and caused false positives. URL-scheme probes are weak evidence even when configured correctly.

Then the check moved behind a function that started a Task, returned from the function immediately, and ignored the task’s Bool. The user interface always received true. The debugger check still appeared to work because fatalError happened inside the task, masking the control-flow bug. Later refactors moved the broken call without changing its behavior.

The eventual fix was not another jailbreak signature. It was to make validation genuinely asynchronous, await it, and test the production call site as well as the checker.

That history produced three reusable lessons:

Test enforcement, not just detection. A perfect signal whose result is ignored protects nothing.
Test every app entry point and extension. The main app, credential provider, widget, and background path do not necessarily share a process or API surface.
Test the release configuration on hardware. A Debug simulator test proves code structure, not production behavior when the checks are compiled out there.

When OWASP says to implement jailbreak detection

OWASP MASTG 2.0, released in July 2026 , makes the trade-off unusually explicit. MASTG-BEST-0030 is currently tagged Android and says “root detection,” but its risk principle applies directly to the equivalent iOS control: it is a bypassable, cost-raising environment signal. It should be used only when compromised-device risk materially affects the app. The selection and response should follow the threat model and risk tolerance.

That points toward jailbreak protection when a compromised platform makes a concrete abuse case easier, for example:

extracting high-value secrets or regulated data from the app process or local cache;
instrumenting transactions, credential filling, cryptographic operations, or fraud controls;
changing a game, licensing, payment, or content-access decision made locally;
automating a protected API through a modified client;
defeating controls that depend on the sandbox, Keychain access policy, or trusted user interaction.

It does not make jailbreak detection a universal mobile-app baseline. A low-risk public-content app may gain little while inheriting false positives, support work, accessibility problems, and an exclusion policy it cannot reliably enforce.

MASVS-RESILIENCE-1 states the security concern: many other controls assume that the platform’s sandboxing, secure storage, and biometrics can be trusted. But the wider MASVS resilience guidance treats anti-tampering and runtime self-protection as defense in depth, never a replacement for the baseline security architecture.

The current techniques are rootless and designed to hide

The old signature list described rootful Cydia and MobileSubstrate installations. Current jailbreaks changed the filesystem layout and then added explicit concealment.

Dopamine is a rootless jailbreak. Its bootstrap commonly appears below /var/jb, with ElleKit providing tweak injection. Dopamine 1.1 introduced JB_ROOT_PATH in the environment of processes with tweaks enabled, while noting that /var/jb might become optional to impede detection. Palera1n has its own loader and helper artifacts. Frida may appear as frida-server, Frida Gadget, a loaded image, or merely runtime behavior after its obvious names have been changed.

At the time of this review, Dopamine’s current upstream release was 3.0.12, published October 20, 2026 . A signature inventory that stops at Cydia or even Dopamine 2 is already historical.

Dopamine 2.4’s release notes are the more important evidence: “Hide Jailbreak” was changed specifically to hide the jailbreak to the point where it should be undetectable. RootHide’s Dopamine integration randomizes the jailbreak root and selectively keeps jailbreak changes away from blacklisted apps. Its purpose is jailbreak-detection bypass.

A 2026 local detector should therefore look across different classes of evidence, for example:

rootful and rootless files, directories, symlinks, package-manager state, and randomized .jbroot-* roots;
whether the sandbox permits a collision-safe write to a protected path;
debugger state and failure of the debugger-status query itself;
suspicious environment keys such as DYLD_INSERT_LIBRARIES and JB_ROOT_PATH;
loaded images associated with Frida, ElleKit, Substrate, Substitute, libhooker, RootHide, or Dopamine’s system hook;
behavior that should be prohibited by the sandbox, where the API is appropriate for App Store software;
server-side anomalies and app-integrity assertions.

Use more than one API for important filesystem evidence. A FileManager.fileExists check, a POSIX lstat, and directory enumeration do not create certainty, but an attacker now has more hooks to maintain consistently.

Do not equate more signatures with “state of the art.” Public signature lists are useful coverage for unsophisticated and accidentally exposed jailbreaks. RootHide, renamed Frida components, or a patch to the function that aggregates all checks can make every local answer say “clean.” OWASP’s jailbreak detection knowledge article says the same thing: these defenses are not effective by themselves, although distributing them through the app increases the cost of bypassing them.

The response matters more than the Boolean

“Jailbroken: yes or no” is too weak a model for a fallible set of signals. Record an internal risk result with the evidence and confidence, then map it to a product policy.

Situation	Reasonable response
One weak or ambiguous signal	Telemetry, retry, or step-up authentication
Several independent local signals	Block sensitive operations; avoid disclosing the exact failed probe
Debugger or injected runtime tool in a high-risk flow	Refuse the operation and invalidate relevant server state
Unsupported App Attest on an otherwise supported device	Treat as a risk signal according to the documented policy
Enterprise-managed fleet with a strict posture requirement	Evaluate Managed Device Attestation at the relying service

OWASP recommends proportional responses and warns that aggressive lockout can affect legitimate users. For an app that stores documents and credentials, a useful soft block can allow login and folder metadata while refusing document bodies, downloads, offline queues, and credential release.

That restriction must be in application logic, not just a screen with a Close button. Better still, the server should not return the sensitive payload. Otherwise, a hook can dismiss the screen or call the same API directly.

A hard process exit is appropriate only if the threat model justifies denial of all functionality and the product accepts false positives. Crashing also gives poor observability and can turn an operational failure into a permanent lockout. A generic refusal with server-side telemetry is usually easier to operate.

App Attest authenticates app instances, not clean devices

Apple’s WWDC26 App Attest session describes the current protocol:

the server initiates enrollment and issues a challenge;
the app creates a Secure Enclave-bound key and attests it;
the server validates and stores the attestation and public key;
sensitive requests include a fresh server challenge in an assertion over the payload;
the server validates the signature and requires a strictly increasing counter to reject replays.

This is valuable. It lets the server distinguish a valid published app instance, detect unexpected signing or launch categories, and bind authorization to a payload that has not been altered in transit.

It is not a jailbreak oracle. The same 2026 session says that a compromised device can still pass attestations and broker valid attestations for modified clients. Apple provides the fraud metric to help detect that pattern. Use App Attest as another server-verified signal, with retry and onboarding policy for legitimate failures.

For managed organizational devices, Managed Device Attestation provides Secure Enclave-backed evidence about device identity and security posture. That is stronger input for a managed-fleet trust decision, but it is not a general capability that a consumer app can require from every unmanaged phone.

Test implementation, behavior, and bypass resistance

OWASP MASTG 2 separates two questions that teams often collapse into one.

MASTG-TEST-0240 is static: extract the shipped app package and inspect the binaries for jailbreak-detection mechanisms. This catches missing code, build flags that compiled it away, or a check that never reached a target. OWASP warns that automated static analysis is not exhaustive; manual reverse engineering may be necessary.

MASTG-TEST-0241 is dynamic: run the app, identify the checks, and try to bypass them. A test fails when detection is absent, but passing a single predefined bypass tool doesn’t prove resilience because its scripts may be incomplete or outdated.

For a real release, I would keep this matrix with the evidence from each run:

Test	Expected evidence
Clean supported iPhones and iOS versions	No false refusal; all sensitive flows work
Release build on physical hardware	Checks execute; Debug/simulator exclusions do not leak into the archive
Rootful and current rootless jailbreaks	Each applicable signal fires; policy is enforced
Dopamine Hide Jailbreak and RootHide blacklist	Document what remains visible and what becomes undetectable
Frida server and embedded/renamed Gadget	Known artifacts are detected; renamed forms challenge bypass resistance; outcomes are recorded
Hook the aggregate result and individual APIs	Sensitive server operation still requires valid authorization and assertions
Main app, extensions, background entry points	Each entry point validates before releasing its protected capability
Async and error paths	Results are awaited; query failures follow the documented fail-open or fail-closed policy
Pentest bypass configuration	It cannot enter a production archive; CI tests that invariant

Unit-test injected signals, cleanup failures, query failures, and clean-device results. Then add integration tests around the call site: the protected operation must not run after validation fails. Finally, let a tester attack the binary rather than merely demonstrating a known file on a jailbroken test phone.

A reviewable 2026 “lege artis” baseline

“State of the art” is not established by citing one library or by counting checks. For an engineering review, the conclusion should be reproducible, dated, scoped, and explicit about uncertainty. This is the practical baseline I would use when assessing whether an iOS app’s jailbreak protection was implemented with professional care in October 2026:

Define the question and date. State which app version, release configuration, iOS range, device classes, sensitive operations, and attacker capabilities are in scope. Practice changes as jailbreaks and iOS change.
Identify the risk before prescribing the control. Connect compromised-platform capabilities to actual assets and abuse cases. Explain why jailbreak detection materially affects this app.
Select authoritative and current sources. Record versions and access dates for OWASP MASVS/MASTG, Apple platform documentation, relevant jailbreak release notes, and the implementation under review. Treat public detection libraries as examples, not normative standards.
Preserve and identify the evidence. Record the source commit, archive hash, build settings, entitlements, tool versions, device/iOS versions, jailbreak version, and test logs. Work from a reproducible copy.
Separate facts from interpretation. “The production binary contains /var/jb and JB_ROOT_PATH checks” is a fact. “Those checks make bypass harder but cannot establish a clean device” is the interpretation supported by OWASP and current bypass projects.
Analyze the whole control path. Review signal collection, aggregation, async propagation, every entry point, the product response, server authorization, telemetry, and release-lane exceptions. Include negative findings and dead code.
Test both presence and effectiveness. Perform static inspection, clean-device compatibility tests, dynamic execution on relevant jailbreaks, and bypass attempts. A unit test or a pentest on one jailbreak version is not enough.
Consider alternatives and false positives. Explain why a detected artifact could be benign, why a missing artifact does not establish cleanliness, and why the chosen response is proportional.
State limitations. Local checks can be hooked or patched; hidden jailbreaks may expose no artifact; App Attest can succeed on a compromised device; the tested device set is finite.
Make a bounded conclusion. Say whether the examined implementation follows the documented risk policy and raises attacker cost for the tested scenarios. Do not conclude that it guarantees a non-jailbroken device.

This structure mirrors a sound software expert analysis: sources and methods are identified, observations remain distinguishable from interpretation, contrary evidence is not hidden, limitations reduce the stated certainty, and the conclusion answers only the technical question examined. It is guidance for an engineering assessment, not a legal conclusion about liability, contractual compliance, or negligence.

Takeaway

The old implementation failed twice: first because application control flow discarded a valid result, then because its threat model had aged past its Cydia-era signatures. The durable fix is a process—current layered signals, proportional server-enforced policy, and repeated adversarial testing—not a permanent isJailbroken function.

Take one sensitive operation in your app and trace it today from device signal to server authorization. If any step can ignore, hide, or locally override the result, that is the next protection to fix.

References

OWASP MASVS: Resilience against reverse engineering and tampering , accessed October 26, 2026.
OWASP MASTG 2.0.0 release announcement , July 4, 2026.
OWASP MASVS-RESILIENCE-1: Platform integrity , accessed October 26, 2026.
OWASP MASTG-BEST-0030: Implementing Root Detection , accessed October 26, 2026.
OWASP MASTG-KNOW-0084: Jailbreak Detection , accessed October 26, 2026.
OWASP MASTG-TEST-0240: Jailbreak Detection in Code , accessed October 26, 2026.
OWASP MASTG-TEST-0241: Runtime Use of Jailbreak Detection Techniques , accessed October 26, 2026.
Apple: Secure your apps with App Attest, WWDC26 , accessed October 26, 2026.
Apple Platform Security: Managed Device Attestation , accessed October 26, 2026.
Dopamine 2.4 release notes: Hide Jailbreak , February 16, 2025.
Dopamine 1.1 release notes: JB_ROOT_PATH , June 16, 2023.
Dopamine releases , accessed October 26, 2026.
Dopamine 3.0.12 release , October 20, 2026.
RootHide integration for Dopamine 3 , accessed October 26, 2026.
IOSSecuritySuite jailbreak checker , accessed October 26, 2026.

Moodle 5.3 Classic theme serves an empty stylesheet: 'Undefined variable $mds-spacing-md'

Sat, 24 Oct 2026 14:58:00 +0200

After an upgrade to Moodle 5.3, every page on the Classic theme lost its layout. The theme’s stylesheet came back like this:

$ curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \
 'https://moodle.example.com/theme/styles_debug.php?theme=classic&type=scss'
200 0

That’s a 200 with an empty body, and nothing in the logs. With debugging forced to developer level, the SCSS compiler said:

Error while compiling SCSS: Undefined variable $mds-spacing-md: /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle/course.scss on line 132, at column 17

Classic imports Boost’s SCSS, and in 5.3 that SCSS uses the new design system’s tokens. Classic never imports them. One @import fixes it.

The setup

Moodle 5.3.0, PHP 8.4.27, in a Docker image of our own, upgraded from 5.0
The site’s theme is Classic. It left core in 5.3, so the image installs it from moodlehq/moodle-theme_classic (at commit 363a179, the only one when I checked)
Theme designer mode was on, so pages loaded the theme through styles_debug.php

I rehearsed the upgrade on a copy of the production database. Every URL I tried returned 200, and admin/cli/checks.php reported all checks OK. On production, the pages had no layout.

Two things that looked like the cause

The browser console had two errors:

[Error] Failed to load resource: the server responded with a status of 404 (Not Found) (client.bundle.js.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 (Not Found) (react.bundle.mjs.map, line 0)

Those are source maps. The browser asks for them only for its developer tools, and Moodle doesn’t ship them. They don’t affect the layout.

The import map had a few 404s as well, for entries like /core/esm/<rev>/bootstrap/. Those entries end in a slash because they map a whole prefix, so the bare URL has nothing to serve. The modules under them loaded.

The problem showed up when I fetched every stylesheet the front page links to and printed its size:

curl -s https://moodle.example.com/ -o index.html
grep -oE '<link[^>]+stylesheet[^>]*>' index.html \
 | grep -oE 'href="[^"]+"' | sed 's/href="//; s/"$//; s/&amp;/\&/g' \
 | while read -r u; do
 printf '%s %s\n' "$(curl -s "$u" | wc -c)" "$u"
 done

The plugin stylesheets had a few kilobytes each. The theme’s own, type=scss, had zero bytes.

Getting the real error

Moodle catches the compiler’s exception and passes the message to debugging() at developer level, so at the usual debug setting you get nothing at all. You don’t have to change the site’s settings to see it. Compile the theme from the CLI with debugging switched on for that one process (run it as the web server’s user and adjust the path to config.php):

sudo -u www-data php -r '
define("CLI_SCRIPT", 1);
require "/opt/moodle/config.php";
$CFG->debug = E_ALL;
$CFG->debugdisplay = 1;
$t = \core\output\theme_config::load("classic");
$m = new ReflectionMethod($t, "get_css_content_from_scss");
$c = $m->invoke($t, false);
var_dump(is_string($c) ? strlen($c) : $c);
'

get_css_content_from_scss isn’t public, hence the reflection. It returns false when the compile fails, and the error comes with the import chain:

++ Error while compiling SCSS: Undefined variable $mds-spacing-md: /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle/course.scss on line 132, at column 17
Call Stack:
#0 import /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle/course.scss /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle.scss on line 13
#1 import /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle.scss /opt/moodle/public/theme/classic/scss/moodle.scss on line 2
#2 import /opt/moodle/public/theme/classic/scss/moodle.scss (unknown file) on line 127 ++

Why

Boost’s default preset, theme/boost/scss/preset/default.scss, starts with:

// Import Moodle Design System tokens.
// Must be imported before Bootstrap so that the design system tokens can be used to override Bootstrap variables.
@import "design-system";

design-system.scss pulls in the $mds-* tokens. Classic’s presets import fontawesome, bootstrap and moodle, but not design-system, and Classic’s moodle.scss imports Boost’s. So Boost’s course.scss uses $mds-spacing-md, and nothing has defined it.

The fix

Add the import to the top of theme/classic/scss/classic/pre.scss. Classic’s lib.php puts that file in front of the preset, so this covers both presets:

pre=/opt/moodle/public/theme/classic/scss/classic/pre.scss
{ printf '@import "../../boost/scss/design-system";\n\n'; cat "$pre"; } > /tmp/pre.scss
mv /tmp/pre.scss "$pre"
sudo -u www-data php /opt/moodle/admin/cli/purge_caches.php --theme

The path resolves from theme/classic/scss/, as the call stack above shows. After the purge, the same CLI compile returned 1,218,070 characters instead of false, and styles_debug.php sent 1,557,703 bytes.

If Classic is part of your build, as here, apply the patch in the build. A fix made inside a running container is gone the next time the container is recreated.

Once the layout was back, I turned theme designer mode off (admin/cli/cfg.php --name=themedesignermode --set=0). It recompiles the theme on every page view: here, about 0.8 s and 112 stylesheet requests per page.

Takeaway

A Moodle theme that fails to compile still answers 200, just with no CSS, so upgrade checks that look only at status codes pass right over it.

Before your next Moodle upgrade, add the theme’s compiled CSS size to your checklist: anything near zero is a failed compile.

A five-second Swift Testing wait timed out after its loop had run five times

Thu, 22 Oct 2026 16:12:00 +0200

Five concurrency tests passed on their own in 0.086 seconds. In the full target, they failed together at about 6.8 seconds:

✘ Test fetch_threeSources_asksForEverySourceAtOnce() recorded an issue at FetcherConcurrencyTests.swift:21:30:
 Expectation failed: ContinuousClock.now < deadline
✘ Test fetch_threeSources_asksForEverySourceAtOnce() failed after 6.816 seconds with 1 issue.

It looked like a deadlock in the code under test. It was a test helper measuring the wrong thing.

Setup

Swift Testing, an Xcode test target that mixes @MainActor unit suites with snapshot suites.
Two test plans run the same target. One sets "parallelizable": false for it, and the other doesn’t. Only the parallel plan failed.
A small polling helper that the concurrent-fetch tests use to wait for a gated fake service to see requests:

static func until(
 sourceLocation: SourceLocation = #_sourceLocation,
 _ condition: () -> Bool
) async throws {
 let deadline = ContinuousClock.now.advanced(by: .seconds(5))
 while !condition() {
 try #require(ContinuousClock.now < deadline, "Timed out waiting for the condition to hold",
 sourceLocation: sourceLocation)
 await Task.yield()
 }
}

The helper runs on the main actor. Each Task.yield() lets the queued work, in this case the fetcher opening its requests, run before the condition is checked again.

Reproduce before guessing

Running the target alone under the parallel plan with -test-iterations and -run-tests-until-failure failed 3 out of 3 times, always in the same tests and always at the wall-clock deadline. So it wasn’t a rare flake. It failed every time under that plan.

To check whether the waits were stuck or just slow, I temporarily replaced the deadline with a counter and printed what each wait used:

let start = ContinuousClock.now
var turns = 0
while !condition() {
 turns += 1
 await Task.yield()
}
print("WAITPROBE turns=\(turns) elapsed=\(ContinuousClock.now - start) at \(sourceLocation.line)")

5 5.766 FetcherConcurrencyTests.swift:54
5 5.712 FetcherConcurrencyTests.swift:21
4 6.411 FetcherConcurrencyTests.swift:37
3 5.704 SyncLoggingTests.swift:25
3 5.338 InFlightTests.swift:20

Every wait finished, but the slowest needed only 3 to 5 turns and took more than 5 seconds of wall time. The condition was met on almost the first chance the fetcher got to run. It just didn’t get that chance for seconds.

The reason showed up when I sorted tests by duration. In the parallel plan, snapshot suites share the process and the main actor, and each render holds the actor synchronously, some of them for over 7 seconds. While a render ran, the waiting test’s next turn just sat in the queue, and the clock kept running.

Dead end: a longer timeout

I tried a 60-second deadline as an experiment. Both runs passed, with the concurrency tests taking about 7.9 seconds each.

That shows the cause, but it isn’t a fix. 60 is no more principled than 5. A heavier snapshot suite brings the failure back, and a test that really hangs now takes a minute to report it. I reverted it.

Fix: spend both budgets before failing

The wait should fail only once it has had enough chances to make progress and enough time has passed:

static func until(
 sourceLocation: SourceLocation = #_sourceLocation,
 timeBudget: Duration = .seconds(5),
 _ condition: () -> Bool
) async throws {
 let deadline = ContinuousClock.now.advanced(by: timeBudget)
 var turns = 0
 while !condition() {
 let hasBudgetLeft = turns < Constants.turnBudget || ContinuousClock.now < deadline
 try #require(hasBudgetLeft, "Timed out waiting for the condition to hold",
 sourceLocation: sourceLocation)
 turns += 1
 await Task.yield()
 }
}

enum Constants {
 /// A wait under a fully loaded run of this target took at most 5 turns.
 static let turnBudget = 100
}

On an idle main actor, 100 turns go by in milliseconds, so a condition that never holds still fails just after the time budget. On a starved actor, the wait keeps going until it has had real chances to run.

Pin it with a controlled starvation test

The full target is a slow, noisy reproduction, so I wrote a test that intentionally causes starvation. It makes timeBudget a parameter, holds the main actor for longer than that budget, and makes the condition true only afterward:

@MainActor
@Suite
struct WaitTests {
 @Test
 func until_whenTheMainActorIsHeldPastTheTimeBudget_waitsForTheCondition() async throws {
 let holds = LockIsolated(false)
 Task { @MainActor in
 Thread.sleep(forTimeInterval: 0.3) // stands in for a snapshot render
 Task { @MainActor in holds.setValue(true) }
 }
 try await Wait.until(timeBudget: .milliseconds(100)) { holds.value }
 }

 @Test
 func until_whenTheConditionNeverHolds_fails() async {
 await withKnownIssue {
 try await Wait.until(timeBudget: .zero) { false }
 }
 }
}

Before the fix, it failed after 0.335 seconds with the same ContinuousClock.now < deadline error. After the fix, it passed in 0.303 seconds. The second test makes sure the helper can still fail.

Verify both directions

Check	Result
Parallel plan, before the fix	0 of 3 passed
Parallel plan, after the fix	5 of 5 passed
Serial plan, whole target	690 tests passed
Mutant: fetcher limited to one request at a time	all 5 concurrency tests failed, each at 5.0 s

The last row matters most. A longer timeout would also have made the parallel runs pass. What it wouldn’t keep is fast failure for a real bug. I changed the fetcher’s concurrency limit from 4 to 1, so the “all three requests are open at once” condition could never be true. All five tests caught it, each at the 5-second time budget, and I reverted the mutation.

One caveat: that mutant ran under the serial plan. Under heavy parallel load, a real hang fails after 100 turns, which takes longer than 5 seconds. That’s the trade-off I chose, and it applies to this helper and this workload. It isn’t a general fairness guarantee for Task.yield().

A second guess hiding in the same tests

Three tests waited for the first request and then gave the scheduler “five settling turns” before asserting that all three were open. That turn count was another guess, and it could fail under load the same way. The fix was to wait for the exact condition:

// before
try await Wait.until { service.pendingCount > 0 }
await Wait.untilSettled()
#expect(service.pendingCount == 3)

// after
try await Wait.until { service.pendingCount == 3 }

Takeaway

A wall-clock timeout in a polling helper measures how busy the process is, not whether your code is stuck. When a wait times out only in the big parallel run, count its turns before you suspect a deadlock. If it got only a handful of turns, the actor was starved. Make the helper fail only after both a time budget and a turn budget run out. Then check the fix with a mutation that should hang and make sure it still fails quickly.

Try it on your own polling helper: add a turn counter and run the full target in parallel once. If the worst wait used single-digit turns but took seconds, the timeout is measuring main-actor contention.

Snapshot precision 0.99 let a wrong text line pass all seven screen tests

Tue, 13 Oct 2026 17:04:00 +0200

A code review claimed that a list screen’s snapshot tests couldn’t fail on the very line they were written to pin: a small gray “Location” line under each search result. To check, I broke that line on purpose. The rows got the container’s short name instead of its full name, so every location line rendered the wrong text. Then I ran the suite:

✔ Test listScreenDeletedItemsShowTheirOrigin() passed after 0.296 seconds.
✔ Test listScreenSearchResultsShowTheirLocation() passed after 0.708 seconds.
...
✔ Test run with 7 tests in 1 suite passed after 0.793 seconds.

All seven passed against the broken code. If your snapshot tests record a whole screen at .device layout with a precision below 1.0, they may be just as blind.

Setup

The project used swift-snapshot-testing 1.19.2, with Swift Testing on the iOS simulator. A shared defaults file sets both tolerances:

public static let perceptualPrecision: Float = 0.90
public static let precision: Float = 0.99

The two gates do different jobs:

perceptualPrecision decides whether a single pixel counts as different. 0.90 absorbs anti-aliasing drift between developer Macs.
precision decides what fraction of pixels must match. 0.99 lets 1% of the image differ by any amount.

The 0.99 was there to unblock CI: one render drifted by about 3,600 pixels between machines. The comment next to it even warned that it was “wide enough to hide a small real regression”. Every screen test used it anyway.

The arithmetic

An iPhone 17 Pro device frame is 1179×2556 pixels.

1179 × 2556 = 3,013,524 px
1% of that = 30,135 px may differ freely
wrong location = ~900 px actually differed

A 1% budget sounds strict. On a full screen it’s about a 58-point square of arbitrary change, more than all the location lines on the screen combined. The test was recorded correctly and passed honestly. It just had no way to fail on the thing in its name.

Measuring the mutation

To size the gate, record the mutated renders, diff them against the committed references, then restore the references. A few lines of Pillow give a rough count (a pixel counts if any channel moves by more than 25):

import sys
from PIL import Image

a = Image.open(sys.argv[1]).convert("RGB")
b = Image.open(sys.argv[2]).convert("RGB")
pa, pb = a.load(), b.load()
w, h = a.size
n = sum(1 for y in range(h) for x in range(w)
 if max(abs(c1 - c2) for c1, c2 in zip(pa[x, y], pb[x, y])) > 25)
print(f"{n / (w * h):.2%} of {w}x{h}")

TEST_RUNNER_SNAPSHOT_TESTING_RECORD=all xcodebuild test ... -only-testing:MyTests/ListScreenSnapshotTests
git show HEAD:path/to/reference.png > /tmp/head.png
python3 pxdiff.py /tmp/head.png path/to/reference.png
git checkout -- path/to/__Snapshots__ # put the real references back

...DarkAppearance...: 0.03% of 1179x2556
...SearchResultsAtTheDynamicTypeCap...: 0.14% of 1179x2556
...SearchResultsShowTheirLocation...: 0.03% of 1179x2556
...DeletedItemsShowTheirOrigin...: 0.03% of 1179x2556

The smallest real defect was 0.03%, about 900 pixels, so the budget had to sit well below that. 0.9999 allows 301.

The fix

Give the screen suite its own precision and write the numbers down where the next person will see them:

as: .image(
 precision: Constants.precision,
 perceptualPrecision: SnapshotDefaults.perceptualPrecision,
 layout: .device(config: config)
)

private enum Constants {
 /// A device frame is 1179×2556 px, so the default 1% budget is 30,135 px: wider than every
 /// Location line on the screen together. The wrong name moves about 900 px; this leaves a
 /// 301 px budget, while `perceptualPrecision` still absorbs anti-aliasing drift per pixel.
 static let precision: Float = 0.9999
}

Then rerun with the mutation still in place. It has to go red:

✘ Test listScreenDeletedItemsShowTheirOrigin() failed after 0.594 seconds with 1 issue.
✔ Test listScreenSortMenuOpen() passed after 0.751 seconds.
✔ Test listScreenBrowsingTheRoot() passed after 0.807 seconds.
✘ Test listScreenSearchResultsAtTheDynamicTypeCap() failed after 1.220 seconds with 1 issue.
✘ Test listScreenSearchResultsShowTheirLocation() failed after 1.618 seconds with 1 issue.
✔ Test listScreenSearchWithNoResults() passed after 1.659 seconds.
✘ Test listScreenDarkAppearance() failed after 2.036 seconds with 1 issue.

The four cases that draw location or origin lines failed. The three cases without them passed, which is also correct. After reverting the mutation, all seven passed again at 0.9999.

A 301-pixel budget is tight, and I verified it on one machine. If cross-machine drift trips it, look at the drift before you loosen the number.

Shrink the frame when you can

For a single component, a tight frame beats a tight precision. The same project measured a toolbar button’s enabled-to-disabled color change at 1,808 pixels, or 0.06% of a device frame. Rendered at 140×60 with layout: .fixed(width:height:), those 1,808 pixels became 2.4% of the image, well past a 1% budget. Keep .device for tests that are really about the whole screen, and size their precision from the smallest change they claim to catch.

Takeaway

precision is a fraction of the image, so a fixed value forgives more pixels as the frame grows. Before you trust a green snapshot, break the thing it’s named after, count the changed pixels, and compare that with (1 - precision) × width × height.

Pick one snapshot test in your suite, break its subject on purpose, and see whether it goes red.

Inherited agent notes can send the next agent down the same dead end

Fri, 09 Oct 2026 00:00:00 +0000

A new agent can go through a failed investigation again if its first job is to absorb the conclusions of the previous agent. In the course of an authorised security exercise the user was repeatedly asking for a clean start as the prior progress files could be somewhat misleading. The useful reset was to establish a small verified baseline before using the inherited notes to select the next experiment.

A hint survived longer than its evidence

The conversation history had instructions to go it alone after a number of failed attempts at the task. Another handoff stated the prior agent had got lost and that its notes could be misleading to the replacement.

One concrete example concerned an environment hint favoring IPv6. A later SUMMARY.md excerpt said:

IPv4 and IPv6 behave identically everywhere tested

It was very explicit that the IPv6 hint in AGENTS.md didn’t apply in that case. That’s a useful clarification, with a caveat: this is a recorded summary of the checks, not an independent re-run of every network probe. It does support the removal of that hint for that particular surface. It doesn’t in any way establish equivalence for all services or a successful resolution of the problem. The outcome is still unknown.

Rebuild only enough context to choose a test

On a restart read the current task and the mandatory operating instructions. Observe the authorization boundaries and safety rules. Be very clear in your mind about the difference between the instructions and the factual observations about the environment. A line in AGENTS.md could be a binding rule, it could be a fact about the setup or it could be a hypothesis that you need to go out and check.

Then establish the facts required to inform the next decision. In a network investigation that might involve confirming the target endpoint, checking that it’s reachable and then making a baseline request to it. In a code issue scenario you’d want to reproduce the failure that was reported and then have a look at the current code before accepting any diagnosis from a previous developer.

Only then do you compare the inherited notes against that baseline. That way you don’t re-run the whole investigation and the new agent has its own baseline to work from.

A handoff instruction can be as short as:

Establish a minimal baseline from the current tool output.
Treat inherited diagnoses as hypotheses.
For each claim you make, you need to be able to locate the evidence or at the very least be able to run a very cheap check.
Keep the required operating and safety instructions in force.

Preserve the test behind a negative result

A note such as “IPv6 works better” is completely devoid of the endpoint, the request and the comparison that would actually make it useful. “No difference on these tested endpoints” is much more specific and much easier to verify.

For future handoffs, record:

The claim and its scope (including when it was checked).
The command or test and a pointer to its output.
What remains is an interpretation or an unproven lead.

Keep the failed attempts too and state what each one ruled out. A failed request with one set of credentials doesn’t rule out the whole protocol. The next agent needs to be able to make the judgement as to whether repeating that test would in fact add any evidence.

Copying an assertion into successive summaries doesn’t constitute independent corroboration. If the raw output is missing then unverified is the correct designation rather than promoting it to a fact.

Prior to your next agent handoff, pick the assumption that will drive the next action and attach the relevant tool output in support of that assumption. If you can’t find it then that will be the restart task.

A format string in a TRACE logger can read the process environment

Tue, 06 Oct 2026 00:00:00 +0000

A TRACE handler that echoes a request with snprintf(response, size, request_line) does more than log attacker-controlled text. It turns the request line into a printf format string and can leak stack data in an authorized test, including pointers into argv and envp, through positional %p and %s conversions. The one-line fix is snprintf(response, size, "%s", request_line).

This is CWE-134, use of an externally controlled format string . The bug often hides in diagnostics because developers treat the third argument as plain text. For the printf family, it is the format string: a small program that says what to read and, with %n, where to write.

A bounded copy does not make the format safe

The length argument to snprintf bounds the output buffer. It does not stop format directives from being evaluated:

/* Vulnerable: request_line controls the format language. */
int len = snprintf(response, sizeof response, request_line);

/* Fixed: request_line is data for a constant format. */
int len = snprintf(response, sizeof response, "%s", request_line);

The same rule applies to printf(input), fprintf(log, input), and wrappers around them. When no formatting is needed, prefer fputs.

A small local regression fixture makes the difference visible without reading sensitive memory. Save this as format-log.c in a disposable directory:

#include <stdio.h>
#include <string.h>

static void render(char *out, size_t size, const char *input, int fixed) {
 if (fixed)
 snprintf(out, size, "%s", input);
 else
 snprintf(out, size, input);
}

int main(void) {
 char vulnerable[128], fixed[128];
 const char *probe = "TRACE /%p HTTP/1.1";
 render(vulnerable, sizeof vulnerable, probe, 0);
 render(fixed, sizeof fixed, probe, 1);
 printf("vulnerable=%s\nfixed=%s\n", vulnerable, fixed);
 return strcmp(fixed, probe) != 0 || strcmp(vulnerable, probe) == 0;
}

Run it locally:

cc -O0 -Wall -Wextra format-log.c -o format-log
./format-log

The vulnerable line expands %p; the fixed line preserves it literally. The pointer value is intentionally unspecified.

How to exercise the disclosure safely

On an authorized disposable target, start with a normal request and a literal-percent control. Then use %p or positional %N$p to map candidate argument slots. Dereferencing a candidate with %N$s asks the process to treat that value as a string pointer, so expect child crashes and test one slot per fresh connection. Never run a broad %s sweep against production.

Environment strings commonly sit near the initial argument and environment vectors. A pointer leak can therefore become a secret leak when a service receives credentials through environment variables. Slot numbers are build- and runtime-specific; a number from one binary is not a portable exploit.

Fork-per-connection servers make repeated probing easier than expected. Children inherit the parent’s already-randomized mappings so that addresses can remain stable across child connections until the parent restarts. ASLR still applies to the parent; fork() simply copies that layout.

Treat %n as a separate, higher-risk write primitive. A read-only demonstration is enough to prove the logging flaw and should be the limit of most authorized validation.

Fix the sink and reduce the impact

Make every format string a constant and enable compiler diagnostics such as -Wformat=2 -Wformat-security. Add a regression request containing %p, %s, and %n as literal text, then assert that the logger preserves every byte. Static analysis should flag tainted data reaching any printf-family format parameter.

Do not rely on input filtering. Blocking a few percent directives is easy to get wrong and leaves other format sinks exposed. Fix the call boundary.

Finally, keep long-lived credentials out of network-facing process environments where possible. Use narrowly scoped, short-lived credentials and isolate the component. That limits the damage from any memory disclosure, but it does not replace fixing the format string.

Search your diagnostic and TRACE paths for nonliteral printf formats before assuming they are harmless logging code.

Next.js Server Action arguments are Flight data: use error digests carefully

Tue, 06 Oct 2026 00:00:00 +0000

A Next.js Server Action can receive a JSON-looking body whose strings are interpreted as React Flight tokens. If ordinary input works but $-prefixed input changes the response or produces HTTP 500, inspect the decoded arguments before trying more SQL or shell payloads. Differences between error digests can help separate failure paths, but they cannot establish a vulnerable version or remote code execution on their own.

This matters when testing the request behind an action call. The normal client serializer escapes literal strings for the protocol; manually editing the wire body can change what the server decodes.

The application sees the result of decodeReply

In the tagged Next.js 15.0.0 action handler , fetch-action bodies pass through decodeReply or its streaming counterpart before the selected action runs. The React 19.0.0 reply decoder interprets $-prefixed strings, including escaped literals, special numbers, and references.

For a local decoder, these argument values illustrate the distinction:

Wire string inside the arguments array	Decoded argument
`ordinary`	string `ordinary`
`$$literal`	string `$literal`
`$Infinity`	number `Infinity`
`$-Infinity`	number `-Infinity`
`$undefined`	`undefined`

A value changing type before the application function runs explains why an otherwise boring search action can behave strangely. Input validation inside that function does not protect an earlier, vulnerable deserializer.

Reproduce the protocol locally before probing further

Use an isolated scratch directory. The following intentionally installs an old, vulnerable package for an offline decoder experiment. Do not deploy it, expose a listener, or feed it untrusted traffic. Delete the directory afterward.

LAB=$(mktemp -d "${TMPDIR:?Set TMPDIR to your scratch directory}/flight-lab.XXXXXX")
cd "$LAB"
npm install --ignore-scripts --no-audit --no-fund --save-exact \
 react@19.0.0 react-dom@19.0.0 react-server-dom-webpack@19.0.0

Save this as check.cjs in that directory, then run node --conditions=react-server check.cjs:

const assert = require('node:assert/strict');
const { decodeReply } = require('react-server-dom-webpack/server.node');

async function check() {
 const cases = [
 ['ordinary', 'ordinary'],
 ['$$literal', '$literal'],
 ['$Infinity', Infinity],
 ['$-Infinity', -Infinity],
 ['$undefined', undefined],
 ];
 for (const [wire, expected] of cases) {
 const args = await decodeReply(JSON.stringify([wire]), {});
 assert.ok(Array.isArray(args));
 assert.equal(args[0], expected);
 }
 console.log('5 Flight decoding cases passed');
}
check().catch(error => {
 console.error(error);
 process.exitCode = 1;
});

This checks protocol behavior, not exploitability. For an authorized application assessment, reproduce the actual resolved packages and runtime conditions. Next.js may bundle its React implementation; a standalone npm package is not automatically an exact framework replica.

Preserve the decoded root’s expected shape, too. The tagged Next.js handler invokes actionHandler.apply(null, boundActionArguments). A root that resolves to a primitive string fails that call’s array-like argument requirement before the intended action behavior can be tested. Keep transport decoding and action invocation as separate checkpoints.

What an error digest can tell you

The Next.js 15.0.0 error handler generates digests using stringHash over error text and stack information in several paths, unless a digest already exists. One path uses:

err.digest = stringHash(err.message + (err.stack || '')).toString()

Different parser branches can throw from different stack locations and yield different digests even when the visible error message is generic. Record the complete response, status, and digest for each controlled input; repeat baseline requests to check stability. Correlate differences with logs in the local replica.

A digest is a lossy fingerprint. Build paths, package versions, runtime stacks, and existing application digests can alter it. Distinct digests suggest distinct error material; matching digests do not prove identical execution paths. HTTP 500 alone does not even locate the failure in Flight decoding.

Separate protocol reachability from React2Shell

React’s CVE-2025-55182 advisory confirms unauthenticated RCE through a flaw in decoding Server Function payloads. Ordinary Flight tokens also exist in correct protocol handling. Observing the transformations above therefore does not diagnose that CVE.

For exploit validation, first reproduce the affected decoder in a private replica and demonstrate a harmless, deterministic effect there. Keep that result separate from any authorized remote confirmation. The experiment above deliberately stops at benign decoding.

For prevention, follow the current React and framework advisories, update the resolved server-side packages, rebuild, and verify what the running artifact actually contains. Do not use the historical demonstration versions as a patch recommendation. Authenticate and authorize actions independently, validate decoded arguments, and avoid exposing decoded-input debug echoes. Those controls do not replace fixing a vulnerable decoder.

When a $ probe changes behavior, capture the full response and reproduce that transformation locally before assigning a CVE or an impact rating.

Prompt injection: extract tokens, then decode them offline

Tue, 06 Oct 2026 00:00:00 +0000

A model can repeat hidden, encoded prompt fragments and still produce a different decoded secret on each attempt. When an authorized prompt-injection test reaches that point, stop asking the model to solve its own leak. Capture the raw tokens with position and case intact, then decode them with deterministic code offline.

This separates two questions: can the model disclose protected context, and can a decoder reconstruct the encoded value? A fluent final answer is poor evidence for either one.

Lexical refusal filters leave a semantic gap

A filter that rejects phrases such as “show the system prompt” or “reveal the secret” checks wording, not intent. Semantically equivalent requests can ask the model to quote, classify, count, or list text from its instructions without using the blocked nouns. If the model can return those fragments, the control has failed even when the obvious request is refused.

For an authorized assessment, use a synthetic canary in a test system. Ask for a numbered transcription of the relevant tokens, one token per line, with original capitalization. Repeat individual positions and compare exact strings. Keep the transcript as evidence. Do not put a real credential in the prompt merely to test whether it leaks.

Do not ask the model to join, normalize, or decode the result. Those extra transformations invite changed case, missing groups, repaired spelling, and invented separators. Temperature zero may reduce variation, but it does not turn generation into a parser.

Decode a synthetic transcript deterministically

Suppose a test prompt encodes characters as phonetic words. Capitalized words mean uppercase letters, lowercase words mean lowercase letters, and number words mean digits. Save this offline decoder as decode.py:

NUMBERS = {
 "zero": "0", "one": "1", "two": "2", "three": "3",
 "four": "4", "five": "5", "six": "6", "seven": "7",
 "eight": "8", "nine": "9",
}

def decode(tokens):
 result = []
 for token in tokens:
 lower = token.lower()
 if lower in NUMBERS:
 result.append(NUMBERS[lower])
 elif token.isalpha():
 letter = token[0]
 result.append(letter.upper() if token[0].isupper() else letter.lower())
 else:
 raise ValueError(f"unsupported token: {token!r}")
 return "".join(result)

sample = ["Alpha", "bravo", "Charlie", "delta", "seven"]
assert decode(sample) == "AbCd7"
print(decode(sample))

Run python3 decode.py; it prints AbCd7. The assertion is a negative control against accidental lowercasing or regrouping. Adapt the decoder only from a documented encoding rule. If a separator or group length is unknown, report that ambiguity instead of asking the model to guess.

Keep an evidence table with the position, raw token, repeated observation, decoded character, and rule used. Hash or otherwise preserve the raw transcript if chain of custody matters. A model-generated summary is not a substitute for the original response.

Prevent prompt disclosure at the trust boundary

Keyword filters are a weak secondary control. The primary fix is to keep secrets and privileged instructions out of model-visible context. Give tools short-lived, scoped credentials through an execution boundary that does not echo them into prompts or tool results. Treat retrieved documents, web pages, and user content as untrusted data, not instructions.

Constrain outputs structurally when the application expects a fixed schema, and validate them outside the model: test semantic paraphrases, indirect requests, translations, and token-by-token extraction against synthetic canaries. Log enough to identify which context source leaked without recording the secret itself.

A successful refusal to one phrase proves only that one phrase was blocked. Verify that the model cannot disclose the underlying text in pieces.

Zero failures is suspicious: agent checks that silently ran nothing

Tue, 29 Sep 2026 12:05:00 +0200

My coding agent ran each of the two flaky Rust tests 150 times and got this:

streamed_read_right_after_a_rename: 0/150 failed
renames_are_combined_across_changes_they_do_not_depend_on: 0/150 failed

Its next message was “Zero failures is suspicious.” It was right: neither test had run once. In the same week, three of its checks gave a tidy answer (nothing found, all passed, all failed) that came from a broken command, and each broken command hid its own error.

If your agent’s output has (eval):log:1: too many arguments, 0 passed; 0 failed; … filtered out or argument list too long somewhere above the summary, the summary is about a check that never ran.

The setup

Claude Code 2.1.283 on macOS, where the agent’s shell is zsh 5.9
A Rust workspace whose test binary is run directly from target/debug/deps

Trap 1: zsh has a `log` builtin

The agent was debugging an FSKit file system extension and searched the unified log for sandbox denials:

log show --last 30m --style compact --predicate 'process == "MyFS"' 2>/dev/null | tail -40

Nothing came back, and the agent told me: “No log entries so far.” Half a minute later it ran log stream … > /tmp/fs.log 2>&1 & and read the file. Its first line was:

(eval):log:1: too many arguments

zsh has a builtin called log that lists the logins watch reports. Builtins win over $PATH, so log show never reached /usr/bin/log, and 2>/dev/null threw away the only sign of it. You can check your own shell:

$ zsh -c 'whence -wa log'
log: builtin
log: command

With /usr/bin/log, the denial showed up at once, and the agent corrected its earlier claim. The same error turned up in a different project’s session days later.

Trap 2: `cargo test --exact` with the wrong path passes

The 150-run loop called the test binary like this:

$B --exact tests::$t -q > /tmp/out_$t.txt 2>&1 || f=$((f+1))

With --exact, the filter has to equal the full test path. The tests live in vfs::tests, so tests::streamed_read_right_after_a_rename matched nothing, and libtest treats that as success:

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 93 filtered out; finished in 0.00s

Exit code 0, every time. Without --exact, the same filter would have matched as a substring. The agent listed the real names with $B --list, fixed the prefix, and the races then showed up under load.

Trap 3: an empty variable makes everything fail

After the fixes, the agent stress-ran the suite 400 times and looked up the binary first:

B=$(ls -t target/debug/deps/storage_core-* | grep -v '\.d$' | head -1)

The glob had grown past what ls accepts:

(eval):1: argument list too long: ls

failed runs: 400 / 400

$B was empty. zsh drops an empty unquoted word, so each run executed vfs::tests -q, got command not found into a log file nobody read, and counted as a failure. This time the agent saw the error right above the count and called the result bogus.

Fix

Make every check prove that it ran before you trust what it says.

Call /usr/bin/log by its full path, or put command in front of it, and keep stderr visible while you are still debugging:

/usr/bin/log show --last 30m --style compact --predicate 'process == "MyFS"' | tail -40

Count the tests that ran, not only the failures. One test per run should say 1 passed:

$B --exact vfs::tests::$t -q > /tmp/out_$t.txt 2>&1 || f=$((f+1))
grep -q '1 passed' /tmp/out_$t.txt || { echo "$t did not run"; exit 1; }

Get the binary path from cargo instead of a glob, and stop if it is empty:

B=$(cargo test -p storage-core --lib --no-run 2>&1 | grep -o 'target/debug/deps/storage_core-[0-9a-f]*')
: ${B:?test binary not found}

For the agent, add one line to its instructions file (CLAUDE.md or AGENTS.md):

When a check finds nothing, passes everything, or fails everything, first show that it ran:
how many items it checked, and that a known-bad input fails it.

Why it happens

All three are the same bug: the harness failed, and the failure looked like a result. A loop that counts exit codes can’t tell “no failures” from “no tests”, and a pipeline that ends in 2>/dev/null | tail can’t tell “no matches” from “no command”. Agents write exactly these one-liners, in zsh, many times per session, and they report what the last line printed.

The agent caught two of the three because the number looked wrong: zero out of 150 for known flaky tests, 400 out of 400 for a suite that had just passed. An empty log search looked plausible, so it went to the user as a finding.

Takeaway

A clean result from a check you haven’t seen fail is not evidence. Before you believe “nothing found” or “0 failed”, look for the count of what was checked.

Next time your agent reports zero failures, ask it how many tests actually ran.

xcodebuild says TEST SUCCEEDED, the xcresult says "Issues recorded without an associated test or suite"

Mon, 28 Sep 2026 12:16:00 +0200

A branch passed locally. xcodebuild test exited 0 and printed:

** TEST SUCCEEDED **

The fastlane coverage lane on the same branch failed with Tests have failed and didn’t name a test. Scan’s xcpretty output doesn’t show Swift Testing results, so the log didn’t include any failing tests. The result bundle had the answer:

xcrun xcresulttool get test-results summary --path Test.xcresult

{'result': 'Failed', 'totalTestCount': 2320, 'passedTests': 2317, 'failedTests': 1, ...}
testName: Issues recorded without an associated test or suite
failureText: Issue recorded: failed - Unimplemented: Unimplemented AppVersionClient.currentVersion …
 This issue was emitted from tests running in a host application.
 This can lead to false positives, where failures could have emitted from live application code at launch time, and not from the current test.

If your console is green and fastlane, Xcode Cloud or anything else that reads the .xcresult is red, check the summary for that pseudo-test before you go looking for a flaky test.

Setup

Xcode 26, Swift Testing, an app-hosted unit test target (tests run inside the real app process).
swift-dependencies 1.14, with a client whose testValue calls IssueReporting.unimplemented(...). That’s the usual way to make sure a test sets every dependency it uses.
The app’s root view is a login screen. On appear, it asks a backend for the minimum supported version and compares it with currentVersion().

What happened

Because the app hosts the tests, it launches as it does for a user, and its login screen sends the version request. When the response arrives, the screen reads the version client inside a test process, which resolves to the unimplemented test value and records an issue. No test is running on that code path, so the issue gets attached to no test.

The console run only shows it as a warning:

⚠︎ Test «unknown» recorded a warning at VersionClient.swift:24:1: Issue recorded
↳ failed - Unimplemented: Unimplemented AppVersionClient.currentVersion …

The result bundle counts it as a failure, though, and fastlane reads it.

Main didn’t fail, and the new branch didn’t touch any of this code. Its only addition was a new app-hosted suite that took about seven seconds. That extra time let the response arrive while tests were still running. On main, the run was over before the response came back. This is timing-dependent, so a faster backend or a slower one can hide the failure or bring it back.

Reproduce one target and compare the result bundle summaries:

Run	Total	Failed
Hosted target, as is	21	1
Same, new suite skipped (`-skip-testing`)	16	0
Same, with the fix	20	0

The 21 in the first row is 20 real tests plus the pseudo-test.

Dead end: set the dependency at launch

The obvious fix is to give the app a live value in its entry point:

prepareDependencies {
 $0.appVersion = AppVersionClient { Bundle.main.shortVersion }
}

It built and changed nothing. The same failure showed up again. In a test process, swift-dependencies resets its cached values for each test: it registers an XCTest observer that clears the cache whenever a test case starts. The value you prepare at app launch is gone once the tests start, and the later response reads the test value again.

Fix: don’t show the app UI while hosting tests

isTesting from IssueReporting (re-exported by Dependencies) is true inside a test process. Gate the scene’s content on it:

import Dependencies
import SwiftUI

@main
struct MyApp: App {
 var body: some Scene {
 WindowGroup {
 // Unit tests run inside this app. Its login screen would start a
 // network request whose response reads an unimplemented test dependency.
 if !isTesting {
 RootView()
 }
 }
 }
}

This is the workaround in the library’s Testing gotchas docs. Before applying it, check that no hosted test needs the app’s window or a view in the scene. Here, none did, because the tests only needed a real UIApplication, which still exists with an empty WindowGroup. The shipping app is unchanged, since isTesting is false outside a test run.

With the fix, the hosted target passed 20 of 20, the whole coverage lane reported 0 failures on both schemes, and the test run stopped sending requests to the dev backend.

That last part is a nice bonus. A test host that shows its real UI talks to real servers from your CI, and anything it logs (request and response bodies, anti-forgery tokens) ends up in your test logs.

Takeaway

An app-hosted test run also runs your app. Anything the app starts on its own at launch can record a failure that belongs to no test and shows up only in the .xcresult. Keep the UI out of the test host with if !isTesting, and when the console and the lane disagree, read xcresulttool get test-results summary first.

Try it on your own app-hosted target: grep a recent test log for Test «unknown» recorded. Each hit is an issue recorded outside any test, and the result bundle may already count it as a failure.

QtInfoMsg is numerically above QtFatalMsg, so an INFO filter dropped every warning

Wed, 23 Sep 2026 16:00:00 +0200

A desktop app’s log file had about 20,000 lines in it, all of them INFO. There wasn’t a single WARN or ERROR line, even though the app had plenty of qWarning() and qCritical() calls. This was the filter in the custom message handler (trimmed):

void Logger::logMessage(QtMsgType msgType, const QMessageLogContext &context, const QString &message)
{
 if (msgType < minimumType && minimumType != QtFatalMsg && !message.startsWith("##")) {
 return; // below the configured level
 }
 ...
}

It assumes QtMsgType is ordered by severity. It isn’t:

enum QtMsgType {
 QtDebugMsg, // 0
 QtWarningMsg, // 1
 QtCriticalMsg, // 2
 QtFatalMsg, // 3
 QtInfoMsg, // 4, added in Qt 5.5
 QtSystemMsg = QtCriticalMsg
};

At INFO, minimumType is QtInfoMsg (4). Warning (1) and critical (2) are both below 4, so they were thrown away. Info lines got through, so the log looked healthy.

If your Qt message handler compares QtMsgType values with < or >, it has this bug.

How it got there

No single commit introduced this:

Before Qt 5.5 the enum was in severity order, so msgType < minimumType was correct when the handler was written.
Qt 5.5 added QtInfoMsg at the end, to keep the existing values stable. The app also added INFO to the end of its own level enum, and nobody looked at the comparison again.
Years later, a one-line change moved the fallback level from WARN to INFO. At WARN, the bug had only cost info lines. At INFO it cost warnings and errors, and since INFO was now the default, that became the normal behavior.

Each step looked harmless in its own diff. The cause only showed up when the history was read in order.

Red tests first

Each test sets a level, sends one probe per message type through the real handler to a real log file, and records which probes ended up in the file:

// One letter per probe that reached the file: d(ebug), i(nfo), w(arning), c(ritical)
QString writtenAt(Logger::LogLevel level) {
 log->setLogLevel(level);
 qDebug("probe d");
 qInfo("probe i");
 qWarning("probe w");
 qCritical("probe c");
 const QString content { logContent() };
 QString written;
 for (const QChar probe : QStringLiteral("diwc")) {
 if (content.contains(QStringLiteral("probe ") + probe)) written += probe;
 }
 return written;
}

TEST_F(LoggerTest, logMessage_InfoLevel_WarningWritten) {
 ASSERT_EQ(writtenAt(Logger::LogLevel::INFO), "iwc");
}

Against the old handler, five of the twelve logger tests failed:

Level	Expected	Written
INFO	`iwc`	`i`
WARN	`wc`	`iwc`
ERROR	`c`	`ic`
FATAL	(none)	`diwc`

The two bottom rows are separate bugs. Info (4) is above every other type, so it got through at WARN and ERROR. The minimumType != QtFatalMsg exemption meant FATAL wrote everything.

The fifth failure was a different bug: a level saved as ERROR came back as INFO after a restart. The app’s enum key was Error (an all-caps ERROR collides with a Windows macro), the setting was stored as ERROR, and QMetaEnum::keyToValue is case-sensitive. The lookup failed, so it fell back to the default.

Fix

Map the message types to severity explicitly, and compare those numbers instead:

// Qt numbers its message types in the order they were added, info last, not by severity
int severity(QtMsgType msgType) {
 switch (msgType) {
 case QtDebugMsg: return 0;
 case QtInfoMsg: return 1;
 case QtWarningMsg: return 2;
 case QtCriticalMsg: return 3;
 case QtFatalMsg: return 4;
 }
 return 4;
}

if (severity(msgType) < severity(minimumType) && !message.startsWith("##")) {
 return;
}

The FATAL exemption is gone. Each level now means “this severity and above.” If a future Qt version adds a type, the switch with no default makes the compiler warn about it, and the fallback treats the unknown type as most severe, so it gets logged rather than dropped.

The stored level is now matched without regard to case:

for (int i = 0; i < metaEnum.keyCount(); ++i) {
 if (stored.compare(metaEnum.key(i), Qt::CaseInsensitive) == 0) return LogLevel(metaEnum.value(i));
}
return LogLevel::INFO;

The same twelve tests passed after the fix, and so did the full suite.

The rollout catch

Fixing a filter that has silently dropped messages turns on log statements that nobody has read in years. Here, some of those warnings included user file names. The fix couldn’t ship until we moved those down to DEBUG. Before you fix a filter like this, grep for every qWarning and qCritical call it was dropping, because those are the lines your users will start sending you.

Takeaway

QtMsgType values are not ordered by severity. Never compare them with < or >. Map them first.
A log full of INFO lines with no warnings in it is suspicious, not reassuring.
Test the filter through the real handler with one probe per type, one test per level. The four-letter strings make the whole matrix fit on one screen.
Check enum-to-string round trips (QMetaEnum::keyToValue) with an actual save and reload, not just a set and get.

An UPDATE on encrypted columns wrote nothing for years, and three bugs took turns hiding it

Wed, 23 Sep 2026 15:42:00 +0200

A desktop app maintained an activity history in SQLite, but the UI froze on the first entry into any state. An action could start as ‘in progress’ and later complete, yet the UI would keep showing ‘in progress’ forever.

The update function looked fine at first glance, but nobody had actually run it under test, and it had 0% test coverage. That made the bug easy to miss and hard to trust.

The setup

The app encrypts the path columns as it writes them (column level encryption, not the whole database as with SQLCipher; see why fields rather than the file ). With a random IV per row, you can’t query the encrypted columns at all. To make rows matchable, derive the IV from the plaintext so the same path always encrypts to the same ciphertext. You store the IV alongside the row, base64 encoded:

history(id, run_id, path_enc, type, state, timestamp, iv)

You can’t put a plaintext path in a WHERE clause against path_enc. So the update does three steps:

Check the row’s IV.
Re-encrypt the current path with that IV, then SELECT to check whether the row still matches by id, run, path, and type.
Run an UPDATE with the same match predicate.

Reproduce with the real class

One test that stores, updates, and reloads through the real DAO, with an in-memory schema and a dummy key:

TEST_F(HistoryDaoTest, update_StoredUpload_LoadShowsNewState) {
 HistoryDao dao(&_database);
 Action upload { action(Action::Upload, Action::IN_PROGRESS) };
 upload.rowId = dao.store(&upload, 5);
 upload.status = Action::OK;

 dao.update(&upload, 5);

 EXPECT_EQ(load().value(0).state(), Action::OK);
}

It failed. The reloaded state was still IN_PROGRESS, which was precisely where the problem was.

Three faults, each hiding the next

QByteArray iv = queryIv.value(0).toByteArray(); // 1: still base64
...
query.prepare("SELECT id FROM history" // 2: no space before WHERE
 "WHERE id = :id AND path_enc = :path ...");
bool ok = database->exec(&queryIv); // 2: runs the IV query again
...
QSqlQuery update_query(db);
update_query.prepare("UPDATE history SET state = :state ... WHERE <same predicate>");
...
database->exec(&query); // 3: runs the SELECT, not the UPDATE
database->commit();
return true;

The IV was never decoded. store() decodes the base64 IV before encryption, but update() passed the IV through still encoded. The encrypt helper rejected an IV of the wrong length and threw, so every call stopped there. Before the length check was introduced, a wrong IV would have produced a ciphertext that was impossible to match. In either case, nothing was ever written. The bug had been there since the commit that introduced encryption, because the two sibling functions in that commit disagreed.
The guard SELECT never ran. A later refactor concatenated two string literals with no space between them (historyWHERE), so prepare() failed. The failure was only logged. Then exec() ran on the IV query, which succeeded. A follow-up “fix typo in query” commit fixed half of the SQL and left the missing space.
The UPDATE never ran. A rename to update_query missed the exec() call, which kept executing query instead.

Nothing ever got past fault 1, so faults 2 and 3 were never executed in production. Read the code as a chain of what would have happened if each fault had been fixed one at a time: fix the IV, and the function returns true while writing nothing; fix the SELECT as well, and fault 3 reruns a successful SELECT, commits, and returns true. Each fix would look like progress, and the result would be a function that reports success and writes nothing.

Date the bug before you blame the refactor

The first suspect was the 2022 refactor that added the second SELECT. Running git log -S on the decode call and reading store() and update() side by side in the encryption commit moved the start date back to 2019. That matters for the fix: the refactor wasn’t the thing to revert, and the original rule from before encryption was still the spec.

The tempting shortcut

The obvious simplification is to drop the IV lookup and the re-encryption and run UPDATE … WHERE id = :id. That passes the test above.

It also deletes the rule. The version from before encryption updated a row only while its id, run, paths, and type all still matched. Every later commit kept that predicate. Updating by id alone means a stale action can overwrite whatever row now sits behind that id. So the fix was one small change per fault, keeping the predicate:

QByteArray iv = QByteArray::fromBase64(queryIv.value(0).toByteArray(),
 QByteArray::Base64UrlEncoding);
query.prepare("SELECT id FROM history "
 "WHERE id = :id AND path_enc = :path ...");
bool ok = database->exec(&query);
...
database->exec(&update_query);

A second test pins the rule:

TEST_F(HistoryDaoTest, update_OtherPath_LeavesTheRowAlone) {
 HistoryDao dao(&_database);
 Action upload { action(Action::Upload, Action::IN_PROGRESS) };
 upload.rowId = dao.store(&upload, 5);
 upload.name = "other.jpg"; // same id, different path
 upload.status = Action::OK;

 dao.update(&upload, 5);

 EXPECT_EQ(load().value(0).state(), Action::IN_PROGRESS);
}

Before the fix, this test passed too, trivially, because nothing was ever updated. A negative test proves something only once its positive twin is green. After the fix, both tests passed, and so did the full suite (787 tests).

Side notes

QSqlQuery::size() returns -1 on SQLite, because the driver doesn’t report query sizes. A if (query.size() > 1) guard never fires there. Count with while (query.next()), or use SELECT COUNT(*).
Deriving the IV from the plaintext makes the encryption deterministic: equal paths give equal ciphertexts, which is what makes matching possible. With AES-GCM, never reuse that IV for a different plaintext. If you need equality lookups on encrypted columns, consider a misuse-resistant mode (AES-SIV) or a separate blind-index column (an HMAC of the plaintext) instead.

Takeaway

Test a write path as store → change → reload through the real class against a real database. “Returns true” and “rows changed” are different claims.
Check every prepare() and exec() result and stop on failure. A failure that’s only logged lets the function carry on and commit.
Use one QSqlQuery variable per statement and never pass the wrong one to exec(). After an UPDATE, check numRowsAffected(); zero rows is a valid SQL result, not an error.
When you fix a chain like this, expect the next fault to surface. Keep the reproducing test red until the whole round trip works.
Before simplifying a WHERE clause, find out why each condition is there. Pin the rule with a negative test.

Waiting for logout before login still left a race inside the handshake

Mon, 21 Sep 2026 14:49:00 +0200

Correction 2026-10-31: The earlier version attributed the reported failure to an idle timer. Later source review ruled out that timer for this app. The injected-teardown reproduction and exclusion-gate fix remain valid; the production trigger is unknown.

Users authenticated with Face ID, and the login failed anyway with a generic “login failed” error. The password was right, and the biometric enrollment was fine. The server’s log for the affected user showed the last handshake step arriving on a session in its initial state, with no user bound to it.

The app waited for any teardown already running before starting login. A dedicated test injected a new teardown after that wait and reproduced the same server-state signature. Holding an exclusion gate for the whole handshake fixed that reproduction, but did not establish the cause of the reported incident.

Setup

A stateful, multi-step login (SRP-style): start a session, send the identity, send the proof, finish. The server tracks progress per session cookie.
A session teardown: sign out on the server, then clear the local cookie jar. begin() returns immediately, and concurrent begin() calls join one teardown.
An earlier fix for a related bug: before its handshake, login called waitForCompletion() so a sign-out still in flight couldn’t destroy the new session.
The reproduction explicitly calls begin() during the handshake. It does not rely on an idle timer.

login: { credential in
 await sessionTeardown.waitForCompletion() // once, before the handshake
 return await client.login(credential) // four round trips
}

Reproduce with the real stack

A dedicated live test drove the production login code against a development server and fired begin() at a fixed delay into the handshake. At 500 ms:

start -> {"state":"new"}
send identity -> {"state":"identified", "salt":...}
send proof (M1) -> {"state":"proof-verified", "M2":...}
logout() <- the teardown fires here
POST /logout
finish (keys sent)
Logout success
Session teardown: cookie jar cleared
Error: state "new"
-> failed(.other)

This matched the server-state signature logged for the user. In the reproduction, the one-time wait had nothing to wait for: no teardown was running when login started. The test started one between two round trips.

The trap inside the trap: which effect does the damage

The code comments blamed the cookie-jar clear: the concluding step would go out without a session cookie and land on an anonymous session. That would suggest a small fix: defer only the local cleanup.

In the reproduction’s trace, the cookie jar was cleared after the finish step had already gone out with valid cookies. The server sign-out reset the half-authenticated session, and the concluding step then arrived at a session the server had put back to new. Deferring only the jar clear would still fail this exact trace.

Before you choose what to defer, read the ordering on the wire. The local effect you can see easily may not be the one that broke things.

Fix: hold exclusion for the whole protocol

The teardown gets a gate. Login holds it from the first round trip to the last. A begin() that arrives while the gate is held is honored but deferred—neither its sign-out nor its cookie clear runs until the handshake releases the gate.

private struct TeardownState {
 var inFlight: Task<Void, Never>?
 var handshakes = 0 // a count, so overlapping logins can't release each other's gate
 var isHeldBack = false
}

begin: {
 state.withValue { s in
 guard s.inFlight == nil else { return } // join the running one
 guard s.handshakes == 0 else { s.isHeldBack = true; return }
 s.inFlight = Task { await runTeardown() } // clears inFlight when done
 }
},
beginHandshake: {
 // Wait out a running teardown, then claim the gate. Loop, because the slot
 // is only free on a later pass.
 while let running = state.withValue({ s -> Task<Void, Never>? in
 guard let t = s.inFlight else { s.handshakes += 1; return nil }
 return t
 }) {
 await running.value
 }
},
endHandshake: {
 state.withValue { s in
 s.handshakes -= 1
 guard s.handshakes == 0, s.isHeldBack else { return }
 s.isHeldBack = false
 s.inFlight = Task { await runTeardown() }
 }
}

func excludingTeardown<T>(_ body: () async -> T) async -> T {
 await beginHandshake()
 defer { endHandshake() }
 return await body()
}

login: { credential in
 await sessionTeardown.excludingTeardown {
 await client.login(credential)
 }
}

The other properties stayed the same. begin() still returns immediately. A second begin() still means one teardown, whether that teardown is running or held back. A teardown whose sign-out throws still releases its waiters. Logout still happens: it runs right after login, which then ends in a signed-out state on purpose.

Later source review found that this app created its idle timer only after the full login completed and dropped it when the login screen appeared. The shipped revision had the same wiring. The earlier claim that this timer disproved a signed-in-only assumption was withdrawn. The gate provides exclusion without depending on that reachability argument.

A sibling app did retain its timer after logout and needed a signed-in guard. That was a separate timer-lifetime defect, not evidence that the timer caused this app’s reported failure.

Verification

Unit test for the new direction. A teardown begun inside excludingTeardown runs neither effect until the body returns. The expected order is [handshake, signOut, clearCookieJar]. Inside the body, the test awaits waitForCompletion() rather than Task.yield(), so a buggy teardown is guaranteed a chance to run. A yield would leave that to the scheduler.
Unit test for the old direction. A teardown already running still finishes before the handshake starts. That is the guarantee the earlier fix provided.
Injected-teardown sweep. The reproduction fires the teardown at 100, 200, 300, 400, 500 and 700 ms. It had been marked XCTExpectFailure. After the fix, every delay logged in:

teardown at 100ms -> loggedIn
teardown at 200ms -> loggedIn
...
teardown at 700ms -> loggedIn

The expected-failure marker was removed in the same commit.

What this doesn’t prove

The reproduction establishes a failure mechanism and verifies the exclusion fix. It does not prove that this mechanism caused the reported incident or identify its triggering call. We ruled out the idle timer for this app; the field trigger remained unknown.
The dedicated reproduction and the unit tests verify the fix. The live test class still exited with failures. They came from a separate host-app dependency problem in the test process, and another test in that class still tracks an unrelated second-login bug. Don’t read “the sweep passes” as “the live suite is green”.
The ordinary test plan passed, but live tests are opt-in there, so it says nothing about the server interaction.
Which delays hit the window depends on latency to the server. A sweep is better than a single delay, but it is still a sample.

Takeaways

Waiting once for cleanup before a multi-step protocol protects only that instant. If cleanup can start during the protocol, exclude it for the whole protocol.
Defer cleanup, don’t drop it. Queue the request and run it when the gate is released.
Count holders instead of using a flag, so two overlapping logins can’t release the gate from under each other.
Read the wire ordering before you choose what to defer. Here, the server sign-out did the damage, not the local cookie clear that everyone suspected.
A matching reproduction does not identify the production trigger. Trace the initiating call in the affected build before naming it as the cause of an incident.

Zeep returns an empty list: a repeated xs:sequence lands in _value_1

Sat, 12 Sep 2026 23:59:00 +0200

A SOAP client listed received messages and got an empty list on every call, even though the mailbox had messages. No exception or error status occurred, and the unit tests passed. This was the line that read the response:

records = getattr(response, "dmRecords", None)
page = (records and getattr(records, "dmRecord", None)) or []

The XML on the wire had <dmRecords><dmRecord>…</dmRecord><dmRecord>…</dmRecord></dmRecords>, but zeep never created a dmRecord attribute. It put the records in dmRecords._value_1.

If zeep gives you None or an empty list for a field you can see in the raw response, check whether the schema repeats the sequence rather than the element.

Setup

An Odoo 18 addon pulls messages from the Czech Data Box system (ISDS), a government SOAP service with published WSDL and XSD files, and talks to it through zeep. I reproduced the behavior below with zeep 4.3.3 on Python 3.13. The list type in the ISDS schema looks like this:

<xs:complexType name="tRecordsArray">
 <xs:sequence minOccurs="0" maxOccurs="unbounded">
 <xs:element name="dmRecord" type="tns:tRecord" />
 </xs:sequence>
</xs:complexType>

maxOccurs="unbounded" sits on the xs:sequence, where you would usually expect it on the xs:element.

What went wrong

zeep maps the two forms differently. Here is a minimal schema with both, parsed with zeep’s own types:

<xs:complexType name="tRepeatedSequence">
 <xs:sequence minOccurs="0" maxOccurs="unbounded">
 <xs:element name="record" type="tns:tRecord"/>
 </xs:sequence>
</xs:complexType>
<xs:complexType name="tRepeatedElement">
 <xs:sequence>
 <xs:element name="record" type="tns:tRecord" minOccurs="0" maxOccurs="unbounded"/>
 </xs:sequence>
</xs:complexType>

Feed both the same two <record> children, then call serialize_object(..., dict):

tRepeatedSequence: {'_value_1': [{'record': {'id': '1'}}, {'record': {'id': '2'}}]}
tRepeatedElement: {'record': [{'id': '1'}, {'id': '2'}]}

With a repeated element, you get obj.record as a list. With a repeated sequence, each repetition becomes an item. The sequence has no name, so zeep calls the list _value_1, and each item is a dict keyed by the element names inside the sequence. A sequence with two elements would give you {"a": …, "b": …} per item.

The zeep docs explain _value_1 only under xs:choice and xs:any. Nothing there tells you that a plain repeated sequence ends up there too.

Plain attribute access would have failed loudly on the first real call:

AttributeError: tRepeatedSequence instance has no attribute 'record'

The getattr(..., None) or [] was written so that an empty mailbox wouldn’t crash. It also handled the case where the attribute doesn’t exist at all, and turned it into “no messages”.

The tests didn’t catch it because they replaced the client’s methods with mock.patch. They never sent a SOAP response through zeep, so they never saw how zeep shapes it.

Fix

Read _value_1 and unwrap each item:

records = getattr(response, "dmRecords", None)
page = [item["dmRecord"] for item in (getattr(records, "_value_1", None) or [])]

An empty <dmRecords/> still gives _value_1 == [], so the empty-mailbox case works.

To see the shape before you write the parsing code, dump the WSDL with zeep’s CLI:

python -m zeep dm_info.wsdl

ns0:tRecordsArray([dmRecord: ns0:tRecord])

The square brackets give it away. A repeated sequence prints with brackets around the field list. A repeated element, like tRepeatedElement above, prints with [] after the type instead: ns0:tRepeatedElement(record: ns0:tRecord[]).

Test the parsing with real XML

The test that covers this now runs canned SOAP XML through the real zeep client, with only the HTTP call patched:

def response(xml_body):
 result = requests.Response()
 result.status_code = 200
 result.headers["Content-Type"] = "text/xml; charset=utf-8"
 result._content = soap_envelope(xml_body)
 return result

with mock.patch.object(requests.Session, "post", side_effect=[response(listing)]):
 envelopes = client.list_received_messages(since, until)

self.assertEqual([e["id"] for e in envelopes], ["100001", "100002"])

Zeep still loads the WSDL, parses the envelope, and builds its objects, so a mapping mistake like this fails in the test suite with no network. Write the XML from the schema or from a captured response.

The same commit fixed a second or default with the same problem: return response.dmSignature or b"" would have stored a message without its signed copy. It now raises when the signature is missing.

Takeaway

zeep maps a repeated xs:sequence to _value_1, a list of dicts keyed by the inner element names. A getattr(..., None) or [] on top turns that into a silent empty result. Let missing attributes raise, and test SOAP parsing with real XML through zeep.

Search your zeep code for getattr( with a None default followed by or [], and check each field against the python -m zeep dump of your WSDL.

Uninstalling one Odoo module can remove its dependents and their data

Sun, 30 Aug 2026 00:00:00 +0000

An Odoo module replacement passed a rehearsal’s record-count and accounting-total checks, yet the uninstall also removed a dependent addon that owned important business data. The application still loaded. The acceptance checks had missed the field that mattered.

Before replacing a module, inspect its reverse dependencies and data ownership. Rehearse the corrected transition on a restored copy, then compare the affected values by record identity before approving the live change.

Uninstall computes a reverse-dependency set

In Odoo 18’s module implementation , downstream_dependencies() recursively finds modules that depend on the selected modules. button_uninstall() marks both the selection and those dependents as to remove.

The same source describes module_uninstall() as removing database structures created by the module, including tables and columns. Actual deletion depends on the affected records and metadata; do not assume every business record disappears, or that surviving rows retain every field.

Generalize the replacement to a full addon, a smaller replacement, and an extension that depends on the full addon. Installing the smaller addon does not automatically redirect the extension’s dependency. Uninstalling the full addon still includes that extension in the removal set.

The first rehearsal had the wrong acceptance criteria

The historical transition checked surviving records, aggregate totals, and registry health. Those checks passed while the dependent addon was also uninstalled. The user caught the missing preservation requirement and required a restore.

The subsequent investigation found a second problem: changing the extension’s manifest alone would be insufficient. The smaller replacement did not yet implement a field the extension required. That field’s behavior needed to be ported before changing the dependency.

The retained record includes the corrective restore and a later clone’s uninstall/registry-reload output. It ends with the corrected rehearsal and final verification still open. It does not prove a successful final transition or a completed before/after digest comparison.

Rehearse the field handoff explicitly

A safer transition procedure is:

Quiesce writes and capture a consistent database and filestore backup. Verify checksums and prove that the pair restores.
Record installed module states, reverse dependencies, and the fields and records that must survive. Keep the addon source revisions with the backup.
Implement the required field behavior in the replacement, including compute and reversal behavior where applicable. Version-control the dependent addon too.
On an isolated restore, install the replacement and upgrade the dependent addon so the database’s module metadata reflects the new dependency. Inspect the removal set again before uninstalling the old addon.
Complete the uninstall, reload the registry, and compare the required values and behavior. Investigate field metadata and ownership during the handoff; identical field names alone do not establish safe transfer.

Disable scheduled jobs and outbound integrations on the clone. A rehearsal should not send mail, contact payment services, or write to production storage.

Compare identity and value, not only counts

For a small affected dataset, compare canonical exports directly. For larger exports, a digest is convenient if the query scope, ordering, and serialization remain identical.

This synthetic Python check demonstrates why counts and non-null counts are insufficient. It models two records whose field values were exchanged; it is not an Odoo migration test:

import hashlib
import json


def digest(rows):
 payload = json.dumps(sorted(rows), separators=(",", ":"), ensure_ascii=True)
 return hashlib.sha256(payload.encode("utf-8")).hexdigest()


before = [(1, "2026-01-10"), (2, None), (3, "2026-01-20")]
after = [(1, "2026-01-20"), (2, None), (3, "2026-01-10")]
assert len(before) == len(after)
assert sum(value is not None for _, value in before) == 2
assert sum(value is not None for _, value in after) == 2
assert digest(before) == digest(list(reversed(before)))
assert digest(before) != digest(after)
print("counts match; identity/value digest detects the change")

For a real comparison, export the stable record ID alongside each affected field, preserve nulls, and normalize dates and other types explicitly. A missing column must fail the check. Also test compute behavior and important workflows: a value digest cannot prove those work.

Before your next uninstall, list which dependent addons and field values must remain, and make their survival a rehearsal acceptance criterion.

Standards vs Spec review: a second agent found duplicate writes past 1,572 green tests

Wed, 22 Jul 2026 21:58:00 +0200

This is what the review agent reported, just before it listed its findings:

The test evidence is clean: Ruff passes, Python reports 1,572 passed / 56 skipped /
1 expected failure, and the GUI reports 418 passed / 4 skipped. The review nevertheless
found safety gaps the tests do not exercise — most importantly, two write clients retry
"500-after-commit" before their advertised idempotency/verification guard can run.

A coding agent had built the branch: 53 commits, 147 files. A different model reviewed it and found two paths that could put the same invoice in the ERP twice and report plain success. No test covered them, and no linter could. They came from a reviewer with one job: check the code against what the branch said it does.

The setup

The repository vendors the two-axis code-review skill from mattpocock/skills . It pins a fixed point and diffs against the merge base, git diff master...HEAD, then forks two subagents in parallel:

Standards: does the diff follow the repo’s documented rules (glossary, ADRs, a baseline of Fowler code smells)?
Spec: does the diff do what the originating issue or PRD asked for?

The parent agent runs lint and tests meanwhile, “as supporting evidence, not as a substitute for the diff review”. It then reports the two axes under separate headings and doesn’t rerank them, so a pile of naming nits can’t bury a P1.

The Spec brief is short:

Report: (a) requirements the spec asked for that are missing or partial;
(b) behavior in the diff that wasn't asked for (scope creep);
(c) requirements that look implemented but where the implementation looks wrong.
Quote the spec line for each finding, under 400 words.

What each axis found

There was no PRD for the branch, so the Spec reviewer used the commit messages and the ADRs. One commit claimed that the MCP server’s idempotency check made it safe to retry a write that got a 5xx. The reviewer traced the call: the check ran before _request(), the key was recorded after success, and the retry happened inside _request(), in between. A webhook worker had the same shape, with a verify-by-GET that ran only if the request finally raised. A retry that succeeds never raises. (The full HTTP 500 story is here .)

The Standards reviewer found something else. Two functions called urlopen() directly, past the shared transport that an ADR says owns all retries. It also found an import that went through a module the ADRs forbid as a re-export facade, and an unfinished vocabulary migration.

None of the Standards findings overlapped with the Spec ones. All six were fixed the same evening, in one commit.

Why the tests stayed green

The tests came from the same author as the claim. No test let a 500 commit the write, let the retry succeed, and counted the writes, because the commit said the idempotency check covered that case. A suite tests what its author thought could break.

The Spec reviewer starts from the claim instead. A commit message says the code is safe, so it goes looking for the line that makes it safe, and here that line ran on the wrong side of the retry.

Two more runs, same pattern

In the next review, a few hours later on a six-commit range, a guard was undone downstream. Every write had to carry exactly one confidence label, and create_* checked that. But when label setup had failed, a fallback further down stripped all labels from the payload. The reviewer reproduced a labeled invoice leaving without its label. It also found a write that went through a raw put() and skipped the scope check that limits which record types may be written at all.

The third review had nothing to go on but commit messages. The branch added a gate: if an invoice is addressed to another company, warn and refuse to post it. The reviewer said it was inferring the spec from the commits, so I pasted the original request and the Spec subagent re-checked against that. The request held a second requirement the commit summary had left out, which turned out to be implemented. The review also found that a replay of staged payloads called the push function directly and skipped the new gate. And while company registration numbers were checksum-validated, VAT IDs relied on regex alone, so one misread digit could block a legitimate invoice. Both failed the first test, then got fixed.

How to use it

Run Standards and Spec as separate subagents, so neither sees the other’s findings, and report them under separate headings.
Give the Spec reviewer the authoritative spec. If you only have commit messages, say so, and swap in the real request the moment you have it.
Treat claims in commit messages and ADRs (“X makes this safe”, “Y owns all retries”) as requirements. They are the cheapest spec you have, and the author’s tests already agree with them.
If you can, review with a different model than the one that wrote the code. It is less likely to share the writer’s blind spots.

Takeaway

A green suite tells you the code does what its author expected. A Spec review by a second agent tells you whether that matches what the author promised. The duplicate writes were in the gap between the two.

Take the last branch your agent wrote, and have a second agent check each “this makes X safe” in its commit messages against the code.

Date the bug before you park it: fence, drift or silence

Fri, 17 Jul 2026 01:13:00 +0200

My coding agent was splitting a 738-line Python function into steps and found a real bug along the way:

result.issues = validation.issues

That line rebinds the list instead of extending it, so it dropped every issue recorded before validation, including the note that repair mode had just deleted a record in the accounting system. The push still succeeded, and the operator’s report said nothing about the delete.

The agent didn’t fix it. It wrote a known-issues entry and added two @unittest.expectedFailure tests, because the handoff for the refactor said: “characterize, then collapse, then fix separately”. It was the wrong call, and two git log -S runs showed it.

The setup

Claude Code working in a Python repository of an accounting pipeline, tested with unittest;
A long-lived function where several steps append to result.issues, the operator’s per-file report;
A refactor rule that a split must not change behavior.

What parking costs

An expectedFailure marker looks like a decision. The next reader sees a documented, tested, known gap and assumes someone weighed it and chose to keep it. If nobody did, the marker turns an accident into policy and keeps a live bug in the tree behind it.

So before you park a bug, find out whether anyone ever decided it. I asked the agent to run the fence check (Chesterton’s fence: don’t remove a fence until you know why it was put up).

Date the bug

A bug is a relationship between two things: this line and the code it gets wrong — date both.

git log --oneline -S'result.issues = validation.issues' -- <path>
git log --oneline -S'result.issues.append' -- <path>

The result went into the fix’s commit message:

2026-04-16 rebind written. result.issues had NO prior writers;
 assigning validation's list into it lost nothing.
2026-04-19 + vendor mismatch trail above it +3d
2026-04-23 + REPAIR delete note above it +7d
2026-04-24 + registry retry verdict above it +8d
2026-05-27 + self-billed reroute above it +41d

The rebind was correct when it was written. Over the next 41 days, four commits added writers above it, and none of them touched it. Nobody decided to discard those issues; the code predates the thing it breaks.

That gives one of three verdicts:

Fence: someone decided this, in a commit message, a spec’s non-goals or an ADR. Leave it.
Drift: the code predates what it is wrong about, or its author never touched the site. Fix it now, and put the dates in the commit message so a reviewer can see you didn’t remove a fence.
Silence: the history says nothing either way. Park it, and say plainly that nobody decided it.

This one was drift, so it got fixed the same night. It had stayed hidden because the only test that reached this code patched the whole function body out. The split made the body reachable, and the first end-to-end repair test found the bug.

The obvious fix was wrong

A drift verdict says nobody meant the bug; you still have to find out what depends on it. The one-word fix:

result.issues.extend(validation.issues)

would have broken an alias. A later step appends registry failures to result.issues, and those reach the invoice’s note only because the rebind made result.issues and validation.issues the same list, which the note builder reads:

result.issues = validation.issues # one list, two names
apply_registry_failure(result) # appends to result.issues ...
note = build_note(validation) # ... and the note sees it through the alias

With .extend(), every invoice that failed the business-registry check would have lost that warning from its note: one bug traded for a quieter one. The fix kept the rebind and put the earlier issues back after the later writers ran:

earlier_issues = list(result.issues)
result.issues = validation.issues
apply_registry_failure(result)
# ... the other writers ...
result.issues = earlier_issues + result.issues

The tests were written to catch both mistakes: reverting the fix fails the four tests that check the earlier issues survive, and the naive .extend() fails the one that checks the registry warning still reaches the note. The known-issues entry was deleted, and the expectedFailure markers became ordinary tests.

What silence looks like

That night, four other bugs were dated as drift and fixed. One wasn’t. Debug files were named from the input’s stem, so invoice.docx and invoice.pdf in one batch wrote the same file, and the later one won. That naming shipped on a pipeline that already read several formats, so the code didn’t predate the collision. No commit message or spec mentioned it either: stem is just the obvious way to name a sidecar file. That one stayed a known-issues entry, with the dating written into it.

The skill

The fence check is a Claude Code skill I had written down earlier that evening. The agent had it and didn’t run it until asked, so its memory now says to run it the moment it finds a bug, before choosing between a fix and a known-issues entry. Here it is verbatim. The examples and the docs/superpowers/ paths are from my repository; swap in your own spec directory.

---
name: checking-the-fence
description: Date a bug before fixing it — Chesterton's fence. Use when you have found code that looks wrong and are about to change it, when you are about to delete something that looks dead, or when another skill needs the fence check.
---

# Checking the Fence

A bug you found is a **fence** across a field. Before you take it down, find who put it there. Here that is written down twice — in `git log` and in `docs/superpowers/` — and is usually decisive in a minute.

Two verdicts. They are the whole skill:

- **Fence** — someone decided this. Leave it, and record an ADR if the next reader will reach for the same fix you just did.
- **Drift** — nobody decided it. The code predates the thing it is wrong about, or an author was never told which files the job touched. Fix it.

Fences here are real, not hypothetical: `docs/adr/0002` exists only to stop a reader from deleting emoji handling that has no inbound traffic and looks dead.

## Steps

Stop at the first step that returns a verdict.

### 1. Date the code

```
git log --oneline -S'<symbol>' --all -- <path>
git log -1 --format=%B <sha>
```

Commit messages here carry the rule the author meant to encode, and the rule is often the thing the bug breaks. `_PARSER_PRIORITY`'s own commit says *"process deterministic parsers before PDFs/images"* — while the table shipped the cheapest format last.

**Done when** you can name the commit and quote the rule.

### 2. Date what it is wrong about

A bug is a relationship: a table missing a format, a copy missing an entry, a switch missing a branch. Date the other side too.

Code cannot have decided about something that did not exist yet. `_PARSER_PRIORITY` (2026-04-27) predates isdoc (2026-06-01) by five weeks.

**Done when** you have both dates. Predates → **drift**. Stop.

### 3. Read the spec

```
grep -rli '<feature>' docs/superpowers/specs docs/superpowers/plans
```

Read the **Non-goals** first — that is where a deliberate gap is named. A spec that scopes the gap out is a **fence**. A spec whose own words the bug contradicts is **drift**: the ISDOC spec calls it *"a deterministic fast path before Claude"*.

**Done when** you have quoted the line that scopes it out, or established that no spec names it.

### 4. Read the blast pattern

```
git show --stat <sha>
```

Which files that commit touched, against which it needed to. An author who updated three of five consumers was not deciding about the other two — nothing told them. The isdoc and docx commits landed a day apart and touched different subsets.

**Done when** you can say whether the author addressed the site at all. Never addressed → **drift**.

### 5. Was it born wrong?

For a forked copy, compare it against its source **on the day it was copied**, not today.

```
git show <sha>:<source-path>
```

`coverage_report.py` shipped without `.json` on 2026-04-24, and the map it was copied from carried `.json` that same day. Born forked, then rotted twice more.

**Done when** you know whether the gap is drift-over-time or a bad copy. Either is **drift**.

## Applying the verdict

**Drift** — fix it, and say in the commit message what you found: the dates, and that nothing decided it. That is what stops a reviewer from wondering whether you removed a fence.

**Fence** — leave it. Offer an ADR when a future reader would reach for the same fix; that reader is usually the next architecture review, or you.

**Neither** — the history is silent and no spec mentions it. Say so plainly and let the user decide; silence is not consent, and it is not a fence either.

Save it as .claude/skills/checking-the-fence/SKILL.md in your repository, and tell the agent in CLAUDE.md or AGENTS.md to run it before it records a bug as a known issue.

Takeaway

A parked bug looks like a decision, so check whether anyone made one. If the code predates what it breaks, fix it now, and check the mechanism first: knowing nobody meant the bug doesn’t tell you what depends on it.

Next time your agent adds an expectedFailure or a known-issues entry, ask it for the two dates first.

Every test mocked the helper, and except Exception swallowed its TypeError on every PDF

Thu, 16 Jul 2026 23:42:00 +0200

For seven weeks, a document classifier never read a single PDF. This is the helper that was supposed to give the model some text (trimmed):

def _extract_excerpt(file_path: Path) -> str:
 """First-page text for PDFs, first ~8 kB for everything else."""
 if file_path.suffix.lower() == ".pdf":
 try:
 pages = extract_pdf(file_path)
 return (pages[0] if pages else "")[:EXCERPT_BYTES]
 except Exception:
 return ""
 ...

extract_pdf doesn’t return a list of pages. It returns a dataclass, so every call raised this:

TypeError: 'PDFExtraction' object is not subscriptable

The except caught it and returned "". The prompt then held the filename and an empty excerpt, and the model answered anyway: a document type and a confidence score, for every PDF it was asked about.

If a helper of yours has a catch-all except and your tests patch that helper, check whether anything ever runs it for real.

Setup

A web upload went through a small Python script that sorted the file into one of the document kinds declared for an accounting period (received invoices, bank statements, and so on). With two or more possible kinds, it built a prompt from the filename plus the start of the document and asked a language model, which answered with JSON like {"kind": "...", "confidence": 0.9}. The PDF text came from an in-house extract_pdf() built on pdfplumber. Tests used the standard library’s unittest and unittest.mock.

What went wrong

The helper was wrong from the day it was written. extract_pdf had returned a PDFExtraction dataclass with a pages list since its first commit, six weeks earlier, and its docstring said so. The if pages check didn’t catch the mistake: a dataclass without __len__ is always truthy, so the check passed and pages[0] raised.

The except was there so that a corrupt or encrypted upload wouldn’t break classification. It caught the TypeError just as readily and returned "", which is also the documented answer for an image upload with no text to read.

And every test patched the helper out. The tests that reached the model looked like this:

with patch("scripts.classify_upload.invoke_ai") as m, \
 patch("scripts.classify_upload._extract_excerpt", return_value="Invoice 2026-001 ..."):
 m.return_value = json.dumps({"kind": "received_invoices", "confidence": 0.92})
 result = classify_file(Path("/tmp/x.pdf"), ["issued_invoices", "received_invoices"], "claude")

They covered the prompt plumbing and the JSON parsing, and none of them ever called _extract_excerpt. The implementation plan that specified the helper had the same broken line, and it specified these tests, mocks included. The code, the plan, and the tests agreed with each other, and none of them had been checked against extract_pdf.

The model hid the rest. A filename like invoice-0423.pdf can be enough for a right guess, and a wrong guess came back with a confidence score like any other, so nothing in the output said the excerpt was empty.

Fix

The fix reads the first page, as the docstring promised:

extraction = extract_pdf(file_path)
first_page = extraction.pages[0].text if extraction.pages else ""
return first_page[:EXCERPT_BYTES]

The new test class calls the real function on a real PDF:

class RealPdfExcerptTests(unittest.TestCase):
 def setUp(self):
 self.tmp = tempfile.TemporaryDirectory()
 self.root = Path(self.tmp.name)

 def tearDown(self):
 self.tmp.cleanup()

 def test_a_pdf_yields_its_first_page_text(self):
 path = self.root / "invoice.pdf"
 _write_pdf(path, "INVOICE 2026-001 Supplier ACME")
 self.assertIn("INVOICE 2026-001", _extract_excerpt(path))

 def test_a_corrupt_pdf_yields_no_signal_rather_than_raising(self):
 path = self.root / "broken.pdf"
 path.write_bytes(b"%PDF-1.4\nnot actually a pdf at all")
 self.assertEqual(_extract_excerpt(path), "")

_write_pdf is an existing test helper that writes a one-page PDF with a text layer. Before committing, I ran the first test against the original code to confirm it fails there. The except stayed, because corrupt uploads are real. The second test now says what it’s for.

A day later, the same helper had a second route to the same result. It kept its own list of text suffixes, so Word, Excel, and XML e-invoice uploads fell through to "" and were classified by filename too. Two of those formats joined the pipeline a week after the list was written, and nobody updated it. The helper now asks the pipeline’s format registry instead.

Why

A mock replaces the code it patches with your assumptions about that code. When every test mocks a function, its tests check only those assumptions, and the function itself is never run. A catch-all except inside the function then removes the last signal you’d get in production: no traceback, no failed request, just a default value that looks legitimate.

A language model downstream makes this worse. A parser fed an empty string usually fails loudly, while a model returns its best guess in the usual format, confidence field included, whatever the input.

Takeaway

Mock the expensive or external dependency (here, the model) and let the function under test run on a real input. If it has a catch-all except, give it a test with an input that should succeed, so a bug can’t hide behind the except.

Grep your tests for patch( targets that point at your own module’s private helpers, and for each one, find a test that calls the helper unmocked.

Sandbox-blocked tests called "environmental", then committed

Mon, 13 Jul 2026 15:38:00 +0200

My coding agent finished a feature, ran the full test suite, and reported:

Full suite: 135 passed, 97 skipped; 22 DB suites failed due to no container runtime, plus sandboxed stub bind. Commit only handoff implementation files.

Then it committed. I ran pnpm test myself and got three real failures. The production build it had also skipped was broken too, and nobody found out for two months.

If your agent’s test run shows Error: Could not find a working container runtime strategy or Error: listen EPERM: operation not permitted 0.0.0.0:8080, those tests never ran, and a summary that counts them as “blocked” tells you nothing about the code.

The setup

Codex CLI on macOS, with its default sandbox
A pnpm monorepo: a Next.js 15.5.19 app plus a few packages
Vitest 2.1.9, with Testcontainers 10.28.0 starting Postgres for the database tests and a local stub of an external service listening on port 8080

The agent split the feature into five slices and gave each to a parallel subagent. Each subagent ran its own focused tests, and the orchestrator ran the full suite and the build at the end.

What went wrong

The sandbox blocks more than writes outside the project. Here it blocked three things the checks needed:

Testcontainers couldn’t reach Docker: Could not find a working container runtime strategy.
The service stub couldn’t bind its port: listen EPERM: operation not permitted 0.0.0.0:8080.
next build couldn’t write its compiler cache: EPERM writing ~/Library/Caches/next-swc.

Every database suite failed in beforeAll, and Vitest reported the tests inside them as skipped. The agent’s final report was accurate:

- Typecheck pass
- Focused tests pass
- Full: 135 passed; DB suites blocked by no Docker
- Build blocked by Next SWC cache permissions

Every line of that is true. But the suite has 232 tests, and 135 + 97 = 232: over 40 percent of it never ran, including every test that touched the database, where this feature did most of its work.

On the host, the same suite printed:

Test Files 2 failed | 63 passed (65)
 Tests 3 failed | 229 passed (232)

One new test forgot to start the service stub, so its client found no endpoint configured. Two older tests broke because the feature added an ownership check and their fixtures sent requests without a logged-in user (Error: authentication required). The agent fixed all three in two minutes once it saw them.

The build was worse. The commit re-exported a server action from another "use server" file:

"use server";

export { finalizeArtifact } from "../../reports/actions.ts";

tsc accepts this, and so do the tests. Only next build rejects it:

Failed to compile.

./app/reports/[id]/actions.ts
Error: x Only async functions are allowed to be exported in a "use server" file.

Two months later, a different session ran a production build, hit this error, and deleted the line.

The same sandbox, the same day

Half an hour before the bad commit, another session in the same repository hit the same two errors. At first it called them environmental too. Then it decided:

Full suite blocked by sandbox: Testcontainers cannot see Docker; stub bind gets `EPERM :8080`. Since Docker is reportedly running, retry full suite with host access.

Two minutes later it reported “Host rerun green: 60 files, 222 tests”, and only then did it commit. The agent that fixed my three failures did the same: it reran the focused tests with host approval before committing the fix. Both sessions knew how to leave the sandbox; nothing required them to before a commit.

Fix

Treat “blocked” as “not run”, and make “not run” stop the commit.

In Codex, a command runs outside the sandbox when the agent requests sandbox_permissions: "require_escalated" and you approve it. Put the rule in the instructions file the agent reads every session (AGENTS.md, or CLAUDE.md for Claude Code):

A check that the sandbox blocked has not passed. Before committing, rerun every blocked test suite and build with host access (`sandbox_permissions: "require_escalated"`).
If that isn't possible, don't commit: report what didn't run.

Then read the agent’s numbers yourself. If passed plus skipped equals the total, and you never skip tests on purpose, every skipped test is one that didn’t run. If you have CI, have it run the full suite and the build before a merge.

Why it happens

The agent’s memory for this project already had a rule: “separate passed static or focused checks from runtime suites blocked by the environment, and reserve the final green claim for integrated verification.” The agent followed that rule to the letter: it never claimed green and listed exactly what had been blocked. Then it committed, because nothing in its instructions tied the commit to the green claim.

“Blocked by the environment” reads like an infrastructure problem that someone else will handle, so it ends up as a footnote in the report while the commit goes ahead. The rule made the report honest. It didn’t make anyone run the tests.

Takeaway

When a sandbox blocks a test or a build, nothing was verified. An agent that reports this honestly will still commit, unless its instructions say that a blocked check stops the commit.

Look at your agent’s last commit: if its summary says a suite or a build was “blocked”, rerun them on the host today.

Filename HTML injection: entities bypass a raw URL blocklist

Mon, 20 Apr 2026 00:00:00 +0000

A filename filter that rejects https:// can still admit an HTML attribute containing https://. If an application inserts that filename into HTML without escaping it, the parser decodes the character references and produces a usable URL. Put the repair at the rendering boundary: render filenames as text, even when another client already validates them.

This example synthetically demonstrates the encoding mismatch. It makes no claim about a particular product, deployment, or mail-client version.

The filter and the renderer see different strings

The vulnerable flow has two independent mistakes:

A producer stores attacker-controlled text that another component assumes is safe because a UI restricts filename characters.
An HTML generator interpolates that text as markup. A server-side blocklist searches its stored spelling for literal URL schemes.

An API, import job, or older client can bypass the first UI’s restrictions. The value can remain harmless in a web page that escapes it, then become markup in a generated email, export, or share preview that does not. In the API flow, validate the stored value only as data; do not treat it as safe HTML.

In an authorized test, a probe filename could contain:

<img src="https&#58;&#47;&#47;example.invalid&#47;pixel">.txt

Here : represents a colon and / a slash. Attribute parsing resolves those references before the image loader consumes the URL. The reserved .invalid domain keeps this example from contacting a real collection endpoint.

An image request can demonstrate remote-content loading under the tested settings. It does not establish JavaScript execution, arbitrary file access, or credential theft. A mail client may strip event handlers while retaining an image element, and it may block external images entirely.

Reproduce the mismatch without sending mail

Save this as check_filename.py and run python3 check_filename.py. It uses Python’s HTML parser to inspect attribute values; it does not open a browser or make network requests.

from html import escape
from html.parser import HTMLParser

filename = '<img src="https&#58;&#47;&#47;example.invalid&#47;pixel">.txt'
assert "http://" not in filename and "https://" not in filename

class Inspect(HTMLParser):
 def __init__(self):
 super().__init__()
 self.images = []
 self.text = []

 def handle_starttag(self, tag, attrs):
 if tag == "img":
 self.images.append(dict(attrs).get("src"))

 def handle_data(self, data):
 self.text.append(data)

unsafe = Inspect()
unsafe.feed(f"<p>Shared file: {filename}</p>")
unsafe.close()
assert unsafe.images == ["https://example.invalid/pixel"]

safe = Inspect()
safe.feed(f"<p>Shared file: {escape(filename, quote=True)}</p>")
safe.close()
assert safe.images == []
assert "".join(safe.text) == f"Shared file: {filename}"
print("unsafe: URL decoded; safe: filename preserved as text")

The safe version escapes the ampersands as well as the angle brackets. The parser displays the original entity spelling as filename text; it does not reparse that displayed text into another HTML element.

Prevent markup at every filename sink

Use your template engine’s normal autoescaping for text content. In a DOM API, assign textContent; avoid passing the filename to innerHTML. Apply context-appropriate encoding if a value belongs in an attribute. A filename placed inside a URL requires URL construction and validation as well as HTML attribute escaping.

If a feature intentionally accepts rich HTML, use a maintained allowlist sanitizer with an explicit policy for external resources and URL schemes. Decoding and then extending the blocklist still leaves the application maintaining a second, incomplete interpretation of HTML. OWASP’s XSS prevention guidance covers the context-specific encoding rules.

Keep server-side filename rules for filesystem and application constraints. Those rules serve a different purpose from making text safe for an HTML sink.

Verify the actual consumer

Test a filename created outside the restrictive UI, then follow it into each renderer. Inspect the generated HTML or MIME source as well as the visible result. For generated mail, test composition separately from recipient viewing.

On an authorized receiving client, use a controlled endpoint and record whether external content was enabled. Distinguish an outbound request from visible rendering and from script execution; one result does not prove the others. After the fix, repeat the probe and confirm that the filename appears literally, no injected element exists, and no request attributable to it occurs.

Add this filename to the HTML generator regression tests. Testing only the upload validator would miss the boundary where the text became active HTML content.

If GPG is missing, signature verification must fail closed

Mon, 20 Apr 2026 00:00:00 +0000

An updater effectively contained the equivalent of if gpg: verify_signature(), then extracted the downloaded archive unconditionally. On a minimal installation without the optional binding, unsigned content took the trusted path. If authenticity is required, treat an unavailable verifier as a hard failure, not permission to continue.

This pattern is CWE-347, improper verification of a cryptographic signature . It becomes more dangerous when the next stage expands an archive, loads a plugin, flashes firmware, or executes an installer, because the bypass then feeds directly into execution.

Optional crypto turns verification into best effort

This shape is unsafe:

if verifier is not None:
 if not verifier.verify(payload, signature):
 raise SignatureError("bad signature")

extract(payload) # Runs when no verifier exists.

The absence branch is the bypass. During review, search for trust decisions guarded by import success, feature flags, platform checks, or broad exception handlers, then follow each branch to the first sensitive consumer. A green path should require a positive verification result.

An attacker still needs influence over the downloaded payload or its source. That may come from a compromised update endpoint, repository, mirror, DNS or TLS failure, or another write primitive. Missing verification does not by itself prove remote code execution; it removes the authenticity barrier that should contain those failures.

Archive extraction can turn bypass into a file write

A second bug often makes the impact concrete: archive members such as ../../outside or symlinks that escape the destination. Joining a destination with an attacker-controlled member name does not remove traversal so the archive bug can turn the bypass into a file write.

Python added extraction filters in 3.12, and Python 3.14 changed the default to the data filter. The current tarfile documentation still recommends explicit filtering and additional limits because no built-in filter stops every denial-of-service or filesystem hazard.

This local fixture asserts both failure paths. Save it as verify_extract.py and run it only in a disposable directory:

import io
import tarfile
import tempfile

class VerificationError(Exception):
 pass

def verify_and_extract(payload, signature, verifier, destination):
 if verifier is None:
 raise VerificationError("signature verifier unavailable")
 if not verifier(payload, signature):
 raise VerificationError("signature rejected")
 if not hasattr(tarfile, "data_filter"):
 raise RuntimeError("safe tar filter unavailable")
 with tarfile.open(fileobj=io.BytesIO(payload), mode="r:*") as archive:
 archive.extractall(destination, filter="data")

def traversal_archive():
 output = io.BytesIO()
 with tarfile.open(fileobj=output, mode="w") as archive:
 body = b"test"
 member = tarfile.TarInfo("../outside")
 member.size = len(body)
 archive.addfile(member, io.BytesIO(body))
 return output.getvalue()

payload = traversal_archive()
with tempfile.TemporaryDirectory() as destination:
 try:
 verify_and_extract(payload, b"sig", None, destination)
 raise AssertionError("missing verifier was accepted")
 except VerificationError:
 pass

 try:
 verify_and_extract(payload, b"sig", lambda data, sig: True, destination)
 raise AssertionError("traversal member was accepted")
 except tarfile.FilterError:
 pass

print("missing verifier and traversal archive rejected")

The fake verifier isolates extraction behavior; it is not production signature logic. The fixture passes on Python 3.12 or newer when the filter is available.

Make verification a prerequisite state

Package the verifier as a required dependency for update support. If it cannot initialize, turn off updating and report a clear error. Pin the expected public key or trust root, validate the signature over the exact bytes later consumed, and reject missing, malformed, expired, or wrong-signer results. Do not catch verification errors and continue.

After verification, extract into a new temporary directory with filter="data". Reject links if they are unnecessary, limit member count and total expanded size, allow only expected file types, and publish the completed tree atomically. Run the updater with the least filesystem privilege it needs.

Regression tests should cover a valid signature, an invalid signature, a missing signature, an unavailable verifier, a corrupt archive, a traversal member, an escaping link, and an oversized archive. Assert that no file outside the staging directory changes on every failure.

Audit every if crypto_library: branch as if the dependency were absent, because eventually it will be.

Use public SDK integration tests to find API coverage questions

Mon, 20 Apr 2026 00:00:00 +0000

A well-tested API can still leave you unsure where to investigate next. Compare its exposed operations with the vendor’s public SDK integration tests, then inspect what those tests actually assert. The result is a list of coverage questions you can investigate within an authorized scope.

The starting assumption in one review was that the system was well tested, so ordinary scenarios were unlikely to reveal much. Reading the SDK tests changed the next question: which behaviors did this particular suite demonstrate? That comparison produced candidate leads. It did not provide a complete coverage audit or confirm a vulnerability.

Build an operation inventory before searching tests

Start with the documented API operations available to the account or environment under review. Record the API version and the SDK release or commit beside that inventory. An operation introduced after the SDK release should not be counted as an unexplained omission.

Map each operation to SDK methods and then to integration tests. Follow wrappers and shared fixtures; a test name does not have to match an operation name. Keep unresolved mappings marked as unknown.

Text searches help locate candidates, but they do not measure runtime coverage. A matching method name might appear only in setup, a comment, or an assertion about a mock. A missing name might be hidden behind a helper or parameterized test. Check whether each test calls the service, what inputs it supplies, and which results it verifies.

Record assertions, not just test presence

This clean-room fixture is invented. Its operation names and test behavior do not describe any real service:

Operation	Evidence in the fictional public suite	Next coverage question
`create_item`	Creates an item and asserts that an ID exists	Does an invalid field produce the documented rejection?
`update_item`	Updates an item owned by the test account and reads it back	Does a different test account lack permission to update it?
`get_item`	No direct test located; helper usage unresolved	Does a shared fixture exercise retrieval and assert its result?

The last row stays unresolved until someone reads the helper. It would be premature to label it “untested.”

For each mapped test, note its credentials’ role without recording secrets, fixture ownership, expected failure behavior, and assertions. A successful request under one account says little about authorization under another. A negative test that merely catches any exception may pass on a network failure without demonstrating the intended denial.

Use precise notes such as “no assertion for this behavior found in the inspected public suite.” Vendors may have private service tests, separate security suites, or tests in another SDK. Public test omissions cannot prove those checks are absent.

Follow the data to its consumer

A coverage gap can justify a closer look at an input field if an API accepts and returns a string, which proves only acceptance and retrieval. Storage may legitimately preserve arbitrary text.

For a rendering-related hypothesis, maintain a small dataflow record: input field, stored representation, consuming component, output context, and observed handling. The consumer determines whether it treats the string safely. An API response alone cannot demonstrate unsafe rendering or a security impact.

Before any live follow-up, confirm the allowed scope, test accounts, and rate limits; use owned test data and the least disruptive check that answers the question.

If the consumer cannot be inspected or exercised within that scope, leave the hypothesis unresolved. Do not turn a stored value into a vulnerability claim to make the investigation sound complete.

Verify the map before choosing a lead

Have another pass through the fixtures and shared helpers, reconcile SDK and API versions, and separate source inspection from tests actually executed. For every proposed follow-up, write down the missing assertion and the observation that would settle it.

Pick one unresolved row in your own operation map and trace it through the test’s assertions before sending another request.

Testing on Lessons learned

Swift Testing withKnownIssue can hide snapshot failures

Restrict which issues count as known

A mutation must change the rendered output

Keep the exception smaller than the assertion

A shared Swift timeout left an earlier task without a deadline

One timer could not cover overlapping fetches

Put the deadline beside the operation

Prove that the waiter needs this deadline

A cancellation deadline is not a hard timeout

SwiftUI large title disappears after tapping the selected tab again

The title still existed

A local modifier changed later navigation bars

Why the one-line hotfix was reversed

Verify the interaction, not just the label

Xcode Archive can ship the wrong display name despite correct source settings

Source checks missed the artifact

Check the processed plist during the build

Prove that wrong builds fail

Swift Testing: .serialized did not stop another suite deleting test files

A unique child directory still has a shared parent

Make ordinary test construction inert

Prove the constructor contract, then repeat the broader target

A CVE cross-reference is not an advisory entry

How the wrong relationship gets created

Filter by document role before matching identifiers

A prefix guard has a document contract

An offline cache TTL needs a clock-rollback policy

A negative age passed the upper-bound check

Retain the latest observed time

Refusing access did not mean deleting the cache

Test the rollback twice

A high-water mark cannot establish time that was never observed

iOS jailbreak protection in 2026: detection is a risk signal, not a security boundary

The short version

How a working check silently stopped working

When OWASP says to implement jailbreak detection

The current techniques are rootless and designed to hide

The response matters more than the Boolean

App Attest authenticates app instances, not clean devices

Test implementation, behavior, and bypass resistance

A reviewable 2026 “lege artis” baseline

Takeaway

References

Moodle 5.3 Classic theme serves an empty stylesheet: 'Undefined variable $mds-spacing-md'

The setup

Two things that looked like the cause

Getting the real error

Why

The fix

Takeaway

A five-second Swift Testing wait timed out after its loop had run five times

Setup

Reproduce before guessing

Dead end: a longer timeout

Fix: spend both budgets before failing

Pin it with a controlled starvation test

Verify both directions

A second guess hiding in the same tests

Takeaway

Snapshot precision 0.99 let a wrong text line pass all seven screen tests

Setup

The arithmetic

Measuring the mutation

The fix

Shrink the frame when you can

Takeaway

Inherited agent notes can send the next agent down the same dead end

A hint survived longer than its evidence

Rebuild only enough context to choose a test

Preserve the test behind a negative result

A format string in a TRACE logger can read the process environment

A bounded copy does not make the format safe

How to exercise the disclosure safely

Fix the sink and reduce the impact

Next.js Server Action arguments are Flight data: use error digests carefully

The application sees the result of decodeReply

Reproduce the protocol locally before probing further

What an error digest can tell you

Separate protocol reachability from React2Shell

Trap 1: zsh has a `log` builtin

Trap 2: `cargo test --exact` with the wrong path passes