Zero failures is suspicious: agent checks that silently ran nothing

·4 mins

My coding agent ran each of the two flaky Rust tests 150 times and got this:

streamed_read_right_after_a_rename: 0/150 failed
renames_are_combined_across_changes_they_do_not_depend_on: 0/150 failed

Its next message was “Zero failures is suspicious.” It was right: neither test had run once. In the same week, three of its checks gave a tidy answer (nothing found, all passed, all failed) that came from a broken command, and each broken command hid its own error.

If your agent’s output has (eval):log:1: too many arguments, 0 passed; 0 failed; … filtered out or argument list too long somewhere above the summary, the summary is about a check that never ran.

The setup

  • Claude Code 2.1.283 on macOS, where the agent’s shell is zsh 5.9
  • A Rust workspace whose test binary is run directly from target/debug/deps

Trap 1: zsh has a log builtin

The agent was debugging an FSKit file system extension and searched the unified log for sandbox denials:

log show --last 30m --style compact --predicate 'process == "MyFS"' 2>/dev/null | tail -40

Nothing came back, and the agent told me: “No log entries so far.” Half a minute later it ran log stream … > /tmp/fs.log 2>&1 & and read the file. Its first line was:

(eval):log:1: too many arguments

zsh has a builtin called log that lists the logins watch reports. Builtins win over $PATH, so log show never reached /usr/bin/log, and 2>/dev/null threw away the only sign of it. You can check your own shell:

$ zsh -c 'whence -wa log'
log: builtin
log: command

With /usr/bin/log, the denial showed up at once, and the agent corrected its earlier claim. The same error turned up in a different project’s session days later.

Trap 2: cargo test --exact with the wrong path passes

The 150-run loop called the test binary like this:

$B --exact tests::$t -q > /tmp/out_$t.txt 2>&1 || f=$((f+1))

With --exact, the filter has to equal the full test path. The tests live in vfs::tests, so tests::streamed_read_right_after_a_rename matched nothing, and libtest treats that as success:

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 93 filtered out; finished in 0.00s

Exit code 0, every time. Without --exact, the same filter would have matched as a substring. The agent listed the real names with $B --list, fixed the prefix, and the races then showed up under load.

Trap 3: an empty variable makes everything fail

After the fixes, the agent stress-ran the suite 400 times and looked up the binary first:

B=$(ls -t target/debug/deps/storage_core-* | grep -v '\.d$' | head -1)

The glob had grown past what ls accepts:

(eval):1: argument list too long: ls

failed runs:      400 / 400

$B was empty. zsh drops an empty unquoted word, so each run executed vfs::tests -q, got command not found into a log file nobody read, and counted as a failure. This time the agent saw the error right above the count and called the result bogus.

Fix

Make every check prove that it ran before you trust what it says.

Call /usr/bin/log by its full path, or put command in front of it, and keep stderr visible while you are still debugging:

/usr/bin/log show --last 30m --style compact --predicate 'process == "MyFS"' | tail -40

Count the tests that ran, not only the failures. One test per run should say 1 passed:

$B --exact vfs::tests::$t -q > /tmp/out_$t.txt 2>&1 || f=$((f+1))
grep -q '1 passed' /tmp/out_$t.txt || { echo "$t did not run"; exit 1; }

Get the binary path from cargo instead of a glob, and stop if it is empty:

B=$(cargo test -p storage-core --lib --no-run 2>&1 | grep -o 'target/debug/deps/storage_core-[0-9a-f]*')
: ${B:?test binary not found}

For the agent, add one line to its instructions file (CLAUDE.md or AGENTS.md):

When a check finds nothing, passes everything, or fails everything, first show that it ran:
how many items it checked, and that a known-bad input fails it.

Why it happens

All three are the same bug: the harness failed, and the failure looked like a result. A loop that counts exit codes can’t tell “no failures” from “no tests”, and a pipeline that ends in 2>/dev/null | tail can’t tell “no matches” from “no command”. Agents write exactly these one-liners, in zsh, many times per session, and they report what the last line printed.

The agent caught two of the three because the number looked wrong: zero out of 150 for known flaky tests, 400 out of 400 for a suite that had just passed. An empty log search looked plausible, so it went to the user as a finding.

Takeaway

A clean result from a check you haven’t seen fail is not evidence. Before you believe “nothing found” or “0 failed”, look for the count of what was checked.

Next time your agent reports zero failures, ask it how many tests actually ran.