Unittest on Lessons learned

Date the bug before you park it: fence, drift or silence

Fri, 17 Jul 2026 01:13:00 +0200

My coding agent was splitting a 738-line Python function into steps and found a real bug along the way:

result.issues = validation.issues

That line rebinds the list instead of extending it, so it dropped every issue recorded before validation, including the note that repair mode had just deleted a record in the accounting system. The push still succeeded, and the operator’s report said nothing about the delete.

The agent didn’t fix it. It wrote a known-issues entry and added two @unittest.expectedFailure tests, because the handoff for the refactor said: “characterize, then collapse, then fix separately”. It was the wrong call, and two git log -S runs showed it.

The setup

Claude Code working in a Python repository of an accounting pipeline, tested with unittest;
A long-lived function where several steps append to result.issues, the operator’s per-file report;
A refactor rule that a split must not change behavior.

What parking costs

An expectedFailure marker looks like a decision. The next reader sees a documented, tested, known gap and assumes someone weighed it and chose to keep it. If nobody did, the marker turns an accident into policy and keeps a live bug in the tree behind it.

So before you park a bug, find out whether anyone ever decided it. I asked the agent to run the fence check (Chesterton’s fence: don’t remove a fence until you know why it was put up).

Date the bug

A bug is a relationship between two things: this line and the code it gets wrong — date both.

git log --oneline -S'result.issues = validation.issues' -- <path>
git log --oneline -S'result.issues.append' -- <path>

The result went into the fix’s commit message:

2026-04-16 rebind written. result.issues had NO prior writers;
 assigning validation's list into it lost nothing.
2026-04-19 + vendor mismatch trail above it +3d
2026-04-23 + REPAIR delete note above it +7d
2026-04-24 + registry retry verdict above it +8d
2026-05-27 + self-billed reroute above it +41d

The rebind was correct when it was written. Over the next 41 days, four commits added writers above it, and none of them touched it. Nobody decided to discard those issues; the code predates the thing it breaks.

That gives one of three verdicts:

Fence: someone decided this, in a commit message, a spec’s non-goals or an ADR. Leave it.
Drift: the code predates what it is wrong about, or its author never touched the site. Fix it now, and put the dates in the commit message so a reviewer can see you didn’t remove a fence.
Silence: the history says nothing either way. Park it, and say plainly that nobody decided it.

This one was drift, so it got fixed the same night. It had stayed hidden because the only test that reached this code patched the whole function body out. The split made the body reachable, and the first end-to-end repair test found the bug.

The obvious fix was wrong

A drift verdict says nobody meant the bug; you still have to find out what depends on it. The one-word fix:

result.issues.extend(validation.issues)

would have broken an alias. A later step appends registry failures to result.issues, and those reach the invoice’s note only because the rebind made result.issues and validation.issues the same list, which the note builder reads:

result.issues = validation.issues # one list, two names
apply_registry_failure(result) # appends to result.issues ...
note = build_note(validation) # ... and the note sees it through the alias

With .extend(), every invoice that failed the business-registry check would have lost that warning from its note: one bug traded for a quieter one. The fix kept the rebind and put the earlier issues back after the later writers ran:

earlier_issues = list(result.issues)
result.issues = validation.issues
apply_registry_failure(result)
# ... the other writers ...
result.issues = earlier_issues + result.issues

The tests were written to catch both mistakes: reverting the fix fails the four tests that check the earlier issues survive, and the naive .extend() fails the one that checks the registry warning still reaches the note. The known-issues entry was deleted, and the expectedFailure markers became ordinary tests.

What silence looks like

That night, four other bugs were dated as drift and fixed. One wasn’t. Debug files were named from the input’s stem, so invoice.docx and invoice.pdf in one batch wrote the same file, and the later one won. That naming shipped on a pipeline that already read several formats, so the code didn’t predate the collision. No commit message or spec mentioned it either: stem is just the obvious way to name a sidecar file. That one stayed a known-issues entry, with the dating written into it.

The skill

The fence check is a Claude Code skill I had written down earlier that evening. The agent had it and didn’t run it until asked, so its memory now says to run it the moment it finds a bug, before choosing between a fix and a known-issues entry. Here it is verbatim. The examples and the docs/superpowers/ paths are from my repository; swap in your own spec directory.

---
name: checking-the-fence
description: Date a bug before fixing it — Chesterton's fence. Use when you have found code that looks wrong and are about to change it, when you are about to delete something that looks dead, or when another skill needs the fence check.
---

# Checking the Fence

A bug you found is a **fence** across a field. Before you take it down, find who put it there. Here that is written down twice — in `git log` and in `docs/superpowers/` — and is usually decisive in a minute.

Two verdicts. They are the whole skill:

- **Fence** — someone decided this. Leave it, and record an ADR if the next reader will reach for the same fix you just did.
- **Drift** — nobody decided it. The code predates the thing it is wrong about, or an author was never told which files the job touched. Fix it.

Fences here are real, not hypothetical: `docs/adr/0002` exists only to stop a reader from deleting emoji handling that has no inbound traffic and looks dead.

## Steps

Stop at the first step that returns a verdict.

### 1. Date the code

```
git log --oneline -S'<symbol>' --all -- <path>
git log -1 --format=%B <sha>
```

Commit messages here carry the rule the author meant to encode, and the rule is often the thing the bug breaks. `_PARSER_PRIORITY`'s own commit says *"process deterministic parsers before PDFs/images"* — while the table shipped the cheapest format last.

**Done when** you can name the commit and quote the rule.

### 2. Date what it is wrong about

A bug is a relationship: a table missing a format, a copy missing an entry, a switch missing a branch. Date the other side too.

Code cannot have decided about something that did not exist yet. `_PARSER_PRIORITY` (2026-04-27) predates isdoc (2026-06-01) by five weeks.

**Done when** you have both dates. Predates → **drift**. Stop.

### 3. Read the spec

```
grep -rli '<feature>' docs/superpowers/specs docs/superpowers/plans
```

Read the **Non-goals** first — that is where a deliberate gap is named. A spec that scopes the gap out is a **fence**. A spec whose own words the bug contradicts is **drift**: the ISDOC spec calls it *"a deterministic fast path before Claude"*.

**Done when** you have quoted the line that scopes it out, or established that no spec names it.

### 4. Read the blast pattern

```
git show --stat <sha>
```

Which files that commit touched, against which it needed to. An author who updated three of five consumers was not deciding about the other two — nothing told them. The isdoc and docx commits landed a day apart and touched different subsets.

**Done when** you can say whether the author addressed the site at all. Never addressed → **drift**.

### 5. Was it born wrong?

For a forked copy, compare it against its source **on the day it was copied**, not today.

```
git show <sha>:<source-path>
```

`coverage_report.py` shipped without `.json` on 2026-04-24, and the map it was copied from carried `.json` that same day. Born forked, then rotted twice more.

**Done when** you know whether the gap is drift-over-time or a bad copy. Either is **drift**.

## Applying the verdict

**Drift** — fix it, and say in the commit message what you found: the dates, and that nothing decided it. That is what stops a reviewer from wondering whether you removed a fence.

**Fence** — leave it. Offer an ADR when a future reader would reach for the same fix; that reader is usually the next architecture review, or you.

**Neither** — the history is silent and no spec mentions it. Say so plainly and let the user decide; silence is not consent, and it is not a fence either.

Save it as .claude/skills/checking-the-fence/SKILL.md in your repository, and tell the agent in CLAUDE.md or AGENTS.md to run it before it records a bug as a known issue.

Takeaway

A parked bug looks like a decision, so check whether anyone made one. If the code predates what it breaks, fix it now, and check the mechanism first: knowing nobody meant the bug doesn’t tell you what depends on it.

Next time your agent adds an expectedFailure or a known-issues entry, ask it for the two dates first.

Every test mocked the helper, and except Exception swallowed its TypeError on every PDF

Thu, 16 Jul 2026 23:42:00 +0200

For seven weeks, a document classifier never read a single PDF. This is the helper that was supposed to give the model some text (trimmed):

def _extract_excerpt(file_path: Path) -> str:
 """First-page text for PDFs, first ~8 kB for everything else."""
 if file_path.suffix.lower() == ".pdf":
 try:
 pages = extract_pdf(file_path)
 return (pages[0] if pages else "")[:EXCERPT_BYTES]
 except Exception:
 return ""
 ...

extract_pdf doesn’t return a list of pages. It returns a dataclass, so every call raised this:

TypeError: 'PDFExtraction' object is not subscriptable

The except caught it and returned "". The prompt then held the filename and an empty excerpt, and the model answered anyway: a document type and a confidence score, for every PDF it was asked about.

If a helper of yours has a catch-all except and your tests patch that helper, check whether anything ever runs it for real.

Setup

A web upload went through a small Python script that sorted the file into one of the document kinds declared for an accounting period (received invoices, bank statements, and so on). With two or more possible kinds, it built a prompt from the filename plus the start of the document and asked a language model, which answered with JSON like {"kind": "...", "confidence": 0.9}. The PDF text came from an in-house extract_pdf() built on pdfplumber. Tests used the standard library’s unittest and unittest.mock.

What went wrong

The helper was wrong from the day it was written. extract_pdf had returned a PDFExtraction dataclass with a pages list since its first commit, six weeks earlier, and its docstring said so. The if pages check didn’t catch the mistake: a dataclass without __len__ is always truthy, so the check passed and pages[0] raised.

The except was there so that a corrupt or encrypted upload wouldn’t break classification. It caught the TypeError just as readily and returned "", which is also the documented answer for an image upload with no text to read.

And every test patched the helper out. The tests that reached the model looked like this:

with patch("scripts.classify_upload.invoke_ai") as m, \
 patch("scripts.classify_upload._extract_excerpt", return_value="Invoice 2026-001 ..."):
 m.return_value = json.dumps({"kind": "received_invoices", "confidence": 0.92})
 result = classify_file(Path("/tmp/x.pdf"), ["issued_invoices", "received_invoices"], "claude")

They covered the prompt plumbing and the JSON parsing, and none of them ever called _extract_excerpt. The implementation plan that specified the helper had the same broken line, and it specified these tests, mocks included. The code, the plan, and the tests agreed with each other, and none of them had been checked against extract_pdf.

The model hid the rest. A filename like invoice-0423.pdf can be enough for a right guess, and a wrong guess came back with a confidence score like any other, so nothing in the output said the excerpt was empty.

Fix

The fix reads the first page, as the docstring promised:

extraction = extract_pdf(file_path)
first_page = extraction.pages[0].text if extraction.pages else ""
return first_page[:EXCERPT_BYTES]

The new test class calls the real function on a real PDF:

class RealPdfExcerptTests(unittest.TestCase):
 def setUp(self):
 self.tmp = tempfile.TemporaryDirectory()
 self.root = Path(self.tmp.name)

 def tearDown(self):
 self.tmp.cleanup()

 def test_a_pdf_yields_its_first_page_text(self):
 path = self.root / "invoice.pdf"
 _write_pdf(path, "INVOICE 2026-001 Supplier ACME")
 self.assertIn("INVOICE 2026-001", _extract_excerpt(path))

 def test_a_corrupt_pdf_yields_no_signal_rather_than_raising(self):
 path = self.root / "broken.pdf"
 path.write_bytes(b"%PDF-1.4\nnot actually a pdf at all")
 self.assertEqual(_extract_excerpt(path), "")

_write_pdf is an existing test helper that writes a one-page PDF with a text layer. Before committing, I ran the first test against the original code to confirm it fails there. The except stayed, because corrupt uploads are real. The second test now says what it’s for.

A day later, the same helper had a second route to the same result. It kept its own list of text suffixes, so Word, Excel, and XML e-invoice uploads fell through to "" and were classified by filename too. Two of those formats joined the pipeline a week after the list was written, and nobody updated it. The helper now asks the pipeline’s format registry instead.

Why

A mock replaces the code it patches with your assumptions about that code. When every test mocks a function, its tests check only those assumptions, and the function itself is never run. A catch-all except inside the function then removes the last signal you’d get in production: no traceback, no failed request, just a default value that looks legitimate.

A language model downstream makes this worse. A parser fed an empty string usually fails loudly, while a model returns its best guess in the usual format, confidence field included, whatever the input.

Takeaway

Mock the expensive or external dependency (here, the model) and let the function under test run on a real input. If it has a catch-all except, give it a test with an input that should succeed, so a bug can’t hide behind the except.

Grep your tests for patch( targets that point at your own module’s private helpers, and for each one, find a test that calls the helper unmocked.