Python on Lessons learned

A CVE cross-reference is not an advisory entry

Wed, 28 Oct 2026 00:00:00 +0000

A vulnerability index can have the correct set of CVE identifiers and still assign them to the wrong advisories. A regex that searches every paragraph also captures narrative references such as “related to CVE-…”. Extract identifiers from declaration blocks, and verify the resulting relationships before using the index to make patch or exposure claims.

The trap is easy to miss during deduplication: the referenced identifier may already have a legitimate entry elsewhere. Removing its false assignment changes the relationships without changing the global distinct-ID count.

How the wrong relationship gets created

Consider two synthetic advisories. Every identifier below is fictional:

Advisory A, component Reader, declares CVE-2099-10001.
Its impact paragraph mentions CVE-2099-10002 as a related issue.
Advisory B, component Writer, declares CVE-2099-10002.

An extractor that carries the current advisory and component into every regex match produces three rows. The false row associates Advisory A and Reader with the second identifier. Both the broken and corrected indexes contain two distinct identifiers.

This error can contaminate a patch matrix: an analyst might attach the wrong affected component or release boundary to a real vulnerability. It requires no malicious input. A crafted narrative mention could produce the same misassociation in a pipeline that ingests untrusted documents, but that is a possible abuse of the parser, not evidence of an exploited advisory feed.

Filter by document role before matching identifiers

For one fixed document layout, a practical repair is to collect complete paragraph text and process only blocks beginning with the declaration marker CVE-. Collecting the whole paragraph matters: identifiers can be split across inline elements, and indentation should not change classification.

Here is a runnable Python example using a deliberately small HTML schema. Save it as check_advisories.py and run python3 check_advisories.py:

import re
from html.parser import HTMLParser

ID = re.compile(r"\bCVE-\d{4}-\d{4,}\b")
FIXTURE = """
<article data-advisory="A" data-component="Reader">
 <p> <strong>CVE-2099-10001</strong> </p>
 <p>Related to <em>CVE-2099-10002</em>.</p>
</article>
<article data-advisory="B" data-component="Writer">
 <p>\n CVE-2099-<span>10002</span> </p>
</article>
"""

class Index(HTMLParser):
 def __init__(self, declarations_only):
 super().__init__()
 self.declarations_only = declarations_only
 self.context = None
 self.parts = None
 self.rows = set()

 def handle_starttag(self, tag, attrs):
 if tag == "article":
 attrs = dict(attrs)
 self.context = (
 attrs["data-advisory"], attrs["data-component"]
 )
 elif tag == "p":
 self.parts = []

 def handle_data(self, data):
 if self.parts is not None:
 self.parts.append(data)

 def handle_endtag(self, tag):
 if tag == "p" and self.parts is not None:
 text = "".join(self.parts).strip()
 if not self.declarations_only or text.startswith("CVE-"):
 for identifier in ID.findall(text):
 self.rows.add((*self.context, identifier))
 self.parts = None
 elif tag == "article":
 self.context = None

def extract(declarations_only):
 parser = Index(declarations_only)
 parser.feed(FIXTURE)
 parser.close()
 return parser.rows

broken = extract(False)
fixed = extract(True)
assert fixed == {
 ("A", "Reader", "CVE-2099-10001"),
 ("B", "Writer", "CVE-2099-10002"),
}
assert broken - fixed == {("A", "Reader", "CVE-2099-10002")}
assert {row[2] for row in broken} == {row[2] for row in fixed}
print(f"relationships: {len(broken)} -> {len(fixed)}; distinct IDs unchanged")

Expected output:

relationships: 3 -> 2; distinct IDs unchanged

A prefix guard has a document contract

This parser is a fixture, not a general advisory importer. A narrative paragraph could itself begin with a CVE identifier; another publisher might declare identifiers in a table or list. For production ingestion, select the publisher’s actual declaration element or structured field, record the source location, and reject or flag layouts the adapter does not recognize. The standard library’s HTMLParser documentation describes its event interface; it cannot infer a publisher’s semantic roles for you.

Keep regression fixtures for narrative references, whitespace, and inline markup. Review added and removed (advisory, component, CVE) tuples during a parser change, including declarations the new selector might accidentally discard.

Before trusting your next vulnerability-index refresh, check one known cross-reference against its owning advisory. A stable distinct-ID total cannot validate that assignment.

OpenPBS array resume: shard by stable task ID

Tue, 20 Oct 2026 00:00:00 +0000

A resumable OpenPBS array can assign the same task to different workers if each worker slices its own remaining-work list. The unsafe implementation was:

todo = [g for g in groups if not completed(g)]
todo = todo[i::n]

We caught this before the array ran and changed ownership to group_id % worker_count. The corrected workers later started at different times and restarted with fewer tasks remaining; the final audit accounted for every group.

The Claude Code, SSH, and OpenPBS setup explains where commands ran and how jobs were submitted. This failure concerns application-level partitioning, not PBS assigning one array index twice.

A pending list changes between starts

The workload was moving from a multi-GPU job to four independently scheduled one-GPU array elements. Each worker skipped groups with stored completion or rejection markers, then selected a shard.

Suppose the immutable task IDs are zero through five. Worker zero starts first and takes every second pending entry:

pending: 0 1 2 3 4 5
worker 0: 0 2 4

It finishes task zero before worker one starts. Worker one’s view is now:

pending: 1 2 3 4 5
worker 1: 2 4

Tasks two and four overlap, while one, three, and five have no owner in that pair of snapshots. Restarting workers creates the same problem. A completion marker can filter old work without stabilizing the positions of everything that remains.

Preserve ownership while filtering completion

The actual correction retained the immutable numeric group ID:

todo = [g for g in todo if g["group"] % n == i]

Equivalently, make the two decisions explicit:

owned = [g for g in groups if g["group"] % n == i]
todo = [g for g in owned if not completed(g)]

In this application, --shard I/N supplied i and n. The submitted script used #PBS -J 0-3 and passed --shard "$PBS_ARRAY_INDEX/4" to the project-specific driver. Each element requested one GPU. Smaller resource requests did not guarantee immediate starts; later queue snapshots still showed waiting elements.

Keep the shard count fixed for a running generation. Changing n changes ownership even with stable IDs. Also, the same shard launched twice still has overlapping ownership; see the separate concurrent-launch failure .

Test different snapshots, then audit the run

This small Python check reproduces the bad assignment and checks the correction:

all_ids = list(range(6))
first_pending = all_ids
later_pending = all_ids[1:]

bad0 = set(first_pending[0::2])
bad1 = set(later_pending[1::2])
assert bad0 & bad1 == {2, 4}

owned0 = {g for g in first_pending if g % 2 == 0}
owned1 = {g for g in later_pending if g % 2 == 1}
assert not owned0 & owned1
assert owned0 | owned1 == set(all_ids)
print("stable ownership checks passed")

The production evidence was broader than a queue disappearing: logs showed staggered starts, three workers restarting with smaller pending counts, and accepted/rejected outcomes covering the design. A final check reported:

missing (neither): []

That established outcome coverage. It was not a separate duplicate-output audit, and the logs did not establish why those workers restarted. Atomic replacement of a shared CSV also does not prove its concurrent snapshots were consistent.

Before your next resumable array, test ownership against two different completion snapshots. Every unfinished task should retain its owner regardless of when that owner starts.

Decode a MikroTik autosupout.rif yourself: 20 lines of Python found the out-of-memory kernel panic

Wed, 14 Oct 2026 18:17:00 +0200

A MikroTik router kept rebooting every 10 minutes to 2.6 hours, with almost no traffic. After each boot, the log said:

system,error,critical router was rebooted without proper shutdown, probably kernel failure
system,error,critical kernel failure in previous boot
system,error,critical out of memory condition was detected
system,error,critical Automatic supout.rif file generated due to service malfunction, please contact MikroTik support and supply the generated file

You don’t have to wait for support. autosupout.rif is base64 with the bits in the other order, wrapped around zlib, and about 20 lines of Python turn it into plain text: the kernel’s last crash dumps, the log, /proc and the full config. That’s where the cause was:

Out of memory and no killable processes...
Kernel panic - not syncing: System is deadlocked on memory

The setup

MikroTik hAP ac² (IPQ4019, 128 MiB RAM, 16 MiB flash)
RouterOS 6.49.5, then 7.23.8 (long-term) with the legacy wireless package; both crashed the same way
Python 3 on a Mac to decode the file

The file format

Copy the file off the router and look at it:

scp admin@192.168.0.1:/autosupout.rif .
file autosupout.rif
head -2 autosupout.rif

autosupout.rif: ASCII text, with CRLF line terminators
--BEGIN ROUTEROS SUPOUT SECTION
uQWZiV3ZAgHnt3X72tNOSb+7xXFYO5HTy0WSEgfKeG3rdsd643J2xjtT37s5kVHKSIbOWiUNJVs9

It looks like base64, and base64 -d turns it into noise. The alphabet is standard, but each group of four characters is read as a 24-bit little-endian number: the first character holds the lowest 6 bits, and the first output byte is the lowest 8 bits. Decoded, each section is its name, a NUL byte, and a zlib stream.

The decoder

import os, re, sys, zlib

ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

def decode(text):
 text = re.sub(r'[^A-Za-z0-9+/=]', '', text)
 out = bytearray()
 for i in range(0, len(text), 4):
 group = text[i:i + 4].rstrip('=')
 value = sum(ALPHABET.index(c) << (6 * j) for j, c in enumerate(group))
 out += value.to_bytes(3, 'little')[:len(group) * 6 // 8]
 return bytes(out)

src, outdir = sys.argv[1], sys.argv[2]
os.makedirs(outdir, exist_ok=True)
sections = re.findall(r'--BEGIN ROUTEROS SUPOUT SECTION\r?\n(.*?)--END ROUTEROS SUPOUT SECTION',
 open(src, encoding='ascii').read(), re.S)
for n, section in enumerate(sections):
 name, _, data = decode(section).partition(b'\0')
 text = zlib.decompressobj().decompress(data)
 safe = re.sub(r'[^\w.-]', '_', name.decode(errors='replace'))
 with open(os.path.join(outdir, f'{n:02d}_{safe}.txt'), 'wb') as f:
 f.write(text)
 print(f'{n:02d} {safe} {len(text)}')

python3 supout.py autosupout.rif supout/

The 6.49.5 file had 76 sections and the 7.23.8 one 87. The useful ones:

Section	What’s in it
`oops`	the kernel log of the last few crashes, each with its timestamp
`log`	the RouterOS log since boot
`resource`	`/system resource` plus `/proc/meminfo`
`.proc`	binary-ish dump of `/proc`, including `slabinfo`, `vmallocinfo`, `buddyinfo` (use `LC_ALL=C grep -a`)
`export`	the full configuration
`.debug`	per-service debug output

What it showed

On 7.23.8, every crash in oops ended the same way, after a process table in which every process had the same score:

[ 691.378048][ T142] [ 150] 0 150 1667 153 28672 0 -1000 parser
...
[ 691.378492][ T142] Out of memory and no killable processes...
[ 691.378509][ T142] Kernel panic - not syncing: System is deadlocked on memory

RouterOS runs every process with oom_score_adj -1000, so the OOM killer has nothing to kill and the kernel panics. Any out-of-memory, however brief, is a reboot. 6.49.5 got there by another route, with the kernel set to panic on any OOM: Kernel panic - not syncing: Out of memory: compulsory panic_on_oom is enabled.

The processes weren’t the problem, at about 0.5 MiB each. resource, written a minute after boot, was:

MemTotal: 118248 kB
MemFree: 7368 kB
SUnreclaim: 52756 kB

One minute into the boot, the kernel already held 52 MiB of unreclaimable memory. The caches in slabinfo add up to only about 10 MiB, so the rest is allocated outside them. In two of the panics, other CPUs were inside the Wi-Fi driver (ath_poll, __w_qos_decrypt_mpdu in the wlan and ath modules), which makes it the main suspect, but turning one radio off didn’t settle it.

The same reading works on any embedded Linux box: check oom_score_adj in the OOM dump, then SUnreclaim against what the process table and slabinfo explain.

Read the timestamps with care

The router has no battery-backed clock. After a crash, it restores an old time, and NTP takes minutes to catch up, so log and supout timestamps right after a reboot were about 7 minutes early. Five “crashes” stamped within two seconds of each other were most likely one boot loop.

The supout itself was written a minute after the crash. The e-mail service tried to send the crash notice before PPPoE and DNS were up, died, and that service failure is what produced the file (.debug shows error: DNS resolve failed [code: 21] and then died).

Before you send it to anyone

The decoded sections held, in plain text, the PPPoE password, the WireGuard private key, a TSIG key, and a bearer token from a dynamic DNS script. Rotate those before you hand the file to MikroTik support or attach it to a forum post.

On a router this tight, delete the file after you copy it. The root of the file system is in RAM, and a 480 KiB supout is RAM the kernel needed:

/file remove [find name~"autosupout"]

Takeaway

A supout.rif holds the kernel log, /proc and your config, and you can read it in a minute. Next time your MikroTik writes one, decode it and read oops before you open a support ticket.

macOS SMB: OSError: [Errno 9] Bad file descriptor after a silent reconnect

Tue, 29 Sep 2026 12:15:00 +0200

A Python script that had encrypted backup archives on a NAS for months started failing on every run:

Traceback (most recent call last):
 File "/Volumes/backup/./encrypt-one.py", line 74, in <module>
 main()
 File "/Volumes/backup/./encrypt-one.py", line 38, in main
 with open(f, "rb") as src:
OSError: [Errno 9] Bad file descriptor

Then the bash script that called it stopped too. Not intermittently: start the job and something fails sooner or later.

If you run long jobs from an SMB share on macOS and get EBADF on files that are plainly there, the cause is probably a session reconnect you never saw, and the kernel log is the only place that records it.

The setup

macOS 27 dropped AFP, so an old NAS that speaks only SMB 2.1 got mounted over SMB instead. A bash script on the share runs encrypt-one.py on a few archives in parallel with xargs -P. Each run streams a 600 MB archive from the share through openssl enc and writes the result back to the same share.

The dead end

The traceback points at open(), so the first move was to open and read that archive by hand. It worked, and so did a read under xargs -P 3. A file that opens fine and a job that fails anyway tells you the file isn’t the problem. Something happens to the handle while it’s open.

The kernel log

smbutil gave the first hint:

$ smbutil statshares -m /Volumes/backup
...
SMB_VERSION SMB_2.1
SESSION_RECONNECT_TIME 2026-09-29 12:01:17
SESSION_RECONNECT_COUNT 4

Four reconnects; the last one was inside the failed run. The kernel log explains what each one does (timestamps shortened, process column removed):

/usr/bin/log show --last 1h --style compact \
 --predicate 'process == "kernel" AND eventMessage CONTAINS[c] "smb"'

11:58:15.502 (smbfs) smb_iod_sendall: id 0 ts > iod_lastrecv && ts > sr_timesent.
11:58:15.502 (smbfs) smb_iod_reconnect: id 0 Starting reconnect with nas.local
11:58:15.502 (smbfs) smb2_smb_parse_change_notify: smb_rq_reply failed 60
11:58:16.523 (smbfs) smb_iod_reconnect: id 0 The negotiate succeeded to nas.local
11:58:17.065 (smbfs) smb2fs_reconnect_dur_handle: Missing durable handle on <archive.tar.bz2.enc.tmp>
11:58:17.173 (smbfs) smb2fs_reconnect_dur_handle: Missing durable handle on <backup.sh>
11:58:17.186 (smbfs) smb2fs_reconnect_dur_handle: Missing durable handle on <archive.tar.bz2>
11:58:17.205 (smbfs) smb_iod_reconnect: id 0: Reconnect completed successfully.
11:58:17.212 (smbfs) smb2_smb_read_write_async: smb_rq_reply failed 9
11:58:17.212 (smbfs) smbfs_do_strategy: archive.tar.bz2: READ failed with an error of 9
11:58:17.283 (smbfs) smbfs_do_strategy: archive.tar.bz2.enc.tmp: WRITE failed with an error of 9

The NAS stopped answering under sustained I/O (error 60 is ETIMEDOUT). macOS dropped the session and reconnected, and that part worked. Then it tried to reattach each open file; the server had nothing to reattach them to, and every read and write on those files failed with error 9, EBADF.

Look at the middle line: backup.sh is on the list. Bash reads a script from disk as it runs, so the script file is open the whole time and goes bad with everything else. That’s why the shell stopped as well.

Why

A durable handle is how an SMB 2.x client keeps an open file across a dropped connection: the server holds the open for a while, and the client reclaims it after reconnecting. This server granted none. So “Reconnect completed successfully” is true for the session and useless for every file that was open at the time. Nothing reaches the application except EBADF on the next read or write.

What fixed it

Stop running long I/O jobs from the share. The pipeline moved to a USB drive, and only finished files go to the NAS. That ended the failures.

If you can’t move off the share, two workarounds got the job past the failing step:

Make bash read the whole script before it starts, so it never goes back to the share for the next line:
```
#!/usr/bin/env bash
{
set -euo pipefail
# ... the whole script ...
exit
}
```
Bash parses the entire { } block before running any of it, and the exit stops it from reading past the closing brace.
Make the I/O survive a reconnect: process the file in segments (64 MiB worked), open, write and fsync each segment separately, and on EBADF, EIO or EAGAIN reopen and retry only that segment. With AES-CBC you can chain the segments by using the last ciphertext block as the next IV, and the output is byte-identical to one openssl enc pass. In the test run on the share, no reconnect happened at all, probably because the per-segment fsync slowed the writes enough for the NAS to keep up.

If the server is a Samba box you control, check whether it grants durable handles: the durable handles option only takes effect with kernel oplocks = no, kernel share modes = no and posix locking = no. Turning off SMB signing on the client (signing_required=no in /etc/nsmb.conf) might also take enough load off a weak NAS CPU to prevent the stalls. Neither was tried here.

Takeaway

On macOS, an SMB reconnect to a server without durable handles invalidates every open file on the share, including a running shell script, and the only trace is Missing durable handle in the kernel log.

If you’ve seen Errno 9 on an SMB mount, run smbutil statshares -a and look at SESSION_RECONNECT_COUNT before you debug your code.

A second launch of a resumable worker: 'OSError: [Errno 39] Directory not empty'

Sat, 26 Sep 2026 00:00:00 +0000

A resumable remote dataset driver failed while cleaning an unfinished task directory:

OSError: [Errno 39] Directory not empty

Another worker was still writing there. The second launch treated an unfinished task as safe to restart, even though the first process already owned it. Checking the surviving worker and validating its affected output recovered that work unit; no locking fix was installed during the session.

This was an early detached-run failure in the Claude Code, SSH, and OpenPBS workflow . Scheduler-managed execution later improved job tracking, but an application still needs to prevent conflicting owners.

The waiting wrapper had not disappeared

The first launch created a detached wrapper that waited for a pilot, then became the production driver. Its structure was:

until grep -q "finished" pilot.log; do sleep 30; done
exec python -u run_dataset.py dataset >> production.log 2>&1

Before the transition, a search for the production process would not find that driver. After exec, a search for the wrapper’s old command name would no longer identify it. Neither snapshot alone described the complete lifecycle.

A later check did not establish that the waiting launch was gone. Another detached production command started. The resulting log contained two identical pending-work counts followed by a traceback from shutil.rmtree(gdir). Both drivers had selected the same unfinished task, and one attempted cleanup while the other’s solver wrote files.

The resume logic used completion markers to skip finished groups. It offered no exclusive claim on an unfinished directory.

Recovery required checking the survivor

The follow-up inspected running processes, restored the affected task’s configuration, and checked the surviving computation. Its original log file had been lost during cleanup. A later driver result reported that the task was accepted.

A broad grep for success also matched text in the traceback. Tightening the check to a result-line pattern distinguished the actual accepted task from an error message that happened to contain matching characters.

That evidence supports recovery of the affected work unit. It does not prove that no intermediate data was lost or that the session did not add a lock. Eventual production completion checked in a later session also cannot establish that overlapping launches became safe.

Claim ownership before destructive resume

For a single-host driver using a filesystem with working advisory locks, a whole-run lock is one possible improvement. On a system with util-linux flock, run from the project directory:

flock -n .production.lock python -u run_dataset.py dataset

Every launcher must use the same stable lock path. Do not delete and recreate the lock file while work is running. The payload named here is the application’s driver, not a standard tool.

This guard is a new recommendation. A small local check can verify contention without running a dataset:

import fcntl
import tempfile

with tempfile.TemporaryDirectory() as directory:
 with open(directory + "/worker.lock", "w") as first:
 fcntl.flock(first, fcntl.LOCK_EX | fcntl.LOCK_NB)
 with open(directory + "/worker.lock", "w") as second:
 try:
 fcntl.flock(second, fcntl.LOCK_EX | fcntl.LOCK_NB)
 except BlockingIOError:
 print("second owner rejected")
 else:
 raise AssertionError("overlapping owner accepted")

Cluster filesystems and multi-node workers need an ownership mechanism tested under their actual locking semantics. Per-task claims can allow parallelism that a whole-run lock forbids. Stable PBS shard assignment solves a different problem: preserving each worker’s task set across starts and restarts. It cannot exclude two copies of the same shard.

Before retrying unfinished work, establish whether a live process already owns it. A missing completion marker only says that completion has not been recorded.

Zeep returns an empty list: a repeated xs:sequence lands in _value_1

Sat, 12 Sep 2026 23:59:00 +0200

A SOAP client listed received messages and got an empty list on every call, even though the mailbox had messages. No exception or error status occurred, and the unit tests passed. This was the line that read the response:

records = getattr(response, "dmRecords", None)
page = (records and getattr(records, "dmRecord", None)) or []

The XML on the wire had <dmRecords><dmRecord>…</dmRecord><dmRecord>…</dmRecord></dmRecords>, but zeep never created a dmRecord attribute. It put the records in dmRecords._value_1.

If zeep gives you None or an empty list for a field you can see in the raw response, check whether the schema repeats the sequence rather than the element.

Setup

An Odoo 18 addon pulls messages from the Czech Data Box system (ISDS), a government SOAP service with published WSDL and XSD files, and talks to it through zeep. I reproduced the behavior below with zeep 4.3.3 on Python 3.13. The list type in the ISDS schema looks like this:

<xs:complexType name="tRecordsArray">
 <xs:sequence minOccurs="0" maxOccurs="unbounded">
 <xs:element name="dmRecord" type="tns:tRecord" />
 </xs:sequence>
</xs:complexType>

maxOccurs="unbounded" sits on the xs:sequence, where you would usually expect it on the xs:element.

What went wrong

zeep maps the two forms differently. Here is a minimal schema with both, parsed with zeep’s own types:

<xs:complexType name="tRepeatedSequence">
 <xs:sequence minOccurs="0" maxOccurs="unbounded">
 <xs:element name="record" type="tns:tRecord"/>
 </xs:sequence>
</xs:complexType>
<xs:complexType name="tRepeatedElement">
 <xs:sequence>
 <xs:element name="record" type="tns:tRecord" minOccurs="0" maxOccurs="unbounded"/>
 </xs:sequence>
</xs:complexType>

Feed both the same two <record> children, then call serialize_object(..., dict):

tRepeatedSequence: {'_value_1': [{'record': {'id': '1'}}, {'record': {'id': '2'}}]}
tRepeatedElement: {'record': [{'id': '1'}, {'id': '2'}]}

With a repeated element, you get obj.record as a list. With a repeated sequence, each repetition becomes an item. The sequence has no name, so zeep calls the list _value_1, and each item is a dict keyed by the element names inside the sequence. A sequence with two elements would give you {"a": …, "b": …} per item.

The zeep docs explain _value_1 only under xs:choice and xs:any. Nothing there tells you that a plain repeated sequence ends up there too.

Plain attribute access would have failed loudly on the first real call:

AttributeError: tRepeatedSequence instance has no attribute 'record'

The getattr(..., None) or [] was written so that an empty mailbox wouldn’t crash. It also handled the case where the attribute doesn’t exist at all, and turned it into “no messages”.

The tests didn’t catch it because they replaced the client’s methods with mock.patch. They never sent a SOAP response through zeep, so they never saw how zeep shapes it.

Fix

Read _value_1 and unwrap each item:

records = getattr(response, "dmRecords", None)
page = [item["dmRecord"] for item in (getattr(records, "_value_1", None) or [])]

An empty <dmRecords/> still gives _value_1 == [], so the empty-mailbox case works.

To see the shape before you write the parsing code, dump the WSDL with zeep’s CLI:

python -m zeep dm_info.wsdl

ns0:tRecordsArray([dmRecord: ns0:tRecord])

The square brackets give it away. A repeated sequence prints with brackets around the field list. A repeated element, like tRepeatedElement above, prints with [] after the type instead: ns0:tRepeatedElement(record: ns0:tRecord[]).

Test the parsing with real XML

The test that covers this now runs canned SOAP XML through the real zeep client, with only the HTTP call patched:

def response(xml_body):
 result = requests.Response()
 result.status_code = 200
 result.headers["Content-Type"] = "text/xml; charset=utf-8"
 result._content = soap_envelope(xml_body)
 return result

with mock.patch.object(requests.Session, "post", side_effect=[response(listing)]):
 envelopes = client.list_received_messages(since, until)

self.assertEqual([e["id"] for e in envelopes], ["100001", "100002"])

Zeep still loads the WSDL, parses the envelope, and builds its objects, so a mapping mistake like this fails in the test suite with no network. Write the XML from the schema or from a captured response.

The same commit fixed a second or default with the same problem: return response.dmSignature or b"" would have stored a message without its signed copy. It now raises when the signature is missing.

Takeaway

zeep maps a repeated xs:sequence to _value_1, a list of dicts keyed by the inner element names. A getattr(..., None) or [] on top turns that into a silent empty result. Let missing attributes raise, and test SOAP parsing with real XML through zeep.

Search your zeep code for getattr( with a None default followed by or [], and check each field against the python -m zeep dump of your WSDL.

A dedup lookup that fails reads as "not found": urllib.parse.quote() is not query escaping

Thu, 23 Jul 2026 01:05:00 +0200

An OCR’d invoice number with an apostrophe in it, say O'BRIEN-1, went into a duplicate check like this one (trimmed):

flt = urllib.parse.quote(f"firma='{firma}' and cisDosle='{cis_dosle}'", safe="")
try:
 r = self.get(f"faktura-prijata/({flt}).json?detail=custom:id,kod&limit=1")
 entries = r.get("winstrom", {}).get("faktura-prijata", [])
 return entries[0] if entries else None
except FlexiBeeError:
 return None

The server rejected the filter with HTTP 400 (“Špatný formát WQL”, bad WQL format). The except turned that into None, the caller read None as “no such invoice yet”, and pushed it again: a second copy of the invoice in the books, and no error anywhere.

If your client puts a filter expression into a URL and wraps it in quote(), the apostrophe problem is probably yours too.

Setup

The API is the REST interface of ABRA Flexi, a Czech ERP, called from Python’s standard library. Its filters are a small query language (WQL) carried in the URL path: evidence/(field='value').json. The ?filter= query-string form is silently ignored, so the path it is used instead.

The values come from OCR and a language model. The invoice number was normalized for dashes and whitespace only, so an apostrophe made it into the filter intact.

What went wrong

quote() did its job. It turned the expression into something safe to put in a URL:

>>> urllib.parse.quote("cisDosle='O'BRIEN-1'", safe="")
'cisDosle%3D%27O%27BRIEN-1%27'

The server URL-decodes the path before it parses the filter, so the parser gets back cisDosle='O'BRIEN-1'. The literal ends after O; the rest is garbage, and the query fails.

Percent-encoding protects the URL, and the server undoes it before the filter language ever sees the value.

Every client in the codebase built its filters this way, including the MCP server. Two call sites even disagreed on safe= ("'=" versus ""), which made no difference, since the server decodes either form back to the same bytes.

The failure stayed hidden because of the except. A lookup that can’t answer reported “not found”, and for a duplicate check, “not found” means “write it”.

Fix

First, find out what the grammar’s escape actually is. No client library for this API escapes it (the PHP one interpolates values raw), so there was nothing to copy. A wrong guess would cause the very duplicate the lookup is there to prevent, so it was settled by asking the server, read-only, in a sandbox company, with values that can’t match any record:

cisDosle='ZZZTEST' balanced -> 200
cisDosle='ZZZ'TEST' raw (the bug) -> 400 Špatný formát WQL … pozice 15
cisDosle='ZZZ\'TEST' backslash -> 200
cisDosle='ZZZ''TEST' SQL-style doubling -> 400
cisDosle='A\\B' doubled backslash -> 400
cisDosle='A\B' lone backslash -> 400

So this grammar escapes an apostrophe with a backslash, the way C does, not by doubling it, the way SQL and OData do. A backslash escapes nothing else, which means a literal backslash can’t be written at all. That’s acceptable for invoice numbers, codes, company IDs, and dates, which never carry one.

Second, build every filter in one module, so the escape exists exactly once:

import urllib.parse

def escape_literal(value: object) -> str:
 return str(value).replace("'", "\\'")

def eq(field: str, value: object) -> str:
 return f"{field}='{escape_literal(value)}'"

def and_(*clauses: str) -> str:
 return " and ".join(c for c in clauses if c)

def evidence_path(evidence: str, expr: str) -> str:
 return f"{evidence}/({urllib.parse.quote(expr, safe='')}).json"

path = evidence_path("faktura-prijata", and_(eq("firma", firma), eq("cisDosle", cis_dosle)))

escape_literal handles the query language and quote handles the URL, each in one place. Against the sandbox, the real client with this filter now gets a 200 for an invoice number with an apostrophe.

Before the fix, the gap was parked as an expectedFailure test. It checks that the quotes in the generated filter balance once either escape form is removed, so it would pass whichever escape the probes picked, and it flipped to an unexpected pass the moment the fix landed.

Why

Any value that goes into a query language needs that language’s escape, and then the transport’s. Percent-encoding is the transport’s. OData $filter and most search APIs set the same trap with their own escape rules, so probe the server instead of guessing.

The other half is the except. The finder in this codebase still maps an error to None, and that remains on its list of known issues. If you write a duplicate check, give it three outcomes: found, not found, and couldn’t tell, and make the caller stop on the third.

An earlier bug in the same codebase took the other route to the same end: a bank-movement duplicate check used LIKE, which this API doesn’t support. The server didn’t reject it; it returned an empty list, so every movement was pushed a second time. The fix was the API’s BEGINS operator, plus a one-off script to delete the copies.

Takeaway

Escape a value for the query language first and for the URL second; quote() only does the second. A duplicate check that turns errors into “not found” creates a duplicate every time it breaks.

Search your code for quote(f" and check each hit for a quoted literal built from user or OCR input.

HTTP 500 that still commits: never retry an ambiguous write

Wed, 22 Jul 2026 22:29:00 +0200

We ran the client’s real retry loop against a fake server and counted the writes on the wire, before and after the fix:

before: push PUT 500 -> 200 writes_on_wire=2 -> reported ok
after: push PUT 500 -> 200 writes_on_wire=1 -> FlexiBeeError(maybe_committed=True)

The fake server played what the real API does now and then: it answers a PUT with 500 Internal Server Error after it has already saved the invoice. The client retried; the second PUT came back 200 OK, and the batch reported success. On a real API, that invoice is on the books twice. If your API can return a 5xx after committing, a retry that succeeds is the worst outcome you can get: you end up with a duplicate and nothing in the logs.

Setup

The API was the REST interface of ABRA Flexi, an ERP, called from Python through urllib and http.client. It returns 500 now and then after the write has gone through. That much was known; one client already looked the record up by document number after a failure.

Five hand-written clients for that API were merged into one transport layer with a shared RetryPolicy. An AI coding agent did the merge, two commits after writing an ADR that warned about duplicate writes.

What went wrong

The policy didn’t know the method

RetryPolicy decided from the response alone, so it retried a 500 on a PUT the same way it retried one on a GET. The test for that case fed in three 500s and checked only the flag on the final error. It pinned the number of retries and never asked whether a write should be retried in the first place.

The safety nets sat outside the loop

The first fix made the method a required argument and turned off retries of ambiguous writes by default. Two callers opted back in, because each had a safety net. In outline:

# MCP server: an idempotency key per tool call
def write_tool(key, payload):
 if idem_seen(key): # checked once, before the first attempt
 return cached_result(key)
 result = request("PUT", url, payload) # retries happen in here
 idem_record(key, result) # recorded only after success
 return result

# webhook worker: look the record up when the write fails
def upsert(doc):
 try:
 return request("PUT", url, doc) # retries happen in here
 except ApiError:
 if find_by_number(doc.number):
 return "committed-via-verify"
 raise

A review five days later, by a different model, caught it: “_idem_seen runs before this function, and _idem_record runs only after success, so neither prevents this retry from duplicating a write.” Likewise, the GET check runs only when request() raises, and a 500 followed by a successful retry never raises. Both checks were correct; the retry loop inside request() simply never consulted them.

Fix

Writes are never retried on an ambiguous outcome, and no caller can opt back in. The error goes up to the caller and says the write may have committed. The policy, trimmed to the part that matters:

WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}

class RetryPolicy:
 def __init__(self, method: str): # required: a default would let the next client forget it
 self._is_write = method.upper() in WRITE_METHODS

 def next_retry(self, status: int, body: str):
 if is_rate_limited(status, body): # the server refused: nothing committed
 return self._backoff("rate-limit")
 if 500 <= status < 600 and not self._is_write:
 return self._backoff("transient-5xx")
 return None # a 5xx on a write: raise, maybe_committed=True

 def next_retry_for_network_error(self):
 # no response at all: the server may have processed it before the connection died
 return None if self._is_write else self._backoff("network")

The new test counts requests, because a retry that succeeds looks exactly like a first attempt that did:

def test_500_on_a_write_is_never_retried(self):
 fake = FakeConn([FakeResponse(500, b"boom"), FakeResponse(200, b"{}")])
 with patch.object(client, "_get_conn", return_value=fake):
 with self.assertRaises(ApiError) as cm:
 client.put("invoice.json", {"invoice": [{}]})
 self.assertEqual(len(fake.requests_made), 1)
 self.assertTrue(cm.exception.maybe_committed)

Reads still retry, and they make up most of the traffic in a batch. A write that comes back maybe_committed is checked by whoever started it, before anything resends it.

Why

The policy asked, “may I retry a 5xx?” For a write, the question is “may I resend something whose outcome is unknown?” Two outcomes leave a write ambiguous: a 5xx, and no response at all. A 429, or a 400 rejecting a stale user role, is the server refusing the write; nothing committed, so a retry can’t duplicate anything.

The HTTP method doesn’t settle it either. RFC 9110 calls PUT idempotent, and urllib3’s default allowed_methods includes PUT, so a status_forcelist=[500, 502, 503] resends it. This API’s PUT to a collection creates a record each time.

If you do want writes retried automatically, the check has to run between the failure and the resend: before every attempt, look the record up or send a key the server itself deduplicates on. A check before the first call or after the last one can’t see the retries in between.

Takeaway

A 500 on a write means you don’t know what happened, so pass that up the stack instead of resending. Test it by counting requests on the wire, since the final result looks the same either way.

Find the retry loop in your own HTTP client and check whether it knows the method it is retrying.

Date the bug before you park it: fence, drift or silence

Fri, 17 Jul 2026 01:13:00 +0200

My coding agent was splitting a 738-line Python function into steps and found a real bug along the way:

result.issues = validation.issues

That line rebinds the list instead of extending it, so it dropped every issue recorded before validation, including the note that repair mode had just deleted a record in the accounting system. The push still succeeded, and the operator’s report said nothing about the delete.

The agent didn’t fix it. It wrote a known-issues entry and added two @unittest.expectedFailure tests, because the handoff for the refactor said: “characterize, then collapse, then fix separately”. It was the wrong call, and two git log -S runs showed it.

The setup

Claude Code working in a Python repository of an accounting pipeline, tested with unittest;
A long-lived function where several steps append to result.issues, the operator’s per-file report;
A refactor rule that a split must not change behavior.

What parking costs

An expectedFailure marker looks like a decision. The next reader sees a documented, tested, known gap and assumes someone weighed it and chose to keep it. If nobody did, the marker turns an accident into policy and keeps a live bug in the tree behind it.

So before you park a bug, find out whether anyone ever decided it. I asked the agent to run the fence check (Chesterton’s fence: don’t remove a fence until you know why it was put up).

Date the bug

A bug is a relationship between two things: this line and the code it gets wrong — date both.

git log --oneline -S'result.issues = validation.issues' -- <path>
git log --oneline -S'result.issues.append' -- <path>

The result went into the fix’s commit message:

2026-04-16 rebind written. result.issues had NO prior writers;
 assigning validation's list into it lost nothing.
2026-04-19 + vendor mismatch trail above it +3d
2026-04-23 + REPAIR delete note above it +7d
2026-04-24 + registry retry verdict above it +8d
2026-05-27 + self-billed reroute above it +41d

The rebind was correct when it was written. Over the next 41 days, four commits added writers above it, and none of them touched it. Nobody decided to discard those issues; the code predates the thing it breaks.

That gives one of three verdicts:

Fence: someone decided this, in a commit message, a spec’s non-goals or an ADR. Leave it.
Drift: the code predates what it is wrong about, or its author never touched the site. Fix it now, and put the dates in the commit message so a reviewer can see you didn’t remove a fence.
Silence: the history says nothing either way. Park it, and say plainly that nobody decided it.

This one was drift, so it got fixed the same night. It had stayed hidden because the only test that reached this code patched the whole function body out. The split made the body reachable, and the first end-to-end repair test found the bug.

The obvious fix was wrong

A drift verdict says nobody meant the bug; you still have to find out what depends on it. The one-word fix:

result.issues.extend(validation.issues)

would have broken an alias. A later step appends registry failures to result.issues, and those reach the invoice’s note only because the rebind made result.issues and validation.issues the same list, which the note builder reads:

result.issues = validation.issues # one list, two names
apply_registry_failure(result) # appends to result.issues ...
note = build_note(validation) # ... and the note sees it through the alias

With .extend(), every invoice that failed the business-registry check would have lost that warning from its note: one bug traded for a quieter one. The fix kept the rebind and put the earlier issues back after the later writers ran:

earlier_issues = list(result.issues)
result.issues = validation.issues
apply_registry_failure(result)
# ... the other writers ...
result.issues = earlier_issues + result.issues

The tests were written to catch both mistakes: reverting the fix fails the four tests that check the earlier issues survive, and the naive .extend() fails the one that checks the registry warning still reaches the note. The known-issues entry was deleted, and the expectedFailure markers became ordinary tests.

What silence looks like

That night, four other bugs were dated as drift and fixed. One wasn’t. Debug files were named from the input’s stem, so invoice.docx and invoice.pdf in one batch wrote the same file, and the later one won. That naming shipped on a pipeline that already read several formats, so the code didn’t predate the collision. No commit message or spec mentioned it either: stem is just the obvious way to name a sidecar file. That one stayed a known-issues entry, with the dating written into it.

The skill

The fence check is a Claude Code skill I had written down earlier that evening. The agent had it and didn’t run it until asked, so its memory now says to run it the moment it finds a bug, before choosing between a fix and a known-issues entry. Here it is verbatim. The examples and the docs/superpowers/ paths are from my repository; swap in your own spec directory.

---
name: checking-the-fence
description: Date a bug before fixing it — Chesterton's fence. Use when you have found code that looks wrong and are about to change it, when you are about to delete something that looks dead, or when another skill needs the fence check.
---

# Checking the Fence

A bug you found is a **fence** across a field. Before you take it down, find who put it there. Here that is written down twice — in `git log` and in `docs/superpowers/` — and is usually decisive in a minute.

Two verdicts. They are the whole skill:

- **Fence** — someone decided this. Leave it, and record an ADR if the next reader will reach for the same fix you just did.
- **Drift** — nobody decided it. The code predates the thing it is wrong about, or an author was never told which files the job touched. Fix it.

Fences here are real, not hypothetical: `docs/adr/0002` exists only to stop a reader from deleting emoji handling that has no inbound traffic and looks dead.

## Steps

Stop at the first step that returns a verdict.

### 1. Date the code

```
git log --oneline -S'<symbol>' --all -- <path>
git log -1 --format=%B <sha>
```

Commit messages here carry the rule the author meant to encode, and the rule is often the thing the bug breaks. `_PARSER_PRIORITY`'s own commit says *"process deterministic parsers before PDFs/images"* — while the table shipped the cheapest format last.

**Done when** you can name the commit and quote the rule.

### 2. Date what it is wrong about

A bug is a relationship: a table missing a format, a copy missing an entry, a switch missing a branch. Date the other side too.

Code cannot have decided about something that did not exist yet. `_PARSER_PRIORITY` (2026-04-27) predates isdoc (2026-06-01) by five weeks.

**Done when** you have both dates. Predates → **drift**. Stop.

### 3. Read the spec

```
grep -rli '<feature>' docs/superpowers/specs docs/superpowers/plans
```

Read the **Non-goals** first — that is where a deliberate gap is named. A spec that scopes the gap out is a **fence**. A spec whose own words the bug contradicts is **drift**: the ISDOC spec calls it *"a deterministic fast path before Claude"*.

**Done when** you have quoted the line that scopes it out, or established that no spec names it.

### 4. Read the blast pattern

```
git show --stat <sha>
```

Which files that commit touched, against which it needed to. An author who updated three of five consumers was not deciding about the other two — nothing told them. The isdoc and docx commits landed a day apart and touched different subsets.

**Done when** you can say whether the author addressed the site at all. Never addressed → **drift**.

### 5. Was it born wrong?

For a forked copy, compare it against its source **on the day it was copied**, not today.

```
git show <sha>:<source-path>
```

`coverage_report.py` shipped without `.json` on 2026-04-24, and the map it was copied from carried `.json` that same day. Born forked, then rotted twice more.

**Done when** you know whether the gap is drift-over-time or a bad copy. Either is **drift**.

## Applying the verdict

**Drift** — fix it, and say in the commit message what you found: the dates, and that nothing decided it. That is what stops a reviewer from wondering whether you removed a fence.

**Fence** — leave it. Offer an ADR when a future reader would reach for the same fix; that reader is usually the next architecture review, or you.

**Neither** — the history is silent and no spec mentions it. Say so plainly and let the user decide; silence is not consent, and it is not a fence either.

Save it as .claude/skills/checking-the-fence/SKILL.md in your repository, and tell the agent in CLAUDE.md or AGENTS.md to run it before it records a bug as a known issue.

Takeaway

A parked bug looks like a decision, so check whether anyone made one. If the code predates what it breaks, fix it now, and check the mechanism first: knowing nobody meant the bug doesn’t tell you what depends on it.

Next time your agent adds an expectedFailure or a known-issues entry, ask it for the two dates first.

Every test mocked the helper, and except Exception swallowed its TypeError on every PDF

Thu, 16 Jul 2026 23:42:00 +0200

For seven weeks, a document classifier never read a single PDF. This is the helper that was supposed to give the model some text (trimmed):

def _extract_excerpt(file_path: Path) -> str:
 """First-page text for PDFs, first ~8 kB for everything else."""
 if file_path.suffix.lower() == ".pdf":
 try:
 pages = extract_pdf(file_path)
 return (pages[0] if pages else "")[:EXCERPT_BYTES]
 except Exception:
 return ""
 ...

extract_pdf doesn’t return a list of pages. It returns a dataclass, so every call raised this:

TypeError: 'PDFExtraction' object is not subscriptable

The except caught it and returned "". The prompt then held the filename and an empty excerpt, and the model answered anyway: a document type and a confidence score, for every PDF it was asked about.

If a helper of yours has a catch-all except and your tests patch that helper, check whether anything ever runs it for real.

Setup

A web upload went through a small Python script that sorted the file into one of the document kinds declared for an accounting period (received invoices, bank statements, and so on). With two or more possible kinds, it built a prompt from the filename plus the start of the document and asked a language model, which answered with JSON like {"kind": "...", "confidence": 0.9}. The PDF text came from an in-house extract_pdf() built on pdfplumber. Tests used the standard library’s unittest and unittest.mock.

What went wrong

The helper was wrong from the day it was written. extract_pdf had returned a PDFExtraction dataclass with a pages list since its first commit, six weeks earlier, and its docstring said so. The if pages check didn’t catch the mistake: a dataclass without __len__ is always truthy, so the check passed and pages[0] raised.

The except was there so that a corrupt or encrypted upload wouldn’t break classification. It caught the TypeError just as readily and returned "", which is also the documented answer for an image upload with no text to read.

And every test patched the helper out. The tests that reached the model looked like this:

with patch("scripts.classify_upload.invoke_ai") as m, \
 patch("scripts.classify_upload._extract_excerpt", return_value="Invoice 2026-001 ..."):
 m.return_value = json.dumps({"kind": "received_invoices", "confidence": 0.92})
 result = classify_file(Path("/tmp/x.pdf"), ["issued_invoices", "received_invoices"], "claude")

They covered the prompt plumbing and the JSON parsing, and none of them ever called _extract_excerpt. The implementation plan that specified the helper had the same broken line, and it specified these tests, mocks included. The code, the plan, and the tests agreed with each other, and none of them had been checked against extract_pdf.

The model hid the rest. A filename like invoice-0423.pdf can be enough for a right guess, and a wrong guess came back with a confidence score like any other, so nothing in the output said the excerpt was empty.

Fix

The fix reads the first page, as the docstring promised:

extraction = extract_pdf(file_path)
first_page = extraction.pages[0].text if extraction.pages else ""
return first_page[:EXCERPT_BYTES]

The new test class calls the real function on a real PDF:

class RealPdfExcerptTests(unittest.TestCase):
 def setUp(self):
 self.tmp = tempfile.TemporaryDirectory()
 self.root = Path(self.tmp.name)

 def tearDown(self):
 self.tmp.cleanup()

 def test_a_pdf_yields_its_first_page_text(self):
 path = self.root / "invoice.pdf"
 _write_pdf(path, "INVOICE 2026-001 Supplier ACME")
 self.assertIn("INVOICE 2026-001", _extract_excerpt(path))

 def test_a_corrupt_pdf_yields_no_signal_rather_than_raising(self):
 path = self.root / "broken.pdf"
 path.write_bytes(b"%PDF-1.4\nnot actually a pdf at all")
 self.assertEqual(_extract_excerpt(path), "")

_write_pdf is an existing test helper that writes a one-page PDF with a text layer. Before committing, I ran the first test against the original code to confirm it fails there. The except stayed, because corrupt uploads are real. The second test now says what it’s for.

A day later, the same helper had a second route to the same result. It kept its own list of text suffixes, so Word, Excel, and XML e-invoice uploads fell through to "" and were classified by filename too. Two of those formats joined the pipeline a week after the list was written, and nobody updated it. The helper now asks the pipeline’s format registry instead.

Why

A mock replaces the code it patches with your assumptions about that code. When every test mocks a function, its tests check only those assumptions, and the function itself is never run. A catch-all except inside the function then removes the last signal you’d get in production: no traceback, no failed request, just a default value that looks legitimate.

A language model downstream makes this worse. A parser fed an empty string usually fails loudly, while a model returns its best guess in the usual format, confidence field included, whatever the input.

Takeaway

Mock the expensive or external dependency (here, the model) and let the function under test run on a real input. If it has a catch-all except, give it a test with an input that should succeed, so a bug can’t hide behind the except.

Grep your tests for patch( targets that point at your own module’s private helpers, and for each one, find a test that calls the helper unmocked.

Tilde pointed at a profile home, so the export landed elsewhere

Mon, 04 May 2026 00:00:00 +0000

The export command finished, but the expected directory was empty. Then Python reported [Errno 2] No such file or directory for a script that existed in the user’s home. The runner’s HOME pointed at a profile-specific directory. Fixing this required explicit paths for both the export destination and the Python environment.

Two homes, two plausible directory trees

An initial attempt to read a helper resolved to a profile-home path where the file did not exist. During the later export, cd ~/transfer succeeded because that directory existed too. A virtual environment had been created there, but the actual transfer script lived under the user’s expected home.

The terminal output settled the discrepancy: printing HOME returned the profile-specific home. Comparing the two transfer directories found the script only under the expected user-home root. Comparing output directories found exported data under that root and an empty directory under the profile home. The report also existed only under the expected root.

Changing directories alone would still leave Python’s Path("~/data").expanduser() following the process’s HOME. The repair replaced tilde-based output defaults with explicit absolute destinations, including separate report and log paths. The script supported overrides such as KB_OUT_ROOT. Running it from the intended transfer directory also selected that directory’s .venv/bin/python.

This was observed in a historical Hermes runner environment. It does not establish that every Hermes profile changes HOME; consult the current Hermes documentation for your setup. A configured profile directory and the home directory you expect are separate facts to check.

Check the process before searching for files

Run these inside the same terminal tool or wrapper that launches the job:

printf 'HOME=%s\n' "$HOME"
id
pwd
python3 -c 'import sys; from pathlib import Path; print(sys.executable); print(Path.home())'

Compare those paths with the expected project root. Limit directory checks to those known roots; a filesystem-wide search adds noise and can expose unrelated private files. A successful cd proves only that a directory exists, not that it contains the intended script or environment.

Shell expansion happens before the child starts

In Bash, unquoted ~/data expands in the calling shell. In env HOME=/somewhere python3 ... ~/data, the argument has already expanded before env changes the child’s environment. Single-quoted '~/data' stays literal; Python must expand it explicitly. On Unix, Path.expanduser() uses HOME when it is set.

This isolated example demonstrates both cases without touching real home directories:

SCRATCH="${TMPDIR:?Set TMPDIR to your scratch directory}/home-path-demo"
mkdir -p "$SCRATCH/caller-home" "$SCRATCH/profile-home"
PYTHON="$(command -v python3)"
export SCRATCH PYTHON
HOME="$SCRATCH/caller-home" bash --noprofile --norc <<'SH'
env HOME="$SCRATCH/profile-home" "$PYTHON" -c \
 'import sys; print(sys.argv[1])' ~/data
env HOME="$SCRATCH/profile-home" "$PYTHON" -c \
 'import sys; from pathlib import Path; print(sys.argv[1]); print(Path(sys.argv[1]).expanduser())' '~/data'
SH

The first command prints the caller’s home path. The second prints the literal ~/data, then the profile-home path.

Pin the interpreter and verify the artifact

Here is a standalone local inventory example. The program reads its input directory as read-only; the output root is explicit. It requires Python’s venv module and uses only scratch fixtures.

SCRATCH="${TMPDIR:?Set TMPDIR to your scratch directory}/home-path-demo"
mkdir -p "$SCRATCH/source" "$SCRATCH/exports"
printf 'fixture\n' > "$SCRATCH/source/sample.txt"
python3 -m venv --without-pip "$SCRATCH/venv"
export DATA_ROOT="$SCRATCH/source" KB_OUT_ROOT="$SCRATCH/exports"
"$SCRATCH/venv/bin/python" <<'PY'
import os
import sys
from pathlib import Path

source = Path(os.environ["DATA_ROOT"])
out = Path(os.environ["KB_OUT_ROOT"])
if not source.is_absolute() or not out.is_absolute():
 raise ValueError("Use absolute input and output roots")
artifact = out / "inventory.txt"
artifact.write_text("\n".join(sorted(p.name for p in source.glob("*.txt"))) + "\n")
assert artifact.read_text() == "sample.txt\n"
print("interpreter:", sys.executable)
print("verified artifact:", artifact.resolve())
PY

For a real export, validate expected contents at the selected destination and check the report there. In the failed run, a pipeline ending in tail returned zero even though Python could not open the script. Command status alone missed both the launch failure and the path mismatch. Check your job’s resolved interpreter and output root before its next run.

If GPG is missing, signature verification must fail closed

Mon, 20 Apr 2026 00:00:00 +0000

An updater effectively contained the equivalent of if gpg: verify_signature(), then extracted the downloaded archive unconditionally. On a minimal installation without the optional binding, unsigned content took the trusted path. If authenticity is required, treat an unavailable verifier as a hard failure, not permission to continue.

This pattern is CWE-347, improper verification of a cryptographic signature . It becomes more dangerous when the next stage expands an archive, loads a plugin, flashes firmware, or executes an installer, because the bypass then feeds directly into execution.

Optional crypto turns verification into best effort

This shape is unsafe:

if verifier is not None:
 if not verifier.verify(payload, signature):
 raise SignatureError("bad signature")

extract(payload) # Runs when no verifier exists.

The absence branch is the bypass. During review, search for trust decisions guarded by import success, feature flags, platform checks, or broad exception handlers, then follow each branch to the first sensitive consumer. A green path should require a positive verification result.

An attacker still needs influence over the downloaded payload or its source. That may come from a compromised update endpoint, repository, mirror, DNS or TLS failure, or another write primitive. Missing verification does not by itself prove remote code execution; it removes the authenticity barrier that should contain those failures.

Archive extraction can turn bypass into a file write

A second bug often makes the impact concrete: archive members such as ../../outside or symlinks that escape the destination. Joining a destination with an attacker-controlled member name does not remove traversal so the archive bug can turn the bypass into a file write.

Python added extraction filters in 3.12, and Python 3.14 changed the default to the data filter. The current tarfile documentation still recommends explicit filtering and additional limits because no built-in filter stops every denial-of-service or filesystem hazard.

This local fixture asserts both failure paths. Save it as verify_extract.py and run it only in a disposable directory:

import io
import tarfile
import tempfile

class VerificationError(Exception):
 pass

def verify_and_extract(payload, signature, verifier, destination):
 if verifier is None:
 raise VerificationError("signature verifier unavailable")
 if not verifier(payload, signature):
 raise VerificationError("signature rejected")
 if not hasattr(tarfile, "data_filter"):
 raise RuntimeError("safe tar filter unavailable")
 with tarfile.open(fileobj=io.BytesIO(payload), mode="r:*") as archive:
 archive.extractall(destination, filter="data")

def traversal_archive():
 output = io.BytesIO()
 with tarfile.open(fileobj=output, mode="w") as archive:
 body = b"test"
 member = tarfile.TarInfo("../outside")
 member.size = len(body)
 archive.addfile(member, io.BytesIO(body))
 return output.getvalue()

payload = traversal_archive()
with tempfile.TemporaryDirectory() as destination:
 try:
 verify_and_extract(payload, b"sig", None, destination)
 raise AssertionError("missing verifier was accepted")
 except VerificationError:
 pass

 try:
 verify_and_extract(payload, b"sig", lambda data, sig: True, destination)
 raise AssertionError("traversal member was accepted")
 except tarfile.FilterError:
 pass

print("missing verifier and traversal archive rejected")

The fake verifier isolates extraction behavior; it is not production signature logic. The fixture passes on Python 3.12 or newer when the filter is available.

Make verification a prerequisite state

Package the verifier as a required dependency for update support. If it cannot initialize, turn off updating and report a clear error. Pin the expected public key or trust root, validate the signature over the exact bytes later consumed, and reject missing, malformed, expired, or wrong-signer results. Do not catch verification errors and continue.

After verification, extract into a new temporary directory with filter="data". Reject links if they are unnecessary, limit member count and total expanded size, allow only expected file types, and publish the completed tree atomically. Run the updater with the least filesystem privilege it needs.

Regression tests should cover a valid signature, an invalid signature, a missing signature, an unavailable verifier, a corrupt archive, a traversal member, an escaping link, and an oversized archive. Assert that no file outside the staging directory changes on every failure.

Audit every if crypto_library: branch as if the dependency were absent, because eventually it will be.

rdflib field notes: ns.format matches nothing, and parse() goes online

Sat, 11 Mar 2023 12:37:00 +0100

UUpdate 2026-11-08: retested on rdflib 7.6.0 and added the ConjunctiveGraph deprecation, pyoxigraph, and the RDF 1.2 and SPARQL 1.2 status.

This lookup returns an empty list, with no exception and no warning:

from rdflib import Graph, Literal, Namespace, URIRef

DCT = Namespace("http://purl.org/dc/terms/")
doc = URIRef("http://example.org/doc")

g = Graph()
g.add((doc, DCT["format"], Literal("text/csv")))

list(g.objects(doc, DCT.format)) # []
list(g.objects(doc, DCT["format"])) # [rdflib.term.Literal('text/csv')]

Namespace subclasses str, so DCT.format is the bound method str.format, and a lookup on it matches nothing. RDF code tends to fail this way: you get fewer triples, and that looks exactly like sparse data.

I collected these while re-reading a crawler I wrote in early 2023, and rebuilt each as a minimal repro on rdflib 6.3.2 and 7.6.0 under Python 3.11. Both versions behave the same unless noted. They show what the code does. I have no production logs that say what any of them cost.

Use `ns["format"]` for names that `str` already owns

format, count, index, join, and every other str method name resolve to the method. title is the exception, special-cased to return a URIRef. Writes are safer than reads here: g.add() raises AssertionError on the method object.

For published vocabularies, import from rdflib.namespace. DCTERMS.format is a real URIRef and DCTERMS.nonsense raises AttributeError. For your own vocabulary, ClosedNamespace rejects unknown terms, but .format on it is still the str method, so keep the brackets.

A typo in the base IRI is still a valid IRI

NS = Namespace("https://example.org/vocab/mediaTyp")
NS.mediaType # https://example.org/vocab/mediaTypmediaType

My crawler shipped this. The branch using it asked for a predicate no document contains. ClosedNamespace produces the same glued IRI, because it validates the term and has no opinion on the base. A one-line test covers it, with an allowlist for the rare namespace that ends differently:

assert str(NS)[-1] in "#/"

A failed parse keeps what it already read

Feed Graph.parse() five good N-Triples lines and one bad one. It raises ParserError, and len(g) is 5. Turtle does the same with BadSyntax. If you parse into a shared graph, a truncated download leaves half a document behind.

Give each document its own store and merge only after the parse succeeds:

import tempfile
from pathlib import Path
from rdflib import Graph, URIRef, plugin
from rdflib.store import Store

def load_document(source, fmt, target):
 with tempfile.TemporaryDirectory() as tmp:
 path = str(Path(tmp) / "graph")
 store = plugin.get("LevelDB", Store)(identifier=URIRef(path))
 graph = Graph(store, identifier=URIRef("urn:doc"))
 graph.open(path, create=True)
 try:
 graph.parse(source, format=fmt)
 target += graph
 finally:
 graph.close()

With a bad document, this raises an exception and target stays empty. The disk store also caps memory. Loading 200,000 synthetic triples (14 MB of N-Triples, one run on Apple Silicon, rdflib 6.3.2, 30 MB baseline after imports):

Store	Time	Peak RSS	Disk
rdflib `Memory`	2.6 s	351 MB	0
`rdflib-leveldb`	15.3 s	85 MB	39 MB
pyoxigraph 0.5.13 `bulk_load`	0.5 s	200 MB	10 MB

Read that as a rough shape and benchmark your own data. rdflib-leveldb isn’t on PyPI, so install it from git. Its last commit is from December 2021, though the snippet above still passes on 7.6.0. On macOS, I used plyvel-ci because plyvel has no Mac wheels. Pyoxigraph has its own API and isn’t a drop-in rdflib store.

A disk store bounds the triples and leaves the input alone. The Turtle parser calls stream.read() on the whole source, with the comment # Not ideal.

`parse()` can go online

Parsing JSON-LD fetches remote @context values. The RDFLib security page says so. Direct cycles are caught (recursive context inclusion after 2 requests). A chain of contexts that each point to a new one isn’t: with Python’s default recursion limit, one small document cost 967 requests to my local server before RecursionError.

Block the network during the parse step:

import urllib.error, urllib.request

class NoNetwork(urllib.request.BaseHandler):
 handler_order = 100 # before the stock HTTPHandler (500)

 def http_open(self, req):
 raise urllib.error.URLError(f"blocked: {req.full_url}")

 https_open = http_open

urllib.request.install_opener(urllib.request.build_opener(NoNetwork()))

Leave out handler_order and the stock handler answers first, so the request goes through. My first attempt did exactly that. The opener is process-global, and the RDFLib docs rank OS-level isolation such as containers above Python-level controls, so use it in addition to those on untrusted input.

Then make the following links a separate, bounded step that you control:

from collections import deque

def crawl(seed, fetch, max_depth=3):
 seen, queue = {seed}, deque([(seed, 0)])
 while queue:
 iri, depth = queue.popleft()
 references = fetch(iri) # fetch, parse in isolation, return IRIs to follow
 if depth < max_depth:
 for ref in references:
 if ref not in seen:
 seen.add(ref)
 queue.append((ref, depth + 1))

On a graph with a cycle and a chain, this visits each IRI once and stops at depth 3.

Keep a workaround’s reason in a test

rdflib #1195 reported that a CONSTRUCT response with application/rdf+xml; charset=UTF-8 broke the plugin lookup. My code registered that literal string as a parser name and pointed a comment at the issue.

Against a fake endpoint sending that header, rdflib 5.0.0 raises PluginException and 6.0.0 works without the workaround (both on Python 3.9). My pin was already 6.0.0 in the commit that added it, so on this path it never changed, and it stayed that way through 6.3.2. Watching the issue wouldn’t have helped: it closed in August 2022 on a PR that only added tests.

A test that exercises the upstream behavior with the workaround off tells you the day it can go. plugin.get() itself still rejects that string on 7.6.0, so strip parameters if you call it directly.

`Dataset` isn’t a drop-in for `ConjunctiveGraph`

ConjunctiveGraph raises a DeprecationWarning on 7.6.0 and the replacement is Dataset, which has a different default. With one quad in a named graph:

Class	`triples()`	`SELECT ?s ?p ?o`
`ConjunctiveGraph()`	1	1
`Dataset()`	0	0
`Dataset(default_union=True)`	1	1

Migrate with default_union=True, unless you want queries limited to the default graph.

On standards, as of November 2026: RDF 1.2 Concepts is a Candidate Recommendation and SPARQL 1.2 Query is a Working Draft, so SPARQL 1.1 remains the Recommendation.

Takeaway

Most of these bugs arrive as an empty result, and nothing downstream complains. For each predicate your pipeline depends on, keep a fixture with one known triple and assert that your code finds it.

Start with this search for unbracketed str names in your own code:

grep -rnE '\b[A-Za-z_]+\.(format|count|index)\b([^(]|$)' --include='*.py' .

Celery ignored task_task_routes: silent failures from growing a crawler on Redis

Sat, 11 Mar 2023 12:19:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

This sat in my crawler’s Celery config:

task_task_routes = {
 "tasks.urgent": {"queue": "celery"},
 "tasks.post": {"queue": "celery:2"},
}

The setting is called task_routes. Celery doesn’t check setting names, so the app started normally, app.conf.task_routes stayed None, and three different tasks all landed in the default queue:

{'celery:1': 3}

That typo arrived with the commit that moved the crawler to Redis priority queues, and it wasn’t the only failure from that stretch that raised nothing. Two small changes catch most of them: compare your settings against Celery’s list of known names, and set task_create_missing_queues = False.

I found these by rereading the code and rebuilt each one against a private Redis, on Python 3.11 with the locked versions (Celery 5.2.7, kombu 5.2.4, redis-py 4.5.4) and on Celery 5.7.0. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

How the crawler got here

Date	Change
2018	Celery on Redis, with the crawl state in Redis too
2020	Mongo for reports; separate priority and regular tasks
2021-09	A monitor task that decides when the crawl is done
2023-02-03	Large structures move from Redis to SQLite, then to Postgres the same night
2023-02-07	Mongo dropped
2023-02-10	Bulk inserts from generators
2023-02-12	Named priority queues; each input parsed into an on-disk graph in a temp directory

Redis ended up mostly as the broker and a few caches. The traps below surfaced in the 2023 part of that table.

Celery ignores settings it doesn’t know

A test finds the typo. DEFAULTS holds every setting name Celery recognizes:

from types import ModuleType

from celery.app.defaults import DEFAULTS

import celeryconfig


def test_celery_settings_are_known():
 unknown = {
 name for name, value in vars(celeryconfig).items()
 if name.islower() and not name.startswith("_")
 and not isinstance(value, ModuleType)
 and name not in DEFAULTS
 }
 assert not unknown

On my config, it reports {'task_task_routes'}, on both Celery versions.

A queue with no consumer looks like a finished crawl

The same commit changed what the worker reads, and the post-processing chain kept an old queue name:

celery -A app worker -Q celery,celery:1,celery:2

chain(step_one.si(), step_two.si()).apply_async(queue="query")

task_create_missing_queues is on by default, so sending to query creates it. In the rebuild, the message sat in that key, and no step ran. Declare your queues and turn creation off:

from kombu import Queue

task_create_missing_queues = False
task_queues = [Queue("celery"), Queue("bulk")]

celery.exceptions.QueueNotFound: "Queue 'query' missing from task_queues"

That protects the sending side only. The -Q list on the worker is still a separate copy, so build it from the same list.

broker_transport_options = {
 "priority_steps": [0, 1, 2],
 "sep": ":",
 "queue_order_strategy": "priority",
}

With these options, kombu stores priority 1 of queue celery queue under the Redis key celery:1. I also named a queue celery:1. A message sent with queue="celery", priority=1 showed up in that key. Priority 0 gets no suffix, so celery:0 never exists.

Pick one scheme: a single queue with priority=, or named queues whose names don’t contain the separator.

`LLEN` 0 doesn’t mean the crawl is done

Every ten minutes, the monitor computed this and declared the crawl done at zero:

enqueued = red.llen("celery") + red.llen("celery:0") + red.llen("celery:1")

It counts the key that never exists and skips celery:2. Even with the right keys, a list length misses work:

State	Monitor	Lists + `HLEN unacked`
Task waiting in `celery:2`	0	1
Task running, `task_acks_late=True`	0	1
Retry waiting on a 6 s countdown	0	1
Chain stranded in `query`	0	0

def outstanding(client, queues=("celery", "celery:1", "celery:2")):
 with client.pipeline() as pipe:
 for queue in queues:
 pipe.llen(queue)
 pipe.hlen("unacked")
 return sum(pipe.execute())

unacked is kombu’s internal bookkeeping hash, and it shows running tasks only because my config sets task_acks_late=True. With early acknowledgment, a running task has already left it. The last row shows the limit of any count: it can’t see a queue you forgot to list.

When you know the fan-out up front, a chord gives a real completion signal. The docs say members must not ignore their results, and I copied that warning into a comment. On the Redis backend, a chord of ignore_result=True members still ran their bodies with all three values, on both versions. I’d follow the docs anyway, since I tested one backend. My tasks also spawned more tasks, which a chord can’t express.

`decode_responses=True` on a shared pool breaks a binary cache

In 2021, my HTTP cache shared the application’s Redis pool, which decoded responses to strings. requests-cache stores pickles, and a pickle starts with byte 0x80. Two identical GETs:

requests-cache	Decoding pool	Separate pool
0.9.8	2 origin hits, one log line	1 origin hit
1.3.3	Second GET raises `UnicodeDecodeError`	1 origin hit

The log line on the old version is all you get:

Unable to deserialize response with key 53630720a0b4bd02: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

The commit that fixed it says what to do: “a separate Redis pool without decoding”:

app_pool = redis.ConnectionPool.from_url(url, decode_responses=True)
cache_pool = redis.ConnectionPool.from_url(url)

`execute()` drains a generator, `group()` doesn’t

I passed generators to session.execute(insert(Row), rows()) to save memory. SQLAlchemy reads them to the end first: on 2.0.7 and 2.1.8, with a SQLite engine, the generator had yielded 5,000 of 5,000 rows before the first INSERT. Batch it yourself:

from itertools import islice


def batches(items, size=1000):
 items = iter(items)
 while batch := list(islice(items, size)):
 yield batch


for batch in batches(rows()):
 session.execute(insert(Row), batch)
session.commit()

With 2,500 rows, the three INSERTs fired after 1,000, 2,000, and 2,500 yields.

group(generator).apply_async() works the other way and publishes as it iterates. With a generator that yielded two signatures and then raised, one task was already in Redis, and the caller got the exception with no GroupResult. Make those tasks safe to run twice. I still prefer group to chunks, which loses retries .

Takeaway

Most of this was introduced or fixed during ten days of refactoring in February 2023, and almost none of it raised an error then. A queue migration needs a written list of each queue, who sends to it, who reads it, and what counts it.

Try the settings test above on your own Celery config.

Celery time_limit on the gevent pool: a 1 s limit let a CPU task run 3 s

Wed, 08 Feb 2023 23:36:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 with gevent 22.10.2, and on Celery 5.7.0 with gevent 26.9.0.

I gave a Celery task time_limit=1 and ran it with -P gevent. The CPU-bound task kept going for 3 s, finished all its work, and was then reported as failed:

state=FAILURE result=TimeLimitExceeded(1) got_after=3.06s

The gevent pool enforces the limit with gevent.Timeout, which can only fire when the task yields to the event loop. CPU-bound code never does. Keep gevent for I/O and run CPU work on a prefork worker.

The workers guide says this in two sentences: the gevent pool “does not implement soft time limits” and “will not enforce the hard time limit if the task is blocking”. It leaves out what that looks like in a task, and how an exception handler makes it worse.

I found this by rereading old code and rebuilt each case on a real worker with a private Redis, on Python 3.11. Both version pairs behaved the same. I have no production logs from that time.

What a 1 s limit does on each pool

Each task does 3 s of work:

Task, 1 s limit	gevent	prefork
CPU loop	Ran the full 3 s, then `TimeLimitExceeded`	Stopped after about 1 s
`time.sleep`	Stopped after about 1 s	Stopped after about 1 s
Sleep inside `try` / `except Exception`	Handler skipped, stopped after about 1 s	Stopped after about 1 s
Sleep inside `try` / bare `except:`, then 3 s more	`SUCCESS` at 4.01 s	`FAILURE` after about 1 s
`soft_time_limit=1` with sleep	Ignored, `SUCCESS` at 3.01 s	`SoftTimeLimitExceeded` after about 1 s

The first row is the nasty one. The work completed, its side effects happened, and the task still reports failure. The timer was pending the whole time and fired at the first switch after the loop.

A CPU-bound task also stalls everything else in that worker process. With -c 8, a task sent during a 3 s CPU loop waited 2.8 s to start, against a few milliseconds on an idle worker. On prefork with -c 2, it still started within a few milliseconds.

Split fetch and parse into separate workers

app.conf.task_routes = {
 "tasks.fetch": {"queue": "fetch"},
 "tasks.parse": {"queue": "parse"},
}

chain(fetch.s(url), parse.s()).apply_async()

celery -A app worker -n fetch@%h -P gevent -c 60 -Q fetch
celery -A app worker -n parse@%h -P prefork -c 2 -Q parse

With time_limit=1 on parse, a chain with a 3 s CPU parse ended in TimeLimitExceeded after 1.1 s, and a 0.1 s parse succeeded. Pass a file path or an object-store key between the two rather than the body.

Why a timeout can’t interrupt CPU work

celery/concurrency/gevent.py in 5.7.0 wraps each task like this:

try:
 with Timeout(timeout):
 return apply_target(target, args, kwargs, callback,
 accept_callback, getpid(),
 propagate=(Timeout,), **rest)
except Timeout:
 return timeout_callback(False, timeout)

Timeout is a timer on the event loop, and everything runs in one thread. The loop gets control only when the current greenlet yields, typically at a monkey-patched socket call or sleep. Prefork stops a task from outside, and the hard limit replaces the child process.

The soft limit is simpler: on_apply takes soft_timeout through **_ and never uses it. The worker log says nothing about that.

A bare `except:` cancels the limit

gevent.Timeout derives from BaseException , so except Exception lets it pass and a bare except: catches it. My code had a fallback like this several calls below the task:

try:
 return lookup(url)
except:
 log.exception("lookup failed, trying the fallback")
 return fallback(url)

The timer fires once. After the handler ate it, the task had no limit at all, and in the rebuild it finished three seconds past the deadline as SUCCESS.

My fix at the time, a commit called “Fail the task on timeout”, added except Timeout at the top of the task. A Timeout stops at the nearest handler that matches, so the nested bare except: on the fallback path kept swallowing it. Change the nested handler:

except Exception:
 log.exception("lookup failed, trying the fallback")
 return fallback(url)

If you need cleanup, catch Timeout and re-raise it. In the rebuild, the cleanup ran at 1.01 s, and Celery recorded TimeLimitExceeded, so there was no need to set the state by hand.

The soft limit goes the other way on prefork: SoftTimeLimitExceeded is an Exception, so a task that moves between pools needs its handlers checked for both.

For a Python loop you own, gevent.sleep(0) on each iteration gives the loop a turn, and the task then stopped at 1.01 s. That can’t help inside a library call that never yields.

The sibling trap on the same crawler: Retry is an Exception, and except Exception eats it .

Takeaway

On the gevent pool, time_limit is only as good as the task’s willingness to yield, so treat it as a limit on I/O waits.

Run grep -rnE --include='*.py' 'except:[[:space:]]*$' . on anything that runs under -P gevent.

Celery self.retry(5) raises TypeError, and except Exception swallows the retry

Tue, 07 Feb 2023 00:01:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

My crawler called Celery’s retry three different ways, and only one did what I meant:

Call	Result
`self.retry(5)`	Task fails with the error below, no retry
`self.retry(timeout=5)`	Retries in 180 s
`self.retry(countdown=5)`	Retries in 5 s

TypeError: 'int' object is not iterable

Even the correct call can go wrong. retry() publishes the new message and then raises celery.exceptions.Retry, an ordinary Exception. An except Exception around it marks the work as failed while the retry is already on its way. Pass countdown= by keyword and let TaskPredicate pass through your broad handlers.

I found these by rereading old code and rebuilt each case on a real worker with a private Redis as broker and result backend, on Python 3.11. Both Celery versions behaved the same. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

Re-raise `TaskPredicate` before `except Exception`

from celery.exceptions import TaskPredicate


@app.task(bind=True, max_retries=2)
def crawl(self, url):
 try:
 fetch(self, url) # calls self.retry(exc=err, countdown=1) on a network error
 except TaskPredicate:
 raise
 except Exception:
 mark_failed(url)
 return "gave up"
 mark_ok(url)
 return "done"

TaskPredicate is the base class of Retry, Ignore and Reject. With this handler, result.get() returned 'done' about 1 s later, after the retry had run.

My handler already knew about one of the three:

except Exception as err:
 if isinstance(err, Ignore):
 raise

Retry needed the same treatment and didn’t get it.

Why the three calls differ

The signature on both versions is:

retry(self, args=None, kwargs=None, exc=None, throw=True, eta=None, countdown=None, max_retries=None, **options)

The first positional parameter is the replacement args for the retried task, so 5 reaches tuple(args) and raises before anything is published.

timeout isn’t a parameter, so it lands in **options and travels with the message. Spying on kombu’s Producer.publish showed timeout=5 arriving there, where it limits how long publishing may take. The delay falls back to default_retry_delay, three minutes by default . Nothing raises. The only hint is the worker log:

Task tasks.crawl[62d0a257-...] retry: Retry in 180s

What `except Exception` does to a retry

Remove the except TaskPredicate branch from the task above and let the first attempt hit a network error:

Time	Event
0.01 s	Retry published, `Retry` caught, URL marked failed
0.01 s	`result.get()` returns `'gave up'`, state `SUCCESS`
1.01 s	Retry runs and marks the URL ok
Later	The backend still says `'gave up'`

The last row surprised me. On the Redis backend, _store_result ignores any write once the stored state is SUCCESS, so the retry’s own return value is dropped. Anyone reading the result sees the attempt that gave up.

Code review won’t catch this reliably, because the placement decides it. A retry inside the try body, even several calls deep, is caught. A retry in a sibling except clause is safe:

try:
 fetch(task, url) # a retry raised in here hits except Exception
except RateLimited as err:
 task.retry(countdown=err.delay) # sibling handlers don't see this one
except Exception:
 mark_failed(url)

My code had both forms in the same function, so some retries worked. Writing raise self.retry(...) changes nothing, and a bare except: catches Retry too. In an older module, a bare except: also caught the TypeError from self.retry(5), which recorded a rate-limited link as a broken one.

A dedupe mark written first makes the retry skip itself

@app.task(bind=True, max_retries=2)
def crawl(self, url):
 if redis_client.sadd("seen", url) == 0:
 return "skipped"
 ...

attempt 0 sadd=1
attempt 1 sadd=0

The retry sees its own mark, returns 'skipped', and the URL is never fetched. Combine that with the swallowed Retry and the URL stays failed for good: in the rebuild, the final status was failed and the result was 'gave up'.

Moving the SADD after the work fixed it: the retry returned 'fetched'. Two concurrent tasks can then fetch the same URL, so the work has to be safe to repeat.

When the retry condition is just an exception type, autoretry_for=(Flaky,), retry_kwargs={"countdown": 1, "max_retries": 2} saves calling retry yourself. The exception has to leave the task body, so a broad handler inside it defeats this as well.

Time limits have a similar problem on the gevent pool, in the opposite direction: Timeout isn’t an Exception .

Takeaway

self.retry() is a raise whose side effect has already happened, so any handler between it and the worker can make your bookkeeping disagree with the queue.

Run grep -rn --include='*.py' '\.retry(' . on your tasks and check each hit for keyword arguments and an enclosing except Exception.

Encrypt sensitive fields inside the ORM

Sat, 30 Oct 2021 18:03:45 +0200

Your database may run where you don’t control the environment: embedded in a desktop app, or in the cloud for a web app. If it stores sensitive data, encrypt those fields so a leaked copy doesn’t expose them. In this article, I show how I did that in two projects, and the drop-in text field for Python ORMs that came out of it.

Encrypted fields in an embedded SQLite

The first case was a desktop app with an SQLite database that only authorized software could read.

I considered encrypting the SQLite file and encrypting fields within it, and I chose the latter. That approach is reusable, and you never have to load the whole database into memory to decrypt it or leave it in plaintext for a moment.

The app already had its database logic, and the goal was to change as little as possible. So I put the encryption inside the ORM, and the rest of the app stayed business as usual.

The details:

Cipher and key: AES-256-CTR with a single key for the whole database. There was no reason for anything more complex. The key is generated on the first run and on the upgrade from the old version.
Storage: binary blobs made querying the database in a terminal for debugging a pain. Base64-encoded ciphertext in a text field works better. A new column holds the row’s initialization vector (IV), a random array of bytes, also base64-encoded.
Migration: it’s not only a schema change. Every row needs the fields encrypted for the updated logic to work.

You can’t query against encrypted fields. The ciphertext is a function of the plaintext, the key, and the IV, and the IV is specific to the row. This is a property of the privacy, not a defect. When you need to query encrypted values, the solution is domain-specific: sometimes you can modify the data or compute hashes. Discuss it with the customer in terms of the security objectives they want to achieve. For the same reasons, the data migration is domain-specific too.

An encrypted payload in MongoDB documents

On another project, the database was MongoDB and had many sensitive fields. I split each document into the sensitive fields and the metadata used for querying. I encrypted the sensitive fields together into a single field. The document also gets the IV and an HMAC over the rest of the document, ciphertext included:

def create_secure_json(form, keys):
 data = get_sensitive(form)
 data_secure_b64, iv_b64 = cryptjson.encrypt_json(data, keys['db'])

 payload = get_metadata(form)
 payload['created'] = str(datetime.datetime.now())
 payload['iv'] = str(iv_b64, 'ascii')
 payload['data'] = str(data_secure_b64, 'ascii')

 hmac_b64 = get_hmac(payload, keys['hmac'])
 payload['hmac'] = str(hmac_b64, 'ascii')
 return payload

This payload is what goes into the database. Use a separate key for the HMAC, and verify it before you decrypt: AES-CBC alone doesn’t detect tampering.

For the encryption, I use the cryptography package from the Python Cryptographic Authority. Encrypting a JSON is no magic: dump it into bytes and encrypt those, here with AES-CBC:

import base64
import secrets

from cryptography.hazmat.primitives import padding
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes


def encrypt_bytes(pt, key):
 iv = secrets.token_bytes(16)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 encryptor = cipher.encryptor()
 padder = padding.PKCS7(128).padder()
 padded_pt = padder.update(pt) + padder.finalize()
 ct = encryptor.update(padded_pt) + encryptor.finalize()
 return base64.b64encode(ct), base64.b64encode(iv)


def decrypt_bytes(b64ct, b64iv, key):
 ct = base64.b64decode(b64ct)
 iv = base64.b64decode(b64iv)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 decryptor = cipher.decryptor()
 padded_pt = decryptor.update(ct) + decryptor.finalize()
 unpadder = padding.PKCS7(128).unpadder()
 return unpadder.update(padded_pt) + unpadder.finalize()

A drop-in encrypted text field

Both solutions were domain-specific. Still, they showed that the use case is common and that encryption in Python is easy to work with. So I wrote cryptbase, a small library that adds an encrypted text field to SQLAlchemy and Django ORM.

Inside, there is a CryptoContainer. You create it from a ciphertext and an IV, or from a plaintext, and it gives you access to the ciphertext, the plaintext, and the IV in a unified way:

container = CryptoContainer(ciphertext=base64.b64decode(ciphertext),
 iv=base64.b64decode(iv),
 encryptor=encryptor)

container = CryptoContainer(plaintext=value, encryptor=self.encryptor)

A separate IV column complicates the migration, and the goal is a drop-in replacement for text fields. The base64 alphabet has no @, so the field stores both values in one column, with @ as the separator:

<base64 ciphertext>@<base64 IV>

The encryptor is an object with a simple interface, so there can be one per encryption method, such as AES256CTREncryptor:

def encrypt(self, plaintext: bytes) -> Tuple[bytes, bytes]:
 pass

def decrypt(self, ciphertext: bytes, iv: bytes) -> bytes:
 pass

In the end, the ORM gets a simple interface:

# SQLAlchemy
sensitive = Column(EncryptedText(key=DB_KEY))

# Django
sensitive = EncryptedTextField(key=DB_KEY)

You work with these fields the same way as with a text field. When the application reads the value, it gets the plaintext from the container. When the value is written to the database, it’s encrypted, and the __str__ of the CryptoContainer joins the base64-encoded ciphertext and IV:

str(CryptoContainer(plaintext=value, encryptor=self.encryptor))

Dynamic DNS for home routers the hacker way

Thu, 28 Oct 2021 17:48:15 +0200

Skip public DDNS services: Python can handle the same job. This post shows how to modify a DNS record in a zone file, upload the zone over SSH, and expose the update as a small web service.

The setup

A MikroTik router with a dynamic public IPv4 address
A name for the router in my own zone, on my primary DNS server
Python with dnspython, paramiko, Flask, and flask_jwt_extended

I first looked at dynamic DNS over a standard protocol, but I had to abandon the idea. MikroTik also offers its cloud service for a similar result, but I find it less convenient than a name in my own zone. That led me to a way to modify the zone file with Python.

The examples cover IPv4 only. Extending them to IPv6 is a small change.

Modify the zone file

The update has three steps: open the zone file, modify the record, and save the file.

def update(zone, name, newip4):
 original = f'zones/{zone}.zone'
 updated = update_dynamic_ip4(original, name, newip4, 3600)
 with open(original, 'w') as file:
 file.write(f'$ORIGIN {zone}.\n')
 file.write(updated)

The record update itself:

Load the zone file.
Find the record.
Replace it, or stop if the address hasn’t changed.
Update the SOA serial.
Return the zone as text.

Don’t forget the SOA serial. Without the bump, the secondary servers of your zone don’t see the change.

dnspython does the job, but its API is not my favorite cup of tea:

from datetime import date

import dns.rdataclass
import dns.rdataset
import dns.rdatatype
import dns.zone


class NoUpdate(ValueError):
 pass


def update_dynamic_ip4(zonefile, name, newip4, ttl):
 z = dns.zone.from_file(zonefile)
 new_data = dns.rdataset.from_text(dns.rdataclass.IN, dns.rdatatype.A, ttl, newip4)
 node = z.find_node(name)
 oldip4 = node.get_rdataset(dns.rdataclass.IN, dns.rdatatype.A).to_text().split()[3]
 if str(oldip4) == newip4:
 raise NoUpdate
 node.replace_rdataset(new_data)

 soa = z.find_node("@").find_rdataset(dns.rdataclass.IN, dns.rdatatype.SOA)
 soa_updated = update_soa(soa)
 z.find_node("@").replace_rdataset(soa_updated)
 return z.to_text(relativize=False)

I keep the serial as a date (%Y%m%d) and a two-digit revision. If the serial is from today, the revision increases by one; otherwise, it starts at zero for today. That allows 100 changes a day, plenty for one dynamic address.

def update_soa(soa):
 soa_parts = list(soa.to_text().split())
 version = soa_parts[-5]
 today = date.today().strftime("%Y%m%d")
 increment = 0
 if version.startswith(today):
 increment = int(version[len(today):]) + 1
 version = f'{today}{increment:02d}'
 soa_parts[-5] = version
 soa_new = " ".join(soa_parts[3:])
 ttl = soa_parts[0]
 return dns.rdataset.from_text(dns.rdataclass.IN, dns.rdatatype.SOA, ttl, soa_new)

Upload the zone file and reload the DNS server

By hand, you’d SSH to the DNS server, replace the file, and reload the server. paramiko does the same:

import paramiko


def upload_zone(localfile, remotefile, remotehost, remoteport, keyfile):
 with paramiko.SSHClient() as client:
 client.load_system_host_keys()
 client.connect(remotehost, remoteport, key_filename=keyfile)
 with client.open_sftp() as ftp:
 ftp.put(localfile, remotefile)
 client.exec_command('nsd-reload') # the reload command of your DNS server

The DNS server’s host key must already be in your known_hosts.

Expose it as a REST API

The router sends a request, and the app updates the zone. The new address comes from the request itself, and the router identifies itself with a JWT. remotefile, remotehost, remoteport, and keyfile come from the configuration.

This example creates a new secret and token each time the app starts, so the router needs the new token after a restart:

import uuid

from flask import Flask, request
from flask_jwt_extended import JWTManager, create_access_token, get_jwt_identity, jwt_required

app = Flask(__name__)
app.config["JWT_SECRET_KEY"] = str(uuid.uuid4())
jwt = JWTManager(app)
with app.app_context():
 token = create_access_token(identity={'name': name, 'zone': zone},
 expires_delta=False)

The endpoint:

@app.route('/api/update/ipv4', methods=['POST'])
@jwt_required()
def update_endpoint():
 identity = get_jwt_identity()
 name = identity['name']
 zone = identity['zone']
 if 'X-Forwarded-For' in request.headers:
 newip = request.headers['X-Forwarded-For']
 else:
 newip = request.remote_addr
 try:
 update(zone, name, newip)
 except NoUpdate:
 return {'status': 'unchanged'}
 upload_zone(f'zones/{zone}.zone', remotefile, remotehost, remoteport, keyfile)
 return {'status': 'updated'}

Most periodic calls find the address unchanged, so NoUpdate is the common path, not an error. Read X-Forwarded-For only when the app runs behind your own reverse proxy that sets it; otherwise, any client can put an arbitrary address there. Before deploying, add input validation and handling for the remaining exceptions.

Once deployed, set up the router to call the endpoint periodically.

SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Tue, 14 Sep 2021 11:35:00 +0200

Update 2026-11-08: rebuilt every repro on rdflib 6.3.2 and 7.6.0, and on Celery 5.2.7 and 5.7.0.

I wanted a resource and three hops of its neighbors from a SPARQL endpoint. This query returns every triple in the store:

CONSTRUCT { <http://example.org/root> ?p1 ?o1 . ?o1 ?p2 ?o2 . ?o2 ?p3 ?o3 . }
WHERE {
 <http://example.org/root> ?p1 ?o1 .
 OPTIONAL { ?o1 ?p2 ?o2 . }
 OPTIONAL { ?o2 ?p3 ?o3 . }
}

Move the second OPTIONAL inside the first, and you get the four triples you meant. A comment in my 2021 crawler says nested optionals are a must, otherwise the query will not finish. This post explains why and covers other ways the crawler could mistreat an endpoint without noticing.

Everything below comes from rereading that code and rebuilding each case against a local fake endpoint, under Python 3.11. It shows what the code does. I have no production logs that say what it cost anyone.

Flat `OPTIONAL`s join an unbound variable against everything

The test store holds 20,005 triples: a root with a literal label, a four-hop chain, and 20,000 unrelated triples. Times are from one run.

Query shape	Solutions	Triples returned	Oxigraph 0.5.13	rdflib 7.6.0
Flat `OPTIONAL`s	20,006	20,005	28 ms	897 ms
Nested `OPTIONAL`s	2	4	1.0 ms	3.2 ms
No `OPTIONAL`	1	3	0.1 ms	1.7 ms

The literal label triggers it. When ?o1 is that literal, the first OPTIONAL can’t match, because a literal is never a subject, so ?o2 stays unbound. The second OPTIONAL then has nothing constraining it. ?o2 ?p3 ?o3 matches every triple, and the CONSTRUCT template writes them all out. Most resources have a label, so most lookups hit this.

Removing the OPTIONALs is the wrong repair: the label disappears, as the last row shows.

I can’t rerun the 2021 endpoint, and I only tested these two engines. I believe this is what “will not finish” meant, since on a store with millions of triples each lookup becomes a full dump.

A variable you never bind does the same thing. I once committed OPTIONALs on ?dataset in a query that bound ?ds, and renamed it 18 minutes later. It’s a valid query, so nothing raises an error.

Nesting fixes the unbound variable and does nothing for hubs. A node with a million outgoing triples still returns them all, so add a LIMIT if a partial answer is acceptable.

Send one `CONSTRUCT` per named graph, then read your copy

Graph(store=SPARQLStore(url)) looks like a local graph, and every value() call on it is an HTTP request. With 50 datasets and three property reads each:

Approach	Requests to the endpoint
Read through the store-backed graph	151
One `CONSTRUCT`, then read `.graph`	1

The graph that .query(...).graph returns lives in a local Memory store, so later reads cost nothing. Scope the query with FROM <graph> and a failure or a retry costs you one named graph.

Prepared queries belong on that local copy too. A prepareQuery() object sent through SPARQLStore dies on a bare AssertionError, because the store only accepts strings. My git history shows I tried the remote version first.

`SPARQLStore` hides what the endpoint tells you

Both rdflib versions behave the same:

Input	Result
`session=requests.Session()`	Accepted, never used
No `headers=`	`User-Agent: Python-urllib/3.11`
`timeout=1`, server hangs 4 s	Waits the full 4 s
429 with `Retry-After`, default GET	`ValueError: You did something wrong formulating either the URI or your SPARQL query`
429, `method="POST"`	Returns the tuple `(429, 'HTTP Error 429: Too Many Requests', None)`, then `AttributeError` on `.graph`

The 7.6.0 connector docstring says extra keyword arguments “can be used to set up timeouts etc.”, but the code reads only params and headers from them. socket.setdefaulttimeout() does work for the whole process.

My extraction query passed session= and no headers=, and logged ValueError at debug level. An endpoint asking for a pause would have produced a debug line and a skipped graph, sent by an anonymous-looking client.

For CONSTRUCT, skip the store and use the SPARQL protocol directly:

import requests
from rdflib import Graph

class Backoff(Exception):
 def __init__(self, seconds):
 self.seconds = seconds

def construct(session, endpoint, query):
 response = session.post(
 endpoint,
 data={"query": query},
 headers={"Accept": "application/n-triples, text/turtle;q=0.9, application/rdf+xml;q=0.8"},
 timeout=(5, 120),
 )
 if response.status_code in (429, 503):
 value = response.headers.get("Retry-After", "") # may also be an HTTP date
 raise Backoff(int(value) if value.isdigit() else 60)
 response.raise_for_status()
 media_type = response.headers["Content-Type"].split(";")[0].strip()
 return Graph().parse(data=response.content, format=media_type)

session = requests.Session()
session.headers["User-Agent"] = "my-harvester/1.0 (+https://example.org/about)"

Against the fake endpoint, this returned all 200 triples and raised Backoff(120) on the 429. The same post() with a 1-second read timeout raised ReadTimeout on the hanging server. Stripping the media type parameters matters for the reason in my rdflib field notes .

Celery `chunks` runs your task as a plain function

task.chunks(args, n) takes a chunk size , and my docstring claimed it was several tasks. Ten graphs with n=4 make three tasks. Items inside a chunk run in sequence, while the chunks run in parallel, so chunks puts no cap on concurrency against the endpoint.

The worse part is retries. I raised self.retry() on the second of ten graphs:

Canvas	Outcome
`task.chunks(zip(graphs), 4)`	Chunk ends in `FAILURE`, no retry, the next two graphs never run
`group(task.s(g) for g in graphs)`	Second graph retried once, all ten finish

Each chunk is a celery.starmap task that executes task(*item). A direct call leaves request.called_directly true, and in that case retry() re-raises the exception and schedules nothing. Celery 5.2.7 and 5.7.0 agree.

Use a group, and give each endpoint its own queue with one worker slot:

@app.task(bind=True, max_retries=5)
def inspect_graph(self, endpoint, graph_iri):
 try:
 graph = construct(session, endpoint, query_for(graph_iri))
 except Backoff as exc:
 raise self.retry(exc=exc, countdown=exc.seconds)
 extract(graph)

group(inspect_graph.s(endpoint, g) for g in graphs).apply_async(queue="endpoint-a")

celery -A harvester worker -Q endpoint-a --concurrency=1

A dereference fallback is traffic too

When fetching an IRI failed, my crawler asked the dataset’s own endpoint for that resource. The idea holds up, since plenty of linked data has IRIs that don’t resolve while the triples sit in an endpoint.

Mine fell back on every failure, each one costing a three-hop CONSTRUCT. That included a robots.txt disallow on the resource, with only the endpoint’s own robots.txt still checked. Send fallbacks through the same queue, and stop when the site said no.

Check IRIs against the SPARQL grammar

Crawled IRIs ended up in <{iri}> inside query strings. rfc3987.match() rejects > and quotes, and accepts http://example.org/a\n, because its pattern ends in $, which matches before a trailing newline. URIRef(x).n3() passes that newline too, along with \r and a newline mid-string, since rdflib’s list of invalid characters has no control characters.

That newline can’t close the IRI, so this isn’t an injection. You get a malformed query and a wasted request. I added .strip() at the time, on one of the code paths. The IRIREF production is short enough to check directly:

FORBIDDEN = set('<>"{}|^`\\') | {chr(c) for c in range(0x21)}

def iri_ref(value):
 if FORBIDDEN & set(value):
 raise ValueError(f"not safe in a SPARQL IRI: {value!r}")
 return f"<{value}>"

Takeaway

Politeness here came down to two things my code got wrong: how much it asked for, and whether it could hear the answer.

Try this on your own harvester: load one unrelated triple into a test store, run each CONSTRUCT, and fail the test if that triple comes back.

Flask @cached decorator traps: POST gets the cached GET, no-store is stored, and X-Last-Modified: None

Fri, 19 Oct 2018 21:26:00 +0200

Update 2026-11-08: I rebuilt every repro on Flask 2.2.3 with Werkzeug 2.2.3 and Flask-Caching 2.0.2, and on Flask 3.1.3 with Werkzeug 3.1.9 and Flask-Caching 2.5.1, under Python 3.11.

A Flask view decorator I kept for years failed in a way that only showed up when a POST got the body of an earlier GET. The view never ran, and the client got a 200:

GET /item 200 "GET ran" X-Cache: MISS
POST /item 200 "GET ran" X-Cache: HIT from Server

The cache key was request.full_path, so any two requests with the same URL shared an entry, regardless of method, user, or Accept header. To fix this, let only anonymous GET and HEAD requests touch the shared entry, build the key from the response inputs, and leave the HTTP headers to Werkzeug.

The decorator entered a 2018 project with a docstring pointing at a Code Review question and Jake Archibald’s caching article . I rebuilt each case from rereading it. My routes were all GET-only and public, so the POST and private cases never fired, and I have no logs showing what the others cost anyone.

A decorator with one shared entry per representation

from functools import wraps
from urllib.parse import urlencode

from flask import make_response, request
from flask_caching import Cache

cache = Cache()


def cached(params=(), offers=("application/json",), server_timeout=300):
 """Server-cache a public view. Declare the query parameters it reads."""

 def decorator(view):
 @wraps(view)
 def wrapper(*args, **kwargs):
 if (
 request.method not in ("GET", "HEAD")
 or request.authorization
 or request.cookies
 ):
 return view(*args, **kwargs)

 query = urlencode([(p, request.args.getlist(p)) for p in params], doseq=True)
 media_type = request.accept_mimetypes.best_match(offers, offers[0])
 key = f"view/{request.path}?{query}#{media_type}"

 response = cache.get(key)
 if response is None:
 response = make_response(view(*args, **kwargs))
 if response.status_code != 200:
 return response
 response.add_etag()
 response.cache_control.public = True
 response.cache_control.no_cache = True
 response.vary.add("Accept")
 cache.set(key, response, timeout=server_timeout)
 return response.make_conditional(request)

 return wrapper

 return decorator

Both decorators show the same probes, identical on both version sets:

Probe	Old decorator	Fixed decorator
POST after GET	Cached GET body, view skipped	View runs
Same URL, cookies for two users	Second user sees the first user’s page	Each sees their own
Anonymous request after a logged-in one	200 with the logged-in body	401
JSON, then `Accept: text/turtle`	JSON for both	Turtle for the second
100 requests with `Cache-Control: no-cache`	100 view runs	0 after the first
100 requests with a junk `?x=N`	100 view runs, 100 entries	0 after the first, 1 entry
`If-None-Match` with the `W/` form of the tag	200	304
304 headers	No `ETag`, `X-Last-Modified: None`	`ETag`, `Cache-Control`, `Vary`

Two caveats. Bypassing any cookie is blunt, since one analytics cookie turns the cache off for that browser, so narrow it to your session cookie. And params must list every argument the view reads, or two different URLs collide again. For per-user caching, put the user ID in the key and send private.

`no-store` and `private` don’t bind your own cache

The bare @cached() sent Cache-Control: no-store and still called cache.set:

GET /nostore 200 Cache-Control: no-store X-Cache: MISS
GET /nostore 200 Cache-Control: no-store X-Cache: HIT from Server

no-store and private are instructions to the browser and to proxies. An entry your application writes to Redis is outside their reach. The arguments cacheable=False and client_only=True read like server switches and only picked a header. With private, the second user received hello alice from the shared entry.

Decorator order makes it worse. With @cached above a login check, a hit never reaches the check: after one logged-in request, an anonymous client got a 200 with account of alice.

The method guard covered writes only

request.method in ["GET", "HEAD"] protected cache.set. The cache.get above it had no guard. A POST on a cold cache runs the view, so a test that only sends POST passes.

Flask-Caching’s @cache.cached(timeout=60) did the same on a GET and POST route, in both versions. Add unless=lambda: request.method != "GET" there.

Any client could pick the miss

if response is not None and request.headers.get("Cache-Control", "") != "no-cache":

The comment says “respect the hard-refresh”. It also hands every client a way to run the expensive view on demand: 100 requests, 100 runs. The comparison is exact, so 100 requests with no-cache, no-store or max-age=0 ran the view once. A junk query parameter does the same and adds an entry per request.

Your application’s cache is part of the origin, and no RFC asks it to obey a client’s request header. Invalidate from the server with cache.delete. The price: a hard refresh shows the cached body until server_timeout passes.

One typo, one bytes object

The 304 branch read a header that was never written:

response.headers.add("X-Last-Modified", str(now))
headers["X-Last-Modified"] = response.headers.get("X-LastModified") # None

Werkzeug turns None into a string, so every 304 carried X-Last-Modified: None and no error. It also lacked the ETag that RFC 9110 requires on a 304.

The tag was binascii.hexlify(os.urandom(4)), which returns bytes:

Werkzeug	`ETag` value
2.2.3 to 2.3.8	`a79d7c42`
3.0.0 and 3.1.9	`b'77155122'`

Neither has the double quotes that the grammar requires. Both round-trip through If-None-Match unchanged, so the change in Werkzeug 3.0 raises no error. The comparison was a literal ==, where If-None-Match calls for weak comparison , and a random tag changes every time the entry refills even if the body doesn’t. add_etag() hashes the body and make_conditional() does the comparison.

`no-cache` means revalidate

The old comments had this right: “no-cache doesn’t mean ‘don’t cache’” and “must-revalidate doesn’t mean ‘must revalidate’”.

Directive	Stored?	Reused without asking the server?
`no-store`	No	No
`no-cache`	Yes	No
`must-revalidate, max-age=900`	Yes	For 900 s, then it must revalidate

Then every call site passed must_revalidate=True, client_timeout=900, which sent no-cache, public, max-age=900. An unqualified no-cache forces validation on every reuse, so the 900 seconds bought nothing. I had the explanation in the file and still passed both.

Takeaway

A server-side entry goes to everyone whose request maps to its key, so list what the response depends on and put each item in the key or refuse to cache.

Try it on your own cached views: GET a URL, POST to it, then GET it with another user’s cookie, and fail the test if any two bodies match.

Python on Lessons learned

A CVE cross-reference is not an advisory entry

How the wrong relationship gets created

Filter by document role before matching identifiers

A prefix guard has a document contract

OpenPBS array resume: shard by stable task ID

A pending list changes between starts

Preserve ownership while filtering completion

Test different snapshots, then audit the run

Decode a MikroTik autosupout.rif yourself: 20 lines of Python found the out-of-memory kernel panic

The setup

The file format

The decoder

What it showed

Read the timestamps with care

Before you send it to anyone

Takeaway

macOS SMB: OSError: [Errno 9] Bad file descriptor after a silent reconnect

The setup

The dead end

The kernel log

Why

What fixed it

Takeaway

A second launch of a resumable worker: 'OSError: [Errno 39] Directory not empty'

The waiting wrapper had not disappeared

Recovery required checking the survivor

Claim ownership before destructive resume

Zeep returns an empty list: a repeated xs:sequence lands in _value_1

Setup

What went wrong

Fix

Test the parsing with real XML

Takeaway

A dedup lookup that fails reads as "not found": urllib.parse.quote() is not query escaping

Setup

What went wrong

Fix

Why

Takeaway

HTTP 500 that still commits: never retry an ambiguous write

Setup

What went wrong

The policy didn’t know the method

The safety nets sat outside the loop

Fix

Why

Takeaway

Date the bug before you park it: fence, drift or silence

The setup

What parking costs

Date the bug

The obvious fix was wrong

What silence looks like

The skill

Takeaway

Every test mocked the helper, and except Exception swallowed its TypeError on every PDF

Setup

What went wrong

Fix

Why

Takeaway

Tilde pointed at a profile home, so the export landed elsewhere

Two homes, two plausible directory trees

Check the process before searching for files

Shell expansion happens before the child starts

Pin the interpreter and verify the artifact

If GPG is missing, signature verification must fail closed

Optional crypto turns verification into best effort

Archive extraction can turn bypass into a file write

Make verification a prerequisite state

rdflib field notes: ns.format matches nothing, and parse() goes online

Use ns["format"] for names that str already owns

A typo in the base IRI is still a valid IRI

A failed parse keeps what it already read

parse() can go online

Keep a workaround’s reason in a test

Dataset isn’t a drop-in for ConjunctiveGraph

Takeaway

Celery ignored task_task_routes: silent failures from growing a crawler on Redis

Use `ns["format"]` for names that `str` already owns

`parse()` can go online

`Dataset` isn’t a drop-in for `ConjunctiveGraph`

`LLEN` 0 doesn’t mean the crawl is done

`decode_responses=True` on a shared pool breaks a binary cache

`execute()` drains a generator, `group()` doesn’t

A bare `except:` cancels the limit

Re-raise `TaskPredicate` before `except Exception`

What `except Exception` does to a retry

Flat `OPTIONAL`s join an unbound variable against everything

Send one `CONSTRUCT` per named graph, then read your copy

`SPARQLStore` hides what the endpoint tells you

Celery `chunks` runs your task as a plain function

`no-store` and `private` don’t bind your own cache

`no-cache` means revalidate