Idempotency on Lessons learned

A dedup lookup that fails reads as "not found": urllib.parse.quote() is not query escaping

Thu, 23 Jul 2026 01:05:00 +0200

An OCR’d invoice number with an apostrophe in it, say O'BRIEN-1, went into a duplicate check like this one (trimmed):

flt = urllib.parse.quote(f"firma='{firma}' and cisDosle='{cis_dosle}'", safe="")
try:
 r = self.get(f"faktura-prijata/({flt}).json?detail=custom:id,kod&limit=1")
 entries = r.get("winstrom", {}).get("faktura-prijata", [])
 return entries[0] if entries else None
except FlexiBeeError:
 return None

The server rejected the filter with HTTP 400 (“Špatný formát WQL”, bad WQL format). The except turned that into None, the caller read None as “no such invoice yet”, and pushed it again: a second copy of the invoice in the books, and no error anywhere.

If your client puts a filter expression into a URL and wraps it in quote(), the apostrophe problem is probably yours too.

Setup

The API is the REST interface of ABRA Flexi, a Czech ERP, called from Python’s standard library. Its filters are a small query language (WQL) carried in the URL path: evidence/(field='value').json. The ?filter= query-string form is silently ignored, so the path it is used instead.

The values come from OCR and a language model. The invoice number was normalized for dashes and whitespace only, so an apostrophe made it into the filter intact.

What went wrong

quote() did its job. It turned the expression into something safe to put in a URL:

>>> urllib.parse.quote("cisDosle='O'BRIEN-1'", safe="")
'cisDosle%3D%27O%27BRIEN-1%27'

The server URL-decodes the path before it parses the filter, so the parser gets back cisDosle='O'BRIEN-1'. The literal ends after O; the rest is garbage, and the query fails.

Percent-encoding protects the URL, and the server undoes it before the filter language ever sees the value.

Every client in the codebase built its filters this way, including the MCP server. Two call sites even disagreed on safe= ("'=" versus ""), which made no difference, since the server decodes either form back to the same bytes.

The failure stayed hidden because of the except. A lookup that can’t answer reported “not found”, and for a duplicate check, “not found” means “write it”.

Fix

First, find out what the grammar’s escape actually is. No client library for this API escapes it (the PHP one interpolates values raw), so there was nothing to copy. A wrong guess would cause the very duplicate the lookup is there to prevent, so it was settled by asking the server, read-only, in a sandbox company, with values that can’t match any record:

cisDosle='ZZZTEST' balanced -> 200
cisDosle='ZZZ'TEST' raw (the bug) -> 400 Špatný formát WQL … pozice 15
cisDosle='ZZZ\'TEST' backslash -> 200
cisDosle='ZZZ''TEST' SQL-style doubling -> 400
cisDosle='A\\B' doubled backslash -> 400
cisDosle='A\B' lone backslash -> 400

So this grammar escapes an apostrophe with a backslash, the way C does, not by doubling it, the way SQL and OData do. A backslash escapes nothing else, which means a literal backslash can’t be written at all. That’s acceptable for invoice numbers, codes, company IDs, and dates, which never carry one.

Second, build every filter in one module, so the escape exists exactly once:

import urllib.parse

def escape_literal(value: object) -> str:
 return str(value).replace("'", "\\'")

def eq(field: str, value: object) -> str:
 return f"{field}='{escape_literal(value)}'"

def and_(*clauses: str) -> str:
 return " and ".join(c for c in clauses if c)

def evidence_path(evidence: str, expr: str) -> str:
 return f"{evidence}/({urllib.parse.quote(expr, safe='')}).json"

path = evidence_path("faktura-prijata", and_(eq("firma", firma), eq("cisDosle", cis_dosle)))

escape_literal handles the query language and quote handles the URL, each in one place. Against the sandbox, the real client with this filter now gets a 200 for an invoice number with an apostrophe.

Before the fix, the gap was parked as an expectedFailure test. It checks that the quotes in the generated filter balance once either escape form is removed, so it would pass whichever escape the probes picked, and it flipped to an unexpected pass the moment the fix landed.

Why

Any value that goes into a query language needs that language’s escape, and then the transport’s. Percent-encoding is the transport’s. OData $filter and most search APIs set the same trap with their own escape rules, so probe the server instead of guessing.

The other half is the except. The finder in this codebase still maps an error to None, and that remains on its list of known issues. If you write a duplicate check, give it three outcomes: found, not found, and couldn’t tell, and make the caller stop on the third.

An earlier bug in the same codebase took the other route to the same end: a bank-movement duplicate check used LIKE, which this API doesn’t support. The server didn’t reject it; it returned an empty list, so every movement was pushed a second time. The fix was the API’s BEGINS operator, plus a one-off script to delete the copies.

Takeaway

Escape a value for the query language first and for the URL second; quote() only does the second. A duplicate check that turns errors into “not found” creates a duplicate every time it breaks.

Search your code for quote(f" and check each hit for a quoted literal built from user or OCR input.

HTTP 500 that still commits: never retry an ambiguous write

Wed, 22 Jul 2026 22:29:00 +0200

We ran the client’s real retry loop against a fake server and counted the writes on the wire, before and after the fix:

before: push PUT 500 -> 200 writes_on_wire=2 -> reported ok
after: push PUT 500 -> 200 writes_on_wire=1 -> FlexiBeeError(maybe_committed=True)

The fake server played what the real API does now and then: it answers a PUT with 500 Internal Server Error after it has already saved the invoice. The client retried; the second PUT came back 200 OK, and the batch reported success. On a real API, that invoice is on the books twice. If your API can return a 5xx after committing, a retry that succeeds is the worst outcome you can get: you end up with a duplicate and nothing in the logs.

Setup

The API was the REST interface of ABRA Flexi, an ERP, called from Python through urllib and http.client. It returns 500 now and then after the write has gone through. That much was known; one client already looked the record up by document number after a failure.

Five hand-written clients for that API were merged into one transport layer with a shared RetryPolicy. An AI coding agent did the merge, two commits after writing an ADR that warned about duplicate writes.

What went wrong

The policy didn’t know the method

RetryPolicy decided from the response alone, so it retried a 500 on a PUT the same way it retried one on a GET. The test for that case fed in three 500s and checked only the flag on the final error. It pinned the number of retries and never asked whether a write should be retried in the first place.

The safety nets sat outside the loop

The first fix made the method a required argument and turned off retries of ambiguous writes by default. Two callers opted back in, because each had a safety net. In outline:

# MCP server: an idempotency key per tool call
def write_tool(key, payload):
 if idem_seen(key): # checked once, before the first attempt
 return cached_result(key)
 result = request("PUT", url, payload) # retries happen in here
 idem_record(key, result) # recorded only after success
 return result

# webhook worker: look the record up when the write fails
def upsert(doc):
 try:
 return request("PUT", url, doc) # retries happen in here
 except ApiError:
 if find_by_number(doc.number):
 return "committed-via-verify"
 raise

A review five days later, by a different model, caught it: “_idem_seen runs before this function, and _idem_record runs only after success, so neither prevents this retry from duplicating a write.” Likewise, the GET check runs only when request() raises, and a 500 followed by a successful retry never raises. Both checks were correct; the retry loop inside request() simply never consulted them.

Fix

Writes are never retried on an ambiguous outcome, and no caller can opt back in. The error goes up to the caller and says the write may have committed. The policy, trimmed to the part that matters:

WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}

class RetryPolicy:
 def __init__(self, method: str): # required: a default would let the next client forget it
 self._is_write = method.upper() in WRITE_METHODS

 def next_retry(self, status: int, body: str):
 if is_rate_limited(status, body): # the server refused: nothing committed
 return self._backoff("rate-limit")
 if 500 <= status < 600 and not self._is_write:
 return self._backoff("transient-5xx")
 return None # a 5xx on a write: raise, maybe_committed=True

 def next_retry_for_network_error(self):
 # no response at all: the server may have processed it before the connection died
 return None if self._is_write else self._backoff("network")

The new test counts requests, because a retry that succeeds looks exactly like a first attempt that did:

def test_500_on_a_write_is_never_retried(self):
 fake = FakeConn([FakeResponse(500, b"boom"), FakeResponse(200, b"{}")])
 with patch.object(client, "_get_conn", return_value=fake):
 with self.assertRaises(ApiError) as cm:
 client.put("invoice.json", {"invoice": [{}]})
 self.assertEqual(len(fake.requests_made), 1)
 self.assertTrue(cm.exception.maybe_committed)

Reads still retry, and they make up most of the traffic in a batch. A write that comes back maybe_committed is checked by whoever started it, before anything resends it.

Why

The policy asked, “may I retry a 5xx?” For a write, the question is “may I resend something whose outcome is unknown?” Two outcomes leave a write ambiguous: a 5xx, and no response at all. A 429, or a 400 rejecting a stale user role, is the server refusing the write; nothing committed, so a retry can’t duplicate anything.

The HTTP method doesn’t settle it either. RFC 9110 calls PUT idempotent, and urllib3’s default allowed_methods includes PUT, so a status_forcelist=[500, 502, 503] resends it. This API’s PUT to a collection creates a record each time.

If you do want writes retried automatically, the check has to run between the failure and the resend: before every attempt, look the record up or send a key the server itself deduplicates on. A check before the first call or after the last one can’t see the retries in between.

Takeaway

A 500 on a write means you don’t know what happened, so pass that up the stack instead of resending. Test it by counting requests on the wire, since the final result looks the same either way.

Find the retry loop in your own HTTP client and check whether it knows the method it is retrying.