Http on Lessons learned

HTTP 500 that still commits: never retry an ambiguous write

Wed, 22 Jul 2026 22:29:00 +0200

We ran the client’s real retry loop against a fake server and counted the writes on the wire, before and after the fix:

before: push PUT 500 -> 200 writes_on_wire=2 -> reported ok
after: push PUT 500 -> 200 writes_on_wire=1 -> FlexiBeeError(maybe_committed=True)

The fake server played what the real API does now and then: it answers a PUT with 500 Internal Server Error after it has already saved the invoice. The client retried; the second PUT came back 200 OK, and the batch reported success. On a real API, that invoice is on the books twice. If your API can return a 5xx after committing, a retry that succeeds is the worst outcome you can get: you end up with a duplicate and nothing in the logs.

Setup

The API was the REST interface of ABRA Flexi, an ERP, called from Python through urllib and http.client. It returns 500 now and then after the write has gone through. That much was known; one client already looked the record up by document number after a failure.

Five hand-written clients for that API were merged into one transport layer with a shared RetryPolicy. An AI coding agent did the merge, two commits after writing an ADR that warned about duplicate writes.

What went wrong

The policy didn’t know the method

RetryPolicy decided from the response alone, so it retried a 500 on a PUT the same way it retried one on a GET. The test for that case fed in three 500s and checked only the flag on the final error. It pinned the number of retries and never asked whether a write should be retried in the first place.

The safety nets sat outside the loop

The first fix made the method a required argument and turned off retries of ambiguous writes by default. Two callers opted back in, because each had a safety net. In outline:

# MCP server: an idempotency key per tool call
def write_tool(key, payload):
 if idem_seen(key): # checked once, before the first attempt
 return cached_result(key)
 result = request("PUT", url, payload) # retries happen in here
 idem_record(key, result) # recorded only after success
 return result

# webhook worker: look the record up when the write fails
def upsert(doc):
 try:
 return request("PUT", url, doc) # retries happen in here
 except ApiError:
 if find_by_number(doc.number):
 return "committed-via-verify"
 raise

A review five days later, by a different model, caught it: “_idem_seen runs before this function, and _idem_record runs only after success, so neither prevents this retry from duplicating a write.” Likewise, the GET check runs only when request() raises, and a 500 followed by a successful retry never raises. Both checks were correct; the retry loop inside request() simply never consulted them.

Fix

Writes are never retried on an ambiguous outcome, and no caller can opt back in. The error goes up to the caller and says the write may have committed. The policy, trimmed to the part that matters:

WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}

class RetryPolicy:
 def __init__(self, method: str): # required: a default would let the next client forget it
 self._is_write = method.upper() in WRITE_METHODS

 def next_retry(self, status: int, body: str):
 if is_rate_limited(status, body): # the server refused: nothing committed
 return self._backoff("rate-limit")
 if 500 <= status < 600 and not self._is_write:
 return self._backoff("transient-5xx")
 return None # a 5xx on a write: raise, maybe_committed=True

 def next_retry_for_network_error(self):
 # no response at all: the server may have processed it before the connection died
 return None if self._is_write else self._backoff("network")

The new test counts requests, because a retry that succeeds looks exactly like a first attempt that did:

def test_500_on_a_write_is_never_retried(self):
 fake = FakeConn([FakeResponse(500, b"boom"), FakeResponse(200, b"{}")])
 with patch.object(client, "_get_conn", return_value=fake):
 with self.assertRaises(ApiError) as cm:
 client.put("invoice.json", {"invoice": [{}]})
 self.assertEqual(len(fake.requests_made), 1)
 self.assertTrue(cm.exception.maybe_committed)

Reads still retry, and they make up most of the traffic in a batch. A write that comes back maybe_committed is checked by whoever started it, before anything resends it.

Why

The policy asked, “may I retry a 5xx?” For a write, the question is “may I resend something whose outcome is unknown?” Two outcomes leave a write ambiguous: a 5xx, and no response at all. A 429, or a 400 rejecting a stale user role, is the server refusing the write; nothing committed, so a retry can’t duplicate anything.

The HTTP method doesn’t settle it either. RFC 9110 calls PUT idempotent, and urllib3’s default allowed_methods includes PUT, so a status_forcelist=[500, 502, 503] resends it. This API’s PUT to a collection creates a record each time.

If you do want writes retried automatically, the check has to run between the failure and the resend: before every attempt, look the record up or send a key the server itself deduplicates on. A check before the first call or after the last one can’t see the retries in between.

Takeaway

A 500 on a write means you don’t know what happened, so pass that up the stack instead of resending. Test it by counting requests on the wire, since the final result looks the same either way.

Find the retry loop in your own HTTP client and check whether it knows the method it is retrying.

Flask @cached decorator traps: POST gets the cached GET, no-store is stored, and X-Last-Modified: None

Fri, 19 Oct 2018 21:26:00 +0200

Update 2026-11-08: I rebuilt every repro on Flask 2.2.3 with Werkzeug 2.2.3 and Flask-Caching 2.0.2, and on Flask 3.1.3 with Werkzeug 3.1.9 and Flask-Caching 2.5.1, under Python 3.11.

A Flask view decorator I kept for years failed in a way that only showed up when a POST got the body of an earlier GET. The view never ran, and the client got a 200:

GET /item 200 "GET ran" X-Cache: MISS
POST /item 200 "GET ran" X-Cache: HIT from Server

The cache key was request.full_path, so any two requests with the same URL shared an entry, regardless of method, user, or Accept header. To fix this, let only anonymous GET and HEAD requests touch the shared entry, build the key from the response inputs, and leave the HTTP headers to Werkzeug.

The decorator entered a 2018 project with a docstring pointing at a Code Review question and Jake Archibald’s caching article . I rebuilt each case from rereading it. My routes were all GET-only and public, so the POST and private cases never fired, and I have no logs showing what the others cost anyone.

A decorator with one shared entry per representation

from functools import wraps
from urllib.parse import urlencode

from flask import make_response, request
from flask_caching import Cache

cache = Cache()


def cached(params=(), offers=("application/json",), server_timeout=300):
 """Server-cache a public view. Declare the query parameters it reads."""

 def decorator(view):
 @wraps(view)
 def wrapper(*args, **kwargs):
 if (
 request.method not in ("GET", "HEAD")
 or request.authorization
 or request.cookies
 ):
 return view(*args, **kwargs)

 query = urlencode([(p, request.args.getlist(p)) for p in params], doseq=True)
 media_type = request.accept_mimetypes.best_match(offers, offers[0])
 key = f"view/{request.path}?{query}#{media_type}"

 response = cache.get(key)
 if response is None:
 response = make_response(view(*args, **kwargs))
 if response.status_code != 200:
 return response
 response.add_etag()
 response.cache_control.public = True
 response.cache_control.no_cache = True
 response.vary.add("Accept")
 cache.set(key, response, timeout=server_timeout)
 return response.make_conditional(request)

 return wrapper

 return decorator

Both decorators show the same probes, identical on both version sets:

Probe	Old decorator	Fixed decorator
POST after GET	Cached GET body, view skipped	View runs
Same URL, cookies for two users	Second user sees the first user’s page	Each sees their own
Anonymous request after a logged-in one	200 with the logged-in body	401
JSON, then `Accept: text/turtle`	JSON for both	Turtle for the second
100 requests with `Cache-Control: no-cache`	100 view runs	0 after the first
100 requests with a junk `?x=N`	100 view runs, 100 entries	0 after the first, 1 entry
`If-None-Match` with the `W/` form of the tag	200	304
304 headers	No `ETag`, `X-Last-Modified: None`	`ETag`, `Cache-Control`, `Vary`

Two caveats. Bypassing any cookie is blunt, since one analytics cookie turns the cache off for that browser, so narrow it to your session cookie. And params must list every argument the view reads, or two different URLs collide again. For per-user caching, put the user ID in the key and send private.

`no-store` and `private` don’t bind your own cache

The bare @cached() sent Cache-Control: no-store and still called cache.set:

GET /nostore 200 Cache-Control: no-store X-Cache: MISS
GET /nostore 200 Cache-Control: no-store X-Cache: HIT from Server

no-store and private are instructions to the browser and to proxies. An entry your application writes to Redis is outside their reach. The arguments cacheable=False and client_only=True read like server switches and only picked a header. With private, the second user received hello alice from the shared entry.

Decorator order makes it worse. With @cached above a login check, a hit never reaches the check: after one logged-in request, an anonymous client got a 200 with account of alice.

The method guard covered writes only

request.method in ["GET", "HEAD"] protected cache.set. The cache.get above it had no guard. A POST on a cold cache runs the view, so a test that only sends POST passes.

Flask-Caching’s @cache.cached(timeout=60) did the same on a GET and POST route, in both versions. Add unless=lambda: request.method != "GET" there.

Any client could pick the miss

if response is not None and request.headers.get("Cache-Control", "") != "no-cache":

The comment says “respect the hard-refresh”. It also hands every client a way to run the expensive view on demand: 100 requests, 100 runs. The comparison is exact, so 100 requests with no-cache, no-store or max-age=0 ran the view once. A junk query parameter does the same and adds an entry per request.

Your application’s cache is part of the origin, and no RFC asks it to obey a client’s request header. Invalidate from the server with cache.delete. The price: a hard refresh shows the cached body until server_timeout passes.

One typo, one bytes object

The 304 branch read a header that was never written:

response.headers.add("X-Last-Modified", str(now))
headers["X-Last-Modified"] = response.headers.get("X-LastModified") # None

Werkzeug turns None into a string, so every 304 carried X-Last-Modified: None and no error. It also lacked the ETag that RFC 9110 requires on a 304.

The tag was binascii.hexlify(os.urandom(4)), which returns bytes:

Werkzeug	`ETag` value
2.2.3 to 2.3.8	`a79d7c42`
3.0.0 and 3.1.9	`b'77155122'`

Neither has the double quotes that the grammar requires. Both round-trip through If-None-Match unchanged, so the change in Werkzeug 3.0 raises no error. The comparison was a literal ==, where If-None-Match calls for weak comparison , and a random tag changes every time the entry refills even if the body doesn’t. add_etag() hashes the body and make_conditional() does the comparison.

`no-cache` means revalidate

The old comments had this right: “no-cache doesn’t mean ‘don’t cache’” and “must-revalidate doesn’t mean ‘must revalidate’”.

Directive	Stored?	Reused without asking the server?
`no-store`	No	No
`no-cache`	Yes	No
`must-revalidate, max-age=900`	Yes	For 900 s, then it must revalidate

Then every call site passed must_revalidate=True, client_timeout=900, which sent no-cache, public, max-age=900. An unqualified no-cache forces validation on every reuse, so the 900 seconds bought nothing. I had the explanation in the file and still passed both.

Takeaway

A server-side entry goes to everyone whose request maps to its key, so list what the response depends on and put each item in the key or refuse to cache.

Try it on your own cached views: GET a URL, POST to it, then GET it with another user’s cookie, and fail the test if any two bodies match.

Http on Lessons learned

HTTP 500 that still commits: never retry an ambiguous write

Setup

What went wrong

The policy didn’t know the method

The safety nets sat outside the loop

Fix

Why

Takeaway

Flask @cached decorator traps: POST gets the cached GET, no-store is stored, and X-Last-Modified: None

A decorator with one shared entry per representation

no-store and private don’t bind your own cache

The method guard covered writes only

Any client could pick the miss

One typo, one bytes object

no-cache means revalidate

Takeaway

`no-store` and `private` don’t bind your own cache

`no-cache` means revalidate