Redis on Lessons learned

Celery ignored task_task_routes: silent failures from growing a crawler on Redis

Sat, 11 Mar 2023 12:19:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

This sat in my crawler’s Celery config:

task_task_routes = {
 "tasks.urgent": {"queue": "celery"},
 "tasks.post": {"queue": "celery:2"},
}

The setting is called task_routes. Celery doesn’t check setting names, so the app started normally, app.conf.task_routes stayed None, and three different tasks all landed in the default queue:

{'celery:1': 3}

That typo arrived with the commit that moved the crawler to Redis priority queues, and it wasn’t the only failure from that stretch that raised nothing. Two small changes catch most of them: compare your settings against Celery’s list of known names, and set task_create_missing_queues = False.

I found these by rereading the code and rebuilt each one against a private Redis, on Python 3.11 with the locked versions (Celery 5.2.7, kombu 5.2.4, redis-py 4.5.4) and on Celery 5.7.0. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

How the crawler got here

Date	Change
2018	Celery on Redis, with the crawl state in Redis too
2020	Mongo for reports; separate priority and regular tasks
2021-09	A monitor task that decides when the crawl is done
2023-02-03	Large structures move from Redis to SQLite, then to Postgres the same night
2023-02-07	Mongo dropped
2023-02-10	Bulk inserts from generators
2023-02-12	Named priority queues; each input parsed into an on-disk graph in a temp directory

Redis ended up mostly as the broker and a few caches. The traps below surfaced in the 2023 part of that table.

Celery ignores settings it doesn’t know

A test finds the typo. DEFAULTS holds every setting name Celery recognizes:

from types import ModuleType

from celery.app.defaults import DEFAULTS

import celeryconfig


def test_celery_settings_are_known():
 unknown = {
 name for name, value in vars(celeryconfig).items()
 if name.islower() and not name.startswith("_")
 and not isinstance(value, ModuleType)
 and name not in DEFAULTS
 }
 assert not unknown

On my config, it reports {'task_task_routes'}, on both Celery versions.

A queue with no consumer looks like a finished crawl

The same commit changed what the worker reads, and the post-processing chain kept an old queue name:

celery -A app worker -Q celery,celery:1,celery:2

chain(step_one.si(), step_two.si()).apply_async(queue="query")

task_create_missing_queues is on by default, so sending to query creates it. In the rebuild, the message sat in that key, and no step ran. Declare your queues and turn creation off:

from kombu import Queue

task_create_missing_queues = False
task_queues = [Queue("celery"), Queue("bulk")]

celery.exceptions.QueueNotFound: "Queue 'query' missing from task_queues"

That protects the sending side only. The -Q list on the worker is still a separate copy, so build it from the same list.

broker_transport_options = {
 "priority_steps": [0, 1, 2],
 "sep": ":",
 "queue_order_strategy": "priority",
}

With these options, kombu stores priority 1 of queue celery queue under the Redis key celery:1. I also named a queue celery:1. A message sent with queue="celery", priority=1 showed up in that key. Priority 0 gets no suffix, so celery:0 never exists.

Pick one scheme: a single queue with priority=, or named queues whose names don’t contain the separator.

`LLEN` 0 doesn’t mean the crawl is done

Every ten minutes, the monitor computed this and declared the crawl done at zero:

enqueued = red.llen("celery") + red.llen("celery:0") + red.llen("celery:1")

It counts the key that never exists and skips celery:2. Even with the right keys, a list length misses work:

State	Monitor	Lists + `HLEN unacked`
Task waiting in `celery:2`	0	1
Task running, `task_acks_late=True`	0	1
Retry waiting on a 6 s countdown	0	1
Chain stranded in `query`	0	0

def outstanding(client, queues=("celery", "celery:1", "celery:2")):
 with client.pipeline() as pipe:
 for queue in queues:
 pipe.llen(queue)
 pipe.hlen("unacked")
 return sum(pipe.execute())

unacked is kombu’s internal bookkeeping hash, and it shows running tasks only because my config sets task_acks_late=True. With early acknowledgment, a running task has already left it. The last row shows the limit of any count: it can’t see a queue you forgot to list.

When you know the fan-out up front, a chord gives a real completion signal. The docs say members must not ignore their results, and I copied that warning into a comment. On the Redis backend, a chord of ignore_result=True members still ran their bodies with all three values, on both versions. I’d follow the docs anyway, since I tested one backend. My tasks also spawned more tasks, which a chord can’t express.

`decode_responses=True` on a shared pool breaks a binary cache

In 2021, my HTTP cache shared the application’s Redis pool, which decoded responses to strings. requests-cache stores pickles, and a pickle starts with byte 0x80. Two identical GETs:

requests-cache	Decoding pool	Separate pool
0.9.8	2 origin hits, one log line	1 origin hit
1.3.3	Second GET raises `UnicodeDecodeError`	1 origin hit

The log line on the old version is all you get:

Unable to deserialize response with key 53630720a0b4bd02: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

The commit that fixed it says what to do: “a separate Redis pool without decoding”:

app_pool = redis.ConnectionPool.from_url(url, decode_responses=True)
cache_pool = redis.ConnectionPool.from_url(url)

`execute()` drains a generator, `group()` doesn’t

I passed generators to session.execute(insert(Row), rows()) to save memory. SQLAlchemy reads them to the end first: on 2.0.7 and 2.1.8, with a SQLite engine, the generator had yielded 5,000 of 5,000 rows before the first INSERT. Batch it yourself:

from itertools import islice


def batches(items, size=1000):
 items = iter(items)
 while batch := list(islice(items, size)):
 yield batch


for batch in batches(rows()):
 session.execute(insert(Row), batch)
session.commit()

With 2,500 rows, the three INSERTs fired after 1,000, 2,000, and 2,500 yields.

group(generator).apply_async() works the other way and publishes as it iterates. With a generator that yielded two signatures and then raised, one task was already in Redis, and the caller got the exception with no GroupResult. Make those tasks safe to run twice. I still prefer group to chunks, which loses retries .

Takeaway

Most of this was introduced or fixed during ten days of refactoring in February 2023, and almost none of it raised an error then. A queue migration needs a written list of each queue, who sends to it, who reads it, and what counts it.

Try the settings test above on your own Celery config.

Celery self.retry(5) raises TypeError, and except Exception swallows the retry

Tue, 07 Feb 2023 00:01:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

My crawler called Celery’s retry three different ways, and only one did what I meant:

Call	Result
`self.retry(5)`	Task fails with the error below, no retry
`self.retry(timeout=5)`	Retries in 180 s
`self.retry(countdown=5)`	Retries in 5 s

TypeError: 'int' object is not iterable

Even the correct call can go wrong. retry() publishes the new message and then raises celery.exceptions.Retry, an ordinary Exception. An except Exception around it marks the work as failed while the retry is already on its way. Pass countdown= by keyword and let TaskPredicate pass through your broad handlers.

I found these by rereading old code and rebuilt each case on a real worker with a private Redis as broker and result backend, on Python 3.11. Both Celery versions behaved the same. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

Re-raise `TaskPredicate` before `except Exception`

from celery.exceptions import TaskPredicate


@app.task(bind=True, max_retries=2)
def crawl(self, url):
 try:
 fetch(self, url) # calls self.retry(exc=err, countdown=1) on a network error
 except TaskPredicate:
 raise
 except Exception:
 mark_failed(url)
 return "gave up"
 mark_ok(url)
 return "done"

TaskPredicate is the base class of Retry, Ignore and Reject. With this handler, result.get() returned 'done' about 1 s later, after the retry had run.

My handler already knew about one of the three:

except Exception as err:
 if isinstance(err, Ignore):
 raise

Retry needed the same treatment and didn’t get it.

Why the three calls differ

The signature on both versions is:

retry(self, args=None, kwargs=None, exc=None, throw=True, eta=None, countdown=None, max_retries=None, **options)

The first positional parameter is the replacement args for the retried task, so 5 reaches tuple(args) and raises before anything is published.

timeout isn’t a parameter, so it lands in **options and travels with the message. Spying on kombu’s Producer.publish showed timeout=5 arriving there, where it limits how long publishing may take. The delay falls back to default_retry_delay, three minutes by default . Nothing raises. The only hint is the worker log:

Task tasks.crawl[62d0a257-...] retry: Retry in 180s

What `except Exception` does to a retry

Remove the except TaskPredicate branch from the task above and let the first attempt hit a network error:

Time	Event
0.01 s	Retry published, `Retry` caught, URL marked failed
0.01 s	`result.get()` returns `'gave up'`, state `SUCCESS`
1.01 s	Retry runs and marks the URL ok
Later	The backend still says `'gave up'`

The last row surprised me. On the Redis backend, _store_result ignores any write once the stored state is SUCCESS, so the retry’s own return value is dropped. Anyone reading the result sees the attempt that gave up.

Code review won’t catch this reliably, because the placement decides it. A retry inside the try body, even several calls deep, is caught. A retry in a sibling except clause is safe:

try:
 fetch(task, url) # a retry raised in here hits except Exception
except RateLimited as err:
 task.retry(countdown=err.delay) # sibling handlers don't see this one
except Exception:
 mark_failed(url)

My code had both forms in the same function, so some retries worked. Writing raise self.retry(...) changes nothing, and a bare except: catches Retry too. In an older module, a bare except: also caught the TypeError from self.retry(5), which recorded a rate-limited link as a broken one.

A dedupe mark written first makes the retry skip itself

@app.task(bind=True, max_retries=2)
def crawl(self, url):
 if redis_client.sadd("seen", url) == 0:
 return "skipped"
 ...

attempt 0 sadd=1
attempt 1 sadd=0

The retry sees its own mark, returns 'skipped', and the URL is never fetched. Combine that with the swallowed Retry and the URL stays failed for good: in the rebuild, the final status was failed and the result was 'gave up'.

Moving the SADD after the work fixed it: the retry returned 'fetched'. Two concurrent tasks can then fetch the same URL, so the work has to be safe to repeat.

When the retry condition is just an exception type, autoretry_for=(Flaky,), retry_kwargs={"countdown": 1, "max_retries": 2} saves calling retry yourself. The exception has to leave the task body, so a broad handler inside it defeats this as well.

Time limits have a similar problem on the gevent pool, in the opposite direction: Timeout isn’t an Exception .

Takeaway

self.retry() is a raise whose side effect has already happened, so any handler between it and the worker can make your bookkeeping disagree with the queue.

Run grep -rn --include='*.py' '\.retry(' . on your tasks and check each hit for keyword arguments and an enclosing except Exception.

Redis on Lessons learned

Celery ignored task_task_routes: silent failures from growing a crawler on Redis

How the crawler got here

Celery ignores settings it doesn’t know

A queue with no consumer looks like a finished crawl

Redis priority steps and queue names share a namespace

LLEN 0 doesn’t mean the crawl is done

decode_responses=True on a shared pool breaks a binary cache

execute() drains a generator, group() doesn’t

Takeaway

Celery self.retry(5) raises TypeError, and except Exception swallows the retry

Re-raise TaskPredicate before except Exception

Why the three calls differ

What except Exception does to a retry

A dedupe mark written first makes the retry skip itself

Takeaway

`LLEN` 0 doesn’t mean the crawl is done

`decode_responses=True` on a shared pool breaks a binary cache

`execute()` drains a generator, `group()` doesn’t

Re-raise `TaskPredicate` before `except Exception`

What `except Exception` does to a retry