Celery on Lessons learned

Celery ignored task_task_routes: silent failures from growing a crawler on Redis

Sat, 11 Mar 2023 12:19:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

This sat in my crawler’s Celery config:

task_task_routes = {
 "tasks.urgent": {"queue": "celery"},
 "tasks.post": {"queue": "celery:2"},
}

The setting is called task_routes. Celery doesn’t check setting names, so the app started normally, app.conf.task_routes stayed None, and three different tasks all landed in the default queue:

{'celery:1': 3}

That typo arrived with the commit that moved the crawler to Redis priority queues, and it wasn’t the only failure from that stretch that raised nothing. Two small changes catch most of them: compare your settings against Celery’s list of known names, and set task_create_missing_queues = False.

I found these by rereading the code and rebuilt each one against a private Redis, on Python 3.11 with the locked versions (Celery 5.2.7, kombu 5.2.4, redis-py 4.5.4) and on Celery 5.7.0. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

How the crawler got here

Date	Change
2018	Celery on Redis, with the crawl state in Redis too
2020	Mongo for reports; separate priority and regular tasks
2021-09	A monitor task that decides when the crawl is done
2023-02-03	Large structures move from Redis to SQLite, then to Postgres the same night
2023-02-07	Mongo dropped
2023-02-10	Bulk inserts from generators
2023-02-12	Named priority queues; each input parsed into an on-disk graph in a temp directory

Redis ended up mostly as the broker and a few caches. The traps below surfaced in the 2023 part of that table.

Celery ignores settings it doesn’t know

A test finds the typo. DEFAULTS holds every setting name Celery recognizes:

from types import ModuleType

from celery.app.defaults import DEFAULTS

import celeryconfig


def test_celery_settings_are_known():
 unknown = {
 name for name, value in vars(celeryconfig).items()
 if name.islower() and not name.startswith("_")
 and not isinstance(value, ModuleType)
 and name not in DEFAULTS
 }
 assert not unknown

On my config, it reports {'task_task_routes'}, on both Celery versions.

A queue with no consumer looks like a finished crawl

The same commit changed what the worker reads, and the post-processing chain kept an old queue name:

celery -A app worker -Q celery,celery:1,celery:2

chain(step_one.si(), step_two.si()).apply_async(queue="query")

task_create_missing_queues is on by default, so sending to query creates it. In the rebuild, the message sat in that key, and no step ran. Declare your queues and turn creation off:

from kombu import Queue

task_create_missing_queues = False
task_queues = [Queue("celery"), Queue("bulk")]

celery.exceptions.QueueNotFound: "Queue 'query' missing from task_queues"

That protects the sending side only. The -Q list on the worker is still a separate copy, so build it from the same list.

broker_transport_options = {
 "priority_steps": [0, 1, 2],
 "sep": ":",
 "queue_order_strategy": "priority",
}

With these options, kombu stores priority 1 of queue celery queue under the Redis key celery:1. I also named a queue celery:1. A message sent with queue="celery", priority=1 showed up in that key. Priority 0 gets no suffix, so celery:0 never exists.

Pick one scheme: a single queue with priority=, or named queues whose names don’t contain the separator.

`LLEN` 0 doesn’t mean the crawl is done

Every ten minutes, the monitor computed this and declared the crawl done at zero:

enqueued = red.llen("celery") + red.llen("celery:0") + red.llen("celery:1")

It counts the key that never exists and skips celery:2. Even with the right keys, a list length misses work:

State	Monitor	Lists + `HLEN unacked`
Task waiting in `celery:2`	0	1
Task running, `task_acks_late=True`	0	1
Retry waiting on a 6 s countdown	0	1
Chain stranded in `query`	0	0

def outstanding(client, queues=("celery", "celery:1", "celery:2")):
 with client.pipeline() as pipe:
 for queue in queues:
 pipe.llen(queue)
 pipe.hlen("unacked")
 return sum(pipe.execute())

unacked is kombu’s internal bookkeeping hash, and it shows running tasks only because my config sets task_acks_late=True. With early acknowledgment, a running task has already left it. The last row shows the limit of any count: it can’t see a queue you forgot to list.

When you know the fan-out up front, a chord gives a real completion signal. The docs say members must not ignore their results, and I copied that warning into a comment. On the Redis backend, a chord of ignore_result=True members still ran their bodies with all three values, on both versions. I’d follow the docs anyway, since I tested one backend. My tasks also spawned more tasks, which a chord can’t express.

`decode_responses=True` on a shared pool breaks a binary cache

In 2021, my HTTP cache shared the application’s Redis pool, which decoded responses to strings. requests-cache stores pickles, and a pickle starts with byte 0x80. Two identical GETs:

requests-cache	Decoding pool	Separate pool
0.9.8	2 origin hits, one log line	1 origin hit
1.3.3	Second GET raises `UnicodeDecodeError`	1 origin hit

The log line on the old version is all you get:

Unable to deserialize response with key 53630720a0b4bd02: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

The commit that fixed it says what to do: “a separate Redis pool without decoding”:

app_pool = redis.ConnectionPool.from_url(url, decode_responses=True)
cache_pool = redis.ConnectionPool.from_url(url)

`execute()` drains a generator, `group()` doesn’t

I passed generators to session.execute(insert(Row), rows()) to save memory. SQLAlchemy reads them to the end first: on 2.0.7 and 2.1.8, with a SQLite engine, the generator had yielded 5,000 of 5,000 rows before the first INSERT. Batch it yourself:

from itertools import islice


def batches(items, size=1000):
 items = iter(items)
 while batch := list(islice(items, size)):
 yield batch


for batch in batches(rows()):
 session.execute(insert(Row), batch)
session.commit()

With 2,500 rows, the three INSERTs fired after 1,000, 2,000, and 2,500 yields.

group(generator).apply_async() works the other way and publishes as it iterates. With a generator that yielded two signatures and then raised, one task was already in Redis, and the caller got the exception with no GroupResult. Make those tasks safe to run twice. I still prefer group to chunks, which loses retries .

Takeaway

Most of this was introduced or fixed during ten days of refactoring in February 2023, and almost none of it raised an error then. A queue migration needs a written list of each queue, who sends to it, who reads it, and what counts it.

Try the settings test above on your own Celery config.

Celery time_limit on the gevent pool: a 1 s limit let a CPU task run 3 s

Wed, 08 Feb 2023 23:36:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 with gevent 22.10.2, and on Celery 5.7.0 with gevent 26.9.0.

I gave a Celery task time_limit=1 and ran it with -P gevent. The CPU-bound task kept going for 3 s, finished all its work, and was then reported as failed:

state=FAILURE result=TimeLimitExceeded(1) got_after=3.06s

The gevent pool enforces the limit with gevent.Timeout, which can only fire when the task yields to the event loop. CPU-bound code never does. Keep gevent for I/O and run CPU work on a prefork worker.

The workers guide says this in two sentences: the gevent pool “does not implement soft time limits” and “will not enforce the hard time limit if the task is blocking”. It leaves out what that looks like in a task, and how an exception handler makes it worse.

I found this by rereading old code and rebuilt each case on a real worker with a private Redis, on Python 3.11. Both version pairs behaved the same. I have no production logs from that time.

What a 1 s limit does on each pool

Each task does 3 s of work:

Task, 1 s limit	gevent	prefork
CPU loop	Ran the full 3 s, then `TimeLimitExceeded`	Stopped after about 1 s
`time.sleep`	Stopped after about 1 s	Stopped after about 1 s
Sleep inside `try` / `except Exception`	Handler skipped, stopped after about 1 s	Stopped after about 1 s
Sleep inside `try` / bare `except:`, then 3 s more	`SUCCESS` at 4.01 s	`FAILURE` after about 1 s
`soft_time_limit=1` with sleep	Ignored, `SUCCESS` at 3.01 s	`SoftTimeLimitExceeded` after about 1 s

The first row is the nasty one. The work completed, its side effects happened, and the task still reports failure. The timer was pending the whole time and fired at the first switch after the loop.

A CPU-bound task also stalls everything else in that worker process. With -c 8, a task sent during a 3 s CPU loop waited 2.8 s to start, against a few milliseconds on an idle worker. On prefork with -c 2, it still started within a few milliseconds.

Split fetch and parse into separate workers

app.conf.task_routes = {
 "tasks.fetch": {"queue": "fetch"},
 "tasks.parse": {"queue": "parse"},
}

chain(fetch.s(url), parse.s()).apply_async()

celery -A app worker -n fetch@%h -P gevent -c 60 -Q fetch
celery -A app worker -n parse@%h -P prefork -c 2 -Q parse

With time_limit=1 on parse, a chain with a 3 s CPU parse ended in TimeLimitExceeded after 1.1 s, and a 0.1 s parse succeeded. Pass a file path or an object-store key between the two rather than the body.

Why a timeout can’t interrupt CPU work

celery/concurrency/gevent.py in 5.7.0 wraps each task like this:

try:
 with Timeout(timeout):
 return apply_target(target, args, kwargs, callback,
 accept_callback, getpid(),
 propagate=(Timeout,), **rest)
except Timeout:
 return timeout_callback(False, timeout)

Timeout is a timer on the event loop, and everything runs in one thread. The loop gets control only when the current greenlet yields, typically at a monkey-patched socket call or sleep. Prefork stops a task from outside, and the hard limit replaces the child process.

The soft limit is simpler: on_apply takes soft_timeout through **_ and never uses it. The worker log says nothing about that.

A bare `except:` cancels the limit

gevent.Timeout derives from BaseException , so except Exception lets it pass and a bare except: catches it. My code had a fallback like this several calls below the task:

try:
 return lookup(url)
except:
 log.exception("lookup failed, trying the fallback")
 return fallback(url)

The timer fires once. After the handler ate it, the task had no limit at all, and in the rebuild it finished three seconds past the deadline as SUCCESS.

My fix at the time, a commit called “Fail the task on timeout”, added except Timeout at the top of the task. A Timeout stops at the nearest handler that matches, so the nested bare except: on the fallback path kept swallowing it. Change the nested handler:

except Exception:
 log.exception("lookup failed, trying the fallback")
 return fallback(url)

If you need cleanup, catch Timeout and re-raise it. In the rebuild, the cleanup ran at 1.01 s, and Celery recorded TimeLimitExceeded, so there was no need to set the state by hand.

The soft limit goes the other way on prefork: SoftTimeLimitExceeded is an Exception, so a task that moves between pools needs its handlers checked for both.

For a Python loop you own, gevent.sleep(0) on each iteration gives the loop a turn, and the task then stopped at 1.01 s. That can’t help inside a library call that never yields.

The sibling trap on the same crawler: Retry is an Exception, and except Exception eats it .

Takeaway

On the gevent pool, time_limit is only as good as the task’s willingness to yield, so treat it as a limit on I/O waits.

Run grep -rnE --include='*.py' 'except:[[:space:]]*$' . on anything that runs under -P gevent.

Celery self.retry(5) raises TypeError, and except Exception swallows the retry

Tue, 07 Feb 2023 00:01:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

My crawler called Celery’s retry three different ways, and only one did what I meant:

Call	Result
`self.retry(5)`	Task fails with the error below, no retry
`self.retry(timeout=5)`	Retries in 180 s
`self.retry(countdown=5)`	Retries in 5 s

TypeError: 'int' object is not iterable

Even the correct call can go wrong. retry() publishes the new message and then raises celery.exceptions.Retry, an ordinary Exception. An except Exception around it marks the work as failed while the retry is already on its way. Pass countdown= by keyword and let TaskPredicate pass through your broad handlers.

I found these by rereading old code and rebuilt each case on a real worker with a private Redis as broker and result backend, on Python 3.11. Both Celery versions behaved the same. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

Re-raise `TaskPredicate` before `except Exception`

from celery.exceptions import TaskPredicate


@app.task(bind=True, max_retries=2)
def crawl(self, url):
 try:
 fetch(self, url) # calls self.retry(exc=err, countdown=1) on a network error
 except TaskPredicate:
 raise
 except Exception:
 mark_failed(url)
 return "gave up"
 mark_ok(url)
 return "done"

TaskPredicate is the base class of Retry, Ignore and Reject. With this handler, result.get() returned 'done' about 1 s later, after the retry had run.

My handler already knew about one of the three:

except Exception as err:
 if isinstance(err, Ignore):
 raise

Retry needed the same treatment and didn’t get it.

Why the three calls differ

The signature on both versions is:

retry(self, args=None, kwargs=None, exc=None, throw=True, eta=None, countdown=None, max_retries=None, **options)

The first positional parameter is the replacement args for the retried task, so 5 reaches tuple(args) and raises before anything is published.

timeout isn’t a parameter, so it lands in **options and travels with the message. Spying on kombu’s Producer.publish showed timeout=5 arriving there, where it limits how long publishing may take. The delay falls back to default_retry_delay, three minutes by default . Nothing raises. The only hint is the worker log:

Task tasks.crawl[62d0a257-...] retry: Retry in 180s

What `except Exception` does to a retry

Remove the except TaskPredicate branch from the task above and let the first attempt hit a network error:

Time	Event
0.01 s	Retry published, `Retry` caught, URL marked failed
0.01 s	`result.get()` returns `'gave up'`, state `SUCCESS`
1.01 s	Retry runs and marks the URL ok
Later	The backend still says `'gave up'`

The last row surprised me. On the Redis backend, _store_result ignores any write once the stored state is SUCCESS, so the retry’s own return value is dropped. Anyone reading the result sees the attempt that gave up.

Code review won’t catch this reliably, because the placement decides it. A retry inside the try body, even several calls deep, is caught. A retry in a sibling except clause is safe:

try:
 fetch(task, url) # a retry raised in here hits except Exception
except RateLimited as err:
 task.retry(countdown=err.delay) # sibling handlers don't see this one
except Exception:
 mark_failed(url)

My code had both forms in the same function, so some retries worked. Writing raise self.retry(...) changes nothing, and a bare except: catches Retry too. In an older module, a bare except: also caught the TypeError from self.retry(5), which recorded a rate-limited link as a broken one.

A dedupe mark written first makes the retry skip itself

@app.task(bind=True, max_retries=2)
def crawl(self, url):
 if redis_client.sadd("seen", url) == 0:
 return "skipped"
 ...

attempt 0 sadd=1
attempt 1 sadd=0

The retry sees its own mark, returns 'skipped', and the URL is never fetched. Combine that with the swallowed Retry and the URL stays failed for good: in the rebuild, the final status was failed and the result was 'gave up'.

Moving the SADD after the work fixed it: the retry returned 'fetched'. Two concurrent tasks can then fetch the same URL, so the work has to be safe to repeat.

When the retry condition is just an exception type, autoretry_for=(Flaky,), retry_kwargs={"countdown": 1, "max_retries": 2} saves calling retry yourself. The exception has to leave the task body, so a broad handler inside it defeats this as well.

Time limits have a similar problem on the gevent pool, in the opposite direction: Timeout isn’t an Exception .

Takeaway

self.retry() is a raise whose side effect has already happened, so any handler between it and the worker can make your bookkeeping disagree with the queue.

Run grep -rn --include='*.py' '\.retry(' . on your tasks and check each hit for keyword arguments and an enclosing except Exception.

SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Tue, 14 Sep 2021 11:35:00 +0200

Update 2026-11-08: rebuilt every repro on rdflib 6.3.2 and 7.6.0, and on Celery 5.2.7 and 5.7.0.

I wanted a resource and three hops of its neighbors from a SPARQL endpoint. This query returns every triple in the store:

CONSTRUCT { <http://example.org/root> ?p1 ?o1 . ?o1 ?p2 ?o2 . ?o2 ?p3 ?o3 . }
WHERE {
 <http://example.org/root> ?p1 ?o1 .
 OPTIONAL { ?o1 ?p2 ?o2 . }
 OPTIONAL { ?o2 ?p3 ?o3 . }
}

Move the second OPTIONAL inside the first, and you get the four triples you meant. A comment in my 2021 crawler says nested optionals are a must, otherwise the query will not finish. This post explains why and covers other ways the crawler could mistreat an endpoint without noticing.

Everything below comes from rereading that code and rebuilding each case against a local fake endpoint, under Python 3.11. It shows what the code does. I have no production logs that say what it cost anyone.

Flat `OPTIONAL`s join an unbound variable against everything

The test store holds 20,005 triples: a root with a literal label, a four-hop chain, and 20,000 unrelated triples. Times are from one run.

Query shape	Solutions	Triples returned	Oxigraph 0.5.13	rdflib 7.6.0
Flat `OPTIONAL`s	20,006	20,005	28 ms	897 ms
Nested `OPTIONAL`s	2	4	1.0 ms	3.2 ms
No `OPTIONAL`	1	3	0.1 ms	1.7 ms

The literal label triggers it. When ?o1 is that literal, the first OPTIONAL can’t match, because a literal is never a subject, so ?o2 stays unbound. The second OPTIONAL then has nothing constraining it. ?o2 ?p3 ?o3 matches every triple, and the CONSTRUCT template writes them all out. Most resources have a label, so most lookups hit this.

Removing the OPTIONALs is the wrong repair: the label disappears, as the last row shows.

I can’t rerun the 2021 endpoint, and I only tested these two engines. I believe this is what “will not finish” meant, since on a store with millions of triples each lookup becomes a full dump.

A variable you never bind does the same thing. I once committed OPTIONALs on ?dataset in a query that bound ?ds, and renamed it 18 minutes later. It’s a valid query, so nothing raises an error.

Nesting fixes the unbound variable and does nothing for hubs. A node with a million outgoing triples still returns them all, so add a LIMIT if a partial answer is acceptable.

Send one `CONSTRUCT` per named graph, then read your copy

Graph(store=SPARQLStore(url)) looks like a local graph, and every value() call on it is an HTTP request. With 50 datasets and three property reads each:

Approach	Requests to the endpoint
Read through the store-backed graph	151
One `CONSTRUCT`, then read `.graph`	1

The graph that .query(...).graph returns lives in a local Memory store, so later reads cost nothing. Scope the query with FROM <graph> and a failure or a retry costs you one named graph.

Prepared queries belong on that local copy too. A prepareQuery() object sent through SPARQLStore dies on a bare AssertionError, because the store only accepts strings. My git history shows I tried the remote version first.

`SPARQLStore` hides what the endpoint tells you

Both rdflib versions behave the same:

Input	Result
`session=requests.Session()`	Accepted, never used
No `headers=`	`User-Agent: Python-urllib/3.11`
`timeout=1`, server hangs 4 s	Waits the full 4 s
429 with `Retry-After`, default GET	`ValueError: You did something wrong formulating either the URI or your SPARQL query`
429, `method="POST"`	Returns the tuple `(429, 'HTTP Error 429: Too Many Requests', None)`, then `AttributeError` on `.graph`

The 7.6.0 connector docstring says extra keyword arguments “can be used to set up timeouts etc.”, but the code reads only params and headers from them. socket.setdefaulttimeout() does work for the whole process.

My extraction query passed session= and no headers=, and logged ValueError at debug level. An endpoint asking for a pause would have produced a debug line and a skipped graph, sent by an anonymous-looking client.

For CONSTRUCT, skip the store and use the SPARQL protocol directly:

import requests
from rdflib import Graph

class Backoff(Exception):
 def __init__(self, seconds):
 self.seconds = seconds

def construct(session, endpoint, query):
 response = session.post(
 endpoint,
 data={"query": query},
 headers={"Accept": "application/n-triples, text/turtle;q=0.9, application/rdf+xml;q=0.8"},
 timeout=(5, 120),
 )
 if response.status_code in (429, 503):
 value = response.headers.get("Retry-After", "") # may also be an HTTP date
 raise Backoff(int(value) if value.isdigit() else 60)
 response.raise_for_status()
 media_type = response.headers["Content-Type"].split(";")[0].strip()
 return Graph().parse(data=response.content, format=media_type)

session = requests.Session()
session.headers["User-Agent"] = "my-harvester/1.0 (+https://example.org/about)"

Against the fake endpoint, this returned all 200 triples and raised Backoff(120) on the 429. The same post() with a 1-second read timeout raised ReadTimeout on the hanging server. Stripping the media type parameters matters for the reason in my rdflib field notes .

Celery `chunks` runs your task as a plain function

task.chunks(args, n) takes a chunk size , and my docstring claimed it was several tasks. Ten graphs with n=4 make three tasks. Items inside a chunk run in sequence, while the chunks run in parallel, so chunks puts no cap on concurrency against the endpoint.

The worse part is retries. I raised self.retry() on the second of ten graphs:

Canvas	Outcome
`task.chunks(zip(graphs), 4)`	Chunk ends in `FAILURE`, no retry, the next two graphs never run
`group(task.s(g) for g in graphs)`	Second graph retried once, all ten finish

Each chunk is a celery.starmap task that executes task(*item). A direct call leaves request.called_directly true, and in that case retry() re-raises the exception and schedules nothing. Celery 5.2.7 and 5.7.0 agree.

Use a group, and give each endpoint its own queue with one worker slot:

@app.task(bind=True, max_retries=5)
def inspect_graph(self, endpoint, graph_iri):
 try:
 graph = construct(session, endpoint, query_for(graph_iri))
 except Backoff as exc:
 raise self.retry(exc=exc, countdown=exc.seconds)
 extract(graph)

group(inspect_graph.s(endpoint, g) for g in graphs).apply_async(queue="endpoint-a")

celery -A harvester worker -Q endpoint-a --concurrency=1

A dereference fallback is traffic too

When fetching an IRI failed, my crawler asked the dataset’s own endpoint for that resource. The idea holds up, since plenty of linked data has IRIs that don’t resolve while the triples sit in an endpoint.

Mine fell back on every failure, each one costing a three-hop CONSTRUCT. That included a robots.txt disallow on the resource, with only the endpoint’s own robots.txt still checked. Send fallbacks through the same queue, and stop when the site said no.

Check IRIs against the SPARQL grammar

Crawled IRIs ended up in <{iri}> inside query strings. rfc3987.match() rejects > and quotes, and accepts http://example.org/a\n, because its pattern ends in $, which matches before a trailing newline. URIRef(x).n3() passes that newline too, along with \r and a newline mid-string, since rdflib’s list of invalid characters has no control characters.

That newline can’t close the IRI, so this isn’t an injection. You get a malformed query and a wasted request. I added .strip() at the time, on one of the code paths. The IRIREF production is short enough to check directly:

FORBIDDEN = set('<>"{}|^`\\') | {chr(c) for c in range(0x21)}

def iri_ref(value):
 if FORBIDDEN & set(value):
 raise ValueError(f"not safe in a SPARQL IRI: {value!r}")
 return f"<{value}>"

Takeaway

Politeness here came down to two things my code got wrong: how much it asked for, and whether it could hear the answer.

Try this on your own harvester: load one unrelated triple into a test store, run each CONSTRUCT, and fail the test if that triple comes back.

Celery on Lessons learned

Celery ignored task_task_routes: silent failures from growing a crawler on Redis

How the crawler got here

Celery ignores settings it doesn’t know

A queue with no consumer looks like a finished crawl

Redis priority steps and queue names share a namespace

LLEN 0 doesn’t mean the crawl is done

decode_responses=True on a shared pool breaks a binary cache

execute() drains a generator, group() doesn’t

Takeaway

Celery time_limit on the gevent pool: a 1 s limit let a CPU task run 3 s

What a 1 s limit does on each pool

Split fetch and parse into separate workers

Why a timeout can’t interrupt CPU work

A bare except: cancels the limit

Takeaway

Celery self.retry(5) raises TypeError, and except Exception swallows the retry

Re-raise TaskPredicate before except Exception

Why the three calls differ

What except Exception does to a retry

A dedupe mark written first makes the retry skip itself

Takeaway

SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Flat OPTIONALs join an unbound variable against everything

Send one CONSTRUCT per named graph, then read your copy

SPARQLStore hides what the endpoint tells you

Celery chunks runs your task as a plain function

A dereference fallback is traffic too

Check IRIs against the SPARQL grammar

Takeaway

`LLEN` 0 doesn’t mean the crawl is done

`decode_responses=True` on a shared pool breaks a binary cache

`execute()` drains a generator, `group()` doesn’t

A bare `except:` cancels the limit

Re-raise `TaskPredicate` before `except Exception`

What `except Exception` does to a retry

Flat `OPTIONAL`s join an unbound variable against everything

Send one `CONSTRUCT` per named graph, then read your copy

`SPARQLStore` hides what the endpoint tells you

Celery `chunks` runs your task as a plain function