SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Tue, 14 Sep 2021 11:35:00 +0200

Update 2026-11-08: rebuilt every repro on rdflib 6.3.2 and 7.6.0, and on Celery 5.2.7 and 5.7.0.

I wanted a resource and three hops of its neighbors from a SPARQL endpoint. This query returns every triple in the store:

CONSTRUCT { <http://example.org/root> ?p1 ?o1 . ?o1 ?p2 ?o2 . ?o2 ?p3 ?o3 . }
WHERE {
 <http://example.org/root> ?p1 ?o1 .
 OPTIONAL { ?o1 ?p2 ?o2 . }
 OPTIONAL { ?o2 ?p3 ?o3 . }
}

Move the second OPTIONAL inside the first, and you get the four triples you meant. A comment in my 2021 crawler says nested optionals are a must, otherwise the query will not finish. This post explains why and covers other ways the crawler could mistreat an endpoint without noticing.

Everything below comes from rereading that code and rebuilding each case against a local fake endpoint, under Python 3.11. It shows what the code does. I have no production logs that say what it cost anyone.

Flat `OPTIONAL`s join an unbound variable against everything

The test store holds 20,005 triples: a root with a literal label, a four-hop chain, and 20,000 unrelated triples. Times are from one run.

Query shape	Solutions	Triples returned	Oxigraph 0.5.13	rdflib 7.6.0
Flat `OPTIONAL`s	20,006	20,005	28 ms	897 ms
Nested `OPTIONAL`s	2	4	1.0 ms	3.2 ms
No `OPTIONAL`	1	3	0.1 ms	1.7 ms

The literal label triggers it. When ?o1 is that literal, the first OPTIONAL can’t match, because a literal is never a subject, so ?o2 stays unbound. The second OPTIONAL then has nothing constraining it. ?o2 ?p3 ?o3 matches every triple, and the CONSTRUCT template writes them all out. Most resources have a label, so most lookups hit this.

Removing the OPTIONALs is the wrong repair: the label disappears, as the last row shows.

I can’t rerun the 2021 endpoint, and I only tested these two engines. I believe this is what “will not finish” meant, since on a store with millions of triples each lookup becomes a full dump.

A variable you never bind does the same thing. I once committed OPTIONALs on ?dataset in a query that bound ?ds, and renamed it 18 minutes later. It’s a valid query, so nothing raises an error.

Nesting fixes the unbound variable and does nothing for hubs. A node with a million outgoing triples still returns them all, so add a LIMIT if a partial answer is acceptable.

Send one `CONSTRUCT` per named graph, then read your copy

Graph(store=SPARQLStore(url)) looks like a local graph, and every value() call on it is an HTTP request. With 50 datasets and three property reads each:

Approach	Requests to the endpoint
Read through the store-backed graph	151
One `CONSTRUCT`, then read `.graph`	1

The graph that .query(...).graph returns lives in a local Memory store, so later reads cost nothing. Scope the query with FROM <graph> and a failure or a retry costs you one named graph.

Prepared queries belong on that local copy too. A prepareQuery() object sent through SPARQLStore dies on a bare AssertionError, because the store only accepts strings. My git history shows I tried the remote version first.

`SPARQLStore` hides what the endpoint tells you

Both rdflib versions behave the same:

Input	Result
`session=requests.Session()`	Accepted, never used
No `headers=`	`User-Agent: Python-urllib/3.11`
`timeout=1`, server hangs 4 s	Waits the full 4 s
429 with `Retry-After`, default GET	`ValueError: You did something wrong formulating either the URI or your SPARQL query`
429, `method="POST"`	Returns the tuple `(429, 'HTTP Error 429: Too Many Requests', None)`, then `AttributeError` on `.graph`

The 7.6.0 connector docstring says extra keyword arguments “can be used to set up timeouts etc.”, but the code reads only params and headers from them. socket.setdefaulttimeout() does work for the whole process.

My extraction query passed session= and no headers=, and logged ValueError at debug level. An endpoint asking for a pause would have produced a debug line and a skipped graph, sent by an anonymous-looking client.

For CONSTRUCT, skip the store and use the SPARQL protocol directly:

import requests
from rdflib import Graph

class Backoff(Exception):
 def __init__(self, seconds):
 self.seconds = seconds

def construct(session, endpoint, query):
 response = session.post(
 endpoint,
 data={"query": query},
 headers={"Accept": "application/n-triples, text/turtle;q=0.9, application/rdf+xml;q=0.8"},
 timeout=(5, 120),
 )
 if response.status_code in (429, 503):
 value = response.headers.get("Retry-After", "") # may also be an HTTP date
 raise Backoff(int(value) if value.isdigit() else 60)
 response.raise_for_status()
 media_type = response.headers["Content-Type"].split(";")[0].strip()
 return Graph().parse(data=response.content, format=media_type)

session = requests.Session()
session.headers["User-Agent"] = "my-harvester/1.0 (+https://example.org/about)"

Against the fake endpoint, this returned all 200 triples and raised Backoff(120) on the 429. The same post() with a 1-second read timeout raised ReadTimeout on the hanging server. Stripping the media type parameters matters for the reason in my rdflib field notes .

Celery `chunks` runs your task as a plain function

task.chunks(args, n) takes a chunk size , and my docstring claimed it was several tasks. Ten graphs with n=4 make three tasks. Items inside a chunk run in sequence, while the chunks run in parallel, so chunks puts no cap on concurrency against the endpoint.

The worse part is retries. I raised self.retry() on the second of ten graphs:

Canvas	Outcome
`task.chunks(zip(graphs), 4)`	Chunk ends in `FAILURE`, no retry, the next two graphs never run
`group(task.s(g) for g in graphs)`	Second graph retried once, all ten finish

Each chunk is a celery.starmap task that executes task(*item). A direct call leaves request.called_directly true, and in that case retry() re-raises the exception and schedules nothing. Celery 5.2.7 and 5.7.0 agree.

Use a group, and give each endpoint its own queue with one worker slot:

@app.task(bind=True, max_retries=5)
def inspect_graph(self, endpoint, graph_iri):
 try:
 graph = construct(session, endpoint, query_for(graph_iri))
 except Backoff as exc:
 raise self.retry(exc=exc, countdown=exc.seconds)
 extract(graph)

group(inspect_graph.s(endpoint, g) for g in graphs).apply_async(queue="endpoint-a")

celery -A harvester worker -Q endpoint-a --concurrency=1

A dereference fallback is traffic too

When fetching an IRI failed, my crawler asked the dataset’s own endpoint for that resource. The idea holds up, since plenty of linked data has IRIs that don’t resolve while the triples sit in an endpoint.

Mine fell back on every failure, each one costing a three-hop CONSTRUCT. That included a robots.txt disallow on the resource, with only the endpoint’s own robots.txt still checked. Send fallbacks through the same queue, and stop when the site said no.

Check IRIs against the SPARQL grammar

Crawled IRIs ended up in <{iri}> inside query strings. rfc3987.match() rejects > and quotes, and accepts http://example.org/a\n, because its pattern ends in $, which matches before a trailing newline. URIRef(x).n3() passes that newline too, along with \r and a newline mid-string, since rdflib’s list of invalid characters has no control characters.

That newline can’t close the IRI, so this isn’t an injection. You get a malformed query and a wasted request. I added .strip() at the time, on one of the code paths. The IRIREF production is short enough to check directly:

FORBIDDEN = set('<>"{}|^`\\') | {chr(c) for c in range(0x21)}

def iri_ref(value):
 if FORBIDDEN & set(value):
 raise ValueError(f"not safe in a SPARQL IRI: {value!r}")
 return f"<{value}>"

Takeaway

Politeness here came down to two things my code got wrong: how much it asked for, and whether it could hear the answer.

Try this on your own harvester: load one unrelated triple into a test store, run each CONSTRUCT, and fail the test if that triple comes back.

Sparql on Lessons learned

SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Flat OPTIONALs join an unbound variable against everything

Send one CONSTRUCT per named graph, then read your copy

SPARQLStore hides what the endpoint tells you