Rdflib on Lessons learned

rdflib field notes: ns.format matches nothing, and parse() goes online

Sat, 11 Mar 2023 12:37:00 +0100

UUpdate 2026-11-08: retested on rdflib 7.6.0 and added the ConjunctiveGraph deprecation, pyoxigraph, and the RDF 1.2 and SPARQL 1.2 status.

This lookup returns an empty list, with no exception and no warning:

from rdflib import Graph, Literal, Namespace, URIRef

DCT = Namespace("http://purl.org/dc/terms/")
doc = URIRef("http://example.org/doc")

g = Graph()
g.add((doc, DCT["format"], Literal("text/csv")))

list(g.objects(doc, DCT.format)) # []
list(g.objects(doc, DCT["format"])) # [rdflib.term.Literal('text/csv')]

Namespace subclasses str, so DCT.format is the bound method str.format, and a lookup on it matches nothing. RDF code tends to fail this way: you get fewer triples, and that looks exactly like sparse data.

I collected these while re-reading a crawler I wrote in early 2023, and rebuilt each as a minimal repro on rdflib 6.3.2 and 7.6.0 under Python 3.11. Both versions behave the same unless noted. They show what the code does. I have no production logs that say what any of them cost.

Use `ns["format"]` for names that `str` already owns

format, count, index, join, and every other str method name resolve to the method. title is the exception, special-cased to return a URIRef. Writes are safer than reads here: g.add() raises AssertionError on the method object.

For published vocabularies, import from rdflib.namespace. DCTERMS.format is a real URIRef and DCTERMS.nonsense raises AttributeError. For your own vocabulary, ClosedNamespace rejects unknown terms, but .format on it is still the str method, so keep the brackets.

A typo in the base IRI is still a valid IRI

NS = Namespace("https://example.org/vocab/mediaTyp")
NS.mediaType # https://example.org/vocab/mediaTypmediaType

My crawler shipped this. The branch using it asked for a predicate no document contains. ClosedNamespace produces the same glued IRI, because it validates the term and has no opinion on the base. A one-line test covers it, with an allowlist for the rare namespace that ends differently:

assert str(NS)[-1] in "#/"

A failed parse keeps what it already read

Feed Graph.parse() five good N-Triples lines and one bad one. It raises ParserError, and len(g) is 5. Turtle does the same with BadSyntax. If you parse into a shared graph, a truncated download leaves half a document behind.

Give each document its own store and merge only after the parse succeeds:

import tempfile
from pathlib import Path
from rdflib import Graph, URIRef, plugin
from rdflib.store import Store

def load_document(source, fmt, target):
 with tempfile.TemporaryDirectory() as tmp:
 path = str(Path(tmp) / "graph")
 store = plugin.get("LevelDB", Store)(identifier=URIRef(path))
 graph = Graph(store, identifier=URIRef("urn:doc"))
 graph.open(path, create=True)
 try:
 graph.parse(source, format=fmt)
 target += graph
 finally:
 graph.close()

With a bad document, this raises an exception and target stays empty. The disk store also caps memory. Loading 200,000 synthetic triples (14 MB of N-Triples, one run on Apple Silicon, rdflib 6.3.2, 30 MB baseline after imports):

Store	Time	Peak RSS	Disk
rdflib `Memory`	2.6 s	351 MB	0
`rdflib-leveldb`	15.3 s	85 MB	39 MB
pyoxigraph 0.5.13 `bulk_load`	0.5 s	200 MB	10 MB

Read that as a rough shape and benchmark your own data. rdflib-leveldb isn’t on PyPI, so install it from git. Its last commit is from December 2021, though the snippet above still passes on 7.6.0. On macOS, I used plyvel-ci because plyvel has no Mac wheels. Pyoxigraph has its own API and isn’t a drop-in rdflib store.

A disk store bounds the triples and leaves the input alone. The Turtle parser calls stream.read() on the whole source, with the comment # Not ideal.

`parse()` can go online

Parsing JSON-LD fetches remote @context values. The RDFLib security page says so. Direct cycles are caught (recursive context inclusion after 2 requests). A chain of contexts that each point to a new one isn’t: with Python’s default recursion limit, one small document cost 967 requests to my local server before RecursionError.

Block the network during the parse step:

import urllib.error, urllib.request

class NoNetwork(urllib.request.BaseHandler):
 handler_order = 100 # before the stock HTTPHandler (500)

 def http_open(self, req):
 raise urllib.error.URLError(f"blocked: {req.full_url}")

 https_open = http_open

urllib.request.install_opener(urllib.request.build_opener(NoNetwork()))

Leave out handler_order and the stock handler answers first, so the request goes through. My first attempt did exactly that. The opener is process-global, and the RDFLib docs rank OS-level isolation such as containers above Python-level controls, so use it in addition to those on untrusted input.

Then make the following links a separate, bounded step that you control:

from collections import deque

def crawl(seed, fetch, max_depth=3):
 seen, queue = {seed}, deque([(seed, 0)])
 while queue:
 iri, depth = queue.popleft()
 references = fetch(iri) # fetch, parse in isolation, return IRIs to follow
 if depth < max_depth:
 for ref in references:
 if ref not in seen:
 seen.add(ref)
 queue.append((ref, depth + 1))

On a graph with a cycle and a chain, this visits each IRI once and stops at depth 3.

Keep a workaround’s reason in a test

rdflib #1195 reported that a CONSTRUCT response with application/rdf+xml; charset=UTF-8 broke the plugin lookup. My code registered that literal string as a parser name and pointed a comment at the issue.

Against a fake endpoint sending that header, rdflib 5.0.0 raises PluginException and 6.0.0 works without the workaround (both on Python 3.9). My pin was already 6.0.0 in the commit that added it, so on this path it never changed, and it stayed that way through 6.3.2. Watching the issue wouldn’t have helped: it closed in August 2022 on a PR that only added tests.

A test that exercises the upstream behavior with the workaround off tells you the day it can go. plugin.get() itself still rejects that string on 7.6.0, so strip parameters if you call it directly.

`Dataset` isn’t a drop-in for `ConjunctiveGraph`

ConjunctiveGraph raises a DeprecationWarning on 7.6.0 and the replacement is Dataset, which has a different default. With one quad in a named graph:

Class	`triples()`	`SELECT ?s ?p ?o`
`ConjunctiveGraph()`	1	1
`Dataset()`	0	0
`Dataset(default_union=True)`	1	1

Migrate with default_union=True, unless you want queries limited to the default graph.

On standards, as of November 2026: RDF 1.2 Concepts is a Candidate Recommendation and SPARQL 1.2 Query is a Working Draft, so SPARQL 1.1 remains the Recommendation.

Takeaway

Most of these bugs arrive as an empty result, and nothing downstream complains. For each predicate your pipeline depends on, keep a fixture with one known triple and assert that your code finds it.

Start with this search for unbracketed str names in your own code:

grep -rnE '\b[A-Za-z_]+\.(format|count|index)\b([^(]|$)' --include='*.py' .

SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Tue, 14 Sep 2021 11:35:00 +0200

Update 2026-11-08: rebuilt every repro on rdflib 6.3.2 and 7.6.0, and on Celery 5.2.7 and 5.7.0.

I wanted a resource and three hops of its neighbors from a SPARQL endpoint. This query returns every triple in the store:

CONSTRUCT { <http://example.org/root> ?p1 ?o1 . ?o1 ?p2 ?o2 . ?o2 ?p3 ?o3 . }
WHERE {
 <http://example.org/root> ?p1 ?o1 .
 OPTIONAL { ?o1 ?p2 ?o2 . }
 OPTIONAL { ?o2 ?p3 ?o3 . }
}

Move the second OPTIONAL inside the first, and you get the four triples you meant. A comment in my 2021 crawler says nested optionals are a must, otherwise the query will not finish. This post explains why and covers other ways the crawler could mistreat an endpoint without noticing.

Everything below comes from rereading that code and rebuilding each case against a local fake endpoint, under Python 3.11. It shows what the code does. I have no production logs that say what it cost anyone.

Flat `OPTIONAL`s join an unbound variable against everything

The test store holds 20,005 triples: a root with a literal label, a four-hop chain, and 20,000 unrelated triples. Times are from one run.

Query shape	Solutions	Triples returned	Oxigraph 0.5.13	rdflib 7.6.0
Flat `OPTIONAL`s	20,006	20,005	28 ms	897 ms
Nested `OPTIONAL`s	2	4	1.0 ms	3.2 ms
No `OPTIONAL`	1	3	0.1 ms	1.7 ms

The literal label triggers it. When ?o1 is that literal, the first OPTIONAL can’t match, because a literal is never a subject, so ?o2 stays unbound. The second OPTIONAL then has nothing constraining it. ?o2 ?p3 ?o3 matches every triple, and the CONSTRUCT template writes them all out. Most resources have a label, so most lookups hit this.

Removing the OPTIONALs is the wrong repair: the label disappears, as the last row shows.

I can’t rerun the 2021 endpoint, and I only tested these two engines. I believe this is what “will not finish” meant, since on a store with millions of triples each lookup becomes a full dump.

A variable you never bind does the same thing. I once committed OPTIONALs on ?dataset in a query that bound ?ds, and renamed it 18 minutes later. It’s a valid query, so nothing raises an error.

Nesting fixes the unbound variable and does nothing for hubs. A node with a million outgoing triples still returns them all, so add a LIMIT if a partial answer is acceptable.

Send one `CONSTRUCT` per named graph, then read your copy

Graph(store=SPARQLStore(url)) looks like a local graph, and every value() call on it is an HTTP request. With 50 datasets and three property reads each:

Approach	Requests to the endpoint
Read through the store-backed graph	151
One `CONSTRUCT`, then read `.graph`	1

The graph that .query(...).graph returns lives in a local Memory store, so later reads cost nothing. Scope the query with FROM <graph> and a failure or a retry costs you one named graph.

Prepared queries belong on that local copy too. A prepareQuery() object sent through SPARQLStore dies on a bare AssertionError, because the store only accepts strings. My git history shows I tried the remote version first.

`SPARQLStore` hides what the endpoint tells you

Both rdflib versions behave the same:

Input	Result
`session=requests.Session()`	Accepted, never used
No `headers=`	`User-Agent: Python-urllib/3.11`
`timeout=1`, server hangs 4 s	Waits the full 4 s
429 with `Retry-After`, default GET	`ValueError: You did something wrong formulating either the URI or your SPARQL query`
429, `method="POST"`	Returns the tuple `(429, 'HTTP Error 429: Too Many Requests', None)`, then `AttributeError` on `.graph`

The 7.6.0 connector docstring says extra keyword arguments “can be used to set up timeouts etc.”, but the code reads only params and headers from them. socket.setdefaulttimeout() does work for the whole process.

My extraction query passed session= and no headers=, and logged ValueError at debug level. An endpoint asking for a pause would have produced a debug line and a skipped graph, sent by an anonymous-looking client.

For CONSTRUCT, skip the store and use the SPARQL protocol directly:

import requests
from rdflib import Graph

class Backoff(Exception):
 def __init__(self, seconds):
 self.seconds = seconds

def construct(session, endpoint, query):
 response = session.post(
 endpoint,
 data={"query": query},
 headers={"Accept": "application/n-triples, text/turtle;q=0.9, application/rdf+xml;q=0.8"},
 timeout=(5, 120),
 )
 if response.status_code in (429, 503):
 value = response.headers.get("Retry-After", "") # may also be an HTTP date
 raise Backoff(int(value) if value.isdigit() else 60)
 response.raise_for_status()
 media_type = response.headers["Content-Type"].split(";")[0].strip()
 return Graph().parse(data=response.content, format=media_type)

session = requests.Session()
session.headers["User-Agent"] = "my-harvester/1.0 (+https://example.org/about)"

Against the fake endpoint, this returned all 200 triples and raised Backoff(120) on the 429. The same post() with a 1-second read timeout raised ReadTimeout on the hanging server. Stripping the media type parameters matters for the reason in my rdflib field notes .

Celery `chunks` runs your task as a plain function

task.chunks(args, n) takes a chunk size , and my docstring claimed it was several tasks. Ten graphs with n=4 make three tasks. Items inside a chunk run in sequence, while the chunks run in parallel, so chunks puts no cap on concurrency against the endpoint.

The worse part is retries. I raised self.retry() on the second of ten graphs:

Canvas	Outcome
`task.chunks(zip(graphs), 4)`	Chunk ends in `FAILURE`, no retry, the next two graphs never run
`group(task.s(g) for g in graphs)`	Second graph retried once, all ten finish

Each chunk is a celery.starmap task that executes task(*item). A direct call leaves request.called_directly true, and in that case retry() re-raises the exception and schedules nothing. Celery 5.2.7 and 5.7.0 agree.

Use a group, and give each endpoint its own queue with one worker slot:

@app.task(bind=True, max_retries=5)
def inspect_graph(self, endpoint, graph_iri):
 try:
 graph = construct(session, endpoint, query_for(graph_iri))
 except Backoff as exc:
 raise self.retry(exc=exc, countdown=exc.seconds)
 extract(graph)

group(inspect_graph.s(endpoint, g) for g in graphs).apply_async(queue="endpoint-a")

celery -A harvester worker -Q endpoint-a --concurrency=1

A dereference fallback is traffic too

When fetching an IRI failed, my crawler asked the dataset’s own endpoint for that resource. The idea holds up, since plenty of linked data has IRIs that don’t resolve while the triples sit in an endpoint.

Mine fell back on every failure, each one costing a three-hop CONSTRUCT. That included a robots.txt disallow on the resource, with only the endpoint’s own robots.txt still checked. Send fallbacks through the same queue, and stop when the site said no.

Check IRIs against the SPARQL grammar

Crawled IRIs ended up in <{iri}> inside query strings. rfc3987.match() rejects > and quotes, and accepts http://example.org/a\n, because its pattern ends in $, which matches before a trailing newline. URIRef(x).n3() passes that newline too, along with \r and a newline mid-string, since rdflib’s list of invalid characters has no control characters.

That newline can’t close the IRI, so this isn’t an injection. You get a malformed query and a wasted request. I added .strip() at the time, on one of the code paths. The IRIREF production is short enough to check directly:

FORBIDDEN = set('<>"{}|^`\\') | {chr(c) for c in range(0x21)}

def iri_ref(value):
 if FORBIDDEN & set(value):
 raise ValueError(f"not safe in a SPARQL IRI: {value!r}")
 return f"<{value}>"

Takeaway

Politeness here came down to two things my code got wrong: how much it asked for, and whether it could hear the answer.

Try this on your own harvester: load one unrelated triple into a test store, run each CONSTRUCT, and fail the test if that triple comes back.

Rdflib on Lessons learned

rdflib field notes: ns.format matches nothing, and parse() goes online

Use ns["format"] for names that str already owns

A typo in the base IRI is still a valid IRI

A failed parse keeps what it already read

parse() can go online

Keep a workaround’s reason in a test

Dataset isn’t a drop-in for ConjunctiveGraph

Takeaway

SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Flat OPTIONALs join an unbound variable against everything

Send one CONSTRUCT per named graph, then read your copy

SPARQLStore hides what the endpoint tells you

Celery chunks runs your task as a plain function

A dereference fallback is traffic too

Check IRIs against the SPARQL grammar

Takeaway

Use `ns["format"]` for names that `str` already owns

`parse()` can go online

`Dataset` isn’t a drop-in for `ConjunctiveGraph`

Flat `OPTIONAL`s join an unbound variable against everything

Send one `CONSTRUCT` per named graph, then read your copy

`SPARQLStore` hides what the endpoint tells you

Celery `chunks` runs your task as a plain function