Encryption on Lessons learned

An UPDATE on encrypted columns wrote nothing for years, and three bugs took turns hiding it

Wed, 23 Sep 2026 15:42:00 +0200

A desktop app maintained an activity history in SQLite, but the UI froze on the first entry into any state. An action could start as ‘in progress’ and later complete, yet the UI would keep showing ‘in progress’ forever.

The update function looked fine at first glance, but nobody had actually run it under test, and it had 0% test coverage. That made the bug easy to miss and hard to trust.

The setup

The app encrypts the path columns as it writes them (column level encryption, not the whole database as with SQLCipher; see why fields rather than the file ). With a random IV per row, you can’t query the encrypted columns at all. To make rows matchable, derive the IV from the plaintext so the same path always encrypts to the same ciphertext. You store the IV alongside the row, base64 encoded:

history(id, run_id, path_enc, type, state, timestamp, iv)

You can’t put a plaintext path in a WHERE clause against path_enc. So the update does three steps:

Check the row’s IV.
Re-encrypt the current path with that IV, then SELECT to check whether the row still matches by id, run, path, and type.
Run an UPDATE with the same match predicate.

Reproduce with the real class

One test that stores, updates, and reloads through the real DAO, with an in-memory schema and a dummy key:

TEST_F(HistoryDaoTest, update_StoredUpload_LoadShowsNewState) {
 HistoryDao dao(&_database);
 Action upload { action(Action::Upload, Action::IN_PROGRESS) };
 upload.rowId = dao.store(&upload, 5);
 upload.status = Action::OK;

 dao.update(&upload, 5);

 EXPECT_EQ(load().value(0).state(), Action::OK);
}

It failed. The reloaded state was still IN_PROGRESS, which was precisely where the problem was.

Three faults, each hiding the next

QByteArray iv = queryIv.value(0).toByteArray(); // 1: still base64
...
query.prepare("SELECT id FROM history" // 2: no space before WHERE
 "WHERE id = :id AND path_enc = :path ...");
bool ok = database->exec(&queryIv); // 2: runs the IV query again
...
QSqlQuery update_query(db);
update_query.prepare("UPDATE history SET state = :state ... WHERE <same predicate>");
...
database->exec(&query); // 3: runs the SELECT, not the UPDATE
database->commit();
return true;

The IV was never decoded. store() decodes the base64 IV before encryption, but update() passed the IV through still encoded. The encrypt helper rejected an IV of the wrong length and threw, so every call stopped there. Before the length check was introduced, a wrong IV would have produced a ciphertext that was impossible to match. In either case, nothing was ever written. The bug had been there since the commit that introduced encryption, because the two sibling functions in that commit disagreed.
The guard SELECT never ran. A later refactor concatenated two string literals with no space between them (historyWHERE), so prepare() failed. The failure was only logged. Then exec() ran on the IV query, which succeeded. A follow-up “fix typo in query” commit fixed half of the SQL and left the missing space.
The UPDATE never ran. A rename to update_query missed the exec() call, which kept executing query instead.

Nothing ever got past fault 1, so faults 2 and 3 were never executed in production. Read the code as a chain of what would have happened if each fault had been fixed one at a time: fix the IV, and the function returns true while writing nothing; fix the SELECT as well, and fault 3 reruns a successful SELECT, commits, and returns true. Each fix would look like progress, and the result would be a function that reports success and writes nothing.

Date the bug before you blame the refactor

The first suspect was the 2022 refactor that added the second SELECT. Running git log -S on the decode call and reading store() and update() side by side in the encryption commit moved the start date back to 2019. That matters for the fix: the refactor wasn’t the thing to revert, and the original rule from before encryption was still the spec.

The tempting shortcut

The obvious simplification is to drop the IV lookup and the re-encryption and run UPDATE … WHERE id = :id. That passes the test above.

It also deletes the rule. The version from before encryption updated a row only while its id, run, paths, and type all still matched. Every later commit kept that predicate. Updating by id alone means a stale action can overwrite whatever row now sits behind that id. So the fix was one small change per fault, keeping the predicate:

QByteArray iv = QByteArray::fromBase64(queryIv.value(0).toByteArray(),
 QByteArray::Base64UrlEncoding);
query.prepare("SELECT id FROM history "
 "WHERE id = :id AND path_enc = :path ...");
bool ok = database->exec(&query);
...
database->exec(&update_query);

A second test pins the rule:

TEST_F(HistoryDaoTest, update_OtherPath_LeavesTheRowAlone) {
 HistoryDao dao(&_database);
 Action upload { action(Action::Upload, Action::IN_PROGRESS) };
 upload.rowId = dao.store(&upload, 5);
 upload.name = "other.jpg"; // same id, different path
 upload.status = Action::OK;

 dao.update(&upload, 5);

 EXPECT_EQ(load().value(0).state(), Action::IN_PROGRESS);
}

Before the fix, this test passed too, trivially, because nothing was ever updated. A negative test proves something only once its positive twin is green. After the fix, both tests passed, and so did the full suite (787 tests).

Side notes

QSqlQuery::size() returns -1 on SQLite, because the driver doesn’t report query sizes. A if (query.size() > 1) guard never fires there. Count with while (query.next()), or use SELECT COUNT(*).
Deriving the IV from the plaintext makes the encryption deterministic: equal paths give equal ciphertexts, which is what makes matching possible. With AES-GCM, never reuse that IV for a different plaintext. If you need equality lookups on encrypted columns, consider a misuse-resistant mode (AES-SIV) or a separate blind-index column (an HMAC of the plaintext) instead.

Takeaway

Test a write path as store → change → reload through the real class against a real database. “Returns true” and “rows changed” are different claims.
Check every prepare() and exec() result and stop on failure. A failure that’s only logged lets the function carry on and commit.
Use one QSqlQuery variable per statement and never pass the wrong one to exec(). After an UPDATE, check numRowsAffected(); zero rows is a valid SQL result, not an error.
When you fix a chain like this, expect the next fault to surface. Keep the reproducing test red until the whole round trip works.
Before simplifying a WHERE clause, find out why each condition is there. Pin the rule with a negative test.

When the server must compute on confidential data, split the data by who may read it

Sun, 21 Jun 2026 20:00:00 +0200

I designed a web app that computes a person’s tax on their investment trades. Professional advisory firms run it for their own clients. Clients connect broker APIs or upload statements, and the server runs lot matching, currency conversion, and the holding-period tests. A client with a messy case can hand it to an advisor.

The confidentiality requirement may sound contradictory. The advisor owes the client professional secrecy from the first contact. The platform operator should not be able to read client data at rest, and the same goes for a stolen backup, a tenant boundary bug, or a subpoena served on the platform. The server still has to compute on all of it, including in background jobs while the user is offline.

Neither usual answer works:

Disk or database encryption protects against a stolen disk. The operator, its DBAs, and a subpoena still get plaintext.
Client-side end-to-end encryption fails because the server is the calculator. Background sync, recomputation after new data, and the advisor’s view all need plaintext on the server. In a web app it is weaker than it sounds anyway: the server ships the JavaScript on every load, so an operator that serves malicious code gets the password.

What worked was a question I had skipped: which fields does the computation actually read?

List what the engine needs

For every realized event, the tax engine needs a date, a quantity, amounts in the reporting currency, fees, an event type, and an asset category (security, crypto, derivative). To match lots, it has to know that two trades are the same instrument. It doesn’t need the instrument’s name, ticker, or ISIN, and it doesn’t need the broker, account number, or person.

One step needs real identity: classifying an instrument into a category. A broker may report a CFD under “stocks”, and a share can be fractional or whole. That happens once, at ingestion, inside a per-broker adapter. The adapter sees identity in memory, emits a de-identified, categorized record, and the identity goes elsewhere.

So the data splits into two zones, each with its own readers:

	Zone 1: tax skeleton	Zone 2: identity vault
Contents	surrogate instrument ID, category, dates, quantity, amounts, fees, event type	surrogate → name/ticker/ISIN, broker, account, personal data
Encrypted under	per-tenant key encryption key in a KMS (envelope encryption)	key chain rooted in the user’s password
Readers	engine (in memory, during compute), the firm after handoff	the user in session, the firm after handoff
Platform operator	ciphertext at rest; if it decrypts, it gets anonymous arithmetic	ciphertext only
Background jobs	read and write	write only, through a public-key layer
Lost password	survives; the case can still be computed and filed	lost unless the user has the recovery code

Zone 1 is intentionally not under the user’s key. Under the user’s key, the background sync, the in-year recompute, and the advisor’s computation would all stop working as soon as the user logged out. Background jobs write to Zone 2 by encrypting to the user’s public key. Only a logged-in user can decrypt.

Surrogate IDs: a keyed hash, scoped per user

Lot matching groups trades by instrument, so the skeleton needs a stable ID that is not the instrument. The design is an HMAC with a per-user key held in the KMS:

// Stable across one user's accounts and years, unlinkable across users.
export async function deriveSurrogateInstrumentId(userId: string, instrument: string, broker: string) {
 if (!instrument || !broker) throw new Error("surrogate requires instrument identity and broker");
 return kmsHmac(`surrogate:user:${userId}`, JSON.stringify([instrument, broker]));
}

Each alternative breaks one requirement:

A plain hash of the ISIN falls to a dictionary attack. The set of listed instruments is small, so hashing them all takes seconds.
One global HMAC key makes the same instrument correlate across every user in a dump.
A random ID per trade cannot be recomputed. Next year’s sync, or a second account at another broker, has to land on the same ID without reading the vault, or carried-in lots stop matching.

The keyed hash protects a database dump. An operator who can call the KMS live can still test “does this user hold this instrument”. That is past the threat model’s at-rest boundary, so write it down instead of claiming more.

Access and liability begin at the same instant

Before handoff, the firm’s advisors have no access to a prospect’s case. For identity, the user’s key enforces that cryptographically. The anonymous skeleton sits under the firm’s own key, so for it, access control is the barrier. It matches the legal position: no engagement contract, no professional liability, only precontractual confidentiality.

Handoff is one atomic event. The engagement contract forms, the firm’s anti-money-laundering (AML) and know-your-customer (KYC) duties start, and the vault’s relevant portion is rewrapped to the firm’s key. Data access and liability start together, so neither exists without the other.

The coding agent’s first draft at this step was wrong. It proposed default-minimal disclosure: share only the vault fields the flagged items needed. I rejected it because an advisor who takes on liability and an AML duty cannot sign off on a case half-blind. The boundary that held was purpose, not field sensitivity: everything the past outcome was built from transfers in full, and anything forward-looking (plans, rebalancing ideas, notes) never transfers. Data minimization still applies, but the line runs between past and future, not through the case.

One password, auto-unlock, never transmitted: pick an aPAKE

The first plan was standard session login plus a client-side KDF for the vault key, with OPAQUE as a later upgrade. That leaves two bad choices:

Two passwords: one for login, sent to the server, and one for the vault, used only in the browser. Users hate it.
One password, still POSTed for login, also used to derive the vault key. It gives auto-unlock, and it quietly hands the operator the vault password on every login. Zone 2 stops protecting anything.

With a standard login, one password cannot also stay off the server. An asymmetric PAKE fixes the login side. OPAQUE (RFC 9807) authenticates without the server learning the password, and gives the client an export_key that the server never sees. Root the vault in it:

// K_vault = HKDF(export_key); unwraps user key → RSA private key → vault entries
function deriveVaultKeyBytes(exportKey: Uint8Array): Uint8Array {
 if (exportKey.length === 0) throw new Error("export_key is required");
 return hkdf(sha256, exportKey, new Uint8Array(0), K_VAULT_INFO, 32);
}

A password reset is a new OPAQUE registration with a new export_key. The mandatory recovery code rewraps the vault chain to the new root, and Zone 1 is untouched. Use a vetted library: the registration and login flows and the export_key handling are now part of the vault’s security.

The tenant boundary belongs in the database

Each firm is a tenant, and one firm’s prospects must never show up in another firm’s queries. A forgotten WHERE firm_id = ? should return nothing, not another tenant’s rows. With Postgres row-level security, three details decide whether it fails closed:

alter table client owner to app_migrator; -- the app role is not the owner
alter table client force row level security; -- the owner obeys policies too
create policy firm_isolation on client
 using (firm_id = current_setting('app.firm_id')::uuid)
 with check (firm_id = current_setting('app.firm_id')::uuid);

await db.transaction().execute(async (tx) => {
 await sql`select set_config('app.firm_id', ${firmId}, true)`.execute(tx); // true = transaction-local
 return work(tx);
});

Table owners bypass RLS unless it is forced. Superusers and BYPASSRLS roles bypass it always, so the app must connect as neither.
current_setting without missing_ok, plus the ::uuid cast, raises when the tenant was never set or is empty. A missing tenant is an error, not an empty match.
set_config(..., true) resets at commit or rollback. With false, a pooled connection carries one firm’s ID into the next request, and a transaction-mode pooler can hand the setting to a different client.

Write down the floors

The design makes specific claims and lists what it does not cover. That list is part of the design:

Plaintext exists in memory during ingestion and compute. The claim is “zero to the operator at rest”, never “the platform never sees it”.
amount ÷ quantity approximates a unit price that can be matched against market data. It is exploitable only by someone who already decrypted Zone 1.
The web JavaScript trust gap stays open. OPAQUE stops offline cracking of a stolen verifier table, not a malicious bundle. A native client is the eventual fix.
An API-connected account reveals which broker it is, because the token names it. A manual upload keeps the broker in Zone 2. The user gets that choice in the UI.
A forgotten password without the recovery code loses the labels. The skeleton survives, and the user is told to keep their own copies of the source statements.

Takeaway

When the server has to be the calculator, don’t choose between “encrypt the disk” and “encrypt everything client-side”. List the readers (user, professional, engine, operator) and what each one needs. Then check whether the computation can run without identity. When it can, identity moves into a zone the computation never opens, and the remaining decisions follow from that: keyed surrogates, a handoff that rewraps keys, an aPAKE for the single password.

In practice, I ran the design as an interview with a coding agent. It asked one question at a time, proposed options with a recommendation, and recorded each decision as an ADR with a “Why this is non-obvious” section: the design a reader would expect, and why it was rejected here. Several recommendations were reasonable but wrong. Two of my corrections removed whole planned pieces: the default-minimal handoff, and an instrument catalog that event-driven classification made unnecessary.

Encrypt sensitive fields inside the ORM

Sat, 30 Oct 2021 18:03:45 +0200

Your database may run where you don’t control the environment: embedded in a desktop app, or in the cloud for a web app. If it stores sensitive data, encrypt those fields so a leaked copy doesn’t expose them. In this article, I show how I did that in two projects, and the drop-in text field for Python ORMs that came out of it.

Encrypted fields in an embedded SQLite

The first case was a desktop app with an SQLite database that only authorized software could read.

I considered encrypting the SQLite file and encrypting fields within it, and I chose the latter. That approach is reusable, and you never have to load the whole database into memory to decrypt it or leave it in plaintext for a moment.

The app already had its database logic, and the goal was to change as little as possible. So I put the encryption inside the ORM, and the rest of the app stayed business as usual.

The details:

Cipher and key: AES-256-CTR with a single key for the whole database. There was no reason for anything more complex. The key is generated on the first run and on the upgrade from the old version.
Storage: binary blobs made querying the database in a terminal for debugging a pain. Base64-encoded ciphertext in a text field works better. A new column holds the row’s initialization vector (IV), a random array of bytes, also base64-encoded.
Migration: it’s not only a schema change. Every row needs the fields encrypted for the updated logic to work.

You can’t query against encrypted fields. The ciphertext is a function of the plaintext, the key, and the IV, and the IV is specific to the row. This is a property of the privacy, not a defect. When you need to query encrypted values, the solution is domain-specific: sometimes you can modify the data or compute hashes. Discuss it with the customer in terms of the security objectives they want to achieve. For the same reasons, the data migration is domain-specific too.

An encrypted payload in MongoDB documents

On another project, the database was MongoDB and had many sensitive fields. I split each document into the sensitive fields and the metadata used for querying. I encrypted the sensitive fields together into a single field. The document also gets the IV and an HMAC over the rest of the document, ciphertext included:

def create_secure_json(form, keys):
 data = get_sensitive(form)
 data_secure_b64, iv_b64 = cryptjson.encrypt_json(data, keys['db'])

 payload = get_metadata(form)
 payload['created'] = str(datetime.datetime.now())
 payload['iv'] = str(iv_b64, 'ascii')
 payload['data'] = str(data_secure_b64, 'ascii')

 hmac_b64 = get_hmac(payload, keys['hmac'])
 payload['hmac'] = str(hmac_b64, 'ascii')
 return payload

This payload is what goes into the database. Use a separate key for the HMAC, and verify it before you decrypt: AES-CBC alone doesn’t detect tampering.

For the encryption, I use the cryptography package from the Python Cryptographic Authority. Encrypting a JSON is no magic: dump it into bytes and encrypt those, here with AES-CBC:

import base64
import secrets

from cryptography.hazmat.primitives import padding
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes


def encrypt_bytes(pt, key):
 iv = secrets.token_bytes(16)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 encryptor = cipher.encryptor()
 padder = padding.PKCS7(128).padder()
 padded_pt = padder.update(pt) + padder.finalize()
 ct = encryptor.update(padded_pt) + encryptor.finalize()
 return base64.b64encode(ct), base64.b64encode(iv)


def decrypt_bytes(b64ct, b64iv, key):
 ct = base64.b64decode(b64ct)
 iv = base64.b64decode(b64iv)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 decryptor = cipher.decryptor()
 padded_pt = decryptor.update(ct) + decryptor.finalize()
 unpadder = padding.PKCS7(128).unpadder()
 return unpadder.update(padded_pt) + unpadder.finalize()

A drop-in encrypted text field

Both solutions were domain-specific. Still, they showed that the use case is common and that encryption in Python is easy to work with. So I wrote cryptbase, a small library that adds an encrypted text field to SQLAlchemy and Django ORM.

Inside, there is a CryptoContainer. You create it from a ciphertext and an IV, or from a plaintext, and it gives you access to the ciphertext, the plaintext, and the IV in a unified way:

container = CryptoContainer(ciphertext=base64.b64decode(ciphertext),
 iv=base64.b64decode(iv),
 encryptor=encryptor)

container = CryptoContainer(plaintext=value, encryptor=self.encryptor)

A separate IV column complicates the migration, and the goal is a drop-in replacement for text fields. The base64 alphabet has no @, so the field stores both values in one column, with @ as the separator:

<base64 ciphertext>@<base64 IV>

The encryptor is an object with a simple interface, so there can be one per encryption method, such as AES256CTREncryptor:

def encrypt(self, plaintext: bytes) -> Tuple[bytes, bytes]:
 pass

def decrypt(self, ciphertext: bytes, iv: bytes) -> bytes:
 pass

In the end, the ORM gets a simple interface:

# SQLAlchemy
sensitive = Column(EncryptedText(key=DB_KEY))

# Django
sensitive = EncryptedTextField(key=DB_KEY)

You work with these fields the same way as with a text field. When the application reads the value, it gets the plaintext from the container. When the value is written to the database, it’s encrypted, and the __str__ of the CryptoContainer joins the base64-encoded ciphertext and IV:

str(CryptoContainer(plaintext=value, encryptor=self.encryptor))