Encrypt sensitive fields inside the ORM

Sat, 30 Oct 2021 18:03:45 +0200

Your database may run where you don’t control the environment: embedded in a desktop app, or in the cloud for a web app. If it stores sensitive data, encrypt those fields so a leaked copy doesn’t expose them. In this article, I show how I did that in two projects, and the drop-in text field for Python ORMs that came out of it.

Encrypted fields in an embedded SQLite

The first case was a desktop app with an SQLite database that only authorized software could read.

I considered encrypting the SQLite file and encrypting fields within it, and I chose the latter. That approach is reusable, and you never have to load the whole database into memory to decrypt it or leave it in plaintext for a moment.

The app already had its database logic, and the goal was to change as little as possible. So I put the encryption inside the ORM, and the rest of the app stayed business as usual.

The details:

Cipher and key: AES-256-CTR with a single key for the whole database. There was no reason for anything more complex. The key is generated on the first run and on the upgrade from the old version.
Storage: binary blobs made querying the database in a terminal for debugging a pain. Base64-encoded ciphertext in a text field works better. A new column holds the row’s initialization vector (IV), a random array of bytes, also base64-encoded.
Migration: it’s not only a schema change. Every row needs the fields encrypted for the updated logic to work.

You can’t query against encrypted fields. The ciphertext is a function of the plaintext, the key, and the IV, and the IV is specific to the row. This is a property of the privacy, not a defect. When you need to query encrypted values, the solution is domain-specific: sometimes you can modify the data or compute hashes. Discuss it with the customer in terms of the security objectives they want to achieve. For the same reasons, the data migration is domain-specific too.

An encrypted payload in MongoDB documents

On another project, the database was MongoDB and had many sensitive fields. I split each document into the sensitive fields and the metadata used for querying. I encrypted the sensitive fields together into a single field. The document also gets the IV and an HMAC over the rest of the document, ciphertext included:

def create_secure_json(form, keys):
 data = get_sensitive(form)
 data_secure_b64, iv_b64 = cryptjson.encrypt_json(data, keys['db'])

 payload = get_metadata(form)
 payload['created'] = str(datetime.datetime.now())
 payload['iv'] = str(iv_b64, 'ascii')
 payload['data'] = str(data_secure_b64, 'ascii')

 hmac_b64 = get_hmac(payload, keys['hmac'])
 payload['hmac'] = str(hmac_b64, 'ascii')
 return payload

This payload is what goes into the database. Use a separate key for the HMAC, and verify it before you decrypt: AES-CBC alone doesn’t detect tampering.

For the encryption, I use the cryptography package from the Python Cryptographic Authority. Encrypting a JSON is no magic: dump it into bytes and encrypt those, here with AES-CBC:

import base64
import secrets

from cryptography.hazmat.primitives import padding
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes


def encrypt_bytes(pt, key):
 iv = secrets.token_bytes(16)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 encryptor = cipher.encryptor()
 padder = padding.PKCS7(128).padder()
 padded_pt = padder.update(pt) + padder.finalize()
 ct = encryptor.update(padded_pt) + encryptor.finalize()
 return base64.b64encode(ct), base64.b64encode(iv)


def decrypt_bytes(b64ct, b64iv, key):
 ct = base64.b64decode(b64ct)
 iv = base64.b64decode(b64iv)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 decryptor = cipher.decryptor()
 padded_pt = decryptor.update(ct) + decryptor.finalize()
 unpadder = padding.PKCS7(128).unpadder()
 return unpadder.update(padded_pt) + unpadder.finalize()

A drop-in encrypted text field

Both solutions were domain-specific. Still, they showed that the use case is common and that encryption in Python is easy to work with. So I wrote cryptbase, a small library that adds an encrypted text field to SQLAlchemy and Django ORM.

Inside, there is a CryptoContainer. You create it from a ciphertext and an IV, or from a plaintext, and it gives you access to the ciphertext, the plaintext, and the IV in a unified way:

container = CryptoContainer(ciphertext=base64.b64decode(ciphertext),
 iv=base64.b64decode(iv),
 encryptor=encryptor)

container = CryptoContainer(plaintext=value, encryptor=self.encryptor)

A separate IV column complicates the migration, and the goal is a drop-in replacement for text fields. The base64 alphabet has no @, so the field stores both values in one column, with @ as the separator:

<base64 ciphertext>@<base64 IV>

The encryptor is an object with a simple interface, so there can be one per encryption method, such as AES256CTREncryptor:

def encrypt(self, plaintext: bytes) -> Tuple[bytes, bytes]:
 pass

def decrypt(self, ciphertext: bytes, iv: bytes) -> bytes:
 pass

In the end, the ORM gets a simple interface:

# SQLAlchemy
sensitive = Column(EncryptedText(key=DB_KEY))

# Django
sensitive = EncryptedTextField(key=DB_KEY)

You work with these fields the same way as with a text field. When the application reads the value, it gets the plaintext from the container. When the value is written to the database, it’s encrypted, and the __str__ of the CryptoContainer joins the base64-encoded ciphertext and IV:

str(CryptoContainer(plaintext=value, encryptor=self.encryptor))

Mongodb on Lessons learned

Encrypt sensitive fields inside the ORM

Encrypted fields in an embedded SQLite

An encrypted payload in MongoDB documents

A drop-in encrypted text field