Linux on Lessons learned

Backing up ~/.claude and ~/.codex without archiving your AI logins

Sat, 31 Oct 2026 22:29:00 +0100

If you back up the whole ~/.claude or ~/.codex folder, the restore works, and the backup now holds everything an infostealer wants. The same folder holds your settings, every session transcript in plaintext, and, on Linux, the login itself:

~/.claude/.credentials.json # Claude Code login, mode 0600
~/.codex/auth.json # Codex login when file storage is in use
~/.claude/projects/*/*.jsonl # transcripts: prompts, tool output, file contents

Split the folder into two lists. Back up what you wrote: settings, instructions, skills, hooks, and memory. Keep transcripts as optional encrypted data. Leave login files out of routine backups and log in again after a restore.

Where AI client state lives on macOS and Linux

I checked these paths against Claude Code 2.1.316, Claude Desktop for Linux 4.4212.0, ChatGPT for macOS 1.2026.183, the Codex desktop app for macOS 26.1027.30824, and Codex source at commit 67f0994. “Documented” means the vendor docs name the path. “Observed” means I found it by inspecting the official build so that it can change in any release.

Client	State root	Login material	Evidence
Claude Code	`~/.claude/` and `~/.claude.json`, both under `CLAUDE_CONFIG_DIR` if set	Keychain on macOS, or `~/.claude/.credentials.json` when the Keychain rejects the write (locked in an SSH session, for example). On Linux, always that file	Documented
Claude Desktop	macOS: `~/Library/Application Support/Claude/`. Linux (beta): `${XDG_CONFIG_HOME:-~/.config}/Claude/`	Electron cookies and Local Storage, extension secrets encrypted with `safeStorage`	MCP config documented, rest observed
ChatGPT desktop, macOS only	`~/Library/Application Support/com.openai.chat/`, `~/Library/Group Containers/2DC432GLL2.group.com.openai.chat/`, `~/Library/Preferences/com.openai.chat.plist`	Keychain items in OpenAI’s access group	Observed
Codex desktop, macOS only	`~/Library/Application Support/Codex/`, plus `~/.codex`	Shared with Codex CLI	Observed
Codex CLI	`${CODEX_HOME:-~/.codex}`. `CODEX_SQLITE_HOME`, `sqlite_home` and `log_dir` can move parts of it	`auth.json` in `file` mode or as the `auto` fallback; `.credentials.json` for MCP OAuth without a keyring	Documented and source

Desktop sessions that run Claude Code write to ~/.claude/, not to the Electron folder, so include both.

What to back up and what to leave out

Treatment	Claude Code	Codex
Back up, versioned	`settings.json`, `CLAUDE.md`, `rules/`, `skills/`, `commands/`, `agents/`, `workflows/`, `output-styles/`, `keybindings.json`, `projects/*/memory/`, `agent-memory/`, `plugins/installed_plugins.json` as a manifest	`config.toml`, `*.config.toml`, `hooks.json`, `rules/`, `skills/`, agent-role files
Back up after a secret check	`~/.claude.json` (personal MCP servers, trust decisions)	MCP `env` blocks in `config.toml`
Optional, encrypted, short retention	`projects//.jsonl`, `subagents/`, `tool-results/`, `history.jsonl`	`sessions/YYYY/MM/DD/rollout-*.jsonl`, `archived_sessions/`, `history.jsonl`
Exclude	`.credentials.json`, `file-history/`, `paste-cache/`, `debug/`, `shell-snapshots/`, `session-env/`, `backups/`	`auth.json`, `.credentials.json`, `log/`, `logs_2.sqlite`, `shell_snapshots/`, `packages/`, caches

For Claude Desktop, keep claude_desktop_config.json after checking its MCP env blocks, plus a list of installed extensions and where each came from. Reinstall extensions instead of restoring their executable caches. The ChatGPT app’s local databases are caches of a server-side account; use the account’s data export for conversations you want to keep.

Why copying the whole folder is the wrong default

Transcripts are plaintext. Anthropic’s docs say that anything passing through a tool is written to the session .jsonl, so a .env file a tool reads or a token a command prints is in there. Codex rollouts record tool activity the same way.

Your backup also keeps transcripts after the client has deleted them. Claude Code deletes transcripts after cleanupPeriodDays (default 30), but keeps sessions started, or last continued in Claude Desktop or Cowork at any age unless you set desktopSessionCleanupPeriodDays. A backup with a year of history still holds a transcript eleven months after the client deleted it.

A restore can also run code. Hooks, MCP server commands, skills, plugins, desktop extensions, and shell snapshots all execute something. If malware edited them, a full restore puts it back on the clean machine.

The Keychain protects less than it seems to. It stops someone who copies the disk offline, but malware running as your user can still request the item. The Linux desktop build can also fall back to Chromium’s basic password store when no wallet is running.

Dead ends worth skipping

Guessing Linux paths from macOS ones. Replacing Library/Application Support with .config works for Claude Desktop, which has a Linux beta. OpenAI ships no Linux desktop app for ChatGPT or Codex. The Codex source has Linux branches, but there is no supported Linux build. On Linux, your ChatGPT state lives in the browser profile.
Copying only the .sqlite files. Codex keeps state in state_5.sqlite, thread_history_1.sqlite, memories_1.sqlite and others . A live copy without the -wal and -shm sidecars can be incomplete or corrupt. Stop the client and copy all three.
Treating the desktop database as the archive. The Codex desktop app can rebuild its index from the rollout JSONL files, but metadata that lives only in the database is lost, back up the JSONL files.
Assuming claude purge cleaned everything. It doesn’t touch shell-snapshots/, backups/ (older copies of ~/.claude.json) or files written by /heapdump. According to Anthropic, a heap snapshot contains the full conversation and any credentials the process held.

Restore into a staging profile

Reinstall the signed apps, plugins, and extensions from their sources.
Restore your config, then read every hook, MCP command, and executable path before the first launch.
Restore transcripts or memory only if you need them.
Log in again. Don’t restore auth.json, .credentials.json, cookies, or Keychain exports.
Resume one session and confirm that it uses the right account and repository.

Verify the backup set

To list what exists and how large it is, without printing any contents, run this on the machine you back up:

for p in "${CLAUDE_CONFIG_DIR:-$HOME/.claude}" "$HOME/.claude.json" \
 "${CODEX_HOME:-$HOME/.codex}" \
 "${XDG_CONFIG_HOME:-$HOME/.config}/Claude" \
 "$HOME/Library/Application Support/Claude" \
 "$HOME/Library/Application Support/Codex" \
 "$HOME/Library/Application Support/com.openai.chat" \
 "$HOME/Library/Group Containers/2DC432GLL2.group.com.openai.chat"; do
 [ -e "$p" ] && du -sh "$p"
done

Then check that no login file ended up in the backup:

find /path/to/mounted/backup \( -name .credentials.json -o -name auth.json \) -print

It should print nothing. If it prints anything, that backup is a credential store: encrypt it, shorten its retention, or rotate the logins.

If malware may already have run

Don’t treat a restore as the cleanup. From a clean device, revoke your Claude and ChatGPT sessions, then rotate API keys, MCP and connector credentials, and any secret that appeared in a transcript or in tool output. Any backup the malware could reach counts as exposed. Rebuild the user profile, restore only config you have reviewed, and log in again.

Caveats

Paths and retention rules change between releases. The ChatGPT and desktop paths come from build inspection, not from a published schema, so check them against your installed versions before you write a backup job. Sources: the .claude directory reference , Claude Code credential management , Codex config and state locations , Codex credential storage , Claude Desktop MCP config paths , and Electron’s app.getPath() .

Back up the config you wrote, and leave out the files the app uses for login. Run the find check above against your latest backup tonight.

Unprivileged LXC containers share the host's per-user kernel limits: 'Too many open files' and 'disk quota exceeded'

Wed, 21 Oct 2026 13:13:00 +0200

One day, two containers on the same host failed with errors that seemed unrelated. systemd in one of them:

systemd[4573]: Failed to create control group inotify object: Too many open files
systemd[4573]: Failed to allocate manager object: Too many open files
systemd[1]: Failed to start user@1000.service - User Manager for UID 1000.

Docker in another, where every container sat at Exited (128):

OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown

No file descriptors had run out, and no disk was full. Both errors come from per-user kernel quotas counted on the host, and every unprivileged container there was drawing from the same two.

The setup

Debian 13, kernel 6.12, LXC 6.0.4
More unprivileged containers than the hardware comfortably runs, all with the same idmap: container root is host UID 1000000
systemd in every container, Docker nested in some

Too many open files: inotify instances

systemd opens inotify instances to watch cgroups and units, and inotify_init1() fails with EMFILE when the user’s instance limit is reached. strerror turns that into “Too many open files”. The limit is fs.inotify.max_user_instances, 128 by default.

Count the instances per user on the host:

sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' 2>/dev/null \
 | cut -d/ -f3 | xargs -I{} ps -o user= -p {} | sort | uniq -c | sort -rn
cat /proc/sys/fs/inotify/max_user_instances

Host root held 23 instances, and uids in the container range (1000000 and up) held 104 between them: 127 of 128. On the first sighting, my coding agent got that far, then dropped it as a side effect of having started too many containers at once. That was half right, since each container’s systemd adds about five instances. A day later it came back.

The lxc package already ships a higher limit, and it wasn’t in effect:

$ cat /etc/sysctl.d/30-lxc-inotify.conf | grep -v '^#'
fs.inotify.max_user_instances = 1024
$ cat /proc/sys/fs/inotify/max_user_instances
128

The file was dated three days after the last boot. /etc/sysctl.d is applied by systemd-sysctl at boot, and the package’s install hadn’t applied it, so the setting sat on disk doing nothing.

Disk quota exceeded: kernel keys

runc gives each container it starts a new session keyring. add_key() fails with EDQUOT when the user’s key quota is full, and strerror calls that “Disk quota exceeded”. The quota is kernel.keys.maxkeys, 200 keys for any uid other than root. The host shows usage per UID:

$ cat /proc/key-users
 0: 127 126/126 107/1000000 2283/25000000
 1000: 12 12/12 12/200 86/20000
1000000: 198 198/198 198/200 4189/20000

198/200 are keys used against the quota. UID 1000000 is root in every container, so all the containers shared those 200 keys, and Docker’s restart attempts had nothing left.

The fix

Raise the limits and apply them right away, not at the next boot. Incus recommends 2000 keys and 2 MB in its server settings ; 1024 inotify instances matches the lxc package.

sudo tee /etc/sysctl.d/60-lxc-limits.conf <<'EOF'
fs.inotify.max_user_instances = 1024
kernel.keys.maxkeys = 2000
kernel.keys.maxbytes = 2000000
EOF
sudo sysctl -p /etc/sysctl.d/60-lxc-limits.conf

With Ansible, ansible.posix.sysctl with sysctl_set: true writes the file and sets the live value in one task.

Then run systemctl reset-failed in the affected container, and docker start for the dead Docker containers. Right after, uid 1000000 held 226 keys, already over the old limit.

Why the two quotas are counted differently

The two quotas look similar, but separate idmaps would fix only one.

Keys are counted per host UID, against global sysctls. Containers that share an idmap share one quota. A separate idmap per container (security.idmap.isolated in LXD and Incus) would give each its own.

inotify instances are counted per user namespace and charged up the chain of parent namespaces, at each level against the uid that created the child namespace. A new namespace’s own limit is INT_MAX, so the limit that bites is the host’s, charged to whoever started the containers. LXC started as root charges host root, which is why the 23 instances of host root’s own daemons counted toward the same 128. Separate idmaps don’t change that.

Takeaway

On a container host, “Too many open files” and “disk quota exceeded” can mean that every container together hit a per-user kernel quota on the host. Check the host’s counters before you debug the container that complained.

Run the find count and cat /proc/key-users on your own container host and compare them with the limits before a container restart finds out for you.

Decode a MikroTik autosupout.rif yourself: 20 lines of Python found the out-of-memory kernel panic

Wed, 14 Oct 2026 18:17:00 +0200

A MikroTik router kept rebooting every 10 minutes to 2.6 hours, with almost no traffic. After each boot, the log said:

system,error,critical router was rebooted without proper shutdown, probably kernel failure
system,error,critical kernel failure in previous boot
system,error,critical out of memory condition was detected
system,error,critical Automatic supout.rif file generated due to service malfunction, please contact MikroTik support and supply the generated file

You don’t have to wait for support. autosupout.rif is base64 with the bits in the other order, wrapped around zlib, and about 20 lines of Python turn it into plain text: the kernel’s last crash dumps, the log, /proc and the full config. That’s where the cause was:

Out of memory and no killable processes...
Kernel panic - not syncing: System is deadlocked on memory

The setup

MikroTik hAP ac² (IPQ4019, 128 MiB RAM, 16 MiB flash)
RouterOS 6.49.5, then 7.23.8 (long-term) with the legacy wireless package; both crashed the same way
Python 3 on a Mac to decode the file

The file format

Copy the file off the router and look at it:

scp admin@192.168.0.1:/autosupout.rif .
file autosupout.rif
head -2 autosupout.rif

autosupout.rif: ASCII text, with CRLF line terminators
--BEGIN ROUTEROS SUPOUT SECTION
uQWZiV3ZAgHnt3X72tNOSb+7xXFYO5HTy0WSEgfKeG3rdsd643J2xjtT37s5kVHKSIbOWiUNJVs9

It looks like base64, and base64 -d turns it into noise. The alphabet is standard, but each group of four characters is read as a 24-bit little-endian number: the first character holds the lowest 6 bits, and the first output byte is the lowest 8 bits. Decoded, each section is its name, a NUL byte, and a zlib stream.

The decoder

import os, re, sys, zlib

ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

def decode(text):
 text = re.sub(r'[^A-Za-z0-9+/=]', '', text)
 out = bytearray()
 for i in range(0, len(text), 4):
 group = text[i:i + 4].rstrip('=')
 value = sum(ALPHABET.index(c) << (6 * j) for j, c in enumerate(group))
 out += value.to_bytes(3, 'little')[:len(group) * 6 // 8]
 return bytes(out)

src, outdir = sys.argv[1], sys.argv[2]
os.makedirs(outdir, exist_ok=True)
sections = re.findall(r'--BEGIN ROUTEROS SUPOUT SECTION\r?\n(.*?)--END ROUTEROS SUPOUT SECTION',
 open(src, encoding='ascii').read(), re.S)
for n, section in enumerate(sections):
 name, _, data = decode(section).partition(b'\0')
 text = zlib.decompressobj().decompress(data)
 safe = re.sub(r'[^\w.-]', '_', name.decode(errors='replace'))
 with open(os.path.join(outdir, f'{n:02d}_{safe}.txt'), 'wb') as f:
 f.write(text)
 print(f'{n:02d} {safe} {len(text)}')

python3 supout.py autosupout.rif supout/

The 6.49.5 file had 76 sections and the 7.23.8 one 87. The useful ones:

Section	What’s in it
`oops`	the kernel log of the last few crashes, each with its timestamp
`log`	the RouterOS log since boot
`resource`	`/system resource` plus `/proc/meminfo`
`.proc`	binary-ish dump of `/proc`, including `slabinfo`, `vmallocinfo`, `buddyinfo` (use `LC_ALL=C grep -a`)
`export`	the full configuration
`.debug`	per-service debug output

What it showed

On 7.23.8, every crash in oops ended the same way, after a process table in which every process had the same score:

[ 691.378048][ T142] [ 150] 0 150 1667 153 28672 0 -1000 parser
...
[ 691.378492][ T142] Out of memory and no killable processes...
[ 691.378509][ T142] Kernel panic - not syncing: System is deadlocked on memory

RouterOS runs every process with oom_score_adj -1000, so the OOM killer has nothing to kill and the kernel panics. Any out-of-memory, however brief, is a reboot. 6.49.5 got there by another route, with the kernel set to panic on any OOM: Kernel panic - not syncing: Out of memory: compulsory panic_on_oom is enabled.

The processes weren’t the problem, at about 0.5 MiB each. resource, written a minute after boot, was:

MemTotal: 118248 kB
MemFree: 7368 kB
SUnreclaim: 52756 kB

One minute into the boot, the kernel already held 52 MiB of unreclaimable memory. The caches in slabinfo add up to only about 10 MiB, so the rest is allocated outside them. In two of the panics, other CPUs were inside the Wi-Fi driver (ath_poll, __w_qos_decrypt_mpdu in the wlan and ath modules), which makes it the main suspect, but turning one radio off didn’t settle it.

The same reading works on any embedded Linux box: check oom_score_adj in the OOM dump, then SUnreclaim against what the process table and slabinfo explain.

Read the timestamps with care

The router has no battery-backed clock. After a crash, it restores an old time, and NTP takes minutes to catch up, so log and supout timestamps right after a reboot were about 7 minutes early. Five “crashes” stamped within two seconds of each other were most likely one boot loop.

The supout itself was written a minute after the crash. The e-mail service tried to send the crash notice before PPPoE and DNS were up, died, and that service failure is what produced the file (.debug shows error: DNS resolve failed [code: 21] and then died).

Before you send it to anyone

The decoded sections held, in plain text, the PPPoE password, the WireGuard private key, a TSIG key, and a bearer token from a dynamic DNS script. Rotate those before you hand the file to MikroTik support or attach it to a forum post.

On a router this tight, delete the file after you copy it. The root of the file system is in RAM, and a 480 KiB supout is RAM the kernel needed:

/file remove [find name~"autosupout"]

Takeaway

A supout.rif holds the kernel log, /proc and your config, and you can read it in a minute. Next time your MikroTik writes one, decode it and read oops before you open a support ticket.