Lxc on Lessons learned

Unprivileged LXC containers share the host's per-user kernel limits: 'Too many open files' and 'disk quota exceeded'

Wed, 21 Oct 2026 13:13:00 +0200

One day, two containers on the same host failed with errors that seemed unrelated. systemd in one of them:

systemd[4573]: Failed to create control group inotify object: Too many open files
systemd[4573]: Failed to allocate manager object: Too many open files
systemd[1]: Failed to start user@1000.service - User Manager for UID 1000.

Docker in another, where every container sat at Exited (128):

OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown

No file descriptors had run out, and no disk was full. Both errors come from per-user kernel quotas counted on the host, and every unprivileged container there was drawing from the same two.

The setup

Debian 13, kernel 6.12, LXC 6.0.4
More unprivileged containers than the hardware comfortably runs, all with the same idmap: container root is host UID 1000000
systemd in every container, Docker nested in some

Too many open files: inotify instances

systemd opens inotify instances to watch cgroups and units, and inotify_init1() fails with EMFILE when the user’s instance limit is reached. strerror turns that into “Too many open files”. The limit is fs.inotify.max_user_instances, 128 by default.

Count the instances per user on the host:

sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' 2>/dev/null \
 | cut -d/ -f3 | xargs -I{} ps -o user= -p {} | sort | uniq -c | sort -rn
cat /proc/sys/fs/inotify/max_user_instances

Host root held 23 instances, and uids in the container range (1000000 and up) held 104 between them: 127 of 128. On the first sighting, my coding agent got that far, then dropped it as a side effect of having started too many containers at once. That was half right, since each container’s systemd adds about five instances. A day later it came back.

The lxc package already ships a higher limit, and it wasn’t in effect:

$ cat /etc/sysctl.d/30-lxc-inotify.conf | grep -v '^#'
fs.inotify.max_user_instances = 1024
$ cat /proc/sys/fs/inotify/max_user_instances
128

The file was dated three days after the last boot. /etc/sysctl.d is applied by systemd-sysctl at boot, and the package’s install hadn’t applied it, so the setting sat on disk doing nothing.

Disk quota exceeded: kernel keys

runc gives each container it starts a new session keyring. add_key() fails with EDQUOT when the user’s key quota is full, and strerror calls that “Disk quota exceeded”. The quota is kernel.keys.maxkeys, 200 keys for any uid other than root. The host shows usage per UID:

$ cat /proc/key-users
 0: 127 126/126 107/1000000 2283/25000000
 1000: 12 12/12 12/200 86/20000
1000000: 198 198/198 198/200 4189/20000

198/200 are keys used against the quota. UID 1000000 is root in every container, so all the containers shared those 200 keys, and Docker’s restart attempts had nothing left.

The fix

Raise the limits and apply them right away, not at the next boot. Incus recommends 2000 keys and 2 MB in its server settings ; 1024 inotify instances matches the lxc package.

sudo tee /etc/sysctl.d/60-lxc-limits.conf <<'EOF'
fs.inotify.max_user_instances = 1024
kernel.keys.maxkeys = 2000
kernel.keys.maxbytes = 2000000
EOF
sudo sysctl -p /etc/sysctl.d/60-lxc-limits.conf

With Ansible, ansible.posix.sysctl with sysctl_set: true writes the file and sets the live value in one task.

Then run systemctl reset-failed in the affected container, and docker start for the dead Docker containers. Right after, uid 1000000 held 226 keys, already over the old limit.

Why the two quotas are counted differently

The two quotas look similar, but separate idmaps would fix only one.

Keys are counted per host UID, against global sysctls. Containers that share an idmap share one quota. A separate idmap per container (security.idmap.isolated in LXD and Incus) would give each its own.

inotify instances are counted per user namespace and charged up the chain of parent namespaces, at each level against the uid that created the child namespace. A new namespace’s own limit is INT_MAX, so the limit that bites is the host’s, charged to whoever started the containers. LXC started as root charges host root, which is why the 23 instances of host root’s own daemons counted toward the same 128. Separate idmaps don’t change that.

Takeaway

On a container host, “Too many open files” and “disk quota exceeded” can mean that every container together hit a per-user kernel quota on the host. Check the host’s counters before you debug the container that complained.

Run the find count and cat /proc/key-users on your own container host and compare them with the limits before a container restart finds out for you.

Rootless Podman backup without app.ini, and a restore that fails with 'Cannot change ownership to uid 100999'

Thu, 15 Oct 2026 17:23:00 +0200

Restoring a Gitea backup to a new host failed on every directory:

tar: data/gitea: Cannot change ownership to uid 100999, gid 100999: Invalid argument
tar: data/gitea/queues: Cannot change ownership to uid 100999, gid 100999: Invalid argument

That one is easy to fix. The worse problem made no noise at all, and I found it just before: the nightly backup didn’t contain app.ini. If you back up a rootless Podman volume with a plain tar as the host user, yours may be missing files too.

The setup

Gitea 1.27 and PostgreSQL ran in rootless Podman on a VPS, with Gitea’s data in a bind mount under ~/gitea/data. The nightly backup ran as the user who owns the containers:

podman exec gitea_db_1 pg_dump -U gitea gitea | bzip2 -z9 > ~/backup/gitea-db-$(date +%y%m%d).sql.bz2
tar c -C ~/gitea/data gitea | bzip2 -z9 > ~/backup/gitea-data-$(date +%y%m%d).tar.bz2

The target was Gitea in Docker, inside an unprivileged LXC container on Debian.

The archive looked fine. It unpacked, and it contained all the repositories.

Compare the archive with the live tree

Before restoring, I compared the archive’s file list with the live data. List the live side from inside the user namespace, so that nothing is hidden from you:

ssh vps.example.com 'cd ~/gitea/data && podman unshare find gitea -type f' | sort > live.txt
tar tjf gitea-data.tar.bz2 | grep -v '/$' | sort > backup.txt
comm -23 live.txt backup.txt > missing.txt
wc -l live.txt backup.txt missing.txt

 12359 live.txt
 12342 backup.txt
 17 missing.txt

Among the 17 were the files Gitea can least do without:

gitea/gitea/conf/app.ini
gitea/gitea/jwt/private.pem
gitea/git/.ssh/authorized_keys
gitea/gitea/avatars/... (3 files)
gitea/gitea/indexers/... (4 files)
gitea/gitea/sessions/... (2 files)

app.ini holds Gitea’s SECRET_KEY, its internal tokens, and the mail settings. private.pem signs its OAuth2 tokens. A restore from this backup would have started with fresh secrets: anything encrypted with the old SECRET_KEY, such as two-factor secrets, would no longer decrypt, and the mailer would be unconfigured.

Why tar skipped them

Rootless Podman maps the container’s UIDs onto the subordinate range in /etc/subuid. With the usual entry:

deploy:100000:65536

container root is you, and container UID 1000 (Gitea’s git user) is host UID 100000 + 999 = 100999. Files that Gitea creates with mode 0600 belong to 100999, and you can’t read them from the host. The repository objects were 0644 in 0755 directories, so tar read those, and the archive looked complete.

tar does complain, on stderr, and exits with status 2. In tar c ... | bzip2 > file, though, the pipeline’s exit status is bzip2’s, which is 0. Without set -o pipefail and someone reading the job’s stderr, nothing tells you.

Fix the backup

Run tar inside the user namespace, where you are root over every mapped UID:

set -o pipefail
podman unshare tar c -C ~/gitea/data gitea | bzip2 -z9 > ~/backup/gitea-data-$(date +%y%m%d).tar.bz2

The archive then records the IDs as the container sees them (1000), not the host’s 100999.

To rescue the files without touching the live host, fetch only the missing ones the same way, then merge them into the old archive with bsdtar’s @archive syntax:

ssh vps.example.com 'cd ~/gitea/data && podman unshare tar -c -T -' < missing.txt > missing.tar
bsdtar -czf gitea-complete.tar.gz @gitea-data.tar.bz2 @missing.tar

Fix the restore

GNU tar run as root restores the owners recorded in the archive (--same-owner is its default for root). Inside an unprivileged LXC container, root is itself a mapped user, and the container’s UIDs usually cover only 0 to 65535. UID 100999 doesn’t exist there, so chown fails with EINVAL, which tar reports as Invalid argument.

Don’t restore owners from an archive made somewhere else. Unpack with --no-same-owner and set the owners the new Gitea expects:

tar -C /var/tmp/gitea-restore --no-same-owner -xzf gitea-complete.tar.gz
cp -a /var/tmp/gitea-restore/gitea/. /srv/gitea/data/
chown -R 1000:1000 /srv/gitea/data/git /srv/gitea/data/gitea
chown -R root:root /srv/gitea/data/ssh

This also covers the merged archive, which carries 100999 for some files and 1000 for others.

After the restore, check against the live instance: the repository count, git fsck on every repository, and the SSH host key fingerprint on the Git SSH port, so existing clones still trust the new host. All three matched.

Takeaway

Under rootless Podman, a host-side tar leaves out the files the container keeps private, which are often its secrets, and the pipeline still exits 0. Comparing the archive’s listing with podman unshare find takes a minute and shows the gap.

Run the podman unshare find and comm check above against your own latest backup and see what comes back.

No DHCP for an LXC container while iptables counts the packets as ACCEPT: firewalld rejects them in its own nftables table

Tue, 13 Oct 2026 22:56:00 +0200

A new LXC container came up with an IPv6 address and no IPv4 address. According to iptables, the host allowed DHCP from the container bridge, and the counter on that rule kept growing:

$ sudo iptables -L INPUT -v -n | grep dpt:67
 0 0 ACCEPT tcp -- lxcbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
 26 7956 ACCEPT udp -- lxcbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67

Twenty-six packets accepted, and dnsmasq, listening on that bridge, never logged a single DHCPDISCOVER.

With nftables, accept only ends the chain it sits in. Any other base chain on the same hook still gets the packet. On this host, the next one belonged to firewalld, and firewalld rejected everything arriving on an interface that wasn’t in any of its zones.

The setup

Debian 13 host with the stock lxc-net bridge: lxcbr0, 10.0.3.0/24, dnsmasq at 10.0.3.1
firewalld 2.3.1 with FirewallBackend=nftables, default zone public
iptables 1.8.11, the nf_tables flavor Debian installs as iptables
an unprivileged Debian 13 container running systemd-networkd with DHCP=true

When lxc-net brings up the bridge, it adds its own iptables rules: ACCEPT for DNS and DHCP on lxcbr0, ACCEPT for forwarding, and a MASQUERADE for the subnet. It knows nothing about firewalld.

Following the packet

The first visible symptom was the container’s bootstrap script, which spent nine minutes waiting on DNS lookups that timed out. From there, each check moved the problem one step along the path:

The container sent requests. With debug logging, systemd-networkd showed DHCPv4 client: DISCOVER in the selecting state.
The host received them. The ACCEPT counter went from 20 to 26 to 29 while I made the client retry with networkctl reconfigure eth0.
dnsmasq had the right socket open: 0.0.0.0%lxcbr0:67.
dnsmasq received nothing. strace showed it sitting in poll(). During a retry, it read only an IPv6 router solicitation, which it answered. That is where the IPv6 address came from.

AppArmor (no denials), reverse-path filtering (loose on the bridge), br_netfilter (not loaded), leftover iptables-legacy tables (empty), and socket buffer drops (RcvbufErrors 0) all came up clean. The host had no tcpdump, and no nft either:

$ sudo nft list ruleset
sudo: nft: command not found

That left iptables as the only view of the firewall, and iptables shows only the tables it created. So the packet passed an ACCEPT at the input hook and never reached the socket. Listing the running services found a second firewall:

firewalld.service loaded active running firewalld - dynamic firewall daemon

$ sudo firewall-cmd --get-zone-of-interface=lxcbr0
no zone

Two tables on one hook

After apt install nftables (the binary is enough; leave nftables.service disabled), you can see both chains on the input hook. Trimmed:

$ sudo nft list chains
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}
}
table inet firewalld {
	chain filter_INPUT {
		type filter hook input priority filter + 10; policy accept;
	}
}

ip filter holds lxc-net’s rules, written through iptables-nft. inet firewalld runs right after it, and an interface that no zone claims falls through to the default zone:

chain filter_INPUT_POLICIES {
	...
	jump filter_IN_policy_allow-host-ipv6
	jump filter_IN_public
	reject with icmpx admin-prohibited
}

public allowed SSH and the DHCPv6 client, but no DHCP server and no DNS, so the DISCOVER hit the final reject. The zone chain also ends with meta l4proto { icmp, ipv6-icmp } accept, which is why the router solicitation got through.

The fix

Put the bridge in a firewalld zone. trusted accepts everything, including forwarded traffic, which the containers need for internet access:

sudo firewall-cmd --zone=trusted --add-interface=lxcbr0
sudo firewall-cmd --permanent --zone=trusted --add-interface=lxcbr0

Or with Ansible:

- name: Trust the container bridge in firewalld
 ansible.posix.firewalld:
 zone: trusted
 interface: lxcbr0
 permanent: true
 immediate: true
 state: enabled

The container got its lease on the client’s next retry, without restarting anything. trusted also lets containers reach every port on the host. If that’s too much, give the bridge a zone of its own that allows only DHCP, DNS, and forwarding. The same applies to any interface that another tool creates, such as a VPN tunnel.

Why the counter lied

The nftables wiki says that “an accept verdict … isn’t necessarily final”, while “drops take immediate effect, with no further rules or chains being evaluated.” Base chains on one hook run in priority order. An accept ends one chain, and a drop or reject in any chain wins. When two firewalls share a host, the stricter one decides, and the permissive one keeps counting packets that never arrive.

Most of the time you won’t see the second table. iptables -L lists only what iptables-nft created, and a host running firewalld may not have the nft command at all. (That missing binary also leaves fail2ban unable to ban anyone .)

Takeaway

A counter on an ACCEPT rule only proves that the packet reached that rule. When an accepted packet still doesn’t arrive, list every base chain on the hook and find the later one that drops it.

If your host runs firewalld alongside anything that writes its own iptables rules, run sudo nft list chains | grep hook and count the chains on input and forward.