Systemd on Lessons learned

Unprivileged LXC containers share the host's per-user kernel limits: 'Too many open files' and 'disk quota exceeded'

Wed, 21 Oct 2026 13:13:00 +0200

One day, two containers on the same host failed with errors that seemed unrelated. systemd in one of them:

systemd[4573]: Failed to create control group inotify object: Too many open files
systemd[4573]: Failed to allocate manager object: Too many open files
systemd[1]: Failed to start user@1000.service - User Manager for UID 1000.

Docker in another, where every container sat at Exited (128):

OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown

No file descriptors had run out, and no disk was full. Both errors come from per-user kernel quotas counted on the host, and every unprivileged container there was drawing from the same two.

The setup

Debian 13, kernel 6.12, LXC 6.0.4
More unprivileged containers than the hardware comfortably runs, all with the same idmap: container root is host UID 1000000
systemd in every container, Docker nested in some

Too many open files: inotify instances

systemd opens inotify instances to watch cgroups and units, and inotify_init1() fails with EMFILE when the user’s instance limit is reached. strerror turns that into “Too many open files”. The limit is fs.inotify.max_user_instances, 128 by default.

Count the instances per user on the host:

sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' 2>/dev/null \
 | cut -d/ -f3 | xargs -I{} ps -o user= -p {} | sort | uniq -c | sort -rn
cat /proc/sys/fs/inotify/max_user_instances

Host root held 23 instances, and uids in the container range (1000000 and up) held 104 between them: 127 of 128. On the first sighting, my coding agent got that far, then dropped it as a side effect of having started too many containers at once. That was half right, since each container’s systemd adds about five instances. A day later it came back.

The lxc package already ships a higher limit, and it wasn’t in effect:

$ cat /etc/sysctl.d/30-lxc-inotify.conf | grep -v '^#'
fs.inotify.max_user_instances = 1024
$ cat /proc/sys/fs/inotify/max_user_instances
128

The file was dated three days after the last boot. /etc/sysctl.d is applied by systemd-sysctl at boot, and the package’s install hadn’t applied it, so the setting sat on disk doing nothing.

Disk quota exceeded: kernel keys

runc gives each container it starts a new session keyring. add_key() fails with EDQUOT when the user’s key quota is full, and strerror calls that “Disk quota exceeded”. The quota is kernel.keys.maxkeys, 200 keys for any uid other than root. The host shows usage per UID:

$ cat /proc/key-users
 0: 127 126/126 107/1000000 2283/25000000
 1000: 12 12/12 12/200 86/20000
1000000: 198 198/198 198/200 4189/20000

198/200 are keys used against the quota. UID 1000000 is root in every container, so all the containers shared those 200 keys, and Docker’s restart attempts had nothing left.

The fix

Raise the limits and apply them right away, not at the next boot. Incus recommends 2000 keys and 2 MB in its server settings ; 1024 inotify instances matches the lxc package.

sudo tee /etc/sysctl.d/60-lxc-limits.conf <<'EOF'
fs.inotify.max_user_instances = 1024
kernel.keys.maxkeys = 2000
kernel.keys.maxbytes = 2000000
EOF
sudo sysctl -p /etc/sysctl.d/60-lxc-limits.conf

With Ansible, ansible.posix.sysctl with sysctl_set: true writes the file and sets the live value in one task.

Then run systemctl reset-failed in the affected container, and docker start for the dead Docker containers. Right after, uid 1000000 held 226 keys, already over the old limit.

Why the two quotas are counted differently

The two quotas look similar, but separate idmaps would fix only one.

Keys are counted per host UID, against global sysctls. Containers that share an idmap share one quota. A separate idmap per container (security.idmap.isolated in LXD and Incus) would give each its own.

inotify instances are counted per user namespace and charged up the chain of parent namespaces, at each level against the uid that created the child namespace. A new namespace’s own limit is INT_MAX, so the limit that bites is the host’s, charged to whoever started the containers. LXC started as root charges host root, which is why the 23 instances of host root’s own daemons counted toward the same 128. Separate idmaps don’t change that.

Takeaway

On a container host, “Too many open files” and “disk quota exceeded” can mean that every container together hit a per-user kernel quota on the host. Check the host’s counters before you debug the container that complained.

Run the find count and cat /proc/key-users on your own container host and compare them with the limits before a container restart finds out for you.

The rollback that saved the host, and killed it 24 hours later

Sat, 17 Oct 2026 21:51:00 +0200

The rollback timer fired, the remote host got its address back, and the journal said all was well:

dhclient[1796120]: bound to 192.168.0.2 -- renewal in 34318 seconds.
systemd[1]: lxc-uplink-rollback.service: Deactivated successfully.

Exactly 24 hours later, to the second, the host dropped off the network and took SSH, the port forwards, and the VPN with it.

When a transient systemd-run unit finishes, systemd kills everything its command left running. If your rollback, or any other command run that way, starts a daemon, the daemon dies with the unit. Here it was dhclient, and the damage only showed when the DHCP lease expired.

The setup

Debian 13, systemd 257, ifupdown with ISC dhclient 4.4.3-P1
allow-hotplug enp0s25 / iface enp0s25 inet dhcp, with a 24-hour lease from the router
The only way in is SSH through a port forward to the host’s address; the console is days away

The dead man’s switch worked

Before a risky network change, an Ansible play arms a timer that undoes it, applies the change, and disarms the timer only if a fresh SSH login succeeds:

- name: Arm rollback timer
 ansible.builtin.command: >-
 systemd-run --collect --unit=lxc-uplink-rollback --on-active=5min
 /bin/sh -c 'ifdown -a --exclude=lo; ip link del br0;
 cp /etc/network/interfaces.rollback /etc/network/interfaces;
 ifup -a; ifup -a --allow=hotplug'

# ... apply the change ...

- name: Drop the cached SSH connection
 ansible.builtin.meta: reset_connection

- name: Prove a fresh SSH login still works
 ansible.builtin.wait_for_connection:
 delay: 10
 timeout: 420

- name: Disarm rollback timer
 ansible.builtin.command: systemctl stop lxc-uplink-rollback.timer

Without meta: reset_connection, Ansible reuses its open SSH connection and a broken change looks fine.

The change moved the NIC into a bridge, br0. The bridge got the old address back, but forwarded SSH from the internet stopped reaching the host. (I never found out why, and it isn’t the point here.) Five minutes later the timer restored the old config, the host came back, and I moved on.

What went wrong

Look at the rollback unit again. Its main process is /bin/sh. ifup starts dhclient, which forks into the background, and then sh exits. The unit is now done, and a finished service gets its cgroup cleaned up: by default (KillMode=control-group), systemd kills every process still in it. That includes the dhclient that had just been “bound”.

A killed dhclient doesn’t release the lease, so the address stays on the interface with a lifetime of 86400 seconds. Nothing renews it. When the lifetime runs out, the kernel removes the address, and avahi is the first to notice:

Oct 17 21:08:42 avahi-daemon[936]: Withdrawing address record for 192.168.0.2 on enp0s25.

The clue was in the journal on day one. When the rollback started, its dhclient logged:

dhclient[1796009]: Removed stale PID file

That stale PID file belonged to the dhclient on br0. Ansible had started it five minutes earlier from another systemd-run unit, and systemd had killed it the same way, but the rollback replaced it before anyone noticed.

At boot, ifup@enp0s25.service starts the same dhclient. That unit stays active (exited), so its cgroup is never cleaned up, which is how the boot-time dhclient had kept renewing for about six weeks.

You can see the behavior in two lines:

sudo systemd-run --unit=demo /bin/sh -c 'sleep 600 &'
sleep 1; pgrep -a sleep # nothing: systemd killed it when sh exited

The fix

To get back in, I went through a VM that had its own macvlan leg on the same NIC and ran sudo dhclient enp0s25. With ssh -J, -i applies only to the last hop, so give the jump host its key through ProxyCommand:

ssh -i ~/.ssh/key -o IdentitiesOnly=yes \
 -o ProxyCommand="ssh -i ~/.ssh/key -o IdentitiesOnly=yes -W %h:%p user@jump.example.com" \
 user@target.example.com

The real fix was to stop depending on a long-running DHCP client. The router reserves the address for the host anyway, so the host now has a static stanza. Ansible applies it without restarting networking, because that would cut off the run and the only way in:

ip address replace 192.168.0.2/24 dev enp0s25 valid_lft forever preferred_lft forever
ip route replace default via 192.168.0.1 dev enp0s25
pkill --signal KILL -f '(^|/)dhclient .*enp0s25$' || true

SIGKILL is deliberate: a dhclient that stops cleanly removes the address it holds, the one this very run is connected through.

If the host has to stay on DHCP, keep daemons out of transient units. Start the daemon through the unit that owns it at boot (here ifup@enp0s25.service), or let the transient unit leave its children alone with systemd-run -p KillMode=process. I didn’t retry the bridge, so I haven’t tested either variant in this rollback.

To check which unit a daemon really belongs to:

cat /proc/$(pgrep -o dhclient)/cgroup

Takeaway

Deactivated successfully only says the unit finished; whatever it started may have died with it. A killed DHCP client fails a whole lease later, long after you have stopped looking.

Search your playbooks and scripts for systemd-run calls that run ifup, dhclient or anything else that forks into the background. Then check which cgroup that process ends up in.