Debian on Lessons learned

Unprivileged LXC containers share the host's per-user kernel limits: 'Too many open files' and 'disk quota exceeded'

Wed, 21 Oct 2026 13:13:00 +0200

One day, two containers on the same host failed with errors that seemed unrelated. systemd in one of them:

systemd[4573]: Failed to create control group inotify object: Too many open files
systemd[4573]: Failed to allocate manager object: Too many open files
systemd[1]: Failed to start user@1000.service - User Manager for UID 1000.

Docker in another, where every container sat at Exited (128):

OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown

No file descriptors had run out, and no disk was full. Both errors come from per-user kernel quotas counted on the host, and every unprivileged container there was drawing from the same two.

The setup

Debian 13, kernel 6.12, LXC 6.0.4
More unprivileged containers than the hardware comfortably runs, all with the same idmap: container root is host UID 1000000
systemd in every container, Docker nested in some

Too many open files: inotify instances

systemd opens inotify instances to watch cgroups and units, and inotify_init1() fails with EMFILE when the user’s instance limit is reached. strerror turns that into “Too many open files”. The limit is fs.inotify.max_user_instances, 128 by default.

Count the instances per user on the host:

sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' 2>/dev/null \
 | cut -d/ -f3 | xargs -I{} ps -o user= -p {} | sort | uniq -c | sort -rn
cat /proc/sys/fs/inotify/max_user_instances

Host root held 23 instances, and uids in the container range (1000000 and up) held 104 between them: 127 of 128. On the first sighting, my coding agent got that far, then dropped it as a side effect of having started too many containers at once. That was half right, since each container’s systemd adds about five instances. A day later it came back.

The lxc package already ships a higher limit, and it wasn’t in effect:

$ cat /etc/sysctl.d/30-lxc-inotify.conf | grep -v '^#'
fs.inotify.max_user_instances = 1024
$ cat /proc/sys/fs/inotify/max_user_instances
128

The file was dated three days after the last boot. /etc/sysctl.d is applied by systemd-sysctl at boot, and the package’s install hadn’t applied it, so the setting sat on disk doing nothing.

Disk quota exceeded: kernel keys

runc gives each container it starts a new session keyring. add_key() fails with EDQUOT when the user’s key quota is full, and strerror calls that “Disk quota exceeded”. The quota is kernel.keys.maxkeys, 200 keys for any uid other than root. The host shows usage per UID:

$ cat /proc/key-users
 0: 127 126/126 107/1000000 2283/25000000
 1000: 12 12/12 12/200 86/20000
1000000: 198 198/198 198/200 4189/20000

198/200 are keys used against the quota. UID 1000000 is root in every container, so all the containers shared those 200 keys, and Docker’s restart attempts had nothing left.

The fix

Raise the limits and apply them right away, not at the next boot. Incus recommends 2000 keys and 2 MB in its server settings ; 1024 inotify instances matches the lxc package.

sudo tee /etc/sysctl.d/60-lxc-limits.conf <<'EOF'
fs.inotify.max_user_instances = 1024
kernel.keys.maxkeys = 2000
kernel.keys.maxbytes = 2000000
EOF
sudo sysctl -p /etc/sysctl.d/60-lxc-limits.conf

With Ansible, ansible.posix.sysctl with sysctl_set: true writes the file and sets the live value in one task.

Then run systemctl reset-failed in the affected container, and docker start for the dead Docker containers. Right after, uid 1000000 held 226 keys, already over the old limit.

Why the two quotas are counted differently

The two quotas look similar, but separate idmaps would fix only one.

Keys are counted per host UID, against global sysctls. Containers that share an idmap share one quota. A separate idmap per container (security.idmap.isolated in LXD and Incus) would give each its own.

inotify instances are counted per user namespace and charged up the chain of parent namespaces, at each level against the uid that created the child namespace. A new namespace’s own limit is INT_MAX, so the limit that bites is the host’s, charged to whoever started the containers. LXC started as root charges host root, which is why the 23 instances of host root’s own daemons counted toward the same 128. Separate idmaps don’t change that.

Takeaway

On a container host, “Too many open files” and “disk quota exceeded” can mean that every container together hit a per-user kernel quota on the host. Check the host’s counters before you debug the container that complained.

Run the find count and cat /proc/key-users on your own container host and compare them with the limits before a container restart finds out for you.

The rollback that saved the host, and killed it 24 hours later

Sat, 17 Oct 2026 21:51:00 +0200

The rollback timer fired, the remote host got its address back, and the journal said all was well:

dhclient[1796120]: bound to 192.168.0.2 -- renewal in 34318 seconds.
systemd[1]: lxc-uplink-rollback.service: Deactivated successfully.

Exactly 24 hours later, to the second, the host dropped off the network and took SSH, the port forwards, and the VPN with it.

When a transient systemd-run unit finishes, systemd kills everything its command left running. If your rollback, or any other command run that way, starts a daemon, the daemon dies with the unit. Here it was dhclient, and the damage only showed when the DHCP lease expired.

The setup

Debian 13, systemd 257, ifupdown with ISC dhclient 4.4.3-P1
allow-hotplug enp0s25 / iface enp0s25 inet dhcp, with a 24-hour lease from the router
The only way in is SSH through a port forward to the host’s address; the console is days away

The dead man’s switch worked

Before a risky network change, an Ansible play arms a timer that undoes it, applies the change, and disarms the timer only if a fresh SSH login succeeds:

- name: Arm rollback timer
 ansible.builtin.command: >-
 systemd-run --collect --unit=lxc-uplink-rollback --on-active=5min
 /bin/sh -c 'ifdown -a --exclude=lo; ip link del br0;
 cp /etc/network/interfaces.rollback /etc/network/interfaces;
 ifup -a; ifup -a --allow=hotplug'

# ... apply the change ...

- name: Drop the cached SSH connection
 ansible.builtin.meta: reset_connection

- name: Prove a fresh SSH login still works
 ansible.builtin.wait_for_connection:
 delay: 10
 timeout: 420

- name: Disarm rollback timer
 ansible.builtin.command: systemctl stop lxc-uplink-rollback.timer

Without meta: reset_connection, Ansible reuses its open SSH connection and a broken change looks fine.

The change moved the NIC into a bridge, br0. The bridge got the old address back, but forwarded SSH from the internet stopped reaching the host. (I never found out why, and it isn’t the point here.) Five minutes later the timer restored the old config, the host came back, and I moved on.

What went wrong

Look at the rollback unit again. Its main process is /bin/sh. ifup starts dhclient, which forks into the background, and then sh exits. The unit is now done, and a finished service gets its cgroup cleaned up: by default (KillMode=control-group), systemd kills every process still in it. That includes the dhclient that had just been “bound”.

A killed dhclient doesn’t release the lease, so the address stays on the interface with a lifetime of 86400 seconds. Nothing renews it. When the lifetime runs out, the kernel removes the address, and avahi is the first to notice:

Oct 17 21:08:42 avahi-daemon[936]: Withdrawing address record for 192.168.0.2 on enp0s25.

The clue was in the journal on day one. When the rollback started, its dhclient logged:

dhclient[1796009]: Removed stale PID file

That stale PID file belonged to the dhclient on br0. Ansible had started it five minutes earlier from another systemd-run unit, and systemd had killed it the same way, but the rollback replaced it before anyone noticed.

At boot, ifup@enp0s25.service starts the same dhclient. That unit stays active (exited), so its cgroup is never cleaned up, which is how the boot-time dhclient had kept renewing for about six weeks.

You can see the behavior in two lines:

sudo systemd-run --unit=demo /bin/sh -c 'sleep 600 &'
sleep 1; pgrep -a sleep # nothing: systemd killed it when sh exited

The fix

To get back in, I went through a VM that had its own macvlan leg on the same NIC and ran sudo dhclient enp0s25. With ssh -J, -i applies only to the last hop, so give the jump host its key through ProxyCommand:

ssh -i ~/.ssh/key -o IdentitiesOnly=yes \
 -o ProxyCommand="ssh -i ~/.ssh/key -o IdentitiesOnly=yes -W %h:%p user@jump.example.com" \
 user@target.example.com

The real fix was to stop depending on a long-running DHCP client. The router reserves the address for the host anyway, so the host now has a static stanza. Ansible applies it without restarting networking, because that would cut off the run and the only way in:

ip address replace 192.168.0.2/24 dev enp0s25 valid_lft forever preferred_lft forever
ip route replace default via 192.168.0.1 dev enp0s25
pkill --signal KILL -f '(^|/)dhclient .*enp0s25$' || true

SIGKILL is deliberate: a dhclient that stops cleanly removes the address it holds, the one this very run is connected through.

If the host has to stay on DHCP, keep daemons out of transient units. Start the daemon through the unit that owns it at boot (here ifup@enp0s25.service), or let the transient unit leave its children alone with systemd-run -p KillMode=process. I didn’t retry the bridge, so I haven’t tested either variant in this rollback.

To check which unit a daemon really belongs to:

cat /proc/$(pgrep -o dhclient)/cgroup

Takeaway

Deactivated successfully only says the unit finished; whatever it started may have died with it. A killed DHCP client fails a whole lease later, long after you have stopped looking.

Search your playbooks and scripts for systemd-run calls that run ifup, dhclient or anything else that forks into the background. Then check which cgroup that process ends up in.

fail2ban that never banned anyone: '/bin/sh: 1: nft: not found'

Fri, 16 Oct 2026 15:33:00 +0200

fail2ban said the address was banned:

$ sudo fail2ban-client status nginx-limit-req
...
 `- Banned IP list:	203.0.113.10

From that address, right after:

$ curl -s -o /dev/null -w '%{http_code}\n' https://example.com/
302

The reason was in /var/log/fail2ban.log all along:

2026-03-14 10:21:05,205 fail2ban.utils [1293]: ERROR 7f5e8fbb0d30 -- exec: nft add table inet f2b-table
2026-03-14 10:21:05,206 fail2ban.utils [1293]: ERROR 7f5e8fbb0d30 -- stderr: '/bin/sh: 1: nft: not found'

On Debian, fail2ban bans through the nft command. A host that runs firewalld can easily lack it. fail2ban then counts failures, lists bans, and blocks nothing, including the sshd jail.

The setup

Debian 13, fail2ban 1.1.0-8, firewalld 2.3.1 with FirewallBackend=nftables
nginx as a reverse proxy in front of a few web apps, SSH forwarded from the router
fail2ban set up by hand, with the stock sshd jail

Debian’s /etc/fail2ban/jail.d/defaults-debian.conf picks the ban action:

[DEFAULT]
banaction = nftables
banaction_allports = nftables[type=allports]

How it showed up

I was adding jails for the web logins, so I tested them from a VPS outside every whitelisted network. Testing from home proves nothing when home is in ignoreip. Twenty-five quick POSTs to a login endpoint:

for i in $(seq 25); do
 curl -s -o /dev/null -w '%{http_code} ' -d client_id=probe https://example.com/login
done

The first 11 got 400 from the app, the rest 429 from nginx’s rate limit, and the nginx-limit-req jail banned the VPS as it should. Then the next request got a 302, as if nothing had happened.

sudo nft list table inet f2b-table answered sudo: nft: command not found. The log held ten lines like the ones above. Each failed ban left two ERROR lines in a log nobody reads, while fail2ban-client status still listed the bans.

The fix

Install the nft command:

sudo apt install nftables
sudo systemctl restart fail2ban

Leave nftables.service disabled, as the package installs it. Debian’s /etc/nftables.conf starts with flush ruleset, so enabling the service would wipe firewalld’s rules at the next boot. fail2ban doesn’t need the service, only the binary. It keeps its bans in a table of its own, inet f2b-table, next to firewalld’s inet firewalld.

On restart, fail2ban restores the bans it has on record, so the ones that had silently failed now take effect:

fail2ban.actions [1649343]: NOTICE [nginx-limit-req] Restore Ban 203.0.113.10

Give it a few seconds before you check. Right after the restart, nft list table inet f2b-table still said No such file or directory. A moment later, it showed the set and the rule:

table inet f2b-table {
	set addr-set-nginx-limit-req {
		type ipv4_addr
		elements = { 203.0.113.10 }
	}
	chain f2b-chain {
		type filter hook input priority filter - 1; policy accept;
		tcp dport { 80, 443 } ip saddr @addr-set-nginx-limit-req reject with icmp port-unreachable
	}
}

This time, curl on the VPS couldn’t connect at all (exit code 7). If you manage the host with Ansible or similar, add nftables to the package list alongside fail2ban so a rebuilt host doesn’t lose it again.

Why the binary was missing

firewalld talks to the kernel through the python3-nftables bindings, not the nft command, so it doesn’t pull in the nftables package. fail2ban only recommends it as an alternative:

$ apt-cache show fail2ban | grep ^Recommends
Recommends: nftables | iptables, whois, python3-pyinotify, python3-setuptools

iptables was already installed (firewalld recommends it), so apt considered the recommendation met. The ban action still called nft.

The same thing can happen with any ban action whose command isn’t on the host: the action fails, and the address stays on the ban list anyway.

Takeaway

Banned IP list shows what fail2ban decided, not what the firewall does. The only proof that a ban works is a blocked connection from the banned address, and grep ERROR /var/log/fail2ban.log tells you why one isn’t.

Get one of your own hosts to ban a machine outside its whitelist, then try to connect from that machine.

No DHCP for an LXC container while iptables counts the packets as ACCEPT: firewalld rejects them in its own nftables table

Tue, 13 Oct 2026 22:56:00 +0200

A new LXC container came up with an IPv6 address and no IPv4 address. According to iptables, the host allowed DHCP from the container bridge, and the counter on that rule kept growing:

$ sudo iptables -L INPUT -v -n | grep dpt:67
 0 0 ACCEPT tcp -- lxcbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
 26 7956 ACCEPT udp -- lxcbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67

Twenty-six packets accepted, and dnsmasq, listening on that bridge, never logged a single DHCPDISCOVER.

With nftables, accept only ends the chain it sits in. Any other base chain on the same hook still gets the packet. On this host, the next one belonged to firewalld, and firewalld rejected everything arriving on an interface that wasn’t in any of its zones.

The setup

Debian 13 host with the stock lxc-net bridge: lxcbr0, 10.0.3.0/24, dnsmasq at 10.0.3.1
firewalld 2.3.1 with FirewallBackend=nftables, default zone public
iptables 1.8.11, the nf_tables flavor Debian installs as iptables
an unprivileged Debian 13 container running systemd-networkd with DHCP=true

When lxc-net brings up the bridge, it adds its own iptables rules: ACCEPT for DNS and DHCP on lxcbr0, ACCEPT for forwarding, and a MASQUERADE for the subnet. It knows nothing about firewalld.

Following the packet

The first visible symptom was the container’s bootstrap script, which spent nine minutes waiting on DNS lookups that timed out. From there, each check moved the problem one step along the path:

The container sent requests. With debug logging, systemd-networkd showed DHCPv4 client: DISCOVER in the selecting state.
The host received them. The ACCEPT counter went from 20 to 26 to 29 while I made the client retry with networkctl reconfigure eth0.
dnsmasq had the right socket open: 0.0.0.0%lxcbr0:67.
dnsmasq received nothing. strace showed it sitting in poll(). During a retry, it read only an IPv6 router solicitation, which it answered. That is where the IPv6 address came from.

AppArmor (no denials), reverse-path filtering (loose on the bridge), br_netfilter (not loaded), leftover iptables-legacy tables (empty), and socket buffer drops (RcvbufErrors 0) all came up clean. The host had no tcpdump, and no nft either:

$ sudo nft list ruleset
sudo: nft: command not found

That left iptables as the only view of the firewall, and iptables shows only the tables it created. So the packet passed an ACCEPT at the input hook and never reached the socket. Listing the running services found a second firewall:

firewalld.service loaded active running firewalld - dynamic firewall daemon

$ sudo firewall-cmd --get-zone-of-interface=lxcbr0
no zone

Two tables on one hook

After apt install nftables (the binary is enough; leave nftables.service disabled), you can see both chains on the input hook. Trimmed:

$ sudo nft list chains
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}
}
table inet firewalld {
	chain filter_INPUT {
		type filter hook input priority filter + 10; policy accept;
	}
}

ip filter holds lxc-net’s rules, written through iptables-nft. inet firewalld runs right after it, and an interface that no zone claims falls through to the default zone:

chain filter_INPUT_POLICIES {
	...
	jump filter_IN_policy_allow-host-ipv6
	jump filter_IN_public
	reject with icmpx admin-prohibited
}

public allowed SSH and the DHCPv6 client, but no DHCP server and no DNS, so the DISCOVER hit the final reject. The zone chain also ends with meta l4proto { icmp, ipv6-icmp } accept, which is why the router solicitation got through.

The fix

Put the bridge in a firewalld zone. trusted accepts everything, including forwarded traffic, which the containers need for internet access:

sudo firewall-cmd --zone=trusted --add-interface=lxcbr0
sudo firewall-cmd --permanent --zone=trusted --add-interface=lxcbr0

Or with Ansible:

- name: Trust the container bridge in firewalld
 ansible.posix.firewalld:
 zone: trusted
 interface: lxcbr0
 permanent: true
 immediate: true
 state: enabled

The container got its lease on the client’s next retry, without restarting anything. trusted also lets containers reach every port on the host. If that’s too much, give the bridge a zone of its own that allows only DHCP, DNS, and forwarding. The same applies to any interface that another tool creates, such as a VPN tunnel.

Why the counter lied

The nftables wiki says that “an accept verdict … isn’t necessarily final”, while “drops take immediate effect, with no further rules or chains being evaluated.” Base chains on one hook run in priority order. An accept ends one chain, and a drop or reject in any chain wins. When two firewalls share a host, the stricter one decides, and the permissive one keeps counting packets that never arrive.

Most of the time you won’t see the second table. iptables -L lists only what iptables-nft created, and a host running firewalld may not have the nft command at all. (That missing binary also leaves fail2ban unable to ban anyone .)

Takeaway

A counter on an ACCEPT rule only proves that the packet reached that rule. When an accepted packet still doesn’t arrive, list every base chain on the hook and find the later one that drops it.

If your host runs firewalld alongside anything that writes its own iptables rules, run sudo nft list chains | grep hook and count the chains on input and forward.

unattended-upgrades on Debian that installs nothing: Ubuntu-style Allowed-Origins

Sat, 26 Sep 2026 19:35:00 +0200

A Debian 13 server had automatic security updates. Ansible had installed unattended-upgrades, enabled it in 20auto-upgrades, and apt-daily-upgrade.timer had run that morning. Then a routine check:

$ apt list --upgradable 2>/dev/null | grep -c security
61

Nothing had failed, and nothing had warned. unattended-upgrades had never found anything it was allowed to install.

If your 50unattended-upgrades on Debian has Allowed-Origins with ${distro_codename}-security, yours does the same.

The config

This is what the Ansible role wrote to /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
 "${distro_id}:${distro_codename}";
 "${distro_id}:${distro_codename}-security";
 "${distro_id}ESMApps:${distro_codename}-apps-security";
 "${distro_id}ESM:${distro_codename}-infra-security";
};

The ESM lines give it away: Extended Security Maintenance is Ubuntu Pro. This is Ubuntu’s stock file, copied into a Debian role by hand or by an LLM. On Debian it expands to “Debian:trixie” and “Debian:trixie-security”, which looks right and matches nothing.

Why it matches nothing

Allowed-Origins is the legacy syntax. unattended-upgrades (2.12 on Debian 13) converts each entry to the newer pattern format in get_allowed_origins_legacy():

allowed_origins.append("o=%s,a=%s" % (substitute(distro_id),
 substitute(distro_codename)))

The part after the colon becomes a=, and the matcher compares a with the archive, which is the Suite: field of the Release file:

elif what in ("a", "suite", "archive"):
 match = fnmatch.fnmatch(origin.archive, value)

On Ubuntu, suite and codename are the same string:

# security.ubuntu.com/ubuntu/dists/noble-security/InRelease
Suite: noble-security
Codename: noble

On Debian, they are not:

# security.debian.org/debian-security/dists/trixie-security/InRelease
Origin: Debian
Label: Debian-Security
Suite: stable-security
Codename: trixie-security

apt-cache policy shows both, as a= and n=:

release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64

So a=trixie-security matches nothing, and neither does a=trixie, because the main archive’s suite is stable. Run through unattended-upgrades’ own matcher against a trixie host’s package lists, the legacy entries come up empty, and Debian’s patterns each hit their archive:

'o=Debian,a=trixie' -> []
'o=Debian,a=trixie-security' -> []
'origin=Debian,codename=${distro_codename},label=Debian' -> ['stable/trixie']
'origin=Debian,codename=${distro_codename}-security,label=Debian-Security' -> ['stable-security/trixie-security']

With no allowed origin, there is nothing to upgrade, and that’s not an error. Each run logs No packages found that can be upgraded unattended and no pending auto-removals and exits cleanly.

The fix

Use Origins-Pattern with codename=, as Debian’s own file does:

Unattended-Upgrade::Origins-Pattern {
 "origin=Debian,codename=${distro_codename},label=Debian";
 "origin=Debian,codename=${distro_codename}-updates,label=Debian";
 "origin=Debian,codename=${distro_codename},label=Debian-Security";
 "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
};

Debian’s stock file has the -updates line commented out. Keep it or drop it as you like.

a=stable-security would match too, but don’t use it. On release day, stable moves to the next Debian release, and a suite-based pattern would follow it. A codename stays with the release you installed.

Then check what unattended-upgrades actually picks:

sudo unattended-upgrade --dry-run -d

The Allowed origins are: line shows the patterns after conversion. Below it, the security packages from apt list --upgradable should show up as upgrade candidates. If the run ends with “No packages found” while apt list --upgradable lists stable-security packages, the origins are still wrong.

With Automatic-Reboot "false", a new kernel or libc still waits for you once updates flow again, so check for /var/run/reboot-required.

Takeaway

On Debian, Allowed-Origins "${distro_id}:${distro_codename}-security" is a no-op that looks like a security policy, because it matches the suite (stable-security), not the codename. Use Origins-Pattern with codename=, and confirm it with --dry-run -d.

Run grep -rn Allowed-Origins /etc/apt/apt.conf.d/ on your Debian hosts, and run the same grep over your configuration management repos, where the copy is most likely to live.