Docker on Lessons learned

Unprivileged LXC containers share the host's per-user kernel limits: 'Too many open files' and 'disk quota exceeded'

Wed, 21 Oct 2026 13:13:00 +0200

One day, two containers on the same host failed with errors that seemed unrelated. systemd in one of them:

systemd[4573]: Failed to create control group inotify object: Too many open files
systemd[4573]: Failed to allocate manager object: Too many open files
systemd[1]: Failed to start user@1000.service - User Manager for UID 1000.

Docker in another, where every container sat at Exited (128):

OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown

No file descriptors had run out, and no disk was full. Both errors come from per-user kernel quotas counted on the host, and every unprivileged container there was drawing from the same two.

The setup

Debian 13, kernel 6.12, LXC 6.0.4
More unprivileged containers than the hardware comfortably runs, all with the same idmap: container root is host UID 1000000
systemd in every container, Docker nested in some

Too many open files: inotify instances

systemd opens inotify instances to watch cgroups and units, and inotify_init1() fails with EMFILE when the user’s instance limit is reached. strerror turns that into “Too many open files”. The limit is fs.inotify.max_user_instances, 128 by default.

Count the instances per user on the host:

sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' 2>/dev/null \
 | cut -d/ -f3 | xargs -I{} ps -o user= -p {} | sort | uniq -c | sort -rn
cat /proc/sys/fs/inotify/max_user_instances

Host root held 23 instances, and uids in the container range (1000000 and up) held 104 between them: 127 of 128. On the first sighting, my coding agent got that far, then dropped it as a side effect of having started too many containers at once. That was half right, since each container’s systemd adds about five instances. A day later it came back.

The lxc package already ships a higher limit, and it wasn’t in effect:

$ cat /etc/sysctl.d/30-lxc-inotify.conf | grep -v '^#'
fs.inotify.max_user_instances = 1024
$ cat /proc/sys/fs/inotify/max_user_instances
128

The file was dated three days after the last boot. /etc/sysctl.d is applied by systemd-sysctl at boot, and the package’s install hadn’t applied it, so the setting sat on disk doing nothing.

Disk quota exceeded: kernel keys

runc gives each container it starts a new session keyring. add_key() fails with EDQUOT when the user’s key quota is full, and strerror calls that “Disk quota exceeded”. The quota is kernel.keys.maxkeys, 200 keys for any uid other than root. The host shows usage per UID:

$ cat /proc/key-users
 0: 127 126/126 107/1000000 2283/25000000
 1000: 12 12/12 12/200 86/20000
1000000: 198 198/198 198/200 4189/20000

198/200 are keys used against the quota. UID 1000000 is root in every container, so all the containers shared those 200 keys, and Docker’s restart attempts had nothing left.

The fix

Raise the limits and apply them right away, not at the next boot. Incus recommends 2000 keys and 2 MB in its server settings ; 1024 inotify instances matches the lxc package.

sudo tee /etc/sysctl.d/60-lxc-limits.conf <<'EOF'
fs.inotify.max_user_instances = 1024
kernel.keys.maxkeys = 2000
kernel.keys.maxbytes = 2000000
EOF
sudo sysctl -p /etc/sysctl.d/60-lxc-limits.conf

With Ansible, ansible.posix.sysctl with sysctl_set: true writes the file and sets the live value in one task.

Then run systemctl reset-failed in the affected container, and docker start for the dead Docker containers. Right after, uid 1000000 held 226 keys, already over the old limit.

Why the two quotas are counted differently

The two quotas look similar, but separate idmaps would fix only one.

Keys are counted per host UID, against global sysctls. Containers that share an idmap share one quota. A separate idmap per container (security.idmap.isolated in LXD and Incus) would give each its own.

inotify instances are counted per user namespace and charged up the chain of parent namespaces, at each level against the uid that created the child namespace. A new namespace’s own limit is INT_MAX, so the limit that bites is the host’s, charged to whoever started the containers. LXC started as root charges host root, which is why the 23 instances of host root’s own daemons counted toward the same 128. Separate idmaps don’t change that.

Takeaway

On a container host, “Too many open files” and “disk quota exceeded” can mean that every container together hit a per-user kernel quota on the host. Check the host’s counters before you debug the container that complained.

Run the find count and cat /proc/key-users on your own container host and compare them with the limits before a container restart finds out for you.

A background agent Docker build: 'open Dockerfile: no such file or directory'

Sun, 30 Aug 2026 00:00:00 +0000

A background Docker build failed immediately with:

ERROR: failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory

The agent had supplied the project directory through the terminal tool’s workdir argument. The retry put cd inside the command itself. For an agent runner whose background path handling is uncertain, bind the directory in the command or make the Dockerfile and build context absolute.

The failed call already supplied workdir

In the historical runner, the background invocation had this shape, with project and image names generalized:

{
 "background": true,
 "workdir": "/srv/app",
 "command": "docker build --progress=plain -t local/app:dev -f Dockerfile ."
}

Docker could not open Dockerfile. The retry started with an explicit cd to the project before running the build. That corrected the launch form; whether the rest of that session succeeded is outside this lesson.

The error alone does not identify the actual launch directory. The record supports a runner-specific path-handling failure, not a rule that background processes generally discard their parent’s directory. Ordinary shell backgrounding with & inherits the shell’s working directory. An agent tool can launch its jobs through a different execution path.

A later check on the runner used to prepare this post found that background jobs preserved the preceding foreground directory, and an explicit background workdir was honored. The historical failure is therefore not a current universal behavior claim.

Bind both Docker paths deliberately

For an existing project at /srv/app, put the directory change and build in the same command:

cd /srv/app &&
 docker build --progress=plain -t local/app:dev -f Dockerfile .

The && prevents the build from running if the directory change fails. Alternatively, remove the directory dependency from both path arguments:

docker build --progress=plain -t local/app:dev \
 -f /srv/app/Dockerfile /srv/app

Making only -f absolute still leaves . as an accidental build context. Likewise, when using Compose, check its file selection and relative-path behavior rather than assuming a successful Docker build proves the next command has the right context.

Probe the actual execution mode

Before a multi-minute job, use a harmless foreground command to change to a known scratch directory:

cd "${TMPDIR:?Set TMPDIR to your scratch directory}" && pwd

Then submit pwd as a separate background tool call without workdir, and read that job’s captured output. Repeat with workdir set to the project root. Those two checks distinguish remembered foreground state from an explicit launch directory without building an image or starting services.

For a real build, print pwd and check test -f Dockerfile inside the same command that launches it. A foreground check in an earlier call cannot establish the background job’s directory.

Separate service startup from readiness checks

The same historical session hit another runner boundary. A foreground command combining docker compose up -d with a health-check loop was rejected with:

This foreground command appears to start a long-lived server/watch process.

The runner directed the agent to launch it with background=true, verify readiness, and run tests separately. That refusal came from the runner, even though Compose’s -d requests detached containers.

Keep service startup in its own call, following the runner’s execution rules. Check readiness in a separate bounded command, and fail explicitly if the deadline expires. Starting a service and proving it healthy are separate results.

Before your next background build, compare its reported directory with the intended build context using the harmless probe above.