Ansible on Lessons learned

The rollback that saved the host, and killed it 24 hours later

Sat, 17 Oct 2026 21:51:00 +0200

The rollback timer fired, the remote host got its address back, and the journal said all was well:

dhclient[1796120]: bound to 192.168.0.2 -- renewal in 34318 seconds.
systemd[1]: lxc-uplink-rollback.service: Deactivated successfully.

Exactly 24 hours later, to the second, the host dropped off the network and took SSH, the port forwards, and the VPN with it.

When a transient systemd-run unit finishes, systemd kills everything its command left running. If your rollback, or any other command run that way, starts a daemon, the daemon dies with the unit. Here it was dhclient, and the damage only showed when the DHCP lease expired.

The setup

Debian 13, systemd 257, ifupdown with ISC dhclient 4.4.3-P1
allow-hotplug enp0s25 / iface enp0s25 inet dhcp, with a 24-hour lease from the router
The only way in is SSH through a port forward to the host’s address; the console is days away

The dead man’s switch worked

Before a risky network change, an Ansible play arms a timer that undoes it, applies the change, and disarms the timer only if a fresh SSH login succeeds:

- name: Arm rollback timer
 ansible.builtin.command: >-
 systemd-run --collect --unit=lxc-uplink-rollback --on-active=5min
 /bin/sh -c 'ifdown -a --exclude=lo; ip link del br0;
 cp /etc/network/interfaces.rollback /etc/network/interfaces;
 ifup -a; ifup -a --allow=hotplug'

# ... apply the change ...

- name: Drop the cached SSH connection
 ansible.builtin.meta: reset_connection

- name: Prove a fresh SSH login still works
 ansible.builtin.wait_for_connection:
 delay: 10
 timeout: 420

- name: Disarm rollback timer
 ansible.builtin.command: systemctl stop lxc-uplink-rollback.timer

Without meta: reset_connection, Ansible reuses its open SSH connection and a broken change looks fine.

The change moved the NIC into a bridge, br0. The bridge got the old address back, but forwarded SSH from the internet stopped reaching the host. (I never found out why, and it isn’t the point here.) Five minutes later the timer restored the old config, the host came back, and I moved on.

What went wrong

Look at the rollback unit again. Its main process is /bin/sh. ifup starts dhclient, which forks into the background, and then sh exits. The unit is now done, and a finished service gets its cgroup cleaned up: by default (KillMode=control-group), systemd kills every process still in it. That includes the dhclient that had just been “bound”.

A killed dhclient doesn’t release the lease, so the address stays on the interface with a lifetime of 86400 seconds. Nothing renews it. When the lifetime runs out, the kernel removes the address, and avahi is the first to notice:

Oct 17 21:08:42 avahi-daemon[936]: Withdrawing address record for 192.168.0.2 on enp0s25.

The clue was in the journal on day one. When the rollback started, its dhclient logged:

dhclient[1796009]: Removed stale PID file

That stale PID file belonged to the dhclient on br0. Ansible had started it five minutes earlier from another systemd-run unit, and systemd had killed it the same way, but the rollback replaced it before anyone noticed.

At boot, ifup@enp0s25.service starts the same dhclient. That unit stays active (exited), so its cgroup is never cleaned up, which is how the boot-time dhclient had kept renewing for about six weeks.

You can see the behavior in two lines:

sudo systemd-run --unit=demo /bin/sh -c 'sleep 600 &'
sleep 1; pgrep -a sleep # nothing: systemd killed it when sh exited

The fix

To get back in, I went through a VM that had its own macvlan leg on the same NIC and ran sudo dhclient enp0s25. With ssh -J, -i applies only to the last hop, so give the jump host its key through ProxyCommand:

ssh -i ~/.ssh/key -o IdentitiesOnly=yes \
 -o ProxyCommand="ssh -i ~/.ssh/key -o IdentitiesOnly=yes -W %h:%p user@jump.example.com" \
 user@target.example.com

The real fix was to stop depending on a long-running DHCP client. The router reserves the address for the host anyway, so the host now has a static stanza. Ansible applies it without restarting networking, because that would cut off the run and the only way in:

ip address replace 192.168.0.2/24 dev enp0s25 valid_lft forever preferred_lft forever
ip route replace default via 192.168.0.1 dev enp0s25
pkill --signal KILL -f '(^|/)dhclient .*enp0s25$' || true

SIGKILL is deliberate: a dhclient that stops cleanly removes the address it holds, the one this very run is connected through.

If the host has to stay on DHCP, keep daemons out of transient units. Start the daemon through the unit that owns it at boot (here ifup@enp0s25.service), or let the transient unit leave its children alone with systemd-run -p KillMode=process. I didn’t retry the bridge, so I haven’t tested either variant in this rollback.

To check which unit a daemon really belongs to:

cat /proc/$(pgrep -o dhclient)/cgroup

Takeaway

Deactivated successfully only says the unit finished; whatever it started may have died with it. A killed DHCP client fails a whole lease later, long after you have stopped looking.

Search your playbooks and scripts for systemd-run calls that run ifup, dhclient or anything else that forks into the background. Then check which cgroup that process ends up in.

"Ansible requires blocking IO on stdin/stdout/stderr" when an AI agent runs Ansible

Tue, 13 Oct 2026 22:34:00 +0200

The first time my coding agent ran Ansible, every command failed before Ansible had read a single file:

ERROR: Ansible requires blocking IO on stdin/stdout/stderr. Non-blocking file handles detected: <stdout>, <stderr>

--syntax-check too, and even ansible --version. If you see this from an agent, a CI job, or a wrapper script, Ansible isn’t broken. The process that started it handed it non-blocking file handles, and a few shell redirections replace them.

The setup

ansible-core 2.20 (Ansible 13.3.0 from Homebrew) on macOS
A coding agent (Claude Code in VS Code) that runs commands in its own shell and captures their output

The check has been in every Ansible CLI entry point since ansible-core 2.14 (2022).

What went wrong

O_NONBLOCK is a flag on the open file, not on the process, so every child that inherits the file descriptor inherits the flag too. The agent’s harness connects the shell’s stdout and stderr to non-blocking pipes, and the shell passes them on unchanged to ansible-playbook.

At startup, Ansible calls os.get_blocking() on each of the three handles and exits if any of them is non-blocking. The list at the end of the message names the handles that failed. That makes it a good diagnostic: the first failure in my session listed only <stderr>, because the command was ansible --version | head -3. The pipe to head had already given stdout a fresh, blocking handle, while stderr still pointed at the agent’s pipe.

Run this in the shell you suspect to see which handles are affected:

python3 -c 'import os; print([os.get_blocking(fd) for fd in (0, 1, 2)])'

False in any position means Ansible will refuse to start there.

Fix

Give Ansible three handles the shell creates itself:

ansible-playbook site.yml </dev/null 2>&1 | cat

| cat gives stdout a new pipe, which is blocking.
2>&1 sends stderr into that same pipe. | cat alone fixes only stdout.
</dev/null replaces stdin. My agent’s stdin was never flagged, but Ansible checks it too, and an agent can’t answer a prompt anyway.

The pipe has a side effect: the command’s exit status is now cat’s, which is 0. A failed playbook looks like a success unless you turn on pipefail (bash and zsh both have it):

set -o pipefail
ansible-playbook site.yml </dev/null 2>&1 | cat

To check the fix without touching your real shell, you can make all three handles non-blocking for a single command and try it:

NB='import os,sys; [os.set_blocking(fd, False) for fd in (0,1,2)]; os.execvp(sys.argv[1], sys.argv[1:])'
(python3 -c "$NB" ansible --version) 2>&1 </dev/zero | cat
# ERROR: ... Non-blocking file handles detected: <stdin>, <stdout>, <stderr>
(python3 -c "$NB" sh -c 'ansible --version </dev/null 2>&1 | cat') 2>&1 </dev/zero | cat
# ansible [core 2.20.1] ...

The outer | cat and </dev/zero keep the flag on throwaway handles, so your terminal stays blocking.

Why Ansible insists

The pull request that added the check explains it. On a non-blocking stdout, a large write can fail partway with [Errno 11] write could not complete without blocking, and Ansible’s output gets cut off, or the run dies with an “Unexpected Exception”. The check turns those intermittent failures into one clear error at startup. The PR names another way to end up there: ssh switches its handles to non-blocking while it runs and doesn’t always switch them back when it exits.

So the same error can turn up wherever a parent process hands down non-blocking pipes, CI runners included, or in a terminal after an ssh that didn’t clean up.

Make the agent remember it

The agent worked this out within two minutes, and later that night it wrote the fix into the project’s skill for provisioning VMs:

Run Ansible from Claude Code with `</dev/null` and pipe its output through `cat`.

The next morning, a session about a different service ran ansible-playbook --syntax-check without the redirections and hit the same error. It never loaded the VM skill, so it never saw the line. A rule about how to run a tool belongs in the file the agent reads in every session, not in the instructions for one task.

Takeaway

“Ansible requires blocking IO” means the process that started Ansible handed it non-blocking file handles. </dev/null 2>&1 | cat gives it blocking ones, and set -o pipefail keeps a failed run from looking like a success.

If your agent runs Ansible, put that line in the instructions file it reads every session (CLAUDE.md or AGENTS.md).

unattended-upgrades on Debian that installs nothing: Ubuntu-style Allowed-Origins

Sat, 26 Sep 2026 19:35:00 +0200

A Debian 13 server had automatic security updates. Ansible had installed unattended-upgrades, enabled it in 20auto-upgrades, and apt-daily-upgrade.timer had run that morning. Then a routine check:

$ apt list --upgradable 2>/dev/null | grep -c security
61

Nothing had failed, and nothing had warned. unattended-upgrades had never found anything it was allowed to install.

If your 50unattended-upgrades on Debian has Allowed-Origins with ${distro_codename}-security, yours does the same.

The config

This is what the Ansible role wrote to /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
 "${distro_id}:${distro_codename}";
 "${distro_id}:${distro_codename}-security";
 "${distro_id}ESMApps:${distro_codename}-apps-security";
 "${distro_id}ESM:${distro_codename}-infra-security";
};

The ESM lines give it away: Extended Security Maintenance is Ubuntu Pro. This is Ubuntu’s stock file, copied into a Debian role by hand or by an LLM. On Debian it expands to “Debian:trixie” and “Debian:trixie-security”, which looks right and matches nothing.

Why it matches nothing

Allowed-Origins is the legacy syntax. unattended-upgrades (2.12 on Debian 13) converts each entry to the newer pattern format in get_allowed_origins_legacy():

allowed_origins.append("o=%s,a=%s" % (substitute(distro_id),
 substitute(distro_codename)))

The part after the colon becomes a=, and the matcher compares a with the archive, which is the Suite: field of the Release file:

elif what in ("a", "suite", "archive"):
 match = fnmatch.fnmatch(origin.archive, value)

On Ubuntu, suite and codename are the same string:

# security.ubuntu.com/ubuntu/dists/noble-security/InRelease
Suite: noble-security
Codename: noble

On Debian, they are not:

# security.debian.org/debian-security/dists/trixie-security/InRelease
Origin: Debian
Label: Debian-Security
Suite: stable-security
Codename: trixie-security

apt-cache policy shows both, as a= and n=:

release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64

So a=trixie-security matches nothing, and neither does a=trixie, because the main archive’s suite is stable. Run through unattended-upgrades’ own matcher against a trixie host’s package lists, the legacy entries come up empty, and Debian’s patterns each hit their archive:

'o=Debian,a=trixie' -> []
'o=Debian,a=trixie-security' -> []
'origin=Debian,codename=${distro_codename},label=Debian' -> ['stable/trixie']
'origin=Debian,codename=${distro_codename}-security,label=Debian-Security' -> ['stable-security/trixie-security']

With no allowed origin, there is nothing to upgrade, and that’s not an error. Each run logs No packages found that can be upgraded unattended and no pending auto-removals and exits cleanly.

The fix

Use Origins-Pattern with codename=, as Debian’s own file does:

Unattended-Upgrade::Origins-Pattern {
 "origin=Debian,codename=${distro_codename},label=Debian";
 "origin=Debian,codename=${distro_codename}-updates,label=Debian";
 "origin=Debian,codename=${distro_codename},label=Debian-Security";
 "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
};

Debian’s stock file has the -updates line commented out. Keep it or drop it as you like.

a=stable-security would match too, but don’t use it. On release day, stable moves to the next Debian release, and a suite-based pattern would follow it. A codename stays with the release you installed.

Then check what unattended-upgrades actually picks:

sudo unattended-upgrade --dry-run -d

The Allowed origins are: line shows the patterns after conversion. Below it, the security packages from apt list --upgradable should show up as upgrade candidates. If the run ends with “No packages found” while apt list --upgradable lists stable-security packages, the origins are still wrong.

With Automatic-Reboot "false", a new kernel or libc still waits for you once updates flow again, so check for /var/run/reboot-required.

Takeaway

On Debian, Allowed-Origins "${distro_id}:${distro_codename}-security" is a no-op that looks like a security policy, because it matches the suite (stable-security), not the codename. Use Origins-Pattern with codename=, and confirm it with --dry-run -d.

Run grep -rn Allowed-Origins /etc/apt/apt.conf.d/ on your Debian hosts, and run the same grep over your configuration management repos, where the copy is most likely to live.