The rollback that saved the host, and killed it 24 hours later

·4 mins

The rollback timer fired, the remote host got its address back, and the journal said all was well:

dhclient[1796120]: bound to 192.168.0.2 -- renewal in 34318 seconds.
systemd[1]: lxc-uplink-rollback.service: Deactivated successfully.

Exactly 24 hours later, to the second, the host dropped off the network and took SSH, the port forwards, and the VPN with it.

When a transient systemd-run unit finishes, systemd kills everything its command left running. If your rollback, or any other command run that way, starts a daemon, the daemon dies with the unit. Here it was dhclient, and the damage only showed when the DHCP lease expired.

The setup

  • Debian 13, systemd 257, ifupdown with ISC dhclient 4.4.3-P1
  • allow-hotplug enp0s25 / iface enp0s25 inet dhcp, with a 24-hour lease from the router
  • The only way in is SSH through a port forward to the host’s address; the console is days away

The dead man’s switch worked

Before a risky network change, an Ansible play arms a timer that undoes it, applies the change, and disarms the timer only if a fresh SSH login succeeds:

- name: Arm rollback timer
  ansible.builtin.command: >-
    systemd-run --collect --unit=lxc-uplink-rollback --on-active=5min
    /bin/sh -c 'ifdown -a --exclude=lo; ip link del br0;
    cp /etc/network/interfaces.rollback /etc/network/interfaces;
    ifup -a; ifup -a --allow=hotplug'

# ... apply the change ...

- name: Drop the cached SSH connection
  ansible.builtin.meta: reset_connection

- name: Prove a fresh SSH login still works
  ansible.builtin.wait_for_connection:
    delay: 10
    timeout: 420

- name: Disarm rollback timer
  ansible.builtin.command: systemctl stop lxc-uplink-rollback.timer

Without meta: reset_connection, Ansible reuses its open SSH connection and a broken change looks fine.

The change moved the NIC into a bridge, br0. The bridge got the old address back, but forwarded SSH from the internet stopped reaching the host. (I never found out why, and it isn’t the point here.) Five minutes later the timer restored the old config, the host came back, and I moved on.

What went wrong

Look at the rollback unit again. Its main process is /bin/sh. ifup starts dhclient, which forks into the background, and then sh exits. The unit is now done, and a finished service gets its cgroup cleaned up: by default (KillMode=control-group), systemd kills every process still in it. That includes the dhclient that had just been “bound”.

A killed dhclient doesn’t release the lease, so the address stays on the interface with a lifetime of 86400 seconds. Nothing renews it. When the lifetime runs out, the kernel removes the address, and avahi is the first to notice:

Oct 17 21:08:42 avahi-daemon[936]: Withdrawing address record for 192.168.0.2 on enp0s25.

The clue was in the journal on day one. When the rollback started, its dhclient logged:

dhclient[1796009]: Removed stale PID file

That stale PID file belonged to the dhclient on br0. Ansible had started it five minutes earlier from another systemd-run unit, and systemd had killed it the same way, but the rollback replaced it before anyone noticed.

At boot, ifup@enp0s25.service starts the same dhclient. That unit stays active (exited), so its cgroup is never cleaned up, which is how the boot-time dhclient had kept renewing for about six weeks.

You can see the behavior in two lines:

sudo systemd-run --unit=demo /bin/sh -c 'sleep 600 &'
sleep 1; pgrep -a sleep    # nothing: systemd killed it when sh exited

The fix

To get back in, I went through a VM that had its own macvlan leg on the same NIC and ran sudo dhclient enp0s25. With ssh -J, -i applies only to the last hop, so give the jump host its key through ProxyCommand:

ssh -i ~/.ssh/key -o IdentitiesOnly=yes \
    -o ProxyCommand="ssh -i ~/.ssh/key -o IdentitiesOnly=yes -W %h:%p user@jump.example.com" \
    user@target.example.com

The real fix was to stop depending on a long-running DHCP client. The router reserves the address for the host anyway, so the host now has a static stanza. Ansible applies it without restarting networking, because that would cut off the run and the only way in:

ip address replace 192.168.0.2/24 dev enp0s25 valid_lft forever preferred_lft forever
ip route replace default via 192.168.0.1 dev enp0s25
pkill --signal KILL -f '(^|/)dhclient .*enp0s25$' || true

SIGKILL is deliberate: a dhclient that stops cleanly removes the address it holds, the one this very run is connected through.

If the host has to stay on DHCP, keep daemons out of transient units. Start the daemon through the unit that owns it at boot (here ifup@enp0s25.service), or let the transient unit leave its children alone with systemd-run -p KillMode=process. I didn’t retry the bridge, so I haven’t tested either variant in this rollback.

To check which unit a daemon really belongs to:

cat /proc/$(pgrep -o dhclient)/cgroup

Takeaway

Deactivated successfully only says the unit finished; whatever it started may have died with it. A killed DHCP client fails a whole lease later, long after you have stopped looking.

Search your playbooks and scripts for systemd-run calls that run ifup, dhclient or anything else that forks into the background. Then check which cgroup that process ends up in.