Rootless Podman backup without app.ini, and a restore that fails with 'Cannot change ownership to uid 100999'
Restoring a Gitea backup to a new host failed on every directory:
tar: data/gitea: Cannot change ownership to uid 100999, gid 100999: Invalid argument
tar: data/gitea/queues: Cannot change ownership to uid 100999, gid 100999: Invalid argument
That one is easy to fix. The worse problem made no noise at all, and I found it just before: the nightly backup didn’t contain app.ini. If you back up a rootless Podman volume with a plain tar as the host user, yours may be missing files too.
The setup #
Gitea 1.27 and PostgreSQL ran in rootless Podman on a VPS, with Gitea’s data in a bind mount under ~/gitea/data. The nightly backup ran as the user who owns the containers:
podman exec gitea_db_1 pg_dump -U gitea gitea | bzip2 -z9 > ~/backup/gitea-db-$(date +%y%m%d).sql.bz2
tar c -C ~/gitea/data gitea | bzip2 -z9 > ~/backup/gitea-data-$(date +%y%m%d).tar.bz2
The target was Gitea in Docker, inside an unprivileged LXC container on Debian.
The archive looked fine. It unpacked, and it contained all the repositories.
Compare the archive with the live tree #
Before restoring, I compared the archive’s file list with the live data. List the live side from inside the user namespace, so that nothing is hidden from you:
ssh vps.example.com 'cd ~/gitea/data && podman unshare find gitea -type f' | sort > live.txt
tar tjf gitea-data.tar.bz2 | grep -v '/$' | sort > backup.txt
comm -23 live.txt backup.txt > missing.txt
wc -l live.txt backup.txt missing.txt
12359 live.txt
12342 backup.txt
17 missing.txt
Among the 17 were the files Gitea can least do without:
gitea/gitea/conf/app.ini
gitea/gitea/jwt/private.pem
gitea/git/.ssh/authorized_keys
gitea/gitea/avatars/... (3 files)
gitea/gitea/indexers/... (4 files)
gitea/gitea/sessions/... (2 files)
app.ini holds Gitea’s SECRET_KEY, its internal tokens, and the mail settings. private.pem signs its OAuth2 tokens. A restore from this backup would have started with fresh secrets: anything encrypted with the old SECRET_KEY, such as two-factor secrets, would no longer decrypt, and the mailer would be unconfigured.
Why tar skipped them #
Rootless Podman maps the container’s UIDs onto the subordinate range in /etc/subuid. With the usual entry:
deploy:100000:65536
container root is you, and container UID 1000 (Gitea’s git user) is host UID 100000 + 999 = 100999. Files that Gitea creates with mode 0600 belong to 100999, and you can’t read them from the host. The repository objects were 0644 in 0755 directories, so tar read those, and the archive looked complete.
tar does complain, on stderr, and exits with status 2. In tar c ... | bzip2 > file, though, the pipeline’s exit status is bzip2’s, which is 0. Without set -o pipefail and someone reading the job’s stderr, nothing tells you.
Fix the backup #
Run tar inside the user namespace, where you are root over every mapped UID:
set -o pipefail
podman unshare tar c -C ~/gitea/data gitea | bzip2 -z9 > ~/backup/gitea-data-$(date +%y%m%d).tar.bz2
The archive then records the IDs as the container sees them (1000), not the host’s 100999.
To rescue the files without touching the live host, fetch only the missing ones the same way, then merge them into the old archive with bsdtar’s @archive syntax:
ssh vps.example.com 'cd ~/gitea/data && podman unshare tar -c -T -' < missing.txt > missing.tar
bsdtar -czf gitea-complete.tar.gz @gitea-data.tar.bz2 @missing.tar
Fix the restore #
GNU tar run as root restores the owners recorded in the archive (--same-owner is its default for root). Inside an unprivileged LXC container, root is itself a mapped user, and the container’s UIDs usually cover only 0 to 65535. UID 100999 doesn’t exist there, so chown fails with EINVAL, which tar reports as Invalid argument.
Don’t restore owners from an archive made somewhere else. Unpack with --no-same-owner and set the owners the new Gitea expects:
tar -C /var/tmp/gitea-restore --no-same-owner -xzf gitea-complete.tar.gz
cp -a /var/tmp/gitea-restore/gitea/. /srv/gitea/data/
chown -R 1000:1000 /srv/gitea/data/git /srv/gitea/data/gitea
chown -R root:root /srv/gitea/data/ssh
This also covers the merged archive, which carries 100999 for some files and 1000 for others.
After the restore, check against the live instance: the repository count, git fsck on every repository, and the SSH host key fingerprint on the Git SSH port, so existing clones still trust the new host. All three matched.
Takeaway #
Under rootless Podman, a host-side tar leaves out the files the container keeps private, which are often its secrets, and the pipeline still exits 0. Comparing the archive’s listing with podman unshare find takes a minute and shows the gap.
Run the podman unshare find and comm check above against your own latest backup and see what comes back.