Rsync on Lessons learned

rsync code 23: atomic archives can still form an incomplete release

Thu, 22 Oct 2026 00:00:00 +0000

A dataset transfer ended with:

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1327) [sender=3.2.5]

The packing job published each finished archive by rename, so final filenames never exposed an archive still being written. The transfer nevertheless began before the collection was complete. Repeating it after packing finished, then checking destination hashes, recovered the release.

This happened in the Claude Code over SSH and OpenPBS workflow : a PBS job produced the archives, while separate remote commands and a human-operated transfer consumed them.

The filename was safe before the collection was ready

The packer used this pattern, with paths generalized:

tar -czf groups_00.tar.gz.part groups/part_00 &&
 mv groups_00.tar.gz.part groups_00.tar.gz

A reader looking for *.tar.gz would only see that archive after its successful creation and rename on the same filesystem. But the producer repeated the operation for several archives, then wrote sidecar files. Nothing prevented a consumer from enumerating the directory between those steps.

Transfer instructions went out while the packer was running. The user’s pasted output ended with to-chk=0/16—the intended release contained seventeen archives plus a checksum manifest and a guide. A later source listing showed the complete set.

The count, transferred size, and packing timeline supported the explanation that the last archive and sidecars didn’t exist when the transfer enumerated its inputs. The earlier detailed rsync errors were not included, so this remained a supported diagnosis rather than a direct readback of each missing-path error.

Code 23 alone cannot identify that cause. The rsync manual defines it as a partial transfer due to error; inspect the preceding errors before retrying mindlessly.

Repeat the transfer after checking the producer

The recovery checked that packing had finished, repeated the transfer, and verified the archives at the destination. With server names and filenames generalized, run from the producer’s archive directory:

rsync -rtv --partial --progress \
 SHA256SUMS guide.txt groups_*.tar.gz metadata.tar.gz \
 researcher@192.168.0.3:incoming/release-01/

Then, on the destination:

cd ~/incoming/release-01 && sha256sum -c SHA256SUMS

The user supplied seventeen OK results. Those checks verified the listed archive bytes, including the archives transferred earlier. They did not verify recipient permissions or the guide’s contents. A separate archive-content check also lacked bc, so its aggregate group-count calculation did not succeed; that check must not be presented as another passing validation.

--partial retains partial transfers. It does not by itself promise efficient offset-based resumption under every transfer configuration.

Add a release-complete barrier

We did not implement a stronger publication protocol during this recovery. For a future run, use a fresh release directory that consumers cannot mistake for an earlier finished release. Publish a completion marker only after validating the expected archive set and sidecars. Consumers should require that marker before enumeration and check hashes after transfer.

A marker needs an immutable release behind it. Reusing a directory with an old marker can announce incomplete new contents as ready; writing the marker early merely moves the race.

Before sending a transfer command, check the producer’s completion evidence and the expected file set. Per-file renames protect readers of each file, but consumers still need a defined point at which the collection is ready.

APFS sparse bundle on SMB: rsync "Operation not supported (45)" and F_FULLFSYNC

Tue, 29 Sep 2026 12:55:00 +0200

A backup job that rsyncs into encrypted disk images on a NAS started printing this for every file (paths elided):

rsync: [generator] failed to set times on …
rsync: [receiver] mkstemp …
rsync: [generator] delete_file: unlink … failed: Operation not supported (45)

Then hdiutil attach for the next image hung for over ten minutes, and the two images whose rsyncs had finished stayed mounted.

If you keep APFS sparse bundles on an SMB share and see error 45 inside them, the files aren’t the problem. The share can’t do F_FULLFSYNC, and APFS can’t commit a single transaction without it.

The setup

macOS 27.0.1 dropped AFP, so an old NAS that speaks only SMB 2.1 got mounted over SMB instead. Each backup source has its own AES-256 sparse bundle on the share, created like this:

hdiutil create -encryption AES-256 -type SPARSEBUNDLE -size 50g \
 -fs APFS -volname app -stdinpass /Volumes/backup/staging/app.sparsebundle

The job attaches a bundle, rsyncs a server’s files into it, and detaches it.

The kernel log

rsync names the files, so the obvious suspects are permissions or the files themselves. The kernel log points at the image instead (timestamps shortened, process column removed):

/usr/bin/log show --last 90m --style compact \
 --predicate 'process == "kernel" AND eventMessage CONTAINS "apfs"'

12:43:53.531 disk7s1 mount-complete volume app, requested by: mount_apfs (pid 82535); parent: mount (pid 82534)
12:43:53.532 disk7s1 Host is remote
...
12:47:45.393 (apfs) tx_flush_thread:1678: disk7 tx_flush(8311) failed, 45, # 297
12:47:45.931 (apfs) tx_flush_thread:1678: disk7 tx_flush(8311) failed, 45, # 298

APFS noticed the image lives on a remote host. From 24 seconds after mounting, it failed to flush a transaction about twice a second, with error 45 (ENOTSUP). The second mounted image did the same. Nothing written inside them was being committed.

The probe

Error 45 from a flush suggested the sync call, not the write. This script tries plain fsync and the two stronger syncs macOS offers. Run it from a directory on the share:

import fcntl, os
p = "_synctest"
fd = os.open(p, os.O_RDWR | os.O_CREAT, 0o600); os.write(fd, b"x" * 4096)
for name, op in (("fsync", None), ("F_FULLFSYNC", 51), ("F_BARRIERFSYNC", 85)):
 try:
 os.fsync(fd) if op is None else fcntl.fcntl(fd, op)
 print(name, "OK")
 except OSError as e:
 print(name, "->", e)
os.close(fd); os.unlink(p)

fsync OK
F_FULLFSYNC -> [Errno 45] Operation not supported
F_BARRIERFSYNC -> [Errno 45] Operation not supported

APFS asks the disk image to flush; the disk image passes that on as F_FULLFSYNC on its band files on the share, and the SMB client says no.

The NAS also had a share called TimeMachineBackup, and Time Machine over SMB needs full sync, so it seemed worth a try. Same result. Its Time Machine backups dated from the AFP days.

What works

Keep the image off the share. The bundles moved to a USB drive, and every healthy one attaches, writes, deletes, and detaches fine there. That’s where the job runs now.

If the image has to stay on the share, use HFS+ inside it. A test bundle on the same share, created with -fs HFS+J instead of -fs APFS, passed everything the job does: rsync with --delete-during, rename, setting times, deletes, detach and reattach.

If the server runs Samba and you control it, fruit:time machine = yes on the share makes vfs_fruit advertise the FULLSYNC volume capability to macOS clients. That should let APFS images commit, but it wasn’t tested here.

Check the images you already have

The failed commits also left damage. When the bundles were copied to the USB drive, one of them wouldn’t mount:

hdiutil: attach failed - no mountable file systems

A read-only check of its volume (fsck_apfs -n) said why:

warning: (oid 0x74e6) om: btn: invalid o_cksum (0xdfbd1ccee26c9006)
** The volume /dev/rdisk8s1 with UUID … was found to be corrupt and cannot be repaired.

Its band files had last been modified during the failed SMB run, and the original on the NAS was identical, so the copy wasn’t to blame. Since it was only a staging mirror, it got recreated empty, and the next run filled it again. After any such episode, attach each image and run diskutil verifyVolume on it.

Why restic didn’t mind

Restic repositories on the same share were unaffected. Restic is written in Go, and Go’s Fsync on macOS tries F_FULLFSYNC and falls back to plain fsync on ENOTSUP. The comment in the Go source names this exact case: “There are scenarios such as SMB mounts where fcntl will fail with ENOTSUP.” APFS inside a disk image has no such fallback: when the full flush fails, it stops committing.

Takeaway

An APFS disk image on an SMB share works only if the share supports F_FULLFSYNC. Without it, every write inside the image fails with error 45, and the failed commits can corrupt the image.

If you’re moving sparse bundles from AFP to SMB, run the probe above on the new share before you mount anything on it.