rsync code 23: atomic archives can still form an incomplete release

·3 mins

A dataset transfer ended with:

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1327) [sender=3.2.5]

The packing job published each finished archive by rename, so final filenames never exposed an archive still being written. The transfer nevertheless began before the collection was complete. Repeating it after packing finished, then checking destination hashes, recovered the release.

This happened in the Claude Code over SSH and OpenPBS workflow : a PBS job produced the archives, while separate remote commands and a human-operated transfer consumed them.

The filename was safe before the collection was ready

The packer used this pattern, with paths generalized:

tar -czf groups_00.tar.gz.part groups/part_00 &&
  mv groups_00.tar.gz.part groups_00.tar.gz

A reader looking for *.tar.gz would only see that archive after its successful creation and rename on the same filesystem. But the producer repeated the operation for several archives, then wrote sidecar files. Nothing prevented a consumer from enumerating the directory between those steps.

Transfer instructions went out while the packer was running. The user’s pasted output ended with to-chk=0/16—the intended release contained seventeen archives plus a checksum manifest and a guide. A later source listing showed the complete set.

The count, transferred size, and packing timeline supported the explanation that the last archive and sidecars didn’t exist when the transfer enumerated its inputs. The earlier detailed rsync errors were not included, so this remained a supported diagnosis rather than a direct readback of each missing-path error.

Code 23 alone cannot identify that cause. The rsync manual defines it as a partial transfer due to error; inspect the preceding errors before retrying mindlessly.

Repeat the transfer after checking the producer

The recovery checked that packing had finished, repeated the transfer, and verified the archives at the destination. With server names and filenames generalized, run from the producer’s archive directory:

rsync -rtv --partial --progress \
  SHA256SUMS guide.txt groups_*.tar.gz metadata.tar.gz \
  researcher@192.168.0.3:incoming/release-01/

Then, on the destination:

cd ~/incoming/release-01 && sha256sum -c SHA256SUMS

The user supplied seventeen OK results. Those checks verified the listed archive bytes, including the archives transferred earlier. They did not verify recipient permissions or the guide’s contents. A separate archive-content check also lacked bc, so its aggregate group-count calculation did not succeed; that check must not be presented as another passing validation.

--partial retains partial transfers. It does not by itself promise efficient offset-based resumption under every transfer configuration.

Add a release-complete barrier

We did not implement a stronger publication protocol during this recovery. For a future run, use a fresh release directory that consumers cannot mistake for an earlier finished release. Publish a completion marker only after validating the expected archive set and sidecars. Consumers should require that marker before enumeration and check hashes after transfer.

A marker needs an immutable release behind it. Reusing a directory with an old marker can announce incomplete new contents as ready; writing the marker early merely moves the race.

Before sending a transfer command, check the producer’s completion evidence and the expected file set. Per-file renames protect readers of each file, but consumers still need a defined point at which the collection is ready.