Openpbs on Lessons learned

rsync code 23: atomic archives can still form an incomplete release

Thu, 22 Oct 2026 00:00:00 +0000

A dataset transfer ended with:

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1327) [sender=3.2.5]

The packing job published each finished archive by rename, so final filenames never exposed an archive still being written. The transfer nevertheless began before the collection was complete. Repeating it after packing finished, then checking destination hashes, recovered the release.

This happened in the Claude Code over SSH and OpenPBS workflow : a PBS job produced the archives, while separate remote commands and a human-operated transfer consumed them.

The filename was safe before the collection was ready

The packer used this pattern, with paths generalized:

tar -czf groups_00.tar.gz.part groups/part_00 &&
 mv groups_00.tar.gz.part groups_00.tar.gz

A reader looking for *.tar.gz would only see that archive after its successful creation and rename on the same filesystem. But the producer repeated the operation for several archives, then wrote sidecar files. Nothing prevented a consumer from enumerating the directory between those steps.

Transfer instructions went out while the packer was running. The user’s pasted output ended with to-chk=0/16—the intended release contained seventeen archives plus a checksum manifest and a guide. A later source listing showed the complete set.

The count, transferred size, and packing timeline supported the explanation that the last archive and sidecars didn’t exist when the transfer enumerated its inputs. The earlier detailed rsync errors were not included, so this remained a supported diagnosis rather than a direct readback of each missing-path error.

Code 23 alone cannot identify that cause. The rsync manual defines it as a partial transfer due to error; inspect the preceding errors before retrying mindlessly.

Repeat the transfer after checking the producer

The recovery checked that packing had finished, repeated the transfer, and verified the archives at the destination. With server names and filenames generalized, run from the producer’s archive directory:

rsync -rtv --partial --progress \
 SHA256SUMS guide.txt groups_*.tar.gz metadata.tar.gz \
 researcher@192.168.0.3:incoming/release-01/

Then, on the destination:

cd ~/incoming/release-01 && sha256sum -c SHA256SUMS

The user supplied seventeen OK results. Those checks verified the listed archive bytes, including the archives transferred earlier. They did not verify recipient permissions or the guide’s contents. A separate archive-content check also lacked bc, so its aggregate group-count calculation did not succeed; that check must not be presented as another passing validation.

--partial retains partial transfers. It does not by itself promise efficient offset-based resumption under every transfer configuration.

Add a release-complete barrier

We did not implement a stronger publication protocol during this recovery. For a future run, use a fresh release directory that consumers cannot mistake for an earlier finished release. Publish a completion marker only after validating the expected archive set and sidecars. Consumers should require that marker before enumeration and check hashes after transfer.

A marker needs an immutable release behind it. Reusing a directory with an old marker can announce incomplete new contents as ready; writing the marker early merely moves the race.

Before sending a transfer command, check the producer’s completion evidence and the expected file set. Per-file renames protect readers of each file, but consumers still need a defined point at which the collection is ready.

OpenPBS array resume: shard by stable task ID

Tue, 20 Oct 2026 00:00:00 +0000

A resumable OpenPBS array can assign the same task to different workers if each worker slices its own remaining-work list. The unsafe implementation was:

todo = [g for g in groups if not completed(g)]
todo = todo[i::n]

We caught this before the array ran and changed ownership to group_id % worker_count. The corrected workers later started at different times and restarted with fewer tasks remaining; the final audit accounted for every group.

The Claude Code, SSH, and OpenPBS setup explains where commands ran and how jobs were submitted. This failure concerns application-level partitioning, not PBS assigning one array index twice.

A pending list changes between starts

The workload was moving from a multi-GPU job to four independently scheduled one-GPU array elements. Each worker skipped groups with stored completion or rejection markers, then selected a shard.

Suppose the immutable task IDs are zero through five. Worker zero starts first and takes every second pending entry:

pending: 0 1 2 3 4 5
worker 0: 0 2 4

It finishes task zero before worker one starts. Worker one’s view is now:

pending: 1 2 3 4 5
worker 1: 2 4

Tasks two and four overlap, while one, three, and five have no owner in that pair of snapshots. Restarting workers creates the same problem. A completion marker can filter old work without stabilizing the positions of everything that remains.

Preserve ownership while filtering completion

The actual correction retained the immutable numeric group ID:

todo = [g for g in todo if g["group"] % n == i]

Equivalently, make the two decisions explicit:

owned = [g for g in groups if g["group"] % n == i]
todo = [g for g in owned if not completed(g)]

In this application, --shard I/N supplied i and n. The submitted script used #PBS -J 0-3 and passed --shard "$PBS_ARRAY_INDEX/4" to the project-specific driver. Each element requested one GPU. Smaller resource requests did not guarantee immediate starts; later queue snapshots still showed waiting elements.

Keep the shard count fixed for a running generation. Changing n changes ownership even with stable IDs. Also, the same shard launched twice still has overlapping ownership; see the separate concurrent-launch failure .

Test different snapshots, then audit the run

This small Python check reproduces the bad assignment and checks the correction:

all_ids = list(range(6))
first_pending = all_ids
later_pending = all_ids[1:]

bad0 = set(first_pending[0::2])
bad1 = set(later_pending[1::2])
assert bad0 & bad1 == {2, 4}

owned0 = {g for g in first_pending if g % 2 == 0}
owned1 = {g for g in later_pending if g % 2 == 1}
assert not owned0 & owned1
assert owned0 | owned1 == set(all_ids)
print("stable ownership checks passed")

The production evidence was broader than a queue disappearing: logs showed staggered starts, three workers restarting with smaller pending counts, and accepted/rejected outcomes covering the design. A final check reported:

missing (neither): []

That established outcome coverage. It was not a separate duplicate-output audit, and the logs did not establish why those workers restarted. Atomic replacement of a shared CSV also does not prove its concurrent snapshots were consistent.

Before your next resumable array, test ownership against two different completion snapshots. Every unfinished task should retain its owner regardless of when that owner starts.

Claude Code over SSH: submit long-running work to OpenPBS

Sat, 26 Sep 2026 00:00:00 +0000

Claude Code can prepare a computation on your laptop and operate a remote cluster through ordinary SSH commands. In this setup, Claude Code ran locally on macOS; source uploads, compilation, queue submissions, and checks ran on a Linux server. OpenPBS handled the long-running computation after submission, so the laptop didn’t need to keep an SSH session open.

The operating rule was explicit: work estimated at one hour or more belonged in PBS. That threshold was a local policy, not an OpenPBS default. Use your site’s rules, including its restrictions on short tests and compilation on login nodes.

Establish the SSH connection yourself

Configure authentication and verify the server’s host key before asking an agent to connect. For example, in your local ~/.ssh/config:

Host compute-login
 HostName 192.168.0.2
 User researcher
 BatchMode yes
 ConnectTimeout 10

The address and account are placeholders. With your site’s actual values and authentication configured, run this from the laptop:

ssh compute-login 'hostname; pwd; command -v qsub; command -v python3'

BatchMode makes missing authentication fail instead of waiting for a password. It does not approve the connection or replace host-key verification. Give Claude permission for the intended remote actions through your normal tool-approval workflow; never paste a private key into the conversation.

Make the execution boundary explicit

A useful project instruction is:

Claude runs on the laptop. Use SSH for remote operations.
Keep this project's remote files in ~/batch-demo.
Submit tasks estimated at one hour or more through OpenPBS.
Follow site policy for shorter tests; do not bypass the scheduler.
Record the job ID after submission. Check logs and expected outputs before reporting completion. Ask before cancellation or deletion.

Save this with the project’s agent instructions. A prose rule still needs checking: our early sessions launched substantial work directly before the PBS rule was captured. Also, having a GPU on a login host does not grant permission to use it.

Submit a small smoke test first

These are generalized setup examples, not a copy of the original scientific workload. Create run.py locally:

import socket
import sys

print(f"worker host: {socket.gethostname()}")
print(f"python: {sys.executable}")
print("smoke test complete")

Create run.pbs beside it:

#!/bin/bash
#PBS -N agent-smoke
#PBS -l select=1:ncpus=1
#PBS -l walltime=00:05:00
#PBS -j oe

set -euo pipefail
cd "${PBS_O_WORKDIR:?PBS submission directory missing}"
exec "$PWD/.venv/bin/python" -u run.py

From the laptop, prepare and submit it:

ssh compute-login 'mkdir -p ~/batch-demo && cd ~/batch-demo && python3 -m venv .venv'
scp run.py run.pbs compute-login:batch-demo/
ssh compute-login 'cd ~/batch-demo && qsub run.pbs'

Record the exact job ID that qsub returns. The batch script selects its directory and interpreter without depending on an activated interactive shell. Install real dependencies according to site policy and check that the environment and project directory are accessible on the execution node. Shared paths and compatible Python installations are prerequisites, not promises made by SSH.

For GPU work, request the site’s documented GPU resources. The original array used select=1:ncpus=2:ngpus=1 per element. GPU visibility variables and binding depend on cluster configuration; inspect them inside an allocation rather than choosing an arbitrary device from the login host.

Follow the job through its outputs

On the login host, use qstat -f with the recorded ID for detailed status. Read the output file named in that status. Sites retaining finished-job history may also support qstat -xf for accounting. The upstream qsub manual documents submission options and PBS_O_WORKDIR; the qstat manual covers status queries.

An empty queue is insufficient evidence of success. For this smoke test, check the reported worker and Python path and the completion line. For production, also verify expected task outcomes and output contents.

The subsequent failures were about execution ownership and verification:

A failed remote build launched an old executable .
A second detached launch collided with resumable work .
PBS array resume needed stable task IDs .
Archive transfer started before the release was complete .

Before handing over a long run, submit the smoke test and check its output on the execution node’s actual environment.