OpenPBS array resume: shard by stable task ID

A resumable OpenPBS array can assign the same task to different workers if each worker slices its own remaining-work list. The unsafe implementation was:

todo = [g for g in groups if not completed(g)]
todo = todo[i::n]

We caught this before the array ran and changed ownership to group_id % worker_count. The corrected workers later started at different times and restarted with fewer tasks remaining; the final audit accounted for every group.

The Claude Code, SSH, and OpenPBS setup explains where commands ran and how jobs were submitted. This failure concerns application-level partitioning, not PBS assigning one array index twice.

A pending list changes between starts #

The workload was moving from a multi-GPU job to four independently scheduled one-GPU array elements. Each worker skipped groups with stored completion or rejection markers, then selected a shard.

Suppose the immutable task IDs are zero through five. Worker zero starts first and takes every second pending entry:

pending: 0 1 2 3 4 5
worker 0: 0 2 4

It finishes task zero before worker one starts. Worker one’s view is now:

pending: 1 2 3 4 5
worker 1: 2 4

Tasks two and four overlap, while one, three, and five have no owner in that pair of snapshots. Restarting workers creates the same problem. A completion marker can filter old work without stabilizing the positions of everything that remains.

Preserve ownership while filtering completion #

The actual correction retained the immutable numeric group ID:

todo = [g for g in todo if g["group"] % n == i]

Equivalently, make the two decisions explicit:

owned = [g for g in groups if g["group"] % n == i]
todo = [g for g in owned if not completed(g)]

In this application, --shard I/N supplied i and n. The submitted script used #PBS -J 0-3 and passed --shard "$PBS_ARRAY_INDEX/4" to the project-specific driver. Each element requested one GPU. Smaller resource requests did not guarantee immediate starts; later queue snapshots still showed waiting elements.

Keep the shard count fixed for a running generation. Changing n changes ownership even with stable IDs. Also, the same shard launched twice still has overlapping ownership; see the separate concurrent-launch failure .

Test different snapshots, then audit the run #

This small Python check reproduces the bad assignment and checks the correction:

all_ids = list(range(6))
first_pending = all_ids
later_pending = all_ids[1:]

bad0 = set(first_pending[0::2])
bad1 = set(later_pending[1::2])
assert bad0 & bad1 == {2, 4}

owned0 = {g for g in first_pending if g % 2 == 0}
owned1 = {g for g in later_pending if g % 2 == 1}
assert not owned0 & owned1
assert owned0 | owned1 == set(all_ids)
print("stable ownership checks passed")

The production evidence was broader than a queue disappearing: logs showed staggered starts, three workers restarting with smaller pending counts, and accepted/rejected outcomes covering the design. A final check reported:

missing (neither): []

That established outcome coverage. It was not a separate duplicate-output audit, and the logs did not establish why those workers restarted. Atomic replacement of a shared CSV also does not prove its concurrent snapshots were consistent.

Before your next resumable array, test ownership against two different completion snapshots. Every unfinished task should retain its owner regardless of when that owner starts.