Ssh on Lessons learned

A failed CUDA build over SSH still launched the old executable

Sat, 26 Sep 2026 00:00:00 +0000

A remote command returned a CUDA compilation error followed by a normal simulation startup banner:

error: calling a __device__ function("__sinf") from a __host__ __device__ function("inflowFluctuation") is not allowed

The simulation started, but it ran an older executable. Semicolons let the launch proceed after nvcc failed. Correcting the source and rebuilding recovered the test; guarding the build-to-launch transition would prevent that misleading result.

The Claude Code and OpenPBS setup explains the local-agent/remote-command boundary. This happened during an early SSH-driven test, before we moved the long-running work into scheduled jobs. The shell mistake also applies inside a PBS script.

Diagnostics did not control execution

The command had this structure, with project paths removed:

nvcc -o solver solver.cu > build.log 2>&1;
grep -A3 error build.log | head;
nohup ./solver case.cfg > run.log 2>&1 < /dev/null &

The build wrote to an executable path that already existed. Compilation failed, leaving a runnable earlier build. Printing matching diagnostics didn’t stop the next command, and backgrounding the program didn’t establish whether the requested source had compiled.

The failed-build result included an ordinary banner. After replacing __sinf with sinf in the host/device function, the rebuilt program’s banner contained the newly added inflowTurb=0.08 parameter and subsequent progress lines. The earlier banner lacked that field. That difference provided evidence of which code path actually ran.

The transcript shows this operational recovery. Its retry still used semicolon-separated build and launch commands, so it did not repair the fail-open shell structure.

Make compiler status a launch prerequisite

For a future build, preserve the compiler’s status explicitly. This generalized Bash pattern builds to a temporary path and only publishes it after success:

#!/bin/bash
set -euo pipefail
cd "${PROJECT_DIR:?Set PROJECT_DIR to the remote project directory}"

candidate=$(mktemp ./solver.build.XXXXXX)
trap 'rm -f "$candidate"' EXIT
if nvcc -o "$candidate" solver.cu > build.log 2>&1; then
 mv -f "$candidate" ./solver
else
 cat build.log >&2
 exit 1
fi
exec ./solver case.cfg

This is a recommended guard, not code installed during the incident. The named source and configuration must exist in the remote project. Use a single build owner: concurrent publishers to ./solver require a separate ownership or versioned-artifact design.

When running remotely, send the whole script through SSH so its directory selection and status checks happen in the same remote shell. Avoid embedding local path expansions accidentally. For batch work, run the payload in the allocated job rather than adding another unscheduled nohup launch.

Reproduce the shell failure without CUDA

On a local machine with a POSIX shell, this harmless fixture leaves an existing command available after a simulated build failure:

sh -c 'solver() { printf "old executable ran\n"; }; false; solver'

It prints old executable ran and exits successfully. The guarded form prevents execution:

sh -c 'solver() { printf "old executable ran\n"; }; false && solver'

That form prints nothing and returns a failure status. For a real compiler test, deliberately introduce an error and verify that no new run starts. Also log a build identifier or other deliberate runtime identity; a healthy-looking startup line only proves that some executable ran.

Check your next remote build command for the exact status dependency between compilation and launch, especially where diagnostic pipelines sit between them.

A second launch of a resumable worker: 'OSError: [Errno 39] Directory not empty'

Sat, 26 Sep 2026 00:00:00 +0000

A resumable remote dataset driver failed while cleaning an unfinished task directory:

OSError: [Errno 39] Directory not empty

Another worker was still writing there. The second launch treated an unfinished task as safe to restart, even though the first process already owned it. Checking the surviving worker and validating its affected output recovered that work unit; no locking fix was installed during the session.

This was an early detached-run failure in the Claude Code, SSH, and OpenPBS workflow . Scheduler-managed execution later improved job tracking, but an application still needs to prevent conflicting owners.

The waiting wrapper had not disappeared

The first launch created a detached wrapper that waited for a pilot, then became the production driver. Its structure was:

until grep -q "finished" pilot.log; do sleep 30; done
exec python -u run_dataset.py dataset >> production.log 2>&1

Before the transition, a search for the production process would not find that driver. After exec, a search for the wrapper’s old command name would no longer identify it. Neither snapshot alone described the complete lifecycle.

A later check did not establish that the waiting launch was gone. Another detached production command started. The resulting log contained two identical pending-work counts followed by a traceback from shutil.rmtree(gdir). Both drivers had selected the same unfinished task, and one attempted cleanup while the other’s solver wrote files.

The resume logic used completion markers to skip finished groups. It offered no exclusive claim on an unfinished directory.

Recovery required checking the survivor

The follow-up inspected running processes, restored the affected task’s configuration, and checked the surviving computation. Its original log file had been lost during cleanup. A later driver result reported that the task was accepted.

A broad grep for success also matched text in the traceback. Tightening the check to a result-line pattern distinguished the actual accepted task from an error message that happened to contain matching characters.

That evidence supports recovery of the affected work unit. It does not prove that no intermediate data was lost or that the session did not add a lock. Eventual production completion checked in a later session also cannot establish that overlapping launches became safe.

Claim ownership before destructive resume

For a single-host driver using a filesystem with working advisory locks, a whole-run lock is one possible improvement. On a system with util-linux flock, run from the project directory:

flock -n .production.lock python -u run_dataset.py dataset

Every launcher must use the same stable lock path. Do not delete and recreate the lock file while work is running. The payload named here is the application’s driver, not a standard tool.

This guard is a new recommendation. A small local check can verify contention without running a dataset:

import fcntl
import tempfile

with tempfile.TemporaryDirectory() as directory:
 with open(directory + "/worker.lock", "w") as first:
 fcntl.flock(first, fcntl.LOCK_EX | fcntl.LOCK_NB)
 with open(directory + "/worker.lock", "w") as second:
 try:
 fcntl.flock(second, fcntl.LOCK_EX | fcntl.LOCK_NB)
 except BlockingIOError:
 print("second owner rejected")
 else:
 raise AssertionError("overlapping owner accepted")

Cluster filesystems and multi-node workers need an ownership mechanism tested under their actual locking semantics. Per-task claims can allow parallelism that a whole-run lock forbids. Stable PBS shard assignment solves a different problem: preserving each worker’s task set across starts and restarts. It cannot exclude two copies of the same shard.

Before retrying unfinished work, establish whether a live process already owns it. A missing completion marker only says that completion has not been recorded.

Claude Code over SSH: submit long-running work to OpenPBS

Sat, 26 Sep 2026 00:00:00 +0000

Claude Code can prepare a computation on your laptop and operate a remote cluster through ordinary SSH commands. In this setup, Claude Code ran locally on macOS; source uploads, compilation, queue submissions, and checks ran on a Linux server. OpenPBS handled the long-running computation after submission, so the laptop didn’t need to keep an SSH session open.

The operating rule was explicit: work estimated at one hour or more belonged in PBS. That threshold was a local policy, not an OpenPBS default. Use your site’s rules, including its restrictions on short tests and compilation on login nodes.

Establish the SSH connection yourself

Configure authentication and verify the server’s host key before asking an agent to connect. For example, in your local ~/.ssh/config:

Host compute-login
 HostName 192.168.0.2
 User researcher
 BatchMode yes
 ConnectTimeout 10

The address and account are placeholders. With your site’s actual values and authentication configured, run this from the laptop:

ssh compute-login 'hostname; pwd; command -v qsub; command -v python3'

BatchMode makes missing authentication fail instead of waiting for a password. It does not approve the connection or replace host-key verification. Give Claude permission for the intended remote actions through your normal tool-approval workflow; never paste a private key into the conversation.

Make the execution boundary explicit

A useful project instruction is:

Claude runs on the laptop. Use SSH for remote operations.
Keep this project's remote files in ~/batch-demo.
Submit tasks estimated at one hour or more through OpenPBS.
Follow site policy for shorter tests; do not bypass the scheduler.
Record the job ID after submission. Check logs and expected outputs before reporting completion. Ask before cancellation or deletion.

Save this with the project’s agent instructions. A prose rule still needs checking: our early sessions launched substantial work directly before the PBS rule was captured. Also, having a GPU on a login host does not grant permission to use it.

Submit a small smoke test first

These are generalized setup examples, not a copy of the original scientific workload. Create run.py locally:

import socket
import sys

print(f"worker host: {socket.gethostname()}")
print(f"python: {sys.executable}")
print("smoke test complete")

Create run.pbs beside it:

#!/bin/bash
#PBS -N agent-smoke
#PBS -l select=1:ncpus=1
#PBS -l walltime=00:05:00
#PBS -j oe

set -euo pipefail
cd "${PBS_O_WORKDIR:?PBS submission directory missing}"
exec "$PWD/.venv/bin/python" -u run.py

From the laptop, prepare and submit it:

ssh compute-login 'mkdir -p ~/batch-demo && cd ~/batch-demo && python3 -m venv .venv'
scp run.py run.pbs compute-login:batch-demo/
ssh compute-login 'cd ~/batch-demo && qsub run.pbs'

Record the exact job ID that qsub returns. The batch script selects its directory and interpreter without depending on an activated interactive shell. Install real dependencies according to site policy and check that the environment and project directory are accessible on the execution node. Shared paths and compatible Python installations are prerequisites, not promises made by SSH.

For GPU work, request the site’s documented GPU resources. The original array used select=1:ncpus=2:ngpus=1 per element. GPU visibility variables and binding depend on cluster configuration; inspect them inside an allocation rather than choosing an arbitrary device from the login host.

Follow the job through its outputs

On the login host, use qstat -f with the recorded ID for detailed status. Read the output file named in that status. Sites retaining finished-job history may also support qstat -xf for accounting. The upstream qsub manual documents submission options and PBS_O_WORKDIR; the qstat manual covers status queries.

An empty queue is insufficient evidence of success. For this smoke test, check the reported worker and Python path and the completion line. For production, also verify expected task outcomes and output contents.

The subsequent failures were about execution ownership and verification:

A failed remote build launched an old executable .
A second detached launch collided with resumable work .
PBS array resume needed stable task IDs .
Archive transfer started before the release was complete .

Before handing over a long run, submit the smoke test and check its output on the execution node’s actual environment.