Agents on Lessons learned

Adapt security skills to your project's risks: from Go to OWASP LLM and iOS

Sun, 25 Oct 2026 21:50:00 +0100

A security skill written for Go covers different questions than one written for an LLM application or an iOS app. I started with golang-security from samber/cc-skills-golang , asked Claude to adapt it using the OWASP Top 10 for LLM Applications, and later carried that approach into production iOS development.

The most useful part of the adaptation was aligning the review questions with the project’s risks and the company’s values.

Start with a review method

I was working in Go, a language I was unfamiliar with. I used my security skills alongside other Go skills to keep the implementation in check.

The security skill provided a repeatable method: separate modes for coding, reviewing changes, and auditing a codebase; questions about trust boundaries and attacker-controlled input; and instructions to trace data through surrounding code before reporting a finding. Its references and checklist gave the agent somewhere to look beyond the main instructions.

Adapt OWASP categories to the application

I asked Claude to create a new skill using the OWASP Top 10 for LLM Applications (2025) , adapted to the project’s LLM and Model Context Protocol (MCP) surfaces.

The resulting llm-security skill kept the review modes and evidence requirements. It mapped the categories to tools, resources, prompts, and content returned to the model. The Go checklist stayed active for the code underneath.

For prompt injection , the reviewer needs to identify whose text reaches the model and what an attacker can control. A returned document body or filename can carry instructions. For excessive agency , the questions concern tool permissions, reachable side effects, and authorization enforced outside the model.

Those questions give the agent a path through the repository. It can follow a tool result back to its data source, then follow a tool call forward to the operation it performs. The skill also asks it to account for existing protections when assessing impact.

Repeat the adaptation for iOS

Later, I applied the same approach to an iOS app using the OWASP Mobile Top 10 (2024) . That mobile security skill became part of production development.

The review method carried over. The mapping changed to platform APIs and the code responsible for them:

Mobile risk	Concrete review question
Improper credential usage	Where are tokens stored, and which Keychain access settings apply?
Insufficient input/output validation	What can a deep link or share-extension input cause the app to do?
Insecure data storage	Which protection classes apply to cached and temporary user files?
Inadequate privacy controls	Where can sensitive data appear outside its intended screen, including logs and notifications?

Repository context matters here. If a shared SDK owns authentication or networking, the skill needs to direct the reviewer into that SDK. Stopping at the changed Swift view would leave the relevant behavior unexamined.

For a mobile adaptation, MASVS supplies verification controls, and MASTG provides testing guidance for checking them.

Turn company values into review rules

Company values need concrete instructions too. For example, a company that prioritizes privacy can require reviewers to trace user data into logging, analytics, and crash reporting. That turns a broad preference into something an agent can inspect.

The mobile skill also records settled product decisions. An intentional tradeoff needs a rationale and the protections that still apply so that each review can assess the implementation against that decision. New evidence or changed assumptions can justify reopening it.

Severity needs the same local reasoning. OWASP provides risk categories; the product’s data, attacker access, and existing controls determine a finding’s impact.

A fresh adaptation request could be:

Use the existing security skill as a template. Keep its review workflow
and evidence requirements. Adapt the relevant OWASP categories to this
repository's entry points, data flows, and platform APIs. Link each check
to the responsible code and a primary reference. Record product decisions
with their rationale. Preserve the language-level checks alongside it.

Tailoring a security skill gives the agent concrete questions about the product it is building. Take one OWASP category in your current skill and map it to the code, data, and product decision the reviewer should examine.

A repeatable agentic development workflow: context, tickets, implementation, review

Sun, 25 Oct 2026 18:25:00 +0100

Giving a coding agent a feature request and accepting the first working diff is easy. Repeating that across projects is where things get messy: the agent has to rediscover the domain, invent missing requirements, decide what “done” means, and review its own assumptions.

Here is the setup I now reuse across stacks. It has supported ten iOS app releases this year, helped clean up an existing app built with loose agent guidance, and helped a Go implementation recover after its first approach fell apart - to name a few examples. The Go work also involved a language stack and MCP technology I had not used before. The process didn’t provide that expertise on its own; it made the gaps visible early enough to address them.

Give the agent durable context

Start with mattpocock/skills and run its repository setup. Keep a short domain vocabulary in CONTEXT.md (or the current upstream convention, GLOSSARY.md). Record consequential design choices as ADRs. The agent can then use the same terms in discussions, tickets, code, and reviews.

For a change whose shape is uncertain, use /grill-with-docs to work through the decisions and write them down. Run /to-spec once the conversation has enough detail. Break the spec into small, ordered tickets with /to-issues in older installations or /to-tickets in the current upstream skills. Then use /implement against those tickets.

The sequence is:

shared vocabulary + ADRs → grill → spec → tickets → implement → review

You can enter the sequence at whichever step the work needs. A small, well-specified fix may already have a ticket. An uncertain feature should be grilled before an agent starts writing code.

Put tickets where the project works

The tracker is an adapter for the process. One project keeps issues in a local docs/TODO.txt; others use Vikunja or Taiga. What matters is that the agent knows where the authoritative spec and tickets live, how to record blockers, and how to update their status. Write those rules into the repository’s agent documentation. A tracker integration skill can handle the API details without changing the development sequence.

This also makes the setup portable. Moving to a new project means teaching the agent that project’s domain, tracker, and checks, rather than rebuilding the workflow.

Make implementation answer to the spec

Implementation includes tests and a diff review. Review the result against two different questions: does the code follow the project’s standards, and does it actually satisfy the originating spec? The latter catches plausible code that solves the wrong problem. A previous review found duplicate-write paths after a large test suite passed because it checked a safety claim against the call path.

Where the project’s rules allow it, use a different model family for review. A typical split is Claude writing the code and GPT reviewing it, since a different family brings different blind spots. Claude reviewing Claude has also proven decent, so one family is still a workable setup when you can only use one provider. For larger changes, a multi-review skill can run distinct axes such as correctness, quality, tests and performance, security, and, for frontend, accessibility. Give each reviewer the same bounded diff and the actual spec; collect actionable findings back into the tracker.

Attach stack-specific skills where they help. Go work can load guidance for concurrency, error handling, testing, and security. iOS work can use Swift and mobile guidance. For exposed web APIs, mobile clients, or MCP tools, apply relevant OWASP-based web, mobile, and LLM security skills during implementation and review. A checklist gives the agent concrete questions to ask; it does not turn a code review into a security certification.

Start with one change: write down the vocabulary and decision, turn the request into a spec and tickets, implement one ticket, and review the diff against the spec and its security boundaries. Adjust the tracker, skills, and review axes to fit the next project.

'Downgrading is unsupported' after a config restore: the update guard was right

Fri, 16 Oct 2026 00:00:00 +0000

After a configuration restore, an updater refused with:

that release predates the gateway runtime these session stores now use. Downgrading is unsupported.

The operator requested an update. The restored configuration selected an older code line than the databases required. Restoring the intended per-install channel corrected target selection; bypassing the guard would have put incompatible code over newer state.

A newer release number can select older capabilities

The checkout followed a development branch. Its session databases already contained runtime-ledger history. The restored configuration backup lacked the install’s channel record, so release selection fell back to the stable channel.

The stable release had a higher patch version, but it came from a release branch without those runtime-ledger tables. Version ordering did not describe the relevant capability ordering.

The historical guard checked two things: whether the proposed target descended from the running commit, and whether a non-descendant target contained the runtime schema marker. It also read the session stores for ledger rows. Its refusal named the affected stores and stopped before moving the checkout.

The implementation explained the consequence: older code could not delete or prune sessions with that runtime history. This was a specific compatibility check, not proof that every forward commit or every future schema change would be safe.

Repair target selection before changing code

The diagnosis followed the configuration lookup:

Identify the actual install path and the channel record associated with it.
Compare that record with the backup that was restored. Check defaults when the record is absent.
Resolve the updater’s proposed target to a commit, rather than relying on its release label.
Compare the target’s ancestry and required state capabilities with the running checkout.
Reassert the intended channel through the updater’s supported configuration command, then run its check-only mode.

The corrected check selected the development branch and reported available forward commits—that verified channel selection. The update itself was deliberately not run, so the record does not establish a completed runtime upgrade.

For a Git-based installation, this read-only check is useful once TARGET is set to the exact resolved commit. Run it inside that installation’s checkout:

git merge-base --is-ancestor HEAD "${TARGET:?Set TARGET to the resolved update commit}"

Exit status 0 means the target descends from the current checkout. Status 1 means it does not; other failures require investigation. A non-descendant can be a valid release-branch target, so this command alone is not a compatibility verdict. Check the particular schema capability named by the refusal.

A config-only restore can create a mixed generation

The database kept its newer state while the restored configuration lost an additive setting. In this implementation, channel records were keyed by install path identity. Restoring an older backup, or moving an install without recreating its record, could therefore change how later updates were selected.

Deployment automation should assert the intended channel after install or restore, then read it back through the supported interface. Do not hard-code a path-derived identifier from another machine. Following a development branch is appropriate only when that is the deliberate operating policy; it is not a universal fix for update errors.

For an intentional rollback, follow the application’s documented procedure for compatible code and pre-upgrade data together. Preserve the current state first, and account for work absent from the older backup. Editing a schema-version value or removing ledger rows merely to silence the guard would discard the evidence it uses to protect the store.

Add an update-target check to your restore drill before allowing the restored installation to update itself.

Cyber classifiers can interrupt offline pentest summaries

Fri, 09 Oct 2026 00:00:00 +0000

An agent spent three minutes reading local transcripts from a finished, authorized security exercise, then stopped with:

This request triggered cyber-related safeguards.

No live target was involved. The job was offline summarization, yet the classifier interrupted it before it could write the requested report. The practical fix was to treat security-history analysis as a resumable data pipeline: retain message provenance, checkpoint extracted evidence, and keep the handoff usable by another approved model or provider.

A flag-shaped string is not a captured flag

The first problem appeared before the classifier fired. A search through about 25 request dumps found dozens of flag-shaped strings. About 20 were spelling, capitalization, spacing, or hyphenation variants of one value.

Most variants appeared only in assistant reasoning and progress notes. The model repeatedly retyped a remembered value and changed it. A checkpoint then copied one mutation into later sessions.

Only strings preserved in tool results represented observed server output. To separate evidence from generation, use this origin-based distinction:

user request instruction or supplied context
assistant text generated claim; unverified
assistant reasoning generated hypothesis; unverified
tool call attempted action
tool result observed output from that action
provider error evidence about the run, not the target

This applies beyond CTF flags. Treat IDs, hashes, error codes, hostnames, and quoted responses in model prose as leads until the first tool result corroborates them. Exact-looking text is still generated text.

Offline processing still triggers cyber classifiers

The retained run used Opus 5.5. It parsed local request dumps, separated real and decoy-looking values, and extracted an existing SMB investigation summary. After that, the next model response was refused under the provider’s real-time cyber safeguard. The session recorded a cyber refusal and pointed to its Cyber Verification Program. The output file did not exist afterward.

Some source dumps also contained earlier cyber_policy failures from another provider along with a prompt to apply for Daybreak access. Because these records were historical input, they still increased the concentration of offensive-security language the summarizer saw.

A related authorized test had a similar shape: generate enough traffic against a non-production environment to trigger a WAF, then verify that the application displays and retries a CAPTCHA correctly. In a newer Opus session, the controlled load-test work was refused. The retained successful session used Opus 4.6 and built the test client, while a later session corrected its cookie handling and the request used to observe the challenge.

The awkward lesson is that task intent and classifier input are different things. In practice, “Summarize these old logs” can contain exploit commands, credentials, flags, and attack narratives. Likewise, “Test the CAPTCHA” can require traffic that resembles denial of service when reduced to request counts.

Make the audit resumable and model-neutral

Do deterministic work before asking a model to synthesize:

Parse each transcript into typed events and preserve the source file, timestamp, role, tool-call ID, and tool-result ID.
Extract exact-match artifacts only from tool results. Store assistant claims separately as unverified leads.
Write incremental JSON or Markdown after each source file. Never leave the only useful state in the model’s context window.
Produce a sanitized handoff containing the authorization scope, completed files, pending questions, and evidence pointers.
Verify the final report exists and reconcile its artifact counts against the extraction index.

Provider access programs are worth the setup when this work recurs. In my experience, GPT Daybreak Blue justified the hassle for authorized security-related tasks. Kimi, GLM, and older model versions have also completed useful work when a newer classifier blocked the same class of jobs. That is a fallback, not a bypass: preserve the same authorization boundary, do not weaken the requested scope, and use models approved for the work.

Before trusting your next transcript summary, choose one exact token from the report and trace it back to the tool result that observed it.

Inherited agent notes can send the next agent down the same dead end

Fri, 09 Oct 2026 00:00:00 +0000

A new agent can go through a failed investigation again if its first job is to absorb the conclusions of the previous agent. In the course of an authorised security exercise the user was repeatedly asking for a clean start as the prior progress files could be somewhat misleading. The useful reset was to establish a small verified baseline before using the inherited notes to select the next experiment.

A hint survived longer than its evidence

The conversation history had instructions to go it alone after a number of failed attempts at the task. Another handoff stated the prior agent had got lost and that its notes could be misleading to the replacement.

One concrete example concerned an environment hint favoring IPv6. A later SUMMARY.md excerpt said:

IPv4 and IPv6 behave identically everywhere tested

It was very explicit that the IPv6 hint in AGENTS.md didn’t apply in that case. That’s a useful clarification, with a caveat: this is a recorded summary of the checks, not an independent re-run of every network probe. It does support the removal of that hint for that particular surface. It doesn’t in any way establish equivalence for all services or a successful resolution of the problem. The outcome is still unknown.

Rebuild only enough context to choose a test

On a restart read the current task and the mandatory operating instructions. Observe the authorization boundaries and safety rules. Be very clear in your mind about the difference between the instructions and the factual observations about the environment. A line in AGENTS.md could be a binding rule, it could be a fact about the setup or it could be a hypothesis that you need to go out and check.

Then establish the facts required to inform the next decision. In a network investigation that might involve confirming the target endpoint, checking that it’s reachable and then making a baseline request to it. In a code issue scenario you’d want to reproduce the failure that was reported and then have a look at the current code before accepting any diagnosis from a previous developer.

Only then do you compare the inherited notes against that baseline. That way you don’t re-run the whole investigation and the new agent has its own baseline to work from.

A handoff instruction can be as short as:

Establish a minimal baseline from the current tool output.
Treat inherited diagnoses as hypotheses.
For each claim you make, you need to be able to locate the evidence or at the very least be able to run a very cheap check.
Keep the required operating and safety instructions in force.

Preserve the test behind a negative result

A note such as “IPv6 works better” is completely devoid of the endpoint, the request and the comparison that would actually make it useful. “No difference on these tested endpoints” is much more specific and much easier to verify.

For future handoffs, record:

The claim and its scope (including when it was checked).
The command or test and a pointer to its output.
What remains is an interpretation or an unproven lead.

Keep the failed attempts too and state what each one ruled out. A failed request with one set of credentials doesn’t rule out the whole protocol. The next agent needs to be able to make the judgement as to whether repeating that test would in fact add any evidence.

Copying an assertion into successive summaries doesn’t constitute independent corroboration. If the raw output is missing then unverified is the correct designation rather than promoting it to a fact.

Prior to your next agent handoff, pick the assumption that will drive the next action and attach the relevant tool output in support of that assumption. If you can’t find it then that will be the restart task.

A failed CUDA build over SSH still launched the old executable

Sat, 26 Sep 2026 00:00:00 +0000

A remote command returned a CUDA compilation error followed by a normal simulation startup banner:

error: calling a __device__ function("__sinf") from a __host__ __device__ function("inflowFluctuation") is not allowed

The simulation started, but it ran an older executable. Semicolons let the launch proceed after nvcc failed. Correcting the source and rebuilding recovered the test; guarding the build-to-launch transition would prevent that misleading result.

The Claude Code and OpenPBS setup explains the local-agent/remote-command boundary. This happened during an early SSH-driven test, before we moved the long-running work into scheduled jobs. The shell mistake also applies inside a PBS script.

Diagnostics did not control execution

The command had this structure, with project paths removed:

nvcc -o solver solver.cu > build.log 2>&1;
grep -A3 error build.log | head;
nohup ./solver case.cfg > run.log 2>&1 < /dev/null &

The build wrote to an executable path that already existed. Compilation failed, leaving a runnable earlier build. Printing matching diagnostics didn’t stop the next command, and backgrounding the program didn’t establish whether the requested source had compiled.

The failed-build result included an ordinary banner. After replacing __sinf with sinf in the host/device function, the rebuilt program’s banner contained the newly added inflowTurb=0.08 parameter and subsequent progress lines. The earlier banner lacked that field. That difference provided evidence of which code path actually ran.

The transcript shows this operational recovery. Its retry still used semicolon-separated build and launch commands, so it did not repair the fail-open shell structure.

Make compiler status a launch prerequisite

For a future build, preserve the compiler’s status explicitly. This generalized Bash pattern builds to a temporary path and only publishes it after success:

#!/bin/bash
set -euo pipefail
cd "${PROJECT_DIR:?Set PROJECT_DIR to the remote project directory}"

candidate=$(mktemp ./solver.build.XXXXXX)
trap 'rm -f "$candidate"' EXIT
if nvcc -o "$candidate" solver.cu > build.log 2>&1; then
 mv -f "$candidate" ./solver
else
 cat build.log >&2
 exit 1
fi
exec ./solver case.cfg

This is a recommended guard, not code installed during the incident. The named source and configuration must exist in the remote project. Use a single build owner: concurrent publishers to ./solver require a separate ownership or versioned-artifact design.

When running remotely, send the whole script through SSH so its directory selection and status checks happen in the same remote shell. Avoid embedding local path expansions accidentally. For batch work, run the payload in the allocated job rather than adding another unscheduled nohup launch.

Reproduce the shell failure without CUDA

On a local machine with a POSIX shell, this harmless fixture leaves an existing command available after a simulated build failure:

sh -c 'solver() { printf "old executable ran\n"; }; false; solver'

It prints old executable ran and exits successfully. The guarded form prevents execution:

sh -c 'solver() { printf "old executable ran\n"; }; false && solver'

That form prints nothing and returns a failure status. For a real compiler test, deliberately introduce an error and verify that no new run starts. Also log a build identifier or other deliberate runtime identity; a healthy-looking startup line only proves that some executable ran.

Check your next remote build command for the exact status dependency between compilation and launch, especially where diagnostic pipelines sit between them.

A second launch of a resumable worker: 'OSError: [Errno 39] Directory not empty'

Sat, 26 Sep 2026 00:00:00 +0000

A resumable remote dataset driver failed while cleaning an unfinished task directory:

OSError: [Errno 39] Directory not empty

Another worker was still writing there. The second launch treated an unfinished task as safe to restart, even though the first process already owned it. Checking the surviving worker and validating its affected output recovered that work unit; no locking fix was installed during the session.

This was an early detached-run failure in the Claude Code, SSH, and OpenPBS workflow . Scheduler-managed execution later improved job tracking, but an application still needs to prevent conflicting owners.

The waiting wrapper had not disappeared

The first launch created a detached wrapper that waited for a pilot, then became the production driver. Its structure was:

until grep -q "finished" pilot.log; do sleep 30; done
exec python -u run_dataset.py dataset >> production.log 2>&1

Before the transition, a search for the production process would not find that driver. After exec, a search for the wrapper’s old command name would no longer identify it. Neither snapshot alone described the complete lifecycle.

A later check did not establish that the waiting launch was gone. Another detached production command started. The resulting log contained two identical pending-work counts followed by a traceback from shutil.rmtree(gdir). Both drivers had selected the same unfinished task, and one attempted cleanup while the other’s solver wrote files.

The resume logic used completion markers to skip finished groups. It offered no exclusive claim on an unfinished directory.

Recovery required checking the survivor

The follow-up inspected running processes, restored the affected task’s configuration, and checked the surviving computation. Its original log file had been lost during cleanup. A later driver result reported that the task was accepted.

A broad grep for success also matched text in the traceback. Tightening the check to a result-line pattern distinguished the actual accepted task from an error message that happened to contain matching characters.

That evidence supports recovery of the affected work unit. It does not prove that no intermediate data was lost or that the session did not add a lock. Eventual production completion checked in a later session also cannot establish that overlapping launches became safe.

Claim ownership before destructive resume

For a single-host driver using a filesystem with working advisory locks, a whole-run lock is one possible improvement. On a system with util-linux flock, run from the project directory:

flock -n .production.lock python -u run_dataset.py dataset

Every launcher must use the same stable lock path. Do not delete and recreate the lock file while work is running. The payload named here is the application’s driver, not a standard tool.

This guard is a new recommendation. A small local check can verify contention without running a dataset:

import fcntl
import tempfile

with tempfile.TemporaryDirectory() as directory:
 with open(directory + "/worker.lock", "w") as first:
 fcntl.flock(first, fcntl.LOCK_EX | fcntl.LOCK_NB)
 with open(directory + "/worker.lock", "w") as second:
 try:
 fcntl.flock(second, fcntl.LOCK_EX | fcntl.LOCK_NB)
 except BlockingIOError:
 print("second owner rejected")
 else:
 raise AssertionError("overlapping owner accepted")

Cluster filesystems and multi-node workers need an ownership mechanism tested under their actual locking semantics. Per-task claims can allow parallelism that a whole-run lock forbids. Stable PBS shard assignment solves a different problem: preserving each worker’s task set across starts and restarts. It cannot exclude two copies of the same shard.

Before retrying unfinished work, establish whether a live process already owns it. A missing completion marker only says that completion has not been recorded.

Claude Code over SSH: submit long-running work to OpenPBS

Sat, 26 Sep 2026 00:00:00 +0000

Claude Code can prepare a computation on your laptop and operate a remote cluster through ordinary SSH commands. In this setup, Claude Code ran locally on macOS; source uploads, compilation, queue submissions, and checks ran on a Linux server. OpenPBS handled the long-running computation after submission, so the laptop didn’t need to keep an SSH session open.

The operating rule was explicit: work estimated at one hour or more belonged in PBS. That threshold was a local policy, not an OpenPBS default. Use your site’s rules, including its restrictions on short tests and compilation on login nodes.

Establish the SSH connection yourself

Configure authentication and verify the server’s host key before asking an agent to connect. For example, in your local ~/.ssh/config:

Host compute-login
 HostName 192.168.0.2
 User researcher
 BatchMode yes
 ConnectTimeout 10

The address and account are placeholders. With your site’s actual values and authentication configured, run this from the laptop:

ssh compute-login 'hostname; pwd; command -v qsub; command -v python3'

BatchMode makes missing authentication fail instead of waiting for a password. It does not approve the connection or replace host-key verification. Give Claude permission for the intended remote actions through your normal tool-approval workflow; never paste a private key into the conversation.

Make the execution boundary explicit

A useful project instruction is:

Claude runs on the laptop. Use SSH for remote operations.
Keep this project's remote files in ~/batch-demo.
Submit tasks estimated at one hour or more through OpenPBS.
Follow site policy for shorter tests; do not bypass the scheduler.
Record the job ID after submission. Check logs and expected outputs before reporting completion. Ask before cancellation or deletion.

Save this with the project’s agent instructions. A prose rule still needs checking: our early sessions launched substantial work directly before the PBS rule was captured. Also, having a GPU on a login host does not grant permission to use it.

Submit a small smoke test first

These are generalized setup examples, not a copy of the original scientific workload. Create run.py locally:

import socket
import sys

print(f"worker host: {socket.gethostname()}")
print(f"python: {sys.executable}")
print("smoke test complete")

Create run.pbs beside it:

#!/bin/bash
#PBS -N agent-smoke
#PBS -l select=1:ncpus=1
#PBS -l walltime=00:05:00
#PBS -j oe

set -euo pipefail
cd "${PBS_O_WORKDIR:?PBS submission directory missing}"
exec "$PWD/.venv/bin/python" -u run.py

From the laptop, prepare and submit it:

ssh compute-login 'mkdir -p ~/batch-demo && cd ~/batch-demo && python3 -m venv .venv'
scp run.py run.pbs compute-login:batch-demo/
ssh compute-login 'cd ~/batch-demo && qsub run.pbs'

Record the exact job ID that qsub returns. The batch script selects its directory and interpreter without depending on an activated interactive shell. Install real dependencies according to site policy and check that the environment and project directory are accessible on the execution node. Shared paths and compatible Python installations are prerequisites, not promises made by SSH.

For GPU work, request the site’s documented GPU resources. The original array used select=1:ncpus=2:ngpus=1 per element. GPU visibility variables and binding depend on cluster configuration; inspect them inside an allocation rather than choosing an arbitrary device from the login host.

Follow the job through its outputs

On the login host, use qstat -f with the recorded ID for detailed status. Read the output file named in that status. Sites retaining finished-job history may also support qstat -xf for accounting. The upstream qsub manual documents submission options and PBS_O_WORKDIR; the qstat manual covers status queries.

An empty queue is insufficient evidence of success. For this smoke test, check the reported worker and Python path and the completion line. For production, also verify expected task outcomes and output contents.

The subsequent failures were about execution ownership and verification:

A failed remote build launched an old executable .
A second detached launch collided with resumable work .
PBS array resume needed stable task IDs .
Archive transfer started before the release was complete .

Before handing over a long run, submit the smoke test and check its output on the execution node’s actual environment.

max_retries_exhausted in a scheduled agent: alert on overdue outcomes, not request dumps

Fri, 25 Sep 2026 00:00:00 +0000

A scheduled agent produced 67 failed-request dumps over roughly two months, yet the archive contains no successful-turn record. The failure reports ended with:

max_retries_exhausted
non_retryable_client_error

Saving the request made diagnosis possible later, but it did nothing to alert an operator to the failed work. To surface unattended jobs, add an independent watcher that alerts when a verified outcome becomes overdue, even if the agent cannot produce a response.

What the failure archive actually proves

The records span five error shapes: a provider-side 503 circuit breaker, a broken pipe, an access-verification service error, a quota rejection, and an invalid-credential response. Counting the dump metadata gives 66 retry-exhaustion failures and one non-retryable client error.

That is evidence of repeated failed requests. It cannot establish that every scheduled invocation failed: successful work might have been recorded elsewhere, and a request can fail after earlier tool actions succeeded. Nor does the absence of a notification in a dump prove that no external monitor existed.

The operational problem exposed by this archive is narrower and useful: failure artifacts alone cannot answer, “When did the job last finish its work?” Because this corpus contains no success receipt to reconcile the errors, it demonstrates no monitoring repair.

Watch the outcome from outside the agent

For each scheduled job, define a durable success receipt tied to its actual acceptance condition. For a capture job, that might mean the destination readback confirms the captured items. A model response saying “done” is not sufficient.

Record at least a run ID, start time, terminal status, and the last verified success time. Keep partial progress separate so a retry can reconcile completed writes instead of duplicating them.

Then run a watcher independently of the model call:

Alert on a terminal failure, including a normalized error class and a run ID.
Alert when the last verified success exceeds the allowed interval plus a bounded grace period.
Treat a missing first success as overdue too. A watcher that only checks an existing timestamp can overlook a job that never worked.
Send the alert through a channel that does not require the failing model provider.

Deduplicate repeated alerts for the same incident, but keep an overdue-success alert active until someone acknowledges or resolves it. If alert delivery fails, track that failure as its own observable status.

Classify before retrying

The archive’s final credential rejection was classified as non-retryable. That classification is a reason to stop automatic retries and request an authentication/configuration check. It does not prove exactly which configuration change caused the rejection.

A broken pipe or service outage can justify bounded retries. Quota failures may require waiting for a published reset time. Preserve the specific provider error class: an HTTP status alone does not show whether the next attempt can succeed.

Pausing should produce an explicit incident state with a resume condition. Otherwise, “automatically disabled” becomes another silent failure.

Test the monitor with deliberately broken runs

In a disposable job, inject a retry-exhaustion failure and a non-retryable credential failure. Confirm both reach the independent alert channel. Suppress the runner entirely and confirm that the overdue-success check still fires. Finally, make the model report success without completing the destination write; that must not refresh the success timestamp.

Keep raw dumps access-controlled and retention-limited. They can contain the complete outgoing prompt and operational context; send a redacted summary and a restricted evidence reference in alerts.

Before leaving a job unattended, deliberately prevent one run from starting and check whether anyone notices.

Hourly agent cron vs. '429 usage_limit_reached': persist the provider's quota reset

Sat, 19 Sep 2026 00:00:00 +0000

An hourly scheduled agent repeatedly received 429 usage_limit_reached while its provider response already supplied a future reset time. One recorded delay was resets_in_seconds: 202825, about 2.35 days. The scheduler needed a durable not-before deadline checked before making another model call.

A failure archive contained 62 quota-related failed-request dumps, plus five failures of other kinds. Those are dump counts, not counts of individual HTTP attempts: a single invocation could retry several times before saving a dump. The archive does not establish that every scheduled invocation failed, and it contains no demonstrated scheduling repair or successful recovery.

A cron tick should consult shared cooldown state

A retry loop inside one invocation cannot govern the next invocation. When a process exits, an in-memory backoff disappears; the next cron tick can start the same futile sequence.

Persist the cooldown at the scope of the exhausted quota, such as the provider account and quota bucket. Every worker sharing that bucket must consult it before the first request. A different job name does not create a different quota.

The HTTP Retry-After specification permits either an HTTP date or a delay in seconds. Provider JSON fields such as resets_at and resets_in_seconds are provider-specific; normalize them in an adapter rather than assuming their units or presence. Use receipt time for relative delays. If multiple applicable constraints exist, do not schedule before any of them.

An illustrative gate, after parsing and validating those fields:

import math


def next_probe(existing, received_at, *, retry_after_at=None,
 quota_reset_at=None, margin=5):
 deadlines = [x for x in (retry_after_at, quota_reset_at)
 if x is not None]
 if not deadlines:
 raise ValueError("no usable deadline; use bounded fallback policy")
 values = [existing, received_at, margin, *deadlines]
 if not all(math.isfinite(x) and x >= 0 for x in values):
 raise ValueError("invalid deadline")
 return max(existing, max(received_at, *deadlines) + margin)


def may_probe(now, not_before):
 return now >= not_before


# Synthetic clock values; these are not recorded provider timestamps.
deadline = next_probe(0, 1000, quota_reset_at=1000 + 202825)
assert deadline == 203830
assert not may_probe(1000 + 3600, deadline)
assert not may_probe(deadline - 1, deadline)
assert may_probe(deadline, deadline)
# A later response cannot shorten an existing cooldown.
assert next_probe(deadline, 2000, retry_after_at=3000) == deadline
# Both applicable constraints survive normalization.
assert next_probe(0, 1000, retry_after_at=5000,
 quota_reset_at=9000) == 9005
print("cooldown boundary assertions passed")

The gate checks timing only. Production code also needs atomic storage updates, a lock or lease to admit one recovery probe, and clock-skew handling. On restart, read the saved deadline; do not silently treat unreadable state as permission to call. Add bounded jitter after the deadline so workers do not all resume together.

Give long waits an operational policy

A near-term transient rate limit can justify a short bounded retry. A multi-day quota exhaustion should explicitly defer the job, with its next eligible probe time visible to an operator. If a provider advertises an implausibly distant reset, escalate instead of silently sleeping forever or truncating the wait and hammering the endpoint.

At the deadline, admit one probe. A new quota rejection updates the cooldown. An authentication rejection needs intervention rather than the same timed retry. A permitted model response still does not prove the scheduled task completed: reconcile destination writes before replaying work that may have partially succeeded.

Test across process boundaries

The executable example verifies the early, boundary, and monotonic-deadline cases. To verify a real scheduler, inject a quota response, terminate the worker, and start a new invocation before reset. Assert that no provider call occurs. Start competing workers at reset and assert that only one obtains the recovery lease. Then inject another quota rejection and confirm that it persists the next deadline.

Stopping repeated calls can hide an overdue job unless monitoring remains independent of the cooldown. The companion post on unattended failure alerts covers that separate requirement. The disappearance of failure dumps after a reset is not a success receipt.

Before the next scheduled run, check whether a provider reset deadline survives a worker restart.

A background agent Docker build: 'open Dockerfile: no such file or directory'

Sun, 30 Aug 2026 00:00:00 +0000

A background Docker build failed immediately with:

ERROR: failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory

The agent had supplied the project directory through the terminal tool’s workdir argument. The retry put cd inside the command itself. For an agent runner whose background path handling is uncertain, bind the directory in the command or make the Dockerfile and build context absolute.

The failed call already supplied workdir

In the historical runner, the background invocation had this shape, with project and image names generalized:

{
 "background": true,
 "workdir": "/srv/app",
 "command": "docker build --progress=plain -t local/app:dev -f Dockerfile ."
}

Docker could not open Dockerfile. The retry started with an explicit cd to the project before running the build. That corrected the launch form; whether the rest of that session succeeded is outside this lesson.

The error alone does not identify the actual launch directory. The record supports a runner-specific path-handling failure, not a rule that background processes generally discard their parent’s directory. Ordinary shell backgrounding with & inherits the shell’s working directory. An agent tool can launch its jobs through a different execution path.

A later check on the runner used to prepare this post found that background jobs preserved the preceding foreground directory, and an explicit background workdir was honored. The historical failure is therefore not a current universal behavior claim.

Bind both Docker paths deliberately

For an existing project at /srv/app, put the directory change and build in the same command:

cd /srv/app &&
 docker build --progress=plain -t local/app:dev -f Dockerfile .

The && prevents the build from running if the directory change fails. Alternatively, remove the directory dependency from both path arguments:

docker build --progress=plain -t local/app:dev \
 -f /srv/app/Dockerfile /srv/app

Making only -f absolute still leaves . as an accidental build context. Likewise, when using Compose, check its file selection and relative-path behavior rather than assuming a successful Docker build proves the next command has the right context.

Probe the actual execution mode

Before a multi-minute job, use a harmless foreground command to change to a known scratch directory:

cd "${TMPDIR:?Set TMPDIR to your scratch directory}" && pwd

Then submit pwd as a separate background tool call without workdir, and read that job’s captured output. Repeat with workdir set to the project root. Those two checks distinguish remembered foreground state from an explicit launch directory without building an image or starting services.

For a real build, print pwd and check test -f Dockerfile inside the same command that launches it. A foreground check in an earlier call cannot establish the background job’s directory.

Separate service startup from readiness checks

The same historical session hit another runner boundary. A foreground command combining docker compose up -d with a health-check loop was rejected with:

This foreground command appears to start a long-lived server/watch process.

The runner directed the agent to launch it with background=true, verify readiness, and run tests separately. That refusal came from the runner, even though Compose’s -d requests detached containers.

Keep service startup in its own call, following the runner’s execution rules. Check readiness in a separate bounded command, and fail explicitly if the deadline expires. Starting a service and proving it healthy are separate results.

Before your next background build, compare its reported directory with the intended build context using the harmless probe above.

Standards vs Spec review: a second agent found duplicate writes past 1,572 green tests

Wed, 22 Jul 2026 21:58:00 +0200

This is what the review agent reported, just before it listed its findings:

The test evidence is clean: Ruff passes, Python reports 1,572 passed / 56 skipped /
1 expected failure, and the GUI reports 418 passed / 4 skipped. The review nevertheless
found safety gaps the tests do not exercise — most importantly, two write clients retry
"500-after-commit" before their advertised idempotency/verification guard can run.

A coding agent had built the branch: 53 commits, 147 files. A different model reviewed it and found two paths that could put the same invoice in the ERP twice and report plain success. No test covered them, and no linter could. They came from a reviewer with one job: check the code against what the branch said it does.

The setup

The repository vendors the two-axis code-review skill from mattpocock/skills . It pins a fixed point and diffs against the merge base, git diff master...HEAD, then forks two subagents in parallel:

Standards: does the diff follow the repo’s documented rules (glossary, ADRs, a baseline of Fowler code smells)?
Spec: does the diff do what the originating issue or PRD asked for?

The parent agent runs lint and tests meanwhile, “as supporting evidence, not as a substitute for the diff review”. It then reports the two axes under separate headings and doesn’t rerank them, so a pile of naming nits can’t bury a P1.

The Spec brief is short:

Report: (a) requirements the spec asked for that are missing or partial;
(b) behavior in the diff that wasn't asked for (scope creep);
(c) requirements that look implemented but where the implementation looks wrong.
Quote the spec line for each finding, under 400 words.

What each axis found

There was no PRD for the branch, so the Spec reviewer used the commit messages and the ADRs. One commit claimed that the MCP server’s idempotency check made it safe to retry a write that got a 5xx. The reviewer traced the call: the check ran before _request(), the key was recorded after success, and the retry happened inside _request(), in between. A webhook worker had the same shape, with a verify-by-GET that ran only if the request finally raised. A retry that succeeds never raises. (The full HTTP 500 story is here .)

The Standards reviewer found something else. Two functions called urlopen() directly, past the shared transport that an ADR says owns all retries. It also found an import that went through a module the ADRs forbid as a re-export facade, and an unfinished vocabulary migration.

None of the Standards findings overlapped with the Spec ones. All six were fixed the same evening, in one commit.

Why the tests stayed green

The tests came from the same author as the claim. No test let a 500 commit the write, let the retry succeed, and counted the writes, because the commit said the idempotency check covered that case. A suite tests what its author thought could break.

The Spec reviewer starts from the claim instead. A commit message says the code is safe, so it goes looking for the line that makes it safe, and here that line ran on the wrong side of the retry.

Two more runs, same pattern

In the next review, a few hours later on a six-commit range, a guard was undone downstream. Every write had to carry exactly one confidence label, and create_* checked that. But when label setup had failed, a fallback further down stripped all labels from the payload. The reviewer reproduced a labeled invoice leaving without its label. It also found a write that went through a raw put() and skipped the scope check that limits which record types may be written at all.

The third review had nothing to go on but commit messages. The branch added a gate: if an invoice is addressed to another company, warn and refuse to post it. The reviewer said it was inferring the spec from the commits, so I pasted the original request and the Spec subagent re-checked against that. The request held a second requirement the commit summary had left out, which turned out to be implemented. The review also found that a replay of staged payloads called the push function directly and skipped the new gate. And while company registration numbers were checksum-validated, VAT IDs relied on regex alone, so one misread digit could block a legitimate invoice. Both failed the first test, then got fixed.

How to use it

Run Standards and Spec as separate subagents, so neither sees the other’s findings, and report them under separate headings.
Give the Spec reviewer the authoritative spec. If you only have commit messages, say so, and swap in the real request the moment you have it.
Treat claims in commit messages and ADRs (“X makes this safe”, “Y owns all retries”) as requirements. They are the cheapest spec you have, and the author’s tests already agree with them.
If you can, review with a different model than the one that wrote the code. It is less likely to share the writer’s blind spots.

Takeaway

A green suite tells you the code does what its author expected. A Spec review by a second agent tells you whether that matches what the author promised. The duplicate writes were in the gap between the two.

Take the last branch your agent wrote, and have a second agent check each “this makes X safe” in its commit messages against the code.

Before building a reference catalog, check whether the attribute reaches the computation

Sun, 21 Jun 2026 21:00:00 +0200

A tax engine for investment accounts has to put every instrument into a category, because the category decides the rules. A real share sold after the holding period can be exempt. A CFD on the same stock is a derivative and is taxed no matter how long you hold it. Brokers don’t make this easy. One reports CFDs under “stocks”. A position can be a fractional share or a whole one. An ETF’s ISIN doesn’t say whether the fund pays out dividends or reinvests them.

During the design session, the coding agent recommended a central instrument catalog: ISIN → {category, asset class, accumulating/distributing, derivative, fractional, …}, seeded from a market data provider and maintained over time. Per-broker adapters would propose a category and reconcile it against the catalog, and anything unresolved would go to a review queue. It argued that the accumulating/distributing flag in particular had to come from the catalog.

I rejected it in two sentences:

If we get the dividend, we have to deal with it. If the fund gets the dividend, it’s non-taxable for the individual.

The attribute decides whether an event exists

A distributing fund pays the investor a dividend, and the broker reports it as a dividend event, which is taxed. An accumulating fund keeps the dividend inside the fund. The investor receives nothing, and the broker reports nothing, so nothing is taxed until the units are sold, and the sale is an ordinary sell event.

The flag the catalog was meant to supply only decides whether an event exists, and the transaction ledger already records whether one did. Removing the flag from the catalog changes no input to the engine. It was also the catalog’s main justification.

The reusable check: for every attribute you are about to maintain as reference data, flip its value and ask which computation input changes.

Attribute	What changes if it flips	Already in the events?	Source
Accumulating / distributing	Whether a dividend event exists	Yes: the event is present or absent	None needed
CFD / real share	The rules applied to the same sell event	No	The broker’s own export (instrument type, leverage)
Fractional / whole share	Whether an exemption is arguable	No	The broker’s idiom
Crypto / token / derivative on crypto	Which rules and caps apply	No	The broker’s idiom

The remaining attributes do change the computation. All of them can be read from the broker that produced the record, so classification belongs in the per-broker adapter. The adapter already parses that broker’s export and knows its quirks. A global catalog would only reconcile against them afterward.

Classify where the source idiom lives, and refuse to guess

The adapter for the broker that labels CFDs as stocks classifies by the type and leverage columns of its own export:

function classify(type: unknown, leverage: unknown): Category | { explanation: string } {
 const kind = header(type);
 if (/cfd|contract for difference/.test(kind)) return "derivative";
 const multiplier = Number(text(leverage).replace(/x$/i, ""));
 if (!kind || !Number.isFinite(multiplier) || multiplier < 1) return { explanation: "Type or Leverage is missing or invalid." };
 if (kind.includes("crypto")) {
 return multiplier === 1 ? "crypto" : { explanation: `Crypto Type conflicts with leveraged position (${text(leverage)}).` };
 }
 if (multiplier > 1 && /(stock|etf|real|cfd|contract)/.test(kind)) return "derivative";
 if (multiplier === 1 && /(stock|etf|real|share|security)/.test(kind)) return "security";
 return { explanation: `Type ${text(type)} and Leverage ${text(leverage)} cannot be classified.` };
}

No default branch picks the most likely category. A row the rules cannot classify becomes a review item (AMBIGUOUS_CATEGORY with the raw row reference) and stays out of the tax computation. A misclassified instrument produces a wrong filing with no warning sign, which is worse than an incomplete one.

The ISIN and ticker stay on the record for cross-checking and audit, but nothing uses them to decide a category.

What dropping the catalog removed

A licensed data feed and the work of keeping up with new listings, delistings, and corrections.
A reconciliation layer between adapter output and catalog values, plus the conflict UI that goes with it.
A global table keyed by ISIN that every user’s trades would look up. In this design, real instrument identity is visible only inside the adapter at ingestion, and the stored record carries a pseudonymous ID. That design is described in When the server must compute on confidential data, split the data by who may read it .

The cost moved into the adapters: classification is now only as correct as each adapter. Every adapter has its own test suite for its broker’s export format, and adding a broker means writing its classification rules and tests.

Caveat: some tax systems tax events that never happened

The shortcut holds where tax follows money the investor actually receives. Some tax systems tax deemed income on fund holdings that paid nothing out. Germany’s Vorabpauschale, UK reporting funds’ excess reportable income and the US PFIC regime are examples. There, the accumulating flag produces an income line that no event carries, and you need the attribute from somewhere. That is exactly the case the flip test catches: the computation input changes even though no event changes.

Takeaway

Before you maintain reference data for an attribute, flip it and see which input of your computation changes. If the only effect is whether an event happens, and your event stream already records that, you don’t need the attribute. If an input does change, get the value from whoever produced the record, and send anything it can’t classify to review.

The agent’s recommendation was internally consistent and sounded responsible. It took someone who knew where the taxable event comes from to see that the attribute changed nothing. When an agent proposes reference data, run the flip test on every attribute before accepting it.

Codex 401 in an agent shell: check credentials before blaming API key precedence

Tue, 05 May 2026 00:00:00 +0000

A regular terminal reported a ChatGPT login, but Codex requests inside an agent’s shell failed with:

HTTP error: 401 Unauthorized

An inherited OPENAI_API_KEY looked suspicious. The useful next step was a controlled credential-context comparison. The recorded investigation never proved that the key overrode the ChatGPT login, and it never showed a successful retry after removing it.

What the May 5 investigation established

The installed CLI was codex-cli 0.128.0. The user reported that codex login status returned Logged in using ChatGPT before launching the agent. Inside the agent, the same status command returned Not logged in, and this request repeatedly failed:

codex exec --skip-git-repo-check "echo hello"

An early environment check found OPENAI_API_KEY present. A separate request to the OpenAI models endpoint, explicitly using that variable, returned HTTP 401. That established that the credential failed that direct probe. It did not establish which credential Codex selected.

Later checks found the variable absent, yet Codex still failed. One retry explicitly supplied an empty OPENAI_API_KEY and also received 401 responses. The chain contains no controlled env -u comparison that restored subscription access. The document-cleanup task continued with Python; Codex authentication was not shown to recover.

Calling this a confirmed case of an environment key shadowing OAuth would overstate the evidence. The login-status mismatch deserved investigation before declaring the subscription token expired or replacing it with an API key.

Compare the launch contexts without exposing secrets

Run these checks locally in both the working terminal and the failing agent shell:

command -v codex
codex --version
codex login status
python3 - <<'PY'
import os
for name in ("HOME", "CODEX_HOME", "OPENAI_API_KEY", "CODEX_API_KEY"):
 print(f"{name}: {'set' if name in os.environ else 'unset'}")
PY

Keep login-status output private if it includes credential details. The Python check prints names and presence only. Do not dump the environment, search credential files, or paste tokens into a troubleshooting conversation.

Compare the actual HOME and CODEX_HOME paths privately, too. Presence alone cannot tell you whether two processes use the same directory. Check the executable and launch configuration for differences before changing authentication.

OpenAI’s authentication documentation distinguishes ChatGPT subscription sign-in from API-key access. It also describes credential storage in a file under CODEX_HOME or an OS credential store. A profile HOME mismatch or unavailable credential store is a separate explanation for a missing login.

To isolate the suspected variable on a Unix-like system, keep the same executable, working directory, HOME, and CODEX_HOME, then compare:

codex login status
env -u OPENAI_API_KEY codex login status

Removing one variable affects only the second process. It does not delete the saved login or change the parent shell. If you intend to make a model request, repeat the same minimal prompt with and without that variable and compare completion and exit status. A status check alone does not prove a request will succeed. This paired request test was not completed in the May investigation.

Verification on October 31: avoid a universal precedence rule

A separate check on 2026-10-31 used installed Codex 0.166.0 with isolated, empty HOME and CODEX_HOME directories. codex login status returned Not logged in both without a key and with a dummy OPENAI_API_KEY. No model request or real credential was used. This checks status behavior without a saved login; it does not test precedence over OAuth.

The upstream authentication loader , inspected that day, has an environment-key precedence branch for CODEX_API_KEY, conditional on that mechanism being enabled and API-key authentication being allowed. That is not a universal rule about OPENAI_API_KEY, nor proof of what version 0.128.0 selected in May.

The earlier trusted-directory error was separate: --skip-git-repo-check bypassed that check and exposed the authentication failure. Initializing unrelated directories as Git repositories would not repair credentials.

Before logging out or deleting a working login, compare the credential context of your two launch paths and change one variable at a time.

Tilde pointed at a profile home, so the export landed elsewhere

Mon, 04 May 2026 00:00:00 +0000

The export command finished, but the expected directory was empty. Then Python reported [Errno 2] No such file or directory for a script that existed in the user’s home. The runner’s HOME pointed at a profile-specific directory. Fixing this required explicit paths for both the export destination and the Python environment.

Two homes, two plausible directory trees

An initial attempt to read a helper resolved to a profile-home path where the file did not exist. During the later export, cd ~/transfer succeeded because that directory existed too. A virtual environment had been created there, but the actual transfer script lived under the user’s expected home.

The terminal output settled the discrepancy: printing HOME returned the profile-specific home. Comparing the two transfer directories found the script only under the expected user-home root. Comparing output directories found exported data under that root and an empty directory under the profile home. The report also existed only under the expected root.

Changing directories alone would still leave Python’s Path("~/data").expanduser() following the process’s HOME. The repair replaced tilde-based output defaults with explicit absolute destinations, including separate report and log paths. The script supported overrides such as KB_OUT_ROOT. Running it from the intended transfer directory also selected that directory’s .venv/bin/python.

This was observed in a historical Hermes runner environment. It does not establish that every Hermes profile changes HOME; consult the current Hermes documentation for your setup. A configured profile directory and the home directory you expect are separate facts to check.

Check the process before searching for files

Run these inside the same terminal tool or wrapper that launches the job:

printf 'HOME=%s\n' "$HOME"
id
pwd
python3 -c 'import sys; from pathlib import Path; print(sys.executable); print(Path.home())'

Compare those paths with the expected project root. Limit directory checks to those known roots; a filesystem-wide search adds noise and can expose unrelated private files. A successful cd proves only that a directory exists, not that it contains the intended script or environment.

Shell expansion happens before the child starts

In Bash, unquoted ~/data expands in the calling shell. In env HOME=/somewhere python3 ... ~/data, the argument has already expanded before env changes the child’s environment. Single-quoted '~/data' stays literal; Python must expand it explicitly. On Unix, Path.expanduser() uses HOME when it is set.

This isolated example demonstrates both cases without touching real home directories:

SCRATCH="${TMPDIR:?Set TMPDIR to your scratch directory}/home-path-demo"
mkdir -p "$SCRATCH/caller-home" "$SCRATCH/profile-home"
PYTHON="$(command -v python3)"
export SCRATCH PYTHON
HOME="$SCRATCH/caller-home" bash --noprofile --norc <<'SH'
env HOME="$SCRATCH/profile-home" "$PYTHON" -c \
 'import sys; print(sys.argv[1])' ~/data
env HOME="$SCRATCH/profile-home" "$PYTHON" -c \
 'import sys; from pathlib import Path; print(sys.argv[1]); print(Path(sys.argv[1]).expanduser())' '~/data'
SH

The first command prints the caller’s home path. The second prints the literal ~/data, then the profile-home path.

Pin the interpreter and verify the artifact

Here is a standalone local inventory example. The program reads its input directory as read-only; the output root is explicit. It requires Python’s venv module and uses only scratch fixtures.

SCRATCH="${TMPDIR:?Set TMPDIR to your scratch directory}/home-path-demo"
mkdir -p "$SCRATCH/source" "$SCRATCH/exports"
printf 'fixture\n' > "$SCRATCH/source/sample.txt"
python3 -m venv --without-pip "$SCRATCH/venv"
export DATA_ROOT="$SCRATCH/source" KB_OUT_ROOT="$SCRATCH/exports"
"$SCRATCH/venv/bin/python" <<'PY'
import os
import sys
from pathlib import Path

source = Path(os.environ["DATA_ROOT"])
out = Path(os.environ["KB_OUT_ROOT"])
if not source.is_absolute() or not out.is_absolute():
 raise ValueError("Use absolute input and output roots")
artifact = out / "inventory.txt"
artifact.write_text("\n".join(sorted(p.name for p in source.glob("*.txt"))) + "\n")
assert artifact.read_text() == "sample.txt\n"
print("interpreter:", sys.executable)
print("verified artifact:", artifact.resolve())
PY

For a real export, validate expected contents at the selected destination and check the report there. In the failed run, a pipeline ending in tail returned zero even though Python could not open the script. Command status alone missed both the launch failure and the path mismatch. Check your job’s resolved interpreter and output root before its next run.