max_retries_exhausted in a scheduled agent: alert on overdue outcomes, not request dumps

·3 mins

A scheduled agent produced 67 failed-request dumps over roughly two months, yet the archive contains no successful-turn record. The failure reports ended with:

max_retries_exhausted
non_retryable_client_error

Saving the request made diagnosis possible later, but it did nothing to alert an operator to the failed work. To surface unattended jobs, add an independent watcher that alerts when a verified outcome becomes overdue, even if the agent cannot produce a response.

What the failure archive actually proves

The records span five error shapes: a provider-side 503 circuit breaker, a broken pipe, an access-verification service error, a quota rejection, and an invalid-credential response. Counting the dump metadata gives 66 retry-exhaustion failures and one non-retryable client error.

That is evidence of repeated failed requests. It cannot establish that every scheduled invocation failed: successful work might have been recorded elsewhere, and a request can fail after earlier tool actions succeeded. Nor does the absence of a notification in a dump prove that no external monitor existed.

The operational problem exposed by this archive is narrower and useful: failure artifacts alone cannot answer, “When did the job last finish its work?” Because this corpus contains no success receipt to reconcile the errors, it demonstrates no monitoring repair.

Watch the outcome from outside the agent

For each scheduled job, define a durable success receipt tied to its actual acceptance condition. For a capture job, that might mean the destination readback confirms the captured items. A model response saying “done” is not sufficient.

Record at least a run ID, start time, terminal status, and the last verified success time. Keep partial progress separate so a retry can reconcile completed writes instead of duplicating them.

Then run a watcher independently of the model call:

  • Alert on a terminal failure, including a normalized error class and a run ID.
  • Alert when the last verified success exceeds the allowed interval plus a bounded grace period.
  • Treat a missing first success as overdue too. A watcher that only checks an existing timestamp can overlook a job that never worked.
  • Send the alert through a channel that does not require the failing model provider.

Deduplicate repeated alerts for the same incident, but keep an overdue-success alert active until someone acknowledges or resolves it. If alert delivery fails, track that failure as its own observable status.

Classify before retrying

The archive’s final credential rejection was classified as non-retryable. That classification is a reason to stop automatic retries and request an authentication/configuration check. It does not prove exactly which configuration change caused the rejection.

A broken pipe or service outage can justify bounded retries. Quota failures may require waiting for a published reset time. Preserve the specific provider error class: an HTTP status alone does not show whether the next attempt can succeed.

Pausing should produce an explicit incident state with a resume condition. Otherwise, “automatically disabled” becomes another silent failure.

Test the monitor with deliberately broken runs

In a disposable job, inject a retry-exhaustion failure and a non-retryable credential failure. Confirm both reach the independent alert channel. Suppress the runner entirely and confirm that the overdue-success check still fires. Finally, make the model report success without completing the destination write; that must not refresh the success timestamp.

Keep raw dumps access-controlled and retention-limited. They can contain the complete outgoing prompt and operational context; send a redacted summary and a restricted evidence reference in alerts.

Before leaving a job unattended, deliberately prevent one run from starting and check whether anyone notices.