Automation on Lessons learned

max_retries_exhausted in a scheduled agent: alert on overdue outcomes, not request dumps

Fri, 25 Sep 2026 00:00:00 +0000

A scheduled agent produced 67 failed-request dumps over roughly two months, yet the archive contains no successful-turn record. The failure reports ended with:

max_retries_exhausted
non_retryable_client_error

Saving the request made diagnosis possible later, but it did nothing to alert an operator to the failed work. To surface unattended jobs, add an independent watcher that alerts when a verified outcome becomes overdue, even if the agent cannot produce a response.

What the failure archive actually proves

The records span five error shapes: a provider-side 503 circuit breaker, a broken pipe, an access-verification service error, a quota rejection, and an invalid-credential response. Counting the dump metadata gives 66 retry-exhaustion failures and one non-retryable client error.

That is evidence of repeated failed requests. It cannot establish that every scheduled invocation failed: successful work might have been recorded elsewhere, and a request can fail after earlier tool actions succeeded. Nor does the absence of a notification in a dump prove that no external monitor existed.

The operational problem exposed by this archive is narrower and useful: failure artifacts alone cannot answer, “When did the job last finish its work?” Because this corpus contains no success receipt to reconcile the errors, it demonstrates no monitoring repair.

Watch the outcome from outside the agent

For each scheduled job, define a durable success receipt tied to its actual acceptance condition. For a capture job, that might mean the destination readback confirms the captured items. A model response saying “done” is not sufficient.

Record at least a run ID, start time, terminal status, and the last verified success time. Keep partial progress separate so a retry can reconcile completed writes instead of duplicating them.

Then run a watcher independently of the model call:

Alert on a terminal failure, including a normalized error class and a run ID.
Alert when the last verified success exceeds the allowed interval plus a bounded grace period.
Treat a missing first success as overdue too. A watcher that only checks an existing timestamp can overlook a job that never worked.
Send the alert through a channel that does not require the failing model provider.

Deduplicate repeated alerts for the same incident, but keep an overdue-success alert active until someone acknowledges or resolves it. If alert delivery fails, track that failure as its own observable status.

Classify before retrying

The archive’s final credential rejection was classified as non-retryable. That classification is a reason to stop automatic retries and request an authentication/configuration check. It does not prove exactly which configuration change caused the rejection.

A broken pipe or service outage can justify bounded retries. Quota failures may require waiting for a published reset time. Preserve the specific provider error class: an HTTP status alone does not show whether the next attempt can succeed.

Pausing should produce an explicit incident state with a resume condition. Otherwise, “automatically disabled” becomes another silent failure.

Test the monitor with deliberately broken runs

In a disposable job, inject a retry-exhaustion failure and a non-retryable credential failure. Confirm both reach the independent alert channel. Suppress the runner entirely and confirm that the overdue-success check still fires. Finally, make the model report success without completing the destination write; that must not refresh the success timestamp.

Keep raw dumps access-controlled and retention-limited. They can contain the complete outgoing prompt and operational context; send a redacted summary and a restricted evidence reference in alerts.

Before leaving a job unattended, deliberately prevent one run from starting and check whether anyone notices.

Hourly agent cron vs. '429 usage_limit_reached': persist the provider's quota reset

Sat, 19 Sep 2026 00:00:00 +0000

An hourly scheduled agent repeatedly received 429 usage_limit_reached while its provider response already supplied a future reset time. One recorded delay was resets_in_seconds: 202825, about 2.35 days. The scheduler needed a durable not-before deadline checked before making another model call.

A failure archive contained 62 quota-related failed-request dumps, plus five failures of other kinds. Those are dump counts, not counts of individual HTTP attempts: a single invocation could retry several times before saving a dump. The archive does not establish that every scheduled invocation failed, and it contains no demonstrated scheduling repair or successful recovery.

A cron tick should consult shared cooldown state

A retry loop inside one invocation cannot govern the next invocation. When a process exits, an in-memory backoff disappears; the next cron tick can start the same futile sequence.

Persist the cooldown at the scope of the exhausted quota, such as the provider account and quota bucket. Every worker sharing that bucket must consult it before the first request. A different job name does not create a different quota.

The HTTP Retry-After specification permits either an HTTP date or a delay in seconds. Provider JSON fields such as resets_at and resets_in_seconds are provider-specific; normalize them in an adapter rather than assuming their units or presence. Use receipt time for relative delays. If multiple applicable constraints exist, do not schedule before any of them.

An illustrative gate, after parsing and validating those fields:

import math


def next_probe(existing, received_at, *, retry_after_at=None,
 quota_reset_at=None, margin=5):
 deadlines = [x for x in (retry_after_at, quota_reset_at)
 if x is not None]
 if not deadlines:
 raise ValueError("no usable deadline; use bounded fallback policy")
 values = [existing, received_at, margin, *deadlines]
 if not all(math.isfinite(x) and x >= 0 for x in values):
 raise ValueError("invalid deadline")
 return max(existing, max(received_at, *deadlines) + margin)


def may_probe(now, not_before):
 return now >= not_before


# Synthetic clock values; these are not recorded provider timestamps.
deadline = next_probe(0, 1000, quota_reset_at=1000 + 202825)
assert deadline == 203830
assert not may_probe(1000 + 3600, deadline)
assert not may_probe(deadline - 1, deadline)
assert may_probe(deadline, deadline)
# A later response cannot shorten an existing cooldown.
assert next_probe(deadline, 2000, retry_after_at=3000) == deadline
# Both applicable constraints survive normalization.
assert next_probe(0, 1000, retry_after_at=5000,
 quota_reset_at=9000) == 9005
print("cooldown boundary assertions passed")

The gate checks timing only. Production code also needs atomic storage updates, a lock or lease to admit one recovery probe, and clock-skew handling. On restart, read the saved deadline; do not silently treat unreadable state as permission to call. Add bounded jitter after the deadline so workers do not all resume together.

Give long waits an operational policy

A near-term transient rate limit can justify a short bounded retry. A multi-day quota exhaustion should explicitly defer the job, with its next eligible probe time visible to an operator. If a provider advertises an implausibly distant reset, escalate instead of silently sleeping forever or truncating the wait and hammering the endpoint.

At the deadline, admit one probe. A new quota rejection updates the cooldown. An authentication rejection needs intervention rather than the same timed retry. A permitted model response still does not prove the scheduled task completed: reconcile destination writes before replaying work that may have partially succeeded.

Test across process boundaries

The executable example verifies the early, boundary, and monotonic-deadline cases. To verify a real scheduler, inject a quota response, terminate the worker, and start a new invocation before reset. Assert that no provider call occurs. Start competing workers at reset and assert that only one obtains the recovery lease. Then inject another quota rejection and confirm that it persists the next deadline.

Stopping repeated calls can hide an overdue job unless monitoring remains independent of the cooldown. The companion post on unattended failure alerts covers that separate requirement. The disappearance of failure dumps after a reset is not a success receipt.

Before the next scheduled run, check whether a provider reset deadline survives a worker restart.