Hourly agent cron vs. '429 usage_limit_reached': persist the provider's quota reset

An hourly scheduled agent repeatedly received 429 usage_limit_reached while its provider response already supplied a future reset time. One recorded delay was resets_in_seconds: 202825, about 2.35 days. The scheduler needed a durable not-before deadline checked before making another model call.

A failure archive contained 62 quota-related failed-request dumps, plus five failures of other kinds. Those are dump counts, not counts of individual HTTP attempts: a single invocation could retry several times before saving a dump. The archive does not establish that every scheduled invocation failed, and it contains no demonstrated scheduling repair or successful recovery.

A cron tick should consult shared cooldown state #

A retry loop inside one invocation cannot govern the next invocation. When a process exits, an in-memory backoff disappears; the next cron tick can start the same futile sequence.

Persist the cooldown at the scope of the exhausted quota, such as the provider account and quota bucket. Every worker sharing that bucket must consult it before the first request. A different job name does not create a different quota.

The HTTP Retry-After specification permits either an HTTP date or a delay in seconds. Provider JSON fields such as resets_at and resets_in_seconds are provider-specific; normalize them in an adapter rather than assuming their units or presence. Use receipt time for relative delays. If multiple applicable constraints exist, do not schedule before any of them.

An illustrative gate, after parsing and validating those fields:

import math


def next_probe(existing, received_at, *, retry_after_at=None,
               quota_reset_at=None, margin=5):
    deadlines = [x for x in (retry_after_at, quota_reset_at)
                 if x is not None]
    if not deadlines:
        raise ValueError("no usable deadline; use bounded fallback policy")
    values = [existing, received_at, margin, *deadlines]
    if not all(math.isfinite(x) and x >= 0 for x in values):
        raise ValueError("invalid deadline")
    return max(existing, max(received_at, *deadlines) + margin)


def may_probe(now, not_before):
    return now >= not_before


# Synthetic clock values; these are not recorded provider timestamps.
deadline = next_probe(0, 1000, quota_reset_at=1000 + 202825)
assert deadline == 203830
assert not may_probe(1000 + 3600, deadline)
assert not may_probe(deadline - 1, deadline)
assert may_probe(deadline, deadline)
# A later response cannot shorten an existing cooldown.
assert next_probe(deadline, 2000, retry_after_at=3000) == deadline
# Both applicable constraints survive normalization.
assert next_probe(0, 1000, retry_after_at=5000,
                  quota_reset_at=9000) == 9005
print("cooldown boundary assertions passed")

The gate checks timing only. Production code also needs atomic storage updates, a lock or lease to admit one recovery probe, and clock-skew handling. On restart, read the saved deadline; do not silently treat unreadable state as permission to call. Add bounded jitter after the deadline so workers do not all resume together.

Give long waits an operational policy #

A near-term transient rate limit can justify a short bounded retry. A multi-day quota exhaustion should explicitly defer the job, with its next eligible probe time visible to an operator. If a provider advertises an implausibly distant reset, escalate instead of silently sleeping forever or truncating the wait and hammering the endpoint.

At the deadline, admit one probe. A new quota rejection updates the cooldown. An authentication rejection needs intervention rather than the same timed retry. A permitted model response still does not prove the scheduled task completed: reconcile destination writes before replaying work that may have partially succeeded.

Test across process boundaries #

The executable example verifies the early, boundary, and monotonic-deadline cases. To verify a real scheduler, inject a quota response, terminate the worker, and start a new invocation before reset. Assert that no provider call occurs. Start competing workers at reset and assert that only one obtains the recovery lease. Then inject another quota rejection and confirm that it persists the next deadline.

Stopping repeated calls can hide an overdue job unless monitoring remains independent of the cooldown. The companion post on unattended failure alerts covers that separate requirement. The disappearance of failure dumps after a reset is not a success receipt.

Before the next scheduled run, check whether a provider reset deadline survives a worker restart.