Observability on Lessons learned

A latest-row query cannot tell you whether every device is current

Thu, 29 Oct 2026 00:00:00 +0000

A dashboard can label an account “current” while that account still has an outdated device. Selecting the latest event per account and application retains one observation; it discards observations from other devices. Describe that selection in the report before claiming that the source telemetry cannot distinguish devices.

The distinction matters during a security review: “latest observed state” and “any observed device needs attention” answer different questions. Correct arithmetic cannot repair a mislabeled metric.

Reproduce the lost device with synthetic data

Run this example with Python 3 and SQLite supporting window functions. The names, dates, and status values are synthetic; current is a supplied classification, not an implementation of version comparison.

import sqlite3

con = sqlite3.connect(":memory:")
con.executescript("""
CREATE TABLE observations (
 event_id INTEGER PRIMARY KEY,
 account TEXT, app TEXT, device TEXT,
 observed_at TEXT, current INTEGER
);
INSERT INTO observations VALUES
 (1, 'account-a', 'demo', 'laptop', '2026-01-01T09:00:00Z', 0),
 (2, 'account-a', 'demo', 'phone', '2026-01-01T10:00:00Z', 1),
 (3, 'account-b', 'demo', 'tablet', '2026-01-01T10:00:00Z', 0),
 (4, 'account-b', 'demo', 'tablet', '2026-01-02T10:00:00Z', 1);
""")

latest_account = """
WITH ranked AS (
 SELECT *, ROW_NUMBER() OVER (
 PARTITION BY account, app
 ORDER BY observed_at DESC, event_id DESC
 ) AS rn
 FROM observations
)
SELECT account, current FROM ranked WHERE rn = 1 ORDER BY account
"""

latest_devices = """
WITH ranked AS (
 SELECT *, ROW_NUMBER() OVER (
 PARTITION BY account, app, device
 ORDER BY observed_at DESC, event_id DESC
 ) AS rn
 FROM observations
)
SELECT account, MAX(CASE WHEN current = 0 THEN 1 ELSE 0 END)
FROM ranked WHERE rn = 1
GROUP BY account, app ORDER BY account
"""

account_rows = con.execute(latest_account).fetchall()
exposure_rows = con.execute(latest_devices).fetchall()
assert account_rows == [('account-a', 1), ('account-b', 1)]
assert exposure_rows == [('account-a', 1), ('account-b', 0)]
print("latest account state:", account_rows)
print("any latest-observed device outdated:", exposure_rows)

Both accounts have a current latest observation. Only account-a has an outdated device among its latest per-device observations. The laptop disappeared from the first query because the phone reported later.

The account-b rows are a negative control. Aggregating every historical outdated event would flag the tablet even after its newer current observation. Rank within each device first, then aggregate the surviving states.

State the measurement contract

An accurate table label might be:

Accounts classified by their latest observed event per application, with ties resolved by event ID.

That supports a claim about selected observations. It does not establish that every device on those accounts is current. Nor does the query establish what the ingestion system could identify before aggregation.

For an exposure report, define the device identity and observation window explicitly. A stable device key is a prerequisite for the second query. An installation identifier that rotates after reinstalling may represent installations rather than physical devices; a shared or missing identifier can merge unrelated observations. Inspect the schema and collection semantics before choosing the partition keys.

The example orders normalized UTC timestamps and breaks ties with a unique event ID. That makes the selection deterministic. It does not prove that a larger ID represents a later real-world event when timestamps tie. SQLite’s window-function documentation describes how ROW_NUMBER() follows the window’s ordering.

Keep unknown devices out of the healthy bucket

Even latest-per-device aggregation describes observed devices. Retired devices, missing telemetry, and a stale last observation need an explicit policy. Define freshness before calling the result a current fleet inventory, and report unknown states separately instead of treating silence as evidence of health.

A reporting correction may leave every displayed count unchanged: the query can stay the same while its title and interpretation become accurate. Calculating a replacement device-level metric is separate work and requires source data that supports it.

Audit the partition keys behind one account-level health table, then check whether its title describes exactly the rows that survive.

Cyber classifiers can interrupt offline pentest summaries

Fri, 09 Oct 2026 00:00:00 +0000

An agent spent three minutes reading local transcripts from a finished, authorized security exercise, then stopped with:

This request triggered cyber-related safeguards.

No live target was involved. The job was offline summarization, yet the classifier interrupted it before it could write the requested report. The practical fix was to treat security-history analysis as a resumable data pipeline: retain message provenance, checkpoint extracted evidence, and keep the handoff usable by another approved model or provider.

A flag-shaped string is not a captured flag

The first problem appeared before the classifier fired. A search through about 25 request dumps found dozens of flag-shaped strings. About 20 were spelling, capitalization, spacing, or hyphenation variants of one value.

Most variants appeared only in assistant reasoning and progress notes. The model repeatedly retyped a remembered value and changed it. A checkpoint then copied one mutation into later sessions.

Only strings preserved in tool results represented observed server output. To separate evidence from generation, use this origin-based distinction:

user request instruction or supplied context
assistant text generated claim; unverified
assistant reasoning generated hypothesis; unverified
tool call attempted action
tool result observed output from that action
provider error evidence about the run, not the target

This applies beyond CTF flags. Treat IDs, hashes, error codes, hostnames, and quoted responses in model prose as leads until the first tool result corroborates them. Exact-looking text is still generated text.

Offline processing still triggers cyber classifiers

The retained run used Opus 5.5. It parsed local request dumps, separated real and decoy-looking values, and extracted an existing SMB investigation summary. After that, the next model response was refused under the provider’s real-time cyber safeguard. The session recorded a cyber refusal and pointed to its Cyber Verification Program. The output file did not exist afterward.

Some source dumps also contained earlier cyber_policy failures from another provider along with a prompt to apply for Daybreak access. Because these records were historical input, they still increased the concentration of offensive-security language the summarizer saw.

A related authorized test had a similar shape: generate enough traffic against a non-production environment to trigger a WAF, then verify that the application displays and retries a CAPTCHA correctly. In a newer Opus session, the controlled load-test work was refused. The retained successful session used Opus 4.6 and built the test client, while a later session corrected its cookie handling and the request used to observe the challenge.

The awkward lesson is that task intent and classifier input are different things. In practice, “Summarize these old logs” can contain exploit commands, credentials, flags, and attack narratives. Likewise, “Test the CAPTCHA” can require traffic that resembles denial of service when reduced to request counts.

Make the audit resumable and model-neutral

Do deterministic work before asking a model to synthesize:

Parse each transcript into typed events and preserve the source file, timestamp, role, tool-call ID, and tool-result ID.
Extract exact-match artifacts only from tool results. Store assistant claims separately as unverified leads.
Write incremental JSON or Markdown after each source file. Never leave the only useful state in the model’s context window.
Produce a sanitized handoff containing the authorization scope, completed files, pending questions, and evidence pointers.
Verify the final report exists and reconcile its artifact counts against the extraction index.

Provider access programs are worth the setup when this work recurs. In my experience, GPT Daybreak Blue justified the hassle for authorized security-related tasks. Kimi, GLM, and older model versions have also completed useful work when a newer classifier blocked the same class of jobs. That is a fallback, not a bypass: preserve the same authorization boundary, do not weaken the requested scope, and use models approved for the work.

Before trusting your next transcript summary, choose one exact token from the report and trace it back to the tool result that observed it.

max_retries_exhausted in a scheduled agent: alert on overdue outcomes, not request dumps

Fri, 25 Sep 2026 00:00:00 +0000

A scheduled agent produced 67 failed-request dumps over roughly two months, yet the archive contains no successful-turn record. The failure reports ended with:

max_retries_exhausted
non_retryable_client_error

Saving the request made diagnosis possible later, but it did nothing to alert an operator to the failed work. To surface unattended jobs, add an independent watcher that alerts when a verified outcome becomes overdue, even if the agent cannot produce a response.

What the failure archive actually proves

The records span five error shapes: a provider-side 503 circuit breaker, a broken pipe, an access-verification service error, a quota rejection, and an invalid-credential response. Counting the dump metadata gives 66 retry-exhaustion failures and one non-retryable client error.

That is evidence of repeated failed requests. It cannot establish that every scheduled invocation failed: successful work might have been recorded elsewhere, and a request can fail after earlier tool actions succeeded. Nor does the absence of a notification in a dump prove that no external monitor existed.

The operational problem exposed by this archive is narrower and useful: failure artifacts alone cannot answer, “When did the job last finish its work?” Because this corpus contains no success receipt to reconcile the errors, it demonstrates no monitoring repair.

Watch the outcome from outside the agent

For each scheduled job, define a durable success receipt tied to its actual acceptance condition. For a capture job, that might mean the destination readback confirms the captured items. A model response saying “done” is not sufficient.

Record at least a run ID, start time, terminal status, and the last verified success time. Keep partial progress separate so a retry can reconcile completed writes instead of duplicating them.

Then run a watcher independently of the model call:

Alert on a terminal failure, including a normalized error class and a run ID.
Alert when the last verified success exceeds the allowed interval plus a bounded grace period.
Treat a missing first success as overdue too. A watcher that only checks an existing timestamp can overlook a job that never worked.
Send the alert through a channel that does not require the failing model provider.

Deduplicate repeated alerts for the same incident, but keep an overdue-success alert active until someone acknowledges or resolves it. If alert delivery fails, track that failure as its own observable status.

Classify before retrying

The archive’s final credential rejection was classified as non-retryable. That classification is a reason to stop automatic retries and request an authentication/configuration check. It does not prove exactly which configuration change caused the rejection.

A broken pipe or service outage can justify bounded retries. Quota failures may require waiting for a published reset time. Preserve the specific provider error class: an HTTP status alone does not show whether the next attempt can succeed.

Pausing should produce an explicit incident state with a resume condition. Otherwise, “automatically disabled” becomes another silent failure.

Test the monitor with deliberately broken runs

In a disposable job, inject a retry-exhaustion failure and a non-retryable credential failure. Confirm both reach the independent alert channel. Suppress the runner entirely and confirm that the overdue-success check still fires. Finally, make the model report success without completing the destination write; that must not refresh the success timestamp.

Keep raw dumps access-controlled and retention-limited. They can contain the complete outgoing prompt and operational context; send a redacted summary and a restricted evidence reference in alerts.

Before leaving a job unattended, deliberately prevent one run from starting and check whether anyone notices.

Hourly agent cron vs. '429 usage_limit_reached': persist the provider's quota reset

Sat, 19 Sep 2026 00:00:00 +0000

An hourly scheduled agent repeatedly received 429 usage_limit_reached while its provider response already supplied a future reset time. One recorded delay was resets_in_seconds: 202825, about 2.35 days. The scheduler needed a durable not-before deadline checked before making another model call.

A failure archive contained 62 quota-related failed-request dumps, plus five failures of other kinds. Those are dump counts, not counts of individual HTTP attempts: a single invocation could retry several times before saving a dump. The archive does not establish that every scheduled invocation failed, and it contains no demonstrated scheduling repair or successful recovery.

A cron tick should consult shared cooldown state

A retry loop inside one invocation cannot govern the next invocation. When a process exits, an in-memory backoff disappears; the next cron tick can start the same futile sequence.

Persist the cooldown at the scope of the exhausted quota, such as the provider account and quota bucket. Every worker sharing that bucket must consult it before the first request. A different job name does not create a different quota.

The HTTP Retry-After specification permits either an HTTP date or a delay in seconds. Provider JSON fields such as resets_at and resets_in_seconds are provider-specific; normalize them in an adapter rather than assuming their units or presence. Use receipt time for relative delays. If multiple applicable constraints exist, do not schedule before any of them.

An illustrative gate, after parsing and validating those fields:

import math


def next_probe(existing, received_at, *, retry_after_at=None,
 quota_reset_at=None, margin=5):
 deadlines = [x for x in (retry_after_at, quota_reset_at)
 if x is not None]
 if not deadlines:
 raise ValueError("no usable deadline; use bounded fallback policy")
 values = [existing, received_at, margin, *deadlines]
 if not all(math.isfinite(x) and x >= 0 for x in values):
 raise ValueError("invalid deadline")
 return max(existing, max(received_at, *deadlines) + margin)


def may_probe(now, not_before):
 return now >= not_before


# Synthetic clock values; these are not recorded provider timestamps.
deadline = next_probe(0, 1000, quota_reset_at=1000 + 202825)
assert deadline == 203830
assert not may_probe(1000 + 3600, deadline)
assert not may_probe(deadline - 1, deadline)
assert may_probe(deadline, deadline)
# A later response cannot shorten an existing cooldown.
assert next_probe(deadline, 2000, retry_after_at=3000) == deadline
# Both applicable constraints survive normalization.
assert next_probe(0, 1000, retry_after_at=5000,
 quota_reset_at=9000) == 9005
print("cooldown boundary assertions passed")

The gate checks timing only. Production code also needs atomic storage updates, a lock or lease to admit one recovery probe, and clock-skew handling. On restart, read the saved deadline; do not silently treat unreadable state as permission to call. Add bounded jitter after the deadline so workers do not all resume together.

Give long waits an operational policy

A near-term transient rate limit can justify a short bounded retry. A multi-day quota exhaustion should explicitly defer the job, with its next eligible probe time visible to an operator. If a provider advertises an implausibly distant reset, escalate instead of silently sleeping forever or truncating the wait and hammering the endpoint.

At the deadline, admit one probe. A new quota rejection updates the cooldown. An authentication rejection needs intervention rather than the same timed retry. A permitted model response still does not prove the scheduled task completed: reconcile destination writes before replaying work that may have partially succeeded.

Test across process boundaries

The executable example verifies the early, boundary, and monotonic-deadline cases. To verify a real scheduler, inject a quota response, terminate the worker, and start a new invocation before reset. Assert that no provider call occurs. Start competing workers at reset and assert that only one obtains the recovery lease. Then inject another quota rejection and confirm that it persists the next deadline.

Stopping repeated calls can hide an overdue job unless monitoring remains independent of the cooldown. The companion post on unattended failure alerts covers that separate requirement. The disappearance of failure dumps after a reset is not a success receipt.

Before the next scheduled run, check whether a provider reset deadline survives a worker restart.