Trace the business task, then debug the agent

The founder opens the cockpit and sees that a proposal has not reached the customer. The model call completed successfully, the worker is healthy and the connector returned a timeout. Is the quote unsent, sent once, or sent twice?

A dashboard of token usage cannot answer that question. The trace needs to connect the business demand, agent episode, reviewed action, dispatch attempt and provider receipt. I want to debug the customer outcome first and descend into model details only when they explain the failure.

Use several identities deliberately

A demand ID names the business task. A run ID names one agent episode. An action ID names a proposed effect. An operation ID remains stable across retries of that effect. Attempt IDs distinguish individual calls. Provider receipt IDs connect the local record to external evidence.

Do not reuse the HTTP request ID for all of these purposes. A job may span several requests and survive a restart. Conversely, one agent run may propose several actions, each with a different approval and recovery state.

{
  "event": "connector.dispatch_uncertain",
  "demand_id": "opportunity-017/quote",
  "run_id": "run-042",
  "action_id": "action-017",
  "operation_id": "send-017",
  "attempt_id": "attempt-002",
  "subject_version": 8,
  "status": "UNKNOWN",
  "next_step": "RECONCILE",
  "elapsed_ms": 3021
}

The elapsed value is an example event field, not a measured platform latency. The event deliberately omits the customer's message body and credentials. An authorised operator can follow the evidence reference to a restricted store when diagnosis requires the content.

Separate spans, events and business history

A span records a timed operation such as retrieval or provider dispatch. An event records an occurrence such as approval becoming stale. Business history records the durable state transition and why it was allowed. Traces may be sampled or retained for less time; the business ledger must not depend on a sampled trace being present.

OpenTelemetry supplies common tracing conventions, including evolving GenAI conventions for model operations. I would map provider calls to those conventions while keeping business identifiers in a documented application namespace. Pin the instrumentation schema used by a release instead of assuming field names never change. OpenTelemetry GenAI spans.

A useful application namespace might contain company.demand.id, company.action.id, company.loop and company.receipt.status. These are proposed local attributes, not official OpenTelemetry names.

Walk through a lost acknowledgement

The local connector fixture makes the ambiguous case reproducible. It creates the provider effect and then raises a timeout. The local store retains uncertainty. A later call looks up the existing effect instead of resending.

from company_lab.store import Store
from company_lab.connectors import FakeProvider, dispatch

store, provider = Store(), FakeProvider()
payload = {"recipient": "[email protected]", "document": "quote-r3"}
uncertain = dispatch(store, provider, "studio", "send-017", payload, lose_ack=True)
recovered = dispatch(store, provider, "studio", "send-017", payload)
assert uncertain["status"] == "UNKNOWN"
assert recovered["status"] == "CONFIRMED"
assert recovered["provider_id"] is not None
assert len(provider.effects) == 1

The incident timeline should show the original approval, dispatch, timeout, reconciliation and final receipt. It should not rewrite the first attempt as a clean success. Preserving the uncertainty explains why delivery took longer and which recovery mechanism worked.

Measure objectives that match the workflow

For ingestion, measure the age of the newest successfully processed source event and the oldest unprocessed event. For review, measure pending age and founder handling time. For dispatch, measure time from approved intent to confirmed or escalated receipt. For recovery, measure how long uncertainty remains unresolved.

A provisional pilot objective could be “every uncertain send is reconciled or assigned to an operator within fifteen minutes during operating hours.” That is a proposed target to test, not an achieved service level. Define the clock, exclusions and owner before reporting performance against it.

Track error classes separately: provider rejection, stale approval, identity ambiguity, missing evidence, policy denial and infrastructure failure. A policy denial can be correct behaviour. Counting all denials as agent failures would incentivise the wrong change.

Redact before telemetry leaves the boundary

Use allowlisted log fields rather than attempting to remove a few known secret names from arbitrary payloads. Hash or tokenise identifiers where appropriate, restrict access to content-bearing traces and set retention by data class.

Be careful with exception messages: SDK errors can include request bodies or URLs containing sensitive parameters. Capture a normalised error code and a restricted diagnostic reference. The public website should never inherit raw production traces as “realistic examples.”

An incident runbook should be short enough to use

First pause the affected external effect class if duplication or disclosure is possible. Then identify the exact operation and current receipt. Check the provider using the documented reconciliation path. Repair the business state through an explicit transition, not a manual edit that erases history. Finally add a regression fixture and verify the recovery path before resuming.

For a stuck outbox, identify whether the record is malformed, the provider is unavailable or the relay lacks permission. Quarantine a poison event with its evidence while allowing unrelated work to proceed under policy. Repeatedly restarting the entire service can make the queue look active without resolving the cause.

The exercise is to reconstruct the lost-acknowledgement demo using only persisted identities and receipts. If the operator needs to read every model token to determine whether a quote was sent, the observability model is missing a business-level fact.