← Series contents · Complete reading view
The run contract: orchestration, lifecycle and idempotency
A messaging webhook can arrive twice. A caller can time out while the worker is still running. A process can be cancelled after the model has requested a tool but before the reply is delivered. These are ordinary distributed-system events. An agent runtime needs ordinary, explicit semantics for them.
The client platform organised execution around four identities: tenant, session, run and attempt. Each answers a different question. The tenant defines the account boundary. The session groups conversation continuity. The run represents handling one admitted event. The attempt identifies an execution envelope within that handling.
Put a stable contract at the front door
Different entry points carry different payloads: chat text, a messaging conversation, a task-status event or an approval decision. Adapters normalise these into an inbound event with authenticated tenant and actor identity, a source, channel information, text and relevant domain context.
The model never chooses the tenant. Nor should a conversation identifier alone become an authorization boundary. A session identity must include the dimensions that determine who can share state. Two accounts using the same external thread identifier must not share a session.
{
"tenant_id": "tenant-example",
"event_id": "message-104",
"idempotency_key": "messaging:message-104",
"source": "messaging",
"actor_type": "guest",
"actor_id": "guest-example",
"channel": "whatsapp",
"channel_conversation_id": "thread-example",
"text": "Is there any update on the repair?"
}
This example puts identity ahead of routing, retrieval and tool execution. Validate identifiers at ingress and derive account scope from the authenticated caller. An external conversation identifier remains a correlation value; it does not grant access to a conversation or its records.
Claim the inbound event before paid work
Admission resolves identity and session before applying the idempotency gate ahead of classification. That ordering matters. If classification can call a model, deduplicating after classification still spends money on duplicate deliveries.
The durable gate uses a unique scope over tenant, idempotency scope and key. It stores a request hash alongside the claim. A duplicate with the same content can replay or refer to the existing run. The same key with different content is a conflict, not an invitation to return an unrelated response.
The following standalone SQL example expresses the admission rule:
INSERT INTO example_event_claims
(tenant_id, scope, key, request_hash, resource_id, status)
VALUES
(:tenant, 'inbound', :key, :hash, :run_id, 'in_progress')
ON CONFLICT (tenant_id, scope, key) DO NOTHING
RETURNING resource_id;
The claim and initial run creation belong in the same database transaction. Otherwise a crash can leave a claim pointing to a run that was never created. PostgreSQL’s conflict handling supplies the atomic insertion primitive; the surrounding transaction and application state rules supply the protocol. PostgreSQL INSERT
This is inbound deduplication. It is not a proof of exactly-once business side effects. Tool writes and downstream projections need their own identities, which we examine in the reliability chapter.
A duplicate is a state-dependent decision
| Existing state | Appropriate behaviour |
|---|---|
| Completed, replay still valid | Return the stored outcome without another model call |
| Running | Report or attach to the existing work; do not create another owner |
| Failed or aborted and retryable | Claim a fresh run using a guarded transition |
| Same key, different request hash | Reject the conflicting request |
| Replay outside its permitted window | Apply the explicit freshness policy |
A retry protocol can rebind a claim for retryable terminal work using a compare-and-swap against the previous owner. Competing retries must not both become the new owner. Refresh the lease or claim timestamp when ownership changes, and preserve the old run as lineage. Otherwise a retried request may appear stale immediately, or its original failure can disappear from the audit history.
An event ID from the channel provider is preferable to a weak fallback derived from text. “Yes” is a valid message today and next week. A hash of the visible text alone cannot distinguish them.
Route and assemble one attempt
After admission, the router produces a model choice, budgets, skill selection and tool allowlist. The context builder combines the resolved persona, live facts, knowledge and conversation state into model input. The orchestrator creates an attempt containing those values and the request-local event sink.
Resolving the model before the loop also makes attribution clearer. Usage for that attempt can be associated with the selected model. If a future design switches models during a run, usage must be stored by model turn rather than pricing the entire total with the final model’s rate.
An attempt carries both the flat system prompt and structured system blocks for provider-specific cache handling. It also carries messages, tool specifications, policy, delivery context and budgets. These are execution inputs, not hidden mutable properties on the harness.
Prepare, send, resolve
The lifecycle envelope validates the attempt, emits its start event, runs the harness under a deadline and classifies the returned outcome. Expected failure and unexpected exception are different paths.
For example, a runner can return waiting_for_approval as a valid outcome. A provider timeout can become a classified timed-out result. An unexpected exception becomes a failed result so that the orchestrator can finalise the run instead of abandoning a running row.
Cancellation deserves separate treatment:
import asyncio
async def execute_with_deadline(runner, attempt, sink, seconds):
try:
return await asyncio.wait_for(
runner.run(attempt, sink), timeout=seconds
)
except asyncio.CancelledError:
# The caller must still be able to stop the task.
raise
The full envelope records cancellation before propagating it. Catching every exception and converting cancellation into an ordinary response can leave work running after the caller believes it stopped. Conversely, propagation without finalisation can strand state. The lifecycle and orchestrator share that responsibility deliberately.
Finalisation is part of correctness
Finalisation stores the result summary, prices captured usage, records domain-specific state updates and emits the terminal run event. Its failure modes deserve tests just as much as tool execution does.
A particularly awkward case is a failure after claiming the event but before context assembly completes. If the run remains running, retries can keep finding the same unusable record. The orchestrator’s failure envelope must cover routing and context building, not only the model loop.
The runtime also records stage durations: routing, context construction, attempt execution, finalisation and total time. This makes a slow call diagnosable without assuming the model caused it.
Runtime completion is not business completion
An iteration budget can be exhausted and still produce a graceful answer. The runtime may classify that as completed because it returned a response successfully. The business request can still be unresolved.
The same distinction appears with approvals and asynchronous workflows. A workflow being accepted is a valid conversational outcome, but the work is not finished. Evaluation and analytics need both the runtime status and the task outcome, including warnings such as budget exhaustion.
Failure drills that expose the contract
Useful tests exercise the same event twice, a conflicting hash, a routing exception after admission, a timeout during tool execution, and cancellation during finalisation. They assert stored states and emitted events, not only the returned text.
For a retry, inspect whether the original run remains available for audit and whether the new run has clear lineage. For concurrent duplicates, prove there is one winner at the database boundary. For a stale replay, prove that the freshness policy does not accidentally replay an old guest answer into a new conversation turn.
These contracts make the rest of the system easier to reason about. Once admission, identity and terminal states are explicit, the model loop can concentrate on its own bounded job.