The model transport and the bounded agent loop

The familiar agent loop is short: call a model, execute its tools, append the results and repeat. The difficult part is deciding exactly what each of those verbs means under streaming, failure and a deadline.

The client runtime kept the loop independent of provider-specific events. A transport adapter translated provider output into a small internal vocabulary. The loop could then focus on conversation progression and tool execution instead of carrying every vendor’s streaming protocol.

A narrow transport contract

A compact provider-neutral vocabulary can represent text deltas, tool-call starts, tool-argument deltas and completion with usage. Vendor-specific adapters translate into that vocabulary; the rest of the runtime consumes the semantic contract.

from dataclasses import dataclass

@dataclass(frozen=True)
class ToolStart:
    call_id: str
    name: str

@dataclass(frozen=True)
class ToolArgsDelta:
    call_id: str
    fragment: str

A provider adapter must preserve call IDs, tool names, complete argument content, stop reasons and token accounting. “Compatible JSON” is not enough if cache usage disappears or two concurrent tool calls become indistinguishable.

The local dispatcher selects an adapter from the resolved provider. Adding a provider means satisfying this stream contract and its tests; the orchestrator should not learn a new set of vendor event names.

Assemble arguments before executing anything

Tool arguments arrive as fragments. A fragment can split a JSON key, a quoted string or a number. It is not executable intent until the relevant call is complete and the accumulated JSON has been parsed and validated.

import json

def parse_call(fragments):
    value = json.loads("".join(fragments))
    if not isinstance(value, dict):
        raise ValueError("tool arguments must be an object")
    return value

assert parse_call(['{"guest', 's":', '4}']) == {"guests": 4}

The call ID is the assembly key. Position alone is unreliable when streams interleave. The result must later be appended with the same call ID so the next model turn can associate each result with the request that produced it.

Malformed arguments should become a structured failure the model can respond to. They should not silently execute as an empty object, particularly for a write tool with defaults.

The loop owns conversation progression

For each model turn, the loop records the request, consumes the stream and accumulates usage. If executable tools were requested, it calls the executor and appends the assistant tool-call message followed by the tool results. If a tool requires approval, the loop returns a waiting outcome. If the model produces a final answer, the loop applies the relevant checks and returns.

The implementation also distinguishes presentation tools. A request to render a component is projected to the UI rather than sent to a backend action handler. That distinction prevents a UI card from being treated as a completed business action.

A model response, an attempt and a run are not interchangeable units. One attempt can contain several model turns. Each turn can request multiple tools. A run can end awaiting approval without having performed the requested mutation.

Bound every source of repetition

The attempt has an iteration limit, output-token constraints and an overall deadline. The loop also bounds repair behaviour. It can ask for a correction when a response claims an unsupported action, when booking facts lack backing or when the model returns an empty answer. These repair passes consume iterations and time; they do not form an unlimited second loop.

Use one shared repair budget across failure classes. Allowing one retry for an empty response, another for an unsupported claim and another for formatting can turn a nominal “one retry” policy into several paid calls. Track both the reason and the total repair work remaining.

When the iteration cap is reached, the product returns a graceful fallback and emits a budget warning. That prevents an uncontrolled loop, but the task remains unresolved. The evaluation layer must treat that warning as an outcome signal rather than counting a polite apology as successful task completion.

Generated text is not automatically deliverable text

The model can begin with “I’ll check that for you” and then request a data tool. Streaming that sentence is often harmless. Streaming “I have cancelled your booking” before the cancellation tool succeeds is not.

The runtime separates raw model text events from user-visible text events. It buffers early narration until it knows whether the turn is answering or entering a data/action step. Presentation calls and executable calls receive different treatment.

This improves the experience, but there is a deeper constraint: a later guard cannot retract bytes already sent to a browser. A synchronous messaging response can be shaped using the final result; an SSE client may already have seen earlier deltas. For claims whose correctness depends on completed actions, the publication boundary must be after the evidence check.

That is a useful design extension for any streamed agent: maintain a draft stream for internal observability and a separately committed answer stream for claims that require verification. Do not assume that checking final_text sanitises every earlier event.

A claim check needs successful evidence

Suppose the model says, “The host has been notified.” Finding a tool named escalate_to_host in the transcript is insufficient. The call may have failed, created only an in-app item or returned no delivered notification channels.

A reliable check uses the tool’s result semantics. Did the action complete, was it merely proposed, and what evidence supports the wording? Typed action receipts make those distinctions explicit. A name or regular-expression match can be a useful warning signal, but it cannot establish that an external action succeeded.

{
  "action": "escalate",
  "record_created": true,
  "notification_status": "not_delivered",
  "allowed_claim": "The issue has been flagged for the host."
}

The receipt is a reference contract. It demonstrates how the business outcome can constrain language without asking another model to guess whether the first model was truthful.

Cancellation crosses every adapter

Cancelling the caller should propagate through the lifecycle, transport and tool layer. A blocked provider iterator, an outbound HTTP request and an executor waiting on a semaphore all need to cooperate with cancellation.

A timeout around only the model request leaves tool execution unbounded. A timeout around only the tool handler leaves queue wait unmeasured. An overall attempt deadline limits the combined path, while component deadlines identify where the time went.

The transport tests should include a stream that stops mid-argument, emits multiple tool calls, returns no text, exceeds a deadline and raises after partial output. Happy-path text streaming is the easiest part of the contract.

Keep the gateway path visible

A gateway can centralise provider access, attribution and budget enforcement. If a design permits direct-provider fallback, that path needs equivalent access controls, metering and spend limits. Otherwise a successful fallback can preserve the answer while losing the accounting and policy guarantees that were assumed at the gateway.

A healthy provider response can therefore coexist with an unhealthy accounting path. Readiness and operational dashboards need to report the route actually used, not simply whether a provider credential exists.

The loop becomes manageable when these boundaries are explicit: providers stream model events, the executor owns actions, the lifecycle owns deadlines, and delivery consumes only the events appropriate for the channel.