PART 01

From an AI concierge to an operations system

A guest asking for the Wi-Fi password and a guest reporting water leaking through the ceiling are both sending a message. Operationally, they are very different requests.

The first needs the correct property, the correct access permissions and a short answer. The second needs a record, an urgency assessment, someone responsible for the work, an escalation path and a way to tell the guest what actually happened. A fluent response is useful in both cases. In the second case, fluency without action makes the situation worse.

This series develops the architecture of a custom agentic harness built at Kuzushi Labs for a client in hospitality. The product combined an AI guest concierge with a host and operations copilot. It connected conversations to reservations, property knowledge, availability, maintenance, staff allocation and internal workflows. The ambition was an operating layer over the hospitality business: a way to request work, perform it through existing systems and inspect the result.

The interesting engineering sits between the conversation and that result.

Start with the people who must use the answer

The requirements were organised around two audiences. The guest concierge handled prospective guests and people with existing stays. The host copilot served the people running the operation. They shared domain services, but they needed different permissions, language and presentations.

This table changes the technical brief. “An assistant that knows our properties” becomes several specific contracts. Property knowledge must be retrieved within scope. Availability must come from the calendar. Staff arrival times must come from booked slots. A proposed task must be described as proposed. An escalation record and a delivered notification must remain distinguishable.

The operator is a first-class user of the product. If changing a prompt requires a developer to edit a container, or investigating a bad response requires reading unrelated log files, the product is unfinished for the team responsible for it.

Convert conversations into acceptance criteria

The useful unit of discovery was a scenario with a starting state, a request, allowed actions and a completion condition. Consider this worked scenario:

Starting state: authenticated guest; one current stay; no open plumbing issue.
Guest: “The bathroom tap will not stop running.”

Required:
  identify the current property;
  record the issue;
  find an appropriate available staff member or escalate;
  report only the assignee and time actually returned.

Forbidden:
  invent an arrival time;
  claim a notification was delivered without delivery evidence;
  create another task when an applicable open issue already exists.

Notice that the completion condition is not “the answer sounds helpful.” It is an operational condition that can be tested. Some checks are deterministic: was a task created, under which tenant, with which property? Others need judgement: was the explanation clear and appropriate to the urgency?

That separation later becomes the evaluation strategy. Requirements should produce the first evaluation cases, before prompts are polished. Otherwise the team ends up measuring whatever the implementation happens to do.

Decide what the model is allowed to decide

The model is good at interpreting flexible language, asking a relevant follow-up and explaining an outcome naturally. The application is responsible for identity, permissions, authoritative data and state transitions.

For booking, the model can understand “the quieter one from the earlier dates.” The application must resolve that reference against what was actually shown and recheck availability before offering a checkout link. For complaints, the model can interpret the description and communicate empathetically. The dispatch chain should not depend on the model remembering six mandatory API calls in the correct order.

The resulting division became a recurring design rule: code owns the operational decision and its record; the model explains the evidence available to it. This does not eliminate model judgement. It places that judgement inside explicit boundaries.

Build on the existing system of record

The hospitality backend already owned reservations, listings, tasks, messaging and payments. Recreating those entities inside an agent framework would create conflicting sources of truth.

The custom Python service therefore owned agent execution, agent configuration and agent-specific state. Domain reads and writes went through the backend’s internal API. The backend remained responsible for its own account scoping and business rules. The agent service could retain a reference to a task without becoming the authority on whether that task was complete.

This boundary was important during integration. A tool can return HTTP 200 while the intended business operation has not happened. A task can be proposed for approval without being created. An accepted parameter can be ignored by an endpoint. A workflow can be queued and later fail. Requirement collection had to include those API semantics, not just endpoint names.

Two assistants, many capabilities

The runtime served two roles: a guest concierge and a host operations copilot. Booking, issue reporting, reservation questions and operational analysis were capabilities attached through skills, prompt fragments and tool sets.

A separate agent was not created for every intent. That would multiply histories, handoffs, prompts and failure modes before the product needed them. A capability could usually be added to the existing persona when the audience, authority and conversation continuity stayed the same.

This also made channel behaviour independent of business capability. A host asking through a messaging channel needed concise text. The same host asking through the web application could receive a structured component. The business operation was shared; presentation changed at the edge.

Define the constraints before selecting infrastructure

Several constraints shaped the implementation directly:

• Requests could be delivered more than once, so ingress needed an idempotency boundary before paid model work.
• Messaging callers had their own wait limits, so the runtime needed deadlines appropriate to the calling channel.
• Multiple clients shared the service, so tenant identity had to travel through every important boundary.
• Business operations could take longer than a conversational turn, so workflow handoff needed an explicit accepted-versus-completed contract.
• Prompts and tools changed during delivery, so the running configuration needed inspection, versioning and controlled reload.
• Bad responses needed to become repeatable tests, so the operator portal needed a route from a run to an evaluation case.

These are architecture inputs. They are also useful questions for the first client workshop: which actions need approval, which data can be stale, who owns a failed task, and how does the team know a guest has been informed?

Success has several denominators

For this type of system, “automation rate” is too broad on its own. An answer-only FAQ and a successfully dispatched repair should not count as equivalent work.

The measurement model should separate eligible requests, correctly resolved requests, requests handed to a person, duplicated actions and reopened issues. It should also distinguish a conversation that reached a terminal runtime state from a task that reached the desired business state.

Task success rate = verified successful tasks / eligible tasks
False completion rate = claimed completions without evidence / claimed completions
Cost per successful task = attributable operating cost / verified successful tasks

These definitions are a measurement framework, not reported client results. The demonstrated product outcomes in this series are the implemented capabilities: scoped tools, durable run records, booking and issue state, operator controls and evaluation workflows. Commercial savings require a before-and-after operational dataset with consistent definitions.

The architecture that follows from the problem

Once the requirements are concrete, the need for a harness becomes clear. A model loop alone does not own duplicate delivery, cancellation, tenancy, approval state, prompt versions or audit delivery. Those responsibilities need a home around the loop.

The next article maps that home: the runtime, the control plane, the domain backend and the console. It also explains which ideas were taken from OpenClaw and Hermes, and why the service retained a stateless execution boundary.

My contribution

My work connected the client’s hospitality workflows to the agent architecture and implementation, including the shared runtime, configurable capabilities, operational tool flows and the control portal used to inspect, evaluate and improve behaviour.

PART 02

Designing the harness: architecture, Hermes and OpenClaw

The first architecture question is where responsibility lives when the model gets something wrong. If every concern sits inside one agent object, it becomes difficult to answer. Was the wrong tenant selected? Was the correct tool missing? Did a prompt change reach the running worker? Did the backend reject an action that the model described as successful?

The client’s platform separated these questions into components with explicit contracts. The model conversation was one component. The harness around it made the conversation callable, bounded and observable.

The four product boundaries

The domain backend, built with Rails and PostgreSQL, owned hospitality records and their rules. The Python agent service, built with FastAPI, owned execution, routing, context assembly, tool dispatch and agent configuration. The Next.js admin console exposed prompts, tools, skills, agents, models, playground sessions, evaluations, observability and spend. A model gateway provided a common operational point for provider access and usage attribution.

The guest-facing web product and messaging integrations were separate entry points into that system. They did not each get their own copy of the agent loop. Their inputs were adapted to a shared event contract, and their outputs were projections of the runtime’s events.

The distinction between the domain database and the agent database matters. Durable agent state does not imply that the agent bypasses the application’s business APIs. Agent runs and configuration belong to the agent service; reservations and task ownership remain with the backend.

Harness, orchestrator and loop are different jobs

The orchestrator prepares and finalises a run. It resolves the session, handles the idempotency gate, obtains routing and context, invokes an attempt and records the outcome.

The harness exposes the execution contract for an attempt. The default implementation delegates to an injected runner. It does not hold a guest’s messages or the current event sink on a shared instance.

The iteration loop performs model turns, assembles streamed tool arguments, executes tools, appends results and decides whether another turn is needed.

The lifecycle envelope wraps the harness with validation, timing, events, deadlines and error classification. Keeping it outside the model loop allows the same failure semantics to apply even when the runner changes.

This separation creates useful replacement seams. A fake transport can exercise routing and tool policy without a model bill. A PostgreSQL store can replace an in-memory store without changing the orchestrator. A synchronous caller and a streaming caller can use the same runtime.

What OpenClaw contributed

The implementation explicitly adapted OpenClaw’s harness boundary and its prepare–send–resolve lifecycle pattern. It also followed an event-oriented approach: execution emits facts that can be projected into different consumer views. OpenClaw’s documented runtime similarly distinguishes session preparation, execution, tools, streaming and lifecycle events. OpenClaw agent loop

The adaptation was selective. This product needed one default harness with replaceable dependencies. It did not need a general selector among many execution backends. Capability methods could remain part of the contract without becoming a framework-sized dispatch layer.

OpenClaw’s gateway also provides a useful reference for keeping channel connectivity and control clients outside the reasoning core. In the client platform, existing backend integrations owned messaging delivery; the agent service accepted normalised requests rather than becoming the owner of every channel connection. OpenClaw gateway architecture

That is an architectural influence, not a claim that the hospitality platform ran on OpenClaw. The runtime was implemented for the client’s service boundaries and domain.

What Hermes contributed—and what changed

Hermes provided a practical reference for a complete agent product: provider resolution, tools, prompt assembly, session state and multiple entry points arranged around the agent conversation. Its architecture is useful precisely because it exposes the surrounding machinery that short agent tutorials omit. Hermes architecture

The local design made two explicit departures. Per-turn callbacks were consolidated into one typed event stream. Session-specific state was passed into execution instead of being kept on the shared harness instance. The service needed to handle unrelated tenants concurrently, so a singleton with mutable conversation fields would be the wrong ownership model.

Hermes’ reusable skills are another useful reference: procedures can be loaded when relevant rather than embedding every procedure in the base prompt. Here, skills became versioned database records with trigger rules, prompt fragments and tool membership. They were managed through the console, with code providing the initial catalog. Hermes skills system

The product did not adopt autonomous rewriting of production procedures as its change-control mechanism. An operational policy about refunds or maintenance needs an accountable owner and evaluation before publication. Learning from a failure meant producing a case, a proposed change and a reviewable comparison.

A map of the selected patterns

The custom work is concentrated in the right-hand column. A general agent framework cannot know that a guest has two active stays, that a task is only proposed, or that a quoted nightly price excludes checkout fees. Those rules belong to the product.

Modularity through contracts

The orchestrator depends on interfaces for the run store, router, context builder and harness. In Python, Protocol expresses this without requiring a deep inheritance tree.

from typing import Protocol

class ContextBuilder(Protocol):
    async def build(self, event, session, route): ...

class AttemptRunner(Protocol):
    async def run(self, attempt, sink): ...

class Harness:
    def __init__(self, runner: AttemptRunner):
        self.runner = runner

    async def run_attempt(self, attempt, sink):
        return await self.runner.run(attempt, sink)

The important property is what the object does not retain: no current_tenant, no current message array and no per-request sink. Those values are parameters. Shared dependencies can be long-lived; request data remains request-local.

This is a compact expression of the execution seam. The full product adds typed models, lifecycle events, cancellation and persistence around it. The modules remain understandable because each has a specific reason to change.

The composition root

Dependencies are wired during application startup in one composition root. It selects configured stores, constructs model transports and clients, loads registries, creates the router and context builder, and connects event sinks.

Requests borrow this graph. They do not reconstruct provider clients or rediscover tool handlers on every turn. Equally, changing the graph must not mutate another request’s input. Shared registries hold configuration; each attempt holds its own resolved input and policy.

Dependency wiring should make the active execution path explicit. A useful architecture review follows one request through identity, context, model access, tools and persistence. That exposes ownership boundaries and unnecessary coupling more clearly than a list of installed dependencies. In a modular harness, replacing a knowledge provider should change its adapter and contract tests, without changing the rules for tenancy or action approval.

The cost of owning a harness

A custom harness buys control over lifecycle, state and domain semantics. It also creates maintenance obligations: provider changes, tool contracts, schema migration, race conditions, configuration compatibility and evaluation infrastructure.

It is justified when those obligations already exist in the product and need explicit ownership. It is less attractive for a simple internal assistant that can operate safely inside an existing framework’s boundaries.

For this platform, the distinguishing requirements were operational action, shared tenancy and an inspectable control plane. Those requirements explain the architecture more convincingly than the phrase “custom agent framework.” The following chapters examine the contracts that made that architecture work.

PART 03

The run contract: orchestration, lifecycle and idempotency

A messaging webhook can arrive twice. A caller can time out while the worker is still running. A process can be cancelled after the model has requested a tool but before the reply is delivered. These are ordinary distributed-system events. An agent runtime needs ordinary, explicit semantics for them.

The client platform organised execution around four identities: tenant, session, run and attempt. Each answers a different question. The tenant defines the account boundary. The session groups conversation continuity. The run represents handling one admitted event. The attempt identifies an execution envelope within that handling.

Put a stable contract at the front door

Different entry points carry different payloads: chat text, a messaging conversation, a task-status event or an approval decision. Adapters normalise these into an inbound event with authenticated tenant and actor identity, a source, channel information, text and relevant domain context.

The model never chooses the tenant. Nor should a conversation identifier alone become an authorization boundary. A session identity must include the dimensions that determine who can share state. Two accounts using the same external thread identifier must not share a session.

{
  "tenant_id": "tenant-example",
  "event_id": "message-104",
  "idempotency_key": "messaging:message-104",
  "source": "messaging",
  "actor_type": "guest",
  "actor_id": "guest-example",
  "channel": "whatsapp",
  "channel_conversation_id": "thread-example",
  "text": "Is there any update on the repair?"
}

This example puts identity ahead of routing, retrieval and tool execution. Validate identifiers at ingress and derive account scope from the authenticated caller. An external conversation identifier remains a correlation value; it does not grant access to a conversation or its records.

Claim the inbound event before paid work

Admission resolves identity and session before applying the idempotency gate ahead of classification. That ordering matters. If classification can call a model, deduplicating after classification still spends money on duplicate deliveries.

The durable gate uses a unique scope over tenant, idempotency scope and key. It stores a request hash alongside the claim. A duplicate with the same content can replay or refer to the existing run. The same key with different content is a conflict, not an invitation to return an unrelated response.

The following standalone SQL example expresses the admission rule:

INSERT INTO example_event_claims
  (tenant_id, scope, key, request_hash, resource_id, status)
VALUES
  (:tenant, 'inbound', :key, :hash, :run_id, 'in_progress')
ON CONFLICT (tenant_id, scope, key) DO NOTHING
RETURNING resource_id;

The claim and initial run creation belong in the same database transaction. Otherwise a crash can leave a claim pointing to a run that was never created. PostgreSQL’s conflict handling supplies the atomic insertion primitive; the surrounding transaction and application state rules supply the protocol. PostgreSQL INSERT

This is inbound deduplication. It is not a proof of exactly-once business side effects. Tool writes and downstream projections need their own identities, which we examine in the reliability chapter.

A duplicate is a state-dependent decision

A retry protocol can rebind a claim for retryable terminal work using a compare-and-swap against the previous owner. Competing retries must not both become the new owner. Refresh the lease or claim timestamp when ownership changes, and preserve the old run as lineage. Otherwise a retried request may appear stale immediately, or its original failure can disappear from the audit history.

An event ID from the channel provider is preferable to a weak fallback derived from text. “Yes” is a valid message today and next week. A hash of the visible text alone cannot distinguish them.

Route and assemble one attempt

After admission, the router produces a model choice, budgets, skill selection and tool allowlist. The context builder combines the resolved persona, live facts, knowledge and conversation state into model input. The orchestrator creates an attempt containing those values and the request-local event sink.

Resolving the model before the loop also makes attribution clearer. Usage for that attempt can be associated with the selected model. If a future design switches models during a run, usage must be stored by model turn rather than pricing the entire total with the final model’s rate.

An attempt carries both the flat system prompt and structured system blocks for provider-specific cache handling. It also carries messages, tool specifications, policy, delivery context and budgets. These are execution inputs, not hidden mutable properties on the harness.

Prepare, send, resolve

The lifecycle envelope validates the attempt, emits its start event, runs the harness under a deadline and classifies the returned outcome. Expected failure and unexpected exception are different paths.

For example, a runner can return waiting_for_approval as a valid outcome. A provider timeout can become a classified timed-out result. An unexpected exception becomes a failed result so that the orchestrator can finalise the run instead of abandoning a running row.

Cancellation deserves separate treatment:

import asyncio

async def execute_with_deadline(runner, attempt, sink, seconds):
    try:
        return await asyncio.wait_for(
            runner.run(attempt, sink), timeout=seconds
        )
    except asyncio.CancelledError:
        # The caller must still be able to stop the task.
        raise

The full envelope records cancellation before propagating it. Catching every exception and converting cancellation into an ordinary response can leave work running after the caller believes it stopped. Conversely, propagation without finalisation can strand state. The lifecycle and orchestrator share that responsibility deliberately.

Finalisation is part of correctness

Finalisation stores the result summary, prices captured usage, records domain-specific state updates and emits the terminal run event. Its failure modes deserve tests just as much as tool execution does.

A particularly awkward case is a failure after claiming the event but before context assembly completes. If the run remains running, retries can keep finding the same unusable record. The orchestrator’s failure envelope must cover routing and context building, not only the model loop.

The runtime also records stage durations: routing, context construction, attempt execution, finalisation and total time. This makes a slow call diagnosable without assuming the model caused it.

Runtime completion is not business completion

An iteration budget can be exhausted and still produce a graceful answer. The runtime may classify that as completed because it returned a response successfully. The business request can still be unresolved.

The same distinction appears with approvals and asynchronous workflows. A workflow being accepted is a valid conversational outcome, but the work is not finished. Evaluation and analytics need both the runtime status and the task outcome, including warnings such as budget exhaustion.

Failure drills that expose the contract

Useful tests exercise the same event twice, a conflicting hash, a routing exception after admission, a timeout during tool execution, and cancellation during finalisation. They assert stored states and emitted events, not only the returned text.

For a retry, inspect whether the original run remains available for audit and whether the new run has clear lineage. For concurrent duplicates, prove there is one winner at the database boundary. For a stale replay, prove that the freshness policy does not accidentally replay an old guest answer into a new conversation turn.

These contracts make the rest of the system easier to reason about. Once admission, identity and terminal states are explicit, the model loop can concentrate on its own bounded job.

PART 04

Skills and on-the-fly composition

“I need a place for next weekend” and “the Wi-Fi is not working” can both mention a property. They should not receive the same operational procedure. One starts a booking inquiry. The other may require troubleshooting and issue reporting for an existing stay.

The routing layer decides which capabilities the turn needs before the model starts working. In the client platform, that decision combined an agent identity, a model tier, skill instructions, message categories and a tool allowlist. The result was an assembled capability for this turn.

Give “skill” a concrete meaning

A skill was a routable bundle of trigger patterns, an instruction fragment and tool names, with context conditions where needed. It was not another autonomous assistant with its own personality and history.

{
  "capability": "example-maintenance-help",
  "audience": "guest",
  "triggers": ["leaking", "not working", "blocked"],
  "requires": "an identified stay",
  "prompt_fragment": "Record the issue and explain the returned outcome.",
  "tools": ["report_issue", "get_task_status", "escalate_to_host"]
}

The bundle keeps a capability separate from a persona. The same guest-facing voice can answer a property question, search inventory and explain a maintenance update without a sequence of artificial handoffs. Validate the bundle before publication: every referenced tool must exist, every prerequisite must have a provider, and the instruction fragment must fit within the audience’s authority.

The skill concept also gives an operator a manageable editing surface. Updating a booking procedure should not require rewriting the entire concierge persona.

A two-tier classifier

The classifier first uses deterministic patterns and context. When that is insufficient or ambiguous, a model-based classifier can supply a more flexible decision. The fallback must be bounded: a routing call that hangs can consume the whole conversational deadline before the main model is invoked.

This is not a claim that keywords are universally better. They are cheap, inspectable and effective for well-defined signals. They are weak on continuation and paraphrase. A guest replying “completely blocked” may be continuing a toilet complaint; the latest words alone are an incomplete classification input.

The design therefore has to account for context preconditions, explicit hints and continuation matching. A broad general-question skill should not absorb a more specific cancellation or complaint merely because it was registered first.

These are classifier failures, so they need a classifier dataset. Looking only at final answer scores hides where the wrong decision entered the pipeline.

Single-skill and multi-skill modes

There are two useful composition policies: select one primary skill, or combine several compatible skills. The correct choice depends on the request. A simple property question may need one capability; a complaint followed by “and what time is checkout?” may require two. Define the policy explicitly and evaluate each mode against the same multi-intent scenarios.

In multi-skill mode, matching capabilities contribute their instructions and tools. The primary classifier-selected skill must remain in the composition even if it came from an LLM decision rather than a regex hit. Otherwise the trace can report the right classification while the attempt receives none of that skill’s instructions.

The runtime also deduplicates identical fragments and tool names. This is not just tidiness. Repeated instructions waste context and can amplify conflicting guidance. Stable ordering makes the assembled prompt easier to compare across runs.

Compose tools within the agent’s authority

Combining skill tool lists must not widen the persona’s permissions. The conceptual rule is:

Effective tools = requested capability tools ∩ agent-permitted tools

Here is a small, executable form of the composition rule used throughout the teaching examples:

def compose_tools(agent_allowlist, skill_toolsets):
    permitted = set(agent_allowlist)
    ordered = dict.fromkeys(
        tool
        for toolset in skill_toolsets
        for tool in toolset
        if tool in permitted
    )
    return list(ordered)

assert compose_tools(
    ["search", "report_issue"],
    [["search"], ["report_issue", "refund", "search"]],
) == ["search", "report_issue"]

The empty case needs explicit semantics. “No tools selected” should remain a read-only conversational route or trigger a deliberate routing failure. If a general-purpose capability needs tools, give it an explicit permitted set. An accidental empty match must never expand authority to the entire catalog.

Exposing a tool is only one control. The executor still checks the allowlist when a call arrives. A model can emit an unexpected tool name even if it was absent from the prompt, and a custom caller can manufacture a tool-call object.

Resolve the model from configuration

The classifier proposes complexity; agent configuration determines the available base and complex models. The router resolves that combination once for the attempt.

A model tier is a product decision, not a synonym for importance. An urgent but structured issue can be handled by a fast model plus a deterministic dispatch flow. A long host analysis may need a stronger reasoning model. Sensitive writes need stronger policy controls regardless of which model is used.

The appropriate routing experiment compares task success, latency and cost by scenario family. Moving every request to a larger model can hide weak tool contracts temporarily while making the whole service slower and more expensive.

On-the-fly composition does not mean arbitrary execution

The system composes from published, known components. It does not allow a guest’s text to install a tool, edit an authorization rule or define a new backend endpoint.

There are three time scales:

1. Deployment time: code handlers, adapters and contracts are shipped.
2. Configuration time: approved prompts, tools, skills, models and agent settings are published and loaded.
3. Request time: the runtime selects and combines those published capabilities for the current event.

Separating these time scales makes “dynamic” precise. The per-turn combination changes quickly. The authority to create executable capabilities remains controlled.

Resolve instruction conflicts at ownership boundaries

A booking skill may say “ask for dates first,” while a base fragment says “search first.” Adding more emphasis to both fragments does not solve the contradiction.

The better fix is to decide which component owns the decision. If a booking flow has explicit rules for missing dates, that behaviour belongs in the flow contract and its tool description. The skill should explain how to use the flow without restating another version of the same policy.

The same applies to tone and formatting. Channel fragments own presentation; skills own the procedure; code owns mandatory state transitions. This is how modularity survives beyond the first few skills.

Inspect the assembled result

The control portal’s assembly view makes the result visible: selected skills, resolved model, prompt-fragment sizes, tool exposure and retrieved context. That is more useful than showing only the name of a skill in a badge.

When a change fails, the first question is whether it reached the attempt. A correct draft that was never published, a published skill that was never reloaded and a routed skill excluded by a precondition produce similar guest-facing symptoms. They require different fixes.

The next article follows the resolved route into prompt and context assembly, where composability meets evidence, freshness and token cost.

PART 05

Prompt assembly is an engineering pipeline

When an agent gives a poor answer, the instruction “improve the prompt” is often too vague to be useful. Which prompt? The base persona, the booking procedure, the host’s property rule, the retrieved knowledge or the channel’s formatting instructions?

The client platform assembled the model input from named components. This made each turn inspectable and made changes more local. It also exposed a practical truth: the quality of the answer depends on the context pipeline as much as on the wording of instructions.

Separate instructions, facts and conversation state

The base persona defines the assistant’s role and general behaviour. A skill fragment defines a procedure. Property knowledge provides authored facts. Domain hydration supplies current reservation or listing data. Conversation state records what has already been offered, selected or reported.

These sources should not be flattened conceptually even if a provider ultimately receives text. Each has a different owner and freshness requirement.

If a retrieved manual contains “ignore the previous instructions,” it remains document content. It does not acquire the authority of a system instruction because it appears inside a prompt block. Scope and authorization still have to be enforced outside the model.

Hydrate independently, assemble deterministically

The context builder uses pluggable hydrators for different evidence sources. Independent reads can run concurrently. Their results are then rendered in a predictable order.

For example, property knowledge, current domain facts and host rules can often be fetched in parallel. Stay resolution may create a dependency: a property-specific retrieval cannot be safely scoped until the relevant stay is known. Parallelising everything blindly would turn a latency optimisation into a correctness bug.

import asyncio

async def hydrate_independent(loaders):
    names = list(loaders)
    results = await asyncio.gather(
        *(loaders[name]() for name in names),
        return_exceptions=True,
    )
    return {
        name: {"value": None, "degraded": True}
        if isinstance(value, Exception)
        else {"value": value, "degraded": False}
        for name, value in zip(names, results)
    }

This small example illustrates independent degradation, not an unlimited permission to continue. A missing optional brand-voice fragment can have a safe fallback. Missing authorization context cannot. Missing availability means the agent cannot claim a property is available.

The runtime records degradation so the operator can distinguish “no knowledge matched” from “the knowledge service could not be reached.” Both can produce an empty block; operationally they are different.

Keep the stable prefix stable

The system prompt is divided into a relatively stable base and a per-turn suffix. A provider supporting prefix caching can reuse the stable part, while current facts and instructions are assembled for the request.

Caching depends on the provider’s rules, including matching prefixes, supported models, minimum sizes and cache lifetime. A cache=True flag in an internal object is an instruction to the adapter, not proof of a cache hit. Hits must be observed in usage records. Anthropic prompt caching

A useful layout is:

Stable:
  persona → domain vocabulary → general operating rules

Per turn:
  date and channel → authenticated context → scoped knowledge
  → live facts → host rules → selected skill → conversation state
  → locale and outbound contract

The ordering should follow dependencies: establish identity before fetching scoped evidence, resolve ambiguous references before requesting resource-specific facts, and separate instructions from data. Record the resolved component versions so the assembled input can be reconstructed without exposing live prompt contents in routine telemetry.

Putting a timestamp at the very beginning of an otherwise cacheable prefix can invalidate reuse. Putting a volatile price inside the stable persona can produce a stale answer. The cache split is therefore both a cost decision and a data-lifetime decision.

Render variables in two passes

Prompt authors can supply defaults for declared variables. The platform then supplies reserved values such as the current date, channel, locale and scoped domain IDs for each turn.

The two-pass design avoids asking every prompt author to reproduce platform context. It also means author-provided defaults must not override authenticated scope. A variable called tenant_id is not an acceptable route for a prompt editor or model to choose another account.

Validate unresolved placeholders before publication. A required variable needs a declared provider and a failure policy; an optional variable needs an explicit fallback. Missing authorization context must stop the action, while a missing style preference can safely use the default voice. This prevents a prompt that renders successfully from being mistaken for one with complete evidence.

Contradictions have a location

Suppose the base says “never claim a booking is confirmed before payment,” while a skill says “confirm the booking after selecting a property.” The model has been given incompatible instructions. A longer prompt makes the conflict harder to inspect.

The correction is to establish ownership: selection returns a checkout opportunity; confirmation comes from a verified booking state. The skill’s language should follow that contract. The operational distinction belongs in the tool result and flow state, with the prompt reinforcing it.

This is why fragment review should include relationships, not just individual text. A perfectly reasonable fragment can become wrong when combined with another one.

Build a context budget around evidence

Context is finite even when the model’s nominal window is large. More material increases prefill time, cost and the chance that important facts are buried.

A useful allocation process reserves output and tool-result space first, then budgets the persona, current state, retrieved evidence and history. Consider an illustrative 16,000-token input envelope with 3,000 tokens of stable instructions, 2,000 of live state, 4,000 of relevant evidence and 5,000 of history. That leaves 2,000 tokens of input headroom for the next tool result. The numbers are a worked allocation, not a measured configuration of the client deployment.

Trimming must preserve evidence relationships. A price without currency, dates or a property reference is smaller but less useful. A tool result without its error status is actively misleading. A summary of previous offers must preserve which option was shown in which search.

The assembly view is a debugging tool

The runtime emits an assembly event describing the model, skills, tools and prompt-fragment breakdown. The admin playground renders that information alongside the conversation.

When a property answer is wrong, inspection can proceed in order: was the stay resolved correctly, was retrieval scoped to that property, did the expected fragment appear, and did the model receive the relevant fact? This is much faster than repeatedly editing the final instruction and hoping the next answer improves.

Prompt text can contain guest information and access details. Routine metrics should carry sizes, hashes and source references. Full prompt capture needs deliberate access and retention controls. Making a system inspectable does not require putting every conversation into every telemetry backend.

Test the assembly, not just the prose

Useful assembly tests compare the selected fragments and tool surface for the same scenario under code-backed and database-backed configuration. They verify scope, ordering, missing-context behaviour and cache-block construction.

A good regression fixture also records the reason a fragment is absent. “No reservation” and “reservation fetch failed” must not accidentally activate the same optimistic booking behaviour.

The result is a prompt pipeline that can be operated like software: inputs are explicit, composition is reproducible and failures can be localised. The next chapter examines the most easily confused of those inputs—memory.

PART 06

Memory: what was said, what was shown and what is true

A guest is shown three properties. Several turns later, after asking about parking and changing the dates, they say: “Let’s go with the second one from the earlier dates.”

The visible transcript may contain property names, but internal IDs are deliberately excluded from guest-facing text. Re-reading that transcript is not enough to determine the exact listing, the search it belonged to or whether its old checkout link is still valid.

This is a memory problem, but a vector database is not the first answer. The missing information is structured operational state.

Four different stores of meaning

The platform distinguishes conversation history, retrieved knowledge, working state and live domain records.

History says what was communicated. Knowledge contains property manuals and FAQs. Working state records the options and actions relevant to the current conversation. Domain records determine whether a reservation exists, a task is assigned or a calendar slot is available.

They can disagree because they represent different times and purposes. A message saying “available” is historical evidence of an offer, not a permanent availability guarantee. A guest’s statement “I paid” is conversational evidence, not the payment system’s state.

Hermes’ persistent memory is a useful external reference for separating durable personal context from the current exchange. The hospitality design needed an additional emphasis: structured state with business identifiers and freshness rules, rather than treating operational facts as free-form remembered prose. Hermes persistent memory

From a recent working set to booking memory

A useful first memory layer recovers recently presented options from completed tool results. It restores the link between the conversation and the exact business objects involved. The next design step is to preserve separate searches, their dates and the order in which choices were shown, so a later search does not silently redefine an earlier reference.

Three common failures show why a flat “recent properties” list is insufficient. Truncation can discard an option the guest still remembers. A new date range can overwrite the meaning of an old ordinal. A stored base price can be mistaken for the calendar’s quoted total. Treat search identity, presentation history and price provenance as distinct fields.

Retention should be bounded by relevance and decision value. Keep searches still referenced by the conversation, expire stale quotes, and summarise older discussion without discarding the identity of unresolved choices. A limit on stored records is a resource policy; it should not silently change what “the second option from earlier” means.

{
  "search_key": "city|2026-10-10|2026-10-13|4",
  "params": {"guests": 4, "nights": 3},
  "shown_upto": 3,
  "total_available": 7,
  "results": [
    {"ordinal": 1, "listing_id": "property-a", "name": "Harbour House"},
    {"ordinal": 2, "listing_id": "property-b", "name": "Garden Apartment"},
    {"ordinal": 3, "listing_id": "property-c", "name": "Courtyard Studio"}
  ]
}

The names and dates here are fictional teaching data. The structural point is that an ordinal is meaningful inside a particular search. It should not be a global “second property” variable that silently changes after every query.

Resolve references, then revalidate facts

Reference resolution and business validation are separate steps. The memory can identify what “the second one” referred to. It cannot establish that the option is still bookable.

A compact resolver can make ambiguity explicit:

def resolve_ordinal(searches, search_key, ordinal):
    search = next((s for s in searches if s["key"] == search_key), None)
    if search is None:
        return {"status": "unknown_search"}
    matches = [r for r in search["results"] if r["ordinal"] == ordinal]
    if len(matches) != 1:
        return {"status": "ambiguous_option"}
    return {"status": "resolved", "listing_id": matches[0]["listing_id"]}

The next step is a fresh calendar read. This is why memory should retain provenance and timestamps, not just a confident sentence. A cached fact can be useful for conversation continuity while still requiring revalidation before an action.

A guest can have more than one stay

The stay flow adds another distinction: the set of stays the guest is entitled to reference and the current focus of the conversation. The backend may attach one reservation to a thread, but the guest may be asking about another legitimate stay.

The resolver should use explicit references, recent choices and confirmation context. When more than one stay is plausible, an action needs clarification. It should not silently apply the complaint or checkout request to whichever reservation happened to be first.

The tool layer receives the resolved focus and the guest’s own stay set. That is stronger than asking the model to keep the correct ID in mind. A property-specific knowledge read must remain inside that scope, and access information can require an additional stay-window check.

Issue state remembers commitments

For maintenance, useful memory includes the issue title, associated task, reservation, assignee, slot, escalation and updates already sent. This lets a later “any news?” refer to a known issue instead of listing all tasks at a property and guessing which belongs to the guest.

The memory record does not replace a current status fetch. It supplies the identifier and conversational continuity for that fetch. The answer can then say what changed since the last update without treating the last update as current truth.

An issue record should preserve the difference between created, proposed, deduplicated and escalated outcomes. Flattening them to success=true loses the information needed for an honest response.

Freshness is specific to the source

Conversation working state changes within a turn and often needs a read after a write. Authored property descriptions can tolerate caching; availability and action status require their own freshness rules. Cache by the decision that consumes the data, rather than applying one expiry to every form of memory.

The right question is not “how long should memory live?” It is “what decision will use this fact, and how stale can the fact be for that decision?” A property description can survive longer than a nightly price. A previous search can remain in memory while every selection is rechecked.

Retention also has a user boundary. State should be keyed by tenant and the appropriate conversation or actor identity. A helpful memory shared across unrelated guests is a data leak, not a feature.

Keep memory behind a capability contract

A memory provider should answer a narrow contract: load the relevant state, apply a validated update, and expose provenance and version. Retrieval can use a search adapter while operational state uses transactional storage. Neither provider should require the model to know where the data lives.

This separation makes replacement practical. Changing a search index should not alter the authority of a reservation record. Replacing a state store should preserve conflict detection and reference resolution. Test these guarantees at the boundary instead of treating a particular memory product as the architecture.

State updates need concurrency rules

A stateless harness does not make the state store race-free. Two turns can read the same version, make different updates and overwrite one another. Booking and issue state therefore need an explicit update strategy: a database transaction, version check, serialised session lane or a carefully designed merge.

One implementation option is to load the latest state inside a database transaction, apply a reducer and increment a version. Another is an optimistic update conditioned on the expected version. Both protect the state update itself; neither automatically serialises external actions across an entire turn. Test retries and competing updates against the required guarantee.

For a reference implementation, make updates reducers over typed state and test them independently. Then test two competing updates against the actual storage implementation. The first tests business logic; the second tests the concurrency guarantee.

What good memory enables

The practical benefit is continuity with evidence: earlier options stay addressable, property facts remain scoped, a later issue update has a known task, and selection can be revalidated against current data.

Memory is valuable when it reduces repeated questions without creating stale authority. That is the standard against which a more elaborate vector or graph layer should be judged.

PART 07

The model transport and the bounded agent loop

The familiar agent loop is short: call a model, execute its tools, append the results and repeat. The difficult part is deciding exactly what each of those verbs means under streaming, failure and a deadline.

The client runtime kept the loop independent of provider-specific events. A transport adapter translated provider output into a small internal vocabulary. The loop could then focus on conversation progression and tool execution instead of carrying every vendor’s streaming protocol.

A narrow transport contract

A compact provider-neutral vocabulary can represent text deltas, tool-call starts, tool-argument deltas and completion with usage. Vendor-specific adapters translate into that vocabulary; the rest of the runtime consumes the semantic contract.

from dataclasses import dataclass

@dataclass(frozen=True)
class ToolStart:
    call_id: str
    name: str

@dataclass(frozen=True)
class ToolArgsDelta:
    call_id: str
    fragment: str

A provider adapter must preserve call IDs, tool names, complete argument content, stop reasons and token accounting. “Compatible JSON” is not enough if cache usage disappears or two concurrent tool calls become indistinguishable.

The local dispatcher selects an adapter from the resolved provider. Adding a provider means satisfying this stream contract and its tests; the orchestrator should not learn a new set of vendor event names.

Assemble arguments before executing anything

Tool arguments arrive as fragments. A fragment can split a JSON key, a quoted string or a number. It is not executable intent until the relevant call is complete and the accumulated JSON has been parsed and validated.

import json

def parse_call(fragments):
    value = json.loads("".join(fragments))
    if not isinstance(value, dict):
        raise ValueError("tool arguments must be an object")
    return value

assert parse_call(['{"guest', 's":', '4}']) == {"guests": 4}

The call ID is the assembly key. Position alone is unreliable when streams interleave. The result must later be appended with the same call ID so the next model turn can associate each result with the request that produced it.

Malformed arguments should become a structured failure the model can respond to. They should not silently execute as an empty object, particularly for a write tool with defaults.

The loop owns conversation progression

For each model turn, the loop records the request, consumes the stream and accumulates usage. If executable tools were requested, it calls the executor and appends the assistant tool-call message followed by the tool results. If a tool requires approval, the loop returns a waiting outcome. If the model produces a final answer, the loop applies the relevant checks and returns.

The implementation also distinguishes presentation tools. A request to render a component is projected to the UI rather than sent to a backend action handler. That distinction prevents a UI card from being treated as a completed business action.

A model response, an attempt and a run are not interchangeable units. One attempt can contain several model turns. Each turn can request multiple tools. A run can end awaiting approval without having performed the requested mutation.

Bound every source of repetition

The attempt has an iteration limit, output-token constraints and an overall deadline. The loop also bounds repair behaviour. It can ask for a correction when a response claims an unsupported action, when booking facts lack backing or when the model returns an empty answer. These repair passes consume iterations and time; they do not form an unlimited second loop.

Use one shared repair budget across failure classes. Allowing one retry for an empty response, another for an unsupported claim and another for formatting can turn a nominal “one retry” policy into several paid calls. Track both the reason and the total repair work remaining.

When the iteration cap is reached, the product returns a graceful fallback and emits a budget warning. That prevents an uncontrolled loop, but the task remains unresolved. The evaluation layer must treat that warning as an outcome signal rather than counting a polite apology as successful task completion.

Generated text is not automatically deliverable text

The model can begin with “I’ll check that for you” and then request a data tool. Streaming that sentence is often harmless. Streaming “I have cancelled your booking” before the cancellation tool succeeds is not.

The runtime separates raw model text events from user-visible text events. It buffers early narration until it knows whether the turn is answering or entering a data/action step. Presentation calls and executable calls receive different treatment.

This improves the experience, but there is a deeper constraint: a later guard cannot retract bytes already sent to a browser. A synchronous messaging response can be shaped using the final result; an SSE client may already have seen earlier deltas. For claims whose correctness depends on completed actions, the publication boundary must be after the evidence check.

That is a useful design extension for any streamed agent: maintain a draft stream for internal observability and a separately committed answer stream for claims that require verification. Do not assume that checking final_text sanitises every earlier event.

A claim check needs successful evidence

Suppose the model says, “The host has been notified.” Finding a tool named escalate_to_host in the transcript is insufficient. The call may have failed, created only an in-app item or returned no delivered notification channels.

A reliable check uses the tool’s result semantics. Did the action complete, was it merely proposed, and what evidence supports the wording? Typed action receipts make those distinctions explicit. A name or regular-expression match can be a useful warning signal, but it cannot establish that an external action succeeded.

{
  "action": "escalate",
  "record_created": true,
  "notification_status": "not_delivered",
  "allowed_claim": "The issue has been flagged for the host."
}

The receipt is a reference contract. It demonstrates how the business outcome can constrain language without asking another model to guess whether the first model was truthful.

Cancellation crosses every adapter

Cancelling the caller should propagate through the lifecycle, transport and tool layer. A blocked provider iterator, an outbound HTTP request and an executor waiting on a semaphore all need to cooperate with cancellation.

A timeout around only the model request leaves tool execution unbounded. A timeout around only the tool handler leaves queue wait unmeasured. An overall attempt deadline limits the combined path, while component deadlines identify where the time went.

The transport tests should include a stream that stops mid-argument, emits multiple tool calls, returns no text, exceeds a deadline and raises after partial output. Happy-path text streaming is the easiest part of the contract.

Keep the gateway path visible

A gateway can centralise provider access, attribution and budget enforcement. If a design permits direct-provider fallback, that path needs equivalent access controls, metering and spend limits. Otherwise a successful fallback can preserve the answer while losing the accounting and policy guarantees that were assumed at the gateway.

A healthy provider response can therefore coexist with an unhealthy accounting path. Readiness and operational dashboards need to report the route actually used, not simply whether a provider credential exists.

The loop becomes manageable when these boundaries are explicit: providers stream model events, the executor owns actions, the lifecycle owns deadlines, and delivery consumes only the events appropriate for the channel.

PART 08

Tools as data, execution as a controlled boundary

A tool definition looks deceptively simple: a name, description and JSON schema. In a product, it also represents a promise about an API, a permission boundary, a timeout, a side effect and a result the model must interpret correctly.

The client platform stored tool definitions as versioned configuration, then compiled them into runtime handlers. This made the tool catalog editable and inspectable without pretending that executable behaviour was just prompt text.

Four tool families

The implementation distinguishes declarative backend bindings, registered Python handlers, asynchronous workflows and client presentation tools.

Most simple API operations can be data-driven. Complicated workflows should not be forced into an increasingly obscure binding language just to keep the catalog “no code.” A small registered handler is often clearer than a declarative structure that has quietly become a programming language.

Compile definitions, do not evaluate arbitrary text

A published tool row contributes its name, description, parameter schema, type and policy metadata. A known compiler selects the appropriate handler type. It does not execute arbitrary Python pasted into an end-user message.

A standalone catalog-search example makes the binding contract concrete:

{
  "name": "example_catalog_search",
  "type": "backend_binding",
  "mutability": "read",
  "method": "GET",
  "path": "/example/catalog",
  "parameter_map": {"search": "q", "limit": "per_page"},
  "scope": ["account_id"]
}

The mapping is important. If the model supplies search but the API reads q, a successful HTTP response can still ignore the requested search. These are contract failures that a generic “tool call succeeded” metric will miss.

The executor pipeline

The executor resolves the tool, strips protected tenant/account keys from model arguments, validates arguments, checks policy, handles confirmation or approval, constructs trusted context and executes under bounded resources.

Tool errors are returned in-band so the model can recover or explain the failure. An invalid date should not crash the entire service. An unknown tool should not be dynamically discovered from the network merely because the model requested it.

Bound execution with a resource-specific concurrency limit and a deadline that includes queue wait. A handler can be fast once admitted while users spend most of their time waiting for a slot. Measure queue duration separately, reject work whose useful deadline has passed, and release capacity when a caller cancels. The performance chapter develops a worked capacity and deadline budget.

Tenant scope comes from the service

The model can propose a booking reference, but it cannot choose an account. Protected tenant/account arguments are removed and authenticated context is supplied separately. Domain-specific bindings and flows then enforce the relevant entity scope.

def scoped_args(model_args, trusted_scope):
    protected = {"tenant_id", "tenantId", "account_id", "accountId"}
    clean = {k: v for k, v in model_args.items() if k not in protected}
    return {**clean, **trusted_scope}

assert scoped_args(
    {"account_id": "other-account", "listing_id": "other-property"},
    {"account_id": "account-a", "listing_id": "property-a"},
)["listing_id"] == "property-a"

This example demonstrates overwrite semantics. It is not a substitute for backend authorization. The backend must verify that the trusted account may access the referenced entity. A parameter dictionary with a correct account_id does not make an unscoped database lookup safe.

Scope policy also depends on the operation. A host inventory search can range over the account’s properties. A guest retrieving access instructions should be pinned to an authorised stay. The same generic “overwrite every ID” rule cannot replace domain-specific authority.

Tool descriptions are decision surfaces

The description tells the model when to call the tool, which prerequisites it needs and what the result means. That makes description changes behaviour changes, even if the schema is unchanged.

For booking, a useful description can explain whether the tool handles missing dates, whether it validates calendar availability and whether a returned link means checkout rather than confirmation. For task creation, it should distinguish a proposal from a created record.

Descriptions should not become competing copies of business rules. If a deterministic flow owns the dispatch ladder, describe the flow’s contract. Do not also ask the model to independently recreate that ladder from six lower-level tools.

Validate the running contract

The operator portal includes a tool-contract surface. Its value is joining three facts: the published definition, the handler actually loaded in the running registry and the backend’s expected parameters.

That join can reveal a dead route, an ignored argument, a missing required parameter, a handler override or a tool absent from the agent allowlist. Each has a different correction. Looking only at a database row cannot tell you which function is executing.

A good contract report records both model-facing names and backend-facing names. It should also show which fields are filled from trusted scope and which must come from the model. This prevents a required parameter from being silently dropped during binding.

Parallel calls need semantic independence

The executor can gather multiple tool calls concurrently behind its semaphore. That improves independent reads. It does not make arbitrary writes safe to parallelise.

Searching properties and retrieving general rules may be independent. Creating a task and assigning a slot to that task are not. A workflow with mandatory order belongs in a composite handler, or must explicitly encode dependencies before execution.

The same issue appears with approvals: a batch containing an approval-gated write and an unrelated write may partially execute before the run pauses. A strict approval design should validate the entire batch before performing irreversible work. That is a stronger guarantee than detecting an approval marker after collecting results.

Return actionable errors

An error should identify what failed and what the model can safely do next. “Invalid request” gives little help. “Check-out must be after check-in; ask for corrected dates” supports recovery.

Errors must not expose credentials or another tenant’s data. Tool traces should preserve enough structured detail to diagnose the failure while redacting sensitive values. The guest-facing explanation can then stay concise.

The tool layer is where flexible language becomes application behaviour. Its quality depends on schema, scope, policy, backend semantics and evidence—not just whether the model selected the right function name.

PART 09

Authority, approvals and the trust boundary

An agent can understand a request perfectly and still be the wrong actor to carry it out. A guest asking for a refund, a host asking for an operational report and an administrator changing a tool definition have different authority even when all three requests arrive as text.

The client platform put tool policy between the model’s proposal and execution. That policy included allowlists, confirmation behaviour, approval handling and trusted scope. The essential design question was which decisions belonged to the conversation and which required application evidence.

A prompt is not authorization

“Only access the current guest’s booking” is useful guidance, but the backend must enforce the same restriction independently. The model can misunderstand context, respond to injected content or simply emit the wrong ID.

The implementation strips protected account and tenant keys from model arguments and supplies context from the runtime. Tool allowlists are checked at execution. Guest-specific flows also use the resolved stay set and focus. Backend account-scoped access remains the final authority on the domain record.

These controls form a chain. Missing any link can undermine the others. For example, a correctly authenticated agent service calling an unscoped backend endpoint still exposes an unsafe operation.

Three different meanings of “yes”

A guest can confirm that the leaking shower is in the upstairs bathroom. They can confirm they want the maintenance request filed. A host can approve an action that policy reserves for an operator. These are different kinds of confirmation.

The tool interface includes conversational confirmation markers, and the runtime has a durable approval path. They should not be conflated. A model-generated confirm=true can represent the model’s interpretation of the conversation; it is not independent proof that an authorized human approved a high-impact mutation.

For consequential actions, the application needs a stored approval bound to the action, actor and scope. A sentence in history is not a sufficient replacement for that record.

Approval resumption is a new execution boundary

When an action needs approval, persist a proposal and return a waiting outcome. Resumption must restore the proposal’s identity and authorization context, then revalidate its preconditions. The approval can be rejected, expire, or become stale because the underlying resource changed.

There are two common execution models: continue a durable workflow, or start a fresh run from the approved proposal. A fresh run can fetch current context, but it must not treat the new model plan as equivalent to the old proposal. A changed action needs a new approval even when the user’s original request has not changed.

Bind authority to the approved action, not to the conversation turn as a whole. For example, approval to extend a reservation by one night must not also authorise cancellation or a different price. The application checks that binding immediately before the mutation, after any fresh planning has taken place.

Bind approval to the proposed action

A reference approval record should include the authenticated tenant, approver, tool name, canonical argument digest, relevant resource version, expiry and current status.

import hashlib
import json

def action_digest(tenant, tool, args, resource_version):
    payload = {
        "tenant": tenant,
        "tool": tool,
        "args": args,
        "resource_version": resource_version,
    }
    wire = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(wire.encode()).hexdigest()

assert action_digest("a", "modify_booking", {"nights": 3}, 7) != \
       action_digest("a", "modify_booking", {"nights": 4}, 7)

This example binds the grant to the action’s meaning. The digest is useful only if the record is stored and checked by trusted code. It is not a credential that the model can grant itself.

On execution, verify that the approval is pending or claimable, belongs to the correct tenant, matches the action and has not expired. If the underlying booking changed after approval, revalidate or request a fresh decision. Approval of an old price should not authorize a different price silently.

Preview and apply are separate phases

Some operations are naturally two-phase. A booking modification can first show the proposed dates, price difference and consequences. Applying the change uses that preview’s validated parameters and current domain state.

This reduces ambiguity. “Yes” can then refer to a concrete proposal with an identity, not whichever plan the model happens to reconstruct. The UI can display the proposal, while messaging can refer to a short action summary.

The same principle applies to bulk host operations. Before changing ten tasks, identify the exact task set. A dynamic query such as “all overdue tasks” can match a different set by the time an approval is used.

Permissions need an audience model

The guest concierge and host copilot have different tool surfaces. An operator console is another audience again. Editing a published tool can change execution for many users, so configuration access is an operational privilege rather than ordinary chat access.

A role model should distinguish viewing traces, running evaluations, trying tools, editing drafts, publishing configuration and approving domain actions. Even if an early deployment combines several roles for a small team, those permissions should remain conceptually separable.

Raw traces can include guest context. A person allowed to edit a prompt does not automatically need access to every guest’s messages. Likewise, an evaluation reviewer can often work with sanitized cases rather than live account data.

Retrieved content remains untrusted input

Property manuals, support messages and tool-returned free text can contain instructions. The model must treat them as evidence, not as authority to change its operating rules.

The stronger protection is structural. A retrieved instruction cannot add a tool to the allowlist, alter the authenticated tenant or mint an approval record. It can still influence language, so evaluation should include adversarial content inside the knowledge and tool-result channels, not only hostile user messages.

For a future general web-fetch or code-execution tool, the trust model changes substantially. Network destinations, filesystem access, execution isolation and credential exposure become additional boundaries. The hospitality runtime’s ordinary backend tools should not be described as an unrestricted computer-use sandbox.

Dry-run mode has a specific guarantee

A dry-run mode can intercept mutating tools and return controlled fixture outcomes. Its scope needs to be explicit: a simulation may still perform real reads unless those are redirected too. Use a test environment with synthetic records when validating complete workflows; a simulated success does not establish that a real backend mutation would succeed.

The guarantee also depends on accurate tool metadata. A mutating endpoint mislabeled as a read defeats a write short-circuit. Tool-contract review therefore belongs alongside dry-run testing.

A dry run, an approved action and a completed action should be visually distinct in the portal. Otherwise an operator can mistake a convincing simulation for operational evidence.

Evaluate authority with negative cases

The most valuable tests are often requests that must not succeed: another tenant’s ID, an expired approval, a changed action after approval, a write requested through a read-only persona, or a duplicate resume.

For each case, assert that the side-effect handler was not called. A polite refusal alone is insufficient; the model could have performed the action before apologising. The authorization result and the conversational result need separate observations.

Authority becomes manageable when it is represented as application state. The model can help a user express intent. The service remains responsible for proving that an allowed actor authorized the specific action being executed.

PART 10

Booking: let the model converse and the system decide

Booking is a good test of whether an agent is connected to a product or merely describing one. A guest can change dates, compare options, ask for more choices and return to an earlier property. Every one of those turns can invalidate an assumption made two messages before.

The client platform moved important booking behaviour into a dedicated flow. The model retained the conversation, but search state, calendar checks, selection and property evidence had explicit application logic.

Search is not availability

A property search finds candidates. A calendar check establishes whether each candidate can satisfy the requested dates, occupancy and booking rules. Returning a search result as “available” without that second step is a semantic error even if the retrieval ranking is excellent.

The booking flow separates candidate discovery from calendar verification. It tracks checked, unavailable and unverified results. It also records whether a scan is complete or truncated. That lets the reply distinguish “no available options were found” from “not every candidate could be checked.”

The distinction matters when a backend is slow. A partial scan may be useful, but the agent should not describe partial evidence as the complete inventory.

Quote the calendar’s total

A listing’s base nightly rate is not a quote for a specific stay. Rates can vary by night, and availability rules can depend on the date range. The flow uses the calendar’s night-by-night information to derive the total for the requested nights.

Consider an illustrative three-night stay at 120, 150 and 150 currency units. The nightly base price of 120 would suggest 360; the actual night total is 420. Fees and taxes may still belong to the checkout calculation. The reply must state what the number includes.

from decimal import Decimal

def nightly_total(rates):
    return sum((Decimal(str(rate)) for rate in rates), Decimal("0"))

assert nightly_total([120, 150, 150]) == Decimal("420")

The arithmetic is simple. The difficult part is carrying the correct dates, currency, inclusion rules and provenance through the tool result and into the sentence the guest reads.

Remember what was shown

The system retains search records with their parameters, results and presentation position. The current design supports paging through available properties rather than letting a small first-page cap hide the rest of the inventory.

The phrase “the second one” refers to a presented option, not the second item in whatever search happened most recently. Recording presentation is therefore part of state management. A tool result can contain ten candidates while the model mentions only three; the memory needs to know what the guest actually saw.

A worked conversation makes the requirement clear:

Guest: Four people, 10–13 October, near the harbour.
System: Search and verify; present options 1–3.
Guest: Any others?
System: Continue the same search and preserve its ordinals.
Guest: What if we arrive a day earlier?
System: Create a distinct search with different calendar evidence.
Guest: The second one from the original dates.
System: Resolve the original search, then recheck that property.

This is why a transcript summary is not a complete booking memory. It may preserve the conversation’s topic while dropping the precise mapping needed for an action.

Selection is a new validation point

The selection flow performs a fresh calendar read before returning a checkout link. A property that was available at discovery time may no longer be available when the guest chooses it.

The response should distinguish a valid checkout opportunity from a confirmed reservation. Giving a link does not prove payment, acceptance or booking completion. The confirmation must come from the domain system’s state.

This can be expressed as an evidence ladder:

The model should not jump to a higher rung because a more confident sentence sounds reassuring.

Property facts must stay with the property

Parking, accessibility, amenities and house rules are not interchangeable across listings. A generic knowledge search without property scope can produce an answer that is plausible for the portfolio but wrong for the selected stay.

The booking flow retrieves listing details, rules, guest-book information and property-scoped knowledge. Guest-book access is also subject to the appropriate redaction and stay privileges. A prospective guest asking about parking does not need an access code.

The model receives the resulting facts in context, but the flow retains their listing identity and retrieval purpose. That helps answer follow-up questions without treating unrelated property evidence as shared knowledge.

Use bounded parallelism for calendar checks

Calendar verification is a natural fan-out operation: several candidate properties can be checked independently. Give it a concurrency budget based on the calendar service’s capacity. That budget serves a different resource from the outer tool-execution limit or the model-call limiter.

The useful performance calculation is the critical path. If 24 independent calendar reads each take an illustrative 250 ms and the effective concurrency is eight, the idealized service time is roughly three batches, or 750 ms, before overhead and tail latency. It is not 250 ms, and real distributions rarely align into perfect batches.

Track queue wait, timeout rate and backend pressure. Raising concurrency can reduce one guest’s scan time while overwhelming the domain API for everyone else.

Persist facts, not unsupported claims

The booking memory keeps verified results, search parameters and references. It can also record what the reply presented. The guard layer checks for booking claims unsupported by tool results or remembered evidence and can request a bounded repair.

This is useful defence, but it should not become the primary source of correctness. If the tool result already clearly distinguishes unavailable, unverified and bookable states, the model has less ambiguity to repair.

A good result contract is often more valuable than another paragraph of instructions. It gives the model a constrained vocabulary for explaining the outcome.

Test the conversation as a state machine

Single-turn tests miss most booking failures. The evaluation set should include changing dates, asking for more results, selecting an earlier option, duplicate property names, unavailable calendars, incomplete scans and returning to a previous search.

Assertions should inspect the selected listing and search parameters, not only the presence of a checkout URL. A syntactically valid URL for the wrong property is a failed task.

Also test the transition from prospect to existing guest. Once a reservation is confirmed, the authority and data needed for access instructions change. The system should derive that transition from domain state, not from the model’s memory of a congratulatory message.

Why the composite flow helped

The flow reduced the number of mandatory operational steps left to open-ended model planning. Search, verification, pagination, state updates and selection checks became explicit behaviour with unit-testable helpers and inspectable results.

The model still handled the part that benefits from language: understanding preferences, clarifying ambiguity and explaining trade-offs. That division made the assistant more useful while making its critical promises easier to verify.

PART 11

From a complaint to work that someone owns

“I have informed the team” is an easy sentence for a model to produce. It is also one of the least useful responses if no one has been assigned, no notification was delivered and the guest has no way to learn what happens next.

The issue workflow in the client platform was designed around the operational chain behind that sentence. A complaint needed a known stay, a record, a dispatch decision, an escalation rule and a persistent reference for later updates.

Turn the complaint into an explicit outcome

The input includes a guest’s description, a category, urgency and the resolved property or stay. The output needs more than a task ID. It should say whether the task was created, proposed or deduplicated; who owns it; whether a slot was booked; when it is due; and what escalation or notification occurred.

The platform’s dedicated issue flow puts the mandatory chain in code. The model interprets the request and explains the returned outcome. This avoids depending on the model to rediscover a dispatch procedure on every complaint.

A worked outcome might be:

{
  "status": "created",
  "task_id": "task-example",
  "assignee": {"name": "Maintenance team", "kind": "load"},
  "slot": null,
  "escalation": {"recorded": true, "notified_channels": []},
  "say": [
    "The issue has been recorded.",
    "An arrival time has not yet been confirmed."
  ]
}

The scenario is illustrative, but the distinctions are implemented concerns. An assigned person without a booked slot is not a scheduled visit. An escalation record without a delivered channel is not evidence that someone received a message.

Start with the right stay

A guest with several stays can report an issue at any of them. Before creating a task, the flow must know which stay the request concerns. The current stay focus, explicit references and recent conversation choices can help; unresolved ambiguity should produce a clarification.

This is also an authorization boundary. A task-status query should resolve against issues associated with the guest’s conversation and stays, not select an arbitrary task at the same property. Shared property identity is too broad to identify the guest’s issue.

The issue memory supplies references for later “any update?” requests. Live task state supplies the answer.

Deduplicate operational work

Messaging retries are one source of duplicates. Repeated guest complaints are another. An inbound idempotency key handles a repeated event, but a guest can send a new event about the same unresolved leak.

The issue flow checks for an applicable existing request and associated task. A valid duplicate can return the current status instead of creating another work order. The deduplication policy needs a defined scope, category and time window; it should not collapse unrelated problems merely because both are plumbing issues.

An evaluation should include a repeated complaint, a genuinely different issue in the same category and a recurrence after closure. The desired behaviour differs across those cases.

Availability is more than a role label

A staff member with the role “maintenance” is not automatically available. The platform’s scheduling API uses working schedules and occupied task intervals. Missing roster schedules can therefore produce no bookable slots even when staff records exist.

That is an integration and data-quality problem, not a prompt problem. Asking the model to “try harder to find staff” cannot create a schedule that the backend does not have.

The dispatch design first looks for suitable availability. When a slot is available, the earliest feasible slot matters; a tie can consider workload. When no slot exists, a configurable ownership ladder can assign responsibility without inventing a visit time.

Distinguish ownership from scheduling

The ladder can fall back through available team members, a property manager and an account owner, depending on configuration and available records. Its purpose is to avoid leaving an issue ownerless while preserving the truth about scheduling.

These distinctions are easy to erase when a handler returns only success=true. They are also precisely what the guest needs to know.

Encode urgency rules where they can be tested

The implementation uses urgency-dependent due times and escalation thresholds. These are product operating policies, not universal hospitality standards. Their values belong in a controlled policy surface and should be validated with the client’s team.

An urgency policy can define shorter response windows for emergency and urgent categories and longer windows for ordinary requests. Escalate when the earliest feasible visit falls outside the relevant window. The mechanism compares the actual dispatch result with a configured policy; the model does not improvise an acceptable delay.

Times should carry timezone and date, especially when a guest and the operator are in different locations. “Tomorrow afternoon” is insufficient state for scheduling even if it is appropriate final wording after conversion.

Status updates have a second delivery path

When the backend changes a task’s state, an event can invoke the concierge to compose an update. The backend then owns channel delivery, including messaging templates and delivery rules.

This path needs the task’s facts and the conversation’s issue record: what was reported, what was promised and what the guest was already told. Otherwise a status update can read like a new, disconnected conversation.

The integration must specify which transitions emit updates. Assignment, appointment changes, completion and cancelled work are different events. Test each transition for correlation, content and delivery acknowledgement. A generic “status changed” hook does not establish that every meaningful operational change reaches the guest.

An asynchronous workflow is only accepted at first

Some requests are delegated to a workflow service because they take longer than the conversational deadline. The agent can report that the request was submitted, using the returned workflow identity. Completion requires a later result.

A downstream 422 or failed job must not disappear behind an earlier “started” marker. The workflow needs terminal status, a correlated event and a failure path visible to the operator. If the guest was told to expect follow-up, that commitment also needs ownership.

This is the operational meaning of an AI concierge: it participates in a workflow whose outcome outlives the sentence it generated.

Evaluate the whole chain

The test matrix should vary roster completeness, urgency, existing issues, multiple stays, declined time slots, failed notifications and backend suggestion mode. Check both the tool results and the final claims.

For example, a case with no staff schedule should not pass merely because a task exists. It should verify the fallback owner, escalation decision and absence of an invented arrival time. A proposed task should not be described as dispatched.

The resulting system is more dependable because the mandatory chain is explicit. The model can be warm and concise while the application keeps a precise account of what was—and was not—done.

PART 12

One event spine, several ways to deliver the result

A web operator wants to see progress as a run unfolds. A messaging integration usually wants a completed response payload. An auditor wants a durable sequence. A metrics system wants counters and durations. These are different views of the same execution.

The client harness emits typed runtime events and attaches consumers to that stream. This replaces a growing collection of special callbacks with a shared vocabulary of facts.

Events describe what happened

The event envelope includes tenant identity and, where available, run, session and attempt IDs. Event classes are frozen at the model level and use explicit type names. Storage adds ordering information when events are persisted.

The vocabulary covers ingress, routing, hydration, assembly, attempts, model requests, tool execution, user-visible text, components and terminal outcomes. This granularity lets a run trace explain where a decision came from.

{
  "type": "tool.call.completed",
  "tenant_id": "tenant-example",
  "run_id": "run-example",
  "attempt_id": "attempt-example",
  "tool_call_id": "call-2",
  "tool_name": "get_task_status",
  "duration_ms": 180,
  "outcome": "ok"
}

The timing is illustrative. The useful contract is the correlation: the event belongs to a particular request and tool call, and its semantic outcome is available without parsing a log sentence.

Frozen event objects reduce accidental reassignment, but nested dictionaries still need disciplined ownership. “Immutable” should not become an excuse to pass a shared mutable payload to several consumers and then modify it later.

The result is a summary; the events are the explanation

A run result can say completed, include final text, usage and selected summary fields. It should not need to duplicate every emitted event. The durable log already contains the sequence.

This keeps the caller’s contract small while preserving debugging depth. A synchronous API can return the answer and a run ID. The operator can follow that ID to routing, model turns and tool calls.

It also avoids a second event history being built independently inside the result object. Two histories assembled by different code paths will eventually disagree.

Project the stream for the channel

The web path uses an SSE projection. It can show status changes, visible text and structured components. The synchronous path uses a buffer projection and returns the completed payload. Both invoke the same orchestrator.

The loop therefore does not need a branch saying “if WhatsApp, run this other algorithm.” Channel differences belong in input adaptation, prompt presentation rules and output projection.

This preserves evaluation parity. A test using the synchronous path exercises the same routing and tool logic as a messaging request. It still does not prove that the external messaging provider delivered the result; delivery is a later boundary.

Presentation tools are not domain actions

The model can request a card, chart or quick reply. The loop recognises these presentation calls and emits component events rather than executing a backend mutation.

A component stream can include start, delta and completion events. The UI should not assume a partially received JSON object is valid component props. It can display a placeholder while accumulating data, then validate the complete payload before rendering the final component.

Component names should come from an allowed registry. A model-provided name should not become arbitrary component import or script execution. The same principle applies to links and rich text: presentation is still an input boundary.

Keep internal narration separate

The raw model stream is useful for investigation. It is not automatically the guest transcript. The runtime distinguishes model text from assistant-visible text so it can suppress inter-step narration before data and action tools.

This prevents an implementation detail such as “calling the booking API now” from becoming a guest-facing message. It also makes the publication boundary explicit for final claims.

A browser cannot unsee a streamed sentence. Where correctness depends on tool evidence, use a committed answer stage or another explicit UI treatment. A final corrected message alone does not erase an earlier incorrect claim.

Sink isolation has limits

One telemetry consumer failing should not break unrelated consumers. The composite sink isolates failures so a metrics or tracing outage does not automatically destroy a guest response.

However, failure isolation is not durability. If the durable event consumer fails and the system continues, the missing evidence needs a visible degraded state or recovery path. Otherwise “sink-first” can be mistaken for “every event is guaranteed persisted.”

A robust design distinguishes best-effort telemetry from required audit facts. For a read-only answer, proceeding with reduced telemetry may be acceptable. For a consequential action, the policy may require durable action evidence before acknowledging success.

Delivery is another state machine

Generated, committed, queued for delivery, accepted by the provider and delivered to the user are different states. The run trace should carry enough correlation to connect them, but the agent service should not invent provider delivery status.

The backend owns the messaging integrations in this architecture. It can apply templates, account modes and provider-specific rules after the agent returns. A task-status update follows the same distinction: composition is not delivery.

This is particularly important for escalation. “An escalation item was recorded” and “the host was notified on a messaging channel” are different observable events. The final language should follow the stronger claim only when the stronger evidence exists.

Backpressure is part of streaming

A slow browser or event sink can turn a fast model stream into a growing memory buffer. Each projection needs a policy for queue limits, cancellation and dropped optional updates.

Status updates can often be coalesced. Tool terminal events and final answers cannot simply disappear. If a client disconnects, decide whether the run should stop or continue as durable background work; make that choice by product contract rather than an accidental socket exception.

Separate channel projections make these behaviours testable. Run the same event sequence through a fast consumer, a slow consumer and a disconnected consumer. The business outcome should remain consistent even when optional progress updates are coalesced or delivery is retried.

Test event meaning and order

Useful tests verify that a tool completion has a corresponding call identity, that terminal outcomes are not emitted twice, that a synchronous response matches the final committed result and that one broken optional sink does not prevent other sinks from receiving events.

For UI components, test partial argument chunks and malformed payloads. For delivery, test retries independently from model execution. Re-sending a message should not rerun a paid model turn or recreate a maintenance task unnecessarily.

The event spine makes the system easier to extend because each consumer interprets the same execution facts. It also makes disagreements visible: a polished answer, a failed tool and a missing notification can no longer hide inside one undifferentiated “success” response.

PART 13

The control plane: changing behaviour without losing control

An operator edits a tool description in the console. A developer ships a corrected handler. A worker continues using the registry loaded at startup. All three people can truthfully say they changed the system, while the next guest still receives the old behaviour.

The client platform addressed this by treating configuration as a product surface with a lifecycle, ownership and runtime inspection. A database editor alone would not have been enough.

Five configuration entities

The control plane contains prompt templates, tool definitions, skill definitions, agent configurations and model definitions. Each has a versioned lifecycle. Runtime reads use published records; drafts are an editing surface, not live behaviour.

Publishing one record can affect others. A skill pointing to an unpublished prompt or missing tool is not a complete deployable capability. Configuration needs dependency validation just as code does.

Code supplies the initial catalog

A control plane can begin with a code-managed catalog and add operator-managed configuration over time. Make the source of every active definition explicit. A fallback is useful only when its conditions and behaviour are known; silently switching catalogs during a failure can change the product while appearing to preserve availability.

The benefit is operational continuity. The risk is ambiguity about the active source. An operator needs to see whether a row came from code, a published database version or a fallback. Otherwise a successful console edit can appear to do nothing.

A deployability view should distinguish configuration-only changes from those requiring a compatible application image. That lets an operator see the next necessary action instead of repeatedly publishing a definition the runtime cannot execute.

Ownership must be explicit and reversible

Represent ownership as current state: an operator-managed definition should not be overwritten by seed reconciliation, while a code-managed definition can follow the release catalog. Transfer ownership through an explicit, audited action.

Make the transfer reversible. Inferring ownership from “someone edited this once” leaves no clear way to return a definition to release management. A current owner, a version and a recorded transfer make the rule explainable to both developers and operators.

The audit log still records history. It should not double as the only current-state authority for ownership.

A useful operator view answers four questions together: what is published, who owns it, what is running and what action would change it. Showing only the latest draft answers none of those completely.

Publication and reload are separate operations

A running process may hold resolved tools and skills in memory. Publishing a configuration record therefore needs a separate activation protocol: build a candidate catalog, validate dependencies, and atomically select it for new work. Avoid incrementally mutating a live catalog while requests are reading it.

Reload failure has different semantics from initial boot. At startup, a documented fallback may permit service availability. During a reload, replacing a working catalog with defaults can silently revert behaviour. Preserve the last known-good revision if the candidate is unavailable, incomplete or invalid; surface the failed activation to operators.

This is a good example of why “fail open” is not a universal reliability strategy. The safe fallback depends on what state is already serving traffic.

Local atomicity is not a distributed release

A process-local lock serialises reload requests for that worker. It does not prove that every replica has loaded the same configuration or that model-gateway reconciliation completed atomically with the local registry swap.

A multi-replica release protocol needs a configuration revision, each worker’s loaded revision and a convergence criterion. A successful reload response from one process is insufficient evidence that every serving replica has activated the candidate.

Pin the relevant definitions to one revision for the whole attempt. Without that boundary, a turn can route using one tool schema and execute against another after a reload. Retain previous immutable revisions until their in-flight users finish, then retire them according to a bounded retention policy.

Some behaviour still ships in code

A registered handler’s executable logic lives in the application image. Editing its database description changes model-facing guidance; it does not rewrite the Python implementation.

The deployability page marks these boundaries. A missing registered handler should be a deployment error, not an invitation to publish a tool that silently does nothing. Similarly, code-only prompt fragments need clear labeling so the console does not promise editability it cannot deliver.

This distinction is especially important for deterministic flows. The booking or issue policy may be partly configured, but the order of API calls and state updates remains code behaviour.

Treat a release as a tuple

To reproduce a run, the application image alone is insufficient. A useful release identity includes code, configuration, model mapping, tool schemas and evaluation data.

{
  "image_digest": "sha256:example",
  "config_revision": "config-42",
  "agent_version": 7,
  "skill_versions": {"example-maintenance-help": 4},
  "tool_schema_digest": "schema-example",
  "model_alias": "example-fast-model",
  "eval_dataset_revision": "hospitality-regression-12"
}

This reference release manifest brings the relevant identities together. Use resolved version identifiers, not mutable aliases alone. A replay should be able to identify the configuration and schemas that produced the original result, even if a provider’s live model behaviour cannot be reproduced exactly.

A practical change workflow

Start with a failing scenario. Identify the responsible layer using the trace. Edit the smallest appropriate surface: prompt, skill, tool mapping or code. Run targeted cases and the relevant regression slices. Compare the candidate to the baseline. Publish, reload, then inspect a fresh assembled turn to verify that the intended version reached execution.

If the change requires code, ship the compatible image before enabling configuration that depends on it. Rollback also needs both sides: previous code and a compatible configuration set. Reverting a container cannot undo an independently published prompt automatically.

The console’s playground and deployment controls support this workflow. They reduce the gap between “the edit was saved” and “the runtime used the edit.”

Test configuration like executable behaviour

Parity tests compare code-backed and database-backed resolutions. Contract checks validate references and handler availability. Reload tests simulate an unavailable database, an empty catalog and a partially failed downstream reconcile. Ownership tests prove that reseeding preserves console-managed rows and that release-to-code works.

Those tests protect a product whose behaviour changes outside normal source deployment. Once prompts, skills and tools are data, the configuration plane is part of the application’s executable surface and deserves the same engineering discipline.

PART 14

Evaluation starts with the operational scenario

An agent can produce a convincing answer while acting on the wrong stay. It can call the expected tool with the wrong dates. It can create a task and then tell the guest a visit was scheduled when no slot exists.

Evaluating only the final sentence misses those failures. The client platform built evaluation around cases with context, conversation turns, expected behaviour and captured runtime evidence.

A case is more than a question

A useful case includes the agent, channel, starting scope, prior history or user turns, deterministic expectations and a qualitative rubric. The same sentence can require a different action for a prospect, a guest with one stay and a guest with several stays.

The database-backed case model supports required and forbidden tools, no-tool expectations, required or forbidden phrases, regular expressions, expected status, minimum response length and limits on iterations, latency and cost. It also carries tags and workflow categories so results can be examined by scenario family.

{
  "title": "A repeated issue should not create another task",
  "audience": "guest",
  "channel": "whatsapp",
  "turns": [
    "The shower is leaking.",
    "It is still leaking. Has anyone been assigned?"
  ],
  "fixture": {"stay": "example-active-stay"},
  "expectations": {
    "minimum_response_characters": 20,
    "forbidden_phrases": ["your booking is confirmed"],
    "rubric": "Use the existing issue and report only evidenced assignment or slot details."
  },
  "tags": ["issues", "multi-turn", "duplicate-work"]
}

This is a teaching case skeleton. A runnable integration case also needs a controlled tenant, stay and backend fixture. “Has a reservation” without an actual test reservation is not enough to establish the intended starting state.

Replay the same runtime path

The live evaluation runner invokes the application’s runtime, while the offline harness can drive the synchronous chat path in bulk and persist reports. The console exposes both cases and results.

Multi-turn cases retain a stable session and accumulate replies as history. This is essential for booking references and issue follow-ups. Replaying each user message as a fresh session tests a different product.

However, replaying text alone does not recreate the world. Calendar availability, task state, configuration and model versions may have changed. A reproducible test records or controls those dependencies. Use fixtures for deterministic contract tests and a dedicated environment for end-to-end integration tests.

Separate three testing layers

A dry-run agent eval cannot prove a real maintenance task was assigned. A backend unit test cannot prove the model explained the outcome correctly. Both are valuable, but their claims should remain separate.

Give every evaluation suite an explicit execution policy: fixture-only, read-only integration, or controlled mutation in a test environment. Classify tools by side effect and verify that classification against their behaviour. A “dry run” label alone does not prove that reads are isolated or that a misclassified write cannot occur.

Put deterministic checks first

The runner combines applicable hard assertions with optional judge results. A failed hard assertion cannot be rescued by a flattering language score.

def passes(assertions, judge_pass=None):
    hard_pass = all(value for value in assertions if value is not None)
    return hard_pass and (judge_pass is None or judge_pass)

assert not passes([True, False], True)
assert passes([True, True], None)

This small example expresses the decision rule, but the selection of assertions is the real work. A required tool name checks intent, not necessarily success. For high-value workflows, add argument, result and backend-state assertions.

For a booking case, compare the selected property and dates. For an issue case, inspect the task outcome and assignee. For authority tests, prove that the forbidden handler was not invoked. A phrase check alone can be gamed accidentally by an answer that repeats the expected words while doing the wrong thing.

Build a failure taxonomy

The useful categories follow the pipeline: wrong route, missing context, wrong scope, stale evidence, invalid tool arguments, failed backend action, unsupported final claim and failed delivery. Each points to a different owner.

The case tags can also capture domain slices: inquiry, existing stay, multiple stays, issue dispatch, host analytics, multilingual phrasing and channel formatting. A global pass rate hides a regression concentrated in a small but consequential slice.

For example, improving FAQ answers can raise the overall score while booking selection gets worse. A release gate should protect critical workflows separately from the aggregate.

Turn real failures into durable cases

The control portal connects a persisted run to case authoring. A reviewer selects the relevant history and evidence, replaces private records with fixtures, and defines the expected behaviour. This shortens the path from an observed failure to a repeatable regression test.

The original response is not automatically the reference answer. The reviewer must decide what should have happened and preserve enough starting state to reproduce the failure. Validate case completeness before accepting it into a release suite: missing history, unavailable dependencies or truncated evidence can produce a misleading pass.

Before sharing a dataset beyond the authorised environment, replace guest identities, property access details and client-specific records with fixtures. Preserve the failure mechanism while removing private data.

Test variance, not only one lucky run

Model behaviour can vary across repeated calls. A scenario that passes once and fails four times is not fixed. Repeat the difficult cases under the same configuration and track failure frequency as well as examples.

A small repeated battery is particularly useful for continuation routing, ambiguous selection and unsupported-action claims. Use a fixed dataset and record model settings, configuration and dependency state. Otherwise a change in the test environment can be mistaken for a prompt improvement.

Repeated evaluation has a cost. Budget it explicitly and use deterministic checks to eliminate simple defects before paying for judge calls.

Release gates need coverage and completeness

An absolute pass-rate floor and a comparison to the baseline answer different questions. The first protects minimum quality; the second detects regression from the current release.

Also check that the run completed the intended case set. A candidate that evaluated only the easy half of the dataset can appear better. Missing cases, infrastructure errors and cancelled cases must be visible in the comparison.

A sensible gate combines critical-case assertions, required dataset coverage and acceptable regression bounds. It should not turn every minor wording difference into a release blocker, nor allow a severe authorization failure to disappear inside a high average.

The next article looks at the subjective layer: how to use judges without allowing a model-generated score to become the only evidence of quality.

PART 15

Judges, human review and regression evidence

An evaluation judge can confidently mark an answer wrong because it never saw the tool result that justified it. It can also reward a clear, friendly explanation that contains an invented operational claim. Adding an LLM judge does not remove the need to design the evaluation carefully.

The client platform exposes judge configuration, results, reasons, human review and baseline comparison in the console. The objective is to make the grading process inspectable, not to replace engineering judgement with a single score.

Score different qualities separately

Groundedness, helpfulness, escalation and format answer different questions. A configurable evaluation layer can vary the rubric, judge model, repeat count and aggregation rule while keeping those dimensions visible to reviewers.

These dimensions should remain distinct. An answer can be helpful but ungrounded, grounded but incomplete, or correct in content but unsuitable for a messaging channel. Collapsing them too early hides the reason a case failed.

A rubric should define the boundary between adjacent scores with domain examples. “Five means excellent” is not a reliable grading contract.

Give the judge the evidence it needs

For operational agents, a judge needs the request, relevant history, final response, selected tool arguments, tool outcomes and applicable context. Otherwise it is judging plausibility rather than support.

Build judge evidence from the correlated run record: the request, selected context, tool arguments, tool outcomes and final reply. Mark omitted or truncated information explicitly. A missing field must not be interpreted as proof that the underlying event did not occur.

A useful evidence packet makes omissions explicit:

{
  "request": "When will someone arrive?",
  "task_result": {"assigned": true, "slot": null},
  "reply": "A team member has been assigned; a time is not yet confirmed.",
  "evidence_complete": true,
  "rubric": "Do not reward an arrival time unless a booked slot supports it."
}

If the slot field is missing because capture was truncated, that is different from an authoritative slot: null. The grader should be allowed to express insufficient evidence rather than inventing certainty.

Groundedness is about claims, not word overlap

An answer can paraphrase a tool result correctly without sharing many words. Conversely, copying an irrelevant property paragraph can have high lexical overlap and still be wrong for the guest.

Groundedness review should identify material claims and their supporting evidence. For this domain, claims about price, dates, availability, assignment, notification and confirmation deserve particular attention. A small adjective about tone is not as consequential as a false booking confirmation.

Deterministic checks can verify many structured claims before the judge is invoked. The judge is most useful for interpretation and completeness where a simple assertion would be brittle.

Compare cases, not just averages

The comparison layer keys results by case slug. It identifies regressions, fixes, unchanged cases and cases present in only one run. Judge-score deltas are displayed alongside pass/fail changes.

Consider a hypothetical ten-case comparison: two regressions, two fixes and six unchanged. The pass rate might remain identical, but the release could be unacceptable if the regressions concern authority and the fixes concern wording.

def regressions(baseline, candidate):
    common = baseline.keys() & candidate.keys()
    return sorted(
        case for case in common
        if baseline[case] is True and candidate[case] is False
    )

assert regressions(
    {"booking": True, "faq": False},
    {"booking": False, "faq": True},
) == ["booking"]

The example is deliberately small. The console comparison additionally retains result IDs, tool information and judge deltas so a reviewer can open the evidence behind the change.

Pairwise judging and position bias

The implemented pairwise path compares candidate and baseline replies, then repeats the comparison with their order swapped. It maps the second verdict back to the original ordering and records whether the judgments agree.

This is useful because judges can favour whichever answer appears first or second. Swapping does not eliminate every bias, but an inconsistent result should not be treated as a confident win.

The comparison should use the same scenario evidence and keep the answer labels neutral. Candidate identity, expected winner and an author’s preferred outcome should not be part of the grading prompt.

A tie is a legitimate result. So is an undecided or inconsistent comparison. Forcing every pair into a winner produces precision the evidence does not support.

Human calibration measures the evaluator

The console supports human labels of pass, fail and unsure. The calibration view compares those labels with the automated verdict and reports agreement, false passes and false fails. In this implementation the automated verdict includes assertions and judges, so that comparison is broader than judge-only calibration.

The most important error depends on the use case. A false pass on an invented confirmation is costly. A false fail on a correct but terse answer wastes review effort. Report both rather than relying on agreement alone.

Suppose, as a worked example, reviewers label 80 outputs. The automated decision agrees on 68, falsely passes 5 and falsely fails 7. Agreement is 85%, but the five false passes need inspection before the evaluator is trusted as a release gate. The example is not a client benchmark.

Review sampling should include random outputs as well as known failures. Reviewing only the worst-looking cases produces a useful debugging queue but a biased estimate of overall agreement.

Keep evaluation spend visible

Judges consume tokens too. The system separates agent cost from judge cost in reports and exposes both in spend views. Multiple judges, repeated samples and order-swapped pairwise comparisons can multiply the bill even when the underlying agent task is cheap.

Start with deterministic assertions and one calibrated judge for the dimensions that need it. Add a panel when disagreement analysis justifies the extra cost. More judges do not automatically create independence if they share similar blind spots or incomplete evidence.

Pin the judge template and model setting with the report. Changing the grader at the same time as the agent makes a baseline comparison difficult to interpret.

Use disagreement to improve the system

When a human and judge disagree, inspect whether the issue is the agent, the rubric, missing evidence or an incorrect fixture. A strict judge is not necessarily a bad judge; a permissive one is not necessarily realistic.

The useful loop is: inspect the trace, correct the case or implementation, update the rubric if needed, and rerun a stable comparison. This keeps the evaluation system connected to actual operational failures rather than turning it into a leaderboard of unexplained numbers.

Judges are most valuable as review tools with evidence and calibration. The release decision remains a combination of hard constraints, scenario coverage, observed regressions and accountable human interpretation.

PART 16

The operator console is part of the agent product

When a guest says “the agent gave me the wrong information,” the team needs to identify the turn, reconstruct the context and see the action evidence. A chart showing total model requests is not enough.

The client platform included a dedicated Next.js control portal alongside the Python runtime. It made configuration, execution and evaluation visible through one operational surface. This was part of delivering the agent product, because someone had to operate the system after the initial implementation.

Organise the console around decisions

The portal includes pages for prompts, tools, skills, agents and models; a playground; tool-contract and deployability views; evaluation datasets and runs; offline reports; observability; and spend.

These pages are useful when they answer a sequence of questions:

1. What happened in this conversation?
2. What did the model receive and which tools were available?
3. Which operation actually executed?
4. Which configuration or code owns the behaviour?
5. How can the failure become a test?
6. Did the candidate change improve it without regressions?

This connects diagnosis to change. A dashboard that ends at a red error count leaves the operator dependent on a developer to continue the investigation.

Read a run as a timeline

The runtime records ingress, normalisation, routing, context hydration, prompt assembly, attempt events, model turns, tools and terminal outcomes. The console can present that sequence with shared identifiers.

A useful investigation starts at the first incorrect fact, not at the final sentence. If the wrong property was selected during stay resolution, the model may have faithfully answered from the wrong property’s data. If the tool failed, changing the response tone will not repair the backend call.

The run view should preserve several distinctions: model-request time versus tool time, raw arguments versus validated arguments, attempted action versus completed action, and runtime completion versus guest delivery.

Assembly inspection narrows the search

The playground’s assembly cards show route and model selection, prompt-fragment breakdown, exposed tools and context details. This is the direct answer to “did my change reach the model?”

An operator can discover that a skill was not selected, a published row was not loaded, a host rule was absent or a tool was filtered out. Without assembly evidence, those failures all look like a model ignoring instructions.

A multi-turn playground also needs a stable conversation identity and accumulated history. Otherwise it cannot reproduce the very behaviours—earlier selections and issue follow-ups—that make a concierge useful.

Four operational signals

The platform combines structured logs, metrics, traces and spend. Each serves a different purpose.

OpenTelemetry provides conventions for model and agent telemetry, but names and stability levels evolve. Pin the instrumentation schema used by the service and preserve the mapping to your own run IDs. OpenTelemetry generative AI conventions

The project’s tracing, metrics, error-reporting and LLM-tracing integrations remain separate consumers of runtime evidence. That allows each tool to serve its purpose without making one vendor’s trace format the application contract.

Measure stages, queues and outcomes

Measure routing, context building, execution, finalisation and total duration. Correlate tool outcomes with those stages so slow hydration, a slow provider and repeated repair calls can be distinguished. Routine dashboards should use logical operation labels; sensitive request details belong in restricted trace storage.

Add queue time wherever a semaphore or worker pool can delay admission. Handler timing alone excludes time spent waiting for permission to execute. A low average handler duration can therefore coexist with slow guest responses under load. Record admission, start and completion separately.

Metrics should use bounded labels such as agent, tool name, source, status and model family. Run IDs, guest IDs and raw URLs with arbitrary parameters belong in trace records, not high-cardinality metrics labels.

Cost belongs next to the run

The observability view includes priced usage by model. The spend view also includes evaluation judge costs. The point is not merely accounting; cost can reveal a behaviour regression.

A prompt change that causes three extra repair passes may preserve answer quality while worsening latency and cost. A classifier unexpectedly routing everything to a complex model is visible in the model distribution. An unknown model showing zero cost needs an explicit unpriced flag, otherwise the dashboard rewards missing metering.

The gateway provides an independent source for reconciliation. A persistent difference between gateway spend and run-attributed spend is a debugging signal: classifier calls, failed attempts, direct-provider bypass or cache accounting may be missing from one side.

A worked incident walkthrough

Imagine a guest reports that the agent promised someone would arrive at 14:00, but the team has no appointment. The investigation should follow a concrete chain:

Find conversation → open run → inspect selected stay
→ inspect report_issue result → check slot and assignee
→ inspect final claim → compare delivery payload
→ save a regression case with the expected wording and state

If the tool returned an assignee but no slot, the response made an unsupported claim. If the tool returned a slot that never reached the scheduling backend, the integration contract is broken. If the backend has the slot but staff were not notified, the delivery/workflow boundary needs attention.

The same visible complaint can have three different causes. A trace connected to domain evidence makes that distinction possible.

Capture enough, but not everywhere

Guest messages, access codes and internal business records should not be copied indiscriminately into logs and span attributes. Use bounded capture, field-aware redaction and separate access controls for detailed model input and output. Public examples can preserve the timing and failure mechanism using synthetic records.

Hashes, lengths, versions and IDs can explain prompt composition without storing the entire prompt in every backend. Detailed evidence can live in an access-controlled store with deliberate retention. Truncation should be marked so an evaluator or operator knows when the record is incomplete.

Redaction also needs domain awareness. Removing fields named password does not catch an access code embedded in a paragraph. Test both structured fields and free-text examples appropriate to the product.

The console should prevent misleading interpretations

A dry-run tool result should be visibly synthetic. A workflow-started event should not look like a completed workflow. A failed trace capture should not look like an empty but successful prompt. An unpriced run should not appear free.

These are interface design decisions with technical consequences. The portal should use the runtime’s semantic fields instead of inferring status from colour or the presence of a response string.

Human review actions also need provenance. Saving a verdict, publishing a prompt and resuming an approval are changes to the operational system. Their actor and time should be recorded just as carefully as model actions.

Close the loop

The strongest feature is the connection from an observed run to an evaluation case and then to a baseline comparison. It turns a one-off support incident into durable engineering knowledge.

The console’s role is to make that path short and inspectable. The agent can then improve through evidence-driven changes, while the team retains a clear view of what is serving users and what each change costs.

PART 17

Latency, concurrency and the cost of a completed task

A fast first token can hide a slow task. A cheap model can become expensive when it repeats tools or needs several repair passes. A service can accept many HTTP requests while only a small number of model streams make progress.

Performance work on an agentic harness needs to measure the complete path: admission, routing, context, model turns, tools, repairs, finalisation and delivery. The client runtime exposes several of these boundaries explicitly, which makes optimisation more precise.

Start with the critical path

For a turn with multiple model iterations, a useful decomposition is:

Total = admission wait + routing + context build
      + Σ(model request + tool batch + repair overhead)
      + finalisation + delivery

Independent hydration calls can overlap. Dependent tool calls cannot. Parallel tool latency is closer to the slowest required call plus scheduling overhead, while sequential tool latency accumulates.

Measure both time to first useful output and time to completed task. A status message saying “checking” can improve perceived responsiveness but does not reduce the time until a guest receives a verified answer.

Control resources independently

Admission, model streams, tool execution and domain fan-out consume different resources. Each needs an explicit concurrency limit, a queue policy and a cancellation rule. Set limits from provider quotas and measured dependency capacity, then validate them with the expected traffic mix.

A semaphore size is not a platform throughput result. Capacity depends on model duration, iteration count, provider quotas, backend latency, replicas and traffic mix. Report measurements with the workload and deployment shape so the reader can interpret what was actually tested.

For a synthetic capacity example, take three worker replicas, a limit of four concurrent composite tools per worker, and a maximum fan-out of six reads per composite tool. The upper bound is 3 × 4 × 6 = 72 simultaneous downstream reads. If the dependency can safely serve only 24, independently tuning each worker is insufficient: use a shared budget or partition the allowance across replicas.

Account for the caller’s deadline

Consider an illustrative caller deadline of 30 seconds. A budget of 4 seconds for routing, 3 for context, 20 for execution and 3 for finalisation and transfer fits within that envelope. These are example allocations for the calculation, not the client’s deployment settings.

If routing consumes 7 seconds and context consumes 4, retaining the full 20-second execution budget leaves too little time to return the result. The execution stage has at most 16 seconds after reserving 3 for finalisation. A component can stay within its own timeout while the end-to-end request still misses the caller’s deadline.

Propagate an absolute request deadline and allocate remaining time at each stage:

def remaining_budget(deadline, now, reserve=0.5):
    return max(0.0, deadline - now - reserve)

assert remaining_budget(30.0, 11.0) == 18.5

The reserve is an illustrative allowance for finalisation and response transfer. In production, compare against a monotonic deadline and pass each stage the smaller of its own limit and the remaining request budget. Include semaphore wait and retries in that accounting. If no useful budget remains, return the defined timeout outcome instead of starting work whose result cannot arrive in time.

Understand nested concurrency limits

A single model turn can request several tools, and one composite booking tool can fan out to several calendars. Limits at these layers govern different resources.

Raising the outer tool limit can multiply inner calendar traffic. Adding replicas multiplies process-local semaphores again. Provider limits and domain-service connection pools are shared constraints, so per-process limits need a deployment-level capacity plan.

An illustrative steady-state calculation uses Little’s Law: average in-flight work equals arrival rate times average time in the system. At two requests per second and eight seconds per request, average in-flight work is sixteen requests. That does not imply sixteen simultaneous provider calls, because requests spend time in other stages, but it gives a first consistency check for admission capacity.

At saturation, queueing dominates. Report queue age and rejected/admitted work, not only handler duration for requests that eventually reached execution.

Price the token classes correctly

Track ordinary input, output, cache-read and cache-creation tokens with model-specific rates. An unknown price should produce an explicit unpriced state. Resolve exact model revisions or deliberate aliases before applying a rate; a similar model name is not sufficient evidence of identical pricing.

Provider usage conventions differ. Some fields include cached input inside total input, while others report disjoint classes. Normalize usage before applying rates, or cache reads can be counted twice.

Using illustrative rates of $1 per million ordinary input tokens, $5 per million output tokens, $0.10 per million cached-read tokens and $1.25 per million cache-write tokens:

The warm-cache saving is about 65.5% for this complete example, not 90% of the whole request. The output cost remains, and creating the cache costs more than an ordinary read under these assumed rates. These are worked arithmetic using an example rate table, not a current provider quote or a client saving claim.

from decimal import Decimal

def cost_usd(tokens, rates_per_million):
    return sum(
        Decimal(tokens.get(kind, 0)) * Decimal(str(rate))
        for kind, rate in rates_per_million.items()
    ) / Decimal(1_000_000)

assert cost_usd(
    {"input": 1000, "output": 400, "cache_read": 8000},
    {"input": 1, "output": 5, "cache_read": 0.1},
) == Decimal("0.0038")

Count the work around the main model

Classification, judges, repair calls, failed requests, retrieval and workflow execution all contribute to cost. The client console separates agent and judge spend, while the model gateway supplies another accounting view.

LiteLLM virtual keys support model access controls, attribution and budgets at the gateway. Those controls apply to traffic that actually goes through the configured gateway path. A direct-provider fallback needs separate visibility and policy. LiteLLM virtual keys

For pricing a product, the useful denominator is a completed task. If a cheaper model reduces per-call cost but doubles unresolved cases, the cost per successful task can worsen. Include human review and escalation workload when comparing operating models.

Optimise by failure mechanism

If context dominates, parallelise independent reads and remove redundant evidence. If the model loops unnecessarily, improve tool contracts or move mandatory sequences into a composite flow. If routing dominates, examine when the classifier calls a model. If output dominates, constrain unnecessary verbosity while preserving the useful answer.

Prompt caching helps repeated stable prefixes. It does not fix slow backend APIs. More replicas help admission capacity. They do not remove a provider quota. A larger model can reduce some reasoning failures, but cannot correct an endpoint that ignores a parameter.

Load-test a realistic mix

Use a workload containing FAQs, booking searches, issue reports, host analysis, duplicates and degraded dependencies. Record latency percentiles, throughput, iteration counts, tool fan-out, queue wait, error rates and cost by scenario class.

Run warm-cache and cold-cache cases separately. Test bursts as well as steady load. Include a slow domain service and provider rate limiting. Confirm that the service remains within its resource limits and produces honest failure outcomes instead of continuing work whose response will be discarded.

Performance is successful when the task finishes correctly within a useful time and cost envelope. The harness makes those dimensions visible enough to improve together rather than optimising one misleading number.

PART 18

Reliability, release discipline and operational value

The most revealing agent test is often a process crash at an inconvenient moment. Kill the worker after a backend write but before the final answer. Restart it and deliver the event again. What happens to the task, the guest response and the audit trail?

That question brings together the whole harness: identities, tool receipts, durable state, events, configuration and delivery. It also explains why a production agent is much larger than its model loop.

Three idempotency boundaries

The system has to reason about at least three distinct duplicate risks.

Inbound idempotency prevents the same event from creating repeated agent runs. Action idempotency prevents repeated attempts from duplicating business mutations. Projection idempotency prevents retries from duplicating audit or downstream records.

One key cannot automatically solve all three. A key based on a run and tool-call identity can deduplicate retries of that call, yet change when a new run repeats the same business intent. Use a stable operation identity at the domain boundary when the intended mutation must survive a replay or a newly generated model plan.

The action key should represent the intended operation, not merely the latest network attempt. For a task create, that might be an application-generated request identity tied to the guest issue. The exact domain policy determines when two requests are duplicates and when they are legitimate new work.

An outbox moves retryable work off the response path

The implementation has a durable outbox for projections such as backend audit delivery, with retry scheduling and terminal failure after a configured number of attempts. This improves on fire-and-forget work that disappears when a process exits.

The outbox does not make delivery exactly once. A worker can send a request and crash before marking it sent. The recipient must deduplicate the retry or tolerate it safely.

A general outbox failure mode is easy to miss: FOR UPDATE SKIP LOCKED protects selected rows only while the transaction holds the locks. Ending that transaction before delivery without persisting ownership allows another worker to select the same pending row. A durable claim or lease makes ownership survive the short selection transaction.

The following standalone outbox pattern records a worker lease and token before network delivery:

WITH candidate AS (
  SELECT id FROM example_delivery_queue
  WHERE (status = 'pending' AND next_attempt_at <= now())
     OR (status = 'processing' AND lease_until < now())
  ORDER BY next_attempt_at
  FOR UPDATE SKIP LOCKED
  LIMIT 20
)
UPDATE example_delivery_queue AS o
SET status = 'processing',
    claim_token = :worker_claim,
    lease_until = now() + interval '60 seconds'
FROM candidate
WHERE o.id = candidate.id
RETURNING o.*;

The worker delivers outside the transaction, then marks completion only if its claim token still matches. Long deliveries need lease renewal or a larger bounded lease. The receiver still needs idempotency because a lease can expire after delivery but before acknowledgement.

Durability has a critical path

Batching event-log writes reduces database round trips but creates a window in which recent evidence exists only in memory. A normal terminal flush does not recover an uncommitted batch after a crash. Keep critical action receipts on a durable path and allow only explicitly optional progress events to use weaker capture guarantees.

Decide which facts must be durable before an action is acknowledged. A payment-like mutation needs a stronger receipt boundary than a transient “thinking” status. The durable business record and the agent’s audit log may be in different databases, so their consistency needs an explicit recovery strategy.

Likewise, independently writing a run result and enqueueing a projection is not automatically a transactional outbox across both operations. If they cannot share a transaction, add reconciliation that finds completed runs missing the expected projection.

Session-state locks solve a narrower problem

A database-backed state reducer can use a transaction-scoped lock or an optimistic version check to prevent updates from overwriting the same old snapshot. This protects the state transition: load current state, apply the reducer, then store the new version.

It does not serialise the entire agent run, including every external API call. Two turns can still observe changing domain state or interleave actions outside that transaction. If the product requires strict conversational turn ordering, introduce a session admission lane or equivalent fencing around the broader workflow.

OpenClaw’s documented session lanes are a useful comparison for that wider execution-order concern. The transferable idea is to define which writer owns the session at each point, rather than assuming stateless request handling removes races. OpenClaw agent loop

Ship the runtime and its configuration together

The release unit includes the Python image, database migrations, compatible configuration, gateway mappings and the console. A changed tool handler can require new configuration, while a new configuration row can require a handler absent from an older image.

A practical sequence is to apply additive migrations, deploy compatible code, validate published configuration, reload workers, verify convergence and run a controlled canary. Rollback must identify the compatible configuration revision as well as the previous image.

Containerised runtime services, durable storage and a separate admin frontend form a portable deployment boundary. Place replicas and dependencies according to latency, failure-domain and operational requirements. Cloud-specific managed services can implement those roles without changing the contracts for identity, configuration or recovery.

Readiness must exercise the real seam

Consider a dependency upgrade that changes a model adapter’s expected response shape. The process still starts and a basic health endpoint remains green, but real requests fail. Compatibility tests at the transport seam catch a failure that process health cannot.

This suggests three different checks. Liveness answers whether the process is running. Readiness checks required local dependencies and configured runtime paths. A controlled synthetic turn verifies the provider and tool seam end to end.

Do not make every frequent readiness probe perform a paid model call. Run a budgeted synthetic transaction on a deliberate cadence or deployment gate, with a scoped test identity and non-mutating fixture. Record its result separately from process health.

Design degraded modes by capability

An optional telemetry outage can permit continued service with an alert. Missing property knowledge should lead to a limited answer or handoff. Missing authorization context should prevent the action. A failed configuration reload should preserve the last working catalog.

The fallback should be specific enough to test. “Graceful degradation” is not an implementation until the team can say what answer, state and event should result from each dependency failure.

Use failure injection at the seams: unavailable database, slow calendar, malformed tool result, provider timeout, cancelled client and failed audit delivery. Check recovery as well as the initial failure.

What was developed

The client product brought together a shared Python harness for two personas, dynamic capability composition, model adapters, scoped tools, booking and issue flows, durable state and a versioned configuration plane. The control portal exposed the assembled turn, tool contracts, runtime traces, evaluation workflows and cost.

These are concrete delivery outcomes. They make the system easier to change, diagnose and operate. Claims about revenue, labour savings or conversion require operational measurement beyond the implementation itself.

A useful business measurement plan compares eligible booking inquiries, verified booking progression, correctly dispatched issues, time to an accountable owner, reopened issues and operator review effort. Keep the traffic mix and definitions consistent across the comparison.

The next reusable layer

The reusable asset is the set of contracts: inbound event, route, attempt, tool, result, runtime event and configuration lifecycle. The hospitality-specific flows remain adapters around those contracts.

A public reference implementation can use a fictional coworking or small-hotel domain with synthetic records. It can demonstrate duplicate delivery, scope rejection, booking-state continuity, approval binding, crash recovery and regression evaluation without exposing client code or live integrations.

Before broadening any harness, test the narrow guarantees under failure: end-to-end deadlines, action-bound approvals, durable delivery claims, immutable per-run configuration and explicit action receipts. These checks establish reusable engineering contracts without depending on a particular client’s routes, storage schema or service layout.

My contribution

My contribution covered translating hospitality workflows into the architecture and implementation of the custom harness, its domain tools and its operator controls. The work connected conversational behaviour with backend state, evaluation evidence and the practical requirements of running and improving the product.