Reliability, release discipline and operational value

The most revealing agent test is often a process crash at an inconvenient moment. Kill the worker after a backend write but before the final answer. Restart it and deliver the event again. What happens to the task, the guest response and the audit trail?

That question brings together the whole harness: identities, tool receipts, durable state, events, configuration and delivery. It also explains why a production agent is much larger than its model loop.

Three idempotency boundaries

The system has to reason about at least three distinct duplicate risks.

Inbound idempotency prevents the same event from creating repeated agent runs. Action idempotency prevents repeated attempts from duplicating business mutations. Projection idempotency prevents retries from duplicating audit or downstream records.

One key cannot automatically solve all three. A key based on a run and tool-call identity can deduplicate retries of that call, yet change when a new run repeats the same business intent. Use a stable operation identity at the domain boundary when the intended mutation must survive a replay or a newly generated model plan.

The action key should represent the intended operation, not merely the latest network attempt. For a task create, that might be an application-generated request identity tied to the guest issue. The exact domain policy determines when two requests are duplicates and when they are legitimate new work.

An outbox moves retryable work off the response path

The implementation has a durable outbox for projections such as backend audit delivery, with retry scheduling and terminal failure after a configured number of attempts. This improves on fire-and-forget work that disappears when a process exits.

The outbox does not make delivery exactly once. A worker can send a request and crash before marking it sent. The recipient must deduplicate the retry or tolerate it safely.

A general outbox failure mode is easy to miss: FOR UPDATE SKIP LOCKED protects selected rows only while the transaction holds the locks. Ending that transaction before delivery without persisting ownership allows another worker to select the same pending row. A durable claim or lease makes ownership survive the short selection transaction.

The following standalone outbox pattern records a worker lease and token before network delivery:

WITH candidate AS (
  SELECT id FROM example_delivery_queue
  WHERE (status = 'pending' AND next_attempt_at <= now())
     OR (status = 'processing' AND lease_until < now())
  ORDER BY next_attempt_at
  FOR UPDATE SKIP LOCKED
  LIMIT 20
)
UPDATE example_delivery_queue AS o
SET status = 'processing',
    claim_token = :worker_claim,
    lease_until = now() + interval '60 seconds'
FROM candidate
WHERE o.id = candidate.id
RETURNING o.*;

The worker delivers outside the transaction, then marks completion only if its claim token still matches. Long deliveries need lease renewal or a larger bounded lease. The receiver still needs idempotency because a lease can expire after delivery but before acknowledgement.

Durability has a critical path

Batching event-log writes reduces database round trips but creates a window in which recent evidence exists only in memory. A normal terminal flush does not recover an uncommitted batch after a crash. Keep critical action receipts on a durable path and allow only explicitly optional progress events to use weaker capture guarantees.

Decide which facts must be durable before an action is acknowledged. A payment-like mutation needs a stronger receipt boundary than a transient “thinking” status. The durable business record and the agent’s audit log may be in different databases, so their consistency needs an explicit recovery strategy.

Likewise, independently writing a run result and enqueueing a projection is not automatically a transactional outbox across both operations. If they cannot share a transaction, add reconciliation that finds completed runs missing the expected projection.

Session-state locks solve a narrower problem

A database-backed state reducer can use a transaction-scoped lock or an optimistic version check to prevent updates from overwriting the same old snapshot. This protects the state transition: load current state, apply the reducer, then store the new version.

It does not serialise the entire agent run, including every external API call. Two turns can still observe changing domain state or interleave actions outside that transaction. If the product requires strict conversational turn ordering, introduce a session admission lane or equivalent fencing around the broader workflow.

OpenClaw’s documented session lanes are a useful comparison for that wider execution-order concern. The transferable idea is to define which writer owns the session at each point, rather than assuming stateless request handling removes races. OpenClaw agent loop

Ship the runtime and its configuration together

The release unit includes the Python image, database migrations, compatible configuration, gateway mappings and the console. A changed tool handler can require new configuration, while a new configuration row can require a handler absent from an older image.

A practical sequence is to apply additive migrations, deploy compatible code, validate published configuration, reload workers, verify convergence and run a controlled canary. Rollback must identify the compatible configuration revision as well as the previous image.

Containerised runtime services, durable storage and a separate admin frontend form a portable deployment boundary. Place replicas and dependencies according to latency, failure-domain and operational requirements. Cloud-specific managed services can implement those roles without changing the contracts for identity, configuration or recovery.

Readiness must exercise the real seam

Consider a dependency upgrade that changes a model adapter’s expected response shape. The process still starts and a basic health endpoint remains green, but real requests fail. Compatibility tests at the transport seam catch a failure that process health cannot.

This suggests three different checks. Liveness answers whether the process is running. Readiness checks required local dependencies and configured runtime paths. A controlled synthetic turn verifies the provider and tool seam end to end.

Do not make every frequent readiness probe perform a paid model call. Run a budgeted synthetic transaction on a deliberate cadence or deployment gate, with a scoped test identity and non-mutating fixture. Record its result separately from process health.

Design degraded modes by capability

An optional telemetry outage can permit continued service with an alert. Missing property knowledge should lead to a limited answer or handoff. Missing authorization context should prevent the action. A failed configuration reload should preserve the last working catalog.

The fallback should be specific enough to test. “Graceful degradation” is not an implementation until the team can say what answer, state and event should result from each dependency failure.

Use failure injection at the seams: unavailable database, slow calendar, malformed tool result, provider timeout, cancelled client and failed audit delivery. Check recovery as well as the initial failure.

What was developed

The client product brought together a shared Python harness for two personas, dynamic capability composition, model adapters, scoped tools, booking and issue flows, durable state and a versioned configuration plane. The control portal exposed the assembled turn, tool contracts, runtime traces, evaluation workflows and cost.

These are concrete delivery outcomes. They make the system easier to change, diagnose and operate. Claims about revenue, labour savings or conversion require operational measurement beyond the implementation itself.

A useful business measurement plan compares eligible booking inquiries, verified booking progression, correctly dispatched issues, time to an accountable owner, reopened issues and operator review effort. Keep the traffic mix and definitions consistent across the comparison.

The next reusable layer

The reusable asset is the set of contracts: inbound event, route, attempt, tool, result, runtime event and configuration lifecycle. The hospitality-specific flows remain adapters around those contracts.

A public reference implementation can use a fictional coworking or small-hotel domain with synthetic records. It can demonstrate duplicate delivery, scope rejection, booking-state continuity, approval binding, crash recovery and regression evaluation without exposing client code or live integrations.

Before broadening any harness, test the narrow guarantees under failure: end-to-end deadlines, action-bound approvals, durable delivery claims, immutable per-run configuration and explicit action receipts. These checks establish reusable engineering contracts without depending on a particular client’s routes, storage schema or service layout.

My contribution

My contribution covered translating hospitality workflows into the architecture and implementation of the custom harness, its domain tools and its operator controls. The work connected conversational behaviour with backend state, evaluation evidence and the practical requirements of running and improving the product.