The control plane: changing behaviour without losing control

An operator edits a tool description in the console. A developer ships a corrected handler. A worker continues using the registry loaded at startup. All three people can truthfully say they changed the system, while the next guest still receives the old behaviour.

The client platform addressed this by treating configuration as a product surface with a lifecycle, ownership and runtime inspection. A database editor alone would not have been enough.

Five configuration entities

The control plane contains prompt templates, tool definitions, skill definitions, agent configurations and model definitions. Each has a versioned lifecycle. Runtime reads use published records; drafts are an editing surface, not live behaviour.

Publishing one record can affect others. A skill pointing to an unpublished prompt or missing tool is not a complete deployable capability. Configuration needs dependency validation just as code does.

Code supplies the initial catalog

A control plane can begin with a code-managed catalog and add operator-managed configuration over time. Make the source of every active definition explicit. A fallback is useful only when its conditions and behaviour are known; silently switching catalogs during a failure can change the product while appearing to preserve availability.

The benefit is operational continuity. The risk is ambiguity about the active source. An operator needs to see whether a row came from code, a published database version or a fallback. Otherwise a successful console edit can appear to do nothing.

A deployability view should distinguish configuration-only changes from those requiring a compatible application image. That lets an operator see the next necessary action instead of repeatedly publishing a definition the runtime cannot execute.

Ownership must be explicit and reversible

Represent ownership as current state: an operator-managed definition should not be overwritten by seed reconciliation, while a code-managed definition can follow the release catalog. Transfer ownership through an explicit, audited action.

Make the transfer reversible. Inferring ownership from “someone edited this once” leaves no clear way to return a definition to release management. A current owner, a version and a recorded transfer make the rule explainable to both developers and operators.

The audit log still records history. It should not double as the only current-state authority for ownership.

A useful operator view answers four questions together: what is published, who owns it, what is running and what action would change it. Showing only the latest draft answers none of those completely.

Publication and reload are separate operations

A running process may hold resolved tools and skills in memory. Publishing a configuration record therefore needs a separate activation protocol: build a candidate catalog, validate dependencies, and atomically select it for new work. Avoid incrementally mutating a live catalog while requests are reading it.

Reload failure has different semantics from initial boot. At startup, a documented fallback may permit service availability. During a reload, replacing a working catalog with defaults can silently revert behaviour. Preserve the last known-good revision if the candidate is unavailable, incomplete or invalid; surface the failed activation to operators.

This is a good example of why “fail open” is not a universal reliability strategy. The safe fallback depends on what state is already serving traffic.

Local atomicity is not a distributed release

A process-local lock serialises reload requests for that worker. It does not prove that every replica has loaded the same configuration or that model-gateway reconciliation completed atomically with the local registry swap.

A multi-replica release protocol needs a configuration revision, each worker’s loaded revision and a convergence criterion. A successful reload response from one process is insufficient evidence that every serving replica has activated the candidate.

Pin the relevant definitions to one revision for the whole attempt. Without that boundary, a turn can route using one tool schema and execute against another after a reload. Retain previous immutable revisions until their in-flight users finish, then retire them according to a bounded retention policy.

Some behaviour still ships in code

A registered handler’s executable logic lives in the application image. Editing its database description changes model-facing guidance; it does not rewrite the Python implementation.

The deployability page marks these boundaries. A missing registered handler should be a deployment error, not an invitation to publish a tool that silently does nothing. Similarly, code-only prompt fragments need clear labeling so the console does not promise editability it cannot deliver.

This distinction is especially important for deterministic flows. The booking or issue policy may be partly configured, but the order of API calls and state updates remains code behaviour.

Treat a release as a tuple

To reproduce a run, the application image alone is insufficient. A useful release identity includes code, configuration, model mapping, tool schemas and evaluation data.

{
  "image_digest": "sha256:example",
  "config_revision": "config-42",
  "agent_version": 7,
  "skill_versions": {"example-maintenance-help": 4},
  "tool_schema_digest": "schema-example",
  "model_alias": "example-fast-model",
  "eval_dataset_revision": "hospitality-regression-12"
}

This reference release manifest brings the relevant identities together. Use resolved version identifiers, not mutable aliases alone. A replay should be able to identify the configuration and schemas that produced the original result, even if a provider’s live model behaviour cannot be reproduced exactly.

A practical change workflow

Start with a failing scenario. Identify the responsible layer using the trace. Edit the smallest appropriate surface: prompt, skill, tool mapping or code. Run targeted cases and the relevant regression slices. Compare the candidate to the baseline. Publish, reload, then inspect a fresh assembled turn to verify that the intended version reached execution.

If the change requires code, ship the compatible image before enabling configuration that depends on it. Rollback also needs both sides: previous code and a compatible configuration set. Reverting a container cannot undo an independently published prompt automatically.

The console’s playground and deployment controls support this workflow. They reduce the gap between “the edit was saved” and “the runtime used the edit.”

Test configuration like executable behaviour

Parity tests compare code-backed and database-backed resolutions. Contract checks validate references and handler availability. Reload tests simulate an unavailable database, an empty catalog and a partially failed downstream reconcile. Ownership tests prove that reseeding preserves console-managed rows and that release-to-code works.

Those tests protect a product whose behaviour changes outside normal source deployment. Once prompts, skills and tools are data, the configuration plane is part of the application’s executable surface and deserves the same engineering discipline.