← Series contents · Complete reading view
The operator console is part of the agent product
When a guest says “the agent gave me the wrong information,” the team needs to identify the turn, reconstruct the context and see the action evidence. A chart showing total model requests is not enough.
The client platform included a dedicated Next.js control portal alongside the Python runtime. It made configuration, execution and evaluation visible through one operational surface. This was part of delivering the agent product, because someone had to operate the system after the initial implementation.
Organise the console around decisions
The portal includes pages for prompts, tools, skills, agents and models; a playground; tool-contract and deployability views; evaluation datasets and runs; offline reports; observability; and spend.
These pages are useful when they answer a sequence of questions:
- What happened in this conversation?
- What did the model receive and which tools were available?
- Which operation actually executed?
- Which configuration or code owns the behaviour?
- How can the failure become a test?
- Did the candidate change improve it without regressions?
This connects diagnosis to change. A dashboard that ends at a red error count leaves the operator dependent on a developer to continue the investigation.
Read a run as a timeline
The runtime records ingress, normalisation, routing, context hydration, prompt assembly, attempt events, model turns, tools and terminal outcomes. The console can present that sequence with shared identifiers.
A useful investigation starts at the first incorrect fact, not at the final sentence. If the wrong property was selected during stay resolution, the model may have faithfully answered from the wrong property’s data. If the tool failed, changing the response tone will not repair the backend call.
The run view should preserve several distinctions: model-request time versus tool time, raw arguments versus validated arguments, attempted action versus completed action, and runtime completion versus guest delivery.
Assembly inspection narrows the search
The playground’s assembly cards show route and model selection, prompt-fragment breakdown, exposed tools and context details. This is the direct answer to “did my change reach the model?”
An operator can discover that a skill was not selected, a published row was not loaded, a host rule was absent or a tool was filtered out. Without assembly evidence, those failures all look like a model ignoring instructions.
A multi-turn playground also needs a stable conversation identity and accumulated history. Otherwise it cannot reproduce the very behaviours—earlier selections and issue follow-ups—that make a concierge useful.
Four operational signals
The platform combines structured logs, metrics, traces and spend. Each serves a different purpose.
| Signal | Useful question | Poor substitute |
|---|---|---|
| Logs | What specific error or decision occurred? | A counter without context |
| Metrics | Is failure or latency increasing across traffic? | Searching individual transcripts |
| Traces/events | Which stage caused this request’s outcome? | A final answer alone |
| Spend | What did this path and model choice cost? | Aggregate token count without rates |
OpenTelemetry provides conventions for model and agent telemetry, but names and stability levels evolve. Pin the instrumentation schema used by the service and preserve the mapping to your own run IDs. OpenTelemetry generative AI conventions
The project’s tracing, metrics, error-reporting and LLM-tracing integrations remain separate consumers of runtime evidence. That allows each tool to serve its purpose without making one vendor’s trace format the application contract.
Measure stages, queues and outcomes
Measure routing, context building, execution, finalisation and total duration. Correlate tool outcomes with those stages so slow hydration, a slow provider and repeated repair calls can be distinguished. Routine dashboards should use logical operation labels; sensitive request details belong in restricted trace storage.
Add queue time wherever a semaphore or worker pool can delay admission. Handler timing alone excludes time spent waiting for permission to execute. A low average handler duration can therefore coexist with slow guest responses under load. Record admission, start and completion separately.
Metrics should use bounded labels such as agent, tool name, source, status and model family. Run IDs, guest IDs and raw URLs with arbitrary parameters belong in trace records, not high-cardinality metrics labels.
Cost belongs next to the run
The observability view includes priced usage by model. The spend view also includes evaluation judge costs. The point is not merely accounting; cost can reveal a behaviour regression.
A prompt change that causes three extra repair passes may preserve answer quality while worsening latency and cost. A classifier unexpectedly routing everything to a complex model is visible in the model distribution. An unknown model showing zero cost needs an explicit unpriced flag, otherwise the dashboard rewards missing metering.
The gateway provides an independent source for reconciliation. A persistent difference between gateway spend and run-attributed spend is a debugging signal: classifier calls, failed attempts, direct-provider bypass or cache accounting may be missing from one side.
A worked incident walkthrough
Imagine a guest reports that the agent promised someone would arrive at 14:00, but the team has no appointment. The investigation should follow a concrete chain:
Find conversation → open run → inspect selected stay
→ inspect report_issue result → check slot and assignee
→ inspect final claim → compare delivery payload
→ save a regression case with the expected wording and state
If the tool returned an assignee but no slot, the response made an unsupported claim. If the tool returned a slot that never reached the scheduling backend, the integration contract is broken. If the backend has the slot but staff were not notified, the delivery/workflow boundary needs attention.
The same visible complaint can have three different causes. A trace connected to domain evidence makes that distinction possible.
Capture enough, but not everywhere
Guest messages, access codes and internal business records should not be copied indiscriminately into logs and span attributes. Use bounded capture, field-aware redaction and separate access controls for detailed model input and output. Public examples can preserve the timing and failure mechanism using synthetic records.
Hashes, lengths, versions and IDs can explain prompt composition without storing the entire prompt in every backend. Detailed evidence can live in an access-controlled store with deliberate retention. Truncation should be marked so an evaluator or operator knows when the record is incomplete.
Redaction also needs domain awareness. Removing fields named password does not catch an access code embedded in a paragraph. Test both structured fields and free-text examples appropriate to the product.
The console should prevent misleading interpretations
A dry-run tool result should be visibly synthetic. A workflow-started event should not look like a completed workflow. A failed trace capture should not look like an empty but successful prompt. An unpriced run should not appear free.
These are interface design decisions with technical consequences. The portal should use the runtime’s semantic fields instead of inferring status from colour or the presence of a response string.
Human review actions also need provenance. Saving a verdict, publishing a prompt and resuming an approval are changes to the operational system. Their actor and time should be recorded just as carefully as model actions.
Close the loop
The strongest feature is the connection from an observed run to an evaluation case and then to a baseline comparison. It turns a one-off support incident into durable engineering knowledge.
The console’s role is to make that path short and inspectable. The agent can then improve through evidence-driven changes, while the team retains a clear view of what is serving users and what each change costs.