From shadow mode to a supervised operating loop

The fastest way to lose confidence in an agent platform is to give it a broad mandate before the first workflow is understood. I would introduce this system through a narrow operating experiment: one business task, a clear acceptance condition, a small set of tools and an explicit review budget.

The schedule below is an illustrative thirty-day plan. Progress depends on passing the gates, not on the date arriving. The platform remains experimental internal R&D; the goal of the pilot is to discover which work can be delegated usefully and where the founder still needs to decide.

Days one to five: map the real work

Choose a workflow such as preparing quotes from inbound enquiries. Sample representative cases, including difficult ones. Record the source inputs, manual steps, decision points, artifacts, active handling time and common corrections.

Define the acceptance condition in business language: “a current quote with the correct recipient, scope, price, delivery assumptions and evidence is ready for founder review.” Do not start with “the agent uses five tools.” Tool calls are implementation details; the reviewed quote is the outcome.

Create a baseline worksheet with task identity, complexity, handling minutes, elapsed waiting time, outcome and correction minutes. Separate unavailable facts from genuine negative evidence. The agent should know when it needs to ask a question rather than invent a requirement.

Days six to ten: run historical and synthetic cases

Build fixtures from permitted, anonymised examples and add synthetic failures that are difficult to collect safely: duplicate events, stale approvals, lost acknowledgements and hostile attachments. Keep the expected outcome separate from the model-visible input.

Run the deterministic contracts first. Then run model episodes with external effects disabled. Inspect every material disagreement with the founder's reference decision and record the reason. A corrected prompt is only one possible fix; the underlying problem may be missing state, an ambiguous tool schema or a weak identity mapping.

pilot_gate:
  workflow: quote_preparation
  phase: offline
  required:
    - scenario_versions_recorded
    - critical_failures_resolved
    - stale_review_rejected
    - duplicate_effect_recovery_exercised
    - founder_can_explain_pending_actions
  measured:
    accepted_tasks: null
    review_minutes: null
    total_cost_minor: null

The unmeasured fields remain null. Filling them with optimistic estimates would make the gate look more complete without providing evidence.

Days eleven to seventeen: shadow live inputs

Connect one read-only source and let the candidate prepare proposals in parallel with normal work. It may read permitted records and write internal drafts, but it cannot contact customers or alter external systems.

Compare its proposal with what actually happened. Capture the evidence it missed, the unnecessary work it attempted and the time required to review it. Keep account and task identities stable so the comparison is reproducible.

Shadow mode should not double the founder's workload indefinitely. Set a daily review budget and sample deliberately if volume is high. A platform that requires constant inspection of every intermediate thought is not yet providing useful operational leverage.

Days eighteen to twenty-four: allow reviewed effects

Enable one low-scope external action after the founder reviews its exact parameters. Recheck current state, permissions, budget and connection status immediately before execution. Record the provider receipt and make uncertainty visible in the cockpit.

Start with an action whose recovery is understood. A calendar draft or reviewed status message may be easier to pilot than a payment or a contractual amendment. Reversibility is useful, but even a reversible action can confuse a customer; still bind it to a current decision.

def may_dispatch(*,review_current,connection_enabled,stop_active,budget_reserved):
    return review_current and connection_enabled and not stop_active and budget_reserved

assert may_dispatch(review_current=True, connection_enabled=True,
                    stop_active=False, budget_reserved=True)
assert not may_dispatch(review_current=True, connection_enabled=True,
                        stop_active=True, budget_reserved=True)

This is a compact gate example, not the full production policy. Real execution also checks authenticated identity, resource scope, content revision and domain-specific conditions. The important behaviour is that a stop control is evaluated at dispatch time, including for previously queued work.

Days twenty-five to thirty: decide whether to widen

Review accepted outcomes, critical failures, founder minutes, unresolved exceptions, total cost and evidence quality. Compare them with the baseline using the same task boundaries. Identify whether the platform saved search time, reduced errors, accelerated a handoff or simply moved work into a review queue.

A sensible decision may be to keep drafting automated and sending reviewed. Another may be to stop an expensive enrichment step. Widen only one dimension at a time: another task class, another connector or a carefully bounded action permission. Changing all three makes it hard to explain what caused an improvement or failure.

A canary needs an eligible task population, stable assignment, a small exposure limit and a direct abort path. It should not include sensitive operations merely to make the autonomy percentage look better.

Keep a current implementation inventory

This inventory is intentionally specific. It lets a reader distinguish what can be run today from the next implementation work without reading the entire repository.

Finish with an operating decision

The pilot report should name the workflow, observation period, task count, failures, supervision, cost, remaining limitations and the next authorised scope. Keep the supporting traces and fixtures. A short report with clear evidence is more valuable than a broad claim that the company now runs itself.

The long-term ambition is still substantial: one founder coordinating sales, delivery, support, finance and improvement through a shared operating platform. The route there is a sequence of dependable loops. Each loop earns more responsibility by keeping its promises, exposing uncertainty and making the founder's decisions easier to carry through.