Operating the platform: capacity, cost and recovery

The administrative assistant is only one part of the workload. A clinic also loads patient screens, searches appointments, receives provider callbacks and runs scheduled outreach. Capacity planning has to protect those operations when model latency or a messaging backlog increases.

The inspected repository contains the ingredients for these paths, but it does not supply a verified production capacity report. The numbers in this chapter are a worked sizing profile for a supervised pilot, followed by the measurements I would use to replace them.

Separate interactive work from background work

I would start with a web tier, API tier, durable worker tier, PostgreSQL, Redis and external model/connector dependencies. The API prepares and confirms actions. Workers handle outbox events, reconciliation and scheduled reminders. Long-lived model calls should not hold database transactions open.

For a small pilot, an initial allocation might be two API instances at 2 vCPU and 4 GiB each, two worker instances at 2 vCPU and 4 GiB each, a managed PostgreSQL instance with 2–4 vCPU and 8–16 GiB, and a small managed Redis deployment. These are starting assumptions, not claimed historical machines or a validated minimum.

If inference is hosted, these application nodes do not need GPUs. If inference is self-hosted, it becomes a separate serving workload with its own memory, context-length, batching and latency measurements. Four billion parameters at four bits require roughly 2 GB for weights alone; that arithmetic excludes runtime overhead, higher-precision components, KV cache and concurrency. It is not a GPU recommendation.

Use arrival rate and service time together

Assume 60 active coordinators, each issuing one assistant request every 45 seconds during a busy period. That is about 1.33 requests per second. A tenfold burst is 13.3 requests per second. At a mean model wait of 2.5 seconds, Little's Law suggests roughly 33 concurrent model requests during that sustained burst, before queueing effects.

active_staff = 60
seconds_between_requests = 45
burst_multiplier = 10
mean_model_seconds = 2.5

arrival_rate = active_staff / seconds_between_requests
burst_rate = arrival_rate * burst_multiplier
in_flight = burst_rate * mean_model_seconds
print(round(burst_rate, 2), round(in_flight, 2))
# 13.33 requests/second, 33.33 mean in-flight requests

I would compare that concurrency with provider request and token limits. More API replicas will not solve a shared model quota. Admission control should return a clear busy state or queue estimate instead of creating an unbounded backlog.

Budget database connections independently

Suppose two API processes and two worker processes each allow eight database connections. That creates a possible 32 application connections, before migrations, monitoring or other services. Reserve headroom and verify pool behavior during timeouts.

An asynchronous Python server can await many network requests, but synchronous database calls still occupy execution resources. I would measure event-loop stalls and thread-pool saturation rather than infer capacity from the async keyword. CPU-bound parsing or large document processing should not share the interactive request path without a measured budget.

For workers, scale on queue age and arrival rate, not only CPU. A worker waiting on a provider can have low CPU while the queue grows. Track the oldest actionable event and the count of unknown outcomes awaiting reconciliation.

Define service objectives at the right boundary

My initial targets would distinguish proposal preparation, confirmation and eventual delivery. A proposal latency objective excludes time waiting for the user. Booking success requires a persisted or externally verified appointment. Delivery latency starts after an eligible message becomes dispatchable.

reference_objectives:
  proposal_ready_p95_ms: 2500
  local_confirmation_p95_ms: 500
  outbox_oldest_ready_event_seconds: 30
  reconciliation_oldest_unknown_seconds: 300
  interactive_availability_monthly: 99.9

These values are proposed pilot targets. They require load tests and operating agreement. A 99.9% availability target allows about 43.2 minutes of unavailability in a 30-day month; whether that is acceptable depends on the clinic's fallback workflow. It does not imply that every external provider meets the same objective.

Instrument the operation, not sensitive content

I would create spans for context loading, interpretation, policy retrieval, proposal persistence, confirmation validation, connector submission and reconciliation. Each span should carry workflow and operation references, version information, duration and outcome. Avoid patient names, phone numbers and full prompts in general metrics.

The original tracing path includes useful hooks, but some counts are derived from presentation artifacts such as cards. I would count actual tool executions and committed effects separately. A response with three cards is not evidence of three successful operations.

Cost per completed task should include model input/output, retries, connector charges, infrastructure and human review time. Divide by successfully completed workflows, then report the failure and handoff rates alongside it. A lower token bill with more staff rework is not necessarily an improvement.

Deploy with recoverable state

I would use separate service identities for API and workers, private database access, managed secret delivery and restricted outbound connections. Schema changes need backward-compatible rollout: add fields, deploy readers/writers that understand them, migrate existing records, then remove obsolete fields later.

Before a release, replay representative workflows against a staging copy containing synthetic data. A canary should limit tenant scope and write capability. Rollback must consider workflow-definition and proposal versions, not only the container image; a previous binary may not understand a newly persisted state.

Run a recovery drill that changes the answer

Stop a worker after provider acceptance but before local acknowledgement. Confirm that the event is reclaimed and the external result reconciled without an unplanned second send. Restore the database to a test environment and verify that pending proposals, outbox events and receipt mappings remain coherent.

I would agree recovery-point and recovery-time objectives explicitly. Backup frequency alone does not prove either. The implementation kit includes the worked calculations and a pilot checklist; it does not present these allocations as benchmarked deployment results.