Running voice agents on AWS and GCP without losing the conversation

A telephone call is a long-lived session with a continuous stream of deadlines. Deploying it as if it were an ordinary stateless HTTP request produces avoidable problems: cold starts, uneven connection placement and calls terminated during an otherwise successful rollout.

We deployed the calling systems with Kubernetes. The central lesson was that the unit of work is a conversation with an owner and a lifetime. You cannot move it between pods as casually as an ordinary HTTP request.

This article develops that deployment model on AWS and GCP: where the audio goes, where models run, how capacity is admitted, and how a release finishes existing calls without breaking them.

The deployment has four resource classes

Do not share a GPU between fine-tuning and live inference merely because it appears underutilized. Training can create memory spikes and scheduling interference. Run experiments on a separate pool.

AWS topology

Figure 10. Reference cloud mapping. Reference mapping of live CPU sessions, GPU speech inference, managed models, relational business state and background work to AWS and GCP services. Scalable SVG.

Keep media workers and self-hosted models near each other. Use private service connectivity internally, scoped workload identity for cloud APIs, and tightly controlled outbound access for external model providers.

The managed model's region matters as much as the cluster's region. A Singapore worker calling a distant model endpoint includes that round trip in every turn. Verify availability and latency for the exact model and account rather than assuming all Bedrock models are available in Singapore.

For externally hosted STT or TTS, trace the outbound provider connection separately. A private GPU endpoint and a third-party API have different failure and capacity boundaries.

Configure connection timeouts deliberately. AWS documents a default ALB idle timeout of 60 seconds. A proposed 120-second setting needs to be coordinated with the application and carrier, using only protocol-supported traffic to keep an idle connection alive. HTTP/2 PING frames do not reset that ALB timeout. Active-session duration and idle timeout are different controls. ALB connection attributes.

Bedrock in a text cascade

Bedrock ConverseStream provides streamed model responses, including content events. It is a candidate interface for the LLM stage. It requires streaming-inference permissions and model-specific support. Bedrock ConverseStream reference.

Here is the text-streaming boundary for a configured Bedrock model:

import os
import boto3

def stream_bedrock_text(user_text: str):
    client = boto3.client(
        "bedrock-runtime",
        region_name=os.environ["VOICE_AWS_REGION"],
    )
    result = client.converse_stream(
        modelId=os.environ["VOICE_BEDROCK_MODEL_ID"],
        messages=[{
            "role": "user",
            "content": [{"text": user_text}],
        }],
        inferenceConfig={"maxTokens": 160, "temperature": 0.2},
    )
    try:
        for event in result["stream"]:
            delta = event.get("contentBlockDelta", {}).get("delta", {})
            if "text" in delta:
                yield delta["text"]
    finally:
        result["stream"].close()

This is a synchronous SDK example. Do not iterate it directly inside the media event loop. A production adapter needs an appropriate asynchronous bridge, bounded output queue, cancellation, error-event handling and complete tool-call assembly. Token limits and temperature are experimental choices, not recommended values for every model.

Bedrock tool integration must retain application-side validation and authorization. A common API does not guarantee identical model behavior or tool reliability.

Bedrock for native speech

Nova 2 Sonic uses Bedrock's bidirectional streaming path, distinct from text ConverseStream. The current getting-started example identifies model ID “amazon.nova-2-sonic-v1:0”, 16 kHz input and 24 kHz output, and documents an eight-minute connection limit with a continuation pattern. Nova 2 Sonic getting started.

The practical implication is an adapter with two concurrent directions, ordered session events, audio conversion and renewal behavior. An ordinary streaming text request cannot substitute for that session.

Keep renewal separate from the telephone connection. The call worker should preserve confirmed state, tool results and pending operations while replacing the model-side session at a safe boundary. Test renewal during silence, speech and tool execution. Do not rely on a long context window to eliminate connection lifecycle constraints.

Choosing an AWS GPU

AWS documents G5 with A10G and G6 with L4 GPUs. These are candidate inference families for smaller speech models. GPU sharing techniques and hardware isolation capabilities differ across families. EKS compute guidance.

Choose the instance only after profiling a representative workload. Four things must fit: weights, activations/KV state, concurrent request state and runtime overhead. CPU audio processing and host memory can also become limiting.

For the initial 0.6B ASR serving experiment, one 24 GB GPU is a reasonable profiling unit, not a promise of forty calls. The measured safe call count determines the number of replicas.

GCP topology

GCP's G2 machine table lists g2-standard-4 with four vCPUs, 16 GB default host memory and one L4 with 24 GB GPU memory; g2-standard-8 provides eight vCPUs and 32 GB host memory with one L4. The latter is a useful starting profiling shape when audio preprocessing needs more host resources. These specifications are not a throughput benchmark. GCP GPU machine types.

Use Workload Identity for cloud access, private inference services and a fixed model-artifact revision. Keep the platform's configuration database separate from high-frequency audio buffers.

Gemini's live-session resumption and compression features must be mapped into the call state machine. A reconnect should not replay a committed business action or discard a caller's corrected amount.

Why not put everything on Cloud Run?

Cloud Run supports WebSockets, but they remain requests subject to a timeout, documented at up to 60 minutes. Reconnects can land on a different instance; affinity is best effort. Open sockets also keep instances active for billing purposes. Cloud Run WebSockets.

It can be appropriate for a control API or a deliberately bounded call-worker design. GKE gives more direct control over warm GPU pools, admission, placement and draining. Neither option should assume that a telephony provider reconnects and replays missing audio exactly as a browser client would.

A Kubernetes example with explicit limits

Start with this deployment template for the CPU call worker. Wire its health endpoints and admission contract into the application, and replace the image with your versioned release:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: voice-call-worker
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: voice-call-worker
  template:
    metadata:
      labels:
        app: voice-call-worker
    spec:
      terminationGracePeriodSeconds: 600
      containers:
        - name: worker
          image: registry.example.com/voice-worker:validated-release
          ports:
            - containerPort: 8080
          env:
            - name: MAX_ACTIVE_CALLS
              value: "10"
          resources:
            requests:
              cpu: "2"
              memory: "2Gi"
            limits:
              memory: "4Gi"
          startupProbe:
            httpGet:
              path: /startup
              port: 8080
            periodSeconds: 2
            failureThreshold: 60
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 2
          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            periodSeconds: 10

The service must handle SIGTERM by stopping new admissions, marking itself unready, draining existing calls and exiting before the deadline. The 600-second grace period is an example; it does not guarantee completion for a longer call. An upstream ingress, node shutdown or carrier can impose a shorter deadline.

Readiness should distinguish “not initialized/draining” from ordinary busy capacity. Use explicit admission and load-aware routing to manage full workers. Merely adding replicas behind a load balancer does not redistribute established WebSockets.

Kubernetes describes pod termination and endpoint removal behavior, but application draining remains necessary. Pod lifecycle.

For GPU pods, request the GPU explicitly, use the appropriate device plugin and node affinity, and expose readiness only after model load and warm-up. Scale on queue/deadline metrics and measured session capacity. GPU utilization alone is not a sufficient signal.

Capacity arithmetic for 40 and 100 calls

Assume, purely for planning, that a profiled CPU worker safely handles ten active calls and the desired operating target is eight.

Forty calls require five workers at the target, plus one spare: six replicas. One worker loss leaves fifty hard slots. One hundred calls require thirteen workers at the target, plus one spare: fourteen. These calculations say nothing about STT, TTS or LLM capacity; those dependencies must independently meet the same demand.

With average call duration four minutes, forty steady active calls correspond to about ten starts per minute using Little's law. If calls become eight minutes long, the same forty slots support only five starts per minute.

At 100 calls with six LLM requests per call per minute, the average demand is 600 requests per minute. At 2,000 input tokens per request that is 1.2 million input tokens per minute, before retries. These assumptions make provider quotas a first-class capacity constraint.

Media bandwidth is rarely the whole problem

For configured 8 kHz, 8-bit μ-law audio, one direction carries 64 kbit/s of raw audio. One hundred full-duplex calls carry 12.8 Mbit/s before protocol overhead; base64 representation alone expands that to about 17.1 Mbit/s. JSON, TLS and packet overhead add more.

LiveKit's published transport benchmark includes 10 audio publishers and 3,000 subscribers on a 16-core GCP c2-standard-16, with about 80% CPU in that scenario. It measures SFU forwarding, not a full speech-recognition, reasoning and synthesis workload. It cannot be used to infer the number of voice agents one GPU or pod can support. LiveKit transport benchmark.

Release and load-test protocol

Warm the fleet before the campaign. Run representative conversations at 10, 20, 40 and 100 concurrent calls, with staggered arrivals and synchronized bursts. Include both short answers and long caller speech.

At every step, capture completion rate, dropped calls, p50/p95/p99 response delay, turn-taking errors, queue age, provider throttles, GPU memory and cost. Repeat with one worker unavailable and a slow downstream service. Record test duration and call count.

A rollout should admit new calls to the candidate version while established calls finish on the previous one. Rollback stops new admissions to the candidate; it does not automatically relocate active media sessions.

Our maximum concurrency test reached 100 in early 2025. When you repeat that test on a newer stack, keep the configuration and the complete call outcomes with the latency trace so you can explain what changed.

Autoscaling needs time to become useful

The gap between “more capacity is desired” and “another call can safely be accepted” contains several steps: the scaling controller observes demand, a pod is scheduled, a node may be created, the image starts, model weights load and the worker passes warm-up.

For an outbound campaign, pre-warm against the scheduled launch. For unexpected inbound traffic, reserve headroom and define a queue or human fallback. Autoscaling cannot rescue a call whose response deadline expired while a GPU node was starting.

The following HPA is a reference fragment for the CPU workers above. It assumes that a custom-metrics adapter exposes a gauge named “voice_active_calls” for each pod:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: voice-call-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: voice-call-worker
  minReplicas: 6
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: voice_active_calls
        target:
          type: AverageValue
          averageValue: "8"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

This is incomplete without metric collection, an adapter and admission-aware routing. Kubernetes documents how the HPA uses custom metrics and periodically adjusts its target. It does not move existing calls between workers. Horizontal pod autoscaling.

An average of eight calls per pod can conceal one pod at ten and another at six. Route new sessions using available admission capacity and reject excess admissions explicitly. Also guard on dependency quotas: spare CPU slots are of little use when the STT account is already throttling.

Scale-down is a separate design problem. A stabilization window reduces oscillation; it does not make every selected pod safe to terminate. Long calls may outlast the termination grace period. A stronger session-aware controller drains chosen workers first and removes them only when empty, subject to a documented hard timeout.

N+1 is not the same as surviving a zone failure

Under the earlier planning assumptions, six workers across three availability zones give two workers per zone. Losing one worker leaves fifty hard slots; losing one zone leaves forty. At forty active calls there is then no hard-slot headroom, and the preferred eight-call operating target has been exceeded.

If the objective is to retain forty slots at the eight-call target after losing any one of three evenly provisioned zones, nine workers—three per zone—leave six workers and forty-eight target slots. This is arithmetic under a hypothetical per-worker benchmark, not an availability guarantee.

Apply the same exercise to GPU capacity, ingress, databases and provider quotas. Nine call workers cannot compensate for a single saturated or unavailable model endpoint. Multi-zone placement also introduces network cost and potentially latency; profile the actual placement rather than assuming redundancy is free.

Measure deadlines and saturation together

A useful call dashboard contains active calls, admission refusals, event-loop lag, per-stage queue age, provider errors and the latency histogram. Separate warm/cold, tool/no-tool and language slices so changing traffic does not masquerade as a regression.

For classic Prometheus histograms, a fleet p95 can be estimated from aggregated buckets:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(voice_response_seconds_bucket{stage="useful_audio"}[5m])
  )
)

Choose bucket boundaries that resolve the relevant range—for example 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.5, 2, 3 and 5 seconds, plus the implicit infinity bucket. This is a proposed schema. The query estimates a quantile from bucket counts; it is not the same as averaging pod-level p95 values. Prometheus histogram guidance.

Show sample count and failed/timed-out turns next to the quantile. A small window with only a few observations is unstable. Failed calls omitted from the histogram still belong in the reliability denominator.

Use bounded labels such as stage, provider, release and language slice. Call IDs, telephone numbers and free-form error text belong in controlled traces or logs, not metric labels that create an unbounded time-series count.

A runbook should connect symptoms to decisions

A retry policy needs a deadline and jitter. A call should not accumulate an unbounded sequence of retries while the customer waits. Decide which failures allow a configured provider fallback, which require a short explanation, and which should transfer the call.

After an incident, preserve a redacted timeline, the triggering load, the first saturated boundary and the change that fixed it. A narrowly explained failure and recovery often demonstrates more platform judgment than an unqualified “scales to 100 calls” claim.

Follow one call through the network

It is easier to assess the cloud design if you follow a single call rather than start with a list of services.

The control request reaches the API over HTTPS. It loads the agent's versioned configuration, checks the campaign and customer context, reserves capacity and asks the telephony provider to start the call. The API response does not own the subsequent conversation.

The media connection arrives through the WebSocket ingress and is assigned to a call worker. That worker owns the call ID, audio format, turn state, provider streams and output writer for the lifetime of the connection. A second worker must not independently start another model conversation for the same active media session.

The inference path connects the worker to STT, the reasoning model and TTS. Self-hosted services stay on private endpoints; managed services use their supported authenticated endpoints. The worker passes a deadline and generation identity through its own adapters even when a provider exposes different names for those concepts.

The completion path persists the call outcome and schedules recording processing, extraction and evaluation. An exhausted reporting queue should delay the report rather than stall the last audio packet.

In an AWS deployment, the external WebSocket can terminate at an ALB before reaching EKS call workers. Private GPU services handle speech inference; the reasoning adapter calls Bedrock where the selected model is available. On GCP, the corresponding path uses a supported WebSocket ingress, GKE call workers, private GPU services and Vertex/Gemini endpoints. Configure the timeout on every hop, including the actual GCP load-balancer backend in use.

The media stream should not travel through SQS or Pub/Sub. Those queues are useful for post-call work and retries; a customer waiting for the next word needs a continuously flowing session.

Workload identity belongs in the architecture

On EKS, associate a Kubernetes service account with an IAM role through EKS Pod Identity and use a supported SDK credential chain. Give the call-worker role the model-invocation permissions it needs; give a recording worker only the relevant object-store and metadata permissions. The EKS Pod Identity Agent supplies temporary credentials. EKS Pod Identity.

On GKE, Workload Identity Federation gives workloads scoped Google Cloud access without embedding service-account key files in images. Enabling the feature is only the identity mechanism; grant the specific resource permissions to the correct workload principal. Workload Identity Federation for GKE.

For supported Bedrock APIs, an interface VPC endpoint can keep the connection on AWS's private networking path. Endpoint policy, IAM policy and security-group rules still need to agree. Verify the specific model API and streaming mode you intend to use. Bedrock VPC endpoints.

I would keep three identities distinct: call serving, model-artifact loading and post-call processing. A call worker should not gain permission to replace model artifacts just because it can invoke inference. Provider API keys belong in the secret system; avoid logging the complete service constructor arguments when diagnosing an adapter.

Make model readiness mean something

A process that responds to HTTP is not necessarily ready to synthesize speech. The sequence should be:

container starts
  → verifies model/tokenizer/codec revisions
  → loads weights
  → allocates runtime state
  → warms representative shapes
  → checks output format
  → becomes ready for inference

Use a startup probe to protect the loading period from an over-eager liveness restart. Google documents this separation in its model-serving examples; the warm-up duration should match the selected runtime and artifact path. GKE model serving and probes.

Here is a deployment fragment for a private speech worker. The node pool is labeled workload=voice-gpu, and the server implements the named endpoints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: speech-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: speech-inference
  template:
    metadata:
      labels:
        app: speech-inference
    spec:
      serviceAccountName: speech-inference
      nodeSelector:
        workload: voice-gpu
      terminationGracePeriodSeconds: 120
      containers:
        - name: model-server
          image: registry.example.com/speech-server:validated-release
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
            limits:
              memory: "24Gi"
              nvidia.com/gpu: 1
          startupProbe:
            httpGet:
              path: /startup
              port: 8000
            periodSeconds: 5
            failureThreshold: 120
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000
            periodSeconds: 2
          livenessProbe:
            httpGet:
              path: /live
              port: 8000
            periodSeconds: 10

The quantities are a starting configuration for profiling. CPU and memory limits describe host resources; the GPU's VRAM is determined by the selected node. The manifest also needs the cloud's GPU driver/device-plugin setup, any node-pool taint tolerations, artifact access and a ClusterIP service.

The server should report ready only after a known text/audio fixture succeeds with the expected shape and sample rate. Liveness should answer whether the worker is functioning, not fail merely because a request queue is temporarily full. Saturation belongs in admission and scheduling.

Move the 2026 TTS work from local to cloud carefully

The separate Solo & Silicon TTS project uses MLX Audio on Apple Silicon. Its MLX-converted Kokoro, Qwen3-TTS and Chatterbox checkpoints do not become CUDA services by copying the local environment onto an L4 machine.

For AWS or GCP GPU serving, select the matching upstream checkpoint and a supported CUDA runtime, then repeat the same narration and pronunciation fixtures. Keep the model family, voice conditioning and generation settings aligned; otherwise you are comparing two different configurations as well as two machines.

Start with one resident model worker and one request. Add cancellation and bounded queues before increasing concurrency. Then measure 2, 4 and 8 simultaneous phrase streams, recording first audio, largest chunk gap, output duration and peak memory. A narration benchmark and a live-call benchmark answer different questions, even when they use the same weights.

Pin model, tokenizer and codec revisions together. Store approved artifacts in a controlled bucket or image layer, depending on their size and release cadence. Loading a changing upstream branch during a production restart makes rollback harder to reason about.

Admission is a resource reservation

The usable capacity of a cascade is bounded by its tightest dependency:

admissible calls ≤ min(
    free call-worker slots,
    safe STT sessions,
    safe TTS demand at the expected speech duty cycle,
    LLM request/token quota,
    business-tool capacity
)

These quantities have different units, so convert them using the observed call mix. A TTS worker serving short confirmations can support a different number of calls from the same worker reading long explanations.

For example, at forty active calls, if the agent speaks 40% of the time, the average output demand is sixteen audio-seconds per wall-clock second. If a profiled replica sustains four audio-seconds per second and the operating target is 65% of that capacity, the simple steady-state estimate is ceil(16 / (4 × 0.65)) = seven replicas. This is an illustrative calculation: bursts, first-audio deadlines and redundancy still need a load test.

Reserve against that capacity before starting an outbound call. Use an expiring lease with renewal and an idempotent release, so a crashed worker does not hold a slot forever and a duplicate disconnect does not release it twice.

A deployment should preserve session ownership

For a release, warm the new version, validate its model/configuration bundle, and send a small share of new calls to it. Existing calls finish on their current worker. Compare the candidate's complete-call outcomes with the control, including provider errors and escalations.

A model-pool release needs the same care. If an ASR stream holds state on a model worker, replacing that worker loses more than a stateless request. Drain existing streams or define how their buffered audio and committed transcript move to a replacement. A database record of the call cannot recreate audio that was never retained.

Rollback stops admitting new calls to the candidate. It should not terminate every current customer conversation just because a newer image was rejected. For a severe failure, the fallback may be a controlled human transfer or callback workflow; that is a product decision the infrastructure has to support.

When I review this design, the useful questions are concrete: who owns the call, who can stop its audio, what survives a worker loss, and how is an uncertain business action reconciled? The cloud service names come after those answers.

Next: the evaluation layer.