Operating RAG across a million financial documents

At one million documents, RAG becomes an operations problem as much as a model problem. A parser can be accurate and still leave a backlog. A vector database can respond quickly while query requests wait behind embedding jobs. A citation can be correct until a source version is overwritten.

For the client platform, ingestion and retrieval were separated into services, Celery handled background work, and Docker definitions packaged the main components. This article develops that architecture into an operating design for a large corpus, including concrete AWS and GCP deployment options. These cloud layouts are current deployment designs, not a claim that every service below was part of the six-week implementation.

Separate the offline and interactive paths

The offline path acquires documents, parses them, validates structure, creates chunks, computes embeddings and writes indexes. It optimises for throughput, cost and recoverability.

The interactive path resolves an analyst’s question, searches the index, reranks evidence, performs any calculations and returns a supported answer. It optimises for predictable latency and task completion.

Sharing code does not require sharing a queue or resource pool. Offline embedding jobs should not evict the query model from memory or saturate the CPU used by the API. Separate worker pools, concurrency limits and resource requests make the boundary explicit.

Design ingestion for replay

Background tasks can be delivered more than once. A worker can finish an external call and crash before recording completion. The next attempt should reuse the result where possible and avoid creating duplicate chunks or charges.

Celery’s task guidance connects late acknowledgement with idempotent task behaviour. Late acknowledgement is useful, but it is not a complete guarantee of exactly-once processing. Celery task documentation

Use a durable job record with states, artifact references and attempt history. A stage acquires a lease, checks whether its output already exists, performs the work, persists the result and commits the transition. A reconciler finds stale leases and incomplete jobs.

job_id
source_document_version
stage + stage_configuration_hash
status: pending / running / succeeded / retryable / quarantined
lease_expires_at
attempt_count
input_artifact_id
output_artifact_id
estimated_cost + actual_usage
last_error_category

Retry transient failures with bounded exponential backoff and jitter. Do not retry invalid files, unsupported formats or a consistently failing parser output indefinitely. Route them to a review queue with the relevant artifact and failure reason.

Queue pages and tokens, not only document counts

A queue with ten thousand one-page releases is different from a queue with ten thousand 200-page annual reports. Scale parsing workers from estimated pages and oldest-job age. Scale embedding workers from queued tokens. Keep a limit on simultaneous provider requests and reserved spend.

For a worked capacity example, suppose the corpus contains 20 million pages. At an effective throughput of 100 pages per second, processing takes about 55.6 hours before additional stages and interruptions. At 25 pages per second, it takes about 9.3 days.

20,000,000 / 100 / 3,600 = 55.56 hours
20,000,000 / 25 / 86,400 = 9.26 days

These are arithmetic scenarios, not measured client throughput. Effective throughput must include provider quotas, retry behaviour, long-document tails and validation work. Doubling workers cannot exceed a shared provider quota.

A six-week deadline therefore needs an early throughput sample across document sizes. A fast ten-page demonstration is not sufficient evidence for completing the full corpus on time.

Publish index versions atomically

Stable point IDs should include document version and chunk identity. Updating a document creates a new version of its chunks; it should not overwrite unrelated points or leave stale passages searchable indefinitely.

For a major embedding change, build a separate collection or named-vector version, validate it and switch the application’s active reference. Qdrant supports collection aliases, which can support controlled switching between index versions. Keep the old version long enough for rollback and for any answer references that still depend on it. Qdrant collections

For incremental updates, maintain a manifest of expected chunks and confirm that all required representations were written before marking the document ready. A partially indexed document should have an explicit state.

Back up object artifacts, metadata and vector indexes with a documented restoration procedure. A backup that has never been restored does not establish recovery time.

Latency needs a critical path

An analyst experiences the complete request, not just the vector search. A worked interactive budget might look like this:

The sum is approximately 3.1 seconds for this illustrative serial budget. It is not a measured service-level objective. Some stages can overlap; some checks must finish before a claim is shown. Percentiles of individual stages cannot simply be added to produce an end-to-end p95.

Instrument a trace from request arrival to completed supported answer. Record queue delay separately from execution time, and break down results by task type, evidence length and concurrency.

The main latency levers are warm models, bounded candidate depth, batched reranking, shorter evidence packets, parallel independent retrieval, connection reuse and sensible caching. Removing evidence validation can make the number look better while making the product worse.

Cache with the correct boundaries

Cache immutable parser artifacts and embeddings aggressively. Query-level caching needs more context:

cache key = normalised question
          + resolved issuer/period/basis
          + access scope
          + corpus snapshot
          + retrieval/generation configuration

A response cached before a new filing arrived may be stale. A response generated for one entitlement scope must not be returned to another. Cache invalidation is therefore part of source identity and access design.

Coalesce identical in-flight requests within the same scope so several users do not launch the same expensive work at once. Bound wait time and ensure cancellation of one subscriber does not cancel work still needed by others.

An AWS deployment shape

A concrete AWS design can place the stateless API and worker pools on EKS, with separate node groups for CPU services and GPU inference where measurements justify it. EKS provides the managed Kubernetes control plane; application sizing and data operations remain engineering responsibilities. Amazon EKS overview

Use S3 for immutable source files and derived parser artifacts, a relational store for job/provenance metadata, and a durable broker configuration for asynchronous work. Qdrant can be operated with persistent storage and explicit backup/replication procedures, or consumed as a managed service if that is the better operational trade-off.

Separate the query and batch embedding deployments. Reserve resources for interactive inference. Keep parser-provider concurrency in an admission service rather than allowing every worker to independently saturate the account quota.

Use workload identity for access to AWS resources, scoped by service role. The ingestion worker needs access to the relevant source/artifact prefixes; the public API does not need broad object-store permissions. Do not bake cloud credentials into images. EKS Pod Identity

Model endpoints may be managed or self-hosted. Verify the model’s regional availability, input limits, data handling and pricing before selecting an endpoint. A containerised open model and a provider OCR API have different throughput, residency and operational requirements.

A GCP deployment shape

The corresponding GCP design uses Cloud Storage for artifacts, Cloud SQL for metadata and GKE for the API, workers and any self-hosted inference services. GKE Standard or Autopilot should be selected based on the workload’s scheduling, hardware and operational needs, rather than assuming one mode fits every stateful or GPU component. GKE Autopilot overview

Workload Identity Federation for GKE provides identity-based access to Google Cloud resources. Give each service the permissions required by its function, and keep tenant access checks in the application as well. GKE workload identity

A Pub/Sub-based ingestion design can be useful, but it is not a drop-in renaming of a Redis/Celery broker. Preserve acknowledgement, retry, lease and idempotency semantics explicitly if the queue technology changes.

Stateless request services can be hosted separately where that simplifies operations. Keep long-running corpus jobs and stateful vector storage on infrastructure suited to their lifecycle. A single cloud label is less useful than a clear account of where state lives and how it is recovered.

Model serving and resource isolation

For self-hosted embeddings or rerankers, pin model and tokenizer revisions in a manifest, download them during an authorised build or startup stage, and expose readiness only after weights are loaded and a small inference succeeds.

Batch by token count as well as request count. Use separate batch windows for offline and interactive traffic. A long batching delay may improve throughput but directly harms query latency. Export queue length, batch tokens, inference time, memory use and error counts.

Before selecting quantisation, compare retrieval or reranking quality on the financial task set. Memory savings are useful only if they do not remove the evidence needed by the analyst.

For the vector database, model raw vectors, index overhead, payloads, replicas and migration space separately. Qdrant documents options for on-disk vectors, quantisation and index tuning; their effects should be tested with the actual filters and candidate depth. Qdrant optimisation guidance

Load-test a representative research workload

Include short lookups, long comparisons, cold and warm caches, concurrent ingestion and unavailable dependencies. Measure completion quality as well as latency. A load test that returns empty answers quickly is not a successful test.

The runbook should explain what happens when the parser quota is exhausted, the vector service is unavailable, a source cannot be opened, a model times out or the iteration budget is nearly spent. Useful degraded behaviour includes returning already available evidence, pausing new ingestion and reporting incomplete coverage explicitly.

The operational objective is predictable research assistance under load, with a clear relationship between capacity, cost and the quality of the evidence delivered.