Update an embedding index while serving customers

This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.

Planning basis and engagement boundary

F — Frame the outcome

Upgrade embeddings/features without disrupting correct current decisions or losing a credible rollback path.

Value measure: End-task quality, freshness and migration cost without serving disruption.

I — Investigate the workflow

Confirm source snapshot/delta semantics, deletion handling and which quality gains justify temporary duplicate storage/compute.

First evidence task: Delete and update records during backfill, then canary and roll back.

E — Engineer the solution

Use isolated versioned index generations, catch-up and coherent query/index switching; budget old/new/build capacity and online interference.

The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.

Contract

Version the embedding model, tokenizer/preprocessing, vector dimension/metric and index configuration as a compatible bundle. Query embeddings must match the index generation; “same dimension” is not sufficient compatibility. Define freshness and rollback windows.

Migration design

Capture a consistent source snapshot and change-stream position; build a shadow index with the new bundle; apply subsequent updates/deletes using versioned idempotent operations; reconcile completeness and quality. Atomically switch an active manifest for a canary cohort, then expand. Pin each query to one coherent query-model/index pair.

Sizing

Ten million 768D float32 vectors require 30.72GB raw/index generation. Old and new raw vectors alone total 61.44GB, before ANN structures, metadata, replicas and build workspace. At a hypothetical effective 1,000 items/s, one full embedding pass takes 10,000s≈2h46m40s; ongoing changes, variable text lengths and API/device limits extend it. Reserve backfill CPU, network and object-store throughput so online serving remains healthy.

Late events

Compare source versions/availability timestamps so stale updates do not overwrite newer records. Deletes use tombstones/manifests to prevent resurrection. A backfill needs delta catch-up before promotion; a high aggregate row count can still hide missing IDs or mixed versions.

Cache and rollback

Cache keys include embedding/index generation and authorization scope. Retain the old bundle and keep its deltas current during the rollback window, or state the bounded staleness/rebuild cost of rollback. Avoid claiming instant rollback after discarding the old representation.

Evaluation

Check exact ID/version completeness, permission/deletion behavior, retrieval recall, end-task quality, serving tails and resource interference. Canary by representative query slices and monitor old/new disagreement with expert adjudication.

L — Launch with evidence

Canary after ID/version reconciliation and representative quality tests. Exercise rollback while late updates and deletes continue.

Critical decision condition: Mixed incompatible embeddings or unreconciled deletes block promotion.

Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.

D — Develop ownership and reuse

Search/data operators own generation manifests and retirement dates; product owns acceptance of changed retrieval behavior.

Candidate reusable asset: Online index migration and completeness contract.

Commercial decision and next commitment

Accountable decision owner: Search/data service release owner.

The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.

Approve only the next exposure level supported by the evidence. The next useful investment is: delete and update records during backfill, then canary and roll back.

Method and references

Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.