PART 2 OF 14 · 7 MIN READ · ARUNANK SHARAN · 4 OCTOBER 2025

A financial corpus begins with identity, time and provenance

Before a financial document can be retrieved, the system needs to know what it is. That sounds obvious until the same earnings release appears as an HTML page, a PDF, an exhibit attached to an 8-K and a copy from a data provider.

At the client’s scale—more than one million financial documents—poor identity handling becomes expensive. Duplicate files increase parsing cost. Duplicate passages crowd out independent evidence. A replaced document can make yesterday’s citation open today’s text.

The ingestion design therefore begins with a document registry, not a text splitter.

Separate the source, the document and the representation

A useful design has three identities:

1. Source record: where the material came from, such as an accession in a filing system or an event in a transcript provider.
2. Document version: the immutable content that was acquired at a particular point in time.
3. Derived representation: parser output, chunk set or embedding set produced from that version with a particular configuration.

These identities solve different problems. The source record supports synchronisation. The immutable document makes citations reproducible. The derived representation allows a parser experiment without pretending that a new filing has arrived.

For public US filings, SEC EDGAR provides submission histories and extracted XBRL data through public data APIs. Its developer guidance currently limits requests to ten per second across a user’s machines, so adding workers is not a way around source-level throttling. Bulk acquisition and incremental updates should share a coordinated rate limit. SEC developer resources

Earnings-call transcripts and research reports need a separate acquisition contract. Source availability does not automatically establish rights to redistribute the full material. The retrieval service and document viewer should enforce the corpus access policy attached to the source.

Financial time is more than a date field

A filing has several relevant dates. The fiscal period it describes can end months before publication. A restatement can arrive later. A downloaded copy can arrive later still.

Take a fictional FY2024 filing published in February 2025 and amended in April. A question asking what was knowable in March must not silently use the April amendment. A question asking for the latest reported figure may need the amendment and an explanation of what changed.

This is a bitemporal design problem: the time described by a fact differs from the time that fact became available. It does not require putting every field into the embedding. It requires keeping the fields available to retrieval, validation and display.

Preserve identity before parsing

A content digest is useful for deduplication, but the digest alone is not a complete business identity. Two licensed sources may distribute identical bytes with different access policies. Keep the digest and source identity separately.

from hashlib import sha256
import json

def document_version_id(source: str, record: str, raw: bytes) -> str:
    identity = {
        "source": source,
        "record": record,
        "content_sha256": sha256(raw).hexdigest(),
    }
    canonical = json.dumps(identity, sort_keys=True, separators=(",", ":"))
    return sha256(canonical.encode()).hexdigest()

def parse_artifact_id(document_id: str, config: dict) -> str:
    canonical = json.dumps(config, sort_keys=True, separators=(",", ":"))
    return sha256((document_id + "\n" + canonical).encode()).hexdigest()

The parser configuration should include the provider, pinned model or parser version, extraction options and normalisation version. A prompt change that alters table extraction should produce a different derived artifact. An unchanged download should not.

The working registry also needs a lifecycle. “The task was accepted by the queue” is different from “the document is searchable.”

discovered → acquired → classified → parsed → validated
                                          ↓
                              chunked → embedded → indexed → ready
                                          ↓
                                    retry / quarantine

ready is the externally meaningful state. A document with half its chunks indexed should not be counted as fully searchable unless the product intentionally exposes partial coverage.

Map this to the client ingestion boundary

The ingestion API accepted an S3 path and metadata including sector, industry, sub-industry, document type and creation date. A Celery task handled the subsequent processing. Document-type dispatch covered annual reports, 10-K/10-Q filings, 8-Ks, earnings calls, conference calls, ESG reports and DEF-14A material.

That separation is useful: the API can acknowledge a job without making the client wait for parsing and embedding. The extension is to make the accepted request a durable record with an idempotency key, rather than relying on a queue task ID as the only evidence that a document exists.

A source connector should acquire and validate the file before submitting its internal object-store reference. Letting an arbitrary request instruct a backend to download any URL creates both reliability and security problems. Allowed sources, bounded file sizes, content-type checks and limited redirects belong at this boundary.

Deduplication without deleting financial meaning

There are three different forms of duplication:

• Identical bytes: usually a safe opportunity to reuse parsing, subject to access boundaries.
• Equivalent content in different layouts: a PDF and HTML filing may support a shared logical document, but their citation coordinates are different.
• Repeated language across distinct filings: boilerplate may be identical while the surrounding reporting context is different.

The third case is where aggressive deduplication can become dangerous. A repeated risk disclosure is still evidence that the disclosure appeared in a particular reporting period. Keep the occurrence metadata even if storage is deduplicated internally.

Near-duplicate grouping is more useful at retrieval time than deleting every similar paragraph during ingestion. The retrieval layer can return the strongest passage from a group while preserving the ability to expand to earlier versions.

Freshness should be visible to the analyst

A search result should make it possible to distinguish “the filing does not contain this” from “the new filing has not been indexed yet.” These are different outcomes.

A useful freshness measure is:

source-to-search lag = indexed_at − published_at
pipeline lag         = indexed_at − acquired_at

Measure both by source and document type. A daily average can hide a backlog of the largest filings, exactly where parsing is most expensive. The dashboard should show oldest unprocessed document, queued pages, failed documents and the fraction of acquired versions that are ready.

For historical research, the answer should carry the effective as-of date. For current research, it should expose missing expected documents rather than implying complete coverage.

Reprocessing is a lineage operation

At a $50,000 iteration budget, a full reparse cannot be a casual maintenance step. The first question should be which derived layer needs to change.

Changing the embedding model requires new embeddings, not necessarily a new parse. Changing chunk boundaries requires new chunks and embeddings, while the original page blocks may remain valid. Correcting a parser’s table output should invalidate downstream artifacts only for affected documents or pages.

This makes the corpus an evolving set of versioned assets. It is the foundation for the cost controls discussed later, and for citations that remain trustworthy after the system improves.