Chunk the evidence, not just the text

Chunking decides what the retrieval system is able to find. Once a table header has been separated from its values, the embedding model cannot reliably put the relationship back. Once an analyst’s question has been separated from management’s answer, a passage can sound much more definite than it was.

For the client, the corpus was varied enough that one universal text-splitting rule was a poor abstraction. The ingestion code exposed document-specific chunker classes, including annual reports, 10-K/10-Q filings, earnings calls, conference calls and proxy material. This provides a useful dispatch boundary. The next step is to make the evidence rules for each class explicit.

The hidden mismatch in a 512-unit chunk

The document path used a 512-word target through its parsing/chunking configuration. The plain-text path used a token-based chunker with a 512-token target and no overlap. These values look similar in configuration and are not equivalent.

The financial BGE model used in the embedding adapter has a 512-token maximum sequence length. A 512-word paragraph can exceed that limit, particularly with financial notation, identifiers and tables. Its trailing content may be truncated before embedding. Financial BGE model card

Even “512 tokens” is incomplete without naming the tokenizer. The text splitter’s GPT-2 token count does not guarantee that the BGE tokenizer will produce the same count. Metadata prepended to the text also consumes capacity.

The operational check should run the actual embedding tokenizer on the final embedding input. Track the fraction of chunks that exceed the model limit before inference. Silent truncation is a data-quality problem, not a harmless model setting.

Use a hierarchy of evidence

A useful financial document hierarchy is:

document version
  └─ section
       ├─ paragraph block
       ├─ table
       │    └─ row group with headers and notes
       └─ speaker turn or question-answer exchange

The retrieval unit and the answer unit do not have to be identical. A small child passage can retrieve the right location. The context assembler can then expand to the relevant parent section, adjacent explanation or table header.

That expansion must remain bounded. Retrieving one row should not automatically inject an entire 200-page report. Define the parent relationship, maximum expansion size and required neighbouring blocks.

The resulting passage should carry its own text plus pointers to the source structure. It should not contain only a copied substring with no way to discover what came before or after it.

Chunk a filing around financial meaning

For a 10-K or annual report, section headings provide useful boundaries. Within a section, keep paragraphs intact where they fit the embedding budget. Preserve table units, row headings, column headings and relevant footnotes with the rows they explain.

A row group might contain:

ExampleCo | FY2024 annual report | Note 7: Debt
Unit: USD millions | Schedule as of 31 December 2024
Maturity year 2025: 120
Maturity year 2026: 180
Footnote: excludes finance lease liabilities

This is longer than the raw rows. It is also a much better retrieval unit for “How much debt matures in 2025, excluding leases?” The exclusion is part of the fact.

For a 10-Q, repeat whether the column represents three months, six months or nine months. For a proxy statement, preserve the relationship between a compensation figure and the footnote that defines it. The same chunk_size cannot express these rules.

Chunk a transcript around turns

A transcript is a conversation. Its headings and speaker labels are structural metadata, not disposable formatting.

Event: ExampleCo Q2 2025 earnings call
Section: Questions and answers
Question — Analyst:
“Does that margin outlook include the restructuring charge?”
Answer — Chief Financial Officer:
“The outlook excludes that charge. We expect it in the third quarter.”

If the answer is indexed alone, “that charge” has no clear referent. If the question is indexed as a management statement, the system may attribute an analyst’s assumption to the company.

A good policy keeps short question-answer pairs together and splits long answers at paragraph boundaries, repeating the event and speaker identity. It should preserve interruptions and corrections rather than merging every nearby utterance into a seamless statement.

Add context without contaminating the evidence

There are three different text fields worth distinguishing:

• source_text: the extracted source passage.
• embedding_text: source plus compact issuer, period and section context.
• display_text: a readable excerpt for search results, with truncation clearly indicated.

Generated summaries can be added as a separate retrieval representation. They should not overwrite the source text or become the quoted evidence. If a generated context label incorrectly says “North America” when the paragraph concerns Europe, retrieval can be distorted even though the original passage is intact.

An enrichment step should therefore preserve its model/configuration version and point back to the underlying source spans. The evaluation should compare retrieval with and without enrichment, particularly for short ambiguous passages.

A token-budgeted packer

The following packing function accepts already meaningful blocks and a tokenizer-specific counting function. It refuses an oversized block so that the caller must split it deliberately—by table rows, sentences or speaker paragraphs—rather than silently cutting it off.

from collections.abc import Callable

def pack_blocks(blocks: list[str], prefix: str,
                count_tokens: Callable[[str], int],
                limit: int = 480) -> list[str]:
    if count_tokens(prefix) >= limit:
        raise ValueError("Metadata consumes the embedding budget")
    output, current = [], []
    for block in blocks:
        if count_tokens(prefix + "\n" + block) > limit:
            raise ValueError("Split this block using its structure first")
        candidate = prefix + "\n" + "\n\n".join(current + [block])
        if current and count_tokens(candidate) > limit:
            output.append(prefix + "\n" + "\n\n".join(current))
            current = []
        current.append(block)
    if current:
        output.append(prefix + "\n" + "\n\n".join(current))
    return output

A 480-token budget is a starting configuration here, leaving room below a 512-token model limit. Count special tokens consistently and verify the final encoded input. The right budget depends on the representation and model, not a universal optimum.

Overlap has a measurable cost

Overlap can recover sentences broken at boundaries. It also creates near-duplicate candidates and increases the index size. With a uniform 400-token window and 80-token overlap, the stride is 320 tokens. For a long text, the approximate chunk-count multiplier is 400 / 320 = 1.25—about 25% more chunks than non-overlapping windows.

That is not automatically a bad trade. It should buy a measurable improvement on boundary-sensitive questions. Structure-aware chunks with parent expansion may achieve the same benefit with less duplication.

When evaluating chunking, keep the questions, parser output, embedding model and retrieval settings fixed. Compare several policies and inspect:

Evaluate difficult slices separately. A chunking policy that improves narrative paragraphs while damaging financial tables is not an unqualified improvement for this product.

Rechunk without paying to parse again

Persist the parser’s block output. A chunking experiment should read those blocks and produce a new chunk-set version. It should not trigger another full paid parse of unchanged documents.

At the client’s iteration budget, that separation has a direct commercial value. It also makes experiments easier to interpret: when only the chunker changes, the quality difference can be attributed to the chunker rather than an unnoticed change in OCR output.