From fifty candidates to an answerable evidence set
Retrieval produces candidates. It does not produce a finished argument.
The client’s retrieval path included query enhancement, dense and sparse representations, a top-candidate search, cross-encoder reranking and LLM synthesis. The reranker used the MiniLM MS MARCO family. This architecture creates a useful separation: a fast stage finds plausible passages, and a more expensive stage considers the query and passage together.
The quality of that second stage depends on what it is allowed to see. A well-chosen model can still fail if the passage was truncated, the query was rewritten incorrectly, or the service passed an empty text field under the wrong key.
A cross-encoder reads the pair together
A bi-encoder embeds queries and passages independently, allowing document vectors to be precomputed. A cross-encoder processes a query–passage pair jointly. It is more expensive across many passages, but useful for reranking a bounded candidate set.
The cross-encoder/ms-marco-MiniLM-L6-v2 model provides a compact baseline. Its scores should be used according to the model’s documented inference contract; a raw relevance score is not a calibrated probability that the financial answer is correct. MiniLM cross-encoder model card
A clear inference shape is:
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
pairs = [(query, passage.text) for passage in candidates]
scores = reranker.predict(pairs, batch_size=16)
ranked = sorted(zip(candidates, scores), key=lambda pair: float(pair[1]),
reverse=True)
Batch size 16 is a starting setting. Measure memory and latency on the chosen hardware, and ensure the configured maximum length matches the intended passage size. The service should expose the proportion of pairs truncated during tokenisation.
Truncation can remove the answer
Consider a candidate with a long issuer/section prefix followed by a table row near the end. If the reranker sees only the first 128 tokens, it may never see the financial value it is being asked to rank.
Tokenise the query and passage as a pair rather than joining them with an arbitrary separator and hoping the model interprets it correctly. Reserve enough space for the query, and apply a deliberate passage-window policy when the evidence exceeds the model limit.
For long evidence, options include scoring several windows, selecting windows around matched terms, or using a longer-context reranker. Each option changes cost and failure modes. A maximum-over-windows score can favour long documents simply because they offer more chances to score highly, so evaluate that effect.
Query enhancement can alter the financial question
The earlier query-enhancement path used a small T5 model with a paraphrasing prompt. The useful principle is to improve retrieval wording while preserving constraints. The dangerous outcome is a rewrite that changes the question.
Original:
“What was net debt excluding leases at the end of FY2024?”
Unsafe rewrite:
“What was the company’s total debt in 2024?”
The rewrite removed cash netting, the lease exclusion and the period-end condition. Better retrieval for the rewritten question would still produce the wrong answer.
Keep the original query and use expansions as additional candidate channels. Resolve issuer, dates, units and exclusions into structured constraints before rewriting. Validate that a proposed expansion does not contradict those constraints.
A useful enhancement record contains the original query, validated constraints, expansion text and the reason for expansion. Logging only the rewritten text makes failures difficult to diagnose.
Build an evidence packet, not a pile of text
The generator should receive passages with explicit identities and context:
[E1]
Issuer: ExampleCo
Document: FY2024 annual report
Section: Consolidated operating results
Period: FY2024; unit: USD millions
Evidence: Revenue 1,240; operating income 186.
Source span: table-2, rows 1–2
[E2]
Issuer: ExampleCo
Document: FY2023 annual report
Period: FY2023; unit: USD millions
Evidence: Revenue 1,000; operating income 150.
Source span: table-1, rows 1–2
The IDs are local handles that the generator can cite. The server owns the mapping from those handles to immutable source references. The model should not invent URLs or PDF coordinates.
This packet also makes evidence completeness visible. If the question requests a comparison and only E1 is present, the system knows what is missing before asking the model to write a polished paragraph.
Allocate context by information value
Suppose a worked configuration reserves 1,500 tokens for instructions and the question, 6,000 for evidence, and 1,500 for the response. These are design budgets, not the context limit of a particular model.
A packer should respect the evidence budget while maintaining required coverage. Start with the highest-value passage for each required evidence group. Then add complementary material if space remains. Avoid spending most of the budget on overlapping excerpts from the same page.
def select_evidence(candidates, required_groups, budget):
chosen, seen, covered, remaining = [], set(), set(), budget
for group in required_groups:
if group in covered:
continue
eligible = [p for p in candidates
if group in p["groups"] and p["id"] not in seen
and p["tokens"] <= remaining]
if not eligible:
return {"status": "missing_evidence", "group": group,
"selected": chosen}
best = max(eligible, key=lambda p: (p["score"], p["id"]))
chosen.append(best)
seen.add(best["id"])
covered.update(best["groups"])
remaining -= best["tokens"]
return {"status": "ready", "selected": chosen,
"remaining_tokens": remaining}
This small greedy example makes missing coverage explicit. A production packer can optimise the full selection rather than relying on this greedy order, and should validate token counts and deduplicate source spans before selection. The key contract is that budget exhaustion does not silently become permission to omit essential support.
Order and redundancy affect answer quality
The original Lost in the Middle study demonstrated that the position of relevant information affected performance for the models and tasks evaluated. It motivates testing evidence placement and long-context behaviour; it is not a universal numerical law for every current model. Lost in the Middle
For financial synthesis, place comparable facts together and keep their labels explicit. Separate management statements from calculated values. Put the evidence required by the question ahead of tangential background.
Evaluate the actual generation model with evidence at different positions and with increasing distractor count. A larger context window allows more text, but the product needs reliable use of that text.
Typed contracts prevent invisible failures
A search service may call its passage field chunk, while a reranker expects content. A model call can succeed with an empty string, leaving the system apparently operational and the rankings meaningless.
Use a shared passage type across retrieval, reranking and synthesis. Validate nonempty text, unique passage ID, source identity, token count and source-span references before inference. For batched scores, assert that the number and order of outputs match the input candidates.
These checks are especially valuable while moving quickly. They turn silent quality degradation into an explicit boundary failure that can be investigated.
Measure the contribution of reranking
Hold retrieval candidates fixed and compare the ordering before and after the cross-encoder. Measure ranking quality and task-complete evidence coverage at the context budget. Then test the final answer with the same generator settings.
Also record reranker latency, batch size, token lengths and truncation. A quality improvement that doubles interactive latency may still be worthwhile for a complex research task, but the trade-off should be visible.
The goal is a small, complete and correctly scoped evidence packet. Once that exists, citations and synthesis have something dependable to work with.