RAGX
Questions about financial filings, answered with citations and measured against a fixed evaluation set.
Current chapter
Archived, with the work and its history preserved.
Status recorded 2026-09-14The idea
Make facts buried in long reports and dense financial tables easier to find and verify.
My contribution
I built a retrieval and answer evaluation harness, table-row descriptions for search, and a deployment pipeline gated by evaluation results. The published case study reports retrieval of 0.97 and answer accuracy of 0.98 on 45 cases.
Along the way
- Undated key point
Measuring each change
The case study traces improvements from the baseline through hybrid search, contextual retrieval, reranking, and table-row descriptions. It also records limitations of reproducibility and testing.
Related writing
Related writing will collect here.
Full case study
RAGX answers questions about 10-K filings — “what was Costco’s 2023 revenue?” — by finding the exact row in the right document, quoting it, and refusing to guess when the answer isn’t there.
The problem
Annual reports run 76–365 pages, and their most valuable facts live in dense financial tables. A wrong-but-confident answer is worse than no answer at all. Most RAG demos hand-wave quality; RAGX takes the opposite stance — quality is a number, not a vibe. I chose financial filings on purpose: the answers are numeric and unambiguous, so the score is honest.
What I built
An eval harness that scores every change against the same 45 cases. Two metrics, kept deliberately separate: did retrieval put every needed fact in front of the model, and was the final answer actually right (numeric-tolerant matching, refusal detection, and an LLM judge for free-form answers). The whole project is that scoreboard climbing:
0.63 / 0.78 baseline → 0.72 / 0.82 hybrid + query decomposition → 0.85 / 0.87 contextual retrieval + reranker → 0.97 / 0.98 with table-row descriptions.
The live /eval dashboard — every change
scored against the same 45 cases.
I treated tables as the boss fight. The biggest jump (0.82 → 0.97 retrieval) came from rewriting each numeric table row into a search-friendly sentence at ingest time — the sentence gets embedded, the raw row is kept for grounding and citation. That single idea is what made the numbers findable.
Swappable seams instead of a framework. Embedder, vector store, generator, reranker, and planner are all interfaces with mock and real implementations, so switching a model or a store is a one-file change. I ruled out heavyweight orchestration until the eval proved I needed it — it never did.
Shipped like production
The CI pipeline runs the 45-case eval before every rollout — below a 0.90 floor on either metric, the deploy stops and production keeps the old version. A change that regresses retrieval quality never reaches users — that’s the MLOps claim, not just DevOps. (0.90, not today’s score: LLM output varies run to run, and a gate that cries wolf gets disabled.)
No service-account key exists anywhere — GitHub Actions authenticates to
GCP via Workload Identity Federation: prove an identity, exchange it for a
short-lived token. Cluster, registry, static IP, and the whole trust chain
live in Terraform, reproducible from an empty GCP project. And the one number
tuned by hand — timeoutSec: 120 — is there because cross-document queries
measured 22–78s against GCP’s 30s default, which cut them off as fake 502s.
Results
- Retrieval 0.97, answer accuracy 0.98 on the 45-case eval — including all three “the answer isn’t in the documents” cases, where the system says I don’t know instead of hallucinating.
- Deployed and public on GKE Autopilot (Vercel kept as a mirror): ~10k chunks across four real filings (Berkshire Hathaway, JPMorgan, Microsoft, Costco).
- Citations are non-negotiable — an uncited answer is a failed answer by definition — and the system is wrapped as an MCP tool, so any agent can call it directly.
- 2,866 lines of TypeScript, 77 commits, three weeks.
What I’d do differently
The eval harness is the test suite — a deliberate call, but it means the non-quality code paths (dashboard, ingestion) have no unit tests, and “0.97” isn’t reproducible from a fresh clone without re-running paid APIs. Next time I’d commit each eval run’s JSON artifact next to the change that produced it, and add plain unit tests around the chunker and scorer — pure functions begging for them.
Two things the cloud layer taught me the hard way: chasing a deprecation
warning, I swapped an Ingress annotation for a field the cluster had no
resource behind — and destroyed the load balancer; the separately reserved
static IP is what made recovery cheap. And a credential rotation failed
silently because the pod restarted before the Secret was replaced — envFrom
config is a snapshot at process start, not a live subscription. The autoscaler
is also still wrong: it watches CPU, but this service saturates on network I/O
while CPU idles — the honest fix is scaling on concurrent requests.