Building Production RAG Systems That Actually Hold Up

A retrieval-augmented generation demo takes an afternoon. A RAG system you can put in front of real users — one that stays accurate, cites its sources, and doesn't make things up — takes a lot more care. Here's where the effort actually goes.

Retrieval is the hard part, not generation

Most RAG quality problems are retrieval problems. If the right context never makes it into the prompt, no model can save you. Pure vector (dense) search is a great default, but on its own it misses exact terms, names, and codes that keyword search nails.

In production I lean on hybrid retrieval: run lexical (BM25) and dense (kNN) search in parallel, then merge the rankings with Reciprocal Rank Fusion. You get the recall of embeddings and the precision of keyword matching.

RRF is refreshingly simple — no tuning of score scales, just ranks:

// Combine BM25 + dense rankings into one fused order.
function reciprocalRankFusion(rankings: string[][], k = 60): Map<string, number> {
  const scores = new Map<string, number>();
  for (const ranking of rankings) {
    ranking.forEach((id, index) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + index + 1));
    });
  }
  return scores;
}

Ground every answer, and show your work

A production assistant should answer from your content, not from the model's memory. Two things make that real:

• Citations. Tie each claim back to the source chunk so users (and you) can verify it.
• Modes. A strict "knowledge-base only" mode for high-stakes questions, and a softer general mode where appropriate — chosen deliberately, not by accident.

You can't improve what you don't measure

The difference between "seems fine" and "is good" is evaluation. Build a small, honest eval set of real questions with expected sources, and track retrieval hit-rate and answer faithfulness as you change chunking, embeddings, or prompts. Without it, every change is a guess.

The takeaway

Production RAG is mostly disciplined retrieval, strict grounding, and continuous evaluation — the generation is the easy 10%. Get the other 90% right and the system earns trust instead of eroding it.