RAG in Production: The Minimal, Correct Implementation for LLM Apps

The Four Failure Modes of Naive RAG

You search your vector database and Claude gets the wrong document. Why?

• Insufficient context for Claude to reason
• Missing surrounding context for accurate understanding

• Wastes Claude's context window with irrelevant information
• Reduces the model's ability to find the key signal

After retrieval, you have 5–10 candidate chunks. But the first chunk isn't necessarily the best match — it's just the one with the highest embedding similarity score.

1. A chunk about "bulk discounts" (high embedding similarity but off-topic)
2. A chunk about "pricing tiers" (exact match, but not retrieved first)
3. A chunk about "support costs"

Without reranking, Claude gets #1 and answers about bulk discounts. With reranking, #2 moves to the top.

Claude has a documented weakness: it loses track of information in the middle of a long context window.

If you pass 10 chunks ordered [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], Claude focuses most on chunks 1–3 and 8–10. Chunks 4–7 are in the "lost in the middle" zone.

Chunking + Hybrid Retrieval

The default that beats everything else in head-to-head benchmarks. Split on structural boundaries in priority order: paragraphs → sentences → words → characters.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,            # ~400-512 tokens
    chunk_overlap=50,          # ~10-20% overlap
    separators=["\n\n", "\n", " ", ""]  # priority order
)

chunks = splitter.split_text(document_text)

Why overlap? If a key sentence sits at a chunk boundary, overlap ensures it appears fully in at least one chunk.

Neither dense nor sparse search alone is optimal. Combine both:

from rank_bm25 import BM25Okapi

tokenized_corpus = [chunk.split() for chunk in corpus]
bm25 = BM25Okapi(tokenized_corpus)

query_tokens = query.split()
bm25_scores = bm25.get_scores(query_tokens)
bm25_results = sorted(enumerate(bm25_scores), key=lambda x: x[1], reverse=True)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = [model.encode(chunk) for chunk in corpus]

query_embedding = model.encode(query)
similarities = model.util.pytorch_cos_sim(query_embedding, embeddings)[0]
dense_results = sorted(enumerate(similarities.tolist()), key=lambda x: x[1], reverse=True)

def reciprocal_rank_fusion(bm25_results, dense_results, k=60):
    """Combine rankings: higher = better. Formula: 1 / (k + rank)"""
    scores = {}
    
    for rank, (idx, _) in enumerate(bm25_results):
        scores[idx] = scores.get(idx, 0) + 1 / (k + rank)
    
    for rank, (idx, _) in enumerate(dense_results):
        scores[idx] = scores.get(idx, 0) + 1 / (k + rank)
    
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

hybrid_results = reciprocal_rank_fusion(bm25_results, dense_results)
top_candidates = [corpus[idx] for idx, _ in hybrid_results[:10]]

The formula 1 / (k + rank) gives high weight to top-ranked results from either method. The k=60 constant prevents any single result from dominating.

After hybrid retrieval, rerank the top-N candidates:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

# Score each (query, chunk) pair
scores = reranker.predict([[query, chunk] for chunk in top_candidates])
reranked = sorted(zip(top_candidates, scores), key=lambda x: x[1], reverse=True)

# Filter by threshold
threshold = 0.5
filtered = [(chunk, score) for chunk, score in reranked if score > threshold]
final_chunks = [chunk for chunk, _ in filtered]

import cohere

co = cohere.ClientV2(api_key="your-key")
results = co.rerank(
    model="rerank-v3.5",
    query=query,
    documents=top_candidates,
    top_n=5
)

Run cross-encoder reranking on the top-N candidates only (not the full corpus) — it's slow for large sets but precise.

Context Assembly and the Lost-in-the-Middle Fix

Research (2024–2026) confirms that Claude performs worst on information in the middle of a long context. If you pass 10 ranked chunks in order [best, 2nd, 3rd, ..., 10th], chunks 4–7 are underutilized.

def build_sandwich_context(chunks_with_scores):
    """Reorder chunks: top chunks at start and end, middle at center."""
    if len(chunks_with_scores) <= 3:
        return [c for c, _ in chunks_with_scores]
    
    # Sort by score
    ranked = sorted(chunks_with_scores, key=lambda x: x[1], reverse=True)
    
    top_count = max(1, len(ranked) // 4)  # Top 25%
    top = ranked[:top_count]
    middle = ranked[top_count:-top_count]
    bottom_ranked = ranked[-top_count:]
    
    # Sandwich: best at start, rest in middle, best again at end
    result = [c for c, _ in top]
    result += [c for c, _ in middle]
    result += [c for c, _ in reversed(bottom_ranked)]  # reversed for variety
    
    return result

# 1. Hybrid retrieval
hybrid_results = reciprocal_rank_fusion(bm25_results, dense_results)
top_candidates = [corpus[idx] for idx, _ in hybrid_results[:10]]

# 2. Rerank
scores = reranker.predict([[query, chunk] for chunk in top_candidates])
reranked = sorted(zip(top_candidates, scores), key=lambda x: x[1], reverse=True)

# 3. Filter by relevance threshold
threshold = 0.5
filtered = [(chunk, score) for chunk, score in reranked if score > threshold]
if not filtered:
    filtered = [reranked[0]]  # Always include at least 1 result

# 4. Build sandwich context
sandwich = build_sandwich_context(filtered)

# 5. Format and send to Claude
context_text = "\n\n".join(f"[Source {i+1}]\n{chunk}" for i, chunk in enumerate(sandwich))

response = client.messages.create(
    model="claude-sonnet-4-6-20250514",
    max_tokens=1024,
    system=f"Answer based on these sources:\n\n{context_text}",
    messages=[{"role": "user", "content": query}]
)

A threshold too low includes irrelevant chunks and causes hallucination. Too high excludes valid chunks and causes "I don't know" responses.

1. Gather 50–100 ground-truth question-answer pairs from your domain
2. Run retrieval with different thresholds
3. Score answers for correctness and hallucination
4. Pick the threshold that minimizes both hallucination rate and "I don't know" rate

RAG adds latency and cost. Skip it if:

• Small, curated data: If your knowledge base is < 5 documents, pass them directly to Claude.
• General knowledge questions: Claude's training already covers most factual knowledge. RAG helps for custom data.
• Creative or brainstorming work: RAG biases Claude toward retrieved content. Let it think freely.
• Real-time data needs: Web search, not RAG, is better for current events.

• Large document corpus (100+ documents)
• Proprietary or domain-specific data
• Your data changes frequently (update the index, not the model)
• You need audit trails (which chunks did Claude use?)