Building RAG Pipelines: From Zero to Production

Retrieval-augmented generation (RAG) lets a language model answer questions using your own documents instead of only its training data. The idea is simple: find the most relevant passages, put them in the prompt, and ask the model to answer from them. Making it reliable takes more work. Here is the pipeline, stage by stage.

1. Ingestion

Start by getting clean text out of your sources: PDFs, wikis, help centers, tickets, databases. This step decides your ceiling. If tables are mangled or headings are lost, no model will recover them.

• Extract structure, not just text: headings, lists and tables
• Keep metadata such as source, URL, author, date and access permissions
• Remove boilerplate like navigation menus and footers
• Re-ingest on a schedule, or on change, so answers stay current

2. Chunking

Documents are split into chunks small enough to retrieve precisely but large enough to carry meaning. A few hundred tokens with some overlap is a reasonable starting point. Better still, split along the document's own structure, such as sections or FAQ entries, and include the section title in each chunk so it makes sense on its own.

3. Embeddings and storage

Each chunk is converted into an embedding vector and stored in a vector database such as pgvector, Pinecone or Weaviate. If you already run PostgreSQL, pgvector is often enough and keeps your stack simple. Store the metadata alongside the vector so you can filter by permissions, product or date.

4. Retrieval

Pure vector search misses exact terms like product codes and error messages. Most production systems use hybrid search: combine keyword search with vector similarity, then merge the results.

Then add a reranking step. Retrieve a broad set, perhaps 20 to 50 candidates, and use a reranker model to pick the best handful for the prompt. This is one of the highest-impact improvements you can make.

Always apply permission filters at retrieval time, so users never see content they should not.

5. Generation

Give the model clear instructions: answer only from the provided context, cite the source for each claim, and say when the context does not contain the answer. Showing citations to users builds trust and makes errors easy to spot.

6. Evaluation

Build a test set of real questions with known good answers and the documents that contain them. Measure two things separately:

• Retrieval quality: did the right passages make it into the context?
• Answer quality: was the answer correct, complete and faithful to the sources?

Separating them tells you where to fix things. Most problems turn out to be retrieval.

7. Production concerns

• Freshness: update the index when source documents change
• Latency: cache frequent queries and keep the prompt lean
• Cost: track tokens per request, and use smaller models where they are good enough
• Observability: log the query, retrieved chunks and answer for every request
• Feedback: let users flag wrong answers and feed them back into the test set

Common pitfalls

• Chunks with no context, such as a paragraph that says "see above"
• Relying on vector search alone for technical or product content
• No "I don't know" path, so the model fills gaps with guesses
• Indexing everything, including outdated and duplicate documents

RAG is the foundation of most useful business agents. If you are building one, we can help you design it, or read our guide to enterprise AI agents.