Retrieval-Augmented Generation (RAG)

Ask a generic LLM about your client’s custom Z-table, and it will answer you — confidently, fluently, and completely wrong, because it was never trained on your system and has no way of knowing that. This is the exact moment every AI proof-of-concept either earns trust or loses it permanently in front of a client.

This lesson is where everything so far has been heading. RAG — Retrieval-Augmented Generation — is how you make a model that knows nothing about your business answer questions about it accurately, without retraining anything at all.

The pipeline: retrieve, augment, generate

Retrieve: embed the question exactly as you learned in Lesson 4, and pull the closest chunks from your vector store. Augment: paste those chunks straight into the prompt, with an explicit instruction — “answer only from the context below; if it isn’t there, say so.” Generate: the model writes an answer grounded in the facts you actually supplied, ideally with citations pointing back to the source chunks.

Be precise about what RAG actually buys you

This is exactly where marketing decks overclaim, so let’s not. RAG grounds answers in current, private data without touching the model’s weights, and it lets you cite sources — genuinely valuable, genuinely real.

But RAG does not eliminate hallucination — the model can still misread or overreach on the very chunks you handed it. It does not retrain anything; nothing is “learned,” which is the subject of the next lesson. And it introduces a brand-new failure mode of its own: retrieval errors. Bad chunking, a stale index, a near-miss match — get retrieval wrong, and the model will confidently answer from the wrong facts with exactly the same fluency it uses for the right ones.

RAG quality is retrieval quality plus prompt quality. Garbage retrieval in, garbage answer out — no matter how good the model is.

Where this shows up in SAP

This is precisely how SAP Joule grounds itself. SAP calls its enterprise flavor of this pattern RAGe — Retrieval-Augmented Generation for enterprise — for unstructured content, paired with the SAP Knowledge Graph for structured business data. The full Joule grounding architecture is coming in Lesson 9; what you’re looking at above is the pipeline running underneath it.

Try it yourself — design it on paper first

Design a RAG flow for “answer questions about our custom Z-tables” before you’d ever write a line of code:

1. Sources — which documents? Data-dictionary exports, design specs, the table definitions themselves?
2. Chunking — one chunk per table, or per field group? What metadata travels with each chunk — table name, module, owner?
3. The augmented prompt template:

   You are an SAP data expert. Answer the question using only the context. <context>{{retrieved_chunk_1}} {{retrieved_chunk_2}}</context> Question: {{user_question}} If the answer isn’t in the context, reply “Not found in the provided documentation.”

That refusal clause is the entire difference between a system that says “I don’t know” and one that quietly invents a Z-table that never existed. It’s the cheapest safety feature you’ll ever add to an AI system, and it’s one line.

Key takeaways

• RAG = retrieve, augment, generate. Fetch the relevant facts, put them in the prompt, then answer — no retraining.
• It grounds answers and enables citations, without touching the model’s weights.
• It reduces, but never eliminates hallucination, and it adds a genuinely new failure mode: retrieval errors.
• Answer quality is capped by retrieval and chunking quality — not by how good the underlying model is.
• In SAP, this shows up as RAGe for unstructured content and the Knowledge Graph for structured data, both inside Joule.

Next: RAG isn’t the only tool in the box. When should you actually fine-tune a model instead — and when is it just an expensive way to solve a problem prompting already solved?