How SAP Joule Works: A Technical Deep Dive into Architecture, Grounding & Agents
Past the chat box, Joule is an orchestration engine: a scenario catalog, a knowledge catalog, RAG-based grounding, data masking, and a skills-and-agents layer that acts on your SAP systems. Here's the full architecture — and an honest look at what happens to your data.
Most explanations of SAP Joule stop at “it’s a copilot you chat with.” That’s the surface. Underneath, Joule is an orchestration engine — it decides what you’re asking, gathers the right context, masks what’s sensitive, calls a model, and then acts on your SAP systems through skills and agents.
This is the deep dive. If you want the two-minute mental model first, read How SAP Joule Actually Works — this post assumes you’ve got that and goes a layer down.
The architecture, end to end
1. Understanding the request: the two catalogs
When you ask Joule something, it first decides whether and how it can help using two catalogs:
- Scenario Catalog — metadata of every scenario, function and skill across the SAP cloud applications you’ve licensed and activated. This is how Joule knows what it’s able to do for you.
- Knowledge Catalog — both SAP’s own knowledge and your company-owned knowledge. This is what Joule retrieves from to answer.
Joule also carries context: which SAP app you’re currently in, your conversation history, and which apps your organisation has switched on. That context is what turns a vague prompt into a precise one.
2. Grounding: the part that makes it enterprise-grade
This is the crux, and it’s worth slowing down on. Before any prompt reaches a language model, Joule grounds it — enriching it with relevant business data so the model answers from your reality, not its training set.
The technique is RAGe (Retrieval-Augmented Generation for enterprise): relevant text is retrieved first, then the model generates an answer constrained to that retrieved context. Architects increasingly describe grounding as a multi-store pattern:
- a Knowledge Graph of your entities and their relationships,
- domain models that encode SAP process logic, and
- company memory — your team’s own accumulated context.
The lesson for anyone planning AI: grounding quality is answer quality. Clean, well-modelled, API-accessible data makes Joule sharp; messy, undocumented data makes it vague or wrong.
3. The model is a swappable component
Joule does not hard-wire one LLM. The underlying models are accessed through AI Foundation and the Generative AI Hub on BTP — a curated set of partner foundation models, behind orchestration features like grounding, prompt templating, data masking, and I/O filtering.
Decoupling the interface from the model is deliberate: SAP can upgrade or swap the model without changing how you work, and it keeps the grounding and governance under SAP’s control rather than the model provider’s. When people argue about “which LLM Joule uses,” they’re focused on the replaceable part.
4. Skills vs Agents: the difference that matters
These two words get used interchangeably and shouldn’t be:
- A Skill is a structured, rule-based capability that calls a specific API action. Deterministic. One job.
- An Agent plans, reasons, and orchestrates multiple skills to handle a multi-step request — with RAG and tool-chaining — running on SAP AI Core with built-in metering, tracing and security.
The timeline shows the shift: the Skill Builder reached general availability in mid-2025; the Agent Builder became generally available in January 2026. That GA is the real “agentic” turning point — from AI that completes a field to AI that completes a task.
What happens to your data (the honest version)
Security is where Joule earns enterprise trust, so here’s the accurate picture, not the marketing one:
- Your data stays in your tenant. Joule accesses it only during authenticated sessions (OAuth 2.0), over TLS 1.2+, and stores no customer data inside Joule itself.
- Sensitive data is masked before it leaves. AI Foundation can anonymise or pseudonymise personal data and filter content before a prompt reaches a partner model. With pseudonymisation, the LLM sees masked tokens and the response is un-masked before it reaches you.
- Partner LLMs are contractually barred from training on, or retaining, your data beyond the inference session.
The one honest caveat to keep in mind: that last point is a contractual guarantee, not a purely technical one. Masked, sanitised prompt content still reaches the provider’s infrastructure to be processed — it’s just not retained or used for training. For most enterprises that’s an acceptable, well-governed trade-off, but it’s worth understanding precisely rather than assuming “nothing ever leaves.”
The 2026 shape of Joule
Post-Sapphire 2026, SAP frames the whole thing as a three-layer model:
- Context — Business Data Cloud + Knowledge Graph + domain models (the grounding foundation).
- Build — Joule Studio 2.0, where you compose skills, agents and workflows.
- Governance — an AI Agent Hub for managing agents with metering, tracing and policy.
The direction is unmistakable: from a single copilot toward a governed fleet of agents that reason and act across systems, including agent-to-agent (A2A) coordination.
What this means for you
Two takeaways fall out of the architecture:
- The model is replaceable; your grounding is not. Stop optimising for the LLM and start optimising for the data and process models Joule grounds against. That’s where the durable advantage is.
- Acting safely needs a clean core. Step 4 of the diagram — skills and agents acting on your tenant — is only safe when those actions run through released APIs against clean, well-governed objects. An agent that posts documents or creates orders is exactly as trustworthy as the custom code beneath it.
That second point is the whole reason AI and S/4HANA migration are the same conversation. Grounding rewards clean data; safe action rewards a clean core.
Where to go next
- The fast version of this: How SAP Joule Actually Works.
- Name every component: a plain-English map of SAP’s AI stack.
- Build one: hands-on Business Partner and Sales Order agents.
- Check your foundation: a free readiness scan shows what’s blocking safe, agent-driven action today.
References
Keep reading
How SAP Joule Actually Works: The Mental Model, Explained Simply
Most SAP professionals know what Joule does. Far fewer can explain how it thinks. Here's the five-step mental model — intent, grounding, reasoning, action — in plain English, and why grounding (not the LLM) is the part that matters for enterprises.
Read more →Your SAP AI Learning Path: From Fundamentals to Building Agents
You don't need a 22-lecture course to get useful with SAP AI. You need a path that ends in something working. Here's a four-stage roadmap — understand, map, build, scale safely — with the exact resources for each step.
Read more →Joule Work: SAP's Bet to Replace App Navigation with Natural Language
At Sapphire 2026, SAP unveiled Joule Work — a configurable agentic workspace on web, desktop and mobile, designed to eventually replace navigating Fiori tiles and transactions. You describe the outcome; Joule orchestrates the agents, workflows and data to deliver it. Here's what it is, how it differs from Joule the copilot, and what it means for your landscape.
Read more →