Evaluating, Testing & Observing AI Agents

Your agent worked perfectly in the demo. Three weeks later, after a quiet model update nobody explicitly told you about, it’s giving subtly different answers to the exact same questions — and the first time anyone notices is when a client does. That gap between “worked in the demo” and “still works in week twelve” is where almost every AI project actually lives or dies, and it’s the part most tutorials skip entirely.

You already have the right instinct for this, even if nobody’s pointed it out yet: this is ABAP Unit and ATC, for AI. You would never ship ABAP without a test suite. Don’t ship an agent without one either.

Why testing AI genuinely is different, not just harder

Recall Lesson 1: AI is non-deterministic. The same input can produce different output. So you cannot assert output == "exactly this string" the way you would in a unit test. You test behaviour and properties instead of exact text — and here’s the part that takes getting used to: a test can flip between two runs of the identical input, and that flip is itself useful information, not a fluke to ignore.

Build an eval set — your golden questions

The core tool is an eval set: a table of representative inputs, each with an expected behaviour — not necessarily an exact expected output — and a pass criterion. Include the ordinary cases the agent should handle every day, the genuine edge cases, and, most importantly, the “must refuse” cases: questions it should decline to answer because the data or the permission simply isn’t there.

That refusal row is the single most valuable test you will ever write. It’s the one that catches hallucination and over-reach before a client does.

How you actually score it

Use deterministic checks wherever you can: did it call the right tool? Is the order number in a valid format? Did it refuse exactly when it should have? Layer in LLM-as-judge for the fuzzier quality questions — a second model grading the answer against an explicit rubric, such as “is this grounded in the supplied context, yes or no?” It’s genuinely powerful, but know its limits going in: the judge is itself non-deterministic, and it can carry its own biases, so use a tightly specified rubric and spot-check its verdicts by hand rather than trusting it blindly. And check groundedness directly — does every claim in the answer trace back to the retrieved context from Lesson 5, or is it fluent but ungrounded, which reads exactly the same to an unsuspecting user?

Regression, drift, and the update nobody warned you about

The model underneath your agent will change — SAP will upgrade it, or you’ll deliberately switch models yourselves — and when it does, behaviour can quietly drift. That’s exactly the scenario in this lesson’s opening line. So your eval set is never a one-time gate you clear once and forget. It’s a regression suite you re-run on every single change, exactly like ATC on a transport before it moves forward. A green suite is your licence to ship — and, just as importantly, a red one is your early warning before a client notices first.

Observability: watching what actually happens inside the loop

In production, you need to watch the reason-act loop from Lesson 7, not just admire the final answer. Tracing shows each step — which tool was called, with what arguments, what came back. Token accounting per step shows exactly where the cost goes, so one runaway agent doesn’t quietly burn through your AI Units overnight. Latency budgets matter because a slow loop is, from a user’s perspective, functionally a failed one, no matter how correct the eventual answer turns out to be.

Without traces, debugging a misbehaving agent is pure guesswork dressed up as engineering. With them, you can point at the exact step that went wrong — and say so with confidence, in a client meeting, instead of hedging.

Where this shows up in SAP

Agents built on SAP’s platform come with built-in metering and tracing out of the box, and the governance layer in the next lesson adds oversight on top of that. But the discipline itself — golden sets, regression on every change, groundedness checks — is yours to bring to the table. SAP hands you the instruments. Reading the dials is still your job.

Try it yourself

Take the agent from Lesson 11 and put it through a real cycle:

1. Write a ten-row eval table: columns for input, expected behaviour, pass criterion. Include at least one must-refuse row.
2. Run all ten, twice.
3. Score the pass rate each run, and note any rows that flipped between the two — those are your live non-determinism risks, not theoretical ones.
4. For one flipped row, write down exactly how you’d stabilise it: a tighter instruction, lower temperature, a deterministic guardrail, or a sharper refusal clause.

You now have a working regression suite — the one thing that actually lets you change the model underneath and still trust what comes out the other side.

Key takeaways

• AI testing checks behaviour and properties, never exact strings, because the output itself is non-deterministic.
• An eval set — golden questions, expected behaviour, and the must-refuse case — is your core tool, and the must-refuse row is the most valuable one in the table.
• Score with deterministic checks, LLM-as-judge (with its limits acknowledged), and groundedness checks, layered together.
• Re-run the eval set as a regression suite on every model change to catch drift early — it’s ATC for AI, not a one-time gate.
• Observability — traces, per-step token cost, latency — is how you debug the loop instead of guessing at it.

Final lesson: now that you can build an agent and prove it works, make it safe, governed, and genuinely Clean-Core compliant.