Embeddings & Semantic Search

Here’s a support ticket that plays out constantly: a user searches your knowledge base for “PO,” gets zero results, and files a complaint that “the search is broken.” The search isn’t broken. It’s doing exactly what keyword search has always done — matching characters — while the answer sits three documents away, in a paragraph that only ever says “purchase order,” never the abbreviation.

You now prompt well (Lesson 3). But prompting alone can’t fix this, because the model still doesn’t know your data exists. The bridge is embeddings — and once this clicks, retrieval-augmented generation, the subject of the very next lesson, stops being a buzzword and becomes obvious.

Turning meaning into geometry

An embedding is a vector — a list of numbers — that represents the meaning of a piece of text. Models are trained specifically so that texts with similar meaning land on similar vectors.

That’s the entire trick, and it’s worth sitting with for a second: language becomes geometry, and “these mean the same thing” becomes “these points are close together.” Closeness is usually measured with cosine similarity, the angle between two vectors — a useful working intuition, but a simplification worth being honest about. Embeddings capture statistical patterns learned from training data, not verified ground truth, so “close” really means “used in similar contexts” — which is usually, but not always, the same as “means the same thing.”

Why this quietly fixes the PO problem

Semantic search embeds your query and every document into the same vector space, then returns whatever is closest to the query — not whatever shares the most letters. “PO” and “purchase order” land near each other because they’re used the same way, so the right document surfaces even with zero shared keywords.

This is the fix for that support ticket. Not a smarter search box. A different underlying representation of what “similar” even means.

Chunking: the boring decision that decides everything

You do not embed an 80-page manual as a single vector — you’d lose every ounce of precision. You split it into chunks, embed each one, and retrieval returns whichever chunks are closest to the question.

This unglamorous step is where most real-world systems quietly succeed or fail. Chunks that are too big produce vague matches and waste context tokens you now know are not free. Chunks that are too small split the real answer across several pieces, so no single chunk is sufficient on its own. Good chunking respects natural boundaries — headings, sections — and carries metadata: source, title, date, so you can filter results and cite them later.

The trap that costs people entire afternoons

Embeddings are model-specific, and they do not mix. A vector produced by one embedding model is meaningless to another model — you cannot compare them, blend them, or search across them. Pick one embedding model, embed your entire corpus with it, and embed every query with that same one. Change embedding models later, and there’s no shortcut: you re-embed everything, from scratch.

Where this shows up in SAP

SAP stores and searches vectors natively in the SAP HANA Cloud Vector Engine — embeddings aren’t a bolt-on add-in, they’re part of the database you already operate. Paired with the Generative AI Hub’s embedding models, this is the SAP-native foundation underneath grounding both Joule and any custom agent you build against your own documents.

Try it yourself

No code required for the core insight — just your own judgment:

1. Take four short SAP-flavored snippets:
   • A) “How to create a sales order”
   • B) “Steps to raise a customer order document”
   • C) “Configuring output determination”
   • D) “Pricing procedure setup”
2. Query: “how do I enter a new order for a customer?”
3. Rank A–D by relatedness by hand. You’ll put A and B at the top — even though B shares almost no keywords with the query at all. You just did semantic ranking in your own head.
4. Optional, about fifteen lines of Python: call any embeddings API, embed all four plus the query, compute cosine similarity, and check that the math agrees with your instinct.

Key takeaways

• An embedding is a vector representing meaning — similar meaning lands on nearby vectors.
• Semantic search finds by meaning, catching the exact matches keyword search structurally cannot.
• Chunking, plus metadata, is the unglamorous decision that quietly makes or breaks the whole system.
• Embeddings are model-specific and non-portable — one embedding model for the whole corpus, no exceptions.
• SAP’s home for vectors is the HANA Cloud Vector Engine, natively inside the database you already run.

Next, we combine embeddings, retrieval, and everything you learned about prompting into the single most important pattern in enterprise AI: RAG.