Inside a Large Language Model: Tokens, Context & Cost

Somewhere in your first AI proof-of-concept, someone on the project — probably you — is going to paste an entire ABAP program into a prompt “just to be thorough,” watch the response come back slow and vague, and then watch the invoice line item a month later. This lesson exists so that person isn’t you.

In Lesson 1 you accepted that generative AI is probabilistic. Now we open the box far enough to reason about it like an engineer, because tokens, context, and cost are the difference between an AI feature that scales and one that quietly bankrupts its own business case.

The unit you think in, whether you like it or not

An LLM does not read words. It reads tokens — chunks of text, often smaller than a word. English averages roughly three-quarters of a word per token, but treat that as a rule of thumb, not a law: the ratio shifts by model, and shifts hard for code, numbers, and non-English text.

Here’s why this matters more than it sounds like it should: tokens are simultaneously the unit the model thinks in and the unit you pay for. Every design decision in this lesson traces back to that one overlap.

No lookup, no memory, just prediction

Two phases, and people confuse them constantly. Training crunches enormous text corpora, once, to set the model’s parameters. Inference is what happens every time you use it: the model predicts the next token, appends it, predicts the next, over and over.

This has a blunt consequence: the model has a knowledge cutoff, and it does not query a database to check facts. It generates statistically likely text. Combine that with what you already know about hallucination from Lesson 1, and you get the exact failure mode that catches people off guard — a model can be extremely fluent about something it is completely wrong about, and nothing in its behavior will tip you off.

The context window is a desk, not a filing cabinet

Every model has a context window — the maximum tokens it can hold in view at once, input and output combined. Two myths worth killing before they cost you a debugging afternoon:

The context window is not memory across sessions. Close the chat, and everything is gone — the model doesn’t remember your last conversation, your project, or you.

And bigger is not automatically better. Stuffing a 5,000-line program into a single prompt can blow past the limit, get silently truncated, or — worse — dilute the model’s attention so badly that answer quality drops even though you gave it more information, not less.

Think of the context window as a desk you rent by the token, not a filing cabinet you own.

What nobody tells you until the invoice arrives

You pay per token, in both directions — the prompt you send and the answer you get back, usually at different rates. This is not a footnote; it is the entire economics of building an AI feature.

Pasting whole documents or full programs into every call, “just in case,” is a very common early mistake — and it’s a leading reason proof-of-concepts that work beautifully in a demo quietly fail to survive contact with a real budget. The fix isn’t “write shorter prompts and hope.” It’s retrieval — sending only the relevant slice of a much larger corpus — which is exactly Lesson 5.

On SAP BTP, this consumption meters from the underlying GenAI tokens and bills through a commercial metric called AI Units — a finite, consumable entitlement. Here’s the one rule to actually remember: AI Unit pricing is not published, and it is negotiated commercially. Anyone — including, politely, you in six months if you forget this lesson — who quotes a client a firm per-unit price without checking is guessing. Size the token usage first. Have the pricing conversation second, with SAP, not with a blog post.

Try it yourself

1. Open any free public tokenizer/token-counter tool.
2. Paste one full ABAP include. Note the token count — it’ll likely surprise you.
3. Now trim it down to just the one subroutine that actually matters, and recount.
4. Multiply each number by an illustrative rate (say, a cent per thousand input tokens — illustrative only, not a real SAP number) and look at the gap.

You’ve just done real capacity planning — the exact skill that keeps a client’s AI feature affordable instead of an expensive cautionary tale in next year’s retrospective.

Key takeaways

• LLMs think, and bill, in tokens — roughly three-quarters of a word in English, but model-dependent. Don’t treat the ratio as exact.
• Training sets the model once; inference generates your answer, one token at a time, with no database lookup behind it.
• The context window is a rented desk for this one call — not memory, and not free to fill.
• You pay per token, both directions. In SAP, that meters into AI Units — never quote a fixed price you haven’t verified.

Next: now that you can reason about cost, it’s time to learn the skill that determines whether you need fewer tokens or more — prompting that actually works.