Imagine you are talking to a smart person who can discuss almost any topic — from writing poetry to analyzing code. You ask a question, and they respond almost instantly. That’s how LLMs — Large Language Models — work. But how exactly do they make decisions and “think”?

In this lesson, you will learn to understand the inner logic of LLMs, discover why they make mistakes, and see how you can influence model behavior through proper prompt structure.

📌 Real-life example: you ask ChatGPT to write a birthday greeting. Why does it sometimes sound too formal or repeat the same phrases? The answer lies in the model’s design and its probabilistic nature. You’ll learn how to change that.

What You’ll Learn in This Article

• What LLMs are and how their architecture works
• What “token prediction” means
• Why models “hallucinate” and how to diagnose it
• How Token Inspector works and what it shows
• How modern models evolve: transformers, RAG, Tool Use
• How to craft prompts that help LLMs stay on track

🧩 Key Terms

• LLM (Large Language Model) — a large language model trained on massive amounts of text to generate human-like responses.
• Token — the smallest unit of text (letter, part of a word, or a word) the model processes.
• Token prediction — the mechanism of choosing the next token based on the previous ones.
• Hallucination — when an LLM confidently “makes up” facts that don’t exist in its training data.
• Transformer — a neural network architecture based on the attention mechanism, forming the foundation of modern LLMs.
• RAG (Retrieval-Augmented Generation) — an approach where the model supplements its answers with information from external sources (like knowledge bases or search engines) before generating text.
• Tool Use — the model’s ability to call external tools such as calculators, APIs, or search engines to complete tasks.
• Token Inspector — a tool that reveals which tokens the model considered during text generation, with probabilities and reasoning.

🔧 How LLMs Work

Large language models (LLMs), such as ChatGPT, Claude, Gemini, and others, don’t work like humans. They don’t understand language in the usual sense and don’t store facts like an encyclopedia. Instead, they rely on a powerful mathematical model that predicts which word (or part of a word) should come next.

📚 Analogy: The Guessing Writer

Imagine a writer who’s only shown the first few words of a sentence:

“This morning I went to…”

Their task is to guess word by word until a coherent phrase emerges.

They rely on:

• life experience (for a model — statistics),
• how people usually finish such sentences,
• and the most probable continuations.

For example:

• “…the store” — likely.
• “…a volcano” — unlikely.
• “…a superconducting resonator” — nearly impossible.

That’s exactly how an LLM works. It doesn’t “know” where you went, but it guesses the most probable continuation based on previously seen texts.

🧠 Under the Hood: Transformers

The foundation of LLMs is an architecture called a Transformer.

It’s not the robot from the movies 🙂, but a type of neural network specifically designed for processing sequences of words.

The key feature of transformers is the attention mechanism. This allows the model not just to “read” text left to right, but to focus more on the words that matter in context.

💡 Analogy: When you read a sentence, you automatically highlight key words. For example, in “Mom washed the window because it was dirty” — you understand that “it” refers to the window, not to mom. That’s exactly what attention does — it helps the model “understand” word relationships.

🏋️ Training: Not Memorization, but Habits

The model doesn’t memorize texts like a person memorizes a poem. Instead, it reads billions of text fragments and learns which patterns occur most often.

Examples of such patterns:

• Questions usually end with a question mark.
• After “if” often comes “then.”
• Recipes often contain “take,” “add,” “mix.”

📦 Analogy: It’s like hearing French spoken thousands of times and beginning to guess how sentences continue — even if you don’t know the meanings. You don’t know exactly, but you sense “the shape” of the language.

🎓 What This Means

• The model is not tied to specific facts, but it’s very good at producing natural-sounding text.
• It can “pretend” to be a programmer, doctor, or teacher — if you guide it properly.
• Everything it does is simply building text token by token, based on a huge probability base and patterns.

🔢 How the Model “Thinks”: Token Prediction

When you type something like:

“Tell me an interesting fact about…”

the model does not know in advance what you want to hear. It doesn’t search the internet, nor query a database. Instead, it does what it does best — predicts which word (token) is most likely to come next.

🎲 The Model Doesn’t Know — It Guesses

LLMs operate via token prediction. This doesn’t mean they understand meaning — they calculate probabilities of continuations.

📚 Analogy: You’re playing “finish the sentence.” Someone says: “Today I ate…”

You guess the most likely continuations:

• “pizza”
• “soup”
• “burger”

That’s exactly what the model does — guessing based on collective experience.

🧩 What Tokens Are

Before “thinking,” the model splits your text into tokens. A token is not always a whole word, it could be:

• a single letter (a),
• part of a word (infor + mation),
• or a full word (data).

Example phrase:

“Interesting fact about space”

Could be tokenized as: ["In", "terest", "ing", " fact", " about", " space"].

LLMs read and generate in tokens, not words.

🧠 How Prediction Works

At each generation step, the model:

1. Looks at all previous tokens.
2. Calculates which tokens could be next.
3. Assigns probabilities.
4. Picks the most probable (or a random one from top-N, depending on settings).

📊 Analogy: Imagine the model holding a probability table:

If using “hard” prediction (like ChatGPT), it chooses the top one. If randomness is enabled (temperature > 0), it may pick a less likely option for variety.

🧠 Why Models “Hallucinate”

When a model starts confidently inventing non-existent facts, it’s called a hallucination. At first glance this looks like a bug, but really: 🔍 hallucination is natural behavior — the model must always predict a token, even if it doesn’t know the answer.

🎲 Analogy: A Smart Storyteller Who Can’t Say “I Don’t Know”

Imagine someone who must answer every question, even if they don’t know. They’re well-spoken, know many facts, and can improvise.

You ask:

“Who wrote the novel Blue Bear?”

There is no such novel. But not wanting to look clueless, they reply:

“Probably… Peter Handke. Sounds like his style.”

That’s how the model behaves: it doesn’t know, but must generate something — so it chooses the most probable continuation based on similar patterns.

📦 Why This Happens Technically

The model has no access to a database of “truth.” It was trained on billions of text fragments, and during generation it:

• does not check facts online,
• does not say “I don’t know” unless explicitly trained to,
• does not refuse to answer unless prompted to.

It just predicts likely words based on the question’s framing.

⚠️ Why It’s Dangerous

• Hallucinations sound convincing — especially to non-experts.
• They can creep into reports, code, emails unchecked.
• The model itself doesn’t know it’s hallucinating — it has no sense of truth.

🛡 How to Reduce Hallucinations

💡 Golden rule: The model doesn’t know — it only pretends to know. Your job is to let it doubt.

1. Rephrase your question

   Instead of: “Who wrote Blue Bear?”

   Ask: “Does a novel called Blue Bear exist? If yes, who wrote it?”

2. Add an instruction

   “If no reliable info exists — say so.”

3. Use external sources

   RAG systems (Retrieval-Augmented Generation) or models with internet access can verify.

4. Add context

   Instead of: “Tell me about John Smith.”

   Say: “Tell me about John Smith — CEO of Acme in 2020 (if such exists).”

🛠 Model Evolution: From Transformers to Tool Use

When the first LLMs like GPT-2 appeared, they could do only one thing: continue text by guessing token after token. It was like smart autocomplete, but without any ability to access the outside world or solve complex tasks step by step.

Modern models such as GPT-4, Claude, and Gemini have become much more flexible and “self-aware” — not because they “got smarter,” but because they gained new ways to interact with external information.

📡 1. Retrieving External Information — RAG

RAG (Retrieval-Augmented Generation) is an approach where the model doesn’t just predict words, it first retrieves relevant information from an external source (e.g., a knowledge base or documentation), and then uses what it found during generation.

📚 Analogy: you don’t remember the exact year a company was founded, but you know how to look it up. Instead of making it up, you search, read the result — and only then answer.

Here’s how it works:

1. You ask a question.
2. The model forms a search query.
3. Snippets are pulled from a knowledge base/search.
4. The model uses these data as “context” and generates an answer.

🔍 Example: Question: “What perks does a Pro plan user get on Stepik?” A model with RAG first finds the relevant document and then produces an accurate, reliable answer — without inventing anything.

🧰 2. Using Tools — Tool Use

Tool Use is when the model doesn’t just “talk,” but delegates parts of the work to external services:

• calls a calculator;
• queries an API;
• plots a chart;
• runs an SQL query.

⚙️ Analogy: You’re an engineer calculating the load on a beam. Instead of doing it all in your head, you:

• enter the formula into a calculator,
• verify values in Excel,
• plot a load chart. The model does the same — if it’s allowed to use “tools.”

🔧 What this enables:

• Higher accuracy: computations are done by a specialized module, not the model itself.
• Complex tasks become solvable — e.g., scheduling, budget planning, graph construction.

🧮 Example: You ask: “Calculate how much remains from $1000 after a 13% tax and split the remainder into 4 categories in a 4:3:2:1 ratio.” The model:

1. Computes the tax — via a calculator.
2. Splits the remainder — with exact arithmetic.
3. Returns the result — nicely formatted.

🧠 3. Chain of Thought (CoT)

Previously, the model answered immediately. But on complex tasks (e.g., math or logic) this often failed. Chain of Thought is a way to have the model reason step by step, as if writing its thoughts on scratch paper.

🧱 Analogy: you’re solving an exam problem. First write the givens, then the formula, substitute values, compute, and only then record the answer. That’s how Chain of Thought works.

You can even trigger CoT manually — just add to your prompt:

“Solve step by step, explaining each action.”

🧩 What this gives:

• Higher accuracy on complex tasks.
• More transparent reasoning.
• Helps diagnose errors, if any.

🔍 Example: Question: “If Masha had 12 apples, gave 3 to Petya, and then half of what remained to Vanya, how many does she have left?”

Without Chain of Thought: “6” (wrong)

With Chain of Thought:

1. Masha had 12.
2. Gave away 3 — 9 left.
3. Gave away half — 4.5, so 4.5 left. ✅ Answer: 4.5

🧪 Combining All Three Approaches

Modern LLMs increasingly use combinations:

• First retrieve from a base (RAG),
• Then reason step by step (CoT),
• Then call the needed tool (Tool Use).

💼 Business scenario: “Prepare a brief sales report for last quarter, compare with the previous one, plot a chart, and draw a conclusion.”

The model can:

1. Find the needed metrics in the base (RAG).
2. Compute percentage changes (Tool Use).
3. Build the report logic step by step (CoT).

🔍 Practice: Token Inspector

When working with an LLM, it often feels like a black box: you enter a prompt, the model outputs a response, and it’s unclear why it chose those exact words. But with tools like Token Inspector, you can literally peek inside the model’s head.

🧠 What Token Inspector Does

Token Inspector is a visualization tool for the generation process. It shows:

• which tokens the model predicted at each step;
• what alternatives it considered;
• with what probability it chose each one;
• how its evaluations shifted depending on context.

📊 It’s like seeing what a person was thinking while choosing the next word in a sentence. For a moment, they might consider “coffee,” “tea,” “juice” — and then pick “coffee.” Token Inspector reveals that whole process.

📚 Analogy: Autocomplete with a Transparent History

Imagine you’re typing a message in a messenger, and it suggests the next word: "I’m going to the..." Options:

• store
• school
• park
• theater

You choose “park” — but the other options were also in your mind. Token Inspector shows all these alternatives, along with how likely they were.

🔬 Why This Matters

1. Prompt Diagnostics

If the model’s answer isn’t what you expected, Token Inspector helps you see:

• where it “went off track”;
• at which token it diverged;
• whether an alternative existed, and how likely it was.

🧠 Example: You ask: “Translate from English to Russian: cat.”

The model replies “котик” instead of “кот.”

In Token Inspector you’ll see it also considered “кот,” “кошка,” “котёнок” — but chose “котик” as most probable.

2. Style Analysis

If you’re building a business-style assistant, Token Inspector helps you understand why the model picks simpler or more formal words, and where you can adjust the prompt.

3. Teaching Students and Teams

It’s a powerful educational tool: you can show colleagues or students that the model isn’t “stupid”, it simply relies on its statistical experience with tokens.

🛠 Where to Use It

• OpenAI Playground (with token log option enabled)
• OpenAI Token Inspector (official tool)
• Prompt Engineering UI or other IDEs with log-probs support
• LangChain or LlamaIndex with query tracing enabled

🚫 Common Mistakes

• Expecting “knowledge of everything”: students assume the model knows the entire Internet. In reality, it only guesses, it doesn’t search.
• Incorrect prompt phrasing: “Who first discovered the Moon?” — triggers a hallucination because it presupposes a fact that doesn’t exist.
• Ignoring context limitations: models have a limited “attention window” — they can’t process more than N tokens (depending on the version).

❓ FAQs

Q: Why does the model confidently lie?

A: Because it predicts the most probable response — not truth.

Q: Can I check sources?

A: Not in plain ChatGPT. Use RAG or plugins.

Q: Does the model learn from me?

A: No. It doesn’t update training from conversations, only simulates memory per session.

📌 Conclusion

Today we learned:

• How LLMs work and why they predict tokens rather than “think.”
• How Token Inspector reveals generation logic.
• Why hallucinations occur and how to reduce them.
• How LLMs evolved with RAG, Tool Use, and reasoning.

These fundamentals will help you write better prompts, understand model behavior, and avoid common pitfalls.

👉 In the next lesson, we’ll explore how to design a proper prompt — from simple commands to structured interfaces.