Imagine you write a query to ChatGPT and get a response. It seems related to the topic, but… not quite what you expected. Or the opposite — it’s perfect: precise, useful, and concise. But how can you understand why one prompt works better than another? How do you compare them objectively?

In this lesson, you’ll learn to measure prompt quality, not just “feel” it. We’ll look at how to evaluate model answers with metrics, how to run A/B prompt experiments, and how to improve prompts using quality analysis.

🧠 What You’ll Learn

• Which metrics are used to assess prompt quality
• How to apply PromptEval and Langfuse when working with LLMs
• What A/B testing of prompts is and how to run it
• Which techniques help objectively verify results
• How to avoid subjectivity when working with LLMs

📚 Key Terms

• Relevance — How well the model’s answer matches the query topic
• Coherence — Logical flow and connectedness of the answer
• Helpfulness — How useful and applicable the answer is to the task
• Latency — Time to generate the answer
• A/B test — A method for comparing two versions of a prompt to find the more effective one
• PromptEval — A tool for automatic assessment of prompt quality
• Langfuse — A platform for analyzing and tracking LLM behavior using real metrics

🧭 Why Metrics Matter

When working with language models — especially in business or product development — intuition alone isn’t enough. The phrase “seems to work fine” won’t help you make decisions, scale solutions, or convince others of their effectiveness.

📌 Imagine:

You’re tuning a car’s autopilot. It turns correctly 7 times out of 10, but you don’t know why it sometimes fails and which mode performs better. Would you trust it on the road? Same with prompts.

🔍 Metrics are needed to:

• 📏 Objectively evaluate answer quality
• 📊 Compare prompt variants (A/B)
• 🔄 Test and improve consistently
• 🛠 Identify weak points and bugs
• 🧪 Iterate — what did the new wording change?

🧠 Analogy: grading homework

A prompt without metrics is like a student grading themselves:

“Well, I think I solved it correctly.”

A prompt with metrics is like a teacher with criteria:

“You stayed on topic (relevance), were clear and logical (coherence), gave practical advice (helpfulness), and did it quickly (latency). Score — 9/10.”

💡 This is especially critical when:

• you are building a customer-facing chatbot (it must be useful, not just chatty),
• you are creating internal tools for employees (speed and accuracy matter),
• you are launching a generative system for clients (you must justify why your prompt is better than competitors’).

Metrics help move from art to engineering: from “by feel” to systematic work with prompts.

📏 Core Metrics

To evaluate prompts systematically, it’s important to understand what exactly we measure. Below are key quality metrics used when working with LLMs.

1. Relevance — staying on topic

What it is: How well the model’s answer addresses your exact query instead of drifting off-topic.

❌ Low relevance example:

Query: “List beginner books for JavaScript”

Answer: “Python is a powerful language to start programming. Here are 5 popular books…”

✅ High relevance example:

“Here are 5 JavaScript books recommended for beginners: ...”

📚 Analogy: You asked the librarian for JavaScript books, and they brought a pile on Java. Useful — but not the point.

2. Coherence — logic and connectedness

What it is: The answer should read as a coherent, logical text, not a random set of fragments.

❌ Low coherence example:

“JavaScript is a language. It browser. You can functions. Compatible with HTML.”

✅ High coherence example:

“JavaScript is a programming language used in browsers. It enables interactive elements on web pages and is closely related to HTML and CSS.”

🧠 Analogy: Like a bad translation: the sentences might be correct, but the logic and flow are broken. Hard to read smoothly.

3. Helpfulness — practical utility

What it is: Even if the answer is on topic and logical, it must help you accomplish what you intended.

❌ Unhelpful example:

“JavaScript is a popular language. Many people use it.”

✅ Helpful example:

“Start with ‘Eloquent JavaScript’, then practice on Codewars — they have beginner-friendly problems.”

🔧 Analogy: You ask how to fix a faucet. One person says, “Water is important.” Another says, “Here’s how to unscrew and replace the gasket.” Which is more helpful?

4. Latency — response speed

What it is: How long the model takes to produce an answer.

Especially critical for:

• support chatbots,
• search assistants,
• real-time scenarios (games, call centers).

❌ Poor latency example:

The answer arrives after 10 seconds — the user has already left.

✅ Good latency example:

The answer arrives in 1–2 seconds — convenient and not annoying.

⚡ Analogy: A waiter brings the check 15 minutes after you ask. Even if the bill is correct — the experience is already spoiled.

These four metrics together give a holistic view of prompt quality. A strong prompt:

• answers your query precisely,
• is logically connected,
• provides useful outcomes,
• and does so quickly.

⚖️ A/B Testing Prompts

A/B testing is a method for comparing the effectiveness of two (or more) versions of the same prompt to determine which works better. This is especially important when:

• you want to increase accuracy or usefulness,
• you’re improving an assistant’s UX,
• you’re building a system where prompts are critical for business results.

🔁 How it works

You create two prompt versions:

• Prompt A — baseline, current, “as is”
• Prompt B — improved: added clarification, output format, examples, steps

Then you:

1. Feed identical inputs (the same user query) to both prompts.
2. Collect both answers.
3. Compare by metrics — e.g., relevance, coherence, helpfulness, latency.
4. Conclude: which prompt performs better?

🧪 Example

❓ Goal: get a list of JavaScript books for a beginner

🔹 Prompt A (baseline):

“Recommend JavaScript books.”

🔸 Prompt B (improved):

“Give 3 JavaScript books for a beginner, each with a short description and difficulty level. Return the answer as a bulleted list.”

📊 Comparison:

➡️ Conclusion: Prompt B wins — slightly slower, but more useful, logical, and structured.

🧠 Analogy: coffee tasting

You run a café and try two cappuccino recipes:

• One — your usual.
• The other — with new milk/espresso proportions.

You serve both to customers without telling which is which. Then you ask: which tastes better, smoother, richer?

📌 That’s A/B testing — a method to choose what truly works better, not what “seems” better.

🛠 Why do this?

• Avoid guesswork. Not “I think,” but “we tested.”
• Compare variants objectively.
• Iteratively improve prompt quality.

A/B testing is especially important when you:

• build an LLM product (bot, editor, search),
• write prompts for a whole team,
• train employees to use LLMs for work tasks.

✅ Tip: don’t change everything at once

For a fair comparison, change only one thing:

• either add examples (few-shot),
• or specify the output format,
• or adjust the query wording.

That way you’ll know what caused the effect.

⚙️ Tools: PromptEval and Langfuse

To analyze prompt quality beyond gut feeling, use dedicated tools. They help evaluate, improve, and track model behavior on both test and real data.

🧪 PromptEval — a “lab” for prompt evaluation

PromptEval is a tool for quick, automated quality checks of LLM outputs.

What it can do:

• load “prompt + answer” pairs (e.g., A and B),
• compare them by key metrics: relevance, coherence, helpfulness, and others,
• produce numeric scores — e.g., “Answer B is 25% better in helpfulness,”
• let you configure custom metrics for your own criteria.

📌 Analogy: You’re grading essays. PromptEval is like an electronic assistant that evaluates style, argumentation, and correctness by patterns — fast and emotion-free.

When to use:

• mass testing of prompts,
• A/B comparison of versions,
• automating analytics in an LLM project.

📊 Langfuse — “Google Analytics” for LLMs

Langfuse is an analytics platform that tracks how people interact with your model in real conditions.

What it provides:

• logs all requests and responses, including time, context, and prompt version,
• visualizes user behavior — who asked what and when,
• shows drop-off points: where users are unhappy, where the prompt failed,
• allows reports and dashboards to improve UX and prompt engineering.

📌 Analogy: Like cameras and reports in a store: you see where customers get lost, where queues form, what doesn’t work — and make decisions from that.

When to use:

• if you have a live LLM product (bot, generator, editor),
• when you need real-time behavior insights,
• to build funnels and track prompt quality.

🧩 Together — ideal

• First, validate hypotheses with PromptEval (fast, offline).
• Then monitor real behavior and metrics via Langfuse (online, in production).

🧠 Techniques to Increase Objectivity

When working with LLMs, it’s not enough to get a “plausible” answer — you need to assess its quality objectively. Below are three techniques to achieve that even without specialized metrics/tools.

1. Clarifying questions — like probing in a dialogue

If the model’s answer is too generic, shallow, or unconvincing, don’t stop. Add a request for argumentation, a source, or a specific output form.

📌 Examples:

• ❌ Query: “Why is it important to test code?”

  Answer: “To avoid bugs.”

• ✅ Follow-up: “Give 3 concrete arguments with examples. Name the types of testing.”

💡 Analogy: like an interview. You don’t accept “Yeah, I’ve done that.” You ask: “Can you give an example? Where did you apply it?”

2. Step-by-step generation — break complexity into stages

Instead of one long request (“Do everything at once”), give a chain of short instructions:

1. “First think about which criteria matter.”
2. “Now propose 3 solution options.”
3. “Compare them by the criteria and pick the best.”

This forces the model to:

• consider the path to a solution rather than guess,
• structure the answer logically instead of dumping a “finished” output,
• produce intermediate steps you can verify.

📌 Analogy: instead of “Write a story,” first “Create a protagonist,” then “Describe the conflict,” then “Draft the ending.”

🧠 This is Chain-of-Thought (CoT), and it significantly improves explainability.

3. Reference answer comparison — template-based checking

If you’re solving a task that has a correct answer (a fact, a calculation, a definition), compare the model’s result with a reference.

📌 Examples:

• ✅ Input: “What is 12*8?”
• ✅ Model answer: 96
• ✅ Reference: 96 → Match: good → Mismatch: refine the prompt or check interpretation

📌 Suitable for:

• fact-checking,
• algorithmic tasks,
• paraphrasing with meaning preserved.

💡 Analogy: you take a test and check against the answer key. A simple but powerful way to see where the prompt fails.

🧨 Common Mistakes

• You can’t measure “taste” — prompts are judged by feelings (“I like it”) without formal criteria.
• Comparison without controls — A/B tests run without fixing inputs and conditions.
• Ignoring latency — in real systems, response speed is critical (especially in UIs).
• Testing on a single example — even a good impression from one answer proves nothing. Use at least 5–10 cases.

❓ FAQs

Q: Can prompt quality evaluation be fully automated?

A: Partially — yes. But subjective aspects (humor, style, tone) still require human judgment.

Q: How do I choose which metric to use?

A: Depends on the goal: in chatbots, helpfulness and latency matter; in QA systems — relevance and precision.

Q: When will A/B testing fail?

A: If you test on an unstable sample or across tasks too different — results won’t be representative.

Q: Can I do it “by eye”?

A: You can — at the prototyping stage. But at scale it leads to regressions and bugs.

📌 Summary

Today you learned that:

• Prompt quality can be measured, not just felt
• There are formal metrics: relevance, coherence, helpfulness, latency
• Tools like PromptEval and Langfuse help evaluate and improve prompts
• A/B tests let you compare phrasings in practice
• A step-by-step approach and clarifying instructions are key to higher accuracy

Next step — learn to use these metrics to optimize entire prompt systems, especially in products and assistants.