The document loaders and splitters guide ended on a question it couldn't fully answer: how do you actually know if a chunk size change helped? "I tried a few queries and it felt better" isn't an answer you can trust, and it definitely isn't one you can repeat every time you touch a prompt. That's what evaluation is for.
The basic loop: dataset, run, score
Evaluation in LangSmith follows the same three-step loop regardless of what you're testing: build a dataset of example inputs (and, where you have them, expected outputs), run your chain against every example in it, and score each result with an evaluator, a function that decides whether a given output was good.
from langsmith import Client
from langsmith.evaluation import evaluate
client = Client()
dataset = client.create_dataset("support-ticket-eval")
client.create_examples(
inputs=[{"text": "My payment failed twice"}, {"text": "How do I reset my password?"}],
outputs=[{"category": "billing"}, {"category": "technical"}],
dataset_id=dataset.id,
)
results = evaluate(
my_chain.invoke,
data="support-ticket-eval",
evaluators=[correctness_evaluator],
)
Where the dataset comes from
You don't need a large, perfectly curated dataset to start. A handful of real examples pulled from actual usage, including the ones that exposed a bug, is more useful than a large synthetic set that doesn't reflect what people actually ask. As your chain runs in production and gets traced (the subject of LangSmith's tracing guide), a natural source of new dataset examples is simply promoting real traces, especially the ones where something went wrong, into your evaluation set.
Evaluators: LLM-graded vs. code-graded
Some things are easy to check with plain code: does the output match an expected category exactly, is the returned JSON valid, is a required field present. Others are inherently judgment calls, is this summary actually faithful to the source, is this tone appropriate, and for those, an LLM-as-judge evaluator (a second model call that scores the first one's output against a rubric) is the standard approach. Most real evaluation setups use both: cheap, deterministic code checks for anything that can be checked mechanically, and LLM-graded checks reserved for the genuinely subjective parts.
That's the series
Across these nine guides you've gone from what LangChain actually is, through the pieces you'll chain together constantly, retrieval and RAG (including the loading and splitting details underneath it), tool-calling agents and structured output, memory, LCEL, and now evaluation. That's the full loop: build something, chain it together properly, and have a real way to know whether your changes are making it better.