Traces tell you what happened. They don't tell you whether it was good. You can stare at a hundred traces and still not know if your new prompt is actually better than the old one, or if that model swap you're considering will quietly make answers worse for 5% of users. That's what evaluations are for.
The two building blocks
LangSmith's evaluation framework rests on two things: datasets and evaluators. A dataset is a collection of examples - typically an input and a reference (expected) output - that represents the cases you actually care about getting right. An evaluator is a function that scores your app's actual output against that reference, or against some other criteria, and returns a score.
Run your app across every example in a dataset and you get an experiment - a snapshot of how your app performed on that exact test set, with a score attached to every example. Change a prompt, run the same dataset again, and you get a second experiment you can compare side by side against the first.
Where datasets actually come from
You could write test cases by hand, and sometimes that's the right move for known edge cases. But the more common - and more realistic - workflow starts from production: capture traces as users hit your app, review them in the dashboard, and when you spot an interesting example (a great answer, a bad one, an edge case you hadn't considered), add it to a dataset with one click. Over time this builds a test set made of real user behavior instead of cases you imagined in advance.
Production traces feed a dataset, which becomes the fixed test set every future experiment is measured against.
Evaluators: not just exact-match
Some things are easy to check with a plain assertion - did the output contain valid JSON, did it stay under a length limit, does a specific field match exactly. LangSmith supports these simple code-based evaluators fine. But most LLM output quality questions aren't that binary: was this summary accurate? Was this customer support reply helpful and on-tone? Did the agent hallucinate a fact that wasn't in its source documents?
For those, LangSmith supports LLM-as-judge evaluators - a second LLM call, given your app's output plus the criteria you care about, that returns a score and a written rationale. It's not perfect, but it scales to thousands of examples in a way manual review never could, and a written rationale for every score means you can audit why it decided what it decided, not just trust a number.
Putting it into a workflow
The pattern that tends to work in practice: capture production traces continuously, periodically pull interesting or broken examples into your dataset, run an experiment against that dataset every time you change a prompt or swap a model, and compare the new experiment's scores against the previous one before shipping the change. This turns "did that prompt edit help or hurt?" from a guess into a number you can look at.
Some teams go further and run evaluators automatically against a sample of live production traces too - not just offline datasets - which catches quality regressions between the deliberate test runs.
What this buys you
Without this, "improving" a prompt is trial and error with no scoreboard - you tweak the wording, it feels better on the three examples you tried, and you ship it hoping nothing broke elsewhere. With a dataset and evaluators in place, every change gets measured against the same fixed set of real examples, so you can actually tell whether you made things better or just moved the problem somewhere you didn't happen to test.