Everything so far in this series has been about inspecting traces you already know to go looking for: a bug someone reported, a test you're running deliberately. Production doesn't work that way. Nobody's going to open the traces dashboard every hour just in case, and the failure that actually matters is usually the one nobody was looking for. That's what monitoring and alerts are for.
The monitoring dashboard: what to actually watch
LangSmith's monitoring view aggregates your traces into charts over time rather than showing you one run at a time: request volume, latency (both average and the slower tail end, which matters more than the average), error rate, and token usage and cost. Watched over days and weeks, these charts are what tell you an app is degrading before a specific user complains about it, a slow creep in latency or a rising error rate is a much earlier warning than the first support ticket.
| Metric | What a bad trend usually means |
|---|---|
| Latency (especially p95/p99) | A slow tool, a provider having a bad day, or a prompt that's grown too long |
| Error rate | An upstream API change, a malformed input type you didn't handle, a rate limit being hit |
| Token usage / cost | A prompt or context window quietly growing, or a loop running more iterations than expected |
Alerts: finding out without watching
Rather than staring at those charts, you set thresholds and let LangSmith notify you when one is crossed, error rate above a set percentage over a rolling window, latency past a cutoff, or cost climbing faster than expected. Alerts can route to email, Slack, or a webhook into whatever on-call system your team already uses, so this plugs into an existing workflow rather than adding a new dashboard someone has to remember to check.
Online evaluation: running evals on live traffic
The evaluations and datasets guide covered running evaluators against a fixed dataset before shipping a change. Online evaluators take the same evaluator functions and run them continuously against a sample of real production traces, automatically, as they come in. This is how you catch a quality regression (not just an error, a response that runs fine but is subtly worse) that a fixed test dataset wouldn't have contained, because it only shows up in the messier distribution of real user inputs.
That's the series
Across these six guides you've gone from what LangSmith is and why observability matters, through traces, runs, and projects, setting up tracing in your own app, evaluating it on purpose instead of by vibes, managing prompts safely with the Playground and Prompt Hub, and now watching all of it continuously once it's actually live. That's the complete loop: build it, trace it, test it, ship prompt changes safely, and know the moment something drifts, not the moment a user tells you.