An agent demo that works on the three examples you tried is not the same thing as an agent you can trust running with minimal supervision. Because agents choose their own sequence of steps, the space of things that can go wrong is much larger than with a fixed script - which means evaluation and guardrails aren't optional polish, they're the actual work of getting from demo to something dependable.
What to actually measure
The single most useful metric for an agent is task success rate on a representative set of test cases - not "does it produce plausible-looking output," but "did it actually accomplish what was asked, verified against a known correct outcome." This requires building an evaluation set up front: a collection of realistic tasks with known correct answers or outcomes, run repeatedly as you change the agent, the same way a test suite is run against changing code. If you've been through this site's RAG 101 series, this is the same discipline as RAG evaluation, applied to end-to-end task completion instead of retrieval quality alone.
Beyond raw success rate, it's worth tracking efficiency (how many steps or tool calls a successful run takes - a big jump can signal the agent is thrashing even when it eventually succeeds) and failure mode (not just that a run failed, but which step it failed at and why, so you know what to actually fix).
Evaluation is a loop, not a one-time check - run the same test set every time the agent changes.
Guardrails: bounding what the agent can do
Because an agent's exact sequence of actions can't be predicted in advance, guardrails exist to bound the space of what it's allowed to do, regardless of what it plans. Practical guardrails include: the iteration ceiling from the agent loop guide, scoped tool permissions (an agent that only needs to read data shouldn't hold write or delete access - the same least-privilege principle covered in this site's MCP 101 series applies here), input validation on tool arguments before they execute, and explicit denylists for especially risky actions regardless of what the model decides.
Human-in-the-loop checkpoints
Not every action an agent might take deserves the same level of trust. A common and effective pattern is tiering actions by risk: low-risk, easily reversible actions (searching, reading, drafting) run autonomously, while higher-risk or irreversible actions (sending an email, making a purchase, deleting data) pause for explicit human approval before executing. This doesn't require making the whole agent supervised - it means identifying the specific steps where a wrong decision is expensive, and inserting a checkpoint only there.
Observability: knowing what actually happened
When something goes wrong in production, you need to be able to reconstruct exactly what the agent saw and decided at each step - not just the final output. Logging every planning decision, tool call, and observation (this is precisely what this site's LangSmith 101 series covers in depth) turns "the agent did something weird" into "the agent called this tool with these arguments because it saw this specific observation," which is the difference between a debuggable system and a black box.