Evaluating and observing agents in production: the discipline that separates a demo from a product

Agent demos are easy to fall in love with. You give it a goal, it reasons through a few steps, calls a tool or two, and returns something that looks like judgement. It feels finished. It is not. A demo is a single sample from a distribution you have barely begun to explore, and the most important property of a production agent, that it behaves acceptably across that whole distribution, is exactly the property a demo cannot show you.

This article is about the work that begins after the first successful run: how to observe what an agent actually does, how to evaluate whether it keeps doing it well, and how to keep the economics of a digital workforce from quietly running away. It is a distinct concern from identity and permissions, which decide what an agent is allowed to do. Here the question is narrower and more stubborn: is it doing it well, can you prove it, and what is it costing you.

A demo proves it can work once

The first successful run is the part that gets the screenshot and the standing ovation in the planning meeting. It is also the cheapest, least informative part of the whole exercise. Production means the agent meets inputs you did not write, did not foresee, and in some cases would never have thought to try: the malformed record, the ambiguous instruction, the tool that times out, the customer who phrases the request in a way your test set never imagined.

An agent is non-deterministic and open-ended. The same prompt can take different paths on different runs, and the space of possible inputs is effectively unbounded. So the question that matters is never “did it work?” It is “how often does it work, on what, and how would I know the day that changes?” Everything after the first successful run, the instrumentation, the test suites, the budgets, the alerts, is the actual job. The demo was the easy part.

Observability: trace every step

You cannot manage what you cannot see, and with an agent there is a lot to see. A single task may involve several model invocations, a chain of reasoning, multiple tool calls each with their own arguments and results, and a final output assembled from all of it. If something goes wrong, “the agent gave a bad answer” is not a diagnosis. The diagnosis lives in the steps.

So capture them. For each task, record the agent’s reasoning, every tool call with its arguments and its result, and the final output, as a structured trace. Amazon Bedrock Agents expose this: their orchestration emits trace data that shows the model’s rationale, the tools (action groups) it chose, the inputs it passed and what came back. Log that trace, for example to Amazon CloudWatch, so it is durable, searchable and tied to a task identifier rather than living only in a response you have already thrown away.

A trace does not need to be elaborate to be useful. What matters is that every step is captured and reconstructable after the fact:

# one task, one trace record: durable and reconstructable
{
  "task_id": "req-8a31f0",
  "input": "reconcile invoice 5567 against the PO",
  "steps": [
    { "reasoning": "need the PO total before comparing" },
    { "tool": "get_purchase_order",
      "args": { "po_id": "PO-4412" },
      "result": { "total": 1840.00 } },
    { "tool": "flag_discrepancy",
      "args": { "delta": 60.00 },
      "result": "ticket-2290 raised" }
  ],
  "output": "Mismatch of $60 flagged for review.",
  "tokens": 3120, "tool_calls": 2
}

The shape matters more than the format: reasoning, each tool call with arguments and result, the final output, and the usage counters that feed the economics. Bedrock Agents emit the equivalent; your job is to log it somewhere durable.

Evaluation: agents are non-deterministic

Because the same input can produce different paths, you cannot validate an agent the way you validate a pure function. Eyeballing one good run tells you almost nothing. What you need is the same instinct a software team brings to a test suite, adapted to a system that does not give the same answer twice.

Build a golden dataset: a representative set of cases, drawn from real and anticipated inputs, each paired with what a good outcome looks like. Run the agent across all of them and assert on outcomes, not on the prose. For a reconciliation agent, the assertion is “the discrepancy was flagged with the right delta,” not “the wording sounded confident.” Where the output is genuinely open-ended and an exact match is too brittle, an LLM-as-judge, a separate model scoring the output against a rubric, can grade quality at scale, provided you spot-check the judge itself.

The headline metric is task success rate: of the representative cases, what fraction reached an acceptable outcome. Measure that, not vibes. Then treat it like any other test suite and regression-test on every change, a new prompt, a new model version, a tweaked tool, because in a non-deterministic system a small change can move the success rate in ways no one intended and no one would notice until a customer did.

# assert on the outcome, not the wording: run the whole golden set
for case in golden_dataset:
    result = agent.run(case.input)
    assert result.flagged == case.expected_flag
    assert abs(result.delta - case.expected_delta) < 0.01

# report the number that matters, every change
success_rate = passed / total   # this is the regression gate

Golden dataset plus outcome assertions plus an LLM-as-judge for open-ended cases. Runtime guardrails sit alongside this, catching unsafe or off-policy output in production rather than in the test suite.

Evaluation before release and guardrails at runtime are complementary, not substitutes. The golden set tells you the agent is good enough to ship; runtime guardrails catch the input the golden set never contained.

Supervised first, autonomy earned

The temptation with a working agent is to hand it the keys at once. The discipline is to make it earn them. Start human-in-the-loop or read-only: the agent proposes, drafts or recommends, and a person approves before anything irreversible happens, or it can observe and answer but not act. While it runs in that mode, you are gathering exactly the data the demo could not give you, the real success rate, on real inputs, with traces to inspect when it falls short.

Then widen scope only as the evidence justifies it. If the success rate holds across a meaningful volume of supervised runs, let the agent act unattended on the low-stakes slice first, keep watching the metric, and expand from there. Autonomy is not a switch you flip on launch day; it is a dial you turn as the numbers earn it. An agent that has handled a few thousand supervised cases at a measured, acceptable success rate has earned a wider remit. One you have watched for an afternoon has not.

The economics of a digital workforce

A traditional service has roughly fixed compute cost per request. An agent does not. It may make several model calls and several tool calls to complete a single task, and a harder or more ambiguous input can trigger more reasoning, more retries and more calls than an easy one. Usage-based pricing means a modest rise in average task complexity can move spend more than you would expect, and because each individual call is cheap, the surprise tends to arrive as a monthly total rather than a single alarming line.

Treat this as a first-class operational concern, not an afterthought. The trace you are already capturing carries the token and tool-call counts; aggregate them into cost per completed task and watch it as a metric in its own right, the same way you watch the success rate. Set budgets and alert when a task, a tenant or a daily total crosses them. The two metrics belong together: a digital workforce is only a good deal when the cost per completed task stays below the value of completing it, and you cannot know that unless you are measuring both sides.

Treat agents as production software

None of this is exotic. It is the ordinary rigour you already apply to any critical system, pointed at a new kind of workload. Monitoring, so you see what is happening. Evaluation, so you know it is still good. Alerting, so a regression or a budget breach reaches a human quickly. Rollback, so a bad prompt or model version can be reversed without drama. Cost control, so the economics stay honest. The novelty of agents is real, but it does not earn them an exemption from operational maturity. It raises the bar, because the system is non-deterministic and acts in the world.

That is the whole distinction. A demo is an agent that worked once, in front of you, on an input you chose. A product is an agent you can observe, evaluate, supervise, budget and roll back, that keeps working on inputs you never saw, and that you can prove is still working tomorrow. The discipline in this article is not the part that comes after the interesting work. On AWS, with Bedrock Agents emitting traces and CloudWatch holding them, it is the work that turns a pilot into something you would actually put your name on.