
How to Evaluate AI Agents on Real Work

By Erin Harris, AI Strategy & Business Transformation Consultant
AI agents should be evaluated on verified task outcomes, appropriate authorization, exception handling and correction effort, not simply the quality of their written answers. OpenAI's November 2 LifeVal research provides a timely reason to update how organizations measure practical assistance. The evaluation uses tasks informed by time-use studies and a survey of 3,995 respondents. Its published examples are research results, not a substitute for testing your own workflows. Source: https://openai.com/index/lifeval/
Start with the business outcome, not the prompt
A service organization might want an agent to prepare a customer appointment. A successful output is not a cheerful confirmation message. It is a correctly identified request, accurate available options, a confirmed authorization boundary and an auditable next step. If the system cannot confirm a slot, the correct result may be a flagged exception rather than an invented booking.
Define a task contract before choosing a model: the starting information, allowed systems, permitted actions, stopping conditions, required evidence and escalation owner. This contract makes it possible to compare an AI-assisted workflow against the current process.
Five evaluation dimensions

First, measure completion accuracy. Specify what constitutes a correct result and how a reviewer will inspect it. A summary may be accurate but incomplete if it omits the one condition that changes a decision.
Second, measure authorization. The agent should be able to distinguish gathering information from sending messages, committing funds, changing records or disclosing sensitive data. Permissions should be enforced by the system, not only described in a prompt.
Third, test exception handling. Include missing documents, contradictory instructions, outdated records and requests outside the agent's authority. A system that asks for help at the right moment may be safer than one that improvises.
Fourth, measure human correction effort. Count the time needed to check outputs, repair errors and resolve handoffs. Apparent speed gains can disappear when verification is expensive.
Fifth, assess traceability. A reviewer should know what source informed the answer, what action occurred and what remains pending. This is particularly important when the agent crosses multiple applications.
A practical pilot sequence
Choose one narrow workflow with an identifiable owner. Document ten to twenty representative cases, including ordinary and difficult situations, without using confidential data in an unapproved environment. Record the human baseline. Run the agent with restricted permissions and human review. Compare correctness, correction time, escalation quality and user experience. Only expand autonomy after the failure modes are understood.
Do not use a benchmark percentage as a forecast of your organization's savings or reliability. Local integrations, policies, data quality and users change the result. The business case should be based on your own measured pilot.
What leadership should decide before rollout
Who can authorize an action? Which steps require a human? What records may the agent access? How are mistakes reversed? What does the user see when the system is uncertain? These are operating-model decisions, not merely technical configuration.
Key Takeaways
- Define a verified outcome before evaluating an agent.
- Test permissions and exceptions as seriously as normal cases.
- Measure review and correction work, not just generation speed.
- Expand autonomy only after a controlled pilot establishes evidence.
An AI Opportunity & Transformation Assessment can help identify which workflows are suitable for this kind of pilot and which need process repair first. Learn more about EH Synergy at https://ehsynergy.com/.

