Evaluating Agent Performance

Evaluating agents is harder than evaluating simple LLM calls. Agents take multiple steps, use tools, and produce intermediate results — so you need metrics that go beyond just "was the final answer correct?" This lesson covers task completion metrics, trajectory evaluation, cost and latency tracking, and benchmarking different agent architectures.

---

Why Agent Evaluation Is Different

Aspect	Simple LLM Call	Agent
Output	Single response	Final answer + full trajectory
Steps	1	Variable (1–50+)
Cost	Predictable	Variable (depends on steps taken)
Latency	Predictable	Variable (depends on tool calls, steps)
Success criteria	Output quality	Output quality + efficiency + safety

---

Task Completion Metrics

Basic: Did the Agent Succeed?

from dataclasses import dataclass

@dataclass
class AgentResult:
    task: str
    final_answer: str
    steps_taken: int
    tools_used: list[str]
    total_tokens: int
    total_cost_usd: float
    latency_seconds: float
    success: bool
    error: str | None = None

def task_completion_rate(results: list[AgentResult]) -> float:
    """Percentage of tasks the agent completed successfully."""
    if not results:
        return 0.0
    return sum(1 for r in results if r.success) / len(results)