Evaluator Agent

Overview

Layer 4 on the Mental Model. The Evaluator reads the full conversation and the Worker’s latest answer, then returns structured EvaluatorOutput: feedback text, whether success criteria are met, and whether the user must clarify. Conditional edges end the graph or loop the worker with that feedback.

Behavior

• Structured judgment. with_structured_output(EvaluatorOutput) forces boolean decisions plus feedback (Evaluator-Optimizer judge side). See Building effective agents.
• Benefit of the doubt. File writes and claimed side effects are trusted unless the answer clearly needs more work.
• Stuck detection. Repeating the same mistakes after prior feedback can flip user_input_needed.
• Gate to finish. route_based_on_evaluation ends when success is met or user input is needed; otherwise the worker retries with feedback_on_work.

Evaluator flow

Worker text reply in; end or retry out. Diagram key: ✦ AI agent · second line = Anthropic workflow · purple = actor · white = service.

---
config:
  theme: default
  themeVariables:
    background: "#ffffff"
    fontSize: "14px"
---
flowchart TB
    person[Person]
    reply[Worker reply]
    evaluator["✦ Evaluator LLM
Evaluator-Optimizer"]
    feedback[Feedback on work]
    worker["✦ Worker"]
    doneNode[Graph finish]

    reply --> evaluator
    evaluator -->|criteria not met| feedback --> worker
    evaluator -->|met or need user| doneNode --> person

    classDef agent fill:#e8f0ee,stroke:#0f3d36,stroke-width:2px,color:#0d0d0d
    classDef service fill:#fafaf8,stroke:#c8c8c2,stroke-width:1px,color:#0d0d0d
    classDef actor fill:#f3eef8,stroke:#6b4f9a,stroke-width:2px,color:#0d0d0d
    class evaluator,worker agent
    class reply,feedback,doneNode service
    class person actor

Gradio shows the worker reply and evaluator feedback together when the superstep returns.