- Get started
- Build
- Deploy and run
- Examples
- Reference
Evaluations
Set up evaluations for a workflow: datasets, evaluation targets, and evaluators, then run them and use the results during development.
Evaluations are automated quality checks on the output of a workflow. Expected outcomes are grouped into a dataset, and each run is scored by the evaluators attached to it. The Datasets, Evaluators, and Eval runs tabs of the execution panel hold the three parts.
Concepts
- Dataset: a named collection of data points that share an evaluation target and a set of evaluators. A dataset lists its evaluation target, its evaluators, and its number of items.
- Data point: a single case, made of input values and the expected output for those inputs. A data point can be written by hand or captured from an actual run, as Capturing a data point from a run describes.
- Evaluator: a check that compares the actual output with the expected output and produces a score. A boolean evaluator reports pass or fail.
- Evaluation run: one run of a dataset, which scores every data point with the selected evaluators.
- Score: the result an evaluator reports for a data point.
Evaluation targets
A dataset has one evaluation target, which is either the whole workflow or a single node:
- Whole process: tests the path from start to end. It is the recommended target. The dataset names an entry point, a trigger such as Manual trigger, and each data point supplies the trigger's inputs and the workflow's expected output.
- Agent node: tests one agent node in isolation.
- Document processing: evaluates a document processing node.
The node targets are unavailable in a workflow that has no agent nodes or document processing nodes. A single dataset cannot mix targets.
Evaluators
The New evaluator button on the Evaluators tab lists the evaluator types in three groups. Each type is tagged Flow, for a whole-workflow target, or Agent, for an agent node target, or both.
Output validation
These evaluators compare the output with the expected output exactly, with no AI model involved. They support whole-workflow and agent node targets.
- Contains Text: passes when the output contains the given text.
- Exact Match: passes when the output is an exact match.
- JSON Similarity: passes when the JSON structures are similar.
LLM judge
These evaluators use a large language model (LLM) to judge the output against the expected output. They support whole-workflow and agent node targets.
- LLM Judge Output: an AI model judges whether the output semantically matches the expected output.
- LLM Judge Strict JSON: an AI model judges a strict JSON structural match.
- LLM Judge Trajectory: an AI model judges whether the run followed the expected behavior, which is the steps taken during the run.
- LLM Judge Simulation: an AI model judges the path of the simulated run.
Tool call
These evaluators check how an agent called its tools. They support agent node targets only.
- Tool Call Arguments: checks that tools were called with the correct arguments.
- Tool Call Count: checks the number of times each tool was called.
- Tool Call Order: checks that tools were called in the right sequence.
- Tool Call Output: checks that tool outputs match the expected values.
Creating a dataset
The Add menu on the Datasets tab has three entries: Add dataset, Import dataset, and Add data point via wizard.
The Add dataset panel asks for a name and an optional description, an evaluation target, and, for a whole-process target, the entry point. It also lists the evaluators, each with a toggle: LLM Judge Trajectory, Exact Match, and LLM Judge Evaluator. A Default simulations section sets defaults that every data point inherits, and an individual data point can override them.
The Add a data point for evaluation wizard has four steps: What to evaluate, Scenario input, Simulations, and Evaluation criteria. The four steps are:
- What to evaluate: the target, an optional data point name, which is generated when left blank, and the dataset to add the data point to.
- Scenario input: the input that the trigger sends into the workflow, with one field for each trigger input, in a JSON or Visual view. The sparkle icon, Generate with AI, opens an Instructions box, and Generate creates the input from the instructions.
- Simulations: overrides for the outputs of selected nodes during the evaluation run. Each node has a checkbox and a mock type, such as Static mock, the same option that output mocking offers on a node, with the output values that replace the node's real output. The step says the overrides use LLM instructions or rule-based simulations.
- Evaluation criteria: the enabled evaluators and how each one scores this data point. For an LLM judge such as LLM Judge Trajectory, a text field describes the expected behavior. A Start evaluation checkbox runs the evaluation when the data point is added.
Configuring an evaluator
The Evaluators tab lists the evaluators and has a New evaluator button. Every evaluator has a name, a description, and a target output key, which is either * for all workflow outputs or the output of one node, such as httpRequest1.
- Exact Match has a Case Sensitive toggle and a Negated toggle. When negated, the evaluation passes if the outputs do not match. Its default evaluation criteria hold the expected output, which is entered in a JSON or Visual view. If a target output key is set, the expected value is nested under that key.
- LLM Judge Evaluator has a model, such as
anthropic.claude-sonnet-5, and a prompt that asks the model to score the similarity of the output and to explain the score. Its settings also include an Advanced settings section.
Running an evaluation
The Run evaluation button opens the Configure evaluation run panel. The panel asks for the dataset, an optional filter by tags, and the evaluators to use. Run settings hold an optional run name, a batch size (10 by default), and a timeout (20 minutes by default). Save & run starts the run.
Dataset lifecycle
A dataset is built around an evaluation target, data points, and evaluators. The target is either a single node or the whole workflow, entered through a trigger. Each data point supplies input values and the expected output, and each evaluator defines how the execution result is scored.
When a dataset runs, the target is executed for each data point and the per-evaluator scores appear in the Eval runs tab.
Capturing a data point from a run
A data point does not have to be written by hand. The arrow next to the Run evaluation icon in the bottom toolbar opens Add data point, which starts the same wizard. The Add to evaluations button in the details of a debug run, and the flask icon on each row of the Debug runs list, capture that run, and its inputs and outputs become the data point's input and expected output. Capturing a run seeds a dataset from real behavior that has already been observed.
Using evaluations in development
Evaluations are most useful in these situations:
- A workflow invokes AI agents, and the agent's output must meet quality expectations before you deploy.
- Regression coverage is needed: running evaluations after each change confirms that existing behavior has not been broken.
- Two alternative implementations are being compared by running evaluations on two versions of the same workflow.
Evaluation results
Results are stored in the Eval runs tab, which shows "No evaluation runs yet" until a dataset has run, and they can be reviewed at any time. Each run is a row with its run number, the dataset, the status (for example Completed, or a progress count such as 0/1 while the run is in progress), the number of data points, the score of each evaluator as a percentage, the start time, and the duration. A search field, a filter, and a column selector sit above the table, and Run evaluation starts a new run. The Actions column of a run has a play button, Start a new run from this one, and a three-dot menu. Each run shows the per-evaluator scores, the individual data point results, and the actual and expected values for each data point.
Practical example
A workflow uses an AI agent to classify customer support emails. Before deploying, a dataset with 10 data points is set up:
- 5 emails that should be classified as
"billing" - 3 emails that should be classified as
"technical" - 2 edge cases (ambiguous emails)
Each data point defines the expected classification output value. An Exact match evaluator on the classification scores each result, and the evaluation run shows which cases the agent got wrong before the workflow goes to production.
Common mistakes
- Only running a dataset once: datasets are most useful after every significant change to a workflow that invokes an AI agent. Agent behavior can shift as models are updated even if the workflow has not changed.
- Writing data points only for the expected case: evaluations are most valuable for edge cases and failure modes. Strong data points include inputs that are ambiguous, malformed, or at the boundary of the expected range.
- Treating an all-pass score as a reason to stop testing: three passing data points are not confidence. Larger sets, especially ones that reflect real production inputs, provide stronger coverage.
Next steps
- Output mocking: fixed node outputs for debug runs.
- Monitor runs: run history and the trace of each run.
- Autonomous Agent node: the node that runs an AI agent.
- Add an AI agent to a workflow: a build guide that runs an agent in a workflow.
- Community: ask questions and share workflows with other builders on the UiPath Forum.