ReliabilityEval

ReliabilityEval parameters, tool call and argument checks, and ReliabilityResult fields.

ReliabilityEval checks the tool calls in a completed agent or team run against expected tool names and arguments.

from agno.eval import ReliabilityEval, ReliabilityResult

Parameters

Parameter	Type	Default	Description
name	Optional[str]	None	Evaluation name.
eval_id	str	Autogenerated UUID	Evaluation ID.
agent_response	Optional[RunOutput]	None	Agent run to check. Provide exactly one of agent_response or team_response.
team_response	Optional[TeamRunOutput]	None	Team run to check, including member responses. Provide exactly one of agent_response or team_response.
expected_tool_calls	Optional[List[str]]	None	Tool names that must appear in the run. Missing names fail the evaluation.
allow_additional_tool_calls	bool	False	Treat tool calls outside expected_tool_calls as allowed extras instead of failures.
expected_tool_call_arguments	Optional[Dict[str, Union[Dict[str, Any], List[Dict[str, Any]]]]]	None	Expected arguments per tool name, matched as a subset of the actual arguments. A list of dicts requires each spec to match at least one call to that tool.
print_results	bool	False	Print the results table after the run.
show_spinner	bool	True	Render the progress spinner. Set to False when the eval runs inside code that must not write to the console.
file_path_to_save_results	Optional[str]	None	File path where the result is saved.
debug_mode	bool	False	Enable debug logs. Defaults to True when the AGNO_DEBUG environment variable is "true".
db	Optional[Union[BaseDb, AsyncBaseDb]]	None	Database where the eval run is stored.
telemetry	bool	True	Log minimal telemetry for the eval run.
result	Optional[ReliabilityResult]	None	Result of the evaluation. Set after a run completes.

Methods

Method	Return type	Description
run(*, print_results=False)	Optional[ReliabilityResult]	Check the response’s tool calls against the expectations. Raises with an async database; use arun() instead.
arun(*, print_results=False)	Optional[ReliabilityResult]	Async version of run().

ReliabilityResult

The evaluation passes when there are no failed tool calls, no missing tool calls, and no failed argument checks.

Field	Type	Description
eval_status	str	"PASSED" or "FAILED".
failed_tool_calls	List[str]	Tool calls outside expected_tool_calls when additional calls are not allowed.
passed_tool_calls	List[str]	Tool calls that matched expected_tool_calls.
additional_tool_calls	List[str]	Tool calls outside expected_tool_calls, recorded when allow_additional_tool_calls=True.
missing_tool_calls	List[str]	Expected tool names that never fired.
failed_argument_checks	List[str]	Tool names whose arguments did not match expected_tool_call_arguments.
passed_argument_checks	List[str]	Tool names whose arguments matched expected_tool_call_arguments.

ReliabilityResult also exposes print_eval(console=None) to render the results table, and assert_passed(), which asserts eval_status == "PASSED" for use in test suites.