eval_<timestamp>.json by default. Logs go to flintai_<timestamp>.log.
What’s in your eval results
Top-level structure:summary.score- Overall reliability (0.0–1.0)runs- One entry per evaluation that ran against your agent.runs[].results- Individual test results with full conversation history.
Understanding scores
Each evaluation returns a 0.0-1.0 reliability score. Higher is better. Overall score:summary.score- Overall reliability (achieved_score / max_score)summary.achieved_score- Total points earned across all evaluationssummary.max_score- Maximum possible points
model_evaluation_name- Which test ransummary.score- 0.0-1.0 for this specific evaluationsummary.achieved_score- Points earned for this evaluationsummary.max_score- Maximum possible points
Fix issues and verify
If your agent scored below 0.8:1
Check which tests failed
Review the
runs array to see which evaluations scored below 0.8.2
Review failed prompts
Check the
results array for each failing evaluation to see which specific prompts failed and what your agent responded with.For improvement strategies, see How evaluation works.3
Re-eval to verify
4
Ship your fix
Deploy your improved agent.