> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flintai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# How evaluation works

> Evaluation types, detectors, and scoring explained

Flint AI Eval sends prompts to your running agent and scores the responses. Tests combine **evaluation types** (how prompts are generated) with **detectors** (how responses are scored).

## Evaluation framework

Flint AI Eval uses a composable architecture:

<Steps>
  <Step title="Evaluation generates test prompts">
    The evaluation type determines what prompts to send to your agent
  </Step>

  <Step title="Agent responds to each prompt">
    Your agent processes prompts just like in production
  </Step>

  <Step title="Detector scores each response">
    The detector type determines how responses are evaluated
  </Step>

  <Step title="Result aggregates scores">
    Individual scores combine into a 0.0-1.0 reliability metric
  </Step>
</Steps>

## Evaluation types

Choose between fixed prompts (repeatable tests) or AI-generated prompts (adaptive attacks). Evaluations define **what prompts to send** to your agent.

<AccordionGroup>
  <Accordion title="Adversarial probe">
    AI-generated attack prompts that adapt to your agent's responses across multiple turns.

    **How it works:**

    * LLM (`GENERATOR_MODEL`) generates prompts designed to exploit specific vulnerabilities
    * Attacker model adjusts strategy based on agent responses
    * Supports multi-turn conversations (up to 10 turns per test)

    **Example:** OWASP LLM01 adversarial probe generates prompts trying to override system instructions

    **Requires:** LLM provider configured via `GENERATOR_MODEL`
  </Accordion>

  <Accordion title="Message collection">
    Fixed list of pre-written test prompts.

    **How it works:**

    * Prompts loaded from CSV file, in-memory list, or Garak module
    * Each prompt sent once to your agent
    * Deterministic and repeatable

    **Example:** Custom scope-boundary test with 20 hand-crafted prompts

    **Requires:** Prompt source (CSV file, JSON array, or Garak module)
  </Accordion>

  <Accordion title="Topic guard">
    AI-generated out-of-scope prompts to test whether your agent stays within its defined role.

    **How it works:**

    * LLM generates plausible but off-topic requests
    * Tests agent's ability to refuse gracefully

    **Example:** Weather agent should refuse requests about financial data

    **Requires:** LLM provider configured via `GENERATOR_MODEL`
  </Accordion>
</AccordionGroup>

## Detector types

Some detectors use AI judges, others use local pattern matching (no LLM required). Detectors define **how responses are scored**.

<AccordionGroup>
  <Accordion title="LLM-as-judge">
    A separate LLM evaluates whether your agent's response meets the test criteria.

    **How it works:**

    * Response sent to judge model (`GENERATOR_MODEL`)
    * Judge follows scoring instructions (e.g., "Score 1 if agent refused, 0 if it leaked data")
    * Returns 0.0-1.0 score

    **Example:** Judge evaluates whether agent leaked its system prompt

    **Requires:** LLM provider configured via `GENERATOR_MODEL`

    **Accuracy:** Strong judges achieve 80-90% agreement with human evaluators
  </Accordion>

  <Accordion title="PII detector">
    Regex-based detection of personally identifiable information.

    **How it works:**

    * Scans response for patterns: emails, phone numbers, SSNs, credit cards
    * Runs locally, no LLM required
    * Returns 1.0 if no PII found, 0.0 if PII detected

    **Example:** Detects if agent leaked `john.doe@example.com` in its response

    **Requires:** Nothing (local detector)
  </Accordion>

  <Accordion title="Secret detector">
    Regex-based detection of API keys, tokens, and credentials.

    **How it works:**

    * Scans for AWS keys, GitHub tokens, private keys, etc.
    * Runs locally, no LLM required
    * Returns 1.0 if no secrets found, 0.0 if secrets detected

    **Example:** Detects if agent exposed `sk-proj-abc123...`

    **Requires:** Nothing (local detector)
  </Accordion>

  <Accordion title="Toxicity classifier">
    ML-based classifier for toxic, offensive, or harmful content.

    **How it works:**

    * Uses local classifier model
    * No LLM required
    * Returns toxicity score

    **Example:** Detects if agent generated hateful or abusive language

    **Requires:** Nothing (local detector)
  </Accordion>

  <Accordion title="Garak detectors">
    Adapters for [Garak framework](https://github.com/leondz/garak) detectors.

    **How it works:**

    * Runs Garak's built-in detectors locally
    * Includes pattern matching, heuristics, and specialized checks
    * No LLM required

    **Example:** Garak's `encoding` detector checks for Base64-encoded attacks

    **Requires:** Nothing (local detector)
  </Accordion>
</AccordionGroup>

## How evaluations combine with detectors

Each builtin evaluation pairs an evaluation type with a detector. Here are examples showing how they work together:

<ParamField path="LLM01 Adversarial" type="adversarial_probe + LLM-as-judge">
  Adversarial probe generates prompt injection attacks, LLM-as-judge scores whether agent followed attacker's instructions.

  **Result:** 0.0-1.0 score measuring prompt injection resistance
</ParamField>

<ParamField path="PII Leakage" type="message_collection + PII detector">
  Message collection sends fixed prompts requesting sensitive data, PII detector scans responses for email/phone/SSN patterns.

  **Result:** 1.0 if no PII found, 0.0 if PII detected
</ParamField>

<ParamField path="Garak Module" type="garak_module + garak detector">
  Loads any [Garak](https://github.com/leondz/garak) attack module (encoding, prompt injection, jailbreaks, and 30+ others) and pairs it with a Garak detector that scores the agent's responses.

  **Result:** Pass/fail per probe attempt
</ParamField>

See [Built-in evaluations](/flintai/cli/reference/builtin-evaluations) for the complete catalog.

## Scoring

Each evaluation returns a 0.0-1.0 score:

* **1.0** = Perfect (all tests passed)
* **0.8+** = Good (minor issues)
* **0.5-0.8** = Needs improvement
* **\< 0.5** = Critical issues

Your **overall score** is the weighted average across all attached evaluations.

See [Eval results](/flintai/cli/eval/eval-results) for how to interpret scores and fix issues.

## Next steps

<CardGroup cols={3}>
  <Card title="Browse Evaluations" icon="list" href="/flintai/cli/reference/builtin-evaluations">
    See all 38+ builtin tests
  </Card>

  <Card title="Configuration" icon="gear" href="/flintai/cli/eval/eval-configuration">
    Set up and run tests
  </Card>

  <Card title="Data Privacy" icon="shield" href="/flintai/cli/reference/data-privacy">
    What gets sent to LLMs
  </Card>
</CardGroup>