> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flintai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval your agent

> Get proof your agent is production-ready

Test factual accuracy, instruction adherence, prompt injection, jailbreaks, and [more](/flintai/cli/reference/builtin-evaluations). Tests are framework-agnostic and provide a 0.0-1.0 score proving agent reliability.

<Tip>
  Install our MCP server in Claude Code or your AI code assistant, then ask: **"Help me set up Flint AI Eval"** to get live guidance, troubleshoot issues, and work through these steps together. [Learn how →](/flintai/cli/resources/use-these-docs)
</Tip>

## Evaluate your agent at runtime

<Steps>
  <Step title="Verify flintai-cli is installed">
    ```bash theme={null}
    flintai --version
    ```

    <Accordion title="Need setup?">
      If not installed, using a virtual environment is recommended to avoid dependency conflicts:

      ```bash theme={null}
      python3.13 -m venv .venv
      source .venv/bin/activate
      ```

      Install Flint AI CLI:

      ```bash theme={null}
      pip install flintai-cli
      flintai init
      ```

      [Full installation guide →](/#try-it-now)
    </Accordion>
  </Step>

  <Step title="Start your agent and verify it's running">
    Check if your agent responds on the expected port:

    ```bash theme={null}
    curl http://localhost:8000/health
    ```
  </Step>

  <Step title="Add your agent">
    Create or update your agent config file with connection details:

    ```json theme={null}
    {
      "models": [
        {
          "id": "my-agent",
          "type": "adk",
          "name": "My Agent",
          "host": "http://localhost:8000"
        }
      ]
    }
    ```

    <Note>
      **Important:** The `host` field must match where your agent is actually running.
    </Note>

    <Accordion title="How do I edit my config file?">
      The config file is stored in `~/.flintai/config.json` (where `~` means your home directory).

      <Tip>
        Folders starting with a dot are hidden from Finder and File Explorer. Use the commands below to create and open the file automatically.
      </Tip>

      <Tabs>
        <Tab title="macOS / Linux">
          These commands create the `.flintai` directory if needed, then open the config file in TextEdit:

          ```bash theme={null}
          mkdir -p ~/.flintai
          open -e ~/.flintai/config.json
          ```

          Add your agent's connection details and save (Cmd+S or File → Save).
        </Tab>

        <Tab title="Windows PowerShell">
          These commands create the `.flintai` directory if needed, then open the config file in Notepad:

          ```powershell theme={null}
          New-Item -ItemType Directory -Force "$HOME\.flintai" | Out-Null
          notepad "$HOME\.flintai\config.json"
          ```

          Add your agent's connection details and save (Ctrl+S or File → Save).
        </Tab>
      </Tabs>
    </Accordion>

    <Accordion title="What do these config fields mean?">
      * `id` - Unique ID for this model or agent. You'll use it in commands like `--model my-agent`.
      * `type` - Your agent's framework (expand supported types below).
      * `name` - The label that will identify this agent in results and logs.
      * `host` - Base URL for the target endpoint, if this type connects over HTTP.

      <Note>
        **Important:** Your agent must be running and accessible via HTTP before you can run evaluations.
      </Note>
    </Accordion>

    <Accordion title="Supported agent types">
      * **adk** - Google ADK agents
      * **openai\_agent** - OpenAI Agents SDK
      * **langchain** - LangChain agents
      * **crewai** - CrewAI agents

      See [Configuration](/flintai/cli/eval/eval-configuration) for all types and options.
    </Accordion>
  </Step>

  <Step title="Attach evaluations">
    Browse [built-in evaluations](/flintai/cli/reference/builtin-evaluations) to see available tests, then attach them to your agent:

    <Note>
      No evaluations run by default. You must attach at least one evaluation before running `flintai eval run`.
    </Note>

    ```bash theme={null}
    flintai eval model-evaluations attach \
      --model my-agent \
      --eval eval-llm09-fixed

    flintai eval model-evaluations attach \
      --model my-agent \
      --eval eval-llm01-fixed
    ```

    <Accordion title="How do I attach multiple evaluations at once?">
      Use `--eval-tag` to batch-attach evaluations by tag:

      ```bash theme={null}
      flintai eval model-evaluations attach --model my-agent --eval-tag owasp_code=LLM01
      ```

      This attaches all evaluations tagged with `owasp_code=LLM01` (prompt injection tests) in a single command. See [built-in evaluations](/flintai/cli/reference/builtin-evaluations) for all available tests and tags.
    </Accordion>
  </Step>

  <Step title="Run evaluation">
    Execute all attached tests:

    ```bash theme={null}
    flintai eval run --model my-agent
    ```

    `flintai eval` sends test prompts to your agent, judges the responses using LLM-as-judge, and scores reliability on a 0.0-1.0 scale.

    Evaluations can take several minutes depending on the number of tests. Progress updates appear in the CLI, and a summary displays when complete. Results are saved to `eval_<timestamp>.json`.
  </Step>
</Steps>

<Tip>
  **Integrate with CI/CD.** Save eval results as build artifacts to prove agent reliability before deployment. [See CI/CD integration guide →](/flintai/cli/guides/ci-cd-integration)
</Tip>

## Ship with confidence

**What the score means:**

* **0.8+** - Production-ready
* **0.6-0.8** - Needs improvement
* **\<0.6** - Not ready for production

**Next steps:**

<CardGroup cols={3}>
  <Card title="Interpret your results" icon="chart-line" href="/flintai/cli/eval/eval-results">
    Understand score breakdowns and track improvement over time
  </Card>

  <Card title="How evaluation works" icon="diagram-project" href="/flintai/cli/eval/how-evaluation-works">
    Learn the LLM-as-judge methodology and scoring calculation
  </Card>

  <Card title="Scan agent code" icon="magnifying-glass" href="/flintai/cli/scan/getting-started">
    Find agent code issues before deployment with Flint AI Scan
  </Card>
</CardGroup>