> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flintai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuration

> Configure models, evaluations, and test assignments

Flint AI Eval tests agent behavior and reliability at runtime. Configuration lives in `~/.flintai/config.json` and defines:

* **What to test** - Your running agent's HTTP endpoint
* **How to test it** - Which evaluations to run
* **When to test** - Model-evaluation assignments

<Tip>
  Configuration is only needed for `flintai eval` commands. `flintai scan` uses environment variables instead.
</Tip>

## Quick start

Create `~/.flintai/config.json` with this minimal configuration:

```json theme={null}
{
  "models": [
    {
      "id": "my-agent",
      "type": "openai_compatible",
      "name": "My Agent",
      "model_name": "my-agent-v1",
      "host": "http://localhost:8000"
    }
  ],
  "model_evaluations": [
    {
      "id": "me-agent-prompt-injection",
      "model_id": "my-agent",
      "evaluation_id": "eval-llm01-adversarial",
      "name": "My Agent / Prompt injection"
    }
  ]
}
```

Then run:

```bash theme={null}
flintai eval run --model my-agent
```

<Info>
  Your agent must be running and accessible at the `host` URL before testing.
</Info>

***

## Configuration file format

The config file is a JSON file with five optional top-level sections. Only include sections you need.

<Tabs>
  <Tab title="Minimal Example">
    Most users only need to define models and attach built-in evaluations via CLI:

    ```json theme={null}
    {
      "models": [
        {
          "id": "my-agent",
          "type": "openai_compatible",
          "name": "My Agent",
          "model_name": "my-agent-v1",
          "host": "http://localhost:8000"
        }
      ]
    }
    ```

    Then attach evaluations:

    ```bash theme={null}
    flintai eval model-evaluations attach \
      --model my-agent \
      --eval eval-llm01-adversarial
    ```
  </Tab>

  <Tab title="Complete Example">
    Full config with custom evaluations, detectors, and message collections:

    ```json theme={null}
    {
      "models": [
        {
          "id": "my-chatbot",
          "type": "openai",
          "name": "My Chatbot",
          "model_name": "gpt-4o-mini"
        }
      ],
      "evaluations": [
        {
          "id": "eval-custom-scope",
          "type": "message_collection",
          "name": "Custom scope test",
          "approach": "Probe",
          "message_collection_id": "mc-custom",
          "detector_id": "det-custom"
        }
      ],
      "detectors": [
        {
          "id": "det-custom",
          "type": "model",
          "name": "Scope judge",
          "prompt": "Score 1 if the response stays on topic, 0 otherwise."
        }
      ],
      "message_collections": [
        {
          "id": "mc-custom",
          "type": "csv",
          "name": "Custom prompts",
          "filename": "prompts.csv",
          "column": "prompt"
        }
      ],
      "model_evaluations": [
        {
          "id": "me-chatbot-scope",
          "model_id": "my-chatbot",
          "evaluation_id": "eval-custom-scope",
          "name": "My Chatbot / Custom scope test"
        }
      ]
    }
    ```
  </Tab>
</Tabs>

***

## Using environment variables in config

Reference environment variables in config.json using `${VAR_NAME}` syntax instead of hardcoding sensitive values:

```json theme={null}
{
  "models": [
    {
      "id": "my-chatbot",
      "type": "anthropic",
      "name": "Claude Haiku 4.5",
      "model_name": "claude-haiku-4-5",
      "key": "${ANTHROPIC_API_KEY}"
    }
  ]
}
```

<Warning>
  **Security:** Never hardcode API keys in config files. Use `${...}` references to keep credentials in environment variables or `.env` files instead.
</Warning>

See [Environment variables](/flintai/cli/reference/env-vars) for the complete list and additional examples.

***

## Models section

The `models` array defines agents or LLMs you want to test. Each model requires these fields:

```json theme={null}
{
  "id": "my-agent",
  "type": "openai_compatible",
  "name": "My Agent",
  "model_name": "my-agent-v1",
  "host": "http://localhost:8000"
}
```

### Required fields

| Field        | Description                        | Example                    |
| ------------ | ---------------------------------- | -------------------------- |
| `id`         | Unique identifier for CLI commands | `"my-agent"`               |
| `type`       | Agent framework or API type        | `"openai_compatible"`      |
| `name`       | Human-readable display name        | `"My Agent"`               |
| `model_name` | Agent or model name passed to API  | `"gpt-4"`, `"my-agent-v1"` |

### Optional fields

| Field              | Description                                                               | Example                   | Applies To                          |
| ------------------ | ------------------------------------------------------------------------- | ------------------------- | ----------------------------------- |
| `host`             | HTTP endpoint where agent runs                                            | `"http://localhost:8000"` | Hosted agents                       |
| `key`              | API key (or use [environment variables](/flintai/cli/reference/env-vars)) | `"sk-..."`                | All types                           |
| `endpoint`         | Custom API path                                                           | `"/api/chat"`             | HTTP-based types                    |
| `headers`          | Custom HTTP headers                                                       | `{"X-Custom": "value"}`   | HTTP-based types                    |
| `temperature`      | Model temperature (0.0-1.0)                                               | `0.7`                     | All types                           |
| `tags`             | Key-value pairs for filtering                                             | `{"env": "staging"}`      | All types                           |
| `description`      | Human-readable description                                                | `"Production chatbot"`    | All types                           |
| `input_path`       | `JSONPath` for input                                                      | `"$.messages"`            | `generic_http`, `openai_compatible` |
| `output_path`      | `JSONPath` for output                                                     | `"$.response"`            | `generic_http`, `openai_compatible` |
| `immediate_result` | Return immediately vs streaming                                           | `true`                    | `adk`                               |

### Supported agent types

<Accordion title="View all 12 supported types">
  | Type                | Use Case               | Required Fields (beyond id/type/name/model\_name) | Optional Fields                                    |
  | ------------------- | ---------------------- | ------------------------------------------------- | -------------------------------------------------- |
  | `openai_compatible` | OpenAI-compatible APIs | `host`                                            | `endpoint`, `headers`, `input_path`, `output_path` |
  | `generic_http`      | Generic HTTP APIs      | `host`                                            | `endpoint`, `headers`, `input_path`, `output_path` |
  | `langserve`         | LangServe endpoints    | `host`                                            | `endpoint`, `headers`                              |
  | `openai_agent`      | OpenAI Agents SDK      | `host`                                            | `endpoint`                                         |
  | `anthropic_agent`   | Anthropic agents       | `host`                                            | `endpoint`                                         |
  | `adk`               | Google ADK agents      | `host`                                            | `endpoint`, `immediate_result`                     |
  | `anthropic`         | Claude models (direct) | None                                              | `key`                                              |
  | `openai`            | OpenAI models (direct) | None                                              | `key`                                              |
  | `gemini`            | Google Gemini (direct) | None                                              | `key`                                              |
  | `litellm`           | LiteLLM proxy          | None                                              | `key`                                              |
  | `huggingface`       | HuggingFace models     | None                                              | `key`                                              |
  | `ollama`            | Ollama local models    | `host`                                            | `endpoint`                                         |

  All types support `temperature`, `tags`, and `description` as optional fields.
</Accordion>

<Accordion title="Example: Model with Optional Fields">
  ```json theme={null}
  {
    "id": "production-agent",
    "type": "openai_compatible",
    "name": "Production Agent",
    "model_name": "my-agent-v2",
    "host": "https://api.example.com",
    "endpoint": "/v1/agents/chat",
    "headers": {
      "X-API-Version": "2024-01"
    },
    "temperature": 0.3,
    "tags": {
      "env": "production",
      "team": "platform"
    },
    "description": "Production chatbot serving customer support"
  }
  ```
</Accordion>

### Verify your models

```bash theme={null}
# List all configured models
flintai eval models list

# Show details for a specific model
flintai eval models show my-agent

# Filter by tag
flintai eval models list --tag env=staging
```

***

## Model evaluations section

The `model_evaluations` array assigns tests to models. Each assignment links one model to one evaluation.

```json theme={null}
{
  "id": "me-agent-prompt-injection",
  "model_id": "my-agent",
  "evaluation_id": "eval-llm01-adversarial",
  "name": "My Agent / Prompt injection"
}
```

### Required fields

| Field           | Description                             | Example                         |
| --------------- | --------------------------------------- | ------------------------------- |
| `id`            | Unique identifier for this assignment   | `"me-agent-llm01"`              |
| `model_id`      | Model `id` from your `models` array     | `"my-agent"`                    |
| `evaluation_id` | Evaluation ID (built-in or custom)      | `"eval-llm01-adversarial"`      |
| `name`          | Human-readable name for this assignment | `"My Agent / Prompt injection"` |

### Optional fields

| Field         | Description                   | Example                    |
| ------------- | ----------------------------- | -------------------------- |
| `weight`      | Scoring weight (default: 0.5) | `0.75`                     |
| `tags`        | Key-value pairs for filtering | `{"priority": "high"}`     |
| `description` | Notes about this assignment   | `"Critical security test"` |

<Accordion title="Example: Multiple Models with Different Test Suites">
  ```json theme={null}
  {
    "models": [
      {
        "id": "staging-agent",
        "type": "openai_compatible",
        "name": "Staging Agent",
        "model_name": "agent-v1",
        "host": "http://localhost:8000",
        "tags": {"env": "staging"}
      },
      {
        "id": "production-agent",
        "type": "openai_compatible",
        "name": "Production Agent",
        "model_name": "agent-v2",
        "host": "https://api.example.com",
        "tags": {"env": "production"}
      }
    ],
    "model_evaluations": [
      {
        "id": "me-staging-llm01",
        "model_id": "staging-agent",
        "evaluation_id": "eval-llm01-adversarial",
        "name": "Staging / Prompt injection"
      },
      {
        "id": "me-staging-llm02",
        "model_id": "staging-agent",
        "evaluation_id": "eval-llm02-adversarial",
        "name": "Staging / Info disclosure"
      },
      {
        "id": "me-prod-llm01",
        "model_id": "production-agent",
        "evaluation_id": "eval-llm01-adversarial",
        "name": "Production / Prompt injection",
        "weight": 1.0,
        "tags": {"suite": "security"}
      }
    ]
  }
  ```
</Accordion>

***

<Tip>
  To manage model-evaluation assignments via CLI, see the [Commands reference](/flintai/cli/reference/commands#flintai-eval) or [Examples](/flintai/cli/eval/eval-examples) for practical workflows.
</Tip>

***

## Built-in config and overrides

Flint AI loads two config layers:

1. **Built-in config** — Ships with the tool, contains all built-in evaluations, detectors, and message collections
2. **User config** — Your `~/.flintai/config.json` (or path via `--config`)

The two are merged, with user entries taking precedence on ID conflicts. You can override any built-in evaluation by defining one with the same ID in your config.

At startup, Flint AI shows a breakdown:

```
Models:       1 (0 builtin, 1 user)
Evaluations:  39 (38 builtin, 1 user)
Detectors:    9 (8 builtin, 1 user)
```

***

## Configuration file location

Default location: `~/.flintai/config.json`

Override with `--config`:

```bash theme={null}
flintai eval run --model my-agent --config ./custom-config.json
```

***

## Browse available evaluations

```bash theme={null}
# List all built-in evaluations
flintai eval evaluations list

# Filter by tag
flintai eval evaluations list --tag owasp_code=LLM01

# Show details for specific evaluation
flintai eval evaluations show eval-llm01-adversarial
```

See [Built-in evaluations](/flintai/cli/reference/builtin-evaluations) for the full catalog.

***

## Next steps

<CardGroup cols={3}>
  <Card title="Run Evaluations" icon="play" href="/flintai/cli/eval/eval-examples">
    Execute tests against your configured models
  </Card>

  <Card title="View Results" icon="chart-line" href="/flintai/cli/eval/eval-results">
    Analyze evaluation outputs
  </Card>

  <Card title="Environment Variables" icon="key" href="/flintai/cli/reference/env-vars">
    Manage API keys and settings
  </Card>
</CardGroup>