Configuration

Flint AI Eval tests agent behavior and reliability at runtime. Configuration lives in ~/.flintai/config.json and defines:

What to test - Your running agent’s HTTP endpoint
How to test it - Which evaluations to run
When to test - Model-evaluation assignments

Configuration is only needed for flintai eval commands. flintai scan uses environment variables instead.

Quick start

Create ~/.flintai/config.json with this minimal configuration:

{
  "models": [
    {
      "id": "my-agent",
      "type": "openai_compatible",
      "name": "My Agent",
      "model_name": "my-agent-v1",
      "host": "http://localhost:8000"
    }
  ],
  "model_evaluations": [
    {
      "id": "me-agent-prompt-injection",
      "model_id": "my-agent",
      "evaluation_id": "eval-llm01-adversarial",
      "name": "My Agent / Prompt injection"
    }
  ]
}

Then run:

flintai eval run --model my-agent

Your agent must be running and accessible at the host URL before testing.

Configuration file format

The config file is a JSON file with five optional top-level sections. Only include sections you need.

Minimal Example
Complete Example

Most users only need to define models and attach built-in evaluations via CLI:

{
  "models": [
    {
      "id": "my-agent",
      "type": "openai_compatible",
      "name": "My Agent",
      "model_name": "my-agent-v1",
      "host": "http://localhost:8000"
    }
  ]
}

Then attach evaluations:

flintai eval model-evaluations attach \
  --model my-agent \
  --eval eval-llm01-adversarial

Full config with custom evaluations, detectors, and message collections:

{
  "models": [
    {
      "id": "my-chatbot",
      "type": "openai",
      "name": "My Chatbot",
      "model_name": "gpt-4o-mini"
    }
  ],
  "evaluations": [
    {
      "id": "eval-custom-scope",
      "type": "message_collection",
      "name": "Custom scope test",
      "approach": "Probe",
      "message_collection_id": "mc-custom",
      "detector_id": "det-custom"
    }
  ],
  "detectors": [
    {
      "id": "det-custom",
      "type": "model",
      "name": "Scope judge",
      "prompt": "Score 1 if the response stays on topic, 0 otherwise."
    }
  ],
  "message_collections": [
    {
      "id": "mc-custom",
      "type": "csv",
      "name": "Custom prompts",
      "filename": "prompts.csv",
      "column": "prompt"
    }
  ],
  "model_evaluations": [
    {
      "id": "me-chatbot-scope",
      "model_id": "my-chatbot",
      "evaluation_id": "eval-custom-scope",
      "name": "My Chatbot / Custom scope test"
    }
  ]
}

Using environment variables in config

Reference environment variables in config.json using ${VAR_NAME} syntax instead of hardcoding sensitive values:

{
  "models": [
    {
      "id": "my-chatbot",
      "type": "anthropic",
      "name": "Claude Haiku 4.5",
      "model_name": "claude-haiku-4-5",
      "key": "${ANTHROPIC_API_KEY}"
    }
  ]
}

Security: Never hardcode API keys in config files. Use ${...} references to keep credentials in environment variables or .env files instead.

See Environment variables for the complete list and additional examples.

Models section

The models array defines agents or LLMs you want to test. Each model requires these fields:

{
  "id": "my-agent",
  "type": "openai_compatible",
  "name": "My Agent",
  "model_name": "my-agent-v1",
  "host": "http://localhost:8000"
}

Required fields

Field	Description	Example
`id`	Unique identifier for CLI commands	`"my-agent"`
`type`	Agent framework or API type	`"openai_compatible"`
`name`	Human-readable display name	`"My Agent"`
`model_name`	Agent or model name passed to API	`"gpt-4"`, `"my-agent-v1"`

Optional fields

Field	Description	Example	Applies To
`host`	HTTP endpoint where agent runs	`"http://localhost:8000"`	Hosted agents
`key`	API key (or use environment variables)	`"sk-..."`	All types
`endpoint`	Custom API path	`"/api/chat"`	HTTP-based types
`headers`	Custom HTTP headers	`{"X-Custom": "value"}`	HTTP-based types
`temperature`	Model temperature (0.0-1.0)	`0.7`	All types
`tags`	Key-value pairs for filtering	`{"env": "staging"}`	All types
`description`	Human-readable description	`"Production chatbot"`	All types
`input_path`	`JSONPath` for input	`"$.messages"`	`generic_http`, `openai_compatible`
`output_path`	`JSONPath` for output	`"$.response"`	`generic_http`, `openai_compatible`
`immediate_result`	Return immediately vs streaming	`true`	`adk`

Supported agent types

View all 12 supported types

Type	Use Case	Required Fields (beyond id/type/name/model_name)	Optional Fields
`openai_compatible`	OpenAI-compatible APIs	`host`	`endpoint`, `headers`, `input_path`, `output_path`
`generic_http`	Generic HTTP APIs	`host`	`endpoint`, `headers`, `input_path`, `output_path`
`langserve`	LangServe endpoints	`host`	`endpoint`, `headers`
`openai_agent`	OpenAI Agents SDK	`host`	`endpoint`
`anthropic_agent`	Anthropic agents	`host`	`endpoint`
`adk`	Google ADK agents	`host`	`endpoint`, `immediate_result`
`anthropic`	Claude models (direct)	None	`key`
`openai`	OpenAI models (direct)	None	`key`
`gemini`	Google Gemini (direct)	None	`key`
`litellm`	LiteLLM proxy	None	`key`
`huggingface`	HuggingFace models	None	`key`
`ollama`	Ollama local models	`host`	`endpoint`

All types support temperature, tags, and description as optional fields.

Example: Model with Optional Fields

{
  "id": "production-agent",
  "type": "openai_compatible",
  "name": "Production Agent",
  "model_name": "my-agent-v2",
  "host": "https://api.example.com",
  "endpoint": "/v1/agents/chat",
  "headers": {
    "X-API-Version": "2024-01"
  },
  "temperature": 0.3,
  "tags": {
    "env": "production",
    "team": "platform"
  },
  "description": "Production chatbot serving customer support"
}

Verify your models

# List all configured models
flintai eval models list

# Show details for a specific model
flintai eval models show my-agent

# Filter by tag
flintai eval models list --tag env=staging

Model evaluations section

The model_evaluations array assigns tests to models. Each assignment links one model to one evaluation.

{
  "id": "me-agent-prompt-injection",
  "model_id": "my-agent",
  "evaluation_id": "eval-llm01-adversarial",
  "name": "My Agent / Prompt injection"
}

Required fields

Field	Description	Example
`id`	Unique identifier for this assignment	`"me-agent-llm01"`
`model_id`	Model `id` from your `models` array	`"my-agent"`
`evaluation_id`	Evaluation ID (built-in or custom)	`"eval-llm01-adversarial"`
`name`	Human-readable name for this assignment	`"My Agent / Prompt injection"`

Optional fields

Field	Description	Example
`weight`	Scoring weight (default: 0.5)	`0.75`
`tags`	Key-value pairs for filtering	`{"priority": "high"}`
`description`	Notes about this assignment	`"Critical security test"`

Example: Multiple Models with Different Test Suites

{
  "models": [
    {
      "id": "staging-agent",
      "type": "openai_compatible",
      "name": "Staging Agent",
      "model_name": "agent-v1",
      "host": "http://localhost:8000",
      "tags": {"env": "staging"}
    },
    {
      "id": "production-agent",
      "type": "openai_compatible",
      "name": "Production Agent",
      "model_name": "agent-v2",
      "host": "https://api.example.com",
      "tags": {"env": "production"}
    }
  ],
  "model_evaluations": [
    {
      "id": "me-staging-llm01",
      "model_id": "staging-agent",
      "evaluation_id": "eval-llm01-adversarial",
      "name": "Staging / Prompt injection"
    },
    {
      "id": "me-staging-llm02",
      "model_id": "staging-agent",
      "evaluation_id": "eval-llm02-adversarial",
      "name": "Staging / Info disclosure"
    },
    {
      "id": "me-prod-llm01",
      "model_id": "production-agent",
      "evaluation_id": "eval-llm01-adversarial",
      "name": "Production / Prompt injection",
      "weight": 1.0,
      "tags": {"suite": "security"}
    }
  ]
}

To manage model-evaluation assignments via CLI, see the Commands reference or Examples for practical workflows.

Built-in config and overrides

Flint AI loads two config layers:

Built-in config — Ships with the tool, contains all built-in evaluations, detectors, and message collections
User config — Your ~/.flintai/config.json (or path via --config)

The two are merged, with user entries taking precedence on ID conflicts. You can override any built-in evaluation by defining one with the same ID in your config. At startup, Flint AI shows a breakdown:

Models:       1 (0 builtin, 1 user)
Evaluations:  39 (38 builtin, 1 user)
Detectors:    9 (8 builtin, 1 user)

Configuration file location

Default location: ~/.flintai/config.json Override with --config:

flintai eval run --model my-agent --config ./custom-config.json

Browse available evaluations

# List all built-in evaluations
flintai eval evaluations list

# Filter by tag
flintai eval evaluations list --tag owasp_code=LLM01

# Show details for specific evaluation
flintai eval evaluations show eval-llm01-adversarial

See Built-in evaluations for the full catalog.

Next steps

Run Evaluations

Execute tests against your configured models

View Results

Analyze evaluation outputs

Environment Variables

Manage API keys and settings

Overview

Flint AI Scan

Flint AI Eval

Guides

Reference

Troubleshooting

Resources

Quick start

Configuration file format

Using environment variables in config

Models section

Required fields

Optional fields

Supported agent types

Verify your models

Model evaluations section

Required fields

Optional fields

Built-in config and overrides

Configuration file location

Browse available evaluations

Next steps

Run Evaluations

View Results

Environment Variables

​Quick start

​Configuration file format

​Using environment variables in config

​Models section

​Required fields

​Optional fields

​Supported agent types

​Verify your models

​Model evaluations section

​Required fields

​Optional fields

​Built-in config and overrides

​Configuration file location

​Browse available evaluations

​Next steps

Run Evaluations

View Results

Environment Variables

Quick start

Configuration file format

Using environment variables in config

Models section

Required fields

Optional fields

Supported agent types

Verify your models

Model evaluations section

Required fields

Optional fields

Built-in config and overrides

Configuration file location

Browse available evaluations

Next steps