Skip to main content
The model is highly susceptible to misuse, generating harmful or malicious content when prompted intentionally or accidentally.

​
How to resolve

Misuse resistance is critical for publicly accessible models and developer-facing APIs, where users may intentionally probe the system to generate malware, phishing templates, or instructions for illegal acts. Because this score indicates a failure to reliably refuse dangerous requests or unsafe outputs, the following suggested application-level safeguards can help block malicious use cases:
  • Implement intent-based guardrails: Use specialized safety filters (for example, Llama Guard, OpenAI Moderation, Vertex AI Safety Filters) to scan inputs for malicious intent, exploit generation, social engineering patterns, or weaponization guidance. Block or escalate high-risk content before it is returned to the user.
  • Strengthen refusal logic: Embed explicit refusal rules into your system prompt for high-risk domains (for example, credential harvesting, malware development, evasion techniques). Ensure these rules cannot be overridden by jailbreak-style prompts.
  • Restrict high-risk capabilities: Limit or disable the model’s ability to generate specific technical outputs, such as shell scripts, executable code, or cryptographic implementations, unless they’re strictly necessary for your application’s core function.
  • Enforce structured output & sandbox execution: Validate all generated outputs against strict schemas to ensure they do not contain hidden malicious payloads. If your application executes model-generated code, do so only in a network-isolated, ephemeral sandbox.

​
Risk

Governance/Compliance Low misuse resistance violates core expectations in:
  • OWASP LLM Top-10 (LLM05: Insecure Output Handling / LLM09 Misinformation): requires strong safeguards against generating malicious or dangerous outputs.
  • NIST AI RMF (MEAS-2): Expects documented controls that prevent abuse and harmful use of AI systems.
  • EU AI Act (Art. 15): Requires high-risk systems to prevent harmful behavior and ensure safe output under all intended uses.
A weak misuse score means the model cannot consistently refuse dangerous requests, failing baseline safety and abuse-prevention requirements. Security A low misuse score means the model may generate harmful or malicious content when probed—such as malware code, exploit templates, or automated attack assistance. These failures show the model does not reliably refuse dangerous requests, creating real operational risk if misused intentionally or accidentally.

​
Explanation

Misuse evaluation measures the model’s resilience when intentionally prompted for harmful output. Low performance suggests the model does not reliably refuse malicious requests.

​
Specifications

Trigger: evaluation.misuse_score < threshold. Severity:
  • CRITICAL: score < 0.2
  • HIGH: 0.2–0.35
  • MEDIUM: 0.35–0.5
  • LOW: 0.5–0.75
Applies to: Models