An apiKey pasted under a provider in promptfooconfig.yaml ends up in git and in every saved eval. A test glob such as file://tests/*.yml, when the files end in .yaml, adds no tests at all, and a suite whose tests have no assert passes in CI whatever the model answers. This skill finds those and the other promptfoo config mistakes below before you commit or run the eval, and tells you how to fix each one.

Your agent runs a scanner on promptfooconfig.yaml, .yml and .json files, other YAML or JSON files that are promptfoo configs by their content (such as a generated redteam.yaml), and the test, scenario, provider and defaultTest files a config loads with file://, globs included. It resolves file:// paths from the config's folder as promptfoo does, checks their exact letter case, reads only the header of CSV test files for __expected columns, and treats {{ env.NAME }} templates, apiKeyEnvar names, docs placeholders and keys for local servers as not secret. It never runs promptfoo, a provider, a script or an eval, and never contacts a model API. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach. Secrets are never printed.

What it checks

BLOCK (fix before shipping)

• P01: API keys, tokens, passwords and private keys written into a config or a file it loads: provider apiKey and other secret fields, env: entries, credential headers, auth blocks, TLS or signing keys, passwords in URLs, and values shaped like real provider keys anywhere, prompts and test vars included
• P02: file:// references to prompts, providers, test files or globs, var files, assertion scripts or transforms that do not exist, or exist only with different letter case
• P03: providers that send credentials to a remote host over plain http:// or ws://

REVIEW (check before release)

• P11: evals with no assertions at all, and tests without assertions in a list where the others have them
• P12: checks that cannot fail: threshold 0 on minimum-score assertions or assert-sets, a test threshold of 0, weight 0 on every assertion, empty contains values, match-everything regexes, javascript or python checks that just return true
• P13: model-graded assertions (llm-rubric, factuality, g-eval, similar and others) with no pinned grader, and similar checks whose grader has no embedding model
• P14: result sharing turned on in the config (sharing: true, a sharing block, commandLineOptions.share), with the fix that actually stops uploads: sharing: false
• P15: provider, assertion, var or extension code loaded from an absolute path or from outside the project, and exec: providers that run a program by absolute path
• P16: secret-named test vars with literal values, email addresses, payment card numbers and US Social Security numbers in vars, and provider keys copied into env with {{ env.NAME }}
• P17: tests with the same description, or the same vars and assertions, in one list
• P18: misspelled assertion types (contain, llm_rubric, json) that promptfoo rejects as unknown

What's included

• The scanner (Python 3.9+, standard library only)
• A rules guide with the reason, safer approach and official promptfoo source for each rule
• A risky and a safe example eval (config, test file and prompt), and the real report the scanner produces on the risky one

Limits

It reads YAML and JSON configs only: promptfooconfig.js and .ts files, command-line flags and values filled in from the environment at run time are not known. Files that a setup script downloads or a hook writes before the eval are reported as missing until they exist. Personal data detection covers email addresses, card numbers and US Social Security numbers only. Official examples without assertions or with the default grader get P11 or P13 by design. NOT FLAGGED is not a guarantee.

Contents

One ZIP file with 17 files: SKILL.md, the scanner in scripts/ (7 Python files), references/rules.md (every rule with its reason and safer approach), and examples/ (7 example files plus report.md, the real scanner output on the risky example).

How to use: This is an agent skill: a folder with a SKILL.md file and Python scripts that an AI coding agent such as Claude Code, Cursor, or Codex CLI can run. Unzip it into your agent's skills folder (for Claude Code: ~/.claude/skills/), then ask your agent to check your project. The scripts also run on their own with Python 3.9 or newer, with nothing to install. They read your files only: no network access, and nothing is sent anywhere.