promptfoo
Test and benchmark LLM apps, prompt quality, security, and hallucinations in CI/CD.
Why consider promptfoo?
promptfoo is an open-source CLI and evaluation engine written in TypeScript by Ian Webster. promptfoo runs automated assertions against LLM outputs (evaluating grading metrics, jailbreak vulnerabilities, PII leakage, and latency) directly within GitHub Actions and GitLab CI.
Learn promptfoo by building
Practical setup notes, real use cases, and copy-ready examples in one focused guide.
In this guide2 sections
Overview of promptfoo
promptfoo runs continuous regression tests and security red-teaming scans on LLM prompt templates.
Quickstart
npx promptfoo@latest initnpx promptfoo@latest evalpromptfoo is licensed under the permissive MIT License.
Related tools
More options with a similar category or technology profile.
diskus
Minimal, fast alternative to du -sh written in Rust using multi-threaded directory traversal.
peco
Simplistic interactive filtering tool for Unix pipelines, process lists, and file trees.
Dapr CLI
Command-line tool for managing Dapr distributed application runtime environments and sidecars.
Freeze
Generate beautiful image screenshots and SVGs of code snippets and terminal outputs.