Topic
LLM Evaluation Frameworks
15 posts on llm evaluation frameworks — part of developer tools on the n4n AI blog.
Writing custom evaluators with OpenAI Evals
Learn to build custom evaluators OpenAI Evals with hands-on code, from dataset setup to running model-graded checks via the CLI. Step-by-step tutorial for engineers shipping LLM evaluations.
What is promptfoo? A guide to prompt regression testing
Promptfoo is an open-source LLM testing framework that adds regression tests for prompts and models. Learn how it works, why it matters, and see examples.
Six open-source LLM evaluation frameworks compared
A practitioner's comparison of six open-source LLM evaluation frameworks—Promptfoo, DeepEval, Ragas, TruLens, Phoenix, and Evalite—for shipping LLM apps.
Running LLM evals on every pull request
A practical guide to implementing continuous evaluation LLM evals pull request pipelines that run automated regression tests on model outputs in CI.
Regression testing prompts before every model deployment
Learn how to implement prompt regression testing before deployment with a CI gate, golden datasets, and resilient model calls to catch quality regressions.
Ragas vs DeepEval for evaluating RAG pipelines
A practical head-to-head comparison of Ragas vs DeepEval for RAG evaluation covering metrics, cost, latency, ergonomics, ecosystem, and limits for engineers.
Ragas metrics explained: relevance and context precision
Ragas metrics for RAG quantify retrieval and generation quality without labeled data. Learn context precision and answer relevancy with code and pitfalls.
Promptfoo vs DeepEval: comparing LLM evaluation frameworks
A pragmatic head-to-head of Promptfoo vs DeepEval across capabilities, cost, latency, ergonomics, and ecosystem for your specific use case.
Golden datasets for LLM evals: how to build one
A practical guide to building golden datasets for LLM evals: sourcing examples, labeling, versioning, and avoiding common pitfalls in eval design.
Getting started with OpenAI Evals for custom test suites
Step-by-step tutorial for getting started with OpenAI Evals: clone the repo, define a custom test suite, run evaluations, and target any OpenAI-compatible gateway.
Evaluating tool-calling accuracy with promptfoo test cases
Step-by-step guide to evaluating tool-calling accuracy promptfoo test cases: define schemas, assert on function calls, and wire checks into CI.
DeepEval metrics explained: G-Eval, faithfulness, and bias
DeepEval metrics explained: a practitioner's breakdown of G-Eval, faithfulness, and bias—how they work, why they matter, and how to use them in code.
Choosing between rule-based and model-graded evals
A practical engineering guide to choosing between rule-based vs model-graded evals for LLM systems, with code, tradeoffs, and a hybrid pipeline.
Building a CI pipeline for LLM evals with promptfoo
Step-by-step guide to building a CI pipeline for LLM evals promptfoo, from config to GitHub Actions, with runnable examples and verification.
Benchmarking GPT-4o and Claude 4.5 with the same eval suite
Hands-on tutorial: build a minimal Python eval harness for benchmarking GPT-4o and Claude 4.5 through one OpenAI-compatible API, with scoring.
More topics in developer tools
- CI/CD Pipelines for LLM Apps15
- Cost Optimization & Model Routing14
- Debugging Hallucinations & Output Quality14
- LangChain Debugging & Observability14
- LLM Observability Platforms14
- Migrating Between LLM Providers14
- Model Deprecation & Version Migration14
- RAG Pipeline Observability14
- Regression Testing for Prompts14
- Compliance & Audit Logging for Regulated Industries13
- CrewAI & AutoGen Multi-Agent Debugging13
- Guardrails & Content Moderation Testing13