Braintrust

About

Braintrust is an evaluation and observability platform for teams shipping AI features — the tooling that catches a confident hallucination or a yes-man answer before it ever reaches a user. The thesis is blunt: large language models agree with you and make things up, and you can't see it until production unless you measure it. A Stanford-led study in Science found chatbots affirm users 49% more than humans do, and endorse 47% of explicitly harmful asks. Braintrust is how engineering teams put a number on that risk and hold the line. It does two jobs in one place. The eval side lets you run experiments against a dataset, compare prompts and models head to head, and score every output with LLM judges, code, or human review. The observability side gives you real-time trace inspection of prompts, responses and tool calls in production, plus a Topics feature that surfaces recurring failure patterns and turns them into new evals. Tie them together with online scoring and quality gates and you get a CI check that blocks a release when answer quality regresses — instead of finding out from an angry customer. The SDKs are framework-agnostic (Python, TypeScript, Go, Ruby, C# and more), and the customer list — Vercel, Notion, Coursera, Dropbox, Replit — signals this is the platform serious teams ship behind. There's a real free Starter tier: 1M trace spans, 10K eval scores, unlimited users, no credit card. Pro is $249/month and Enterprise is custom; the full platform is free for academic and non-commercial open-source work. Verdict: if you ship anything LLM-powered to real users, Braintrust is the difference between hoping your model is accurate and proving it. It won't make a sycophantic model honest — but it will catch the moment it isn't. Read our breakdown of why AI chatbots act like yes-men for the research behind the risk, and pair Braintrust with an install-time checker like the Fact-Check Skill to verify claims before your model agrees with them.