LiftmyCVLiftmyCV
BR

Eval Engineer

BraintrustOn-siteSan Francisco, US
AI job summary

Braintrust is looking for an Eval Engineer to creatively evaluate new AI capabilities. This position will involve designing experiments to test AI ideas, turning those ideas into measurable benchmarks, and publishing results. The engineer will work at the intersection of engineering and experimentation, focusing on model comparisons, dataset creation, and understanding AI behavior. Candidates should have experience with LLM or agent evaluation systems, and be passionate about AI methodology and reproducibility.

Job description

About the company

Braintrust is the AI observability platform. By connecting evals and observability in one workflow, Braintrust gives builders the visibility to understand how AI behaves in production and the tools to improve it. Teams at Notion, Stripe, Zapier, Vercel, and Ramp use Braintrust to compare models, test prompts, and catch regressions — turning production data into better AI with every release.

About the role

We’re hiring an Eval Engineer to design and run creative evaluations of new AI capabilities. Your job is to turn emerging AI ideas into measurable experiments and publish the results for the developer ecosystem. When new models, agents, or frameworks appear, everyone has opinions about what works but few people actually test them. This role exists to change that. You’ll design experiments that compare models, prompts, and agent architectures against real tasks. You’ll build the datasets, scoring logic, and evaluation harnesses. Then you’ll publish the results so builders understand what actually works. This role sits at the intersection of engineering, experimentation, and technical storytelling.

What you’ll own

Industry evals

  • Design and run evaluations of new AI capabilities
  • Compare frontier models, agent systems, and tool workflows
  • Turn emerging ideas into measurable benchmarks

Eval design

  • Define datasets, tasks, and scoring logic for experiments
  • Design realistic workloads that reflect production environments
  • Create tests that expose failure modes and edge cases

Experiment implementation

  • Build evaluation harnesses using Braintrust
  • Run comparisons across models, prompts, and agent approaches
  • Analyze traces, outputs, and failure patterns

Creative test construction

  • Invent novel ways to stress test AI systems
  • Design scenarios that break agents, prompts, and model reasoning
  • Build adversarial or complex datasets that reveal weaknesses

Technical content

  • Write technical posts explaining evaluation methodology and results
  • Share datasets and scoring logic so experiments are reproducible
  • Help establish better evaluation patterns for the industry via courses

Evaluation playbooks

  • Develop reusable eval patterns for agents, RAG systems, and LLM apps
  • Create open source reference implementations developers can adopt
  • Contribute examples and guides that help teams build better evals

What great looks like

  • You’re an engineer who likes testing systems more than building features
  • You enjoy breaking things and understanding why they fail
  • You can design experiments that isolate meaningful differences between approaches
  • You understand how LLMs, agents, and RAG systems actually work
  • You write clearly for technical audiences
  • You ship experiments quickly and iterate often
  • You care about methodology and reproducibility
  • You’re curious, creative, and opinionated about how AI should be evaluated

What you’ve done

  • Built or contributed to evaluation systems for LLM or agent applications
  • Designed experiments comparing models, prompts, or AI architectures
  • Written Python code to run tests across models or APIs
  • Built datasets or scoring logic for AI quality measurement
  • Investigated model failures or unexpected behaviors
  • Published technical blog posts, research notes, or engineering write-ups
  • Built prototypes quickly to test ideas

If you want to help the industry understand how to measure AI systems and design the evaluations everyone else learns from, this is the role.

Benefits include

  • Medical, dental, and vision insurance
  • Daily lunch, snacks, and beverages
  • Flexible time off
  • Competitive salary and equity
  • AI Stipend

Equal opportunity

Braintrust is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.

Job details

Published
March 13, 2026
Job source
ashby
Workplace
On-site · San Francisco, US
First seen
March 14, 2026
AI match explained

See how well you match this job

Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.

Sign up to see your matchAlready have an account? Log in

More at Braintrust

Similar jobs from this company

None listed.

Across the market

Similar jobs from other companies

Third-party content disclaimer

Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.

LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.

LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.

If you prefer, you may apply directly via the original listing for transparency.