
Member of Technical Staff, Evals
Join Profound, a rapidly growing AI marketing platform, as an Evals Engineer. You'll be responsible for building and maintaining evaluation systems that ensure AI accuracy. Collaborate with marketing teams and leverage real user data to develop insightful metrics and benchmarks. Your role will involve designing scoring systems, analyzing AI performance, and implementing experimental methodologies. Ideal candidates have over 3 years of experience in evaluation systems, strong Python skills, and an understanding of large language models. This role is suited for those who thrive in fast-paced environments.
Job description
Profound is the fastest-growing AI marketing platform, used by 15% of the Fortune 500 including Comcast, Estée Lauder, and Walmart, and innovators like Ramp, MongoDB, and Figma. It is where marketing teams do the work: research, briefs, content, campaigns, reporting, all in one place, alongside agents that carry the load with them. Built on 2B+ real user prompts and growing, Profound’s data shows you what people ask AI and what the models say back. Profound's AI Marketer joins that data with deep context about our customers to find the opportunities worth acting on, and then puts agents to work: automating reporting, drafting content, auditing pages for readability, running competitive research. It measures what works and gets better every day. As an Evals Engineer, you will own how Profound knows its AI is right. At Profound, measurement is the product: when a Fortune 500 CMO sees their brand’s visibility drop in ChatGPT, they act on that number. No public benchmark measures whether a model attributed a citation correctly or ran a competitive analysis well, so we build our own. You’ll decide what gets measured, why, and how results shape what we ship. Every product runs through the evals you design, so your impact compounds across all of them.
What you’ll do
- Build Profound’s eval system: golden datasets, scorers and LLM judges, programmatic verifiers, regression alerts, and dashboards
- Write evals for every pipeline behind the product: brand extraction, visibility scoring, sentiment, FactCheck, and citation classification
- Define what a great, mediocre, and broken AI Marketer run looks like, and measure it across multi-step tool use, analytical correctness, and content quality
- Study how answer engines behave across runs, platforms, regions, and personas, and design the sampling and statistics that keep customer-facing numbers trustworthy
- Close the loop from real usage: collect customer signals, cluster failure modes, and turn regressions into permanent guardrails
- Build tooling for debugging model and agent behavior, plus the labeling workflows and annotation guidelines behind our ground truth
- Make evals part of how we decide things: wired into CI, launch reviews, and model selection
Who you are
- 3+ years building evaluation or measurement systems, whether AI evals, experimentation, ranking, or search quality
- Strong Python and data skills, with a track record of shipping production systems
- Deep familiarity with LLM evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and significance testing
- Sound experimental design, and the instinct to separate signal from noise in non-deterministic systems
- Strong opinions about how models and agents behave, and you keep up with current research
- You believe a good eval is clear, reliable, and hard to game
- Work that shows how you think: shipped benchmarks, datasets, open-source tools, or publications
- Comfortable operating in a fast-paced startup environment with rapid iteration cycles
Nice to have (optional):
- Experience with search, retrieval, citation, or fact-verification systems
- Publications at venues like NeurIPS, ICML, ICLR, ACL, or COLM
Location
This is an on-site role in our NYC or SF office, designed for builders who thrive on speed, iteration, and meaningful impact. For this role, the expected base salary range is $215,000 to $300,000. Profound’s total compensation package is designed to be competitive and includes base salary, equity, and a full range of benefits and perks. Final compensation will depend on factors such as your skills, experience, qualifications, and location, and will be determined during the interview process. Our recruiting team will share more details about the full compensation package and benefits as you move through hiring. #LI-PRO Note: All official communication from Profound will come from a @tryprofound.com email address. If you're contacted by anyone using a different domain, please disregard and report it as spam.
Job details
- Published
- October 30, 2026
- Last checked
- November 2, 2026
- Experience
- Staff
- Job source
- ashby
- First seen
- October 30, 2026
Skills mentioned in this job
LiftmyCV exclusive data
Profound hiring intelligence
Profound currently has 18 active jobs tracked by LiftmyCV. 6 new jobs in the past 7 days.
Verified November 2, 2026
Active openings
18
Currently tracked by LiftmyCV
New in 7 days
6
15 new jobs in the past 30 days
Published salaries
56%
Median disclosed salary: $196,250
Average opening age
16 days
Across 18 active openings
Workplace breakdown
Hiring countries
Top hiring cities
Most in-demand roles
- • Compensation Manager
- • Corporate Development Lead
- • Data Acquisition Lead
- • G&A Recruiter
- • GTM, Compensation Manager
- • Head of Design
Frequently requested skills
Median disclosed salary: $196,250 · 56% of jobs publish a salary
As of November 17, 2026
Hiring insights calculated from active jobs tracked by LiftmyCV.
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inMore at Profound
Similar jobs from this company
- Member of Technical Staff, Internal ToolingProfoundNew York, US
- Member of Technical Staff, TLM, Internal ToolingProfoundNew York, US
- Member of Technical Staff, Program ManagerProfoundNew York, US
- Member of Technical Staff, Engineering ManagerProfoundNew York, US
- Member of Technical Staff, Post TrainingProfoundNew York, US$230,000–$325,000 per year
Across the market
Similar jobs from other companies
Related job searches
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.