Member of Technical Staff - Evals
P-1 AI is pioneering engineering AGI with its product, Archie, an AI engineer designed to learn and adapt to engineering tasks. As a Member of Technical Staff - Evals, you will oversee the evaluation processes that ensure Archie meets industry standards, collaborating with engineering experts and industrial partners. This remote role, based in the US or Canada, offers opportunities for in-person collaboration approximately once every six weeks. Ideal candidates will possess experience in software testing, metric design, Python programming, and a passion for rapid innovation in a startup environment.
Job description
About P-1 AI:
We are building an engineering AGI. We founded P-1 AI with the conviction that the greatest impact of artificial intelligence will be on the built world. Our first product is Archie, an AI engineer capable of quantitative intuition over physical product domains and engineering tool use. Archie initially performs at the level of an entry-level design engineer but rapidly gets smarter and more capable. We aim to put an Archie on every engineering team at every industrial company on earth. Our founding team includes the top minds in deep learning, model-based engineering, and industries that are our customers. We closed a $23 million seed round led by Radical Ventures that includes a number of other AI and industrial luminaries (from OpenAI, DeepMind, etc.).
About the Role:
In this role, you’ll be responsible for the evals that we use to ensure that Archie is learning and retaining the skills needed to successfully perform its engineering work, and to benchmark it against industry skill expectations. Working within a small, tightly-knit team of high-performers, you’ll be principally responsible for clearly defining, implementing, and validating these, including input from our engineering experts and industrial partners. You’ll also be responsible for translating these eval tests into multiple formats for use with different types of AI and non-AI systems and agents. This role is remote and you can be based anywhere in the US or Canada, where you must have existing work authorization. You will be expected to travel to our San Mateo office for co-working sessions approximately one week out of every six. If you are already located in the Bay Area or are interested in relocation, you are of course welcome to work out of our San Mateo office. Our AI team is based in the San Mateo office, so there would be some benefit to you being in-office at least part of the time.
What you’ll do:
- Implement and operate the system for organizing, transforming, running, grading, and reporting on eval benchmarks.
- Design and execute the process by which we develop and QA our evals, incorporating contributions from our own engineering team, industrial partners, and subject-matter experts.
- Ensure that evals run effectively within our CI/CD system, continuously benchmarking our evolving AI platform and the experiments we’re performing around it.
- Create methods for detecting and testing for common quality challenges of AI, including hallucinations, undesirable stochasticity, and regressions.
- Be a technical leader in the consistent implementation and organization of automated tests across other areas of our technology stacks.
Who you are:
- Experience in constructing comprehensive test suites for software and/or AI systems, including coordinating the contributions of others.
- Experience designing metrics to evaluate systems and visualize their performance, including differences across successive generations.
- Experience in developing, managing, and running evals against LLM-based systems is a strong plus.
- Good communication skills with a variety of stakeholders (AI researchers, domain experts, application developers).
- Proficiency in Python programming, complex modules and modern software development tools and practices (Git, CI/CD, etc.).
- Ability to thrive in a fast-paced, dynamic startup environment.
Our Values:
- Mission obsession & urgency: We are obsessed with building engineering AGI as quickly as possible. We also recognize that as a startup, speed is our most precious competitive advantage. We are constantly asking ourselves what we can do to go faster. We make tradeoffs and sacrifices (personally and in the workplace) in exchange for speed.
- Intellectual excellence & curiosity: We ask “what if?” and experiment liberally. We always look for better ways of doing something. We read voraciously. We challenge each other to be better. We surround ourselves with A players and we actively and unapologetically reject B players (and even B+ players⸺because they tend to surround themselves with C players).
- Shipping discipline: We treat production with respect. We test and demo our product constantly. We listen attentively to our customers, users, and stakeholders, and we respect our commitments to them. We also respect our commitments to each other and will go the extra mile (or ten or one hundred) to honor them.
- Ownership: We all have significant ownership stakes in the company and operate in founder mode. We believe in hierarchical requirements but not in hierarchical information flows. If we see that something is broken or can be done better, we flag it and we fix it. We encourage each other to play with and fix anything and everything... but there’s a clear owner for everything.
Interview Process:
- Initial screening call (30 mins)
- Biographical/behavioural interview (45 mins)
- Technical interview (75 mins)
- CEO interview (30 mins)
Compensation:
Salary: $170k - $200k. This role includes a significant equity component. We are an early-stage startup, so we favor equity over cash in our current compensation philosophy. This role is best suited for candidates who value long-term ownership and impact over short-term cash optimization. Our benefits include healthcare, dental, and vision insurance, 401k with employer matching, unlimited PTO.
Job details
- Published
- May 11, 2026
- Job source
- ashby
- Workplace
- Remote · US
- First seen
- May 11, 2026
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inMore at P-1 AI
Similar jobs from this company
None listed.
Across the market
Similar jobs from other companies
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.