
Member of Technical Staff, Pretraining Evaluations
Hark is an AI company focused on developing advanced multimodal models and hardware that facilitate human-machine interaction. The Omni team is tasked with building evaluation systems for AI capabilities across various modalities. This role involves developing benchmarks, maintaining evaluation tools for model performance, and collaborating with researchers to analyze training results. Ideal candidates should have experience in model evaluation systems, strong statistical skills, and proficiency in Python engineering. Successful applicants will contribute to pioneering AI experiences beyond text.
Job description
About Hark
Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persistent memory. We're pairing that intelligence with next-generation hardware to create a universal interface between humans and machines. While today's AI largely operates through chat boxes and decade-old devices, Hark is focused on what comes next: agentic systems that interact naturally with people and the real world. To get there, we're developing multimodal models and next-generation AI hardware together - designed from the ground up as a single, unified interface for a new era of intelligent systems.
About the Role
The Omni team at Hark is building the next generation of AI experiences beyond text, enabling models to understand and generate content across multiple modalities, including text, audio, and beyond. Our goal is to create seamless, real-time multimodal intelligence that powers intuitive and immersive user experiences. As part of the Omni team, you will build the multimodal foundation models at the core of this effort. You'll build the systems used to evaluate Hark's models during pretraining and mid-training. You'll track held-out loss, measure capabilities across text and multimodal tasks, and make comparisons between training runs reproducible. Your analysis will help researchers choose training recipes, select checkpoints, and identify regressions.
Responsibilities
- Build and maintain an evaluation harness for held-out loss, performance across data domains, and capabilities in text, code, math, and reasoning.
- Design evaluations for early checkpoints and smaller models using log-likelihood scoring, few-shot prompts, and generated completions.
- Assess benchmark coverage and saturation, quantify uncertainty in results, and work with data researchers to detect training-data contamination.
- Version datasets, prompts, graders, and inference settings so results can be reproduced and compared across checkpoints and training runs.
- Analyze experiments and work with researchers to identify useful recipe changes, select checkpoints, and prioritize capability improvements.
- Develop new benchmarks for capabilities that existing evaluations do not adequately measure.
- Extend evaluation tools to image, video, speech, and audio tasks in collaboration with multimodal researchers.
Requirements
- Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
- Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
- Strong statistical judgment, including uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.
- Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and the intended capability.
- Strong Python engineering skills, including building reliable evaluation pipelines that run efficiently across many checkpoints.
- A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.
Bonus Qualifications
- Experience designing benchmarks with human ratings or model-based graders.
- Experience studying capability scaling or predicting larger-model performance from smaller experiments.
- Experience with code execution sandboxes, math verifiers, or other programmatic graders.
- Experience evaluating checkpoints during pretraining or mid-training.
- Multimodal evaluation experience: audio, speech, vision, or interleaved inputs.
- Experience evaluating model quality under constrained inference budgets.
Compensation
The US base salary range for this full-time position is between $180,000 - $450,000 annually. The pay offered for this position may vary based on several individual factors, including job-related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.
Job details
- Published
- November 10, 2026
- Last checked
- November 18, 2026
- Experience
- Staff
- Job source
- greenhouse
- Workplace
- On-site · San Jose, US
- First seen
- November 11, 2026
Skills mentioned in this job
LiftmyCV exclusive data
Hark hiring intelligence
Hark currently has 15 active jobs tracked by LiftmyCV. 1 new jobs in the past 7 days.
Verified November 18, 2026
Active openings
15
Currently tracked by LiftmyCV
New in 7 days
1
9 new jobs in the past 30 days
Published salaries
80%
Median disclosed salary: $282,500
Average opening age
55 days
Across 15 active openings
Workplace breakdown
Hiring countries
Top hiring cities
Most in-demand roles
- • Business Development Manager
- • Finance Manager
- • Industrial Designer
- • Lead Brand Designer
- • Member of Technical Staff, Computer-Use Agents
- • Member of Technical Staff, Distributed Training Performance
Frequently requested skills
Median disclosed salary: $282,500 · 80% of jobs publish a salary
As of November 18, 2026
Hiring insights calculated from active jobs tracked by LiftmyCV.
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inMore at Hark
Similar jobs from this company
- Member of Technical Staff, Multimodal Pretraining DataHarkSan Jose, USOn-site$180,000–$450,000 per year
- Member of Technical Staff, Mid-trainingHarkSan Jose, USOn-site$180,000–$450,000 per year
- Member of Technical Staff, Pretraining Architecture and ScalingHarkSan Jose, USOn-site$180,000–$450,000 per year
- Technical Customer Experience LeadHarkSan Jose, USOn-site$120,000–$170,000 per year
- Member of Technical Staff, LLM & ProductionHarkSan Jose, USOn-site$180,000–$450,000 per year
- Member of Technical Staff, Distributed Training PerformanceHarkSan Jose, USOn-site$180,000–$450,000 per year
Across the market
Similar jobs from other companies
- Member of Technical Staff - PretrainingBlack Forest LabsFreiburg, DEHybrid
- Member of the Technical Staff, PretrainingOutput BiosciencesNew York, USHybrid
- Member of Technical Staff, PretrainingMirendilSan Francisco, US$300,000–$500,000 per year
- Member of Technical Staff, PretrainingHarkSan Jose, USOn-site
- Member of Technical Staff, Pretraining ScienceRadical NumericsSan Francisco, USHybrid
- Member of Technical Staff, Pretraining DataMirendilSan Francisco, US$300,000–$400,000 per year
Related job searches
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.