Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote
Job description
Role: AI Engineer – Agent Evaluation Specialist Location: Los Angeles, CA (Hybrid) Duration: 3+ Months (Possible Extension) Primary Focus: Hands-on Agent Evaluation
Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation
We are looking for an AI Engineer with strong, practical experience in evaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.
Key Requirements (Must-Have):
- LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
- Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
- Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
- Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Critical Screening Criteria:
Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development. Agent evaluation is the core requirement and will be the primary selection criterion. All your information will be kept confidential according to EEO guidelines.
Job details
- Published
- November 5, 2026
- Last checked
- November 16, 2026
- Job source
- smartrecruiters
- Workplace
- Hybrid · Los Angeles, US
- First seen
- November 5, 2026
LiftmyCV exclusive data
TMS LLC hiring intelligence
TMS LLC currently has 9 active jobs tracked by LiftmyCV. 2 new jobs in the past 7 days.
Verified November 16, 2026
Active openings
9
Currently tracked by LiftmyCV
New in 7 days
2
7 new jobs in the past 30 days
Published salaries
—
Not published by this employer yet
Average opening age
15 days
Across 9 active openings
Workplace breakdown
Hiring countries
Top hiring cities
Most in-demand roles
- • RPA Architect
- • Cisco Network Architect
- • Hiring
- • Hiring: Principal IAM Architect
- • Java AS400 Engineer - Remote
- • Senior Business Analyst- Commercial P&C/ MGA
Frequently requested skills
As of November 17, 2026
Hiring insights calculated from active jobs tracked by LiftmyCV.
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inMore at TMS LLC
Similar jobs from this company
Across the market
Similar jobs from other companies
- Remote Insurance Agent - No ExperienceAmerican Income Life Insurance CompanyRemote
- Customer Experience EvaluatorCXGPrague, CZRemote
- Agent Evaluation InternIMAGE FRAME INVESTMENT (UK) LIMITEDUnited Kingdom-LondonOn-site
- AI Agent EvaluatorHippocratic AIUSRemote
- AI Agent EvaluatorAnyone AIARRemote
- Immediate Hiring No Experience Needed Remote WorkAO Globe LifeRemote
Related job searches
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.