AI Agent Evaluator
Hippocratic AI is looking for experienced Nurses and Nurse Practitioners to evaluate AI-driven clinical agents. This part-time, remote role involves assessing conversations and recommendations surrounding acute symptom triage, diabetes management, and travel health guidance to ensure clinical accuracy and patient safety. Responsibilities include reviewing AI agent interactions, providing structured feedback, and collaborating with clinical teams. Candidates must have at least 5 years of clinical experience and relevant licenses. This flexible contract position requires up to 10 hours per week.
Job description
AI Agent Development & Evaluation
Primary Care, Acute Symptom Triage & Travel Health | Contract, up to 10 hrs/week
About the Role
We are seeking Nurses & Nurse Practitioners with clinical experience in one or more of the following areas:
- acute symptom management
- CHF and hypertension management
- diabetes management
- travel health
to serve as a subject matter expert supporting the evaluation of AI-driven clinical agents. In this role, you will evaluate agent conversations and recommendations spanning acute symptom triage and escalation decisions, insulin/diabetes titration, and pre-travel health guidance, ensuring clinical accuracy, guideline adherence, and patient safety. This is a part-time, remote contract position (1099) requiring up to 10 hours per week, with flexible scheduling.
Key Responsibilities
- Review recorded and simulated AI agent interactions involving acute symptom triage and assess the appropriateness of escalation and de-escalation decisions.
- Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence.
- Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment.
- Identify clinical safety concerns, edge cases, and gaps in the agent's triage logic or clinical reasoning.
- Score and annotate agent performance using structured evaluation rubrics.
- Provide clear, actionable written feedback to clinical and product teams to inform agent refinement.
- Participate in periodic case review and calibration sessions with other clinical SMEs to ensure consistent evaluation standards.
- Help develop and refine test scenarios used in ongoing agent quality assurance.
Qualifications
- Registered Nurses: active, unrestricted RN License. Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP).
- Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting.
- Clinical experience managing patients with diabetes, including insulin titration.
- Strong, current working knowledge of relevant triage protocols and clinical practice guidelines.
- Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions.
- Comfortable working independently in a remote, asynchronous contract role.
- Strong written communication skills for delivering structured clinical feedback.
Recommended
It is not a requirement that candidates have all of the recommended criteria
- Experience in telephone triage or nurse advice line settings.
- Experience in travel medicine, occupational health, or pre-travel consultation.
- Familiarity with CDC and other travel medicine guidelines.
- Prior experience in clinical quality assurance, utilization review, or case review.
- Familiarity with AI-assisted clinical tools or digital health platforms.
Engagement Details
- Contract (1099) position, up to 10 hours per week.
- Fully remote; flexible scheduling.
- Hourly compensation, commensurate with experience.
- Engagement expected to run in support of ongoing agent development and FDA evaluation milestones.
Job details
- Published
- August 6, 2026
- Last checked
- November 17, 2026
- Job source
- ashby
- Workplace
- Remote · US
- First seen
- August 6, 2026
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inMore at Hippocratic AI
Similar jobs from this company
None listed.
Across the market
Similar jobs from other companies
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.