Medical Domain Expert – AI Quality Assessment
About the Role
A high-impact AI evaluation initiative is seeking experienced health and medical professionals to contribute to the improvement of advanced AI systems. This work focuses on ensuring the reliability, safety, and factual integrity of AI-generated outputs in sensitive, high-stakes healthcare contexts.
This opportunity is ideal for individuals with strong domain expertise who can critically assess technical content and apply clinical or regulatory judgment. It suits professionals comfortable working independently in structured evaluation environments with clear scoring frameworks.
The work involves reviewing AI-generated responses, identifying inaccuracies or risks, and producing structured evaluations supported by evidence. Success in this role depends on precision, analytical thinking, and the ability to interpret healthcare standards in dynamic scenarios.
What You'll Do
- Design realistic prompts reflecting real-world healthcare inquiries from professionals and consumers
- Evaluate AI-generated responses for factual accuracy and clinical correctness
- Identify misleading reasoning, fabricated claims, or unsupported references
- Score and rank outputs using predefined evaluation rubrics
- Provide detailed written justifications supported by domain evidence
- Apply healthcare regulations and guidelines when assessing outputs
Requirements
- Master’s degree or higher in Health, Medicine, or a related field
- Professional experience in a clinical, advisory, or applied healthcare role
- Strong understanding of healthcare standards, regulations, or clinical guidelines
- Excellent written communication and critical reasoning skills
- Ability to work independently with consistent output quality
- Comfort using structured evaluation frameworks and scoring systems
- Availability for approximately 20 hours per week
- Preferred: Experience reviewing technical or AI-generated content
- Preferred: Familiarity with AI tools or evaluation platforms such as ChatGPT