Internal Medicine Clinical Reasoning Expert (AI Evaluation)
About the Role
A leading healthcare AI initiative is seeking Internal Medicine experts to contribute to the development of advanced clinical reasoning systems. This opportunity focuses on improving how AI models interpret, analyze, and respond to complex patient-care scenarios using evidence-based medical knowledge.
This opportunity is ideal for board-certified Internal Medicine physicians, hospitalists, subspecialists, final-year residents, and fellows who possess strong diagnostic reasoning skills and a commitment to clinical excellence. Candidates should be comfortable evaluating medical content and applying current standards of care.
The work involves creating realistic clinical cases, producing high-quality reference answers, assessing AI-generated outputs, and providing structured feedback to support model improvement. Attention to clinical accuracy, guideline alignment, and clear reasoning is critical to success in this role.
What You'll Do
- Design realistic Internal Medicine clinical scenarios and diagnostic cases
- Develop attending-level reference responses based on evidence-based practice
- Evaluate AI-generated medical outputs using structured assessment criteria
- Assess diagnostic reasoning, treatment recommendations, and guideline adherence
- Provide detailed written feedback to improve model performance
- Review clinical content for accuracy, consistency, and completeness
- Participate in onboarding activities and specialty calibration sessions
- Support quality assurance efforts related to healthcare reasoning datasets
Requirements
- Board certification in Internal Medicine with an active, unrestricted medical license (for attending physicians)
- Final-year Internal Medicine residency status or board eligibility following recent completion of training (for residents)
- Board-certified or board-eligible status in primary specialty with an active, unrestricted medical license (for fellows)
- Strong expertise in diagnostic reasoning, treatment planning, and patient management
- Ability to apply evidence-based guidelines to clinical decision-making
- Excellent written communication and analytical evaluation skills
- Experience reviewing or creating medical education, assessment, or clinical documentation materials
- Ability to work independently in a fully remote, asynchronous environment
- Availability for approximately 20 hours per week, with flexibility based on project demand
- Eligibility to work as an independent contractor in the United States or Canada
- Comfort using digital collaboration and documentation tools
- Experience contributing to AI evaluation, medical content development, healthcare research, or clinical quality initiatives preferred
- Familiarity with generative AI tools and clinical AI applications preferred