Job Title: Medical Evaluation Specialist
Role Type: Contractor
Location: Remote
Job Overview
We are seeking Medical Evaluation Specialists, including medical students, residents, physicians, and qualified biomedical professionals, to contribute clinical expertise to a project focused on evaluating and improving next-generation AI systems.
In this role, you will create and validate challenging medical questions and answers designed to test advanced clinical reasoning. Your work will help assess whether AI systems can accurately interpret medical evidence, synthesize complex information, and apply nuanced clinical judgment.
No prior AI experience is required—your medical knowledge, research skills, and clinical reasoning are what matter most.
Scope of Work
- Create original, high-difficulty medical question-and-answer pairs covering areas such as diagnosis, pathophysiology, pharmacology, clinical guidelines, and clinical decision-making.
- Develop questions that require synthesis, interpretation, and genuine medical reasoning rather than simple factual recall.
- Research and verify answers using primary literature, clinical guidelines, systematic reviews, and authoritative medical references.
- Document the rationale behind answers and provide appropriate supporting citations.
- Evaluate AI-generated responses for clinical accuracy, completeness, reasoning quality, and adherence to available evidence.
- Identify questions that are too straightforward and refine them to increase complexity while maintaining clinical validity.
- Review questions and answers for ambiguity, unsupported assumptions, factual errors, and inconsistencies.
- Ensure all deliverables are written with clarity, precision, and defensibility.
- Incorporate reviewer feedback and adapt work to evolving project guidelines and quality standards.
- Contribute insights that help establish rigorous benchmarks for medical AI evaluation.
Required Skills
- Medical Training
- Research & Source Triangulation
- Attention to Detail
- Written Precision
- Analytical Thinking
- Clinical Reasoning
- Medical Literature Review
- Evidence Synthesis
- Self-Direction & Reliability
- Medical Question Development
Preferred Qualifications
- Current or recent medical training or clinical practice as a medical student, resident, or physician, or equivalent expertise in a relevant biomedical discipline.
- Demonstrated ability to locate, interpret, compare, and synthesize information from primary research and clinical guidelines.
- Strong written English skills with exceptional attention to detail.
- Ability to create engaging, well-structured, challenging, and clinically nuanced questions.
- Strong understanding of evidence-based medicine and clinical reasoning.
- Ability to work independently, manage deadlines, and consistently deliver high-quality work in a remote environment.
- Previous experience with medical question writing, peer review, clinical education, medical content evaluation, or research is a plus.
- Familiarity with AI/ML systems, AI evaluation, or medical AI applications is advantageous but not required.
Compensation Structure
Compensation is output-based, with contributors paid per task that meets the applicable project specifications and quality standards. The time required to complete each task may vary depending on experience and individual workflow.
Minimum submission requirements apply, and selected contributors are expected to complete a minimum number of tasks per week.
Start Timeline & Availability
Roles are typically filled within 48 hours. Selected contributors should be prepared to begin their first assignments within 24–48 hours of completing onboarding.