Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: Generalist Annotator — Health AI Conversation Quality Evaluation Type:Contract Compensation:$20–$160/hour Location:Remote
Role Responsibilities
Evaluate the relevance and clarity of AI responses. Ensure the AI answers the question in understandable language for non-experts.
Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.
Qualifications
Must-Have
Strong written English skills.
Ability to explain judgments in two to four specific sentences.
Careful reading to notice omissions in responses.
Comfort with applying a detailed rubric consistently.
Reliable availability for time-boxed batches.
Preferred
Experience in annotation, human-feedback, evaluation, or content review.
Background in explaining technical concepts to non-experts.
Experience with rubric-based grading or quality assurance.
Important
Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.
Application Process (Takes 20–30 mins to complete)
Upload resume
AI interview based on your resume
Submit form
Resources & Support
For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
For any help or support, reach out to: support@mercor.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#hiringmercor
Numbers & Facts
Location
New York, New York (Remote)
Skills
Artificial Intelligence (AI)unmatched
Benchmarkingunmatched
Data Setsunmatched
English Languageunmatched
Healthcare Qualityunmatched
Research Laboratoryunmatched
Training/Teachingunmatched
Writing Skillsunmatched
🎯
Be found by employers
5,500+ employers search our resume database daily. Add yours to get found by recruiters looking for candidates like you.
Level up your application
Professional resume templates
Browse dozens of recruiter approved resume templates, layouts and formats. Choose your favorite and make it your own in minutes.