We are sharing a specialised full-time consulting opportunity for US-based linguists, instructional designers, and technical writers experienced in developing clear evaluation guidelines, structured rubrics, and human-rating instructions for generative AI programmes.
This role supports a high-impact generative AI initiative focused on translating complex and potentially ambiguous programme requirements into precise, practical guidance for human evaluators. Selected professionals will develop rater-ready instructions across domains such as finance, retail, insurance, legal, and sports while resolving contradictions, defining edge cases, and improving consistency throughout evaluation workflows.
Key Responsibilities
Rater Guideline Development
- Translate programme requirements into clear, structured, and actionable instructions for human evaluators
- Develop guidelines that can be applied consistently across standard scenarios and complex edge cases
- Define terminology, rating criteria, decision rules, exceptions, and escalation pathways
- Ensure instructions are accessible to raters while preserving necessary domain-specific precision
Rubric & Evaluation Framework Design
- Design detailed scoring rubrics for evaluating generative AI outputs
- Establish measurable criteria covering correctness, relevance, reasoning quality, completeness, and instruction adherence
- Create examples and counterexamples illustrating different performance levels
- Align evaluation frameworks with programme objectives and quality standards
Ambiguity & Consistency Review
- Review draft specifications for ambiguity, contradiction, missing information, and inconsistent terminology
- Identify instructions that may lead to conflicting interpretations across raters
- Revise guideline sets until they can be applied reliably with minimal escalation
- Document concrete before-and-after improvements to written requirements and evaluation instructions
Cross-Domain Instructional Translation
- Convert specifications from finance, retail, insurance, legal, sports, and other specialist domains into rater-ready guidance
- Collaborate with subject matter experts to understand domain-specific terminology and professional judgment
- Preserve important technical nuance while making instructions clear to non-specialist evaluators
- Maintain consistent structure and quality across multiple domain-specific guideline sets
Ideal Profile
Strong candidates may have:
- At least 3 years of professional experience in linguistics, instructional design, technical writing, content design, or a closely related field
- Direct experience developing or refining guidelines and rubrics for human evaluators in generative AI, RLHF, or model-assessment programmes
- Demonstrated ability to resolve ambiguity and contradiction in complex written specifications
- Experience translating specialist requirements into clear and practical instructions
- Ability to work effectively across multiple subject-matter domains
- A portfolio or concrete examples showing measurable improvements to guidelines, rubrics, or instructional materials
- Demonstrable professional growth and increasing responsibility
- Reliable availability for at least 35 hours per week during weekdays
Educational Background
- A degree in linguistics, instructional design, education, communications, technical writing, language studies, or a related field is highly relevant
- Graduate-level education in applied linguistics, learning design, human-computer interaction, or information design may be helpful
- Equivalent professional experience in AI evaluation, technical documentation, or guideline development may also be considered
- Training in assessment design, taxonomy development, content strategy, or quality assurance may be valuable
Nice to Have
- Experience supporting large language model evaluation, reinforcement learning from human feedback, or AI training-data programmes
- Familiarity with annotation platforms, human-feedback workflows, and rater calibration processes
- Experience developing domain-specific guidance for finance, insurance, retail, legal, sports, or comparable fields
- Knowledge of controlled language, information architecture, taxonomy design, or content governance
- Experience conducting guideline usability tests or analysing inter-rater consistency
- Familiarity with version control, documentation systems, and structured authoring tools
- Previous collaboration with researchers, programme managers, engineers, and subject matter experts
Why This Opportunity
- Apply linguistic and instructional-design expertise to an advanced generative AI initiative
- Influence the clarity and reliability of human evaluation processes
- Develop guidelines used across a wide range of professional subject-matter domains
- Solve complex problems involving ambiguity, edge cases, and evaluation consistency
- Join a full-time remote engagement with competitive hourly compensation
Contract Details
- Full-time W-2 contingent employment arrangement
- Fully remote role available to candidates based in the United States
- Expected commitment of at least 35 hours per week during weekdays
- Competitive rates between $40–$60 per hour depending on expertise and project scope
- Direct experience developing rater guidelines or rubrics for generative AI or RLHF programmes is required
- Applicants should be prepared to provide concrete examples of guideline or specification improvements
- Immediate availability is preferred
- Work may include onboarding, calibration, documentation review, and ongoing guideline refinement
- Project scope and duration may be adjusted according to programme requirements and performance
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.