We are sharing a specialised full-time consulting opportunity for experienced QA and test engineers with strong expertise in test-case design, end-to-end debugging, quality assurance, Python, and complex technical evaluation workflows.
This role supports the development of advanced agentic evaluation benchmarks for frontier AI models. Selected professionals will review complex multi-step tasks, test reference solutions, identify ambiguity and grading gaps, debug technical environments, and develop repeatable quality processes that keep benchmark results accurate and trustworthy.
Key Responsibilities
Test Case Design
- Create comprehensive test cases confirming that benchmark tasks function as intended
- Design positive, negative, boundary, and edge-case tests
- Validate task requirements, expected outputs, reference solutions, and grading logic
- Identify scenarios that may produce incorrect or misleading evaluation results
- Ensure tests measure the intended technical capability accurately
Benchmark Task Review
- Review complex multi-step tasks and reference solutions before finalisation
- Identify ambiguous instructions, inconsistent requirements, missing assumptions, and incomplete acceptance criteria
- Run tasks independently to confirm reproducibility and expected behaviour
- Assess whether grading standards are clear, fair, and technically defensible
- Provide actionable feedback to task authors and researchers
Technical Debugging
- Investigate failures across Python scripts, test harnesses, repositories, and task environments
- Diagnose unexpected behaviour within unfamiliar codebases
- Reproduce reported issues and isolate their underlying causes
- Correct or document environment, dependency, logic, and validation problems
- Use Git-based workflows to support structured review and collaboration
Quality Process Development
- Develop practical checklists and repeatable review procedures for benchmark quality
- Improve consistency across task validation, testing, and approval workflows
- Document findings clearly so authors can resolve issues efficiently
- Track recurring defects and recommend preventive quality measures
- Collaborate closely with researchers, task authors, and other technical reviewers
Benchmark Integrity & Shortcut Detection
- Examine AI agent runs for unintended shortcuts, loopholes, and grading weaknesses
- Identify cases where models can receive credit without completing the intended reasoning or technical work
- Test whether benchmark tasks remain robust across alternative approaches
- Strengthen evaluation criteria to maintain reliable and meaningful benchmark scores
- Distinguish valid solution diversity from unintended task exploitation
Ideal Profile
Strong candidates may have:
- At least 1 year of experience in test engineering, quality assurance, software engineering, research engineering, or a related technical role
- Demonstrated experience designing test cases and quality-review processes
- Strong end-to-end debugging skills across complex technical systems
- Working proficiency in Python and Git
- Comfort navigating unfamiliar codebases, repositories, and execution environments
- Exceptional attention to detail and strong written documentation habits
- Ability to identify ambiguity, edge cases, hidden assumptions, and quality gaps
- Capacity to work independently through open-ended technical problems
- Reliable availability for approximately 35 hours per week
Educational Background
- A master's degree or PhD in a STEM field is highly relevant
- Equivalent practical experience in an engineering-intensive or research-intensive domain may also be considered
- Academic or professional experience involving computer science, software engineering, machine learning, mathematics, statistics, or a related technical field may strengthen an application
- Technical research, open-source contributions, testing projects, or substantial engineering work may also be valuable
Nice to Have
- Experience with AI training, model evaluation, or quality review of AI-generated work
- Familiarity with agentic systems and multi-step AI benchmarks
- Background testing machine learning, research, or data-processing workflows
- Experience developing automated test suites or validation scripts
- Familiarity with CI/CD systems, test harnesses, containers, or reproducible environments
- Experience reviewing reference solutions, grading logic, or technical rubrics
- Knowledge of adversarial testing, failure-mode analysis, or benchmark design
- Prior collaboration with AI research or evaluation teams
Why This Opportunity
- Serve as the quality backbone for advanced agentic AI benchmarks
- Ensure complex evaluation tasks are accurate, reproducible, and resistant to shortcuts
- Apply testing and debugging expertise to technically challenging AI research workflows
- Work closely with researchers and task authors on benchmark improvement
- Help protect the reliability of evaluation results for frontier AI systems
- Participate in a structured full-time remote role with competitive hourly compensation
Contract Details
- Full-time W-2 contingent employment opportunity
- Fully remote within the United States
- Expected commitment of approximately 35 hours per week
- Competitive rates between $55–$85 per hour depending on expertise and project scope
- Individual task reviews may require one to two days of focused technical work
- Work may include test design, benchmark review, Python debugging, quality-process development, and shortcut detection
- Close collaboration with research and task-development teams
- Engagement scope and duration may evolve according to project requirements and performance
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.