We are sharing a specialised part-time consulting opportunity for experienced Software Engineers with hands-on open-source contribution or maintainer experience and strong expertise in repository-level code review, testing, debugging, and software quality evaluation.
This role focuses on reviewing software-engineering benchmark tasks for correctness, reproducibility, and grading integrity. Selected engineers will audit repository-level assignments, reference patches, test harnesses, containerised environments, and evaluation logic while identifying technical flaws, unintended shortcuts, and weaknesses in task design.
Key Responsibilities
Repository-Level Code Review
- Review software engineering tasks built around real code repositories
- Assess whether task requirements are technically clear, complete, and reproducible
- Evaluate repository state, dependencies, configuration, and expected behaviour
- Identify ambiguities or implementation issues that could affect task validity
- Apply practical engineering judgement to realistic codebase-level problems
Reference Patch Auditing
- Review reference patches for correctness and completeness
- Determine whether proposed solutions appropriately address the underlying software issue
- Identify unintended behavioural changes, incomplete fixes, or unsupported assumptions
- Compare reference implementations against task requirements and expected outcomes
- Assess whether alternative valid implementations are treated fairly
Test Harness & Grading Review
- Audit test runners and automated evaluation logic
- Assess whether tests accurately measure the intended behaviour
- Identify missing coverage, brittle assertions, or grading inconsistencies
- Verify that evaluation criteria appropriately distinguish correct from incorrect solutions
- Review benchmark tasks for reliable and repeatable scoring
Reproducibility & Environment Validation
- Evaluate whether tasks can be reproduced consistently across clean environments
- Review dependency installation, build processes, configuration, and runtime requirements
- Assess Docker-based isolation and containerised execution
- Identify environmental dependencies or hidden assumptions affecting reproducibility
- Verify that tasks execute reliably under their intended setup
Benchmark Integrity
- Identify potential answer leakage, unintended shortcuts, or reward-hacking opportunities
- Evaluate whether benchmark structure exposes information that makes tasks artificially easy
- Review task and grading design for loopholes or exploitable behaviours
- Assess whether successful completion genuinely demonstrates the intended engineering capability
- Recommend improvements where benchmark integrity is compromised
Software Testing & Debugging
- Investigate failing or inconsistent benchmark tasks
- Review stack traces, logs, test failures, and repository behaviour
- Identify root causes of technical issues
- Distinguish task defects from legitimate implementation failures
- Assess whether debugging and validation processes follow sound engineering practices
Open-Source Engineering
- Apply experience from contributing to or maintaining open-source software
- Evaluate repository conventions, contribution patterns, and realistic development workflows
- Review patches with the perspective of an experienced contributor or maintainer
- Assess whether proposed changes would meet reasonable code-review expectations
- Apply practical judgement derived from real-world pull request and repository experience
Multi-Language Code Evaluation
- Review software written in Python
- Evaluate tasks involving at least one additional ecosystem such as Java, Go, TypeScript, or C++
- Assess code structure, tests, implementation choices, and repository conventions across languages
- Identify language-specific implementation or testing issues
- Apply consistent engineering standards across different technology stacks
Rubric-Based Evaluation
- Assess benchmark tasks against structured technical criteria
- Provide clear written explanations supporting evaluation decisions
- Reference specific code, tests, patches, or execution behaviour when identifying issues
- Apply grading standards consistently across assignments
- Distinguish substantive benchmark defects from minor implementation differences
Ideal Profile
- 3+ years of professional software engineering experience
- Demonstrated open-source contribution or maintainer experience, such as merged pull requests, committer responsibilities, or maintainer roles
- Strong ability to review repository-level software changes
- Experience auditing reference patches, test runners, and automated test suites
- Comfortable evaluating Docker-based isolation and reproducible development environments
- Strong understanding of software testing, debugging, and code-review practices
- Ability to identify answer leakage, reward hacking, or other benchmark-integrity issues
- Strong proficiency in Python
- Professional fluency in at least one additional language such as Java, Go, TypeScript, or C++
- Familiarity with SWE-Bench Verified or similar repository-level software engineering benchmarks is preferred
- Maintainer or contributor history with established Python open-source projects is highly valued
- Previous code-review, software evaluation, or task-grading experience is advantageous
- Strong written communication and ability to provide precise technical feedback
Engagement Details
- Part-time independent contractor engagement
- Fully remote within the United States
- Flexible scheduling based on project requirements
- Compensation: $60–$80/hour
- Work focuses on repository-level software evaluation, reference-patch review, testing, reproducibility, benchmark integrity, and technical quality assessment
- Projects may be extended, shortened, or concluded based on project needs and performance
- Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party
- H1-B and STEM OPT support is unavailable for this engagement
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.