Senior AI/ML Test and Evaluation Engineer

Senior AI/ML Test and Evaluation Engineer

Location: Washington, DC; Denver, CO; or Colorado Springs, CO preferred (hybrid). Highly qualified candidates outside these locations may also be considered.  

Work Authorization: U.S. citizenship required

Clearance: An active U.S. security clearance is strongly preferred. Candidates without an active clearance may be considered for unclassified work but must be eligible to obtain and maintain a clearance. 

Salary Range: $145,000–$250,000 USD, dependent on experience level and location

About the Role

We're looking for a Senior AI/ML Test and Evaluation Engineer to build and operate the benchmarking and evaluation capability at the core of an AI platform. This is a role for someone who is more interested in what a model gets wrong than in what it gets right.

You build the evaluation harnesses — automated metrics paired with structured human expert judgment, applied to candidate models and to the agentic workflows built on top of them. You develop repeatable methodologies for comparing performance against current operational baselines, which means the comparison holds up when someone runs it again in six months with a different model. And you document the limitations and surface the failure modes that matter, including the ones nobody asked about.

Your reports go to senior stakeholders and inform decisions about which capabilities are ready to field. That's the weight of the job: a benchmark that looks good and hides a failure mode is worse than no benchmark at all, and you're the check against that.

This is hands-on engineering on an open-source toolchain.

This position is contingent upon contract award. Travel of up to 15% may be required, primarily to Government facilities and between company locations.

Key Responsibilities

• Design, implement, and operate benchmark execution and evaluation harnesses for AI models and agentic workflows
• Develop evaluation methodologies that combine automated metrics with structured human subject matter expert judgment
• Curate and recommend candidate benchmarks based on mission needs and document the provenance of ground-truth and reference data
• Produce defensible evaluation reports comparing candidate capabilities with current mission workflows, including documented limitations and failure modes
• Define and contribute to common standards for benchmark expression, ingestion, and reporting
• Support partner organizations and vendors as they integrate their capabilities with shared evaluation standards
• Build lightweight expert-scoring workflows and measure inter-reviewer agreement for judgment-based evaluations
• Participate in structured feedback sessions with mission end users and incorporate findings into the platform and evaluation methodology
• Develop reference notebooks and example workflows that enable data-science-capable analysts to run, interpret, and extend evaluations
• Document technical approaches, evaluation results, and key decisions for Government stakeholders and internal teams

Required Skills & Experience

• U.S. citizenship and eligibility to obtain and maintain a U.S. security clearance
• 6+ years of software engineering or machine learning engineering experience, including 3+ years evaluating, benchmarking, or deploying ML models in production or applied research environments
• Strong Python proficiency in a machine learning or data science context
• Hands-on experience with common ML frameworks and tooling, such as PyTorch and the Hugging Face ecosystem
• Experience developing or using model evaluation harnesses, benchmark suites, or test and evaluation frameworks
• Experience designing evaluation metrics and applying appropriate statistical rigor when interpreting and reporting results
• Experience building repeatable and auditable evaluation pipelines with documented data provenance
• Experience evaluating large language models or agentic workflows using task-based, metric-based, or judgment-based scoring
• Strong written communication skills, including the ability to clearly explain evaluation methodologies, results, limitations, and failure modes to technical and nontechnical stakeholders
• Ability to work effectively in an evolving environment and translate mission needs into practical evaluation approaches
• Bachelor’s degree in computer science, mathematics, engineering, or a related field, or equivalent practical experience

Nice to Have

• Active U.S. security clearance
• Prior AI/ML evaluation or test and evaluation experience supporting the Department of Defense, Intelligence Community, or another federal customer
• Experience designing human-in-the-loop evaluations, measuring inter-rater reliability, or facilitating structured expert adjudication
• Experience defining or implementing benchmark interchange formats or evaluation standards used across multiple organizations
• Familiarity with intelligence analysis workflows or other high-stakes analytical domains
• Experience working directly with Government stakeholders, mission users, or external technical partners
• Contributions to open-source machine learning, benchmarking, or evaluation projects

What We Offer

• Medical, Dental & Vision – 100% paid for employees, 75% for dependents
• 401(k) Match – Up to 5% with full vesting after 2 years
• Unlimited PTO – With a required minimum of 15 days off annually
• Fully Remote Setup – Includes up to $3,000 equipment reimbursement
• Continuous Education –  Includes up to $500 reimbursement
• Disability & Life Insurance – 100% employer-paid
• HSA & FSA Options – With monthly HSA contributions from OpenTeams