AI Test Engineer

Mindlance

Ann Arbor, MI

Apply

JOB DETAILS

SKILLS

Agile Programming Methodologies, Amazon Web Services (AWS), Analysis Skills, Application Framework, Application Programming Interface (API), Artificial Intelligence (AI), Artificial Intelligence (AI) Agents, Automation, Automation Engineering, Benchmarking, Best Practices, Bug Tracking/Defect Management, Cloud Applications, Cloud Computing, Computer Science, Continuous Deployment/Delivery, Continuous Integration, Distributed Computing, Docker, Engineering, GCP (Good Clinical Practices), GitHub, Home Automation, Information Technology & Information Systems, Integration Testing, Internet Application, JUnit, Java, JavaScript, Jenkins, Load Testing, MCP - Microsoft Certified Professional, Machine Learning, Mail Services, Microsoft C# (C Sharp), Microsoft Windows Azure, Model Validation, Performance Testing, Production Support, Pytest, Python Programming/Scripting Language, Quality Engineering, Quality Metrics, Regression Testing, Regulatory Compliance, Risk, Risk Management, Root Cause Analysis, Search Ranking, Security Architecture, Selenium, Software Development Lifecycle (SDLC), Software Engineering, Software Testing, Stress Testing, System Integration (SI), System Validation, Technical/Engineering Design, Test Automation, Test Case, Test Data, Test Harness, Test Plan/Schedule, Test Strategy, Test Tools, Testing

LOCATION

Ann Arbor, MI

POSTED

4 days ago

Work Location: Ann Arbor, MI

6 months 100% onsite. After 6 months 4 days on site and 5th day remote. Candidate will pick their remote day.

Mid-Level AI Software Test Engineer
Position Summary
We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions. This role combines traditional software quality engineering with emerging AI validation techniques to ensure AI systems are reliable, safe, accurate, performant, and production-ready.
The ideal candidate has a strong software testing background, experience building automated test fraeworks, and an interest in AI technologies such as LLMs, agents, RAG systems, MCP integrations, and machine learning models. AI engineering organizations increasingly emphasize AI evaluation frameworks, automation, reliability, governance, and production quality, making testing a critical function in AI delivery.
Key Responsibilities
AI Quality Engineering

Develop comprehensive testing strategies for AI applications, platforms, and services.
Validate AI model outputs for accuracy, consistency, reliability, and safety.
Design and execute functional, integration, end-to-end, regression, and performance tests for AI solutions.
Create test cases for prompt-driven, agentic, and retrieval-based AI workflows.
Validate AI guardrails, business rules, permissions, and governance controls.
Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations.

Automation

Build and maintain automated test frameworks for AI applications.
Develop automated evaluation pipelines for AI responses and workflows.
Integrate AI testing into CI/CD pipelines.
Implement automated quality scoring and regression detection.
Create reusable test data, mocks, simulators, and validation frameworks.

Platform & Integration Testing

Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities.
Validate integrations with external systems, data sources, and enterprise services.
Verify performance, reliability, scalability, and resiliency of AI workloads.
Execute load and stress testing for AI services.

Collaboration

Partner with software engineers, AI engineers, product owners, architects, and security teams.
Participate in design reviews and provide quality feedback during development.
Contribute to test strategy, quality standards, and best practices.
Support production readiness reviews and defect triage activities.

Required Qualifications

Bachelor's degree in Computer Science, Software Engineering, Information Systems, or related field.
3–6 years of software testing, QA automation, or quality engineering experience.
Experience developing automated test solutions using:

o Python
o Java
o JavaScript/TypeScript
C#

Experience with API testing and automation tools.
Strong understanding of:

o Test automation
o Software development lifecycle (SDLC)
o Agile methodologies
o CI/CD pipelines

Experience testing distributed systems, web applications, and APIs.

Preferred Qualifications

Experience testing:

o Generative AI applications
o LLM-based systems
o AI agents
o RAG applications
o MCP-based integrations

Familiarity with:

o OpenAI
o Claude
o Gemini
o Azure OpenAI

Experience building evaluation and benchmarking frameworks for AI solutions.
Experience testing cloud-native applications on Azure, AWS, or GCP.
Knowledge of responsible AI, AI governance, and AI risk management practices. AI initiatives are typically expected to align with enterprise governance, risk, privacy, and compliance requirements.

Technical Skills
AI Testing

Prompt Testing
Response Evaluation
Hallucination Detection
Agent Workflow Validation
RAG Validation
AI Safety Testing
AI Regression Testing
AI Benchmarking

Quality Engineering

Test Automation
API Testing
Integration Testing
Performance Testing
Load Testing
Security Testing
Defect Analysis
Root Cause Investigation

Tools & Technologies

Selenium
Playwright
Postman
JUnit / PyTest
GitHub Actions
Jenkins
Docker
Kubernetes
Cloud Platforms (Azure, AWS, GCP)

Success Measures

Establishes reliable automated AI testing coverage.
Detects AI quality issues before production deployment.
Reduces regression defects across AI releases.
Improves confidence in AI model and agent behavior.
Enables safe, scalable delivery of AI-powered features.
Ensures AI solutions meet quality, security, and governance requirements.

Example Projects

Testing AI chat assistants and copilots
Validating MCP tools and AI agent workflows
Evaluating RAG search quality and grounding accuracy
Automating AI response evaluation frameworks
Testing AI-powered trading assistants and workflow automation
Performance testing AI services and orchestration platforms

Typical Level
Experience: 3–6 years software quality engineering, with 1–3 years exposure to AI/ML or Generative AI technologies.

Equivalent Titles:

AI Software Test Engineer
AI Quality Engineer
Generative AI Test Engineer
AI Automation Engineer
AI Validation Engineer
Software Engineer in Test (AI) (SDET-AI)

“Mindlance is an Equal Opportunity Employer and does not discriminate in employment on the basis of – Minority/Gender/Disability/Religion/LGBTQI/Age/Veterans.”

About the Company

Mindlance

Resume Resources

Free Resume Templates Free Resume Builder