QA Engineer for AI Products

We are seeking a QA Engineer for AI Products for a six-month contract position in Bellevue, Washington. This is a hybrid role focused on testing and evaluating large language model (LLM) and generative AI features, with hands-on responsibilities across model validation, test automation, and quality assurance frameworks.

Responsibilities

• Design and execute test plans for AI/ML-driven features, including model outputs, prompts, and integrated application behavior
• Build and maintain automated test suites covering functional, regression, integration, and API testing
• Evaluate model outputs for accuracy, consistency, bias, hallucination, and edge-case failures
• Develop evaluation frameworks and golden datasets/test cases to benchmark model performance over time
• Test prompt engineering changes, model version upgrades, and fine-tuning outputs for regressions
• Perform adversarial and red-team style testing to surface safety, security, and robustness issues
• Validate data pipelines feeding into AI models, including data quality, schema, and drift detection
• Collaborate with data scientists and ML engineers to define acceptance criteria and quality metrics for models
• Test latency, scalability, and reliability of AI services under load
• Contribute to CI/CD pipelines, integrating automated and model-evaluation tests
• Design and execute test cases for LLM-based products such as chatbots, copilots, RAG systems, and AI agents, with focus on non-deterministic and generative outputs
• Build evaluation suites using AI/LLM evaluation frameworks, including scoring rubrics and golden datasets
• Validate prompt changes and model/version upgrades against baseline evaluation sets
• Write evaluation scripts, test harnesses, and data validation logic in Python

Qualifications

• Required

• 5+ years of quality assurance or testing experience

• Hands-on experience testing LLM/generative AI products

• Expertise in Python for test automation and evaluation scripting

• Proficiency in SQL and data validation techniques

• Practical experience with AI/LLM evaluation frameworks such as Ragas, DeepEval, LangSmith, Promptfoo, OpenAI Evals, or TruLens

• Working knowledge of evaluation metrics for generative AI, including hallucination rate, faithfulness/groundedness, relevance, answer correctness, toxicity/bias scoring, and semantic similarity measures

• Experience with prompt regression testing and validating prompt changes against baseline evaluation sets

• Test automation and automated test suite development

• Preferred

• Prompt engineering or prompt testing experience

• CI/CD pipeline automation experience

• Exposure to Azure, Snowflake, or cloud data platforms