AI QA Engineer

We are seeking an AI Quality Engineer to join a high-impact team delivering AI-enabled capabilities within enterprise systems for a leading wealth management client. This role will help ensure that AI-powered applications—including LLM-based assistants, retrieval-augmented generation (RAG) solutions, machine learning models, and intelligent workflows—are reliable, accurate, secure, and ready for enterprise use.

The AI Quality Engineer will define and execute quality strategies across the AI solution lifecycle, from data preparation and model evaluation through integration testing, production monitoring, and continuous improvement. The ideal candidate brings strong quality engineering discipline, hands-on experience testing complex enterprise applications, and a practical understanding of AI/ML or LLM-driven systems.

This is an opportunity to shape repeatable AI quality practices in a visible, regulated environment where accuracy, user trust, operational reliability, and measurable business outcomes are critical. The role aligns with established AI risk-management expectations that emphasize ongoing testing, evaluation, verification, and validation throughout the AI lifecycle.nist+1

Key Responsibilities

• Design and execute test strategies for AI-powered applications, including functional, integration, regression, usability, performance, and non-functional testing.
• Validate LLM-based solutions for accuracy, relevance, consistency, groundedness, safety, latency, and end-to-end task completion.
• Test retrieval-augmented generation solutions, including document ingestion, retrieval quality, prompt behavior, response quality, citations or source alignment, and orchestration workflows.
• Develop test cases, evaluation datasets, benchmark scenarios, and expected outcomes for prompts, retrieval pipelines, model-driven workflows, and end-user interactions.
• Identify model failure modes, hallucinations, edge cases, bias concerns, workflow breakdowns, and integration issues; document findings and recommend corrective actions.
• Perform data validation activities, including data preprocessing checks, feature validation, test-data quality assessment, and data-flow testing.
• Support validation of machine learning and AI models using relevant metrics, business scenarios, and acceptance criteria.
• Test APIs, data pipelines, system integrations, workflows, and dependencies supporting AI-enabled business processes.
• Partner with AI engineers, software engineers, product owners, business analysts, and stakeholders to define quality standards, acceptance criteria, and release-readiness requirements.
• Contribute to automation of testing, AI evaluations, regression checks, monitoring, and quality reporting across development and production environments.
• Monitor production feedback, user behavior, operational metrics, and model performance to identify opportunities for improvement.
• Document defects, test results, risks, quality decisions, methodologies, and release recommendations.
• Support quality governance, including traceability, defect management, root-cause analysis, and evidence for release approvals.
• Help establish scalable AI quality engineering practices, controls, and reusable testing assets for enterprise delivery.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Systems, Data Science, or a related technical field.
• 6+ years of experience in quality engineering, QA automation, software testing, test engineering, or a related technical discipline.
• Experience testing enterprise applications, APIs, data pipelines, workflow-based systems, or integrated technology platforms.
• Experience evaluating AI-, machine learning-, or LLM-enabled solutions, including prompt behavior, output quality, model-driven workflows, or automated decision-support capabilities.
• Strong knowledge of software-testing methodologies, including functional, integration, regression, end-to-end, performance, and user-acceptance testing.
• Experience creating test plans, test cases, test data, defect reports, quality metrics, and release-readiness documentation.
• Understanding of defect management, root-cause analysis, risk assessment, and quality reporting.
• Familiarity with QA automation frameworks, test-management tools, CI/CD-aligned testing, and Agile delivery practices.
• Ability to work effectively with both technical teams and business stakeholders.
• Strong written and verbal communication skills, with the ability to clearly communicate quality risks, trade-offs, and recommendations.

Preferred Qualifications

• Experience testing LLM applications, AI assistants, generative AI solutions, intelligent agents, or RAG-based systems.
• Experience with LLM evaluation, prompt testing, hallucination testing, retrieval validation, AI observability, or automated evaluation frameworks.
• Familiarity with Python, SQL, REST APIs, JSON, scripting, or automation tools used for test automation and data validation.
• Experience creating synthetic datasets, golden datasets, benchmark scenarios, adversarial test cases, or AI evaluation frameworks.
• Exposure to model monitoring, production observability, drift detection, model-performance metrics, or feedback-loop design.
• Familiarity with responsible AI, AI governance, privacy, security, model risk, or compliance controls.
• Experience in financial services, wealth management, consulting, insurance, healthcare, or another regulated environment.
• Experience working with cloud-based AI services or platforms such as Azure AI, AWS Bedrock, Google Vertex AI, OpenAI, or similar tools.

Final compensation offered is based on multiple factors such as the specific role, hiring location, as well as individual skills, experience, and qualifications. In addition to competitive salaries, we offer a comprehensive benefits package.