QA Test Engineer with AI/ML

Job Overview:

Pay Range: $57.40hr - $62.40hr

Requirement/Must Have:

• Detailed knowledge of Salesforce applications, ML model lifecycle, and Generative AI concepts including LLMs, prompt engineering, RAG (Retrieval-Augmented Generation), and vector databases.
• 3+ years of relevant testing experience with a strong background in Software QA with cloud technologies, including 3+ years of experience with hands-on exposure to AI/ML application testing.
• 2+ years of relational database experience including generating complex queries, setting up database connections, and automating query processing, with familiarity in querying ML feature stores or data pipelines.
• Experience in automated and manual testing in cloud environments, particularly for AI/ML and Gen AI applications, including validation of model outputs, embeddings, and LLM-generated responses.
• Experience designing test strategies for non-deterministic systems, including techniques for evaluating accuracy, relevance, and safety of Gen AI outputs.
• 1+ years of experience with BDD/Selenium and automation tools, with a plus for experience in AI testing frameworks.
• Experience using Agile team collaboration and requirements management tools (Jira, Confluence).
• Strong verbal and written communication skills and problem-solving abilities, with the ability to clearly articulate AI/ML testing findings to both technical and non-technical stakeholders.
• Strong aptitude to learn and adapt to emerging AI/ML technologies and evolving Gen AI tooling.
• Familiarity with AWS AI/ML services (SageMaker, Bedrock) or equivalent cloud AI platforms is a plus.
• Bachelor's degree required; technical degree preferred.

Responsibilities:

• Conduct deep technical analysis and testing of AI/ML and Generative AI systems, including validation and verification of model outputs, prompt behavior, and end-to-end application workflows.
• Participate in Business Requirements reviews and collaborate with systems analysts and developers to fully test application changes, including AI/ML model integrations and Gen AI features.
• Understand functional specifications and scope of releases to design robust testing harnesses that ensure quality delivery of AI-powered applications.
• Write complex SQL queries for retrieval of data from multi-database environments, including querying ML feature stores and model metadata.
• Apply expert systems analysis and design techniques to complex enterprise systems, with a focus on validating Gen AI outputs, LLM responses, and ML model predictions.
• Maintain broad knowledge of data sources/flow, interactions of complex systems, and the capabilities and limitations of AI/ML systems, including model drift, hallucination detection, and bias evaluation.
• Work closely with Product Owners, developers, Data Scientists, and stakeholders to understand AI/ML product specifics and create corresponding test strategies, including evaluation frameworks for generative outputs.
• Execute manual and exploratory testing on frontend (UI) and backend (API) systems, including AI/ML inference endpoints and Gen AI prompt pipelines.
• Develop and maintain testing suites for a large-scale Salesforce environment integrated with AI/ML capabilities such as Salesforce Einstein and external LLM services.
• Responsible for front-end and back-end testing for applications built on the Salesforce platform, including AI-driven features and automated decision systems.

Skills:

• Salesforce.
• Machine Learning.
• Generative AI.
• LLM.
• Prompt Engineering.
• RAG.
• Vector Databases.
• SQL.
• Oracle.
• Cloud Technologies.
• BDD.
• Selenium.
• Jira.
• Confluence.
• AWS SageMaker.
• AWS Bedrock.

Qualification And Education:

• Bachelor's degree required; technical degree preferred.