QA Automation Engineer – Agentic AI Testing

Position Overview

We are seeking a QA Automation Engineer to help build and validate a next-generation agentic AI application within a growing AI-driven HCM software ecosystem.

This is not a traditional QA automation role. The application uses large language models (LLMs) and AI agents to dynamically determine application behavior and workflows, creating testing challenges that cannot be solved through conventional scripted automation alone.

The team is building a new Python-based testing and evaluation framework from the ground up specifically for agentic AI systems. This includes LLM-as-judge evaluations, agent state checks, behavioral validation, and testing approaches designed for non-deterministic AI outputs.

Python and hands-on AI/LLM testing experience are the primary requirements for this role. Playwright is used for front-end automation but is a secondary/supporting skill rather than the core focus.

Key Responsibilities

• Design, build, and maintain automated testing and evaluation solutions for agentic and LLM-driven applications.
• Help build a new Python-based AI testing and evaluation framework from the ground up, including defining patterns, standards, and approaches as the platform evolves.
• Develop LLM-as-judge evaluators and other automated methods for assessing AI-generated outcomes.
• Validate agent states, conversational flows, tool interactions, outputs, and expected behaviors across multi-step agentic workflows.
• Develop testing strategies for non-deterministic LLM behavior where conventional scripted pass/fail assertions are insufficient.
• Evaluate AI responses for factors such as accuracy, relevance, consistency, groundedness, and appropriate behavior.
• Test agent behavior across expected paths, edge cases, failure scenarios, and variable outputs.
• Use Playwright for front-end/UI automation where appropriate while maintaining a separate Python-based evaluation layer for AI behavior.
• Partner closely with software engineers and AI developers to identify risks, troubleshoot failures, and improve product quality.
• Help establish testing standards, evaluation methodologies, and automation practices for an evolving agentic AI platform.
• Take shared ownership of quality within a collaborative engineering model rather than operating within a traditional siloed QA organization.

Required Qualifications

• Strong hands-on Python programming and automation experience.
• Experience testing AI/LLM-based, conversational AI, or agentic systems.
• Experience with or strong knowledge of LLM evaluation techniques, including LLM-as-judge, state/behavior validation, or similar approaches.
• Experience validating non-deterministic AI behavior where expected outcomes cannot always be represented through conventional scripted assertions.
• Ability to design and build testing or evaluation frameworks, rather than solely executing within an established automation framework.
• Strong understanding of software testing, automation principles, debugging, and root-cause analysis.
• Ability to design test scenarios for complex, multi-step application and agent workflows.
• Strong problem-solving skills and ability to operate effectively in an early-stage, greenfield environment where standards and processes are still being defined.
• Comfortable working in a distributed quality model, with QA responsibility shared closely across engineering rather than owned solely by a formal QA function.

Preferred Qualifications

• Experience with Playwright for front-end automation.
• Experience with PyTest, Behave, or similar Python testing frameworks.
• Experience testing AI agents, tool/function calling, multi-turn conversations, or complex agentic workflows.
• Experience evaluating LLM outputs for groundedness, hallucination, relevance, correctness, consistency, or completeness.
• Experience testing guardrails, failure scenarios, and AI validation controls.
• Experience validating agent tool usage and downstream system interactions.
• API and database testing experience.
• Experience integrating automated testing and evaluation into CI/CD pipelines.
• Experience within HCM, HR technology, payroll, or related enterprise software.

Team & Environment

This is an early-stage, greenfield AI initiative where engineering conventions, testing standards, and evaluation frameworks are actively being developed.

The team operates using an AI-SDLC model, with small engineering pods working closely with AI agents rather than relying on traditional, highly structured development and QA workflows. Quality ownership is collaborative and distributed across the team.

The broader product is more than a chatbot. It is an expanding agentic HCM software ecosystem, with a conversational interface serving as one entry point into a growing set of AI-driven HR and workforce capabilities.

Ideal Candidate

The strongest candidate will be a Python-focused QA Automation Engineer with genuine hands-on AI/LLM testing experience who is excited about helping define how agentic applications should be tested.

You should understand that testing an AI agent requires more than confirming whether a predefined workflow completes successfully. You will help determine whether an agent behaved appropriately, reached the correct state, used the right tools, produced an acceptable result, and performed reliably across inherently variable interactions.

This role is particularly well suited for someone who enjoys building frameworks rather than simply using them, solving testing problems that do not yet have established answers, and working closely with engineers to shape how agentic AI applications are evaluated and validated at scale.