Lead AI Engineer – Agentic Test Automation

JOB DESCRIPTION

1) Agentic test automation foundation (reusable patterns + reference implementations)

Design and implement agentic testing patterns that can be adopted by multiple Underwriting teams (and later other domains).

Create reference implementations (sample repos / templates) demonstrating:

o Test generation assistance (from requirements, APIs, contracts, schemas)

o Test maintenance assistance (auto-updating selectors/contracts, flaky test triage)

o Failure analysis assistance (root cause suggestions, log correlation, defect drafting)

Establish a standard architecture for test code organization, tagging, data management, and execution across UI + API + service layers.

2) Coverage standards, templates, and governance

• Define and publish coverage standards (what "good" looks like) including:

o Minimum coverage expectations by service/component

o Test type mix (unit vs API vs UI vs contract vs integration)

o Risk-based prioritization and traceability to requirements

• Provide templates usable across teams:

o Test plan templates

o Test case/spec templates (Gherkin-style or equivalent)

o Definition of Ready / Definition of Done quality checklists

• Create a scalable tagging/metadata strategy (e.g., feature, service, risk, priority, data sensitivity) to support reporting and quality gates.

3) GenAI-assisted reporting and quality insights across microservices

• Build automated reporting that aggregates test + service data across multiple microservices, such as:

o Test execution results (Karate/Playwright + CI runs)

o Service health signals (logs/metrics/traces if available)

o Defect signals (issue tracker metadata if available)

• Generate GenAI-driven summaries:

o Release readiness narratives

o Failure clustering and trend analysis

o "What changed?" insights (commit/PR correlation)

• Produce outputs consumable by engineering leadership and teams (dashboards, markdown summaries in PRs, artifacts in CI).

4) "Quality gates" via agents

• Build automated review agents that evaluate user stories/requirements for minimum required clarity and data before development/testing starts:

o Required fields present (acceptance criteria, testable outcomes, data needs, dependencies)

o Ambiguity detection and missing edge cases

o Data/privacy considerations and environment needs

• Integrate gates into workflow (PR checks, issue templates, GitHub Actions) to reduce churn and rework.

Required Technical Skills (must-have)

GenAI / LLM + agentic development

• Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning, guardrails).
• Experience with prompting patterns, structured outputs (JSON schemas), evaluation, and reducing hallucinations.
• Ability to design agent workflows for:

o Test generation/augmentation

o Requirements review and completeness validation

o Report generation and summarization

GitHub platform + GHCP (Copilot) for engineering workflows

• Strong proficiency with GitHub Copilot in day-to-day development.
• Deep experience with GitHub platform capabilities:

o GitHub Actions (CI/CD pipelines, reusable workflows, composite actions)

o PR checks, branch protections, CODEOWNERS, templates

• Automation via GitHub APIs/webhooks (as needed)

Test automation engineering (framework expertise)

• Advanced experience designing and implementing automation with:

o Karate (API testing, contract-like checks, data-driven testing, mocks)

o Playwright (UI automation, selectors strategy, parallelization, trace/video artifacts)

• Strong understanding of test design and coverage:

o Happy path scenarios

o Negative/validation scenarios

o Edge/boundary scenarios

o Data setup/teardown strategies and test isolation

Cross-service reporting and data aggregation

• Proven ability to aggregate and normalize results from multiple microservices and multiple pipelines.
• Experience producing actionable automated reports (trend analysis, failure clustering, service correlation).

Automated requirements review agents

• Experience implementing automated checks that validate:

o Acceptance criteria completeness

o Required test data and environment dependencies

o Non-functional requirements (performance, security, observability) when applicable