Our client is a fast-growing healthcare technology company building AI voice agents that answer every call for physician practices scheduling appointments, routing patients, and resolving routine requests around the clock. They are looking for a Senior Automation Test Engineer who can plug in and build rapidly, owning the test strategy and automation infrastructure for their voice AI platform and building the frameworks that make non-deterministic AI behavior measurable and regressions obvious.
Responsibilities
Design, build, and maintain automated test infrastructure across unit, integration, end-to-end, and load/performance layers for a real-time voice AI platform
Build evaluation and regression harnesses that score conversation outcomes and intent fulfillment for non-deterministic, LLM-driven agent behavior
Generate synthetic conversation data and de-identified patient scenarios to expand test coverage for rare edge cases
Develop test frameworks and tooling that enable the broader engineering team to write and run reliable tests efficiently
Define and enforce quality standards and testing best practices across engineering teams
Build simulation and regression frameworks for real-time voice and conversational AI scenarios, including heavy rules-based logic layered on top of nondeterministic AI behavior
Own test environments and CI/CD pipeline integration for automated test execution
Investigate and triage production quality issues, driving root cause analysis and prevention
Develop testing approaches that satisfy HIPAA and healthcare compliance requirements
Ramp quickly: within the first 30 days, understand the tech stack and business, and begin shaping a test suite strategy for the platform
Skills/Requirements
10+ years in software quality engineering with a strong focus on test automation
Proven experience designing and owning test automation frameworks from the ground up
Proficiency in Python, TypeScript, or equivalent, with strong scripting and tooling skills
Experience with CI/CD integration (GitHub Actions, Jenkins, or equivalent)
Hands-on AI evaluation experience testing AI-powered applications, including prompt and model regression testing
Demonstrated ability to test non-deterministic conversational flows, designing assertions and evaluation criteria that tolerate variation without masking real regressions
Experience generating synthetic data at scale realistic conversation transcripts, edge-case inputs, and de-identified patient scenarios
Ability to move quickly and independently in a fast-scaling environment with minimal ramp-up time
Nice to have: experience with real-time systems, telephony, or voice platforms; LLM evaluation tooling and prompt/model versioning; evaluating speech-to-text or text-to-speech quality; load/performance testing for high-throughput systems; HIPAA-regulated environment experience; contract testing or service virtualization
Numbers & Facts
Location
Birmingham, AL (Remote)
Salary
$125,000–$150,000 Per Year
Skills
Artificial Intelligence (AI)unmatched
Artificial Intelligence (AI) Agentsunmatched
Best Practicesunmatched
Calendar Managementunmatched
Continuous Deployment/Deliveryunmatched
Continuous Integrationunmatched
Design Flowsunmatched
Develop Test Toolsunmatched
GitHubunmatched
HIPAA (Health Insurance Portability and Accountability Act)unmatched
Healthcareunmatched
High Throughputunmatched
Hospital/Healthcare Constructionunmatched
Jenkinsunmatched
Load Testingunmatched
Machine Toolunmatched
Patient Assessmentunmatched
Performance Testingunmatched
Problem Solving Skillsunmatched
Python Programming/Scripting Languageunmatched
Quality Assurance Methodologyunmatched
Quality Engineeringunmatched
Quality Metricsunmatched
Regression Testingunmatched
Regulatory Complianceunmatched
Root Cause Analysisunmatched
Scripting (Scripting Languages)unmatched
Software Engineeringunmatched
Telephone Skillsunmatched
Telephonyunmatched
Test Automationunmatched
Test Designunmatched
Test Plan/Scheduleunmatched
Test Scenariounmatched
Test Strategyunmatched
Test Suiteunmatched
Test Toolsunmatched
Testingunmatched
Tolerance Analysisunmatched
Unit Testunmatched
Virtualizationunmatched
Web Client Plug-insunmatched
🎯
Be found by employers
5,500+ employers search our resume database daily. Add yours to get found by recruiters looking for candidates like you.
Level up your application
Professional resume templates
Browse dozens of recruiter approved resume templates, layouts and formats. Choose your favorite and make it your own in minutes.