AI Quality Assurance Specialist Sr. Lead

Job Title: AI Quality Assurance Specialist Sr. Lead

Work Place Flexibility: Hybrid

Legal Entity: Entergy Services, LLC

• The preferred location for this role is New Orleans, LA. Other locations within Entergy's service territory may be considered*

JOB SUMMARY/PURPOSE

This role blends traditional software QA skills with advanced evaluation methodologies for modern AI models, including LLMs and ML-driven product features. As a member of the team, you will work closely with experienced engineers and machine learning experts to qualify and refine features powered by vision, language, and cross-modal models. You will be responsible for designing rigorous evaluation strategies for both objective and subjective ML behaviors, creating reliable automated testing pipelines, and developing LLM-driven evaluators that complement human judgement. The ideal candidate is self-directed, creative, and comfortable with ambiguity, with strong technical and interpersonal skills. They have hands-on experience testing ML models directly, defining qualitative scoring rubrics, building reproducible evaluation frameworks, and ensuring that AI behavior is safe, consistent, and aligned with product vision.

JOB DUTIES/RESPONSIBILITIES

• Create detailed, comprehensive, and well-structured test plans and test cases for traditional and ML-based solutions. Estimate, prioritize, plan, and coordinate testing activities.
• Design, develop, and execute automation scripts. Identify, record, document, and track bugs. Perform thorough regression testing when bugs are resolved.
• Develop and maintain robust testing frameworks for machine learning models
• Develop and apply testing processes for new and existing products to meet client needs.
• Monitor debugging process results. Investigate the causes of non-conforming software. Track quality assurance metrics, such as defect densities and open defect counts.
• Collaborate with internal teams (e.g. developers, product managers, and designers) to identify testing requirements.
• Identify and prioritize quality control issues from an end-to-end perspective.
• Participate in scrum ceremonies such as refinement, demo, and release planning.

MINIMUM REQUIREMENTS

Minimum education required of the position: Bachelor's degree in computer science, MIS, business administration, or other technical area of study OR equivalent work experience; advanced degree is a plus.

Minimum experience required of the position:

Specialist I: 0-3 years of professional experience.

Specialist II: At least 2 years of professional experience, at least 1 year of experience testing ML/GenAI solutions.

Specialist III: At Least 4 years of professional experience, at least 2 years of experience testing ML/GenAI solutions.

Specialist, Sr: At least 6 years of professional experience, at least 3 years of experience testing ML/GenAI solutions.

Sr Lead: At least 8 years of professional experience, at least 4 years of experience testing ML/GenAI solutions.

Minimum knowledge, skills and abilities required of the position:

• Strong knowledge of software QA methodologies, tools, and processes.
• Familiarity working with Agile development teams.
• Good communications skills; ability to effectively communicate verbally and in writing with both non-technical and technical staff.
• Proficient in both white box and black box testing methods.
• Ability to work both independently and as part of a team.

Desired knowledge, skills, and abilities:

• Working knowledge of automated testing tools.
• Highly proficient SQL and scripting skills.
• Experience testing AI/ML systems in cloud-based environments.
• Experience integrating automated AI testing into CI/CD pipelines (e.g., GitHub Actions or similar tools).
• Experience with monitoring and logging systems for post-deployment model validation.
• Advanced experience testing generative AI systems, including LLMs for accuracy, bias, safety, and hallucinations.
• Familiarity with RAG evaluation workflows and vector databases (e.g., FAISS, Pinecone, Weaviate).
• Experience with prompt engineering, adversarial prompting, and synthetic data generation.

Any certificates, licenses, etc. required of the position:

• Any ServiceNow, Scrum, or ITIL certifications are a plus.

Primary Location: Texas-Woodlands Texas : The Woodlands || Louisiana : New Orleans || Mississippi : Jackson

Job Function: Corporate

FLSA Status: Professional

Relocation Option:

Union description/code: NON BARGAINING UNIT

Number of Openings: 1

Req ID: 123598

Travel Percentage:Up to 25%

An Equal Opportunity Employer, Minority/Female/Disability/Vets. Please click here to view the EEO page, or see statements below.

EEO Statement: The Entergy System of Companies provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, gender, sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, amnesty, or status as a protected veteran in accordance with applicable federal, state and local laws. The Entergy System of Companies complies with applicable state and local laws governing non-discrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment including, but not limited to, recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.

The Entergy System of Companies expressly prohibits any form of unlawful employee harassment based on race, color, religion, sex, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status. Improper interference with the ability of the Entergy System of Company employees to perform their expected job duties is absolutely not tolerated.

Accessibility: Entergy provides reasonable accommodations for online applicants. Requests for a reasonable accommodation may be made orally or in writing by an applicant, employee, or third party on his or her behalf. If you are an individual with a disability and you are in need of an accommodation for the recruiting process please click here and provide your name, contact number, the accommodation requested and the requisition number that you are requesting the accommodation for. Employee Services will contact you regarding your request.

Additional Responsibilities: As a provider of essential services, Entergy expects its employees to be available to work additional hours, to work in alternate locations, and/or to perform additional duties in connection with storms, outages, emergencies, or other situations as deemed necessary by the company. Exempt employees may not be paid overtime associated with such duties.

Know Your Rights: Workplace Discrimination is Illegal

The non-confidential portions of the affirmative action program for individuals with disabilities and protected veterans shall be available for inspection upon request by any employee or applicant for employment. Please contact HRCompliance@entergy.com to schedule a time to review the affirmative action plan during regular office hours.

WORKING CONDITIONS:

As a provider of essential services, Entergy expects its employees to be available to work additional hours, to work in alternate locations, and/or to perform additional duties in connection with storms, outages, emergencies, or other situations as deemed necessary by the company. Exempt employees may not be paid overtime associated with such duties.

Please note: Authorization to work in the United States is a precondition to employment in this position. Entergy will not sponsor candidates for work visas for this position.