Role: Product Analyst
Location - Phoenix, AZ is preferred but remote can be considered
Duration - 6+ months
Skills -
Required Qualifications
3+ years experience drafting detailed Jira user stories, maintaining product backlogs, and defining acceptance criteria based on business requirements in an Agile product team or equivalent environment.
2+ years experience in data analysis with the ability to interpret AI performance metrics and translate findings into actionable product recommendations.
1+ year experience working with LLM-based applications, including understanding of hallucination patterns, output quality issues, grounding techniques, bias detection, and mitigation strategies.
1+ year hands-on experience executing structured AI agent evaluation plans, tracking iterations, maintaining golden datasets, and performing regression testing.
1+ year experience supporting the development, evaluation, and continuous improvement of AI-powered products, AI agents, copilots, or generative AI solutions.
Experience translating AI product requirements into prompts, agent workflows, business rules, and measurable success criteria.
Experience defining AI product success metrics and measuring improvements in output quality, user satisfaction, task completion, or productivity.
Experience partnering with engineering, UX, data science, and business stakeholders to prioritize, evaluate, and refine AI product capabilities.
Preferred Qualifications
Experience in HR technology, learning technology, content creation platforms, or knowledge management solutions as a Business Analyst, Product Analyst, Product Owner, or equivalent role.
Hands-on experience building and executing specialized AI test cases, evaluating LLM outputs, prompt variations, and agentic workflows.
Hands-on experience with prompt engineering, prompt optimization, and evaluation of AI-generated content quality, accuracy, completeness, and consistency.
Experience designing evaluation rubrics, benchmark datasets, golden datasets, and quality measurement frameworks for AI-generated outputs.
Experience conducting side-by-side model comparisons, experimentation, or A/B testing to improve AI product performance.
Familiarity with Retrieval Augmented Generation (RAG), grounding strategies, hallucination mitigation, and content sourcing concepts.
Familiarity with automated AI evaluation tools, frameworks, and testing methodologies.
Experience with human-in-the-loop workflows and designing review, validation, and approval processes for AI-generated outputs.
Experience working with AI governance, responsible AI practices, bias detection, content safety, and model quality evaluation.
Experience evaluating content quality against defined business, compliance, instructional design, or organizational standards.
Strong written and verbal communication skills, with the ability to document evaluation methodologies, findings, recommendations, and trade-offs for both technical and business audiences.
Numbers & Facts
Location
Pheonix, AZ
Skills
A/B Testingunmatched
Agile Programming Methodologiesunmatched
Analysis Skillsunmatched
Artificial Intelligence (AI)unmatched
Artificial Intelligence (AI) Agentsunmatched
Atlassian JIRAunmatched
Benchmarkingunmatched
Business Analysisunmatched
Communication Skillsunmatched
Content Developmentunmatched
Content Managementunmatched
Continuous Improvementunmatched
Customer Satisfactionunmatched
Data Analysisunmatched
Data Scienceunmatched
Data Setsunmatched
Establish Prioritiesunmatched
Improvement Metricsunmatched
Instructional Designunmatched
Knowledge Managementunmatched
Organizational Development/Managementunmatched
Performance Managementunmatched
Performance Metricsunmatched
Presentation/Verbal Skillsunmatched
Product Backlogunmatched
Productivity Managementunmatched
Quality Assurance Methodologyunmatched
Quality Managementunmatched
Quality Metricsunmatched
Regression Testingunmatched
Safety/Work Safetyunmatched
Test Caseunmatched
Testingunmatched
User Interface/Experience (UI/UX)unmatched
Writing Skillsunmatched
🎯
Be found by employers
5,500+ employers search our resume database daily. Add yours to get found by recruiters looking for candidates like you.
Level up your application
Professional resume templates
Browse dozens of recruiter approved resume templates, layouts and formats. Choose your favorite and make it your own in minutes.