Product Analyst, AI Solutions

Description

Product Analyst, AI Solutions

The Product Analyst, AI Solutions is responsible for shaping, optimizing, and maintaining advanced AI-driven features and user experiences within our product suite. This role sits at the intersection of business analysis, AI tuning, and user experience-combining expertise in software requirements gathering, prompt engineering, and AI model evaluation. The Analyst is heavily focused on refining how AI systems behave in real-world scenarios, ensuring outputs are accurate, trustworthy, and closely aligned with client needs.

You partner with AI Engineers, Product Managers, UX/UI Designers, QA, and Subject Matter Experts (SMEs) to extract user requirements, translate those into actionable User Stories and evaluation metrics, and continuously tune AI systems for best-in-class performance and usability.

This role is pivotal in ensuring our AI solutions provide maximum business value, aligning advanced technology with end-user needs while driving continuous improvement and reliability. The Analyst produces high-quality deliverables using systematic processes, with impact across multiple projects, deadlines, and teams.

Principal Duties

AI Tuning & Output Quality

• Design, evaluate, and refine prompts, templates, and conversation flows for reliability, cost effectiveness, and user alignment.
• Maintain prompt libraries, track changes, and manage versioning.
• Systematically test AI model outputs for accuracy, consistency, compliance, and tone, applying rerankers, guardrails, and validation layers to ensure business rules are met.
• Collaborate with engineering and QA to promptly integrate feedback and resolve model drift or new scenarios.

Requirements Gathering & Documentation

• Proactively gather requirements of moderate scope and complexity from diverse sources (clients, market analysis, SMEs) to drive product development and AI enhancements.
• Document software and system requirements, including user stories, program functions, test cases, and steps required for new features or modifications.
• Ensure all documentation supports future maintainability and growth of AI-enabled features.

Evaluation, Metrics, & Continuous Improvement

• Establish systematic evaluation frameworks for AI model outputs, leveraging quantitative and qualitative benchmarks.
• Run experiments, A/B tests, and structured evaluations to compare tuning strategies and track performance metrics (accuracy, coverage, hallucination rates, latency).
• Translate business and user feedback into actionable improvements, owning continuous iteration for released solutions.

Collaboration & UX Alignment

• Act as a bridge between technical and non-technical stakeholders, ensuring clear, effective communication and alignment.
• Work closely with product, development, and UX/UI teams to ensure AI-driven features fit seamlessly into business workflows, are intuitive and user-friendly, and meet client needs.
• Help shape the overall user experience for AI interactions, promoting explainability and transparency.

Education, Experience and Special Skills

• Bachelor's degree in Computer Science, Management Information Science, Business, or related field OR equivalent work experience.
• 1+ years' experience in software, AI/ML, or related product analysis roles.
• Experience with AI/ML applications (LLMs, NLP, machine learning pipelines).
• Strong understanding of prompt engineering and generative AI interaction design.
• Basic knowledge of data modeling, relational database concepts, and SQL.
• Familiarity with model evaluation techniques and performance metrics.
• Proficiency with Python or JavaScript for prototyping and evaluation.
• Experience with Agile SCRUM development processes.
• Excellent planning, organization, and interpersonal communication skills.
• Ability to work independently and collaboratively across functional groups.
• Experience in ERP systems, enterprise software, or complex business domains.
• Familiarity with RAG (retrieval-augmented generation), embeddings, MLOps, and responsible AI practices.
• Experience with fine-tuning methods (LoRA, QLoRA, PEFT), running structured evaluation series, or human-in-the-loop systems.