Data Analyst
hybrid in Scottsdale, AZ
i need to send 2 more ppl - they need to have sql, python and visual studio
Job Description
The recently shared job description focuses on
deterministic testing, data validation, and testing scenarios pertains to these additional openings.
Role Overview
The primary focus of these positions is validating AI-generated responses and ensuring data accuracy. Responsibilities include:
- Reviewing questions submitted by users.
- Tracing the underlying code and data sources used to generate responses.
- Verifying the accuracy of the logic, code, and resulting data.
- Testing and validating outputs to ensure quality and reliability.
The Good candidates should possess:- Advanced SQL expertise (required).
- Robust Python development skills (required).
- Experience with Visual Studio Code (VS Code) and related development environments.
- Robust analytical and critical-thinking abilities.
- The ability to understand user intent, not just the literal wording of a request, when validating responses.
Key Responsibilities 1. Deterministic Testing & Data Validation
Validate generative AI tool outputs for structured, rules-based use cases by
reconciling results against trusted data sources and established SQL-based
metrics
Ensure consistency, explainability, and auditability of outputs by confirming
alignment with existing data pipelines and query logic
Expand and maintain test coverage across prioritized use cases to establish a
Robust , high-confidence baseline for the platform
Partner with data engineering and analytics teams to identify and resolve
discrepancies in underlying data or logic
2. Non-Deterministic Testing & Scenario Evaluation
Design and execute scenario-based testing for more complex, AI-driven outputs
where direct validation is not always possible
Evaluate results based on intent accuracy, reasonableness, and confidence
thresholds rather than exact match validation
Prioritize testing across higher-risk and high-impact use cases using curated
question sets and real-world scenarios
Identify patterns in output variability and drive iterative refinement to improve
reliability and user trust
3. Human-in-the-Loop Review & Continuous Monitoring
Conduct ongoing review of generative AI tool interactions post-launch, validating
outputs and ensuring quality across all user scenarios
Identify edge cases, inconsistencies, and emerging risks, and escalate findings
to product and engineering teams
Synthesize insights from testing and live usage to inform enhancements, training
data improvements, and governance practices
Serve as an accountable reviewer, providing a critical control point for
responsible AI deployment and continuous improvement
Required Skills & Experience
Robust SQL skills required.
Robust analytical background with experience in data validation, SQL, and
analytics workflows
Ability to assess outputs both quantitatively (data accuracy) and qualitatively
(reasonableness, business context)
Demonstrated critical thinking and sound judgment, especially in ambiguous or
non-deterministic environments
Experience working with large datasets, reporting tools, or analytics platforms
Preferred Qualifications
Exposure to AI/ML or generative AI tools and associated testing or validation
frameworks
Experience in scenario-based testing, UAT, or model validation
Familiarity with financial services, retirement, or plan sponsor analytics