Senior Data QA Engineer – ETL, Databricks & AI/ML

Location: Louisville, KY / Dallas, TX
Experience: 8–10 Years

Role Overview

Seeking a Senior Data QA / Automation Engineer with strong experience in ETL/Data Testing, Databricks, Spark/PySpark, Python, SQL, and AI/ML validation. The ideal candidate will be responsible for validating enterprise data pipelines, Databricks platforms, ML workflows, integrations, and automated quality processes.

Key Responsibilities

• Perform end-to-end ETL/ELT, data quality, and Source-to-Target validation.

• Validate data transformations, aggregations, business rules, lineage, schemas, metadata, and Delta Lake.

• Test CDC, incremental loads, batch and streaming pipelines.

• Validate Databricks Notebooks, Workflows, Jobs, Delta Live Tables, Unity Catalog, and Bronze/Silver/Gold data layers.

• Perform Spark/PySpark transformation and data quality testing.

• Validate AI/ML models, predictions, feature engineering, model inputs/outputs, and MLOps workflows.

• Test data drift, schema drift, model deployment, MLflow/Databricks ML/Azure ML integrations, and GenAI/RAG workflows where applicable.

• Perform API, integration, contract, and end-to-end data flow testing.

• Develop automated testing frameworks and reusable utilities using Python and SQL.

• Integrate automated tests and quality gates into CI/CD pipelines.

• Generate test reports, metrics, and quality validation results.

Required Skills

• 8–10 years of QA/Automation Testing experience.

• 6+ years of ETL/Data Testing experience.

• 3+ years of Databricks/Spark/PySpark testing.

• Strong Python and SQL programming skills.

• Hands-on AI/ML Model Testing or MLOps validation experience.

• Strong experience with data warehouses, data lakes, APIs, and cloud data platforms.

• Experience building and maintaining test automation frameworks.

• Strong understanding of data quality, reconciliation, validation, and integration testing.

Preferred

• Experience with MLflow, Azure ML, Delta Lake, Unity Catalog, CI/CD, and GenAI/RAG testing.

• Experience testing large-scale structured and semi-structured datasets.

• Strong understanding of enterprise data architecture and data pipelines.