Test Automation Engineer/Lead (Databricks, ETL/Data) with AI/ML Exp.

Test Automation Engineer/Lead (Databricks, ETL/Data) with AI/ML Exp.

8+Years

Louisville, KY / Dallas, TX/ Chicago, IL

Contract

Required Skills:

6+ years of experience in ETL/Data Testing.

3+ years of experience testing Databricks or Spark-based platforms.

Hands-on experience in AI/ML Model Testing or MLOps validation.

Strong coding experience in Python and SQL.

Databricks & Data Engineering Testing

Test Bronze, Silver, and Gold layer data transformations within Databricks environments.

Verify workflow orchestration, scheduling, and dependency management across Databricks jobs and pipelines.

Perform performance, scalability, and reliability validation of Spark and PySpark transformations.

Validate data quality frameworks, rules, and monitoring mechanisms implemented in Databricks.

Ensure data accuracy, completeness, consistency, and lineage across data pipelines.

AI/ML Testing

Validate AI/ML model integrations within enterprise applications and data ecosystems.

Verify model scoring results, prediction outputs, and business rule compliance.

Perform data drift, schema drift, feature drift, and model validation testing.

Validate feature engineering pipelines, training datasets, and inference datasets.

Execute model input/output verification against expected business outcomes and acceptance criteria.

Validate explain ability, traceability, auditability, and regulatory compliance requirements.

Test model deployment workflows using MLflow, Databricks ML, Azure ML, or similar MLOps platforms.

Validate Generative AI and RAG (Retrieval-Augmented Generation) workflows, including prompt accuracy, response quality, grounding, and hallucination detection.

Integration Testing

Perform end-to-end integration testing across:

Databricks

Data warehouses

APIs and microservices

AI/ML platforms

Cloud services

Reporting and analytics systems

Validate data movement and transformations between source systems, data lakes, ML models, and downstream applications.

Perform API testing, contract testing, and interface validation.

Ensure seamless interoperability across enterprise data and AI ecosystems.

Test Automation

Design and develop automation frameworks for:

ETL/Data Pipeline Testing

API Testing

Data Validation

AI/ML Model Validation

Build reusable test utilities, libraries, and accelerators using Python.

Integrate automated test suites into CI/CD pipelines.

Develop automated quality gates and release validation mechanisms.

Generate automated execution reports, dashboards, test metrics, and quality insights.

Support continuous testing and shift-left quality practices across the software delivery lifecycle.

Key Skills

Databricks, Spark, PySpark

Delta Lake, Medallion Architecture (Bronze/Silver/Gold)

Python

SQL

ETL/Data Warehouse Testing

API Testing

AI/ML Testing

MLflow, Azure ML

GenAI/RAG Validation

CI/CD Integration

Test Automation Frameworks

Data Quality and Data Governance

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.