Job Title: Senior DataOps Engineer (Contractor) Azure Databricks
Remote
Role Summary
We are hiring a Senior DataOps Engineer (Contractor) to drive enterprise DataOps capabilities, ensuring reliable, scalable, and production-ready data pipelines on Azure Databricks platforms. This role focuses on pipeline orchestration, CI/CD automation, monitoring, and data platform operations to support analytics and AI/ML workloads.
Key Responsibilities
Build and manage end-to-end data pipelines using Azure Databricks (PySpark, SQL)
Implement workflow orchestration (Airflow / ADF / Databricks Workflows)
Establish DataOps practices: monitoring, alerting, and production support
Drive CI/CD pipelines and environment promotion (Dev $B"*(B Prod)
Ensure data quality, lineage, and governance compliance
Manage incident resolution, RCA, and SLA adherence
Optimize pipeline performance, scalability, and cost
DataOps function ensures consistent, reliable data delivery and operational stability across platforms
Required Skills
Azure Databricks (Mandatory) PySpark, SQL
Azure Services ADF, ADLS, Synapse
Data Orchestration Airflow / ADF / Databricks Workflows
CI/CD Tools Azure DevOps / GitHub / Git
Strong expertise in ETL/ELT pipelines, DataOps, and production support
Experience with data quality, monitoring, and observability frameworks
Preferred Skills
Experience with Snowflake / BigQuery / multi-cloud environments
Exposure to MLOps / AI data pipelines
Knowledge of Medallion Architecture (Bronze Silver Gold)
Domain Experience (Good to Have)
US Healthcare domain knowledge:
o Claims, Member, Provider, Eligibility data
o Familiarity with FHIR / HL7 / EMR data models