Sr. Azure Data Engineer with Databricks

Job Description Data Engineer (Onshore)
Location: Philadelphia, PA (Hybrid: 3 Days Per Week Onsite)
Experience Required: 7+ Years

Position Overview
We are seeking an experienced Data Engineer to support enterprise data modernization and analytics initiatives. This individual will be responsible for designing, developing, and optimizing scalable cloud-based data solutions leveraging Azure, Databricks, and modern Data Lake architectures.
The ideal candidate will have strong hands-on experience building enterprise-grade data pipelines, implementing data transformation frameworks, and supporting large-scale analytics platforms. This role requires close collaboration with business stakeholders, data architects, analysts, and product teams to enable data-driven decision-making across the organization.
This is a hybrid role requiring onsite presence in the Philadelphia office at least three days per week.

Key Responsibilities
Data Engineering & Development
Design, develop, and maintain scalable ETL/ELT pipelines supporting enterprise data and analytics initiatives.
Build data ingestion, transformation, and orchestration frameworks using Azure-native technologies.
Develop and optimize data processing solutions using Databricks and PySpark.
Implement and maintain Delta Lake and Lakehouse architectures.
Ensure data quality, accuracy, lineage, and reliability across all data assets.
Perform root cause analysis and troubleshoot data-related issues.
Azure Cloud Data Platform

Develop solutions leveraging:

Azure Databricks

Azure Data Lake Storage (ADLS Gen2)

Azure Data Factory (ADF)

Azure Synapse Analytics

Azure SQL Database

Support data migration, modernization, and cloud transformation efforts.

Optimize performance, scalability, and cost efficiency of cloud-based data platforms.
Data Modeling & Analytics Enablement

Build and maintain logical and physical data models.

Create curated data layers to support business intelligence, reporting, and advanced analytics workloads.

Partner with business users and data consumers to understand requirements and translate them into scalable technical solutions.
DevOps & Operational Excellence

Implement CI/CD processes for data engineering solutions.

Support automated deployment and release management processes.

Develop monitoring, ing, and observability capabilities for data pipelines.

Follow enterprise data governance and security standards.
Agile Delivery

Participate in Agile/Scrum ceremonies including sprint planning, daily stand-ups, retrospectives, and backlog refinement.

Collaborate effectively with Product Owners, Architects, QA teams, and business stakeholders.

Contribute to continuous improvement initiatives and engineering best practices.

Required Qualifications
Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
7+ years of experience in Data Engineering.
5+ years of hands-on experience with Azure data technologies.
2+ years of experience in Healthcare, Managed Care, Health Plans, or Payer organizations.

Strong expertise in:
Azure Databricks
Azure Data Lake Storage (ADLS)
Azure Data Factory (ADF)
PySpark
Python
SQL
Extensive experience building enterprise-scale data pipelines and data integration solutions.
Strong understanding of Data Warehousing, Data Lake, and Lakehouse concepts
Experience working with large volumes of structured and unstructured data.
Experience with Git, CI/CD, and DevOps practices.
Strong analytical, troubleshooting, and problem-solving skills.

Preferred Qualifications
Experience working with:
Claims Data
Membership Data
Provider Data
Care Management Data
Experience with Power BI or other reporting tools.
Familiarity with FHIR, HL7, or healthcare interoperability standards.
Experience with Azure DevOps and Infrastructure as Code (IaC).
Exposure to AI/ML data pipelines and advanced analytics environments.

Required Technical Skills
Must Have
Azure Databricks
Azure Data Lake (ADLS Gen2)
Azure Data Factory (ADF)
PySpark
Python
SQL
Delta Lake
Data Modeling
ETL/ELT Development
Azure Cloud Platform
Git
CI/CD
Agile/Scrum

Azure Databricks

Azure Data Lake Storage (ADLS)

Azure Data Factory (ADF)

PySpark

Python

SQL