Databricks Data Engineer

We are seeking a Data Engineer for a contract position (6+ months) based in Malvern, Pennsylvania. In this role, you will design, build, and optimize enterprise-scale data pipelines using Databricks and related cloud technologies. You will orchestrate data extraction from diverse sources, migrate workloads to modern platforms, and provide production support to ensure system reliability and data integrity across financial data ecosystems.

Responsibilities

• Design and implement high-performance, scalable ETL and ELT pipelines using Databricks, Apache Spark, and AWS data services
• Orchestrate data extraction and transformation from legacy and modern data sources
• Migrate data processing jobs from EMR to Databricks
• Develop and maintain CI/CD pipelines for data infrastructure
• Collaborate with business analysts and stakeholders to ensure data availability, quality, and integrity
• Provide expert-level production support and troubleshooting for data platforms
• Create documentation and job aids to support team operations and training
• Implement dimensional data modeling and data lake architectures (Medallion Architecture)

Qualifications

• Required

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline

• 8+ years of enterprise Data Engineering experience

• Strong hands-on Python development experience

• Advanced SQL programming skills

• Extensive experience with Apache Spark and PySpark

• Strong hands-on experience with Databricks

• Hands-on experience with AWS data services: S3, AWS Glue, Athena, Lambda, Step Functions, and EventBridge

• Experience building enterprise ETL and ELT pipelines

• Strong understanding of distributed data processing

• Experience working with large-scale cloud data platforms

• Strong knowledge of dimensional data modeling

• Experience implementing CI/CD pipelines

• Experience with Git and GitHub Actions

• Familiarity with Agile/Scrum methodologies

• Excellent communication and stakeholder management skills

• Preferred

• Financial Services or Asset Management industry experience

• Knowledge of Apache Iceberg, Delta Lake, or Hudi

• Experience with Apache Airflow

• Knowledge of Kafka or Kinesis streaming

• Experience supporting AI/ML and advanced analytics platforms

• AWS Professional or Specialty Certifications