Databricks/AwS Engineer

Databricks Python Developer

Experience: 7+ Years

Location: Bloomfield, CT (Remote / Hybrid)

Contract

Job Summary

We are seeking a Senior Databricks Developer with strong expertise in Databricks, Python, and cloud-based data engineering, preferably within the healthcare domain. The ideal candidate will have extensive experience building scalable data pipelines, developing cloud-native analytics solutions, and working in Agile delivery environments. Experience with AWS and healthcare data platforms is highly desirable.

Key Responsibilities

• Design, develop, and optimize large-scale data processing solutions using Databricks.
• Build and maintain robust ETL/ELT pipelines using Python and Databricks.
• Develop scalable data ingestion, transformation, and integration frameworks from multiple data sources.
• Leverage Databricks features including Delta Lake, Spark SQL, Unity Catalog, Workflows, and ML capabilities.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Work with AWS services to support cloud-based data platforms and analytics workloads.
• Implement data quality checks, monitoring, performance tuning, and operational support processes.
• Participate in architecture discussions and contribute to best practices for data engineering and cloud adoption.
• Ensure compliance with security, governance, and data management standards.
• Actively participate in Agile ceremonies including sprint planning, backlog grooming, daily standups, sprint reviews, and retrospectives.

Required Skills & Qualifications

Mandatory Skills

• 7+ years of overall IT experience in Data Engineering and Analytics.
• Strong hands-on experience with Databricks development and administration.
• Strong Python programming skills for data processing, automation, and analytics.
• Experience with Apache Spark (PySpark) and distributed data processing.
• Strong experience in designing and developing ETL/ELT pipelines.
• Experience working with structured and unstructured data.
• Hands-on experience with Git version control and CI/CD practices.
• Strong understanding of data modeling, data warehousing, and performance optimization.
• Mandatory experience working in Agile/Scrum delivery models.
• Excellent analytical, problem-solving, and communication skills.

Preferred Skills

• Working knowledge of AWS services such as:

o S3

o Glue

o Lambda

o EC2

o IAM

o CloudWatch

o Redshift (preferred)

• Experience with healthcare data ecosystems.
• Knowledge of FHIR, HL7, Claims, Membership, Provider, or Clinical data domains is an advantage.
• Exposure to data governance, data quality, and security frameworks.

Preferred Experience

• Healthcare/Payer/Provider industry experience is a strong advantage.
• Experience supporting enterprise-scale cloud data platforms.
• Exposure to DevOps and DataOps practices.
• Experience mentoring junior developers and leading technical workstreams.

Education

• Bachelor''s or Master''s degree in Computer Science, Information Technology, Engineering, or related field.

Nice to Have

• Databricks Certifications.
• AWS Cloud Certifications.
• Healthcare domain certifications or relevant project experience.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.