Sr. Data Engineer

• Must be authorized to work in the U.S. without the need for employment-based visa sponsorship now or in the future. We are unable to sponsor applicants for U.S. work visa status for this opportunity (no sponsorship is available for H-1B, L-1, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT or any other employment-based visa).

About the Role

We are seeking a highly skilled Senior Data Engineer to join our Data Engineering & Platform organization. This role is ideal for someone passionate about building scalable data solutions, enabling cross-functional data delivery, and raising engineering excellence across the organization. You will work on an advanced Databricks-on-AWS data platform and play a key role in transforming our data ecosystem-designing and building resilient pipelines, implementing CI/CD, and contributing to platform modernization efforts.

As a senior member of the team, you will partner with data engineers, analytics teams, product teams, and platform engineering to build high-quality, production-grade data solutions that power enterprise analytics, data science, and operational workloads.

Key Responsibilities:

Data Engineering & Pipeline Development

• Design, build, maintain, and optimize scalable data pipelines and ELT/ETL processes to ingest, process, and store large volumes of data from various sources.
• Architect robust batch and streaming solutions using PySpark, Spark SQL, and Databricks Jobs.
• Monitor and troubleshoot data pipelines and infrastructure to ensure high availability and performance.
• Ensure reliability, observability, and SLAs through monitoring, alerting, and automated recovery patterns.
• Implement and maintain data integration solutions, including APIs and data connectors, to facilitate data exchange between systems.

Cloud Platform Engineering

• Build and manage data solutions on AWS, leveraging services such as S3, Lambda, Glue, IAM, EC2, EKS, Step Functions, and EventBridge.
• Apply cloud engineering best practices for cost optimization, reliability, and security.

CI/CD, DevOps, and Infrastructure-as-Code

• Implement and maintain CI/CD pipelines using:

• Databricks Asset Bundles

• Terraform (IaC)

• GitHub Actions / Azure DevOps / Jenkins (depending on internal stack)

• Own deployment automation and environment management for data workloads and platform components.

• Contribute to reusable Terraform modules and engineering standards.

Data Governance and Data Quality

• Maintain data quality through validation frameworks (e.g., DQX, Great Expectations, Databricks expectations, custom frameworks).
• Ensure data quality, integrity, and security by implementing best practices for data governance and management.
• Implement governance and lineage through Unity Catalog, cataloging standards, and metadata best practices.

Platform Modernization

• Contribute to initiatives such as:
• Migration of pipelines from legacy integration tools to Lakeflow / Lakehouse-native ingestion
• Improving infrastructure automation
• Enhancing data observability, testing, and deployment workflows

Collaboration & Technical Leadership

• Serve as a technical mentor to junior engineers.
• Partner closely with data analysts, data scientists, and business stakeholders to deliver enterprise data assets.
• Contribute to architecture decisions, design reviews, and platform roadmap conversations.

Required Qualifications

• 7+ years of data engineering experience building production-grade data pipelines.
• Deep expertise in Databricks, Spark, and Delta Lake.
• Hands-on experience with AWS cloud services.
• Strong proficiency in Python, SQL, and distributed computing patterns.
• Experience with Terraform and cloud-native CI/CD.
• Strong understanding of data modeling, orchestration, and data quality frameworks.
• Ability to operate autonomously while collaborating in a team environment.

Additional Qualifications

• Experience with Lakeflow, Databricks Workflows, or ingestion modernization initiatives.

• Knowledge of software engineering best practices:

• test automation

• versioning

• code reviews

• Git workflows

• Familiarity with event-driven architecture and streaming technologies.

• Experience supporting Data Science and ML platform teams.

• P&C Insurance domain knowledge preferred

#LI-RL1

IND1

Click here for some insight into our culture!

The Baldwin Group will not accept unsolicited resumes from any source other than directly from a candidate who applies on our career site. Any unsolicited resumes sent to The Baldwin Group, including unsolicited resumes sent via any source from an Agency, will not be considered and are not subject to any fees for any placement resulting from the receipt of an unsolicited resume.