Senior Data Engineer

Senior Data Engineer

10+Years

Chicago, IL

Contract

Summary:

We are seeking highly experienced Senior Data Engineers with strong expertise in building scalable cloud-native data platforms and enterprise data solutions. The ideal candidate will have extensive experience designing and developing end-to-end ETL/ELT pipelines using Python, PySpark, SQL, Databricks, Azure Data Factory, Infrastructure as Code (Pulumi) and Azure cloud technologies to support large-scale data processing and , and production-grade data pipelines and platforms.

Key Responsibilities

• Design, build, and support end-to-end data pipelines (ingestion, transformation, validation, publishing).
• Develop and optimize SQL, PySpark, Databricks transformations for large datasets.
• Build production-grade Python/Pyspark components (reusable modules, logging, error handling, testing).
• Create and maintain Azure Data Factory (ADF) pipelines (triggers, parameterization, monitoring, failure handling).
• Work within Azure environments (ADLS Gen2, Azure SQL, resource groups, portal operations).
• Provision and maintain Azure components using Pulumi (Infrastructure as Code).
• Participate in code reviews, documentation, and operational support (triage + root cause analysis).

Must-have Skills (Required)

• ETL / ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting.
• SQL: Advanced querying (CTEs, views, joins, complex query logic) and performance tuning for transformations and validation.
• Python: Production-quality development (modular code, testing, logging, integration with APIs/files, CICD, Unit Test/Integration test automation, Code Coverage).
• PySpark: Distributed transformations and performance optimization (joins, partitions, debugging), CICD, Unit Test/Integration test automation, Code Coverage.
• Azure Data Factory (ADF): Build/operate ADF pipelines, parameterization, triggers, monitoring, retry/error handling; integrate with Databricks/ADLS.
• Databricks: Develop and operationalize notebooks/jobs/workflows; Delta Lake patterns; basic cluster/job configuration. New features - Genie Spaces, Delta Share, Unity Catalog, AI functions, Agent bricks and advanced optimization techniques.
• Azure Fundamentals + Pulumi: Hands-on with ADLS Gen2, Azure Portal, Storage Explorer, Resource Groups, Azure SQL, Azure Batch and familiarity integrating with Azure OpenAI. Able to use/maintain Pulumi scripts for provisioning and managing Azure resources across environments.
• Experience with Azure API Services and Azure Function Apps using Python.

Nice-to-have Skills

• R: Ability to support/translate validation rules with SQL scripts and create data quality reports.
• TypeScript: Useful for pulumi pipeline to create Azure components.
• Java: Useful for integration with existing services/components.
• .NET: Useful for integration with existing services/components.
• Angular / Spring Boot: Minor troubleshooting or coordination with app teams.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.