Senior Data Engineer

Position Description

We are seeking a Senior Data Engineer to lead the design and implementation of end-to-end data solutions within a modern Azure-based cloud environment. This hands-on, technical role will partner closely with data scientists, analysts, and business stakeholders to ensure that data is accurate, accessible, and optimized for analytics, reporting, and operational needs. You will play a key role in building Data Lakehouse, scalable data pipelines and integrations while championing best practices, mentoring junior engineers, and leading strategic data initiatives across teams.

Key Responsibilities

• Architect and implement secure, scalable data pipelines using Azure Data Factory, Azure Functions, and Azure Data Lake Storage
• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks (PySpark/Spark SQL) to ingest, transform, and process large volumes of structured and unstructured data
• Build and manage data pipelines using Azure services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure Event Hubs, and Azure SQL Database
• Implement data quality, validation, and monitoring frameworks to ensure reliability and accuracy of data pipelines
• Collaborate cross-functionally with data scientists, analysts, and business stakeholders to understand data requirements and deliver fit-for-purpose data solutions
• Lead modernization and optimization efforts, improving pipeline performance, maintainability, and scalability
• Support CI/CD practices for data pipelines using tools such as Azure DevOps, Git, and Databricks Repos
• Detect and resolve data quality issues; implement automated audits and monitoring processes
• Troubleshoot and resolve production data pipeline issues, ensuring high availability and minimal downtime
• Act as a technical leader on schema design, performance tuning, and Azure data architecture
• Mentor and support junior engineers across data engineering, analytics, and BI teams
• Participate in and lead code reviews, promoting clean, well-documented, and testable code
• Stay current on trends in data engineering and cloud technologies, identifying opportunities to innovate

Minimum Requirements

• 8+ years of hands-on experience in data engineering, including designing and implementing enterprise-scale data solutions.
• 2+ years of experience developing and operating Azure cloud-native data platforms.

Critical Skills

• Expertise in MS SQL Server, Python (pandas, PySpark), Azure Data Factory, Azure Functions and Azure Data Lake Storage.
• Strong expertise with Azure Databricks, including Spark (PySpark/Scala), Delta Lake, and cluster/job optimization.
• Solid understanding and hands-on experience building Data Lakehouse architecture
• Experience working with a variety of file formats (e.g., CSV, JSON, XML, Parquet).
• Experience with version control (Git) and CI/CD pipelines for data engineering workflows
• Familiarity using REST APIs for data extraction and integration.
• Proven experience designing and implementing data solutions.
• Strong understanding of cloud architecture, data warehousing and modern data stack components.

Additional Skills & Qualifications

• Demonstrated ability to perform root cause analysis on data and processing issues
• Strong problem-solving skills with the ability to explain technical concepts to non-technical audiences
• A successful history of manipulating, processing and extracting value from large disparate datasets
• Experience with Big Data technologies such as Databricks, Spark, or Azure Synapse
• Knowledge of CI/CD workflows, version control, and agile development practices
• Familiarity with data governance, privacy, and compliance frameworks
• Experience with data warehousing, analytics tools, and BI platforms
• Familiarity with streaming data technologies (Azure Event Hubs, Kafka, Structured Streaming)

Education

• 4-year degree in computer science, engineering or other related IT field of study, or equivalent professional work experience

Physical Requirements

• General office demands
• Prolonged periods of sitting at a desk and working on a computer.
• Frequent reaching, handling, and fine manipulation for using office equipment, filing, and managing paperwork.
• Manual dexterity sufficient to operate a keyboard, mouse, and other office tools.
• Occasional standing, walking, and bending.
• Ability to lift up to 10-20 pounds occasionally.
• Vision abilities required include close vision for computer work and reading documents.
• Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.