Lead Data Engineer

Argyllinfotech

NULL, IL(remote)

Apply

JOB DETAILS

SKILLS

Agile Programming Methodologies, Amazon Simple Storage Service (S3), Amazon Web Services (AWS), Analysis Skills, Apache Spark, Artificial Intelligence (AI), Automation, Best Practices, Business Skills, Cisco Unity, Cloud Architecture, Cloud Computing, Cloud Storage, Communication Skills, Computer Programming, Consulting, Continuous Deployment/Delivery, Continuous Integration, Data Analysis, Data Management, Data Modeling, Data Processing, Data Quality, Data Warehousing, Database Extract Transform and Load (ETL), DevOps, Distributed Computing, Ecosystems, Engineering, Enterprise Architecture, Git, GitHub, Identify Issues, Information Technology & Information Systems, Leadership, Machine Learning, Mentoring, Metadata, Microsoft Windows Azure, Multiplatform/Cross-Platform, Network Architecture/Engineering, Organizational Skills, Performance Analysis, Performance Tuning/Optimization, Presentation/Verbal Skills, Problem Solving Skills, Python Programming/Scripting Language, Requirements Management, SQL (Structured Query Language), Scalable System Development, Software Development, Team Lead/Manager, Team Player, Technical Leadership, Time Management

LOCATION

NULL, IL

POSTED

2 days ago

Job Title: Lead Data Engineer (AWS | Databricks | Python | Spark)

Location: 100% Remote (Preferred: Chicago, IL / Chicagoland Area)
Client Location: Chicago, IL
Duration: 6+ Months Contract (Contract-to-Hire)
Start Date: 07/20/2026
End Date: 12/31/2026
Position Type: Contract (C2C)
No. of Positions: 1

Position Overview

We are seeking a highly skilled Lead Data Engineer to join a growing Data & Analytics team responsible for building and modernizing an enterprise-scale AWS + Databricks Lakehouse platform. This is a hands-on technical leadership role where you will architect, develop, and optimize scalable data solutions that power analytics, reporting, and AI initiatives across the organization.

The ideal candidate has extensive experience designing enterprise data platforms, building robust ETL/ELT pipelines, and working with cloud-native technologies such as Databricks, Apache Spark, Delta Lake, Unity Catalog, AWS, SQL, and Python.

This position offers the opportunity to influence enterprise data architecture while mentoring engineering teams and collaborating closely with business stakeholders.

Key Responsibilities

Design, develop, and maintain scalable enterprise data pipelines using Databricks and AWS.
Lead the architecture and implementation of modern Lakehouse solutions.
Build high-performance ETL/ELT pipelines supporting both batch and streaming workloads.
Develop data ingestion, transformation, and orchestration frameworks.
Optimize Spark workloads for performance, scalability, and cost efficiency.
Implement Delta Lake best practices for reliable and governed data management.
Design and maintain enterprise data models supporting analytics and reporting.
Collaborate with business stakeholders to translate requirements into scalable technical solutions.
Establish engineering standards, coding best practices, and data governance processes.
Monitor pipeline performance and troubleshoot production issues.
Mentor junior engineers while remaining actively involved in hands-on development.
Support CI/CD automation and DevOps best practices across the data platform.

Required Technical Skills

Databricks

Apache Spark
Delta Lake
Unity Catalog
Databricks Workflows
Lakehouse Architecture
Notebook Development
Performance Optimization

AWS Cloud

Amazon S3
IAM
VPC & Networking
Cloud Storage
Security & Access Management
Cloud-native Architecture

Programming

Python
SQL
PySpark
Spark SQL
Data Processing Frameworks

Data Engineering

ETL / ELT Development
Data Pipelines
Data Integration
Data Transformation
Batch Processing
Streaming Data Processing
Distributed Computing

Data Architecture

Lakehouse Architecture
Medallion Architecture
Data Modeling
Data Warehousing
Metadata Management
Data Governance
Data Quality

DevOps & Automation

CI/CD Pipelines
Git
Azure DevOps or GitHub Actions
Infrastructure Automation
Monitoring & Logging

Required Qualifications

10+ years of overall IT experience.
5+ years of hands-on Data Engineering experience.
Strong experience designing enterprise-scale ETL/ELT solutions.
Expert-level experience with Databricks, Apache Spark, Delta Lake, and Unity Catalog.
Strong AWS experience including S3, IAM, networking, and cloud architecture.
Advanced SQL and Python programming skills.
Experience building batch and streaming data pipelines.
Strong understanding of distributed computing concepts.
Experience implementing Lakehouse and Medallion architecture.
Experience with Agile software development methodologies.
Strong problem-solving and analytical skills.
Excellent communication and stakeholder management abilities.

Preferred Qualifications

Experience leading Data Engineering teams.
Experience supporting AI, Machine Learning, or Advanced Analytics platforms.
Knowledge of enterprise data governance frameworks.
Experience with infrastructure automation and cloud optimization.
Exposure to modern analytics platforms and reporting ecosystems.
Experience working in highly collaborative Agile environments.

Soft Skills

Strong leadership and mentoring abilities.
Excellent communication and presentation skills.
Ability to translate business needs into technical solutions.
Self-driven with strong ownership and accountability.
Comfortable working in fast-paced, evolving environments.
Strong organizational and time-management skills.
Collaborative mindset with a focus on delivering business value.

About the Company

Argyllinfotech

Resume Resources

Free Resume Templates Free Resume Builder