Lead Data Engineer

Role: Lead Data Engineer Azure Databricks

Location: Charlotte, North Carolina 28273
Work Arrangement: 5 days onsite
Employment Consideration: Contract to Hire

Let's create our future together at The AES Group!

About The AES Group:

The AES Group is a premier technology and engineering consulting company that has been bringing businesses and talent together for over 20 years to deliver innovative solutions that have the greatest positive impact on society. AES has helped over 40 business enterprises, including Fortune 500 companies, engage their customers, empower their employees, and transform their business operations with the power of cloud, data, AI, engineering, and other emerging technologies.

Why Join This Opportunity?

• Lead high-impact enterprise data engineering initiatives.
• Guide and mentor a growing team of data engineers.
• Work extensively with the latest Azure Databricks capabilities and modern cloud data technologies.
• Contribute to large-scale data modernization and legacy migration programs.
• Collaborate with technical leaders and business stakeholders on mission-critical data solutions.
• Gain exposure to Power BI, MicroStrategy, Azure DevOps, Delta Lake, and advanced data governance frameworks.

Position Overview

A leading organization is seeking an experienced Lead Data Engineer to join its expanding enterprise data team. This role will lead the design, development, and optimization of scalable data platforms while mentoring engineers and establishing data engineering best practices.

The ideal candidate will bring strong, recent experience with the latest version of Azure Databricks, along with hands-on expertise in Python, PySpark, SQL, ETL/ELT development, and modern data warehouse architecture. Strong communication and team leadership skills are essential.

Key Responsibilities

• Lead the design, development, and optimization of scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, SQL, and Delta Lake.
• Provide technical leadership, guidance, and mentorship to a team of data engineers.
• Enhance CI/CD pipelines using Azure DevOps, Git, Terraform, and automated deployment frameworks.
• Establish engineering standards that improve development efficiency, code quality, testing, and release management.
• Ensure data accuracy, quality, and consistency through effective integration, transformation, validation, and cleansing processes.
• Use Python and PySpark for data manipulation, workflow automation, performance optimization, and custom data solutions.
• Support migrations from legacy data environments to modern cloud-based data platforms.
• Collaborate with business stakeholders, architects, analysts, and engineering teams to translate data requirements into scalable technical solutions.
• Apply data security and governance practices using Unity Catalog, Azure role-based access control, managed identities, and data access controls.
• Support enterprise reporting platforms, primarily Power BI, with exposure to MicroStrategy environments.
• Participate in an on-call rotation and assist with after-hours production issues when required.
• Perform additional technical and leadership responsibilities as assigned.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, Information Technology, or a related discipline, or equivalent professional experience.
• At least three years of hands-on experience with the Azure Databricks platform.
• Strong experience using recent Azure Databricks features and capabilities.
• At least five years of data warehouse analysis, architecture, and design experience.
• Strong knowledge of data warehouse methodologies, dimensional modeling, operational data stores, and enterprise data integration.
• At least five years of experience developing ETL/ELT solutions using Python, PySpark, and SQL.
• Hands-on experience with Terraform, Git, CI/CD pipelines, and Azure DevOps.
• Demonstrated experience leading and mentoring data engineering teams.
• Strong written, verbal, technical, and stakeholder communication skills.
• Ability to manage changing priorities and solve complex technical problems creatively.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional certification.
• Experience with Unity Catalog, Azure RBAC, managed identities, and enterprise data governance.
• Familiarity with Power BI architecture and reporting solutions.
• Familiarity with MicroStrategy architecture.
• Experience supporting regulated, quality-focused, or compliance-driven environments.

Ready to make an impact? Apply now and join us on our journey!