Lead Data Engineer

Introduction

As a key figure in our data engineering team, you will play a critical role in designing and implementing scalable data pipelines and architectures. You will leverage your expertise in Python and PySpark to enhance our data processing capabilities, with a particular focus on migrating data from Snowflake to Databricks. This position requires a strong background in big data technologies and excellent leadership skills.

Required Skills & Qualifications

• Minimum of 8 years of experience in data engineering, with a strong focus on Databricks and big data technologies.
• Proficient in Python and PySpark, with a proven track record of building data pipelines and ETL processes.
• Experience with Snowflake and a demonstrated ability to migrate data solutions to Databricks.
• Strong understanding of data modeling, data warehousing concepts, and distributed computing.
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
• Excellent problem-solving skills and the ability to work in a fast-paced environment.
• Strong communication and interpersonal skills, with the ability to collaborate effectively with technical and non-technical stakeholders.
• Bachelor’s degree in Computer Science, Engineering, or a related field; advanced degree preferred.
• Prior work experience at client or in client's Industry.
• Applicants must be able to work directly for the company on W2.
• Previous experience working with large datasets in a high-volume environment.
• Knowledge of machine learning frameworks and analytics tools is a plus.
• Experience in Agile methodologies and DevOps practices.

Day-to-Day Responsibilities

• Lead the design, development, and optimization of data pipelines and ETL processes in Databricks.
• Collaborate with cross-functional teams to gather requirements and translate them into robust data solutions.
• Migrate existing data solutions from Snowflake to Databricks, ensuring minimal disruption and high data integrity.
• Utilize Python and PySpark to build and maintain scalable data processing frameworks.
• Implement best practices for data governance, security, and performance optimization.
• Mentor and guide junior data engineers, fostering a culture of continuous learning and improvement.
• Monitor system performance and troubleshoot issues to ensure optimal data flow and availability.
• Stay updated with industry trends and emerging technologies related to data engineering and analytics.

Company Benefits & Culture

• Inclusive and diverse work environment.
• Opportunities for professional growth and development.
• Flexible work arrangements to support work-life balance.

For immediate consideration please click APPLY to begin the screening process with Alex.