Lead Data Engineer

Role: Lead Data Engineer

Location: Charlotte, NC (Hybrid)

FULLTIME

We're seeking an experienced pipeline-centric data engineer to put it to good use in building out ETL and Data Operations framework (Data Preparation / Normalization and Ontological processes).

Technical Skills:

• Lead the design, development, and maintenance of scalable data pipelines and ETL processes, ensuring data integrity and accessibility for business intelligence and advanced analytics.
• Architect and manage robust data platforms on the AWS ecosystem, leveraging services like Glue, PySpark, Apache Iceberg, IAM, S3, and Secrets Manager to build a secure and efficient data infrastructure.
• Provide technical guidance and mentorship to a team of data engineers, fostering a culture of high performance and continuous learning.
• Utilize deep expertise in various RDBMS (e.g., MySQL, Db2, PostgreSQL, Snowflake) and different data formats (e.g., JSON, Parquet) to drive strategic data initiatives.
• Champion the adoption of modern data technologies such as Apache Iceberg with AWS Glue to optimize data lake performance and analytics capabilities.
• Apply advanced proficiency in Generative AI to automate and streamline data analysis, development, and documentation processes.
• Business Acumen and Stakeholder Communication
• Translate complex business challenges into clear, data-driven solutions, effectively communicating technical concepts and project progress to both technical and non-technical stakeholders, including senior leadership.
• Act as a key liaison between the data engineering team and business units, providing data-backed insights and recommendations that directly influence business strategy.
• Manage concurrent projects in a dynamic, research-oriented environment, ensuring timely delivery and high-quality outcomes.
• Experience in insurance domain preferrable
• AWS Data Engineering certification good to have

Key Responsibilities

• Collaborate with business analysts and stakeholders to translate business needs into comprehensive source-to-target (S2T) data mappings.
• Lead the design and development of robust data pipelines, using your deep understanding of existing ETL frameworks to build scalable and efficient solutions.
• Analyze and understand diverse source systems and data formats to create and validate synthetic datasets for testing and development.
• Develop automated data validation tools using Python to ensure data integrity across various ETL layers.
• Ensure Industry best practice is followed in all aspects of project.
• Develop robust development testing approach to minimize the quality, integration and user testing issues.
• Leverage generative AI to enhance data analysis, accelerate development, Unit Testing and streamline documentation.
• Apply statistical analysis to large datasets, uncovering key patterns, identifying potential challenges, and generating actionable insights.
• Partner with business leaders to understand their challenges and provide data-driven recommendations that improve processes and inform strategic decisions.

Qualification:

• Somebody who has at least 12+ years of data engineering experience has played Lead Data Engineer role.
• Bachelor's degree (or equivalent) in computer science, information technology, engineering, or related discipline
• Education qualification: Any degree from a reputed college

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.