Technical Lead Data Engineering

Technical Lead Data Engineering
Oakland, CA (remote)
12+ Months Contract
Potential to convert to perm

Job Description
Client is looking for an experienced Technical Lead Data Engineering with strong expertise in Python, PySpark, SQL, and Palantir Foundry. The ideal candidate should have hands-on experience designing and developing scalable batch and streaming data pipelines, data transformation workflows, and data models. The candidate will be responsible for leading a team of data engineers, providing technical guidance, designing data engineering solutions, optimizing pipeline performance, and collaborating with business and technical stakeholders.

Key Responsibilities

• Lead the design, development, and implementation of scalable data pipelines using Palantir Foundry.
• Develop and optimize ETL/ELT pipelines using Python, PySpark, Spark SQL, and SQL.
• Build and manage batch and real-time/streaming data processing workflows.
• Design data models, datasets, and ontology objects within Palantir Foundry.
• Implement data quality checks, data validation, monitoring, and alerting mechanisms.
• Optimize data pipelines for performance, scalability, and reliability.
• Work with multiple data sources and develop data integration solutions.
• Lead technical discussions, code reviews, sprint planning, and solution design.
• Collaborate with Product Owners, Business Analysts, and cross-functional teams.
• Mentor junior and mid-level data engineers and establish engineering best practices.
• Prepare technical design documents, architecture diagrams, and implementation plans.

Required Technical Skills

• Strong hands-on experience with Palantir Foundry.
• Strong programming experience in Python and PySpark.
• Advanced SQL and Spark SQL expertise.
• Experience building ETL/ELT data pipelines.
• Experience with batch and streaming data processing.
• Knowledge of data modeling, data transformation, and data integration.
• Experience with Palantir Foundry Ontology, datasets, and pipeline development.
• Experience implementing data quality, monitoring, and pipeline error handling.
• Strong understanding of data engineering architecture and performance optimization.

Leadership Skills

• Experience leading and mentoring data engineering teams.
• Ability to translate business requirements into technical solutions.
• Experience with Agile/Scrum methodologies and sprint management.
• Strong communication, stakeholder management, and problem-solving skills.