Lead Data Platform Engineer - Enterprise, Data & AI

In This Role, You Will...

• Scaling self-healing data pipelines that automatically handle schema evolution, detect anomalies, execute circuit breakers and recover from failures without manual intervention.
• Define and enforce company-wide data governance, access, automated data quality testing, schema evolution policies and metadata management to maintain high-fidelity data assets.
• Implement intelligent storage lifecycle strategies, resource throttling and query optimization to minimize compute overhead while delivering high-throughput, low-latency data access for analytics and AI agents. With a focus on Databricks vs EMR (AWS) cloud optimization (cost and performance).
• Build automated CI/CD deployment templates, environment isolation, testing frameworks and version control standards, enabling rapid, reliable and zero-downtime deployments

Qualifications

• 10+ years of hands-on experience in Data Platform Engineering or Software Engineering with a proven track record of architecting and scaling production-grade data foundations.
• Deep expertise in scaling and optimization, query tuning, compute resource allocation and implementing efficient compute-storage lifecycle policies to minimize infrastructure costs.
• Experience implementing enterprise security standards, Role-Based Access Control (RBAC), Active Directory/IAM roles and fine-grained data masking, with strong familiarity supporting enterprise compliance requirements (e.g., SOX, audit trail controls, data retention policies).
• Track record of establishing enterprise data governance frameworks, automated schema evolution controls, data quality audits and near real-time observability/alerting.
• Hands-on experience building automated CI/CD pipelines, environment isolation (branch testing, rollback mechanisms), version control and automated deployment testing for data assets.

Bonus Qualifications