Databricks Platform Engineer

Role: Engineer

Required Skills: Apache Spark

Qualification: BACHELOR OF COMPUTER SCIENCE

Experience: 8 - 10 Years

Function: TECHNOLOGY

Apply By: 2026-12-11

Data Engineering Responsibilities

Design and build batch data pipelines using Apache Spark on EKS and Databricks, processing data from NAS and upstream source systems into Iceberg tables on S3

Own the Bronze Silver Gold DLT pipeline architecture for assigned business domains

Write and maintain DDL for Iceberg tables following the CDP naming standard (catalog.schema.table with region, LOB, and layer components)

Integrate with existing loading framework (Autosys OpenShift S3 Airflow DAG trigger chain) and extend it as the platform evolves

Work with Data Stewards and DBA team to ensure schema design meets data classification requirements (INT, PII, CPII, CSPII)

Implement and validate Unity Catalog governance controls - column masking, row-level filters, and lineage tracking - for tables under your ownership

Contribute to the migration of pipeline artifacts from DBFS to UC Volumes as part of the platform's governance uplift programmer

DataOps Responsibilities

Own the operational health of pipelines in your domain - monitoring, alerting, failure triage, and restart procedures

Build and maintain job monitoring integration between Databricks / Airflow and the team's observability stack (Splunk, CloudWatch, MSK audit events)

Define and document incident runbooks for common failure patterns: EKS resource exhaustion, S3 VPC Endpoint connectivity, Spark executor out-of-memory, DLT pipeline schema evolution failures

Manage deployment of pipeline code and DAG updates through the Lightspeed CI/CD pipeline - understand what goes to S3 config bucket, what goes to the DAG bucket, and what goes to EFS

Participate in on-call rotation for production pipeline incidents

Work with the platform team on capacity planning for EKS cluster - there is no YARN queue here, resource exhaustion means job failure

Maintain data quality monitoring and reconciliation between on-prem and cloud outputs during the migration period

What We Are Looking For

Must Have

Overall 10 years' experience; 4+ years of data engineering experience, with at least 2 years working with Spark in a production environment

Solid understanding of cloud networking fundamentals - VPC, subnets, VPC Endpoints, Security Groups. You do not need to be a network engineer, but you need to understand why a Spark pod's network path matters and what breaks when it is wrong

Hands-on experience with AWS services: S3, EKS, IAM, KMS, SQS, MSK

Experience with Apache Iceberg or Delta Lake in production - not just awareness, but actual table design and maintenance

Familiarity with Kubernetes - you do not need to administer a cluster, but you need to understand pod l ifecycle, resource limits, and how to read a pod describe output when something fails

Comfortable reading and writing Python and SQL at a production standard

Understanding of IAM and credential management - why static keys are unacceptable, how STS AssumeRole works, what Vault does

Strong Preference

Experience with Databricks, specifically Unity Catalog, MLflow, and DLT

Experience working in a regulated financial services environment - you understand why data classification, audit trails, and access controls are not optional extras

Familiarity with Airflow (DAG design, KubernetesPodOperator, trigger chain design)

Experience with Spark performance tuning - you know what data skew looks like, when to add executors vs CPU, and how to read a Spark history server

Exposure to model governance or MLOps - understanding how MLflow Model Registry, Champion/Challenger alias management, and MRM review processes fit together

Not Required But Noticed

Experience with Lakebase or other OLTP-on-lakehouse technologies

Familiarity with Collibra or similar enterprise data governance platforms

Experience designing UC group naming conventions or RBAC frameworks at scale

What You Will Not Be Doing

Building dashboards or BI reports

Managing the Databricks platform itself - that sits with the platform team

The Environment

Compute: EKS (Spark), OpenShift (on-prem loading framework), Databricks (UC, MLflow, DLT)

Storage: S3 (Iceberg), EFS (intermediate artifacts), NAS (source files)

Orchestration: Airflow (MWA), Autosys (on-prem, being phased out)

Messaging: MSK (Kafka), SQS, SNS

Catalog: HMS on Aurora migrating to AWS Glue Unity Catalog

Security: Vault (credential injection), KMS (encryption), PrivateLink (no public internet for data paths), CloudTrail (audit)

CI/CD: Lightspeed (Citi internal), Bitbucket / GitHub

What Good Looks Like in This Role

In the first 90 days, you should be able to:

Explain the end-to-end data flow from NAS to an Iceberg table in S3, naming every component the data touches and the network path it takes

Run a Spark job on EKS independently, debug a failure from the pod logs, and know when to look at the Spark History Server vs CloudTrail

Identify a governance gap in a UC table configuration and propose a fix

In six months, you should own at least one business domain end-to-end - pipeline design, operational health, schema governance, and incident response.

Salary Range: $125,000 to $140,000 per year