SDE III

SDE III/IV - Data Platform Engineering

Mission

Architect a self-serve Data "Platform-as-a-Product" powering InMobi's global-scale data ecosystem. Integrate OSS tools, proprietary services, and Cloud/SaaS into unified infrastructure. Requires deep data engineering

expertise (batch/streaming pipelines, data modeling, query optimization, governance) combined with platform engineering to build production-grade solutions for data engineers and analysts.

Core Responsibilities

Design & Development- Bridge OSS tools (Spark, Flink, Airflow, Iceberg), internal services, and cloud offerings into cohesive data platform infrastructure. Build intuitive platform integrations enabling push-button data workflows.

Scale Engineering- Operate distributed systems processing petabytes of data daily. Own multi-region Kubernetes infrastructure with elastic scalability and fault tolerance.

Performance Optimization- Optimize compute utilization (Spark/Flink clusters, Velox/Gluten acceleration) for large-scale batch and real-time streaming with sub-second latency.

Observability & Data Quality- Build comprehensive telemetry (metrics, logs,traces) and data quality frameworks for 24/7 uptime. Enforce SLAs/SLOs with automated incident response and data validation.

Required Skills & Experience (Must-Have)

• 7-10 years building, optimizing, and operating production data platforms
• Deep data engineering fundamentals: data modeling, partitioning strategies, query optimization
• Distributed compute: Spark (PySpark/Scala), Flink streaming, performance tuning at petabyte scale
• Data lake architecture: Iceberg table format, Polaris catalog, schema evolution, time travel
• Orchestration: Airflow DAG development, dependency management, SLA monitoring
• Data transformation: DBT modeling, testing, documentation, incremental builds
• Data quality: Great Expectations, dqueue validation frameworks, drift detection
• Query acceleration: Velox, Gluten integration, columnar formats (Parquet, ORC)
• Data governance: OpenMetadata catalog, lineage tracking, access control
• Kubernetes platform development: operators (Spark/Flink), Yunikorn scheduler, multi-tenancy, autoscaling
• Cloud infrastructure: GKE multi-region clusters, GCS object storage, hybrid cloud/on-prem architecture
• Programming: Python, PySpark, Scala for data pipelines and platform tooling
• IaC: Terraform, Helm, GitOps for reproducible deployments
• CI/CD: Automated testing, deployment pipelines for data platform components

Good-to-Have

• Experience building cloud data platform / control plane development
• Advanced observability: Prometheus/Grafana, Loki, Firehydrant integration
• Real-time streaming: Kafka integration, exactly-once semantics, backpressure handling
• Cost optimization: Resource allocation, query optimization, storage tiering strategies