This is a hybrid role based in Hyderabad, India.
Role Summary
Seeking a hands-on senior Python engineer to own data pipelines end to end (ingestion, transformation, orchestration, quality, and delivery) at enterprise scale. Pipelines feed analytics, search, and GenAI/Agentic AI platforms.
Responsibilities
- Design, build, and own production data pipelines in Python (batch and near-real-time).
- Operate orchestrated workflows (AWS Step Functions, Airflow, Dagster, or Prefect) with retries, idempotency, and backfills.
- Model data at scale: partitioning, Parquet/Avro/Iceberg, schema evolution, CDC.
- Establish data quality and observability (Great Expectations/Pandera, SLAs, lineage, alerting).
- Build document-processing and RAG retrieval pipelines (chunking, embeddings, vector and hybrid search).
- Build serving APIs with FastAPI, Flask, or Django REST.
- Set Python engineering standards; mentor junior engineers; liaise with business stakeholders.
Qualifications
- 8–12 years of software engineering, 5–6+ years building production data pipelines in Python.
- Expert modern Python (3.10+), typing, asyncio/concurrency, pandas/Polars, PyArrow, PySpark/Dask/Ray.
- Strong SQL and data modeling; Snowflake, Redshift, Iceberg/Delta, dbt.
- Strong AWS: Lambda, Step Functions, S3, Glue, EMR, IAM, CloudWatch; IaC (CDK/Terraform).
- pytest, packaging (uv/Poetry), CI/CD.
- Strongly preferred: RAG, AWS Bedrock, LangChain/LlamaIndex, agentic frameworks, MCP servers, GenAI evaluation.
- Nice to have: Kafka/Kinesis/Flink streaming, C#/.NET, regulated-industry experience.