Principal Databricks Developer

Do you want to be part of an inclusive team that works to develop innovative therapies for patients? Every day, we are driven to develop and deliver innovative and effective new medicines to patients and physicians. If you want to be part of this exciting work, you belong to Astellas!

Astellas Pharma Inc. is a pharmaceutical company conducting business in more than 70 countries around the world. We are committed to turning innovative science into medical solutions that bring value and hope to patients and their families. Keeping our focus on addressing unmet medical needs and conducting our business with ethics and integrity enables us to improve the health of people throughout the world. For more information on Astellas, please visit our website at www.astellas.com.

This position is based in Bengaluru and might require some on-site work.

Astellas' Global Capability Centres - Overview

Astellas' Global Capability Centres (GCCs) are strategically located sites that give Astellas the ability to access talent across various functions in the value chain and to co-locate core capabilities that are currently dispersed. Our three GCCs are located in India, Poland and Mexico.

The GCCs will enhance our operational efficiency, resilience and innovation potential, enabling a timely response to changing business demands.

Our GCCs are an integral part of Astellas, guided by our shared values and behaviors, and are critical enablers of the company's strategic priorities, sustainable growth, and commitment to turn innovative science into VALUE for patients.

Purpose and Scope:

The Principal Databricks Developer serves as a senior technical leader responsible for shaping how complex data applications, pipelines, and distributed processing workloads are built and operated on the Databricks Platform. This role combines deep hands‑on engineering expertise with architectural thinking, ensuring that our data solutions are not only well‑implemented, but also scalable, maintainable, and aligned with the long‑term direction of our data ecosystem. You will design advanced PySpark and SQL frameworks, lead the development of high‑performance data flows, and guide the team in applying best practices around Delta Lake, job orchestration, and cloud integration.

As a principal contributor, you will take ownership of the most challenging technical initiatives-solving problems at scale, driving improvements in performance and reliability, and setting the standards that others follow. You will partner closely with Data Engineering, Platform Engineering, Data Science, Architecture, and other technical teams to evaluate solution designs, influence engineering decisions, and mentor developers in modern data engineering practices.

Responsibilities and Accountabilities:

• Develop & Maintain Scalable Data Pipelines: Architect, build, and optimize ETL/ELT pipelines using PySpark, Spark SQL, Auto Loader, and Delta Live Tables to support end to end ingestion and transformation.
• Implement Robust Lakehouse Architecture: Design and enhance Medallion Layers (Bronze/Silver/Gold) data models, applying Delta Lake features such as schema evolution, CDF, Optimize, and Z Ordering to deliver performant, reliable, and cost efficient data layers.
• Integrate Data Across Cloud Platforms: Ingest and harmonize structured, semi structured, and unstructured data from multiple cloud environments including Azure, AWS, GCP, and enterprise object storage.
• Develop Reusable Engineering Frameworks: Create and maintain reusable Python, PySpark, and YAML based libraries and patterns to standardize ingestion, transformation, automation, and engineering workflows across teams.
• Enforce Data Quality & Governance: Implement and operationalize automated data validation frameworks (DLT expectations, data contracts) while applying Unity Catalog governance covering permissions, lineage, external locations, and PII/PHI controls.
• Lead CI/CD & Deployment Automation: Lead Azure DevOps and Databricks Asset Bundles (DABs) to establish automated build, test, and deployment workflows; ensure source control discipline and promote engineering best practices.
• Optimize Performance & Cost Efficiency: Tune Spark workloads by applying partitioning, caching, and join optimization strategies; leverage Photon, serverless SQL, and cluster right sizing to improve runtime performance and reduce compute costs.
• Collaborate with Data & Platform Teams: Partner closely with Data Scientists, Analysts, SMEs, and Platform Engineering teams to translate requirements into scalable data solutions and align on architectural, governance, and operational standards.
• Operationalize Data Science Workflows: Convert prototype notebooks into production ready pipelines, support feature engineering and batch/real time scoring and manage MLflow tracking and model registry operations.
• Develop Lightweight Analytical Applications: Build small-scale applications using Streamlit, Shiny, or Gradio to support internal stakeholders with interactive data products and insights.
• Provide Technical Leadership & Mentorship: Lead design reviews, guide junior/mid-level engineers, and champion best practices to elevate engineering quality and technical execution across the organization.
• Lead and manage a team of Databricks engineers, providing direction, prioritization, and delivery oversight across sprint commitments and platform objectives.
• Coach, mentor, and develop engineers through regular 1:1s, capability plans, technical guidance, and structured feedback to build a high-performing engineering culture.
• Lead cross-team alignment on Databricks standards (CI/CD, governance, data quality, and operational readiness), ensuring consistent adoption across domains and delivery teams.