Responsibilities:
• Advise on the Databricks reference architecture covering Lakeflow, Delta Lake, Unity Catalog Mosaic AI, Databricks SQL, and Databricks Apps
• Provide hands-on guidance on Databricks full-stack solution design
• Advise on design of both data movement (e.g. batch, real-time streaming) and zero-copy patterns
• Guide platform migrations from legacy SQL and document stores onto Databricks and related cloud data platforms
• Recommend security, governance, and compliance-as-code: Unity Catalog ABAC, governed tags, row filters/column masks, key management, networking security and compliance security profiles.
• Advise on design of segregated/tiered data pool architectures aligned to the organization's data classification policy and applicable regulatory requirements, including cross-tier data movement
• Recommend FinOps and performance standards, workload and cost management
• Mentor data engineers, analysts, and AI/ML teams participate in architecture reviews and contribute to standards
• Work with information & cyber security, and other teams as needed for compliance to policies and standards
• Evaluate emerging Databricks capabilities and recommend adoption strategies, including endpoint integrations with third parties as they evolve
Required:
• 7+ years designing, building, and operating Databricks workloads with medallion layer architecture
• Hands-on experience with Delta Lake, Unity Catalog,, Spark Structured Streaming, Lakeflow (or DLT/Live Tables), Databricks SQL, MLflow, and Mosaic AI Model Serving
• Strong experience in cloud architecture supporting the platform (VPC/VNet, Security, S3/ADLS, KMS, etc.)
• Proven experience migrating enterprise data workloads from legacy/on prem data warehouse/lake to Databricks
• Production streaming and CDC experience
• Depth in security and governance: Unity Catalog policies (ABAC), classification, encryption, access auditing, compliance-to-control translation
• Hands-on experience designing multi-tier Databricks architectures driven by data classification requirements
• Strong Python and SQL, Spark, Notebooks
• Excellent verbal and written communication skills
Nice to Have:
• Active Databricks certifications
• GenAI and agent systems on Databricks experience
• Delta Sharing and Databricks Clean Rooms for controlled, governed data exchange across organizational/cross-tier/system boundaries
• Familiarity with Oracle EBS and SAP enterprise data
• Experience with SAP BDC/DataSphere and Databricks integration to SAP BDC, with zero-copy patterns
• Lakeflow Designer, Databricks Apps, Genie, Databricks One
• CI/CD, Infrastructure-as-Code (e.g. Terraform) for data platforms
• Experience in designing, building, supporting BI reporting layers
• Data Quality and Data Governance experience
• Prior experience in regulated industries (e.g. utilities, life sciences, government services) and exposure to related frameworks (e.g. FERC/NERC, HIPAA, GxP, GDPR, FedRAMP, ITAR, etc.)