Senior Databricks Developer

Job Description: Seeking an experienced Senior Databricks Developer to lead the design, development, optimization, and management of enterprise-scale cloud data engineering solutions using Azure Databricks and Azure data services. The ideal candidate should possess strong expertise in Azure Databricks, Azure Data Lake, ADLS Gen 2, ETL/ELT development, cloud-native data pipelines, and enterprise data architecture. This role requires leading end-to-end data engineering initiatives, building scalable pipelines for structured, semi-structured, API, and unstructured datasets, supporting analytics and AI workloads, and collaborating with business and technical stakeholders to deliver reliable, scalable, and cost-efficient data solutions. The candidate will also drive data governance, quality standards, performance tuning, metadata management, and continuous improvement across cloud data environments.

Key Responsibilities:

• Lead the design, development, and implementation of end-to-end data pipelines on Azure using Azure Databricks, Azure Data Lake, and ADLS Gen 2
• Develop scalable ETL/ELT pipelines supporting structured, semi-structured, API-based, and unstructured data sources
• Collaborate with Data Scientists, Business Analysts, architects, and stakeholders to translate business requirements into scalable technical data solutions
• Optimize data workflows for performance, scalability, reliability, and cost-efficiency within cloud environments
• Establish, maintain, and enforce enterprise data quality, governance, validation, and compliance standards
• Provide technical guidance for data modeling, schema design, metadata management, and analytics-ready data structures
• Support enterprise analytics, reporting, AI, and machine learning workloads through optimized data engineering solutions
• Monitor, troubleshoot, and resolve issues related to production data pipelines and cloud-based integrations
• Recommend performance improvements and optimization strategies for enterprise data workflows
• Evaluate emerging Azure and Databricks technologies and recommend enhancements to enterprise data architecture

Additional Responsibilities:

• Support enterprise cloud modernization and analytics transformation initiatives
• Participate in architectural discussions, technical reviews, and solution planning sessions
• Collaborate with engineering and analytics teams to improve scalability and operational performance
• Maintain technical documentation, standards, and best practices for enterprise data engineering initiatives
• Support continuous improvement and innovation within Azure cloud and Databricks environments

Required Skills:

• Strong expertise in Azure Databricks and enterprise data engineering
• Experience designing and developing ETL/ELT pipelines for enterprise environments
• Strong experience with Azure Data Lake and ADLS Gen 2
• Experience working with structured, semi-structured, API-based, and unstructured datasets
• Strong knowledge of data modeling, schema design, and metadata management
• Experience with cloud performance optimization, monitoring, and troubleshooting
• Strong understanding of data governance, validation, and data quality standards
• Experience supporting analytics, AI, and machine learning workloads using cloud data platforms
• Strong collaboration, analytical, and problem-solving skills

Qualifications:

• Senior-level experience in cloud-based data engineering and Azure Databricks environments preferred
• Experience collaborating with business, analytics, and technical teams required
• Experience supporting enterprise analytics and AI-driven workloads preferred
• Strong cloud optimization and enterprise-scale data architecture experience preferred

Certifications:

• Microsoft Azure certification preferred
• Databricks certification preferred