Job Description
Responsibilities & Skills: 12 YRS Exp- Design and implement enterprise-scale solutions on the Databricks Lakehouse Platform.
- Architect end-to-end data pipelines for batch and real-time processing using Apache Spark and PySpark.
- Develop scalable data ingestion, transformation, and data quality frameworks.
- Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
- Build and optimize data warehouses, data marts, and analytical solutions.
- Implement data governance, security, lineage, and access controls using Unity Catalog.
- Develop and support AI/BI dashboards, semantic models, and self-service analytics solutions.
- Configure and optimize Genie Spaces to enable natural language business queries and conversational analytics.
- Design and deploy Generative AI and RAG-based solutions using Databricks Mosaic AI and Vector Search.
- Collaborate with business users to translate requirements into scalable data and AI solutions.
- Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
- Lead cloud-native implementations across Azure environments.
- Define architecture standards, best practices, and reusable design patterns.
- Mentor data engineers, analysts, and architects on Databricks technologies and platform adoption.
- Lead migration initiatives from legacy data warehouses and analytics platforms to Databricks.
- Build and maintain Genie Spaces for business self-service analytics.
- Create semantic models, metrics, and trusted data assets for AI-driven reporting.
- Develop natural language-to-SQL analytics solutions using Databricks Genie.
- Implement RAG solutions using enterprise data and Vector Search.
- Optimize AI/BI dashboards and conversational analytics experiences.
- Troubleshoot Spark performance, query optimization, and workload management.
- Automate data validation, monitoring, and governance controls.
- Support AI use cases using Mosaic AI model serving and inference endpoints.
Technical Skills- Databricks Lakehouse Platform
- Apache Spark, PySpark, Spark SQL
- Python, SQL
- Delta Lake, Delta Live Tables, Lakeflow
- Unity Catalog
- Databricks AI/BI and Genie
- Mosaic AI, Vector Search, RAG
- Data Modeling (Dimensional & Data Vault)
- Structured Streaming
- Data Quality and Data Governance
- Azure
- Terraform, Git, Azure DevOps, Jenkins
- REST APIs and Data Integration
- Performance Tuning and Cost Optimization
Certification : Azure Databricks certified Data Eng professional