Databricks Engineer

KEY RESPONSIBILITIES
• Design, develop, and deliver end-to-end data engineering pipelines for both batch and real-time streaming workloads.
• Design and implement cloud-based data lakehouse platforms integrating data from diverse enterprise sources.
• Develop scalable ETL/ELT pipelines using PySpark, Python, Scala, and SQL.
• Build and maintain real-time data processing and streaming pipelines for operational and analytical use cases.
• Design and implement robust data models, including 3NF, dimensional modeling, and enterprise data warehouse solutions.
• Develop incremental data loading frameworks and metadata-driven data ingestion pipelines.
• Implement data quality frameworks, validation processes, and data governance standards.
• Implement and manage Databricks Unity Catalog, including fine-grained security, permissions, and access controls.
• Develop and manage Databricks solutions using Delta Live Tables (DLT), Auto Loader, Structured Streaming, and Databricks Workflows.
• Integrate Databricks pipelines with orchestration platforms such as Apache Airflow.
• Implement CI/CD pipelines and deployment automation following DevOps best practices.
• Optimize Spark jobs, Databricks workloads, data pipelines, and compute resources for performance and cost efficiency.
• Troubleshoot complex data engineering and pipeline issues and provide effective solutions.
• Provide technical and architectural guidance to data engineering teams.
• Collaborate with cross-functional teams, business stakeholders, and clients to understand requirements and translate them into scalable technical solutions.
• Contribute to data engineering standards, best practices, reusable frameworks, and development methodologies.