Databricks Admin SME

Position Overview
Client is seeking an experienced Databricks Admin Subject Matter Expert (SME) to manage, optimize, and secure enterprise-scale Databricks environments. The ideal candidate will be responsible for administration, governance, performance optimization, security implementation, and operational excellence of the Databricks platform while supporting business-critical data engineering and analytics workloads.
This role requires deep expertise in Databricks workspace administration, cloud integrations, user access governance, security best practices, and platform scalability across large enterprise environments.

Key Responsibilities
Databricks Platform Administration

• Manage and administer Databricks workspaces across development, QA, and production environments.
• Configure clusters, cluster policies, job scheduling, and workspace settings.
• Monitor system performance, usage trends, and optimize platform costs.
• Implement workspace provisioning, maintenance, upgrades, and troubleshooting.

Security & Governance

• Design and manage role-based access control (RBAC), Unity Catalog, and data governance frameworks.
• Integrate Databricks with enterprise identity providers (Azure AD, AWS IAM, Okta, etc.).
• Ensure compliance with organizational security policies and industry standards.
• Configure secure networking, private link, VNET injection, and encryption protocols.

Cloud Integration & Infrastructure

• Collaborate with cloud infrastructure teams to manage Databricks deployments on Azure, AWS, or GCP.
• Configure storage integrations (ADLS, S3, Blob Storage).
• Support CI/CD automation using Terraform, ARM templates, or Infrastructure as Code (IaC).
• Maintain disaster recovery and backup strategies.

Monitoring & Optimization

• Set up monitoring dashboards, logging, and alerting for proactive issue resolution.
• Optimize cluster sizing, auto-scaling, and job performance.
• Conduct capacity planning and cost governance initiatives.
• Troubleshoot platform, networking, and integration issues.

Collaboration & Leadership

• Act as the primary SME for Databricks administration and operational support.
• Partner with data engineers, architects, security teams, and business stakeholders.
• Provide best practices, technical guidance, and mentoring to internal teams.
• Support enterprise data modernization initiatives.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field.
• 15+ years of experience in data platform administration or cloud data engineering.
• 4+ years of hands-on Databricks administration experience.
• Strong expertise with:
  • Databricks Workspace Administration
  • Unity Catalog
  • Cluster Management
  • Security & Governance
  • Azure/AWS/GCP cloud ecosystems
  • Terraform / IaC
  • Identity and Access Management
• Experience with:
  • Spark
  • Delta Lake
  • SQL
  • Python
  • Monitoring tools
  • DevOps processes
• Knowledge of networking, cloud security, and compliance standards.