Job Description
We are seeking a Senior Databricks Tech Lead / Architect for an onsite engagement. This is NOT a paper architect role. The ideal candidate must be a hands-on coding practitioner who can architect scalable data solutions, write and review production-quality code, and provide technical direction to an offshore development team. You will be the primary technical authority for our Databricks platform and own both delivery and design quality.
Key Responsibilities
Design and implement end-to-end data lakehouse solutions on Databricks (Delta Lake, Unity Catalog, Workflows)
Write hands-on production-quality PySpark, Python, and SQL code — not just review or approve
Architect medallion (Bronze/Silver/Gold) data pipelines and ensure they meet performance and reliability standards
Lead and direct offshore/nearshore development teams: daily standups, code reviews, sprint planning, and technical mentorship
Define and enforce coding standards, branching strategies, and CI/CD practices for the data engineering team
Collaborate with business stakeholders, data architects, and product owners to translate requirements into technical designs
Optimize Spark jobs, cluster configurations, and cost management on Azure/AWS/GCP cloud environments
Integrate Databricks pipelines with upstream/downstream systems including Kafka, ADF, Airflow, dbt, and REST APIs
Drive best practices in data quality, data governance, and metadata management
Proactively identify technical debt and lead remediation efforts
Serve as the go-to escalation point for all Databricks technical issues within the engagement
Required Qualifications
8+ years of overall data engineering experience; 4+ years specifically on Databricks platform
Deep hands-on expertise with PySpark, Spark SQL, Delta Lake, and Databricks Workflows
Proven experience architecting large-scale data pipelines in a cloud environment (Azure preferred; AWS or GCP acceptable)
Demonstrated experience leading and directing offshore or distributed engineering teams
Proficiency in Python and SQL — must be able to produce and review code, not just advise
Strong grasp of data modeling, dimensional modeling, and lakehouse design patterns
Experience with CI/CD tools (Azure DevOps, GitHub Actions) and infrastructure-as-code (Terraform, ARM)
Databricks Certified Associate Developer or Professional certification is a strong plus
Excellent communication and stakeholder management skills for both technical and business audiences