Job Title: Databricks Solution Architect
Location: King of Prussia, PA – Onsite
Experience Level: 12+Years of experience
Job Summary
We are looking for an experienced Databricks Solution Architect to design, architect, and lead the implementation of enterprise-scale data and AI solutions using the Databricks Data Intelligence Platform. The ideal candidate should have strong hands-on experience with Databricks, Lakehouse architecture, Apache Spark, Delta Lake, Unity Catalog, cloud platforms, data engineering, and modern data/AI architectures.
The candidate will work closely with business stakeholders, data engineers, developers, cloud teams, and enterprise architects to translate business requirements into scalable, secure, high-performance Databricks solutions.
Databricks' architecture guidance emphasizes Lakehouse architecture, governance, security, reliability, performance, cost optimization, and interoperability.
Key Responsibilities
Design and implement enterprise-grade Databricks Lakehouse architectures across development, testing, and production environments.
Define target-state architecture for data engineering, analytics, BI, machine learning, and AI workloads.
Architect solutions using Databricks, Delta Lake, Apache Spark, Unity Catalog, Databricks SQL, Workflows, and Lakehouse Federation.
Design and implement enterprise data governance, security, access control, lineage, and data-sharing strategies using Unity Catalog.
Lead migration of legacy data platforms, data warehouses, and Hive Metastore environments to Databricks/Unity Catalog.
Develop scalable data ingestion and processing architectures for batch and streaming workloads.
Provide technical leadership for performance tuning, scalability, reliability, and cost optimization of Databricks environments.
Design cloud-native solutions leveraging AWS, Azure, or GCP and integrate Databricks with cloud storage, networking, security, and identity services.
Establish Infrastructure as Code (IaC) and deployment strategies using Terraform and CI/CD. Databricks currently recommends Terraform for automating workspace, networking, storage, and Unity Catalog infrastructure.
Collaborate with DevOps teams to implement automated deployment pipelines and environment promotion.
Provide technical guidance to data engineers and development teams and conduct architecture/design reviews.
Work directly with customers and stakeholders to understand requirements and present architecture options and recommendations.
Create architecture diagrams, technical design documents, standards, and implementation roadmaps.
Troubleshoot complex production issues and provide architectural recommendations for resolution.
Stay current with emerging Databricks Data + AI capabilities, Generative AI, ML, and Lakehouse technologies.
Required Skills
12+ years of experience in Data Engineering, Data Architecture, Cloud Architecture, or related fields.
5+ years of strong Databricks experience, including architecture and implementation.
Strong hands-on experience with:
Databricks Lakehouse Platform
Apache Spark / PySpark
Delta Lake
Unity Catalog
Databricks SQL
Databricks Workflows / Jobs
Delta Live Tables / Lakeflow
Data governance and security
Strong understanding of Data Lake, Data Warehouse, and Lakehouse architectures.
Experience designing enterprise-scale batch and real-time/streaming data pipelines.
Strong programming experience with Python and/or Scala.
Strong SQL development and performance-tuning skills.
Experience with at least one major cloud platform:
AWS
Microsoft Azure
Google Cloud Platform
Experience with cloud storage technologies such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage.
Experience with Terraform/IaC and CI/CD.
Strong understanding of networking, IAM, authentication, encryption, and cloud security.
Excellent communication, presentation, documentation, and stakeholder-management skills.
Preferred Skills
Databricks certifications are highly preferred.
Experience with MLflow, Model Serving, Mosaic AI, and Generative AI.
Experience with LLM/RAG architectures and AI/ML platforms.
Experience with Kafka or other event-streaming technologies.
Experience with data integration tools such as Informatica, ADF, dbt, or equivalent.
Experience with Power BI/Tableau and enterprise analytics architectures.
Experience with enterprise data governance and compliance requirements.
Experience leading architecture workshops and technical discussions with senior stakeholders.
Strong understanding of data modeling, dimensional modeling, metadata management, data lineage, and master data concepts.