Azure Databricks Data Engineer

Azure Databricks Data EngineerOnly qualified Azure Databricks Data Engineer candidates located near Atlanta, GA to be considered due to the position requiring an onsite presence.Education: Bachelor's degree in computer science, Information Technology, or related field. Databricks and Azure Certifications (Strongly preferred).Required Skills, Experience, & Abilities:Strong hands-on knowledge of Azure Databricks and the Azure platform, with proven experience operating a production Lakehouse at scale.Proficiency in Databricks SQL, Python, and other relevant programming languages.Deep Azure knowledge: ADLS Gen2, Key Vault, Networking (VNet/Private Endpoints), Event Hub.Deep understanding of advanced data warehouse concepts (dimensional modeling, SCD, surrogate key management) is required.Experience with data modeling, dimensional modeling, and temporal data structures in an operational ETL environment.Unity Catalog security & lineage.Strong business problem framing and stakeholder communication.Experience establishing engineering standards, performing design/code reviews, and mentoring data engineers.Excellent problem-solving and communication skills.Ability to work independently and collaboratively in a fast-paced environment.Responsibilities:Act as the technical design authority for an existing Databricks ETL team, working alongside customer business, development, Databricks, and DevOps teams.Support and evolve the production Lakehouse architecture: Delta medallion layers, Databricks SQL Warehouses, Unity Catalog security, and data lineage.Define and enforce design standards, reusable patterns, and best practices across the existing pipeline estate and all new development.Design new ingestion, streaming, and batch pipelines and review team designs against the medallion (bronze/silver/gold) architecture.Drive continuous optimization of the running platform: cluster sizing, autoscaling, Photon usage (if decided), query performance, Delta Liquid Clustering/Z-Ordering, VACUUM strategies, and DBU cost governance.Translate business questions into analytical models, KPIs, and reusable semantic layers.Mature CI/CD (Azure DevOps/GitHub), observability, and security (RBAC, row/column-level policies) across existing pipelines.Evaluate and drive adoption of new Databricks capabilities (Lakeflow, Delta Live Tables, serverless compute, AI/BI) where they add value.Conduct architectural reviews, code audits, and mentoring sessions to raise the team's engineering bar and ensure adherence to standards and scalability.Guide the team in version control, automated testing, and release discipline for Databricks notebooks, jobs, and workflows.**Only those lawfully authorized to work in the designated country associated with the position will be considered.****Please note that all Position start dates and duration are estimates and may be reduced or lengthened based upon a client's business needs and requirements.**