Our client is seeking a skilled and passionate Databricks Engineer to design and build scalable, cloud-based data pipelines for enterprise analytics platforms. The role focuses on creating robust ETL/ELT workflows using Databricks and Spark, with opportunities to leverage advanced features such as Delta Live Tables (DLT) and cloud-native tools. The ideal candidate has strong hands-on experience in PySpark and SQL, and a proven ability to build pipelines from the ground up while collaborating with cross-functional teams.KEY RESPONSIBILITIESDesign and develop scalable batch and streaming data pipelines in Databricks using PySpark, SQL, and Delta Lake.Build ETL/ELT processes to transform and integrate data from diverse sources into unified analytical datasets.Leverage Delta Live Tables (DLT) and Databricks Workflows to automate and orchestrate pipelines (preferred).Optimize pipelines for performance, cost-efficiency, and reusability by applying Spark tuning and partitioning strategies.Collaborate with architects, data modelers, and analysts to implement business logic into pipelines.Implement data quality checks, monitoring, and observability practices across pipelines.Participate in Agile delivery processes, including sprint planning, code reviews, and deployments.Work with cloud platform teams to ensure proper storage, networking, and security setup.Deliver high-quality outputs, meet deadlines, and maintain strong client relationships.Provide consultative guidance, seek creative solutions, and proactively identify opportunities for improvement.Stay current with evolving technology trends and best practices in data engineering.REQUIRED SKILLS & EXPERIENCE5+ years of experience in data engineering, including at least 3+ years with Databricks.Strong hands-on experience with Apache Spark and/or Delta Live Tables (DLT).Proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.Experience designing and implementing ETL/ELT workflows in cloud environments (Azure or AWS).Familiarity with Delta Lake, Unity Catalog, and Databricks Jobs/Workflows.Experience with structured and semi-structured data from APIs, cloud storage, and enterprise systems.Strong understanding of modern data architecture concepts (Medallion architecture, lakehouse patterns).Knowledge of Git-based version control, CI/CD practices, and infrastructure automation.Excellent problem-solving and communication skills, particularly in client-facing settings.Bachelor's degree in Computer Science, Information Systems, or equivalent experience.PREFERRED QUALIFICATIONSExperience with Delta Live Tables (DLT) for declarative pipeline development.Exposure to IDMC (Informatica Intelligent Data Management Cloud) or other integration platforms.Databricks or cloud provider certifications (Azure/AWS).Familiarity with data governance tools (Unity Catalog, Purview, Collibra).Experience with REST APIs and streaming technologies (Kafka, Event Hubs, etc.).CERTIFICATIONSDatabricks Certified Data Engineer or equivalent cloud certifications (preferred).EDUCATIONBachelor's degree in computer science, Information Systems, or a related field (Master's degree preferred).#J-18808-Ljbffr
| Location | Houston, TX |