Amazon Web Services (AWS), Artificial Intelligence (AI), Automation, Cisco Unity, Continuous Deployment/Delivery, Continuous Integration, Data Analysis, Data Management, Data Modeling, Data Quality, Database Extract Transform and Load (ETL), Dimensional Modeling, Distributed Computing, Engineering, GCP (Good Clinical Practices), Git, Microsoft Windows Azure, Performance Tuning/Optimization, Programming Tools, Quality Monitoring, SQL (Structured Query Language), Scalable System Development
About the Role
We are seeking an experienced Senior Data Engineer with deep expertise in Databricks to design, develop, and optimize enterprise-scale data pipelines. You'll play a key role in building scalable Lakehouse solutions while ensuring reliable, high-quality data for analytics, reporting, and downstream applications.
Key Responsibilities
- Design and build scalable ETL/ELT pipelines using Databricks
- Develop PySpark and Spark SQL transformations
- Build Bronze, Silver, and Gold data layers using Medallion Architecture
- Implement Delta Live Tables, Workflows, and Unity Catalog
- Optimize Spark jobs for scalability and performance
- Design dimensional data models for analytical workloads
- Implement data quality validations and monitoring
- Collaborate with analytics, engineering, and architecture teams
- Build reusable frameworks and automation
- Support CI/CD deployment pipelines and Git-based development
Required Qualifications
- 7–10+ years of Data Engineering experience
- Extensive hands-on Databricks experience
- Strong PySpark programming
- Advanced Spark SQL skills
- Experience with Delta Lake
- Experience implementing Medallion Architecture
- Hands-on Delta Live Tables
- Unity Catalog experience
- Strong SQL optimization and performance tuning
- Experience with AWS, Azure, or GCP
- CI/CD and Git experience
- Strong understanding of distributed data systems
- Experience designing enterprise data models
Preferred Qualifications
- Databricks certification
- Experience with streaming data
- Experience building reusable data frameworks
- Experience with enterprise governance
- Exposure to AI-assisted development tools including Databricks AI Genie
#LI-WJ1
#PSI2