Databricks Consultant

Job Title: Databricks Consultant
Location: Houston, MS — Hybrid (2–3 days)
Duration: 6 months
GBaMS ReqID: 10926146
Experience: 7+ years

Role Summary

• Design, develop, and implement scalable data solutions using the Databricks platform.
• Apply strong expertise in:
  • Data Engineering
  • Cloud Platforms
  • Apache Spark
  • Modern Data Architectures
• Collaborate closely with business and technical stakeholders to understand requirements and deliver effective solutions.

Key Responsibilities

• Design and develop data pipelines using:
  • Databricks
  • Apache Spark
  • PySpark
• Build and maintain ETL/ELT processes for large-scale data ingestion and transformation.
• Implement Data Lake and Lakehouse architectures on Databricks.
• Integrate data from multiple sources, including:
  • Databases
  • APIs
  • Cloud storage
• Optimize Spark jobs and Databricks workloads for:
  • Performance
  • Scalability
  • Cost efficiency
• Collaborate with:
  • Business users
  • Solution architects
  • Developers
  • Technical stakeholders
• Develop data models to support:
  • Reporting
  • Analytics
  • AI/ML initiatives
• Implement standards for:
  • Data quality
  • Data governance
  • Data security
  • Data monitoring
• Support production deployments.
• Perform troubleshooting and performance tuning.
• Prepare technical documentation.
• Provide knowledge transfer and technical guidance to project teams.

Required Skills

• Strong hands-on experience with Databricks.
• Expert-level knowledge of:
  • Apache Spark
  • PySpark
  • SQL
• Experience with cloud platforms such as:
  • Azure
  • AWS
  • GCP
• Strong knowledge of:
  • Delta Lake
  • Unity Catalog
  • Databricks Workflows
• Experience with:
  • Data Warehousing
  • Data Lakes
  • Lakehouse architecture
• Strong understanding of:
  • ETL/ELT frameworks
  • Data modeling
• Familiarity with:
  • CI/CD
  • Git
  • DevOps practices
• Excellent problem-solving and communication skills.

Preferred Skills

• Experience with:
  • Azure Data Factory (ADF)
  • Azure Synapse
  • AWS Glue
• Knowledge of:
  • Machine Learning on Databricks
  • MLOps on Databricks
• Experience with streaming technologies:
  • Apache Kafka
  • Structured Streaming
• Databricks or cloud platform certifications.

Qualifications

• Bachelor’s degree in:
  • Computer Science
  • Information Technology
  • Engineering
  • Related technical field
• 5+ years of experience in:
  • Data Engineering
  • Big Data
• 2+ years of hands-on Databricks implementation experience.

Nice to Have

• Databricks Certified Data Engineer Associate/Professional certification.
• Experience in industry domains such as:
  • Utilities
  • Energy
  • Banking
  • Healthcare
  • Retail
• Experience with Agile delivery methodologies.

Primary Technologies

• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Azure
• AWS
• GCP
• Azure Data Factory (ADF)
• Apache Kafka
• Git
• CI/CD