Job Summary (Azure Databricks Architect Oil & Gas Industry, Houston TX Onsite):
- Lead the design, development, and optimization of enterprise-scale data platforms and analytics solutions on Azure Databricks.
- Drive and implement data engineering best practices across the team.
- Architect and develop scalable data pipelines and ETL/ELT frameworks using Azure Databricks and PySpark.
- Optimize data ingestion, transformation, and consumption layers for performance and reliability.
- Design data models, data dictionaries, and reusable data products to support business needs.
- Ensure data governance, data quality, security, and compliance standards are met.
- Optimize Spark workloads, Databricks jobs, and overall platform performance.
- Collaborate with business stakeholders, data scientists, and cross-functional teams to translate business requirements into technical solutions.
- Mentor and guide data engineers and junior team members in best practices and technical solutions.
- Drive CI/CD automation and DevOps best practices for data platforms.
- Support Data Lakehouse architecture and enterprise analytics initiatives.
- Utilize and manage tools including Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Delta Lake, Databricks Workflows, Azure DevOps, SQL/T-SQL, and Python.
- Lead enterprise-scale Databricks implementations and cloud data transformation programs.
- Experience with performance tuning, automation frameworks, and technical governance.
- Preferred exposure to Unity Catalog, Airflow, Power BI, Lakehouse Architecture, DataOps, and MLOps.
- Strong stakeholder management and client-facing communication skills.
- Minimum 10 years of experience in Data Engineering/Data Platform delivery, with strong leadership experience in Azure-based data solutions, preferably in the Oil & Gas sector.