Senior Data Engineer
Location: Glendale, CA (2-3 days/week) Type of Hire: Contract Rate: $90.60
As a Senior Data Engineer, you will be pivotal in transforming data into actionable insights. Collaborate with our dynamic team of technologists to develop cutting-edge data solutions that drive innovation and fuel business growth. You will own and operate the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines, platform governance, and solution architecture across AWS, Kubernetes, and Airflow. Your expertise will be essential in explaining Spark architecture, recommending best-fit Databricks solutions to stakeholders, and optimizing our data-driven decision-making processes. If you're passionate about leveraging data to make a tangible impact, we welcome you to join us in shaping the future of our organization.
Key Responsibilities:
Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python
Meet with stakeholders to gather requirements and translate them into scalable data platform solutions
Understanding of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations
Ability to explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutions
Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrations
Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy tooling
Build and maintain Kubernetes containers and containerized utilities supporting deployed data platform services
Apply networking knowledge to troubleshoot connectivity and integration errors across platform components
Perform platform administration: provision and remove access, assess resource utilization, monitor platform health and cost, and evaluate stakeholder requests
Collaborate with engineers, architects, and product managers to drive Core Data platform success; participate in agile/scrum ceremonies
Maintain documentation of platform changes, standards, and pipeline configurations to support data quality and governance
Qualifications:
5+ years of data engineering experience developing and operating large-scale data pipelines
Deep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala
Strong understanding of Spark architectureexecutors, stages, partitioning, shuffle, and performance tuningwith ability to explain tradeoffs to technical and non-technical stakeholders
Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting
Proficient in SQL with advanced performance tuning capabilities
Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines