Sr. Data Engineer- Databricks

Prolim is hiring for a Data Engineer- Databricks for aleading client

Location: Austin, TX
Contract: Long term

I.  DESCRIPTION OF SERVICES

8 or more years of experience, relies on experience and judgment to plan and accomplish goals, independently performs a variety of complicated tasks, a wide degree of creativity and latitude is expected.

Understands business objectives and problems, identifies alternative solutions, performs studies and cost/benefit analysis of alternatives. Analyzes user requirements, procedures, and problems to automate processing or to improve existing computer system: Confers with personnel of organizational units involved to analyze current operational procedures, identify problems, and learn specific input and output requirements, such as forms of data input, how data is to be; summarized, and formats for reports. Writes detailed description of user needs, program functions, and steps required to develop or modify computer program. Reviews computer system capabilities, specifications, and scheduling limitations to determine if requested program or program change is possible within existing system.

The Databricks Engineer should design, develop, and optimize scalable data solutions on Databricks, leveraging PySpark or Scala for large-scale data processing. Build and maintain ingestion pipelines, Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting. Develop robust data models and implement data quality, validation, and governance frameworks. Create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights. Optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency.

II.  CANDIDATE SKILLS AND QUALIFICATIONS

Required Qualifications
8+ Years of general IT experience supporting the design, development, deployment, or delivery of technology solutions.
8+ Years of experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
8+ Years of experience in data warehousing and dimensional data modeling using star and snowflake schemas.
8+ Years of experience using SQL and Python (or Scala) for large-scale data processing.
8+ Years of experience designing and developing dashboards and applications natively within Databricks (such as Databricks SQL dashboards and Databricks Apps).
8+ Years of experience implementing data governance, data quality, and data security practices.
8+ Years of experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) to build and manage production data pipelines.
8+ Years of experience with Delta Lake, medallion architecture (Bronze/Silver/Gold), data lakehouse design, and scheduling offline jobs with Lakeflow Jobs (formerly Databricks Workflows) or Airflow.
8+ Years of experience demonstrating excellent verbal and written communication skills when presenting insights to technical and business stakeholders.

Preferred Qualifications
1+ Year of preferred experience working in public sector or state government environments.
1+ Year of preferred experience with a Databricks certification (such as Databricks Certified Data Engineer Associate or Professional).
1+ Year of preferred experience with CI/CD practices and Git-based workflows for data pipelines.