DataBricks Data Engineer

Location: New York, NY (Hybrid – 4 days onsite per week - Tues/Wed anchor days)
Job Type: Contract (6 Months)
Schedule: Monday – Friday, 8:00 AM – 5:00 PM

About the Opportunity

We are seeking an experienced DataBricks Data Engineer to join a growing team supporting a new enterprise data initiative. This role is responsible for designing, developing, and optimizing scalable data engineering solutions using Databricks to support analytics, reporting, enterprise data pipelines, and Sales Incentive Compensation (SIC) processes.

The ideal candidate has strong experience with Databricks, modern cloud data platforms, ETL/ELT development, and enterprise data architecture, along with the ability to partner with business stakeholders to deliver reliable, high-performing data solutions.

Responsibilities

• Design, develop, and maintain scalable data pipelines using the Databricks platform.
• Build and optimize Delta Lake tables, Delta Live Tables, Unity Catalog, notebooks, workflows, and enterprise data models.
• Design data architectures that support reporting, analytics, and enterprise business processes.
• Develop ETL/ELT solutions to ingest, transform, validate, and deliver high-quality data.
• Support Sales Incentive Compensation (SIC) processes, including quota management, attainment calculations, commission processing, and payout reporting.
• Partner with Sales Operations, Finance, IT, and business stakeholders to translate business requirements into scalable technical solutions.
• Maintain and optimize existing Databricks environments by improving automation, integration, data quality, and overall performance.
• Troubleshoot, test, and resolve issues within data pipelines while ensuring reliability and scalability.
• Integrate Databricks with cloud platforms and enterprise data sources.
• Create and maintain technical documentation for data models, pipeline architecture, and development standards.
• Stay current with Databricks capabilities and recommend improvements that enhance performance and efficiency.

Required Qualifications

• Minimum of 7 years of data engineering experience.
• At least 3 years of hands-on Databricks development experience.
• Strong experience with:
  • Databricks
  • Delta Lake
  • Delta Live Tables
  • Unity Catalog
  • Databricks Workflows
• Advanced SQL skills.
• Strong programming experience with Python and/or Scala.
• Experience building ETL/ELT pipelines, enterprise data models, and scalable data architectures.
• Experience integrating Databricks with Azure, AWS, or Google Cloud Platform.
• Experience working with enterprise data integration technologies such as Azure Data Factory, Event Hubs, Kafka, or similar tools.
• Experience supporting Sales Incentive Compensation (SIC) processes, including incentive plans, quota management, commission calculations, attainment tracking, and payout processing.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work independently while collaborating effectively within a team environment.

Preferred Qualifications

• Experience with incentive compensation platforms such as Anaplan, Oracle ICM, Varicent, or similar solutions.
• Databricks Certified Data Engineer Associate or Professional certification.
• Experience designing enterprise-scale cloud data solutions.

Key Skills

• Databricks
• Python
• SQL
• Apache Spark
• ETL / ELT
• Delta Lake
• Data Modeling
• Cloud Data Platforms
• Azure Databricks
• Data Engineering

Interview Process

• 2–3 interview process
• First interview: Virtual via Zoom
• Second interview: Onsite
• Final interview may be conducted based on business needs.

What We're Looking For

We're looking for a collaborative Data Engineer who enjoys solving complex data challenges and building scalable, enterprise-grade data solutions. The ideal candidate is a team player with strong technical expertise, excellent communication skills, and a passion for delivering high-quality, reliable data platforms in a fast-paced environment.