DataBricks Data Engineer

Location: New York - 1166

Introduction

We are seeking a skilled and experienced professional to join our team, responsible for designing, developing, and implementing data engineering solutions using the Databricks platform. Your work will support data ingestion, transformation, performance reporting, analytics, sales incentive compensation processing, and enterprise data pipeline processes.

Required Skills & Qualifications

• At least 7 years of experience in data engineering, with a minimum of 3 years hands-on experience building and managing Databricks solutions.
• Extensive experience with Databricks platform components: Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
• Strong proficiency in Python and/or Scala for data engineering; SQL expertise required.
• Strong experience with core data engineering practices including data ingestion, transformation (ETL/ELT), data modeling, and pipeline orchestration.
• Functional and technical experience with Sales Incentive Compensation (SIC) — including incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing.
• Applicants must be able to work directly for Artech on W2.

Preferred Skills & Qualifications

• Familiarity with SIC platforms (e.g., Anaplan, Oracle ICM, Varicent) and experience integrating them with data engineering platforms is a strong plus.
• Experience integrating Databricks with cloud platforms (Azure, AWS, or GCP) and data sources such as Azure Data Factory, Event Hubs, Kafka, or equivalent.
• Databricks Certified Data Engineer Associate or Professional certification is a plus.

Day-to-Day Responsibilities

• Collaborate with business stakeholders to understand their data requirements and translate them into scalable technical solutions.
• Design and develop Databricks-based data pipelines, including data models, Delta Lake tables, and dashboards/notebooks.
• Maintain and optimize all current Databricks workspaces and pipelines; capture and deliver requirements for new or improved functionality.
• Design and implement data pipelines to support Sales Incentive Compensation (SIC) processes, including quota management, attainment calculations, commission processing, and payout reporting.
• Provide technical support and troubleshooting for Databricks-based solutions, working closely with end-users and analysts to address their needs and resolve any data-related problems.

Company Benefits & Culture

• Opportunity to work on a new team and project.
• Collaborative and supportive team environment.
• Exposure to cutting-edge data engineering technologies and practices.

For immediate consideration, please click APPLY to begin the screening process with Alex.