Company Description: IMCS Group is one of the fastest growing MWBE (Minority Woman Owned Enterprise) staffing firms in the U.S. We focus on bringing a Diversity Recruitment approach to Fortune 500 companies within North America and EMEA region contingent labor programs. IMCS Group excels in providing top talent in IT, Healthcare, Engineering, Finance, Light Industrial, Contact Center, and Clerical. We have offices in the US, Canada, Mexico, and India in which we can provide support to our clients.
Description: Job Title: Lead Data Engineer
Work Location & Reporting Address: Bellevue, WA 98006 (2 days a week in Office)
Vendor Rate: ***
Contract duration: 12 months
Target Start Date: 25 Sep 2026
Does this position require Visa independent candidates only? Yes
Interview Process (Is face to face required?): F2F or Team Video Call
Job Details:
Minimum years of experience required: 8+
Certification needed: NA
Must Have Skills: Azure Data factory, Azure Databricks
Nice to Have Skills: Scala, Python (PySpark), SQL
Detailed Job Description
Key Responsibilities
"Design, build, and maintain PySpark/SQL pipelines in Azure Databricks for batch and streaming data.
"Develop robust ingestion from Azure Data Lake Storage (ADLS Gen2), Azure Synapse/SQL, Event Hub, Kafka, and REST/JSON sources.
"Optimize Spark jobs (partitioning, caching, broadcast joins, AQE) for performance and cost.
"Implement monitoring and alerting (cluster/job metrics, driver/executor logs).
"Use Databricks Repos, notebooks, and modular PySpark projects with unit tests (pytest).
"Build CI/CD pipelines (e.g., Azure DevOps, GitHub Actions) for jobs, notebooks, and infrastructure-as-code (Terraform/ARM/Bicep).
"Manage environments (dev/test/prod), secrets/Key Vault, and configuration promotion.
Required Qualifications (Intermediate Level)
"6+ years in data engineering; 4+ years hands-on with Azure Databricks and Spark.
"Strong PySpark and SQL skills: DataFrames, joins, window functions, UDFs, incremental loads.
"Practical experience with Delta Lake, Unity Catalog, and Databricks Jobs/Workflows.
"Familiarity with Azure services: ADLS Gen2, Azure Key Vault, Event Hub, Azure SQL/Synapse.
"Version control (Git) and CI/CD experience; basic testing practices (pytest).
"Ability to optimize Spark jobs and troubleshoot: skew, shuffle, OOM, driver/executor tuning.
"Solid understanding of data modeling (star schema, medallion/lakehouse), partitioning, and file formats (Parquet/JSON).
"Airflow, Azure Data Factory orchestration.
"Terraform for Databricks & Azure resources.
"Basic Scala and/or SQL Warehouses (Databricks SQL) for BI.
Education
"Bachelor s/Master s in Computer Science, Engineering, or related field (or equivalent experience).
Certifications (Optional but Valued)
"Databricks: Data Engineer Associate/Professional
"Microsoft Azure: DP-203 (Data Engineering on Microsoft Azure), AZ-900 (Fundamentals)
Tools & Tech Stack (Typical)
"Languages: Python (PySpark), SQL
"Databricks: Notebooks, Jobs/Workflows, Repos, Unity Catalog, Delta Lake, MLflow
"Azure: ADLS Gen2, Key Vault, Event Hub, Synapse/SQL, Monitor/Log Analytics
"DevOps: Git, Azure DevOps/GitHub Actions, Terraform/Bicep
Custom Fields:
Name: Intake Call Requested
Value: true
Name: Intake Call Completed
Value: true
EEOC Disclaimer: IMCS Group is an equal opportunity employer and makes employment decisions without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability status, age, or any other status protected by law. IMCS Group is committed to maintaining an inclusive workforce, where employees are hired, retained, compensated, and promoted based on their contributions to our Company.