Technology Lead | Cloud Integration | Azure Data Factory (ADF)

"Project Niche Skills"
"VD profiles are allowed"

Job Title: Technology Lead | Cloud Integration | Azure Data Factory (ADF)
Work Location: RedmondWA98052
####
Contract duration: 12
Target Start Date: 14 Sep 2026
Does this position require Visa independent candidates only? No
**hybrid work set-up**


Job Details:

Must Have Skills:
Python, SQL, (bonus: Scala), Apache Spark, PySpark, Spark SQL, Azure Databricks (notebooks, Jobs/Workflows, clusters), Azure Synapse, Delta Lake, Parquet, ADLS Gen2

Nice to have skills:
ETL/ELT, MERGE/upsert, partitioning, Spark performance tuning, Git, Azure DevOps / CI/CD

Detailed Job Description
Techniques (Data Engineering Practices)

ETL / ELT
Extract data from source systems
Transform/clean/reshape into usable form
Load into target tables (ELT = transform after loading, inside the platform)


MERGE / Upsert
Update existing rows when records already exist
Insert new rows when they don't
Handled in a single atomic operation (core Delta Lake pattern)


Partitioning
Split large tables by a column (e.g., date, source type)
Enables partition pruning scan only relevant data
Improves query speed and reduces cost


Spark Performance Tuning
Handle data skew and manage shuffles
Caching / persistence of reused datasets
Right-size partitions; use broadcast joins
Cluster sizing and resource optimizationorm

Minimum years of experience
8-10 years

Certifications Needed :No

Top 3 responsibilities you would expect the Subcon to shoulder and execute
Partitioning Split large tables by a column e.g., date, source type Enables partition pruning scan only relevant data Improves query speed and reduces cost
Spark Performance Tuning Handle data skew and manage shuffles Caching persistence of reused datasets Rightsize partitions use broadcast joins Cluster sizing and resource optimization
MERGE Upsert Update existing rows when records already exist Insert new rows when they dont Handled in a single atomic operation core Delta Lake pattern

Interview Process (Is face to face required?)
No

Any additional information you would like to share about the project specs/ nature of work

Project Code: Master project code for Accounts Plaform