Job Title: Technology Lead Cloud Integration | Azure Data Factory (ADF)
Location: Houston, TX 77079
Contract Duration: 6 Months
Experience: 8 10 Years
Certification: Not Required
Job Summary
Seeking an experienced Azure Data Engineer with strong expertise in Azure Data Factory, Azure Databricks, PySpark/Scala, and SQL. The candidate will design, develop, and optimize scalable data pipelines and cloud-based data integration solutions.
Mandatory Skills
Strong experience in Azure Data Factory (ADF), including pipelines, data flows, triggers, integration runtimes, parameterization, and error handling.
Hands-on experience with Azure Databricks, notebooks, jobs/workflows, PySpark/Scala, and Delta Lake.
Advanced SQL skills, including CTEs, window functions, query optimization, and large dataset processing.
Proficiency in Python or Scala, with experience in modular coding and unit testing.
Experience with ADLS Gen2, Delta Lake, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.
Knowledge of Git, branching strategies, CI/CD, and environment promotion for ADF and Databricks.
Nice to Have
Key Responsibilities
Design, develop, and maintain Azure Data Factory pipelines and Databricks workflows.
Build and optimize data transformation solutions using PySpark/Scala and SQL.
Perform coding, development, testing, and performance tuning.
Manage CI/CD deployments across environments.
Conduct UI testing and coordinate with stakeholders to deliver data integration solutions.