Azure Data Engineer Databricks / PySpark / AI
Location: Chicago, IL Onsite
Experience: 9+ Years
Employment Type: Contract
Job Summary
We are looking for an experienced Azure Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, SQL, and Azure cloud technologies. The ideal candidate should have strong experience in data engineering, data warehousing, data modeling, and cloud migration projects.
One of the resources should also have exposure to AI-driven data pipelines, Machine Learning, and Recommendation Systems to support AI/ML initiatives.
Required Skills
- 9+ years of overall IT/industry experience with strong cloud technology experience.
- 8+ years of hands-on experience with:
- Azure Databricks
- PySpark
- Azure SQL
- SQL development
- Strong hands-on experience in Azure Databricks development.
- Strong expertise in PySpark and SQL development.
- Experience with Data Warehousing and Data Modeling concepts.
- Experience developing and integrating cloud-based data solutions.
- Strong understanding of ETL/ELT data pipelines and data engineering processes.
- Experience working on cloud migration projects, preferably Azure migration.
- Experience working in Agile methodology and offshore/onshore delivery models.
- Strong SQL skills, including complex queries, optimization, joins, CTEs, stored procedures, and performance tuning.
- Ability to gather requirements, understand business needs, and participate in technical/design discussions.
- Strong communication and collaboration skills.
AI / Machine Learning Preferred
At least one candidate should have experience with:
- AI-driven data pipelines
- Machine Learning data preparation and pipelines
- Recommendation Systems / Recommendation Engines
- Feature engineering and ML data processing
- Integration of ML models with data pipelines
- Exposure to Azure ML or similar cloud-based ML platforms is a plus.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
- Develop efficient SQL-based data solutions using Azure SQL.
- Build and maintain data warehouse and data lake solutions.
- Perform data transformation, cleansing, integration, and validation.
- Develop and optimize Databricks notebooks and PySpark jobs.
- Support cloud migration and modernization initiatives.
- Collaborate with architects, developers, business teams, and offshore/onshore teams.
- Gather business and technical requirements and translate them into scalable data solutions.
- Participate in solution design, development, testing, deployment, and production support.
- Support AI/ML initiatives by preparing data pipelines and datasets for Machine Learning and recommendation use cases.
- Troubleshoot and optimize data pipelines for performance, reliability, and scalability.