Senior Python PySpark Data Engineering Lead AI & Data Platforms

Job Title

Job Description Required Experience, Knowledge & Skills: Eight plus years of overall experience in large-scale application development with a recent mandatory platform for the secure and scalable deployment of AI agents into application contexts. Minimum of five plus years of proven experience in a Python and PySpark engineering lead role focused on building enterprise-grade, high-volume ELT/ETL processes using PySpark. Hands-on experience with YAML, JSON, FAST API or Spring Boot, GitHub Copilot. Proven experience developing and automating microservice integrations to support data-intensive applications. Proficiency in at least one programming language commonly used for data analytics and engineering, such as Python or Scala. Strong SQL skills and experience with various relational databases. Deep understanding of data modeling, data warehousing concepts, Data Mesh architecture, and data federation. Excellent communication, collaboration, and problem-solving skills.

Responsibilities: Design, develop, and maintain scalable, enterprise-grade AI agents, supporting ELT/ETL processes to handle large data volumes using Python, FAST API, Microservices, PySpark, Kafka, and the Databricks ecosystem. Develop, deploy, and automate microservice integrations to support data-intensive applications, ensuring scalability, resilience, and maintainability using cloud native infrastructure and OpenShift or Kubernetes architecture, including CI/CD pipelines. Ensure data quality, integrity, and security throughout the entire data lifecycle. Contribute to the continuous improvement of data engineering processes, standards, and best practices within the team.

**Only those lawfully authorized to work in the designated country associated with the position will be considered.**

**Please note that all Position start dates and duration are estimates and may be reduced or lengthened based upon a client's business needs and requirements.**