Senior Data Software Engineer/ AI Agents, Azure, Spark

Back to Search

Senior Data Software Engineer/ AI Agents, Azure, Spark

Remote in Georgia, & 4 others

Data Software Engineering

apply

FacebookLinkedInSend via email

Looking for something else?

Find a vacancy that works for you. Send us your CV to receive a personalized offer.

Find me a job

Location-specific conditions & benefits*

Choose an option

We are looking for an experienced Senior Data Software Engineer with strong hands-on expertise in Azure and the Spark ecosystem, primarily focused on building and maintaining data transformation pipelines.

The candidate should be comfortable with any Spark-adjacent technology (PySpark, Spark SQL, Scala Spark, Databricks, Synapse, etc.) rather than being locked into one specific flavor. Experience with Microsoft Fabric is a strong plus but not a requirement.

The ideal candidate combines great technical skills with leadership capability, contributing to architecture, design, development, and mentoring of engineering teams - preferably in complex enterprise or financial services environments.

Responsibilities

• Lead the design, development, and optimization of scalable data engineering solutions on Azure, using Spark-based processing (PySpark, Spark SQL, Scala, or equivalent)

• Own end-to-end data transformation pipelines, including ingestion, transformation, storage, and analytics

• Work with Azure-native data services such as Data Factory, Databricks, and Synapse

• Support high-performance data access patterns using Cosmos DB (NoSQL API) where applicable

• Collaborate with data scientists, AI engineers, and product stakeholders to enable data-driven analytics and insights

• Mentor and guide junior engineers, setting coding standards and best practices

• Ensure data quality, security, governance, and performance across platforms

• Contribute to technical decision-making and solution architecture discussions

Requirements

• 3+ years of experience in data engineering roles, preferably within complex enterprise or financial services environments

• Expertise in Azure cloud data services, including Data Factory, Databricks, and Synapse

• Proficiency in the Spark ecosystem, including PySpark, Spark SQL, and Scala Spark

• Background in designing and maintaining end-to-end data transformation pipelines covering ingestion, transformation, storage, and analytics

• Skills in ensuring data quality, security, and governance across large-scale platforms

• Capability to contribute to solution architecture and technical decision-making

• Competency in mentoring engineering teams and setting coding standards and best practices

• Understanding of high-performance data access patterns using Cosmos DB (NoSQL API)

• English proficiency at an Upper-Intermediate level (B2) or higher

Nice to have

• Hands-on experience with Microsoft Fabric and OneLake (Delta / OpenLake)

• Familiarity with financial instruments and financial services data

• Exposure to AI-assisted development tools such as GitHub Copilot and awareness of industry-standard LLMs

• Knowledge of Data Science fundamentals and collaboration experience with DS teams