Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

Back to Search

Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

Remote in Georgia, & 4 others

Data Software Engineering

apply

FacebookLinkedInSend via email

Looking for something else?

Find a vacancy that works for you. Send us your CV to receive a personalized offer.

Find me a job

Location-specific conditions & benefits*

Choose an option

We are looking for a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as Data Factory, Databricks, Event Hub, and Cassandra, to a cloud-agnostic platform deployable on-premises.

Responsibilities

• Implement streaming and batch Spark pipelines running on Kubernetes

• Lead the implementation of objects within the data generator framework

• Oversee deployment and testing across local and development environments

• Drive improvement of the current, actively changing solution

• Conduct bug investigation and resolution

• Perform unit testing to ensure solution quality

• Participate in refinement, planning, and demo sessions

• Guide and mentor team members throughout the implementation phase

Requirements

• 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes

• Proficiency in Python

• Familiarity with the data engineering domain and cloud-agnostic platform migrations

• Capability to dive deeper into new technologies in a short time during active implementation phases

• Ability to work independently with syncs with a team lead

• Strong communication skills and proactiveness

• Showcase of being a hands-on team player and technical leader

• Proficiency in English at a B2+ level

Nice to have

• Knowledge of Java

• Familiarity with Kafka Connect, KSQL, and Kafka Streams

• Skills in Ansible and Argo CD

• Understanding of TDD