Kafka Expert with Agentic AI Skills

Location: St. Louis, MO 63131 (Onsite)
Duration: 12 Months
Experience: 5 8 Years

Job Summary

We are seeking an experienced Kafka Expert with Agentic AI Skills to design, build, and optimize real-time streaming solutions and AI-powered autonomous workflows. The ideal candidate should have strong expertise in Apache Kafka, event-driven architectures, and hands-on experience with Agentic AI frameworks such as LangChain, AutoGen, CrewAI, or Semantic Kernel. The role involves building scalable data pipelines, managing Kafka infrastructure, and collaborating with AI and engineering teams to deliver high-performance solutions.

Required Skills

• 5 8 years of experience in software engineering or data engineering.
• Strong hands-on experience with Apache Kafka (Kafka Streams, Kafka Connect, Brokers).
• Experience designing, implementing, and managing Kafka-based streaming architectures.
• Expertise in Kafka performance tuning, scalability, reliability, upgrades, and migrations.
• Strong programming skills in Python or Java.
• Hands-on experience with Agentic AI and LLM frameworks such as LangChain, AutoGen, CrewAI, or Semantic Kernel.
• Experience building autonomous AI workflows and AI-driven decision systems.
• Strong understanding of event-driven architecture, microservices, and real-time data processing.
• Experience developing and maintaining real-time streaming/data pipelines.
• Ability to troubleshoot and monitor high-volume Kafka environments.

Preferred Skills

• Experience with AWS, Azure, or GCP.
• Familiarity with monitoring and analytics tools such as Grafana, Prometheus, ELK Stack, Datadog, or Splunk.
• Experience with Docker and Kubernetes.
• Knowledge of MLOps, AI deployment pipelines, RAG architectures, and vector databases.
• Experience integrating Kafka with cloud-native data services.
• Experience working in large-scale enterprise environments.

Key Responsibilities

• Design, implement, and manage Kafka-based streaming architectures.
• Build and optimize real-time data pipelines and event-driven applications.
• Develop and integrate Agentic AI solutions using modern LLM frameworks.
• Manage Kafka components including Kafka Connect, Schema Registry, MirrorMaker, Confluent Replicator, REST Proxy, and Kafka infrastructure.
• Perform Kafka upgrades, patching, migrations, and capacity planning.
• Monitor system performance, ensure scalability, reliability, and high availability.
• Create architecture diagrams, operational documentation, runbooks, and monitoring alerts.
• Collaborate with AI, data engineering, and application development teams to deliver scalable real-time solutions.