Note: The ideal candidate is not simply a traditional Databricks Data Engineer. We are looking for someone who can combine: Data Engineering + Azure Databricks + Python/PySpark + Generative AI + LLMs + AI Agents + Automation
Must-Have Skills:
Azure Databricks | Azure Cloud | Python | SQL | PySpark | LLMs | Generative AI | AI Agents | Prompt Engineering | CI/CD | Data Automation
Responsibilities & Skills:
Design, develop, and deploy intelligent AI agents that automate data engineering workflows within Azure Databricks.
Build LLM-powered agents for code generation, code review, refactoring, optimization, and troubleshooting of Python, PySpark, and SQL workloads.
Develop AI-driven solutions to automate data quality validation, testing, reconciliation, and QA processes.
Build agents to support metadata management, data lineage, governance, and compliance automation within Databricks.
Develop intelligent automation for CI/CD pipelines, including test-case generation, deployment validation, release checks, and rollback processes.
Implement AI-powered monitoring and automation for data pipeline failures, anomalies, and root-cause analysis.
Develop reusable prompt engineering frameworks, agent patterns, tools, and orchestration workflows for data engineering use cases.
Integrate LLMs and AI agents with Databricks, Azure services, APIs, data platforms, and enterprise workflows.
Work with Data Engineers, Data Architects, QA teams, and Platform teams to identify high-value automation opportunities.
Establish standards and best practices for AI agent development, testing, deployment, observability, security, and governance.
Build production-grade automation solutions with appropriate logging, error handling, monitoring, testing, and documentation.
Evaluate emerging Generative AI, agentic AI, and automation frameworks and recommend technologies that provide measurable business or engineering value.
Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related technical discipline. Master's degree is a plus.
10+ years of experience in data engineering, software engineering, cloud data platforms, or related technical roles.
Strong hands-on experience with Azure Databricks and modern cloud data platforms.
Proven experience developing AI/LLM-powered automation, intelligent agents, or Generative AI solutions.
Strong programming skills in Python and SQL with experience developing production-quality applications.
Strong hands-on experience with PySpark and distributed data processing.
Experience with Databricks notebooks, Jobs/Workflows, Delta Lake, and Databricks Asset Bundles.
Experience with LLM APIs, prompt engineering, agent orchestration, and AI frameworks.
Experience with technologies/frameworks such as Azure OpenAI, OpenAI, LangChain, AutoGen, Databricks Agent Framework/Agent Bricks, or comparable agentic AI technologies.
Hands-on experience with CI/CD and automation, including GitHub Actions and/or Azure DevOps Pipelines.
Experience building data quality, testing, QA automation, or data validation frameworks.
Strong understanding of software engineering principles including Git, testing, version control, documentation, code quality, and observability.