Senior Databricks AI Engineer
Atlanta, GA - Onsite
Contract
$60
Databricks, PySpark, Delta Lake, Medallion Architecture (Bronze Layer/Silver Layer/Gold Layer), Python, Machine Learning, LLM, RAG, LangChain / LangGraph, Cloud (Azure Databricks, Azure Storage (ADLS), Azure Data Factory (ADF)), SQL, DevOps, Git & CI/CD
Position Summary
We are seeking a highly skilled Senior Databricks AI Engineer to design, develop, and deploy scalable data and AI solutions using the Databricks Lakehouse Platform. The ideal candidate will have expertise in Data Engineering, Machine Learning, Generative AI, and cloud technologies, with hands-on experience building enterprise-grade AI applications and data platforms.
Key Responsibilities
- Data Engineering
- Design and develop scalable data pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.
- Build ELT/ETL solutions for structured and unstructured data.
- Implement Medallion Architecture (Bronze, Silver, Gold layers).
- Optimize Databricks jobs, clusters, and workflows for performance and cost efficiency.
- Manage data ingestion from multiple cloud and on-premise sources.
- AI & Machine Learning
- Develop and deploy Machine Learning models using Databricks ML and Python.
- Implement MLOps practices for model training, deployment, monitoring, and governance.
- Build predictive analytics solutions and recommendation engines.
- Create feature stores and reusable ML pipelines.
- Generative AI
- Design and develop GenAI applications using:
- Large Language Models (LLMs)
- Retrieval Augmented Generation (RAG)
- Vector Databases
- AI Agents and Multi-Agent Frameworks
- Integrate Databricks Mosaic AI, LangChain, LangGraph, and OpenAI/Azure OpenAI services.
- Fine-tune and evaluate foundation models.
- Develop enterprise chatbots, knowledge assistants, and intelligent automation solutions.
- Cloud & Architecture
- Implement solutions on Azure, AWS, or GCP.
- Design secure and scalable Lakehouse architectures.
- Work with Unity Catalog, Delta Sharing, and data governance frameworks.
- Collaborate with business stakeholders, architects, and data scientists to deliver AI-driven business solutions.
Required Skills
- Databricks Lakehouse Platform
- PySpark, Spark SQL
- Python
- Delta Lake
- Databricks Workflows
- Databricks Unity Catalog
- Azure Data Factory / AWS Glue
- SQL and Data Modeling
- Git, CI/CD, DevOps
- AI & GenAI Skills
- Machine Learning
- Deep Learning
- LLMs
- RAG Architecture
- Prompt Engineering
- LangChain / LangGraph
- Vector Databases (FAISS, Pinecone, ChromaDB)
- Databricks Mosaic AI
- MLflow
- Cloud Platforms
- Microsoft Azure (Preferred)
- AWS
- Google Cloud Platform
Qualifications
- Bachelor''s or Master''s degree in Computer Science, Data Science, Engineering, Information Technology, or related field.
- 6+ years of experience in Data Engineering.
- 3+ years of experience with Databricks.
- Experience delivering AI/ML or Generative AI solutions in production environments.
- Databricks Certified Data Engineer or Machine Learning certification preferred.
Preferred Experience
- Healthcare, Life Sciences, Banking, Retail, or Manufacturing domain experience.
- Experience with Azure OpenAI Service.
- Knowledge of Agentic AI and AI Governance.
- Experience in enterprise-scale data modernization programs.
Key Deliverables
- Scalable Databricks data pipelines.
- Enterprise AI and GenAI solutions.
- Real-time analytics and AI-powered insights.
- AI-enabled automation and decision-support systems.
- Secure and governed Lakehouse architecture.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.