Location: position will sit on-site in either Warren, MI or Austin, TX 4 days per week
Please only submit candidates that are able to work on-site 4 days per week in either Austin or Warren.
Please make sure candidates are comfortable with conversion to full time employment after 6 months on contract.
We are needing candidates who have strong experience in Databricks on either Azure or GCP. Candidates must have very strong, hands on experience with Databricks, PySpark and SQL.
We are augmenting an internal team with contract contractors who have strong Azure Databricks and ML data platform skills. Our staff already owns platform standards, Unity Catalog governance, and architecture; we need additional capacity to deliver lakehouse pipelines, catalogued engineering and enterprise datasets, and GenAI-ready data products (embeddings, retrieval, batch enrichment) in parallel with other priorities. Assignments may span multiple programs not a single fixed product so candidates should be comfortable ramping on new domains and codebases quickly.
Responsibilities
- Design and build production data pipelines on Databricks (PySpark/SQL, Delta Lake, Unity Catalog, Workflows/Jobs) with idempotency, testing, monitoring, and clear lineage.
- Model bronze / silver / gold layers and data contracts (schemas, SLAs, quality checks) for analytics and downstream applications.
- Develop ML-oriented data products: feature stores or curated tables, embedding generation, chunking/indexing for RAG, batch classification/enrichment, and evaluation of retrieval/data quality.
- Integrate with Azure services (ADLS Gen2, Key Vault, managed identities, private connectivity) following enterprise security and compliance patterns.
- Partner with application and ML teams to expose stable APIs, tables, and volumes that power agentic workflows (tool-backed assistants, multi-step orchestration) without owning full UI or real-time serving unless scoped.
- Document data dictionaries, runbooks, and handoff materials; participate in code review and agile ceremonies with full-time staff.
Required qualifications
- 5+ years in data engineering, including 2+ years production Databricks on Azure.
- Strong Python (PySpark, pandas); SQL; experience operationalizing jobs at scale.
- Hands-on Delta Lake and Unity Catalog (schemas, grants, environments, promotion patterns).
- Experience building pipelines that feed ML/GenAI use cases: embeddings, vector or hybrid search inputs, labeled datasets, batch inference outputs.
- Git, CI-friendly repo layout (notebooks + packages), unit/integration testing for data pipelines.
- Strong communication; effective in staff augmentation models (embedded with internal leads, shared standards).