Job Title: Data Engineer with Databricks & AWS
Location: Santa Clara, CA Onsite Need only Locals
12+ Months Contract
Position Summary
We are seeking a Senior Data Engineer with strong experience designing, developing, and supporting scalable data pipelines using Databricks and AWS. The engineer will support business-critical data solutions across supply chain, capacity planning, and global sourcing.
The ideal candidate will have hands-on experience with S3 ingestion, Databricks, Delta/Iceberg tables, ETL frameworks, data quality, SQL/Athena validation, CI/CD, production troubleshooting, and data-layer architecture.
Key Responsibilities
- Design, develop, and support scalable data pipelines in Databricks and AWS.
- Build and maintain data ingestion pipelines using Amazon S3.
- Onboard new data feeds and perform schema validation and data quality checks.
- Develop and manage Bronze, Silver, and Gold data layers.
- Design and maintain Delta Lake / Apache Iceberg tables.
- Enhance ETL frameworks and improve pipeline reliability and scalability.
- Perform data validation using SQL and Amazon Athena.
- Troubleshoot production data pipeline issues and ensure timely resolution.
- Monitor production pipelines and datasets using Splunk.
- Implement and support CI/CD deployments for data engineering workflows.
- Support migration of legacy Hive workloads to unified ODP platforms.
- Maintain business-critical reporting datasets and ensure data accuracy and availability.
- Collaborate with business stakeholders, data engineering teams, and technical teams to understand requirements and deliver reliable data solutions.
Required Skills
- Strong hands-on experience as a Senior Data Engineer / Data Pipeline Engineer.
- Expertise in Databricks and AWS.
- Strong experience developing scalable ETL/data pipelines.
- Experience with Amazon S3 and cloud-based data ingestion.
- Experience with Delta Lake and/or Apache Iceberg.
- Strong SQL skills and experience with Amazon Athena.
- Experience with Bronze/Silver/Gold data architecture.
- Experience with schema validation and data quality frameworks.
- Production troubleshooting and incident resolution experience.
- Experience with CI/CD and deployment automation.
- Experience with Splunk or similar monitoring/logging tools.
- Experience working with business-critical datasets and reporting pipelines.
Preferred / Nice-to-Have
- Experience with Hive-to-lakehouse/platform migrations.
- Experience supporting supply chain, capacity planning, or global sourcing data use cases.
- Experience with unified ODP/data platform environments.
- Strong stakeholder management and communication skills.
Thanks
Sri Vardhan Chilakamukku
Infobahn SoftWorld Inc.