Senior Data Engineer - IoT
Location: Fort Mill, SC
Schedule: Monday-Friday | Primarily On-Site with flexibility for 1-2 days remote
Compensation: $185,000-$190,000 base salary + 7.5%-15% bonus
Position Overview
Our client is seeking a Senior Data Engineer - IoT to take a leading role in the development and long-term direction of its IoT data platform. This is a newly created position for a highly hands-on engineer who brings significant experience working with large-scale IoT, machine, or telemetric data.
This individual will help drive the continued migration of IoT data into Databricks, while designing and building a scalable data platform that supports real-time and near-real-time reporting, analytics, data science, AI/ML, customer-facing applications, and sustainability initiatives.
This is an opportunity to have meaningful ownership over the organization's IoT data strategy-not simply implement an existing specification. The ideal candidate will be comfortable moving between hands-on engineering, architecture, technical guidance, and business-facing discussions.
Key Responsibilities
- Design, build, and maintain end-to-end IoT data pipelines from edge devices into Databricks using Bronze/Silver/Gold medallion architecture.
- Lead the migration of IoT data from a third-party application into a scalable Databricks environment.
- Develop and optimize streaming ingestion pipelines using technologies such as Apache Kafka, Azure Event Hubs, and Databricks Structured Streaming.
- Establish data engineering best practices, including pipeline design patterns, data quality frameworks, schema governance, and scalable development standards.
- Architect and optimize Delta Lake tables, including partitioning, Z-ordering, and data lifecycle management for performance and cost efficiency.
- Collaborate with infrastructure and cloud teams on Databricks clusters, Unity Catalog, Delta Live Tables, and related cloud technologies.
- Develop solutions capable of supporting multi-terabyte, high-volume telemetry and time-series data.
- Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
- Ensure IoT data is structured and optimized for downstream analytics, AI/ML, reporting, and customer-facing use cases.
- Mentor junior and mid-level data engineers through code reviews, pair programming, technical guidance, and architectural direction.
- Troubleshoot production pipeline issues, perform root-cause analysis, and implement long-term solutions.
- Evaluate emerging technologies and recommend improvements to the organization's IoT and data engineering platform.
- Represent the data engineering team in technical and business discussions and help shape the organization's long-term IoT data strategy.
Required Qualifications
- 5+ years of hands-on experience working with IoT, machine, telemetric, or similarly large-scale device data.
- Strong senior-level data engineering experience, preferably in environments involving high-volume or real-time data.
- Strong proficiency in Python and SQL, with significant experience developing production-grade Spark/PySpark solutions.
- Experience designing and implementing data pipelines for high-frequency telemetry, time-series data, or device-generated data.
- Experience with Databricks and/or other large-scale cloud data platforms.
- Strong understanding of data architecture, pipeline design, data quality, and schema governance.
- Experience working with streaming or near-real-time data architectures.
- Ability to lead technical discussions and translate business requirements into scalable technical solutions.
- Strong communication and interpersonal skills, with the ability to work effectively with engineers, data scientists, analysts, and business stakeholders.
- Ability to provide technical guidance and mentorship to other engineers.
Preferred Qualifications
- Hands-on experience with Databricks, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
- Experience with Apache Kafka, Azure Event Hubs, or similar streaming technologies.
- Experience with fleet telematics platforms such as Geotab or similar IoT/connected-vehicle platforms.
- Experience working with Azure cloud infrastructure and infrastructure-as-code concepts.
- Familiarity with Azure Key Vault and Azure Data Factory.
- Experience with dbt or other modern data transformation tools.
- Experience supporting multi-terabyte data environments.
- Databricks Certified Data Engineer Associate or Professional certification.
- Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field, or equivalent professional experience.
- Experience within the rental, transportation, fleet, logistics, automotive, manufacturing, or connected-device industries is a plus.
What You'll Bring
The successful candidate will be more than a data pipeline developer. You'll bring the technical depth to work hands-on with complex IoT data while also providing architectural direction and helping shape how the organization approaches IoT data for years to come.
You are a hands-on technical leader who enjoys solving complex data challenges, mentoring engineers, collaborating with stakeholders, and turning large volumes of machine-generated data into solutions that drive meaningful business outcomes.
Why This Opportunity?
- Own the direction of IoT data engineering within a growing enterprise environment.
- Help design a scalable, modern IoT data platform rather than simply implementing an established specification.
- Play a key role in the continued migration and expansion of IoT data within Databricks.
- Work at the intersection of IoT, data engineering, analytics, AI/ML, customer-facing solutions, and sustainability.
- Provide technical leadership while remaining hands-on with the technology.
- Join a collaborative data organization with opportunities to influence architecture, engineering standards, and future technology decisions.
Category Code: JN008
#LI-DM1
#CRGSearchJobs
#LI-Onsite
Seniority Level: Mid-Senior Level