Data Pipeline Engineer
Updated Job Description Years of Experience
8 years of software development in data modeling, database design & optimization, scalable data pipelines for image/video (multimedia) data, and full-stack development using Python and SQL
5 years of development on data warehouse and analytics solutions (e.g., Airflow, Trino, Spark, Snowflake)
5 years of development on cloud-based data processing pipelines
Demonstrated software development skills using Python
Preferred: Agentic AI development experience
Education
Bachelor's or Master's degree (or above) in Computer Science, Software Engineering, or a related field
Responsibilities
Unified Metadata & Schema Design: Design and build standardized schema that unify file indexing, metadata, and enriched metadata across heterogeneous storage platforms (e.g., a document-store-based file platform, a relational per-protocol file platform, and other data collection platforms) and multiple dataset with differing file/metadata structures.
Data Pipeline Engineering for Multimedia Data: Build and maintain efficient, scalable, monitored data pipelines that process image and video data at scale within a unified data platform, and support downstream consumer dataset generation.
Database Design & Optimization: Create and maintain database schemas across SQL, vector, and unstructured data stores; apply normalization and denormalization appropriately; tune queries and storage for performance, reliability, and query efficiency at scale, addressing the known challenge of indexing/querying file- and frame-level metadata beyond classic DB solutions.
Dashboards & Data Quality Monitoring: Develop dashboards to track data ingestion progress, surface metadata gaps, detect invalid/inconsistent metadata, and fix data issues early.
Federated Ecosystem Enablement: Support the broader goal of a federated data ecosystem where independently contributed tables are connected, discoverable, and queryable through one consistent data schema interface.
Full-Stack Development: Build front-end and back-end components of data-driven applications/dashboards, develop APIs, and integrate data from multiple sources/technologies into a single consumer-facing interface.
Data Security & Compliance: Ensure data handling meets security and privacy compliance requirements across platforms and studies.
Agentic AI Prototyping (stretch/preferred): Prototype and develop agentic AI capabilities to assist with data analysis and processing workflows on top of the unified data interface.
Preferred Skills
Proficiency with multimedia processing tools (FFmpeg, GStreamer, OpenCV) and familiarity with AI/ML frameworks (TensorFlow, PyTorch) for video analysis
Experience with Spark, Airflow, Trino, Snowflake, Iceberg-based lakehouse/data platforms
Experience designing schema layers that separate interface from implementation to allow backward-compatible schema evolution
Experience with config-driven/declarative pipeline and schema design
Agentic AI development experience
Numbers & Facts
Location
Sunnyvale, CA
Salary
$100 Per Hour
Skills
Analysis Skillsunmatched
Application Programming Interface (API)unmatched
Artificial Intelligence (AI)unmatched
Cloud Computingunmatched
Computer Scienceunmatched
Customer Support/Serviceunmatched
Data Analysisunmatched
Data Collectionunmatched
Data Managementunmatched
Data Modelingunmatched
Data Processingunmatched
Data Qualityunmatched
Data Setsunmatched
Data Warehousingunmatched
Database Administrationunmatched
Database Designunmatched
Database Optimizationunmatched
Database Technologyunmatched
Ecosystemsunmatched
FFmpegunmatched
Information/Data Security (InfoSec)unmatched
Metadataunmatched
Multimediaunmatched
Multiplatform/Cross-Platformunmatched
Privacy Controlsunmatched
Prototypingunmatched
Python Programming/Scripting Languageunmatched
Quality Monitoringunmatched
Regulatory Complianceunmatched
Reporting Dashboardsunmatched
SQL (Structured Query Language)unmatched
Snowflake Schemaunmatched
Software Developmentunmatched
Software Engineeringunmatched
Unstructured Dataunmatched
User Interface/Experience (UI/UX)unmatched
Workflow Analysisunmatched
🎯
Be found by employers
5,500+ employers search our resume database daily. Add yours to get found by recruiters looking for candidates like you.
Level up your application
Professional resume templates
Browse dozens of recruiter approved resume templates, layouts and formats. Choose your favorite and make it your own in minutes.