Request ID: 101204Title: Senior Data EngineerLocation: Raleigh/Pheonix/Dallas (Onsite)Duration: 6 MonthsSalary Range: $40 - $45 an hour on W2 or C2C
Job DescriptionResponsibilities:
Data Pipeline Architecture & Development
- Design and implement scalable, resilient data pipelines using Snowflake features including Snowpipe, Tasks, Streams, Dynamic Tables, and advanced SQL.
- Build and maintain DBT models with strong testing, documentation, and lineage.
- Develop Python ingestion frameworks for files and APIs, including schema validation, retries, and metadata capture.
- Engineer ingestion solutions for CSV, fixed-width multi-record layouts, JSON, XML, Excel, and semi-structured formats.
- Design Mainframe VSAM data ingestion patterns for complex EBCDIC data formats.
Schema Drift & Schema Evolution
- Detect, analyze, and manage schema drift across file, API, and replicated database sources.
- Implement metadata-driven schema evolution strategies to ensure downstream stability.
- Coordinate schema changes through controlled CI/CD workflows.
Database Replication & CDC
- Configure and manage Qlik Replicate tasks for CDC and full-load replication from Oracle, SQL Server, and DB2.
- Ensure idempotent, auditable, and recoverable replication pipelines with strong monitoring and reconciliation.
Data Governance, Security & Tokenization
- Implement and maintain Snowflake Data Masking policies, including dynamic masking, conditional masking, and role-based masking rules.
- Apply Protegrity tokenization for sensitive data fields across ingestion and transformation layers.
- Enforce RBAC, data access controls, and governance standards across Snowflake and supporting systems.
Orchestration & Automation
- Build and schedule workflows using Astronomer Airflow, ensuring dependency management, retries, SLAs, and observability.
- Integrate pipelines with enterprise DevOps processes using GitLab and Azure DevOps for CI/CD automation.
Version Control & Code Quality
- Manage code repositories using GitLab, including branching strategies, merge requests, code reviews, and approvals.
Monitoring, Alerting & Performance Optimization
- Implement monitoring and alerting for ingestion pipelines, schema drift, replication, and transformation workloads.
- Optimize Snowflake compute, storage, and query performance.
- Scale ingestion pipelines to meet evolving data volume and latency requirements.
Required Qualifications:- 6-8 years of hands-on experience in Data Engineering.
- Deep expertise in Snowflake, including:
- Data masking policies
- RBAC
- Performance tuning
- Advanced SQL
- Strong experience with Qlik Replicate for CDC and database replication.
- Excellent proficiency in Python and PySpark for ingestion frameworks and automation.
- Hands-on experience with DBT Cloud and Astronomer Airflow.
- Experience with schema drift detection and schema evolution patterns.
- Experience with GitLab and CI/CD pipelines.
- Familiarity with Protegrity or similar data protection platforms.