Sr Data Engineer

• IT IS 100 % ON SITE POSITION IN Mc lean VA
  

KEY REQUIRED SKILLS:

• PySpark & Python for data pipeline development, Snowflake & AWS
  

• Scalable Architecture: Design and build scalable batch and streaming data pipelines using PySpark for large-scale data processing.
  
  
• Modernization: Migrate legacy, on-premises ETL workloads (e.g., IBM DataStage, Informatica) to high-performing PySpark and Snowflake cloud pipelines.
  
• Data Transformation: Write production-grade PySpark code to read from Amazon S3 (Parquet/Delta files), execute complex transformations, and process massive datasets efficiently.
  
  
• Deduplication: Design and implement robust deduplication strategies for high-volume datasets using PySpark.
  Platform Engineering: Build and manage Snowflake warehouses, schemas, and data models optimized for enterprise analytics and business intelligence reporting.
  
  
• Iceberg Tables: Design and implement Apache Iceberg tables in Snowflake to support open lakehouse architectures and data interoperability.
  Incremental Processing: Build and maintain Snowflake Dynamic Tables and Materialized Views to enable near real-time analytics and query acceleration.
  
  
• PySpark Tuning: Optimize distributed Spark jobs by leveraging partitioning, caching, broadcast joins, and shuffle optimization.
  Snowflake Optimization: Tune Snowflake workloads using clustering keys, micro-partition pruning, query profiling, precise warehouse sizing, and strategic result caching.
  
  
• Cost Management: Continuously monitor and optimize Spark jobs, Snowflake queries, and AWS infrastructure to balance speed and cloud expenditure.
  
  
• Data Quality: Implement data validation, lineage tracking, and monitoring solutions across all pipeline stages to ensure high data integrity.
  Cross-Functional Collaboration: Partner closely with data architects, business analysts, Technical Program Managers (TPMs), and corporate stakeholders to deliver dependable data products.
  
  
• Technical Documentation: Author comprehensive technical designs, data schemas, and operational runbooks to ensure every pipeline is maintainable and audit-ready.
  

Requirements

• Experience: 6+ years of hands-on data engineering experience in large-scale enterprise environments.
  
  
• PySpark Expertise: Deep proficiency in building distributed data processing pipelines, handling S3 Parquet/Delta files, and implementing complex transformations and deduplication logic.
  
  
• Snowflake Proficiency: Strong hands-on experience with SnowSQL, Snowpipe, Streams, Tasks, and Role-Based Access Control (RBAC). Proven track record establishing Iceberg tables, Dynamic Tables, and Materialized Views.
  
  
• AWS Cloud Ecosystem: Robust working knowledge of AWS services, including S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, and Redshift.
  
  
• Advanced SQL & Python: Mastery of advanced SQL techniques (window functions, CTEs, complex joins) alongside strong Python programming skills for automation, scripting, and orchestration utilities.
  
  
• Orchestration & Architecture: Solid understanding of data warehousing, ELT/ETL patterns, data lakes, and lakehouse architectures using tools like Airflow or AWS Step Functions.
  
  
• Communication: Strong verbal and written communication skills with the ability to articulate technical decisions clearly to both technical peers and business leaders.
  

• Industry Experience: Prior experience working within heavily regulated environments such as financial services, mortgage banking, or GSE programs (Fannie Mae / Freddie Mac).
  
  
• ETL Migration: Hands-on experience with legacy ETL frameworks (e.g., IBM DataStage) to support modernization initiatives.
  
  
• DevOps & CI/CD: Familiarity with continuous integration and continuous deployment pipelines for data infrastructure (Git, Jenkins, GitHub Actions, Terraform).
  
  
• Data Quality Frameworks: Exposure to automated data quality and validation frameworks (e.g., Great Expectations, dbt testing suites).
  
  
• Streaming Analytics: Knowledge of real-time streaming platforms like Apache Kafka or AWS Kinesis.
  
  
• Professional Certifications: AWS Certified Data Analytics, AWS Certified Solutions Architect, or SnowPro Core/Advanced certifications.