Data Engineer Hybrid Malvern, PA 2 OPENINGS ASSESSMENT REQUIRED
Senior Data Engineer
Job Summary
We are seeking a talented Senior Data Engineer to join our infrastructure support team. This role is focused on designing and developing robust AWS utilities that enable our data engineers and analysts to seamlessly query across multiple data sources, specifically bridging the gap between S3-based tables and Amazon Redshift. As we transition to a multi-technology data environment, you will be responsible for building resilient, enterprise-grade solutions that prioritize reliability and quick recovery.
Key Responsibilities
Design & Development: Lead the design and development of custom AWS utilities that allow analysts to query disparate data sources (S3 and Redshift) with minimal effort and high productivity.
Pipeline Engineering: Build and maintain scalable data pipelines using AWS Glue, PySpark, and Spark SQL to ingest and transform data across platforms.
Resiliency & Scalability: Apply an "enterprise-grade " development mindset. Build solutions that are resilient, scalable, and capable of quick automated recovery in the event of failure.
Utility Creation: Create tools to streamline the user experience for analysts, such as building automated processes to convert SQL outputs for multi-source tables.
Infrastructure Ops: Utilize CloudFormation to manage and deploy infrastructure components.
Required Technical Skills
AWS Core: Deep proficiency in AWS Glue, S3, Lambda, IAM, and Step Functions.
Data Engineering: Expert-level knowledge of PySpark and Spark SQL.
Querying & SQL: Extensive experience with Athena and Presto; strong SQL skills for complex data manipulation.
Infrastructure as Code: Proven experience with CloudFormation for operational excellence.
Development Mindset: Ability to write clean, maintainable code for automation and custom utility development.
Dremio: Experience with Dremio implementation (reflections, views, using system tables).
Required Qualifications
6-8+ years of Data Engineering or IT experience.
4+ years of AWS cloud development experience.
Expert Python programming skills.
Advanced SQL development experience.
Strong hands-on experience with Apache Spark / PySpark.
Experience building cloud-native ETL pipelines.
Strong understanding of distributed computing.
Experience designing scalable data models.
Experience with Git and CI/CD pipelines.
Strong analytical and problem-solving skills.
Build scalable and auto recoverable pipelines.
Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.
Nice to Have
Tableau: Familiarity with Tableau for end-user data consumption.
Numbers & Facts
Location
Malvern, PA
Skills
AWS Lambdaunmatched
Amazon Simple Storage Service (S3)unmatched
Amazon Web Services (AWS)unmatched
Analysis Skillsunmatched
Apache Sparkunmatched
Automationunmatched
Cloud Computingunmatched
Computer Programmingunmatched
Computer Scienceunmatched
Continuous Deployment/Deliveryunmatched
Continuous Integrationunmatched
Customer/Client Researchunmatched
Data Analysisunmatched
Data Managementunmatched
Data Modelingunmatched
Database Extract Transform and Load (ETL)unmatched
Distributed Computingunmatched
Establish Prioritiesunmatched
Gitunmatched
Home Automationunmatched
Information Technology & Information Systemsunmatched
Multiplatform/Cross-Platformunmatched
Problem Solving Skillsunmatched
Python Programming/Scripting Languageunmatched
Query Analysisunmatched
SQL (Structured Query Language)unmatched
Scalable System Developmentunmatched
Tableauunmatched
User Interface/Experience (UI/UX)unmatched
Writing Skillsunmatched
🎯
Be found by employers
5,500+ employers search our resume database daily. Add yours to get found by recruiters looking for candidates like you.
Level up your application
Professional resume templates
Browse dozens of recruiter approved resume templates, layouts and formats. Choose your favorite and make it your own in minutes.