AWS Lambda, Amazon Simple Storage Service (S3), Amazon Web Services (AWS), Best Practices, Change Management, Cloud Computing, Coaching, Code Reviews, Communication Skills, Continuous Deployment/Delivery, Continuous Integration, Cross-Functional, Data Management, Data Processing, Data Quality, Data Sets, Data Storage, Database Extract Transform and Load (ETL), Database Replication, Documentation, Establish Priorities, Identify Issues, Information/Data Security (InfoSec), Lead Generation, Leadership, Mentoring, Needs Assessment, Options Analysis, Project Execution, Python Programming/Scripting Language, Requirements Management, Risk Analysis, Risk Management, Scalable System Development, Technical Delivery, Technical Writing, Time Management
Job Title: Data Engineer
Duration- Fulltime Permanent
Location: Malvern, PA, 19355 / Charlotte, NC, 28262 (Hybrid /requiring 3 days onsite Mandate (Tuesday, Wednesday, and Thursday) (Local Only)
Job Description:
Data Engineer
Python, Glue ETL, Attunity, Pyspark, AWS (S3, IAM, Lambda, Cloud Formation)(must)
Must Have Technical/Functional Skills
Python, Glue ETL, Attunity, Pyspark, AWS (S3, IAM, Lambda, Cloud Formation)
Roles & Responsibilities
· Design, develop, and maintain scalable ETL/data integration pipelines using AWS Glue, PySpark, and Python.
· Develop and optimize data transformation workflows to process large datasets from multiple source systems.
· Utilize Attunity for real-time and batch data replication between on-premises and cloud environments.
· Build and manage data lakes and storage solutions using Amazon S3, ensuring data quality, security, and governance.
· Implement serverless data processing solutions using AWS Lambda and integrate with other AWS services.
· Configure and manage IAM roles, policies, and permissions to ensure secure access to AWS resources.
· Automate infrastructure provisioning and deployment using AWS CloudFormation.
· Monitor, troubleshoot, and optimize ETL jobs, data pipelines, and cloud resources for performance and reliability.
· Collaborate with business, analytics, and application teams to gather requirements and deliver data solutions.
· Support CI/CD processes, perform code reviews, and maintain technical documentation following best practices.
Generic Managerial Skills, If any
· Planning & Execution – Effectively prioritize tasks, estimate effort, and ensure timely delivery of project milestones while balancing multiple commitments.
· Stakeholder Management – Build strong relationships with business teams, clients, and leadership through proactive communication and expectation management.
· Communication – Convey technical and non-technical information clearly through meetings, presentations, documentation, and status updates.
· Collaboration – Foster teamwork across cross-functional groups, encourage knowledge sharing, and contribute to a positive team environment.
· Decision Making – Analyze options, assess risks, and make informed decisions that align with both technical and business objectives.
· Leadership & Ownership – Take accountability for deliverables, drive initiatives proactively, and lead by example to achieve team goals.
· Risk & Issue Management – Identify potential risks early, address blockers proactively, and implement mitigation strategies to ensure smooth project execution.
· Customer & Business Focus – Understand customer needs and business priorities, ensuring technical solutions deliver measurable value.
· Mentoring & Coaching – Support team growth through guidance, code reviews, knowledge sharing, and skill development initiatives.
· Project & Change Management – Adapt effectively to evolving requirements, manage priorities during change, and ensure successful delivery outcomes.