Senior Data Engineer, BI United Nations Federal Credit Union
New York, NY30+ days ago
- $115,000 Per Year
- Full-time
Want to know if you’re a fit?
Upload your resume and let our AI show you.
Upload your resume and let our AI show you.
Skills
- Amazon Simple Storage Service (S3)unmatched
- Amazon Web Services (AWS)unmatched
- Analysis Skillsunmatched
- Apacheunmatched
- Apache Cassandraunmatched
- Apache Hadoopunmatched
- Apache Hiveunmatched
- Apache Sparkunmatched
- Application Programming Interface (API)unmatched
- Architectural Servicesunmatched
- Best Practicesunmatched
- Big Dataunmatched
- Cloud Computingunmatched
- Cloud Storageunmatched
- Code Reviewsunmatched
- Communication Skillsunmatched
- Computer Scienceunmatched
- Continuous Deployment/Deliveryunmatched
- Continuous Integrationunmatched
- Cost Effectiveness Analysisunmatched
- Customer Support/Serviceunmatched
- Data Analysisunmatched
- Data Managementunmatched
- Data Modelingunmatched
- Data Processingunmatched
- Data Qualityunmatched
- Data Scienceunmatched
- Data Setsunmatched
- Data Warehousingunmatched
- Database Extract Transform and Load (ETL)unmatched
- Dimensional Modelingunmatched
- Electronic Medical Recordsunmatched
- Financial Servicesunmatched
- GCP (Good Clinical Practices)unmatched
- Geneticsunmatched
- Gitunmatched
- HDFS (Hadoop Distributed File System)unmatched
- Identify Issuesunmatched
- Information Technology & Information Systemsunmatched
- Information/Data Security (InfoSec)unmatched
- Machine Learningunmatched
- Microsoft Windows Azureunmatched
- MongoDBunmatched
- NoSQLunmatched
- Problem Solving Skillsunmatched
- Process Managementunmatched
- Process Modelingunmatched
- Python Programming/Scripting Languageunmatched
- Quality Monitoringunmatched
- SQL (Structured Query Language)unmatched
- Scalable System Developmentunmatched
- Source Code/Configuration Management (SCM)unmatched
- Staff Developmentunmatched
- Training Data Setsunmatched
Description
Software Guidance & Assistance, Inc., (SGA), is searching for a Data Engineer - PySpark for a right to hire assignment with one of our premier financial services clients in Rutherford, NJ.
Responsibilities :
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy
Responsibilities :
- Responsible for designing, developing, and maintaining robust and scalable data pipelines using PySpark.
- Design, build, and optimize data pipelines using PySpark to extract, transform, and load (ETL) data from various sources into data lakes and data warehouses.
- Develop and maintain scalable data processing jobs and frameworks using Apache Spark with Python (PySpark).
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality data solutions.
- Implement data quality checks, monitoring, and alerting for data pipelines to ensure data accuracy and reliability.
- Optimize existing PySpark jobs for performance, efficiency, and cost-effectiveness.
- Manage and process large datasets, ensuring data governance, security, and compliance.
- Troubleshoot and resolve issues in data pipelines and data processing jobs.
- Participate in code reviews, contribute to architectural discussions, and promote best practices in data engineering.
- Stay informed about new PySpark features, big data technologies, and industry best practices.
- Document data pipelines, data models, and processes.
- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related quantitative field.
- 7-10 years of experience as a Data Engineer, with significant experience specifically in PySpark.
- Strong proficiency in Python programming.
- Extensive experience with Apache Spark, including Spark SQL, Spark Streaming, and DataFrame API.
- Solid understanding of data warehousing concepts, dimensional modeling, and ETL principles.
- Proficiency in SQL for data querying and manipulation.
- Experience with big data technologies suchation as Hadoop, HDFS, Hive, or similar.
- Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services (e.g., S3, ADLS, Google Cloud Storage, EMR, Databricks, Glue).
- Experience with version control systems (e.g., Git).
- Excellent problem-solving, analytical, and communication skills.
- Requires a strong understanding of big data technologies, data warehousing concepts, and the ability to work with large datasets to support analytics, reporting, and machine learning initiatives.
- Master's degree in a related field.
- Experience with workflow orchestration tools (e.g., Apache Airflow, Azure Data Factory, AWS Step Functions).
- Knowledge of stream processing technologies (e.g., Kafka, Kinesis).
- Experience with NoSQL databases (e.g., MongoDB, Cassandra, DynamoDB).
- Familiarity with data governance tools and practices.
- Experience in a CI/CD environment.
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy
Numbers & Facts
| Location | Rutherford, NJ |
Similar Jobs
- New!
Senior Data Engineer Mitchell Martin
New York, NY5 days ago- $130,000–$170,000 Per Year
- New!
Data Engineer ECLARO
Rutherford, NJ1 day ago Data Engineer -601/602 Pinnacle Technical Resources
Jersey City,, New Jersey30+ days ago- $65–$70 Per Hour
- Contractor
- New!
Vice President - Senior Data Engineer Barclays
Whippany, NJ6 days ago- $170,000 Per Year
Mechanical Engineer (Data Centers) Jobot
New York, NY9 days ago- $110,000–$155,000 Per Year