Job Title: Big data Engineer
Location: McLean, VA (Hybrid – Tuesday through Thursday onsite)
Duration: 5 Months (Possibility of Extension and Conversion)
Interview Process: In-person interview in McLean, VA – Two back-to-back interview rounds, approximately 1.5 hours
Required Experience: 5+ years, ideally 5–10 years
Work Arrangement: Candidates must be comfortable working onsite in McLean, VA, Tuesday through Thursday, and attending the in-person interview.
Job Summary
We are seeking a hands-on Senior Big Data Engineer to design, develop, maintain, and support large-scale data pipelines for transaction data and personal loan data. The engineer will work on initiatives that bring data from multiple lines of business into a centralized ecosystem to support Anti-Money Laundering (AML), fraud detection, and regulatory reporting use cases.
The ideal candidate will have strong experience in Python, Apache Spark, SQL, AWS, and modern data engineering platforms, along with experience troubleshooting data pipelines, supporting production environments, and resolving data-related issues.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Python and Apache Spark to process large volumes of transaction and financial data.
- Ingest, transform, validate, and publish data from multiple lines of business to support downstream AML, fraud detection, and regulatory reporting applications.
- Build and enhance data pipelines to integrate personal loan data into the existing data ecosystem.
- Develop and manage data workflows using Apache Airflow for orchestration, scheduling, and dependency management.
- Work with Databricks, Delta Lake, and Snowflake for data processing, storage, and publishing.
- Write and optimize complex SQL queries to support data transformation, analysis, validation, and troubleshooting.
- Monitor, troubleshoot, and resolve data pipeline failures, data quality issues, and production incidents.
- Provide ongoing production support for existing data pipelines and ensure their reliability, performance, and stability.
- Identify and remediate vulnerabilities and address production-related technical issues.
- Collaborate with engineering teams and business stakeholders to understand data requirements and deliver reliable data solutions.
- Perform hands-on coding, data analysis, debugging, and problem-solving across data engineering initiatives.
- Contribute to system design and technical improvements to enhance pipeline performance, scalability, and maintainability.
Required Qualifications
- 5+ years of hands-on experience in Data Engineering, ideally within the 5–10-year range.
- Strong programming experience in Python.
- Hands-on experience with Apache Spark for large-scale data processing.
- Strong SQL skills and experience working with relational databases.
- Experience developing, maintaining, and supporting end-to-end data pipelines.
- Experience with AWS cloud services and cloud-based data engineering solutions.
- Experience with Apache Airflow or similar data orchestration tools.
- Hands-on experience with Databricks, Delta Lake, and/or Snowflake.
- Strong understanding of data ingestion, transformation, processing, and publishing.
- Experience troubleshooting pipeline failures, resolving production issues, and supporting live data workflows.
- Strong analytical, debugging, and problem-solving skills.
- Ability to work in a hands-on development environment and collaborate with multiple engineering teams.
Preferred Qualifications
- Experience working with financial transaction data, personal loan data, or other large-scale financial datasets.
- Familiarity with AML, fraud detection, or regulatory reporting data use cases.
- Experience identifying and remediating application or data pipeline vulnerabilities.
- Experience with API development and data integration.
- Previous experience working in large enterprise data engineering environments.
Interview Process
Candidates should be prepared for technical assessments covering:
- Python Coding: Hands-on coding and problem-solving exercises.
- System Design: Data pipeline architecture and technical design.
- SQL and Data Engineering: SQL queries, data engineering concepts, and real-world troubleshooting scenarios.
Important: The role is primarily focused on Data Engineering and pipeline development, maintenance, and production support. Machine Learning experience is not required. API development is a plus but is not a primary responsibility.