Senior Data Architect/Lead Data Engineer

Immediate need for a talented Senior Data Architect/Lead Data Engineer. This is a 12 months contract opportunity with long-term potential and is located in Jersey City NJ(Hybrid). Please review the job description below and contact me ASAP if you are interested.

Job ID: 26- 20307

Pay Range: $60/hr - $68/hour.  Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).

Key Responsibilities:

• Enterprise & Surveillance Architecture
• Define targetstate architecture and strategic roadmap for surveillance data processing platforms leveraging Hadoop, Spark, PySpark, and Python.
• Establish standard architectural patterns for alert generation, enrichment, aggregation, and reporting pipelines.
• Ensure architecture aligns with enterprise technology standards and Surveillance control expectations.
• Solution Design & Delivery Support
• Translate surveillance business requirements (e.g., market misconduct detection, regulatory coverage) into scalable technical designs.
• Review and approve detailed technical designs, ensuring alignment with functional intent, regulatory requirements, and architecture standards.
• Provide hands on architectural guidance to engineering teams during development, testing, and implementation.
• Design and standardization of Spark/PySpark frameworks supporting surveillance alert generation and enrichment.
• Modernization and optimization of large scale Hadoop surveillance workloads to improve performance, stability, and control coverage.
• Implementation of enterprise consistent architecture patterns enabling audit readiness, lineage, and regulatory traceability.
• Support end to end delivery across the SDLC, minimizing rework and technical debt.
• Architecture Governance & Leadership
• Participate in and lead architecture review forums and design walkthroughs.
• Mentor senior engineers and promote adoption of standard frameworks and best practices.
• Influence enterprise surveillance platform strategy through architecture governance
• Documentation & Communication
• Maintain highquality documentation, including business requirements, data definitions, and process flows.
• Proactively communicate risks, dependencies, and potential timeline impacts to stakeholders.
• Non Functional Requirements & Production Readiness
• Own nonfunctional requirements including performance, scalability, resiliency, SLAs, auditability, and disaster recovery.
• Define and enforce production readiness criteria, including monitoring, alerting, logging, and operational handoff.
• Partner with support and operations teams to ensure platform stability and timely issue resolution
• Data Controls, Risk, and Compliance
• Ensure designs meet Surveillance data control requirements: completeness, accuracy, timeliness, lineage, and traceability.
• Align architecture with enterprise security standards, including encryption, access control, and PII handling.
• Support internal audits, regulatory exams, and model governance reviews by providing architectural documentation and control evidence.
• Regular Points of Contact
• Surveillance Technology teams to clarify requirements and prioritize architectural decisions
• Enterprise Architecture, Information Security, and Production Support teams
• Coordinate with platform, data governance, and vendor teams as required
• Direct impact on strengthening risk and compliance controls
• Exposure to complex financial markets and Dodd Frank regulatory requirements
• High visibility collaboration across compliance and technology organizations
• An environment that values innovation, accountability, and measurable impact

Key Requirements and Technology Experience: 

• Key skills; Python, Hadoop, PySpark
• 10 years of experience in data engineering, platform engineering, or architecture roles
• Deep expertise in PySpark, including performance tuning and optimization
• Strong Python development experience in large scale distributed environments
• Solid knowledge of Hadoop ecosystem (HDFS, Hive/Impala, YARN)
• Proven experience designing and governing enterprise, regulator facing data platforms
• Strong understanding of non functional requirements and production support models
• Clear written and verbal communication skills with ability to influence across organizations
• Financial services experience, particularly in Market Surveillance, AML, Fraud, or Risk Technology
• Experience supporting regulatory or audit facing platforms
• Kafka and Spark Structured Streaming exposure
• Familiarity with orchestration tools (Airflow, Control M, Oozie)
• Knowledge of data governance, lineage, and data quality controls
• Characteristics of a Top Performer
• Demonstrates strong ownership and accountability in regulator facing environments
• Balances strategic architecture thinking with practical delivery execution
• Proactive in identifying risks, control gaps, and optimization opportunities
• Effective communicator across engineering, compliance, and leadership audiences

Our client is a leading Financial Industry, and we are currently interviewing to fill this and other similar contract positions. If you are interested in this position, please apply online for immediate consideration.

Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

By applying to our jobs you agree to receive calls, AI-generated calls, text messages, or emails from Pyramid Consulting, Inc. and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy here.