Job Title: Data Architect OCP (OpenShift) / IAM Data Modernization
Location: Dallas, TX or Charlotte, NC (100% Onsite)
Duration: 6-12 Months
Objective:
Migration of an on-premises SQL data warehouse to a modern enterprise Data Lake platform, enabling analytics and GenAI use cases.
The platform leverages PySpark-based processing, CI/CD pipelines, and containerized deployments on OpenShift (OCP), with GCP as a preferred cloud platform, to deliver scalable, secure, and high-performance data solutions.
About Program/Project:
The IAM Data Modernization program focuses on transforming legacy data platforms into a scalable and cloud-compatible architecture.
Key Highlights:
Integration Scope: 30+ source systems with multiple downstream integrations.
Capabilities: Metrics, reporting, advanced analytics, and GenAI use cases (NL querying, summarisation, cross-domain insights).
Benefits:
Scalable and resilient data platform.
High-performance semantic and analytics layer.
Single source of truth for enterprise-wide reporting and analytics.
Role Summary:
We are looking for a Data Architect with strong expertise in OpenShift (OCP), PySpark, and CI/CD pipelines to design and govern scalable data platforms.
The role requires defining end-to-end data architecture, containerised deployment patterns, orchestration strategies (Airflow/Autosys), and platform standards, along with hands-on involvement in implementation.
Key Responsibilities:
Data Architecture & Platform Design:
Define enterprise data architecture for IAM data lake and analytics platform.
Design scalable, modular, and containerised data pipeline architectures on OCP.
Establish data models, schema governance, and data lifecycle strategies.
Define best practices for data partitioning, performance optimisation, and cost efficiency.
OpenShift (OCP) & Platform Engineering:
Architect and govern containerised data workloads on OpenShift (OCP).
Define standards for deployment, scaling, and workload isolation.
Collaborate with DevOps teams for platform engineering and infrastructure alignment.
Big Data & Processing (PySpark Focus):
Define architecture for PySpark-based batch and near real-time processing pipelines.
Provide guidance on distributed processing design, optimisation, and performance tuning.
Establish reusable frameworks for ETL/ELT processing.
Data Ingestion & Orchestration:
Architect data ingestion frameworks (batch, streaming, CDC).
Define orchestration strategies using Airflow/Autosys.
Implement standards for retry, backfills, dependency management, and error handling.
DevOps/CI-CD:
Define and oversee CI/CD strategy for data and platform deployments.
Enable automation of build, test, and deployment processes.
Ensure integration of CI/CD pipelines with OCP-based environments.
Cloud & Data Platforms (Preferred):
Provide architecture guidance for GCP-based data platforms (preferred, not mandatory).
Define integration patterns for cloud-native and on-premise hybrid environments.
Guide teams on cloud migration strategies and modern data platform adoption.
Data Governance, Quality & Observability:
Data quality, validation, and lineage.
Metadata management and cataloguing.
Establish monitoring, logging, alerting, and SLOs for platform reliability.
Ensure compliance with data security and audit requirements.
Stakeholder Collaboration.
Work closely with client architects, IAM teams, and business stakeholders.
Translate business requirements into scalable technical architecture.
Provide architectural guidance and mentorship to engineering teams.
Required Skills - Core Skills (Must Have):
OpenShift (OCP)/Kubernetes-based platforms.
PySpark/Spark ecosystem.
CI/CD implementation for data platforms.
Airflow/Autosys orchestration tools
Solid Understanding Of:
Data lake architectures (layered models).
ETL/ELT design patterns.
Distributed data processing concepts.
Data Engineering & Storage:
Data formats: Parquet, ORC, Avro.
Partitioning and performance tuning.
Large-scale data modelling for analytics.
Cloud (Preferred Not Mandatory).
Experience with Google Cloud Platform (GCP) (preferred).
Exposure to services like BigQuery, Dataproc, Dataflow, GCS is a plus.
Observability & Reliability:
Monitoring, logging, alerting frameworks.
Dashboards, SLOs, and operational runbooks.
Good To Have:
Experience with IAM domain/cybersecurity data.
Understanding of data security and access control frameworks.
Exposure to GenAI-enabled data platforms.
Experience in Agile delivery and team leadership.
Experience:
10 14+ years in Data Architecture/Data Engineering.
Strong experience in OCP, PySpark, CI/CD, and orchestration frameworks.
Prior experience in data modernization/migration programs.
Education:
Bachelor's/Master's in Computer Science, Information Systems, or equivalent.
Certifications (Preferred):
OpenShift/Kubernetes certifications.
GCP certifications (preferred, not mandatory).
Regards,
Himanshu Rawat
himanshu@carmansg.com