Consulting Partner - Data Engineering & Modern Data Platforms
Required Experience
- 15+ years of experience in data engineering, data platforms, analytics, or technology architecture.
- 10+ years of deep hands-on technical experience in data engineering and data platform technologies.
- Demonstrated experience architecting and delivering complex enterprise-scale data platforms.
- Strong hands-on experience with Spark/PySpark and SQL.
- Strong practical experience with at least one modern cloud data platform such as Databricks or Snowflake.
- Experience with modern data transformation frameworks such as dbt.
- Experience with workflow orchestration technologies such as Airflow.
- Strong understanding of distributed computing, data processing performance, and scalability.
- Experience leading large-scale data modernization or migration programs.
- Experience managing technical teams distributed across multiple locations.
- Ability to engage directly with engineers, architects, platform teams, and executive stakeholders.
Preferred Experience
- Cloud-to-cloud data platform migration.
- Migration of large PySpark estates.
- Legacy ETL modernization.
- Stored procedure modernization to dbt or modern transformation frameworks.
- Databricks platform engineering.
- Data platform governance and security.
- Large-scale data migration involving hundreds of terabytes or petabyte-scale environments.
- AI-assisted engineering and migration accelerators.
- Building reusable engineering frameworks and accelerators.
- Experience with platform reliability, FinOps, and performance optimization.
Roles & Responsibilities
We are seeking a highly accomplished Consulting Partner with deep hands-on Data Engineering and Modern Data Platform expertise to lead complex enterprise data transformation programs.
This is a technical leadership role, not a purely advisory or relationship-management position. The successful candidate will combine executive consulting capabilities with strong architecture and engineering expertise and must have demonstrated hands-on experience designing, building, modernizing, and troubleshooting large-scale data platforms and pipelines.
The Consulting Partner will work with client executives to shape transformation strategies while remaining sufficiently close to technology and engineering teams to influence architecture, solve complex technical problems, review implementations, and guide critical delivery decisions.
The role requires an individual who can move seamlessly from an executive discussion with a CIO or CDO to a deep technical discussion involving Databricks, Spark/PySpark, dbt, SQL, cloud architecture, data pipelines, orchestration, performance optimization, and large-scale data migration.
Key Responsibilities
- Technical Consulting & Architecture Leadership
- Serve as the senior technical advisor for complex data engineering and data platform transformation programs.
- Define enterprise data platform, lakehouse, and data engineering architectures.
- Lead architecture assessments and develop modernization strategies for legacy and cloud-based data ecosystems.
- Design scalable architectures covering data ingestion, transformation, storage, orchestration, governance, security, and consumption.
- Review and challenge architecture decisions made by engineering and solution teams.
- Establish technology standards, engineering principles, and reference architectures.
- Identify technical risks and scalability challenges early in the program lifecycle.
- Provide hands-on technical guidance for complex architecture and engineering issues.
- Hands-On Data Engineering Leadership
The Consulting Partner must maintain strong hands-on technical depth and be capable of working directly with engineering teams when required.
Responsibilities include:
- Reviewing and optimizing PySpark and Spark applications.
- Analyzing query and pipeline performance issues.
- Reviewing complex SQL transformations.
- Designing and reviewing dbt models, transformation frameworks, and testing strategies.
- Troubleshooting complex data processing and distributed computing issues.
- Reviewing data partitioning, clustering, file sizing, and storage optimization strategies.
- Providing technical direction on Delta Lake, table design, and data lifecycle management.
- Reviewing Airflow DAGs and orchestration architecture.
- Helping teams resolve critical production issues and performance bottlenecks.
- Conducting architecture and code reviews for high-risk or complex components.
The role does not require day-to-day individual contributor coding but requires the ability to get hands-on with technology when necessary and independently validate engineering decisions.
- Complex Program Technical Leadership
- Lead the technical direction of large-scale data engineering and modernization programs.
- Oversee programs involving hundreds or thousands of pipelines, jobs, tables, and datasets.
- Define technical migration strategies for:
o Legacy ETL to modern ELT
o span>Stored procedures to dbt
o Legacy Spark platforms to Databricks
o Cloud-to-cloud data platform migration
o Autosys and legacy scheduling to Airflow
o Data warehouse modernization
o Data lake and lakehouse implementation
- Define migration patterns, factory approaches, automation strategies, and validation frameworks.
- Establish technical governance across multiple engineering teams and workstreams.
- Resolve cross-platform and cross-domain technical dependencies.
- Define technical KPIs for scalability, reliability, performance, data quality, and cost efficiency.
- Modern Data Platform Expertise
The candidate should have strong practical experience across several of the following technologies:
Data Engineering
- Apache Spark
- PySpark
- SQL
- Python
- Distributed data processing
- Data pipeline design and optimization
Modern Data Platforms
- Databricks
- Delta Lake
- Snowflake
- Data lakehouse architectures
- Data warehouse modernization
Transformation & Orchestration
- dbt
- Apache Airflow
- Astronomer
- Enterprise workload orchestration
- CI/CD for data engineering
Cloud
- AWS, Azure and/or GCP
- Cloud storage and data services
- IAM and security architecture
- Networking for data platforms
- Infrastructure as Code
- Terraform
Governance & Reliability
- Unity Catalog
- Data catalog and metadata management
- Data lineage
- Data quality
- Data observability
- DataOps
- Executive Consulting & Client Leadership
- Act as a trusted technical advisor to CIOs, CDOs, CTOs, and senior Data and Analytics executives.
- Translate complex technical issues into business implications and executive decisions.