Python/PySpark Data Enginee

Role : Python/PySpark Data Engineer

Location : Charlotte NC 28202 (100% Onsite)

Hire Type (FTE/Contract): FTE / C2H

Rate : $60/hr.

Salary : $120K

Role Summary

We are looking for a strong Python/PySpark Data Engineer to build scalable data ingestion and integration pipelines across enterprise data sources and metadata platforms. The engineer will work with metadata catalogs, databases, APIs, and batch/Realtime integration patterns while contributing to enterprise data management and data warehousing initiatives.

Key Responsibilities

• Develop robust data engineering solutions using Python, PySpark and Apache Spark.

• Consume and integrate metadata/data from Ab Initio Enterprise Meta>Environment (EME/EMH), Oracle databases, and ServiceNow APIs.

• Perform source-to-target mapping, data transformation, validation, and reconciliation.

• Build and maintain batch data ingestion and integration pipelines.

• Work with data warehouse concepts, data models, metadata management, data lineage and data quality.

• Integrate technical and business metadata from multiple enterprise sources.

• Understand existing/custom data engineering frameworks and quickly develop solutions within them.

• Troubleshoot performance, data quality, and integration issues across large datasets.

Required Skills

• Strong hands-on expertise in Python-based data engineering.

• Strong experience with PySpark / Apache Spark and large-scale data processing.

• Experience working with Oracle/SQL and relational databases.

• Experience consuming REST APIs and integrating API-based data sources.

• Good understanding of ETL/ELT, source-to-target mapping, data warehousing and dimensional/relational data modeling.

• Understanding of metadata management, business/technical metadata, lineage and data governance concepts.

• Experience building production-grade batch ingestion/integration pipelines.

• Ability to understand and work with custom enterprise data frameworks.

Good to Have

• Experience with real-time/streaming integration, such as Kafka or Spark Structured Streaming.

• Exposure to Google Cloud Platform (GCP) and its data services.

• Experience with enterprise metadata/catalog solutions, particularly Ab Initio metadata environments.

• Experience integrating with ServiceNow.

Ideal Profile: A hands-on engineer who is particularly strong in Python + PySpark, understands enterprise data and metadata ecosystems, and can quickly learn existing frameworks rather than relying only on off-the-shelf ETL tools.