Indent : PSL307991-15-1
Role : Python backend engineer with PySpark/Spark
Location : Charlotte, NC (Hybrid)
Rate : $57/hr
Role Summary
- Senior backend engineer building the core integrations between Unity and the enterprise Spark engine and SparkFlow framework
- Designs distributed, high-throughput systems that power data processing and integration workflows at enterprise scale
- Operates with significant autonomy, setting technical direction within a regulated, large-scale engineering environment
Key Responsibilities
Platform & Integration Engineering
- Design, build, and operate integrations between Unity and the Spark engine and SparkFlow framework
- Develop robust APIs, SDKs, and connectors enabling reliable data movement and workflow orchestration
- Implement abstractions that simplify SparkFlow adoption for downstream platform consumers
Distributed Systems Design
- Architect scalable, fault-tolerant systems handling large data volumes and concurrent workloads
- Apply best practices for performance tuning, caching, partitioning, backpressure, and resilience
- Drive design reviews, RFCs, and architectural decisions for core platform components
Reliability & Operations
- Build systems with first-class observability (metrics, logs, traces) and SLOs
- Establish operational excellence: runbooks, alerting, capacity planning, and incident response
- Champion automated testing, and progressive delivery practices
Required Qualifications
- 6+ years of backend engineering experience, with deep expertise in distributed systems
- Expert-level Python; strong API design (REST, gRPC) and asynchronous programming
- Proven track record delivering large-scale, production-grade systems
- Strong fundamentals in concurrency, data structures, and systems performance experience
Preferred Qualifications
- Hands-on experience with Apache Spark (PySpark, Spark SQL, Structured Streaming)
- Experience building or integrating with workflow/data orchestration frameworks (Airflow, Autosys)
- Exposure to greenfield platform builds in regulated financial enterprises
- Experience with event-driven architectures (Kafka) and lakehouse formats (Iceberg, Delta, Hudi)
Technical Skills
- Languages: Python (expert), Scala or Java (plus), SQL
- Distributed Compute: Apache Spark
- Storage & Data: S3/ADLS/GCS, Iceberg, Delta Lake, Parquet, Postgres
- Streaming & Messaging: Kafka, Pub/Sub
- APIs & Services: REST, gRPC, GraphQL, OpenAPI
- Containers & Cloud: Docker, Kubernetes, Azure/GCP, Terraform
- CI/CD: GitHub Actions
Soft Skills & Leadership Expectations
- Strong technical judgment and ability to operate