Python Engineer- Remote

About the Role
We are looking for a Senior Data Engineer to design, build, and operate cloud-native data pipelines that ingest, validate, standardize, and enrich healthcare provider data on Google Cloud Platform. You will own config-driven transformation pipelines, batch and event-driven processing on GCP, and implement data quality controls.

Key Responsibilities
• Design, build, and maintain scalable ETL/ELT pipelines in Python and SQL on GCP (Cloud Storage, Pub/Sub, Cloud Functions, Cloud Run, BigQuery, Cloud SQL for PostgreSQL).
• Develop and orchestrate batch workflows with Apache Airflow / Cloud Composer.
• Build distributed processing jobs with PySpark/Spark on Dataproc for large-volume healthcare datasets.
• Drive CI/CD and infrastructure-as-code using GitHub Actions

Required Qualifications
• 8+ years of data engineering experience, including 4+ years on Google Cloud Platform.
• Expert Python (pandas, NumPy, PySpark) and advanced SQL (BigQuery, PostgreSQL).
• Hands-on experience with BigQuery, Cloud Storage, Pub/Sub, Cloud Functions/Cloud Run, Cloud SQL, and IAM/service-account security.
• Production experience with Cloud Composer / Apache Airflow DAG design and operations, Dataproc / Apache Spark for distributed batch processing.

Preferred Qualifications
• Experience integrating LLM/AI/ML services into data pipelines
• Kubernetes (GKE/EKS) and event-driven/serverless architecture experience.


GCP Data And AI Automation: Need for GCP-oriented engineers to automate provider verification and other transactional processes, while also developing interfaces that allow users to review AI-generated results and evidence.
Provider Verification: The GCP team evaluates provider information from state websites, including license status, specialty, location, and adverse actions. When providers must be removed from a network because of licensing or disciplinary issues, the process helps prevent inaccurate network records and related regulatory exposure.
.
AI-Assisted Matching: Many state websites have outdated interfaces and limited search capabilities. When an NPI is unavailable, AI helps identify the correct provider from less precise information such as a name, state, and specialty. The processing is primarily transactional data comparison with AI used to automate steps that would otherwise require manual review.
.
Review Applications: The team is creating quick Streamlit-based interfaces so users can evaluate whether AI outputs are drifting, making correct assumptions, and producing appropriate results. Longer term, Java developers may support more structured front-end or dashboard experiences for daily evidence review.