Job Summary: We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for creating equivalent PySpark log pipelines, rule configuration files, unit tests, and data validation checks, and deploying these pipelines using Terraform. A strong background in data engineering, particularly with large-scale log analytics, is essential. Requirements:
5-10+ years of experience as a PySpark/SQL Engineer, with a strong focus on data engineering and analytics.
Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks.
Hands-on experience with Terraform for deploying data infrastructure.
Proficient in PySpark for large-scale data processing and transformation.
Familiarity with cloud platforms such as AWS or Azure is preferred.
Strong analytical skills and attention to detail when working with complex datasets.
Proven ability to work effectively in collaborative, cross-functional teams.
Excellent verbal and written communication skills in English.
Responsibilities:
Design and build threat detection pipelines using PySpark, SQL, and Databricks.
Support the migration of detection rules and content from a legacy platform to Databricks.
Create and maintain PySpark log pipelines and associated rule configuration files.
Write unit tests to ensure pipeline accuracy and stability.
Perform data validation checks to ensure data integrity.
Deploy pipelines and infrastructure using Terraform.
Optimize existing data workflows and queries for performance and scalability.
Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives.
Numbers & Facts
Location
San Francisco, CA (Remote)
Salary
$76.55–$129.16 Per Hour
Skills
Amazon Web Services (AWS)unmatched
Analysis Skillsunmatched
Cloud Computingunmatched
Communication Skillsunmatched
Cross-Functionalunmatched
Data Analysisunmatched
Data Processingunmatched
Data Qualityunmatched
Data Setsunmatched
Detail Orientedunmatched
English Languageunmatched
Microsoft Windows Azureunmatched
Presentation/Verbal Skillsunmatched
SQL (Structured Query Language)unmatched
Sales Pipelineunmatched
Security Analysisunmatched
Team Playerunmatched
Test Plan/Scheduleunmatched
Unit Testunmatched
Validation Testingunmatched
Writing Skillsunmatched
🎯
Be found by employers
5,500+ employers search our resume database daily. Add yours to get found by recruiters looking for candidates like you.
Level up your application
Professional resume templates
Browse dozens of recruiter approved resume templates, layouts and formats. Choose your favorite and make it your own in minutes.