QA Automation with Spark

Title: QA Automation with Spark

Location: Bothell, WA

Duration: 6-12 months

GC or $No Visa Sponsership$ only

$45/hr on W2

F2F Interview

Required Details:

Experience: 3-7 years

Job Summary

We are looking for a skilled QA Automation Engineer with experience in Apache Spark and data testing to design, develop, and execute automated testing solutions for data-intensive applications and ETL/data pipelines. The ideal candidate will have strong knowledge of QA methodologies, automation frameworks, SQL, Python/Java, and Spark-based data processing.

Key Responsibilities

• Design and develop automated test scripts and frameworks for data pipelines and Spark applications.
• Perform functional, regression, integration, system, and end-to-end testing.
• Validate Apache Spark jobs, including data transformations, aggregations, joins, and business rules.
• Perform data validation and reconciliation between source and target systems using SQL.
• Test ETL/ELT pipelines and ensure data quality, accuracy, completeness, and consistency.
• Develop automation scripts using Python, Java, or Scala.
• Execute Spark jobs and validate input/output datasets, transformations, and processing results.
• Identify, analyze, and troubleshoot defects in applications and data pipelines.
• Integrate automated tests into CI/CD pipelines using tools such as Jenkins, GitHub Actions, or Azure DevOps.
• Work closely with developers, data engineers, business analysts, and product teams to resolve issues.
• Create and maintain test plans, test cases, automation scripts, and defect reports.
• Monitor test execution and provide quality metrics and status reports.
• Participate in Agile/Scrum ceremonies and contribute to continuous improvement of QA processes.

Required Skills

• 3+ years of experience in QA Automation / Software Testing.
• Hands-on experience with Apache Spark / PySpark.
• Strong SQL skills for data validation and backend testing.
• Experience with Python, Java, or Scala.
• Knowledge of automation frameworks such as Selenium, PyTest, TestNG, or similar.
• Experience testing ETL/data pipelines and large datasets.
• Understanding of Spark concepts such as DataFrames, transformations, actions, joins, partitions, and Spark SQL.
• Experience with Git and CI/CD tools.
• Good understanding of SDLC, STLC, defect management, and Agile methodologies.
• Strong analytical and problem-solving skills.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.