Data Engineer (Tech Lead)

Job description:
Seeking an experienced Data Engineer with 10+ years of expertise in Python, PySpark, AWS, SQL, Kafka, and Apache Flink to design, develop, and support scalable cloud-based data pipelines and real-time data processing solutions.

Technical Skill Sets: Python, PySpark, AWS Lambda, Amazon S3, AWS Glue, Kinesis, SQL, Apache Flink, Confluent Kafka, AI-LLM, Gen-AI, Java (Optional)

ROLES AND Responsibilities

1. Advanced Architecture & System Design

A Tech Lead is primarily responsible for the overall platform vision and ensuring systems do not break under scale.

• Distributed Computing: Mastery of frameworks like Apache Spark or Ray for massive-scale parallel data processing.
• Streaming & Event-Driven Architecture: Deep understanding of real-time pipeline design using Kafka, Kinesis, or Flink.
• Cloud Infrastructure: Expertise in at least one major public cloud (AWS), specifically understanding storage/compute decoupling and cost optimization.

2. Core Programming & Database Management

Leads set coding standards and review code, requiring complete fluency in the fundamentals.

• SQL: Advanced mastery for metrics computation, window functions, and query performance tuning across relational and columnar databases (e.g., Snowflake, Redshift, BigQuery).
• Scripting Languages: High proficiency in Python or Scala for writing reusable pipeline code and interacting with APIs.
• Data Storage: Deep familiarity with both columnar/analytical stores and NoSQL databases (e.g., DynamoDB, Cassandra).

3. Pipeline Orchestration & DevOps

Ensuring pipelines run smoothly, idempotently, and securely in production.

• Workflow Orchestration: Ability to architect Directed Acyclic Graphs (DAGs) in tools like Apache Airflow or Prefect.
• CI/CD & Infrastructure as Code (IaC): Applying software engineering principles to data by using Docker, Kubernetes, and Terraform.
• Data Governance & Security: Implementing Role-Based Access Control (RBAC), data masking, and compliance frameworks.

4. Leadership & Soft Skills

Tech Leads also mentor junior engineers, estimate project timelines, and translate ambiguous business needs into concrete technical specifications.

• Mentorship & Code Review: Fostering a collaborative development environment and enforcing style guidelines.
• System Observability: Building logging, monitoring, and alerting mechanisms so the team knows exactly when and why pipelines fail.

Essential Skills

• Python
• PySpark
• AWS Lambda
• Amazon S3
• AWS Glue
• Kinesis
• SQL
• Apache Flink
• Confluent Kafka
• Java (Optional)

Skills

• Digital: Python
• Digital: Amazon Web Service (AWS) Cloud Computing
• Digital: Kafka
• Advanced Java Concepts
• Digital: PySpark

• Inclusive and diverse work environment
• Opportunities for professional growth and development
• Comprehensive health and wellness benefits