Data Engineering Lead

Role: Data Engineering Lead

Location: New York, NY

Fulltime

Salary Range: $135,000-$160,000/Year

Job Description

You'll need to have:

• Bachelor's or advanced degree in Computer Science, Data Engineering, or a related field.
• 4-15 years of professional data engineering experience.
• Strong Python, SQL, and Spark (PySpark) skills, and/or Kafka.
• Snowflake (Snowpipe, Tasks, Streams) as a complementary warehouse.
• Databricks (Delta formats, workflows, cataloging) or equivalent Spark platforms.
• Hands-on experience building ETL/ELT with Prefect (or Airflow), dbt, Spark, and/or Kafka.
• Experience onboarding datasets to cloud data platforms (storage, compute, security, governance).
• Familiarity with Azure/AWS/GCP data services (e.g., S3/ADLS/GCS; Redshift/BigQuery; Glue/ADF).
• Git-based workflows CI/CD and containerization with Docker (Kubernetes a plus).

It is a bonus to have:

• Advanced APIM practices (custom policies, OAuth2/JWT, mTLS, private endpoints) and Azure AD integration.
• Integrating datasets into MCP tools/providers for LLM/agent applications; familiarity with frameworks such as LangChain or LlamaIndex.
• Data observability/quality tools (e.g., Great Expectations, Monte Carlo, Datafold) and strong lineage practices.
• Exposure to financial datasets and controls (PII handling, encryption, masking).

Desired Skills: Python | Spark | Tableau

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.