Data Engineer

Job Description

Job Role: Data Engineer

Job Type: Full Time

Job Location: Quincy, MA (Onsite)

Salary Range: $100000 to $120000/Annum + Full Time Benefits

Must Have Technical/Functional Skills

• Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience
• Must have 8+ years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql
• Must have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sql
• Must have strong experience with Databricks, including developing and Eligible to workimizing spark jobs, data transformations, and data processing workflows
• Must have experience and a good understanding on the below topics
• Databricks delta lake storage
• Unity catalog
• Autoloader
• Lakehouse architecture
• Medallion architecture
• Databricks serverless Eligible to workions
• Dataset and dataframe
• Databricks clusters configuration and sizing
• Databricks job orchestration
• Spark structured streaming
• Must have experience with complex and large volume data streaming and data transformations using Azure Databricks
• Must have extensive experience in datawarehouse / data lakehouse implementation
• Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services The candidate should be able to provide data ingestion, streaming and transformations solutions and do development work Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous Good to have experience with containerization technologies such as docker and Kubernetes

Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination, based solely on their qualifications, ability, competence and performance. This email and its attachments may contain confidential or proprietary information and is intended only for the recipient(s). If you received this message in error, please disregard it and notify the sender. If you no longer wish to receive our communications, you may unsubscribe at any time.

Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.