Data Engineer / Data Architect

Job Title: Senior Data Engineer / Data Architect Databricks

Location: Charlotte, NC

Work Arrangement: Onsite

Job Type: Contract

Industry: Insurance

Experience Level: 12+ Years

Job Summary

We are seeking a highly experienced Senior Data Engineer / Data Architect with deep Databricks hands-on expertise and strong Insurance domain experience to design, build, and implement scalable enterprise data solutions.

The ideal candidate brings 12+ years of experience across Databricks, Apache Spark, Python, SQL, data architecture, and modern cloud platforms. You will collaborate with business stakeholders, cross-functional engineering teams, and analysts to architect next-generation data platforms and analytics solutions.

Key Responsibilities

• Design, build, and optimize enterprise-scale data architecture and pipelines using Databricks, Apache Spark, Python, and SQL.

• Architect robust data lakes, lakehouse platforms, and data warehouses utilizing Delta Lake, Delta Live Tables (DLT), Unity Catalog, and Databricks Workflows.

• Develop high-performance batch and streaming ETL/ELT data pipelines, including data ingestion, transformation, cleansing, and validation.

• Optimize Spark jobs, SQL queries, and complex workloads for high performance and cost efficiency.

• Implement enterprise data governance, data quality standards, security policies, lineage, and access controls.

• Translate complex business requirements into scalable technical data architectures and engineering solutions.

• Troubleshoot production data issues, provide root-cause analysis, and mentor engineering team members.

Insurance Domain Responsibilities

• Apply strong domain knowledge across Property & Casualty (P&C), Life, or Health Insurance data (e.g., Policy, Claims, Premium, Billing, Underwriting, Rating, Coverage, Loss, and Risk).

• Integrate data across core insurance applications such as policy administration, claims management, underwriting engines, and billing systems.

Technical Skills & Qualifications

• Total Experience: 12+ years of progressive experience in Data Engineering, Data Architecture, or Enterprise Data Solutions.

• Core Technologies: Databricks (Workspace, PySpark, Delta Lake, DLT, Unity Catalog, SQL, CI/CD), Apache Spark, Python, and advanced SQL.

• Data Architecture: Expertise in Data Lakehouse, Data Warehousing, ETL/ELT design, and dimensional/reusable data modeling.

• Cloud Infrastructure: Strong hands-on experience with major cloud platforms (Azure preferred; AWS or GCP highly valued) and integration tools (e.g., Azure Data Factory, ADLS, AWS S3, Glue, Snowflake, Kafka, Airflow).

• Software Practices: Proficiency with Git, CI/CD automated deployments, and Agile/Scrum delivery.

• Education: Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred Qualifications

• Databricks Certified Data Engineer / Data Architect.

• Major Cloud Provider Certification (Azure / AWS / GCP).

• Experience with P&C insurance platforms, master data management (MDM), and real-time streaming architectures.

Key Skills / Keywords

Senior Data Engineer, Data Architect, Databricks, Azure Databricks, Apache Spark, PySpark, Python, SQL, Delta Lake, Delta Live Tables, Unity Catalog, Lakehouse, ETL, ELT, Data Modeling, Data Architecture, Azure, AWS, GCP, Snowflake, Kafka, Insurance Domain, P&C Insurance, Claims, Policy, Underwriting, Data Governance, Data Integration.