Data Lead/Architect (Databricks)

Apolis

Charlotte, NC(remote)

Apply

JOB DETAILS

SALARY

$60–$65 Per Hour

SKILLS

ARM (Advanced RISC Machine), Acceptance Testing, Analysis Skills, Apache, Apache Spark, Autoscaling, Broadcasting, Cisco Unity, Code Reviews, Cost Control, Data Management, Database Extract Transform and Load (ETL), DevOps, Git, Hubs, Input/Output, Mentoring, Microsoft SQL Server, Microsoft Windows Azure, Performance Tuning/Optimization, Process Improvement, Production Systems, Python Programming/Scripting Language, Root Cause Analysis, SQL (Structured Query Language), System Architecture

LOCATION

Charlotte, NC(remote)

POSTED

2 days ago

Role: Data Lead/Architect (Databricks)
Location: Remote
Duration: 6+ months

Primary Skill Required for the Role: Databricks
Level Required for Primary Skill: Advanced (6-9 years experience)

Responsibilities:
§ Design, develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks
§ Write and tune PySpark / Spark SQL notebooks for large-scale data transformation
§ Architect end-to-end data solutions across dev → UAT → prod environments using Unity Catalog
§ Lead and drive design discussions with client architects and other counterparts
§ Collaborate with different teams on data contracts and schema agreements
§ Lead design and optimization of high-volume data pipeline.
§ Define and enforce data engineering standards — naming conventions, partitioning strategies, cluster configurations, Spark tuning
§ Drive performance optimization — AQE tuning, liquid clustering, broadcast joins, shuffle partition management
§ Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies
§ Conduct root cause analysis on production incidents and implement permanent fixes
§ Mentor junior and mid-level engineers through code reviews and pair programming
§ Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs)

Mandatory Skills

• Python, PySpark, Spark SQL, SQL Server
• Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
• Databricks (Delta Lake, Unity Catalog, Workflows)
• Apache Airflow
• Git / Azure DevOps
• Deep Spark internals (DAG optimization, spill analysis, skew handling)
• Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
• Unity Catalog governance (row/column security, external locations, system tables)
• IaC — Terraform, Azure ARM templates

About the Company

Apolis

Since 1996, RJT has provided successful SAP, Oracle, and IT consulting solutions and staffing services to clients around the world. The new Apolis brings you the same personalized service fortified with a greater array of IT solutions, global expertise, and cost-management strategies.

We are a global IT consultancy that seamlessly integrates experts and leading-edge solutions into your organization so you can focus on what really matters.

COMPANY SIZE

500 to 999 employees

INDUSTRY

Computer/IT Services

EMPLOYEE BENEFITS

Paid Sick Days, Employee Referral Program, Employee Events, Retirement / Pension Plans

WEBSITE

https://www.apolisrises.com/

Resume Resources

Free Resume Templates Free Resume Builder