Databricks Engineer :: Bethesda, MD (Onsite)

Databricks Engineer

Bethesda, MD - Onsite

Duration: 4+ months

Day-to-day Responsibilities:

• Design, build, and maintain scalable data pipelines using Databricks on Microsoft Azure.
• Integrate, transform, and manage large volumes of structured data for advanced analytics.
• Collaborate with data engineers, Finance SMEs, clients, and stakeholders to deliver secure and reliable data solutions.
• Ensure solutions are performance-optimized and compliant with data governance and security practices.

Key Responsibilities:

• Design and develop robust ETL pipelines using Databricks.
• Integrate and ingest data from multiple sources into Databricks.
• Implement data validation, quality checks, and transformation processes.
• Monitor and troubleshoot data pipeline performance, resolving bottlenecks and issues.
• Document data engineering processes and workflows for transparency and maintainability.
• Ensure data security, access management, and privacy frameworks aligned with federal compliance.
• Adopt best practices in data engineering, including medallion architecture and CI/CD automation.
• Contribute to data governance initiatives through data cataloging and metadata management.
• Drive continuous improvement and performance optimization efforts.

Required Skills:

• Databricks Data Engineer: Advanced
• DevOps: Intermediate
• Microsoft Azure: Intermediate
• Python: Advanced

Expected Deliverables:

• Python Databricks Notebooks for data ingestion.
• Support for Databricks on Azure Parquet files.
• Integration with Microsoft Power BI and Qlik SaaS reporting applications.