Position: - Databricks Lead/Architecture.
Location: - Remote (Need to be in the office in Washington, DC, once a Quarter for Meetings).
Duration: - Long Term.
Job Description:
Architecture & Design:
- Define and design end-to-end data Lakehouse architecture leveraging Databricks, Delta Lake, and cloud-native services.
- Create reference architectures for batch, real-time, and streaming data pipelines.
- Architect data ingestion, curation, storage, and governance frameworks.
- Ensure the platform is scalable, secure, and optimized for performance and cost.
- Establish standards for data lineage, metadata management, and compliance.
- Work with enterprise architects to align Databricks solutions with the overall cloud strategy (AWS).
Leadership & Delivery:
- Lead and mentor a team of data engineers in building robust data pipelines.
- Collaborate with data scientists, BI teams, and business stakeholders to enable advanced analytics and AI/ML use cases.
- Drive adoption of DevOps and CI/CD practices for data engineering.
- Review designs and solutions to ensure adherence to architectural principles.
Implementation & Optimization:
- Build, optimize, and manage large-scale PySpark/SQL pipelines in Databricks.
- Enable real-time data processing through Kafka, Kinesis, or Event Hubs.
- Implement security best practices, including RBAC, data masking, and encryption.