Cloud Data Engineer – AWS / Databricks / Power BI

Job Title: AWS Cloud Data Engineer AWS / Databricks / Power BI

Location: Remote (United States)
Experience: 10+ Years

Job Summary

We are looking for an experienced, hands-on Cloud Data Engineer with strong expertise in Databricks, AWS Cloud Services, Python, AWS Lambda, Amazon S3, MongoDB, and Power BI.

The ideal candidate will have extensive experience developing data pipelines, building ETL/ELT workflows, processing structured and semi-structured data, integrating cloud-based data services, and developing business intelligence dashboards.

The candidate should also have experience with GitHub, GitHub Actions CI/CD pipelines, and Agile development methodologies.

This is a hands-on development position, not an architecture or management role. We are looking for an individual contributor who can actively write code, develop data pipelines, troubleshoot technical issues, and deliver production-ready data solutions.

Mandatory Technical Skills

• Databricks: Hands-on experience with Databricks notebooks, data processing, transformations, and pipeline development.

• AWS Cloud: Strong experience developing and integrating cloud-based data solutions using AWS services.

• Python: Proficiency in Python programming, scripting, automation, and ETL/ELT development.

• AWS Lambda: Experience developing serverless functions and implementing event-driven data processing workflows.

• Amazon S3: Experience with data storage, file processing, data organization, and integration with data pipelines.

• MongoDB: Experience with querying, extracting, integrating, and processing semi-structured data.

• Power BI: Strong experience in dashboard development, data modeling, DAX, Power Query, and reporting.

• GitHub & GitHub Actions: Experience with source control, branching, pull requests, code reviews, and CI/CD pipeline automation.

• Agile Methodology: Experience working in Agile environments, participating in sprint planning, daily stand-ups, and collaborative development.

Preferred Skills

• Experience with Tableau dashboard development and data visualization.

• Hands-on experience with both Power BI and Tableau.

• Strong knowledge of Databricks Delta Lake, Apache Spark, and PySpark.

• Experience with AWS Glue, IAM, CloudWatch, and other AWS data services.

• Knowledge of data warehousing, data modeling, and data quality validation.

• Strong SQL development and query performance optimization skills.

• Experience with automated testing and monitoring of data pipelines.

Key Responsibilities 1. Cloud Data Engineering

• Design, develop, and maintain scalable data pipelines using Databricks, Python, and AWS services.

• Build data ingestion and transformation workflows for structured and semi-structured data.

• Process, manage, and integrate datasets stored in Amazon S3.

• Integrate MongoDB with cloud-based data processing workflows.

• Develop AWS Lambda functions for automated and event-driven data processing.

• Troubleshoot pipeline failures, data inconsistencies, and performance issues.

2. Databricks & Data Processing

• Develop and maintain Databricks notebooks, jobs, and data transformation workflows.

• Use Python and PySpark to process, cleanse, validate, and transform large datasets.

• Implement reusable data processing logic and efficient ETL/ELT workflows.

• Perform data quality checks, error handling, and pipeline monitoring.

• Optimize data processing performance, scalability, and reliability.

3. Power BI & Business Intelligence

• Develop, maintain, and enhance interactive Power BI dashboards and reports.

• Build data models, DAX measures, calculated columns, and transformations using Power Query.

• Connect Power BI to cloud data sources and curated datasets.

• Translate business requirements into meaningful reports and visualizations.

• Validate reporting accuracy and troubleshoot data discrepancies.

• Support Tableau dashboard development when required.

Note: Power BI is mandatory. Experience with both Power BI and Tableau is highly preferred.

4. CI/CD & Development Practices

• Use GitHub for source control, branching strategies, pull requests, and code reviews.

• Develop and maintain automated CI/CD workflows using GitHub Actions.

• Support automated testing, deployment, and release management processes.

• Follow secure coding standards and development best practices.

• Participate in Agile ceremonies and collaborate with data engineering, BI, QA, and application development teams.

Required Qualifications

• 10+ years of overall IT experience, with strong experience in cloud data engineering and development.

• Hands-on experience with Databricks, AWS, Python, AWS Lambda, Amazon S3, and MongoDB.

• Proven experience developing and supporting ETL/ELT pipelines and data transformation workflows.

• Strong Power BI development experience, including DAX, Power Query, data modeling, and dashboard development.

• Experience with GitHub Actions and CI/CD automation.

• Good understanding of Agile development methodologies.

• Strong SQL, debugging, data validation, and problem-solving skills.

• Ability to independently develop, test, troubleshoot, and maintain production-ready data solutions.