We are looking for a Cloud & Data Engineer to help build, optimize, and operate our Voice of Customer (VoC) AI Platform.
The platform ingests structured and unstructured data from multiple enterprise sources with varying degrees of signal-to-noise ratio and is primarily built on Microsoft Azure, with data processing and ETL/ELT workloads powered by Azure Databricks.
This is a hands-on role spanning data engineering, cloud infrastructure, and platform reliability.
Key Responsibilities
Data Engineering
Build and maintain ETL/ELT pipelines for structured and unstructured data sources.
Develop data-processing workloads using Azure Databricks, Python/PySpark, and SQL.
Optimize Databricks jobs, compute, and data layouts for performance and cost.
Work with Delta Lake and Unity Catalog for data storage, governance, and access control.
Implement data quality checks, monitoring, error handling, and recovery mechanisms.
Cloud & Platform Engineering
Build, operate, and optimize Azure services including App Service, Functions, Blob/ADLS storage, Azure SQL DB for Postgres, Key Vault, Container Registry and Azure Monitor.
Experience with cloud AI services including vector search and embedding pipelines.
Improve platform performance, scalability, reliability, security, and cost efficiency.
Implement secure identity, and access management using Azure managed identities, Key Vault, secrets managed via scopes.
Support development, staging, and production environments.
Automation & Operations
Improve platform observability through logging, monitoring, dashboards, and alerting.
Automate infrastructure and deployments using Infrastructure as Code (IaC) and CI/CD.
Troubleshoot issues across cloud services, storage, databases, networking, and data pipelines.
Support production incidents and root-cause analysis.
Collaborate with application, AI/ML, and data engineering teams.
Skills:
Required Qualifications
3 5 years of experience in data engineering, cloud engineering, or a related role.
Hands-on experience with Microsoft Azure, Databricks, APIs, and object storage.
Strong skills in Python/PySpark, SQL.
Experience building and operating production data ETL/ELT pipelines.
Experience with relational and semi-structured/unstructured data, database technologies.
Experience with Spark, Delta Lake, and lakehouse architectures.
Working knowledge of Databricks Unity Catalog and data-access controls.
Understanding of cloud security concepts such as RBAC, managed identities, service principals, and Key Vault.
Experience with Git, CI/CD, monitoring, troubleshooting, and performance optimization.
Preferred Qualifications
Experience with Terraform or ARM templates.
Experience with Azure DevOps or GitHub Actions.
Familiarity with Azure networking, including VNets and private endpoints.
Experience with Azure Data Factory, Databricks Workflows, or Airflow.
Familiarity with data platforms supporting AI/ML, NLP, analytics, or GenAI applications.