Required 4 Years Relevant certifications and coursework may substitute for formal education/experience requirements Required 4 Years Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions Required 4 Years ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM) Required 4 Years Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT Required 4 Years Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials Required 4 Years Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI Required 4 Years Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control Required 4 Years Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning Required 4 Years Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views Required 4 Years Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2 Required 4 Years Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers) Required 4 Years Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing Required 4 Years ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault Required 4 Years Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles Required 4 Years Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs Required 4 Years Strong written and verbal communication skills Required Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks Required 4 Years Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus) Highly desired Microsoft Certified: Azure Data Engineer Associate (DP-203) if already held Highly desired Power BI connectivity to Synapse (Import vs. Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy.