Data Lake Schema Drift Detection

Overview

This flow guards Azure Data Lake datasets against silent schema drift. When a new file arrives in a monitored dataset path, the flow infers its schema from the CSV header, compares it to the registered baseline schema stored in Dataverse, and classifies any change as additive (new columns) or breaking (removed columns). On drift it sets the dataset's promotion status to Blocked, opens a drift ticket in Dataverse, and alerts the data-engineering channel in Teams with the exact column diff. On first sight of a dataset it registers the baseline automatically. Why it matters: silent schema changes break downstream pipelines and reports. Detecting drift at ingestion stops bad data from propagating and gives data engineering a precise, actionable diff.

Use Case

A data engineering team ingests curated CSV extracts into a Data Lake zone. They want automatic protection against unexpected schema changes from upstream producers: new columns should be allowed (configurable), but removed or renamed columns must block promotion and raise an alert before consumers break.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Event-driven vs scheduled

Trigger_ListFiles polls metadata hourly by default. Lower the recurrence interval for faster detection, or pair with an Event Grid blob-created subscription for true push.

Allow / block additive

Flip flowlibs_AllowAdditive to false to enforce a strict data contract where any new column also blocks promotion.

File formats

The header-inference logic targets CSV. For Parquet/JSON, replace Infer Header Row with a footer/metadata parse.

Auto-evolve

Extend the additive-allowed branch to update the registry (increment flowlibs_schemaversion) so the baseline tracks approved additions automatically.

Ticketing

Swap or supplement Create Drift Ticket with an Azure DevOps / Jira work item if drift should open a tracked engineering ticket.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

trim(replace(first(split(string(body('Read_New_File_From_Data_Lake')), decodeUriComponent('%0A'))), decodeUriComponent('%0D'), ''))

@not(contains(variables('varRegisteredColumns'), item()))

@not(contains(variables('varNewColumns'), item()))

or(length(Filter_Added) > 0, length(Filter_Removed) > 0)

flowlibs_datasetpath eq '<dataset path>'