Data Lake Raw-Zone Ingestion and Cataloging

Overview

This flow operationalizes raw-zone intake for a lakehouse on Azure Data Lake (Gen1). On a schedule it scans the raw/landing zone, reads each file, validates it, registers it in a Dataverse data catalog (file name, extension/schema, date partition, source system, row count, status), then promotes valid files to a date-partitioned curated zone and moves invalid ones to a quarantine zone. It finishes by posting a run summary to the data-engineering Teams channel.

Why it matters: ungoverned raw-zone dumps become data swamps. Cataloging every file and enforcing zone promotion keeps the lake organized, auditable, and trustworthy, with a single correlation id tracing each ingestion batch end to end.

Ships Off (demo). All-connector reference implementation - no HTTP fallbacks.

Use Case

A data-engineering team lands files into a Data Lake raw zone from upstream systems (ERP exports, partner feeds, IoT batches) and needs them cataloged, validated, and promoted reliably without manual babysitting - a Dataverse catalog of everything that arrived, automatic promotion of conformant files, automatic quarantine of bad ones, and a per-run Teams summary.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Trigger style

Replace the Recurrence with the Data Lake list-files trigger (or Event Grid blob-created) for near-real-time intake.

Validation depth

Extend the valid check with schema-conformance (header match, column count, JSON/XML parse) beyond extension + non-empty.

Row counting

Compose Row Count handles CSV; add branches for JSON/TSV/XML to populate the row count per format.

Partitioning

The curated path uses yyyy/MM/dd; switch to source- or entity-based partition folders by editing Compose Curated Path.

Heavy transforms

For large files, trigger Azure Data Factory/Databricks from the valid branch instead of processing content in the flow.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

body('List_Raw_Zone_Files')?['FileStatuses']?['FileStatus']

toLower(last(split(items('Apply_To_Each_File')?['pathSuffix'],'.')))

if(equals(outputs('Compose_File_Extension'),'csv'),sub(length(split(replace(string(body('Read_File_Content')),decodeUriComponent('%0D'),''),decodeUriComponent('%0A'))),1),0)

concat(variables('varCuratedZonePath'),'/',formatDateTime(utcNow(),'yyyy/MM/dd'),'/',items('Apply_To_Each_File')?['pathSuffix'])