Data Factory Run Orchestration and Dependencies

Overview

This flow orchestrates a dependency-ordered sequence of Azure Data Factory pipelines from a single, observable Power Automate Cloud flow. It reads a JSON dependency graph (the DAG), triggers each stage's ADF pipeline in order, polls each run to completion, gates downstream stages on upstream success, writes a per-stage lineage/audit row to Dataverse, and posts an end-to-end run report to Microsoft Teams.

Why it matters: cross-pipeline dependencies are hard to express inside a single ADF pipeline. A flow orchestrator gives flexible, observable end-to-end control with a governed audit trail and a single notification surface.

Ships Off (demo).

Use Case

A data team runs several ADF pipelines that must execute in a fixed dependency order (ingest raw to transform curated to publish marts). Each stage must finish successfully before the next begins; a failure should halt the remaining stages and surface immediately. This flow coordinates that sequence, records every stage outcome, and reports the whole run to Teams.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Conditional stages

Add a runIf flag to each DAG entry and wrap Start_Stage_Pipeline in a Condition to skip stages based on upstream output.

Fan-out / fan-in

Split the DAG into independent branches and raise concurrency (or parallel Scopes), joining before the report. The build runs sequentially for deterministic ordering.

Backfill

Parameterize each stage's parameters (e.g. date windows) in the DAG and pass them to CreatePipelineRun.

Retry transient failures

Add a retry policy around Start_Stage_Pipeline when the run status is a transient failure.

Event/on-demand start

Replace the nightly recurrence with an HTTP, Dataverse, or Event Grid trigger.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

json(parameters('EnvVar_PipelineDag'))

equals(variables('varOverallOutcome'),'Succeeded')

or(equals(variables('varRunStatus'),'Succeeded'),or(equals(variables('varRunStatus'),'Failed'),equals(variables('varRunStatus'),'Cancelled')))

div(coalesce(body('Get_Run_Status')?['durationInMs'],0),1000)