Data Factory Source Arrival Watchdog

Overview

This solution prevents Azure Data Factory pipelines from running on missing inputs. On a schedule it checks whether expected source files have arrived in an Azure Blob landing folder before the ADF load pipeline is due. If the expected files are present, it triggers the pipeline and posts a success notice to Microsoft Teams. If they are missing, it computes how overdue the source is against a configurable daily deadline and either posts an informational within-grace notice or - once the grace period is exceeded - posts a high-importance escalation naming the upstream data provider to chase.

Why it matters: running a pipeline on incomplete inputs produces bad or partial data. A source watchdog ensures the load only fires when inputs are ready, and gives the team early, actionable warning when a provider is late.

Ships Off (demo).

Use Case

A data team depends on partner/source files landing in a blob folder before a nightly ADF load. The watchdog runs on an interval through the load window: it confirms arrival, kicks off the pipeline when ready, and escalates to the provider if a source is overdue beyond an agreed grace period.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Per-source SLA / multiple files

Extend the expected prefix into a list and loop the filter/condition per expected file, tracking a missing array.

Different deadline cadence

Change the recurrence frequency and the deadline hour / grace minutes values.

Partial run

In the ready branch, pass pipeline parameters to CreatePipelineRun to load only the sources that have arrived.

Provider ping by email

Add an Outlook SendEmailV2 in the escalation branch using the provider email.

Run-status follow-up

Add ADF GetPipelineRun after CreatePipelineRun to confirm the run started, using the returned runId.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

greater(length(body('Filter_Expected_Files')), 0)

startsWith(item()?['DisplayName'], variables('varExpectedPrefix'))

addHours(startOfDay(utcNow()), variables('varDeadlineHour'))

div(sub(ticks(utcNow()), ticks(outputs('Compute_Deadline'))), 600000000)