Data Factory Deployment and CI/CD Promotion

Overview

This solution adds governed ALM to Azure Data Factory. On a schedule it detects the latest publish on the adf_publish branch, requests a release approval, triggers the Azure DevOps deployment pipeline that promotes the ARM template to the target environment with environment-specific parameters, validates the target Data Factory, and records the outcome as an Azure DevOps work-item comment.

Why it matters: manual ADF promotion across dev to test to prod is error-prone. An approval-gated, parameterized promotion pipeline gives safe, repeatable, auditable releases - and a correlation id traces a single promotion end-to-end.

Ships Off (demo).

Use Case

A data engineering team has a Dev Azure Data Factory connected to Git. When an author clicks Publish, ADF writes ARM templates to the adf_publish branch. This flow turns that publish event into a controlled promotion: an approver gates the release, an Azure DevOps pipeline deploys the ARM template to the target environment with per-environment parameter files (stopping/starting triggers inside the pipeline), the target factory is validated, and the promotion decision is logged on a tracking work item.

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Make it event-driven

Replace the Recurrence trigger with an Azure DevOps service hook (push to adf_publish) or a Request (HTTP) trigger called from the publish pipeline.

Trigger management

The deployment pipeline should stop ADF triggers before deploy and restart them after, per ADF ALM best practice.

Multi-stage promotion

Parameterize the target environment label and the deploy pipeline to chain test to prod, requiring a separate approval per stage.

Rollback

On validation failure, run the deploy pipeline against the previous commit id captured in the release manifest.

Per-environment parameters

Pass an environment-specific ARM parameter file name into the pipeline templateParameters so each target gets its own linked-service / IR values.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

@{guid()}

@{first(body('Get_Latest_Publish_Commit')?['value'])?['commitId']}

@equals(body('Request_Promotion_Approval')?['outcome'], 'Approve')

concat('{"resources":{"repositories":{"self":{"refName":"refs/heads/', variables('varPublishBranch'), '"}}},"templateParameters":{"targetEnvironment":"', variables('varTargetEnvironment'), '","factoryName":"', variables('varTargetFactory'), '","correlationId":"', variables('varCorrelationId'), '"}}')