Data Factory Deployment and CI/CD Promotion
When an Azure Data Factory ARM template is published from collaboration, the flow promotes it across environments - running an approval, deploying via the Azure DevOps pipeline with environment-specific parameters, validating the deployment, and notifying the team. Adds governed ALM to Data Factory.
Provided as-is, without warranty of any kind. Review and test each pattern in a non-production environment before deploying it to live automations. See our Terms.
Overview
This solution adds governed ALM to Azure Data Factory. On a schedule it detects the latest publish on the adf_publish branch, requests a release approval, triggers the Azure DevOps deployment pipeline that promotes the ARM template to the target environment with environment-specific parameters, validates the target Data Factory, and records the outcome as an Azure DevOps work-item comment.
Why it matters: manual ADF promotion across dev to test to prod is error-prone. An approval-gated, parameterized promotion pipeline gives safe, repeatable, auditable releases - and a correlation id traces a single promotion end-to-end.
Ships Off (demo).
Use Case
A data engineering team has a Dev Azure Data Factory connected to Git. When an author clicks Publish, ADF writes ARM templates to the adf_publish branch. This flow turns that publish event into a controlled promotion: an approver gates the release, an Azure DevOps pipeline deploys the ARM template to the target environment with per-environment parameter files (stopping/starting triggers inside the pipeline), the target factory is validated, and the promotion decision is logged on a tracking work item.
Flow Architecture
Check For New ADF Publish
Recurrence (every 4h)Polls for a new publish on the adf_publish branch. Replace with an Azure DevOps service hook for event-driven promotion.
Initialize Correlation Id & Config
Initialize variableMints a correlation id and loads subscription, resource group, target factory, ADO org/project/repo, publish branch, ARM template path, deploy pipeline id, environment label, approver, and work item id from env vars.
Get Latest Publish Commit
Azure DevOps - HttpRequest (GET)Reads the most recent commit on the adf_publish branch - the ARM template version being promoted.
Get Published ARM Template
Azure DevOps - HttpRequest (GET)Retrieves the ARMTemplateForFactory.json artifact content from the publish branch.
Request Promotion Approval
Approvals - StartAndWaitForAnApprovalRelease gate before promoting to the target environment, using a composed release manifest.
Deploy + Validate (Approved)
Azure DevOps - HttpRequest (POST)Triggers the deployment pipeline with environment-specific parameters, validates the target factory via GetDataFactory, and comments success on the work item.
Record Rejection (Rejected)
Azure DevOps - HttpRequest (POST)Records a rejected promotion as a work-item comment so the decision is auditable.
Environment Variables
| Schema name | Type | Default | Description |
|---|---|---|---|
| flowlibs_AdfSubscriptionId | String | <your-subscription-id> | Azure subscription id of the target Data Factory. |
| flowlibs_AdfResourceGroup | String | rg-data-prod | Resource group of the target Data Factory. |
| flowlibs_AdfFactoryName | String | adf-enterprise-prod | Name of the Data Factory being promoted to / validated. |
| flowlibs_AzureDevOpsOrgName | String | MyOrg | Azure DevOps organization name. |
| flowlibs_AzureDevOpsProject | String | MyProject | Azure DevOps project hosting the ADF repo and deploy pipeline. |
| flowlibs_AzureDevOpsRepoId | String | MyRepo | Repository holding the ADF collaboration and publish branches. |
| flowlibs_ApproverEmail | String | you@yourcompany.com | Release approver who gates the promotion. |
| flowlibs_AdfPublishBranch | String | adf_publish | Branch holding the published ARM templates. |
| flowlibs_AdfArmTemplatePath | String | /ARMTemplateForFactory.json | Repo path of the factory ARM template. |
Connectors & Connections
| Connector | API name | Actions used |
|---|---|---|
| Azure Data Factory | shared_azuredatafactory | GetDataFactory |
| Azure DevOps | shared_visualstudioteamservices | HttpRequest |
| Approvals | shared_approvals | StartAndWaitForAnApproval |
Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.
Customization Guide
Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.
- Make it event-driven
- Replace the Recurrence trigger with an Azure DevOps service hook (push to adf_publish) or a Request (HTTP) trigger called from the publish pipeline.
- Trigger management
- The deployment pipeline should stop ADF triggers before deploy and restart them after, per ADF ALM best practice.
- Multi-stage promotion
- Parameterize the target environment label and the deploy pipeline to chain test to prod, requiring a separate approval per stage.
- Rollback
- On validation failure, run the deploy pipeline against the previous commit id captured in the release manifest.
- Per-environment parameters
- Pass an environment-specific ARM parameter file name into the pipeline templateParameters so each target gets its own linked-service / IR values.
Key Expressions
The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.
EXPR.01Correlation id
Minted first and propagated into the approval, pipeline body, and comments.
EXPR.02Latest published commit
Commit id of the ARM template being promoted.
EXPR.03Approval gate
Branches on the approval outcome.
EXPR.04Deploy pipeline body (ADO)
Run-pipeline body pinning the publish branch and passing environment parameters.
Comments
Sign in to join the conversation.
Sign inNo comments yet. Be the first to share your experience with this flow.