Multi-Step Pipeline with Queue Handoffs

Overview

FlowLibs - Multi-Step Pipeline with Queue Handoffs is a three-flow reference implementation of the event-driven microflow architecture pattern in Power Automate. Each flow performs one stage of a classic ETL pipeline - validate -> transform -> load - and the stages are decoupled with Azure Storage Queues so each step can scale, fail, retry, and evolve independently of the others. The Loader stage gates the final write with a Power Automate Approval so a human stays in the loop on what actually lands in the system of record.

The brief: chain multiple flows via Azure Queues so that Step 1 validates, Step 2 transforms, and Step 3 loads. This solution does exactly that - three separate cloud flows in one Dataverse solution, glued together by two queues (pipeline-validated and pipeline-transformed) and traced end-to-end with a single correlationId GUID.

Use Case

Use this pattern when you need to decouple long, brittle, or differently-scaled stages, survive partial failures, fan out without rewrites, and trace audit-grade lineage through a multi-stage pipeline. A 30-second validation, a 2-second transform, and a 10-minute approval shouldn't share one flow run timer or one connection's throttling budget - splitting them across queue-decoupled stages lets each scale and recover on its own clock. Every message carries the same correlationId from the SharePoint submission all the way through to the final loaded record and the approval response.

The flow is ideal for teams that:

• Decouple long, brittle, or differently-scaled stages so each runs on its own timer and throttling budget
• Survive partial failures - if one stage breaks, upstream stages keep filling the queue and downstream drains the backlog once fixed
• Fan out without rewrites - adding a new stage is a new flow that subscribes to an existing queue, no upstream edits
• Trace audit-grade lineage end-to-end via a single correlationId GUID minted at the front of the pipeline

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Activate the connections

Open each flow in the designer once and authorize the four connection references (SharePoint, Azure Queues, Outlook, Approvals). Until then the flows show 'Add an authorized connection' banners - expected for a demo solution shipping Off.

Provision the Azure Storage Queues

Create two queues in the Azure Storage account named in flowlibs_AzureQueueStorageAccountName. Default queue names are pipeline-validated and pipeline-transformed. The Azure Queues connector auto-creates queues on first PutMessage_V2, but pre-creating them avoids surprises.

Re-point the SharePoint lists

If you don't keep FlowLibs - Pipeline Submissions and FlowLibs - Pipeline Loaded, update the two list-name env vars and let the designer re-bind columns. The Loader's Create_Loaded_Item action contains hardcoded column names (Title, CorrelationId, Amount, RequesterEmail, Department, Priority, ApprovalDecision, Approver, ApprovalComments, SourceListItemId, LoadedAt) - those must match the destination list.

Tune the recurrences

Both pollers run every 5 minutes. For higher throughput shorten the cadence; for cost-sensitive deployments lengthen it. The Loader's visibilitytimeout of 1800s assumes the approver responds within 30 minutes; raise it for slow approvers.

Set the high-value threshold

flowlibs_PipelineHighValueThreshold defaults to 10000. Items at or above this amount are marked priority: "High" in the transformed envelope. To gate ONLY high-value items, wrap Request_Pipeline_Approval in an If on body('Parse_Transformed_Payload')?['highValue']

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

@{guid()}

@{coalesce(triggerOutputs()?['body/RequesterEmail'], '')}

@if(greaterOrEquals(outputs('Compose_Amount'), variables('varHighValueThreshold')), 'High', if(greaterOrEquals(outputs('Compose_Amount'), 1000), 'Standard', 'Low'))

@coalesce(body('Get_Validated_Messages')?['QueueMessagesList']?['QueueMessage'], createArray())

@first(body('Request_Pipeline_Approval')?['responses'])?['responder']?['email']