Cosmos DB Document Validation and Enrichment

Overview

This flow keeps Azure Cosmos DB data clean and complete. On a schedule it polls a source container for documents that have not yet been processed, validates each against a configurable required-fields ruleset, enriches valid documents via an external API (geocode / FX / lookups) and writes the enriched version back, and dead-letters invalid documents (with the list of missing fields) into a separate container. A Microsoft Teams summary reports the counts each run. A single correlationId (minted with guid() in the first action) is stamped on every written document and the Teams summary so a run can be traced end to end. Ships Off — going live requires only authorizing the connections and setting the environment variable values.

Use Case

A Cosmos-backed application accepts documents from many sources, with inconsistent completeness. The data team wants every document validated against a minimum schema, missing reference data filled in automatically, and anything that cannot be validated quarantined with a reason so it can be fixed and replayed — all without touching the app.

The flow is ideal for teams that:

• Validates every document against a configurable required-fields ruleset
• Enriches valid documents via an external API and writes them back to Cosmos
• Quarantines invalid documents to a dead-letter container with the missing-field reasons
• Loop-safe: each document is flagged so subsequent runs never reprocess it

Flow Architecture

Environment Variables

Connectors & Connections

Note — All connections are referenced as solution connection references; the flow is portable between environments as long as a connection is mapped at import time.

Customization Guide

Almost every realistic variant of this flow can be implemented by changing environment variable values. A few cases require small edits inside the flow definition — those are called out explicitly below.

Validation rules

Edit flowlibs_ValidationRules to add/remove required fields; no flow edit needed. Extend Compose Missing Fields for format/type checks.

Enrichment

Point flowlibs_EnrichApiBase at the real API and map its response fields into Compose Enriched Document. Only write high-confidence enrichment if the API returns a confidence score.

Replay

After fixing a dead-lettered document, clear its _enriched flag (or copy it back to the source container) and the next run reprocesses it.

Trigger

Swap the hourly recurrence for the Cosmos change-feed trigger if near-real-time processing is required.

Concurrency

The loop runs sequentially (concurrency = 1) so the counters are exact; raise it for throughput if you replace the counters with a different aggregation.

Key Expressions

The flow is intentionally light on Power Fx / WDL gymnastics — the heaviest expressions are the branch-name concatenation and the approval outcome check. They are listed below in the order they appear in the flow.

@concat('SELECT * FROM c WHERE (NOT IS_DEFINED(c["', variables('varEnrichedField'), '"]) OR c["', variables('varEnrichedField'), '"] = false)')

@filter(outputs('Compose_Required_Fields'), equals(trim(coalesce(string(items('Apply_to_each_Document')?[item()]), '')), ''))

@equals(length(outputs('Compose_Missing_Fields')), 0)

@setProperty(setProperty(setProperty(setProperty(setProperty(items('Apply_to_each_Document'), variables('varStatusField'), 'validated'), variables('varEnrichedField'), true), 'enrichment', body('Enrich_Via_HTTP')), 'enrichedAt', utcNow()), 'correlationId', variables('varCorrelationId'))