RPT File Pipeline

An event-driven demo on Azure. A small web page generates a .RPT report file, drops it into a source blob store, and Azure automatically moves it through Event Grid → Service Bus → an Azure Functions worker → a destination blob store — showing every step with timestamps in real time.

---

What it does (the flow)

flowchart LR
    U[Browser] -->|filename| WEB[Web App - FastAPI / App Service]
    WEB -->|1-2 upload .rpt via PE| A2[(App2Blob)]
    A2 -->|3 BlobCreated| EG[Event Grid system topic]
    EG --> SB[[Service Bus queue: blob-events]]
    SB -->|4 trigger| FN[Functions worker - Flex Consumption]
    FN -->|5 read via PE| A2
    FN -->|6 copy via PE| PB[(ProdBlob)]
    WEB -->|7 poll every 5s| PB
    WEB & FN -->|step + time| ST[(State table: pipeline)]

Loading

1. Generate – Enter a filename in the web UI; the app builds <name>.rpt with dummy content.
2. Upload – The file is written to the App2Blob storage account over a Private Endpoint.
3. Event – A BlobCreated event is raised by an Event Grid system topic.
4. Route – Event Grid delivers the event to a Service Bus queue (blob-events).
5. Receive – An Azure Functions worker (Service Bus trigger) picks up the message.
6. Copy – The worker downloads the file from App2Blob (Private Endpoint) and writes it to ProdBlob (Private Endpoint).
7. Display – The web UI polls ProdBlob every 5 seconds and lists file name, size, and time. Every step (1–6) is recorded with a UTC timestamp and shown as a live timeline.

All storage access uses managed identity (no keys). Both apps are VNet-integrated, and all three storage accounts sit behind Private Endpoints.

---

Repository layout

.
├─ src/
│  ├─ web/            # FastAPI frontend (App Service)
│  │  └─ app/         # main.py, storage.py, report.py, templates/, static/
│  └─ worker/         # Azure Functions worker (Service Bus trigger)
│     ├─ function_app.py
│     └─ shared.py
├─ infra/             # Bicep infrastructure-as-code
│  ├─ main.bicep
│  ├─ main.parameters.json
│  └─ modules/resources.bicep
├─ azure.yaml         # Azure Developer CLI (azd) config
└─ docs/
   └─ Manual-Build-Guide.md   # build everything by hand in the Azure Portal

---

Prerequisites

Install once (Windows shown; macOS/Linux equivalents exist):

winget install --id Python.Python.3.11 -e
winget install --id Microsoft.AzureCLI -e
winget install --id Microsoft.Azd -e
winget install --id Microsoft.Azure.FunctionsCoreTools -e

You also need an Azure subscription you can create resources in.

Governed subscriptions (e.g. MCAPS): Azure Policy may force every storage account to disable public network access and shared keys. This project already handles that (Private Endpoints on all storage + user-assigned identities), so it deploys cleanly under those policies.

---

Deploy it (recommended: Azure Developer CLI)

Step 1 — Clone and open

git clone https://github.com/jvivek2k1/AMA_FILE_TEST.git
cd AMA_FILE_TEST

Step 2 — Sign in

azd auth login
az login

Step 3 — Create an environment (pick a region that supports Functions Flex Consumption)

azd env new rptpipe --location westus2
azd env set AZURE_SUBSCRIPTION_ID <your-subscription-id>

Step 4 — Provision + deploy everything

azd up

This creates all resources from infra/ and deploys both services. It takes ~5–8 minutes.

If provisioning fails once on the Event Grid → Service Bus step (Managed identity for event subscription does not have authorization…), that's RBAC propagation timing. Just run it again:

azd provision
azd deploy

Step 5 — Get the web URL

azd env get-values | Select-String WEB_URL

Open that URL in your browser.

---

Run / use it (step by step)

1. Open the web URL from Step 5 (e.g. https://web-xxxx.azurewebsites.net).
2. In the filename box type a name, e.g. dailysales, and click Generate & Upload.
3. Watch the Pipeline steps panel:
   • Steps 1–3 (source web) appear immediately.
   • Steps 4–6 (source worker) appear within a few seconds.
4. The ProdBlob contents table auto-refreshes every 5 seconds and shows dailysales.rpt with its size and last-modified time. Click view to see the file content.

---

Check that it works

Option A — In the browser (easiest)

After clicking Generate & Upload, you should see all 6 steps filled in with timestamps, and the file appear in the ProdBlob contents table. That confirms the full pipeline ran.

Option B — From the command line

$web = (azd env get-values | Select-String WEB_URL).ToString().Split('=')[1].Trim('"')

# 1) health check -> {"status":"ok"}
Invoke-RestMethod "$web/health"

# 2) trigger the pipeline
Invoke-RestMethod "$web/api/generate" -Method Post -ContentType application/json -Body '{"filename":"check1"}'

# 3) wait a few seconds, then confirm steps 4-6 were recorded by the worker
Start-Sleep 15
(Invoke-RestMethod "$web/api/status?filename=check1.rpt").steps |
  Select-Object step, source, name, time | Format-Table

# 4) confirm the file landed in ProdBlob
(Invoke-RestMethod "$web/api/prod-files").files | Format-Table

Success = you see six step rows (three web, three worker) and check1.rpt in the file list.

Option C — Check the Azure resources

• Storage → ProdBlob → Containers → reports: the copied .rpt files are listed.
• Service Bus → blob-events → Overview: message counts increment when you generate files.
• Function App → Monitoring → Log stream: shows the worker processing each event.
• App Service → Log stream: shows the web app requests.

---

Troubleshooting

Symptom	Likely cause	Fix
Steps 1–3 fail in the UI	Web app can't reach App2Blob / state table	Confirm id-web role assignments + VNet integration + blob/table Private Endpoints
Steps 4–6 never appear	Event not reaching queue or function	Check Event Grid subscription, Service Bus Data Sender on the system topic, Data Receiver on the function identity
azd provision Event Grid auth error	RBAC propagation timing	Re-run azd provision
Worker deploy 403 on storage	Deployment storage unreachable	Ensure stfunc* Private Endpoints (blob/queue/table) exist and the Function App VNet integration is on

---

Build it manually instead

Prefer clicking through the Azure Portal? Follow the full step-by-step guide: docs/Manual-Build-Guide.md.

---

Clean up (stop all charges)

azd down --purge

Or delete the resource group in the Portal: Resource groups → rg-rptpipe → Delete resource group.