Honor .dcomp allowlist and round-trip sql_variant

.gitignore

@@ -59,3 +59,14 @@ logs/
 
 # Argfiles that may contain passwords (keep sample/ fixtures, not real ones)
 *.argfile
+
+# Local captured jobs, backups, and Devart project files — never commit
+_data/
+MARS_PROD_AZURE_TO_ONPREM_DATA_SYNC.bat
+*_PROD_*.bat
+*.dcomp
+*.scomp
+*.bak
+!tests/fixtures/*.dcomp
+!tests/fixtures/*.scomp
+UPDATE_AsOfDate.sql

README.md

@@ -51,6 +51,7 @@ ArtSync replaces the executables. `.bat` files, Task Scheduler entries, and argf
 | `schemacompare … /sync` | Yes | 0 applied, 112 nothing to sync |
 | `schemacompare … /sync:file.sql` | Yes | 101; target untouched |
 | `datacompare /source … /target … /sync` | Yes | Same 100 / 101 / 0 / 112 / 108 / 40 pattern |
+| `/compfile:.dcomp` + `/sync` | Yes (data) | CLI `/source` `/target` override project connections |
 | `/argfile` + CLI override | Yes | CLI wins, matching Devart precedence |
 | `/filter:.scflt` | Yes | 114 on bad XML |
 | `/report` + `/reportformat HTML\|XML\|CSV` + `/log` | Yes | 107 report I/O, 106 log I/O; passwords redacted |
@@ -62,7 +63,7 @@ These are not bugs. They exit 10 or leave the target alone **by design**.
 
 | Item | Behavior | Why |
 |---|---|---|
-| `/compfile` (`.scomp` / `.dcomp`) | Exit 10 | Do not invent project XML until a captured file is in `tests/fixtures/` |
+| `/compfile` (`.scomp`) | Exit 10 | Schema project files are not loaded in v1 |
 | `backup:` / `snapshot:` / `scriptsfolder:` endpoints | Exit 10 | Live databases only in v1 |
 | `/reportformat:XLS` | Exit 10 | Excel reports are out of scope |
 | Extra tables only on the target | Left in place | DacFx `DropObjectsNotInSource` stays false — dropping extras by default is unsafe |
@@ -110,6 +111,18 @@ Integration tests skip unless `ARTSYNC_INTEGRATION=true`.
 
 That starts SQL Server 2022, seeds `artsync_src` / `artsync_tgt`, and runs the schema + data suite (FK graphs, data types, reports, apply). See [tests/fixtures/README.md](tests/fixtures/README.md).
 
+### MARS lab (Azure SQL + local restore)
+
+Disposable stand-ins for the captured Azure → on-prem job. Not production. See [infra/README.md](infra/README.md).
+
+```bash
+./scripts/lab-azure.sh   # Azure SQL MARS (westus2, serverless)
+./scripts/lab-up.sh      # Docker SQL + restore _data/*.bak as [MARS]
+./scripts/lab-smoke.sh   # both sides reachable
+```
+
+`.env` is gitignored. Tear down with `./scripts/lab-down.sh`.
+
 ### Publish the three executable names
 
 **Windows (PowerShell):**
@@ -134,7 +147,7 @@ CI publishes `linux-x64`, `win-x64`, `osx-x64`, and `osx-arm64` on every green m
 
 ### Drop-in on Windows scheduler
 
-Replace the Devart executable on `PATH`, or update the scheduled task program from `….com` to `….exe` of the same basename. Confirm the job uses live `/source` and `/target`, not `/compfile` or `backup:` endpoints.
+Replace the Devart executable on `PATH`, or update the scheduled task program from `….com` to `….exe` of the same basename. Live `/source` and `/target` still override `/compfile` connections. `.scomp` and `backup:` endpoints are not supported.
 
 Recommended first production cutover:
 
@@ -223,7 +236,8 @@ All engine code sits behind thin interfaces. `ArtSync.Cli` depends only on `IArg
 | `/sync` / `/sync:<file>` | Complete |
 | `/report` + `/reportformat:HTML\|XML\|CSV` | Complete (XLS → exit 10) |
 | `/log` | Complete (passwords redacted) |
-| `/compfile` (`.scomp` / `.dcomp`) | Exit 10 until a real file is in `tests/fixtures/` |
+| `/compfile` (`.dcomp`) | Complete — connections, compare/sync options, exclude masks, `Included="False"` objects. CLI wins. |
+| `/compfile` (`.scomp`) | Exit 10 — schema projects not loaded in v1 |
 | `/source backup:…` | Exit 10 — out of scope in v1 |
 | Extra objects on the target (schema drop) | Not dropped (`DropObjectsNotInSource` stays false) |
 | Azure SQL Database | Supported |

docker-compose.lab.yml

@@ -0,0 +1,11 @@
+# Lab overlay: mount captured backups and give SQL Server more RAM.
+# Usage: docker compose -f docker-compose.yml -f docker-compose.lab.yml up -d
+#
+# CI keeps using docker-compose.yml only so GitHub Actions does not create an empty _data/.
+
+services:
+  sqlserver:
+    environment:
+      MSSQL_MEMORY_LIMIT_MB: "4096"
+    volumes:
+      - ./_data:/bak:ro

infra/README.md

@@ -0,0 +1,49 @@
+# ArtSync lab
+
+Disposable **source** (Azure SQL) and **target** (Docker SQL Server) for the captured MARS job. Not production.
+
+| Side | What | How |
+|---|---|---|
+| Target (on-prem stand-in) | SQL Server 2022 in Docker, database `MARS` | `./scripts/lab-up.sh` restores `_data/*.bak` |
+| Source (Azure stand-in) | Azure SQL Database `MARS`, serverless | `./scripts/lab-azure.sh` |
+
+`.env` at the repo root is gitignored. `_data/` stays gitignored.
+
+## One-time
+
+```bash
+# 1. Azure SQL server + database + firewall for this IP
+./scripts/lab-azure.sh
+
+# 2. Local SQL + restore the captured backup (several minutes)
+./scripts/lab-up.sh
+```
+
+Resource group: `rg-artsync`. Azure SQL defaults to **westus2** (this Microsoft Partner Network subscription blocks SQL in eastus / eastus2). SKU: serverless `GP_S_Gen5_1` (auto-pause after 60 minutes).
+
+## Compare without applying
+
+```bash
+set +H
+set -a && source .env && set +a
+dotnet run --project src/ArtSync.Cli -- \
+  /datacompare \
+  /compfile:"$PWD/_data/TheSync!.dcomp" \
+  /source connection:"$ARTSYNC_SRC_CS" \
+  /target connection:"$ARTSYNC_TGT_CS"
+```
+
+Empty Azure vs restored local will show many source-only / different rows until you seed Azure (schema then data `/sync` local → Azure, or import a bacpac). Do not `/sync` Azure → local until you mean to overwrite the restored copy.
+
+## Tear down Azure
+
+```bash
+./scripts/lab-down.sh
+# also drop local SQL volume: ARTSYNC_LAB_DOWN_DOCKER=1 ./scripts/lab-down.sh
+```
+
+## Smoke test
+
+```bash
+./scripts/lab-smoke.sh
+```

infra/lab-tables-only.scflt

@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Lab filter: tables/schemas/keys only. Procs, views, users, and Azure-hostile types stay off. -->
+<FiltersCollection>
+  <Filter ObjectName="Table" Checked="True" />
+  <Filter ObjectName="Schema" Checked="True" />
+  <Filter ObjectName="Index" Checked="True" />
+  <Filter ObjectName="PrimaryKeyConstraint" Checked="True" />
+  <Filter ObjectName="ForeignKeyConstraint" Checked="True" />
+  <Filter ObjectName="DefaultConstraint" Checked="True" />
+  <Filter ObjectName="CheckConstraint" Checked="True" />
+  <Filter ObjectName="UniqueConstraint" Checked="True" />
+  <Filter ObjectName="UserDefinedTableType" Checked="True" />
+  <Filter ObjectName="UserDefinedDataType" Checked="True" />
+  <Filter ObjectName="View" Checked="False" />
+  <Filter ObjectName="Procedure" Checked="False" />
+  <Filter ObjectName="ScalarFunction" Checked="False" />
+  <Filter ObjectName="TableValuedFunction" Checked="False" />
+  <Filter ObjectName="DmlTrigger" Checked="False" />
+  <Filter ObjectName="DatabaseDdlTrigger" Checked="False" />
+  <Filter ObjectName="User" Checked="False" />
+  <Filter ObjectName="Role" Checked="False" />
+  <Filter ObjectName="DatabaseRole" Checked="False" />
+  <Filter ObjectName="ApplicationRole" Checked="False" />
+  <Filter ObjectName="SqlAssembly" Checked="False" />
+  <Filter ObjectName="XmlSchemaCollection" Checked="False" />
+  <Filter ObjectName="Synonym" Checked="False" />
+  <Filter ObjectName="Sequence" Checked="False" />
+  <Filter ObjectName="DatabaseOptions" Checked="False" />
+  <Filter ObjectName="Filegroup" Checked="False" />
+  <Filter ObjectName="Permission" Checked="False" />
+  <Filter ObjectName="RoleMembership" Checked="False" />
+  <Filter ObjectName="ExtendedProperty" Checked="False" />
+  <Filter ObjectName="PartitionFunction" Checked="False" />
+  <Filter ObjectName="PartitionScheme" Checked="False" />
+  <Filter ObjectName="FullTextCatalog" Checked="False" />
+  <Filter ObjectName="Certificate" Checked="False" />
+</FiltersCollection>

infra/lab.env.example

@@ -0,0 +1,22 @@
+# ArtSync lab (do not commit the real .env)
+#
+# Copy to ../.env and fill the Azure password after ./scripts/lab-azure.sh
+
+ARTSYNC_RESOURCE_GROUP=rg-artsync
+ARTSYNC_LOCATION=westus2
+ARTSYNC_SQL_ADMIN=artsyncadmin
+ARTSYNC_SQL_PASSWORD=
+
+# Filled by lab-azure.sh
+ARTSYNC_AZURE_SERVER=
+ARTSYNC_AZURE_DB=MARS
+
+# Local Docker SQL (on-prem stand-in)
+ARTSYNC_LOCAL_SERVER=localhost,1433
+ARTSYNC_LOCAL_DB=MARS
+ARTSYNC_LOCAL_USER=sa
+ARTSYNC_LOCAL_PASSWORD=ArtSync_Test@2026
+
+# Filled by lab-azure.sh (quoted; values contain spaces/semicolons)
+# ARTSYNC_SRC_CS="Server=….database.windows.net,1433;Database=MARS;User ID=artsyncadmin;Password=…;Encrypt=True;TrustServerCertificate=False;Connection Timeout=60"
+# ARTSYNC_TGT_CS="Server=localhost,1433;Database=MARS;User ID=sa;Password=ArtSync_Test@2026;TrustServerCertificate=True;Connection Timeout=30"

infra/main.bicep

@@ -0,0 +1,65 @@
+@description('ArtSync lab: Azure SQL as the cloud source stand-in. Not production.')
+param location string = resourceGroup().location
+param administratorLogin string
+@secure()
+param administratorLoginPassword string
+param clientIpv4 string
+param databaseName string = 'MARS'
+param skuName string = 'GP_S_Gen5_1'
+param maxSizeBytes int = 34359738368
+
+var serverPrefix = 'artsyncdev'
+
+resource sqlServer 'Microsoft.Sql/servers@2023-08-01-preview' = {
+  name: '${serverPrefix}${uniqueString(resourceGroup().id)}'
+  location: location
+  properties: {
+    administratorLogin: administratorLogin
+    administratorLoginPassword: administratorLoginPassword
+    version: '12.0'
+    publicNetworkAccess: 'Enabled'
+    minimalTlsVersion: '1.2'
+  }
+}
+
+resource allowAzure 'Microsoft.Sql/servers/firewallRules@2023-08-01-preview' = {
+  parent: sqlServer
+  name: 'AllowAzureServices'
+  properties: {
+    startIpAddress: '0.0.0.0'
+    endIpAddress: '0.0.0.0'
+  }
+}
+
+resource allowClient 'Microsoft.Sql/servers/firewallRules@2023-08-01-preview' = {
+  parent: sqlServer
+  name: 'AllowLabClient'
+  properties: {
+    startIpAddress: clientIpv4
+    endIpAddress: clientIpv4
+  }
+}
+
+resource marsDb 'Microsoft.Sql/servers/databases@2023-08-01-preview' = {
+  parent: sqlServer
+  name: databaseName
+  location: location
+  sku: {
+    name: skuName
+    tier: 'GeneralPurpose'
+    family: 'Gen5'
+    capacity: 1
+  }
+  properties: {
+    collation: 'SQL_Latin1_General_CP1_CI_AS'
+    maxSizeBytes: maxSizeBytes
+    autoPauseDelay: 60
+    minCapacity: json('0.5')
+    requestedBackupStorageRedundancy: 'Local'
+  }
+}
+
+output sqlServerName string = sqlServer.name
+output sqlServerFqdn string = sqlServer.properties.fullyQualifiedDomainName
+output databaseName string = marsDb.name
+output adoNet string = 'Server=tcp:${sqlServer.properties.fullyQualifiedDomainName},1433;Database=${marsDb.name};User ID=${administratorLogin};Encrypt=True;TrustServerCertificate=False;Connection Timeout=60'

scripts/lab-azure.sh

@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+# Provision rg-artsync + Azure SQL (serverless) and write repo-root .env
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+ENV_FILE="$REPO_ROOT/.env"
+
+if [[ -f "$ENV_FILE" ]]; then
+    set +H
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+RG="${ARTSYNC_RESOURCE_GROUP:-rg-artsync}"
+LOC="${ARTSYNC_LOCATION:-westus2}"
+ADMIN="${ARTSYNC_SQL_ADMIN:-artsyncadmin}"
+DB_NAME="${ARTSYNC_AZURE_DB:-MARS}"
+
+sql_location_available() {
+    local loc="$1"
+    az rest --method get \
+        --url "https://management.azure.com/subscriptions/$(az account show --query id -o tsv)/providers/Microsoft.Sql/locations/${loc}/capabilities?api-version=2023-08-01-preview" \
+        --query status -o tsv 2>/dev/null | grep -qx Available
+}
+
+if ! sql_location_available "$LOC"; then
+    echo "==> SQL provisioning not available in $LOC; trying westus2 / centralus / southcentralus…"
+    LOC=""
+    for cand in westus2 centralus southcentralus westus westus3 westeurope uksouth; do
+        if sql_location_available "$cand"; then
+            LOC="$cand"
+            break
+        fi
+    done
+    if [[ -z "$LOC" ]]; then
+        echo "ERROR: no Azure SQL region available on this subscription." >&2
+        exit 1
+    fi
+fi
+
+PASS="${ARTSYNC_SQL_PASSWORD:-}"
+if [[ -z "$PASS" ]]; then
+    PASS="ArtSync_Lab_$(openssl rand -hex 10)!"
+fi
+
+IP="$(curl -sS --max-time 15 https://api.ipify.org)"
+if [[ ! "$IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+    echo "ERROR: could not detect public IPv4 (got: $IP)" >&2
+    exit 1
+fi
+
+echo "==> Subscription: $(az account show --query name -o tsv)"
+echo "==> Resource group: $RG"
+echo "==> SQL location: $LOC"
+echo "==> Client IP: $IP"
+
+if [[ "$(az group exists --name "$RG")" == "true" ]]; then
+    echo "==> Resource group $RG already exists ($(az group show -n "$RG" --query location -o tsv))"
+else
+    az group create --name "$RG" --location "$LOC" --tags project=artsync purpose=lab >/dev/null
+fi
+
+echo "==> Deploying Azure SQL (serverless GP_S_Gen5_1)…"
+OUT_JSON="$(az deployment group create \
+    --resource-group "$RG" \
+    --template-file "$REPO_ROOT/infra/main.bicep" \
+    --parameters \
+        location="$LOC" \
+        administratorLogin="$ADMIN" \
+        administratorLoginPassword="$PASS" \
+        clientIpv4="$IP" \
+        databaseName="$DB_NAME" \
+    --query properties.outputs \
+    --output json)"
+
+FQDN="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["sqlServerFqdn"]["value"])' <<<"$OUT_JSON")"
+SERVER="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["sqlServerName"]["value"])' <<<"$OUT_JSON")"
+
+SRC_CS="Server=$FQDN,1433;Database=$DB_NAME;User ID=$ADMIN;Password=$PASS;Encrypt=True;TrustServerCertificate=False;Connection Timeout=60"
+TGT_CS="Server=localhost,1433;Database=MARS;User ID=sa;Password=ArtSync_Test@2026;TrustServerCertificate=True;Connection Timeout=30"
+
+export RG LOC ADMIN PASS FQDN DB_NAME SRC_CS TGT_CS
+umask 077
+python3 - "$ENV_FILE" <<'PY'
+import json, os, sys
+path = sys.argv[1]
+pairs = [
+    ("ARTSYNC_RESOURCE_GROUP", os.environ["RG"]),
+    ("ARTSYNC_LOCATION", os.environ["LOC"]),
+    ("ARTSYNC_SQL_ADMIN", os.environ["ADMIN"]),
+    ("ARTSYNC_SQL_PASSWORD", os.environ["PASS"]),
+    ("ARTSYNC_AZURE_SERVER", os.environ["FQDN"]),
+    ("ARTSYNC_AZURE_DB", os.environ["DB_NAME"]),
+    ("ARTSYNC_LOCAL_SERVER", "localhost,1433"),
+    ("ARTSYNC_LOCAL_DB", "MARS"),
+    ("ARTSYNC_LOCAL_USER", "sa"),
+    ("ARTSYNC_LOCAL_PASSWORD", "ArtSync_Test@2026"),
+    ("ARTSYNC_SRC_CS", os.environ["SRC_CS"]),
+    ("ARTSYNC_TGT_CS", os.environ["TGT_CS"]),
+]
+with open(path, "w") as f:
+    f.write("# Generated by scripts/lab-azure.sh — do not commit\n")
+    for k, v in pairs:
+        f.write(f"{k}={json.dumps(v)}\n")
+os.chmod(path, 0o600)
+PY
+
+echo "==> Wrote $ENV_FILE"
+echo "==> Azure SQL: $SERVER / $FQDN / $DB_NAME"
+echo "==> Local target: localhost,1433 / MARS (after ./scripts/lab-up.sh)"
+echo
+echo "Next: ./scripts/lab-up.sh   # restore the .bak locally"
+echo "Then: source .env and run ArtSync Azure → local (compare-only first)."