From a2d7c1950d3129c68f3744372014a4963d82408a Mon Sep 17 00:00:00 2001 From: Ricardo Vega Date: Mon, 24 Aug 2026 18:28:00 -0500 Subject: [PATCH] Honor .dcomp mapping as an allowlist and round-trip sql_variant so /sync can leave mapped tables identical. Also add the disposable MARS lab (Azure SQL + Docker restore) and Azure-friendly schema compare options used to prove the loop. Co-authored-by: Cursor --- .gitignore | 11 + README.md | 20 +- docker-compose.lab.yml | 11 + infra/README.md | 49 +++ infra/lab-tables-only.scflt | 37 ++ infra/lab.env.example | 22 ++ infra/main.bicep | 65 ++++ scripts/lab-azure.sh | 115 ++++++ scripts/lab-down.sh | 28 ++ scripts/lab-smoke.sh | 37 ++ scripts/lab-up.sh | 54 +++ .../DevartConnectionString.cs | 71 ++++ src/ArtSync.Cli/CliApp.cs | 10 + src/ArtSync.Cli/HelpText.cs | 2 +- src/ArtSync.Compat/ArgvParser.cs | 18 +- src/ArtSync.Compat/CompFileLoader.cs | 330 ++++++++++++++++++ src/ArtSync.Compat/KnownOptions.cs | 18 + src/ArtSync.Data/CanonicalPayload.cs | 13 +- src/ArtSync.Data/DataOperationHandler.cs | 17 +- src/ArtSync.Data/SqlDataApplier.cs | 2 +- src/ArtSync.Data/SqlDataCompare.cs | 2 +- src/ArtSync.Data/SqlDataScripter.cs | 20 +- src/ArtSync.Data/SqlValueFormatter.cs | 57 +++ src/ArtSync.Schema/ArtSync.Schema.csproj | 1 + src/ArtSync.Schema/DacFxSchemaCompare.cs | 68 +++- src/ArtSync.Schema/SchemaOperationHandler.cs | 7 - .../ArtSync.Cli.Tests.csproj | 4 + tests/ArtSync.Cli.Tests/CliAppTests.cs | 48 +++ .../ArgvParserGoldenTests.cs | 105 ++++++ .../ArtSync.Compat.Tests.csproj | 4 + .../CompFileParserTests.cs | 199 +++++++++++ .../DevartConnectionStringTests.cs | 46 +++ .../CanonicalPayloadTests.cs | 22 +- .../DataHandlerUnitTests.cs | 6 +- .../DevartConnectionStringBuildTests.cs | 24 ++ tests/ArtSync.Data.Tests/SqlFkGraphTests.cs | 21 ++ .../SqlValueFormatterTests.cs | 18 + .../SchemaHandlerUnitTests.cs | 4 +- tests/fixtures/README.md | 30 +- tests/fixtures/sample.dcomp | 64 ++++ 40 files changed, 1622 insertions(+), 58 deletions(-) create mode 100644 docker-compose.lab.yml create mode 100644 infra/README.md create mode 100644 infra/lab-tables-only.scflt create mode 100644 infra/lab.env.example create mode 100644 infra/main.bicep create mode 100755 scripts/lab-azure.sh create mode 100755 scripts/lab-down.sh create mode 100755 scripts/lab-smoke.sh create mode 100755 scripts/lab-up.sh create mode 100644 src/ArtSync.Abstractions/DevartConnectionString.cs create mode 100644 src/ArtSync.Compat/CompFileLoader.cs create mode 100644 tests/ArtSync.Compat.Tests/CompFileParserTests.cs create mode 100644 tests/ArtSync.Compat.Tests/DevartConnectionStringTests.cs create mode 100644 tests/ArtSync.Data.Tests/DevartConnectionStringBuildTests.cs create mode 100644 tests/fixtures/sample.dcomp diff --git a/.gitignore b/.gitignore index a6e3376..2f46793 100644 --- a/.gitignore +++ b/.gitignore @@ -59,3 +59,14 @@ logs/ # Argfiles that may contain passwords (keep sample/ fixtures, not real ones) *.argfile + +# Local captured jobs, backups, and Devart project files — never commit +_data/ +MARS_PROD_AZURE_TO_ONPREM_DATA_SYNC.bat +*_PROD_*.bat +*.dcomp +*.scomp +*.bak +!tests/fixtures/*.dcomp +!tests/fixtures/*.scomp +UPDATE_AsOfDate.sql diff --git a/README.md b/README.md index 1d4347c..b055bbb 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,7 @@ ArtSync replaces the executables. `.bat` files, Task Scheduler entries, and argf | `schemacompare … /sync` | Yes | 0 applied, 112 nothing to sync | | `schemacompare … /sync:file.sql` | Yes | 101; target untouched | | `datacompare /source … /target … /sync` | Yes | Same 100 / 101 / 0 / 112 / 108 / 40 pattern | +| `/compfile:.dcomp` + `/sync` | Yes (data) | CLI `/source` `/target` override project connections | | `/argfile` + CLI override | Yes | CLI wins, matching Devart precedence | | `/filter:.scflt` | Yes | 114 on bad XML | | `/report` + `/reportformat HTML\|XML\|CSV` + `/log` | Yes | 107 report I/O, 106 log I/O; passwords redacted | @@ -62,7 +63,7 @@ These are not bugs. They exit 10 or leave the target alone **by design**. | Item | Behavior | Why | |---|---|---| -| `/compfile` (`.scomp` / `.dcomp`) | Exit 10 | Do not invent project XML until a captured file is in `tests/fixtures/` | +| `/compfile` (`.scomp`) | Exit 10 | Schema project files are not loaded in v1 | | `backup:` / `snapshot:` / `scriptsfolder:` endpoints | Exit 10 | Live databases only in v1 | | `/reportformat:XLS` | Exit 10 | Excel reports are out of scope | | Extra tables only on the target | Left in place | DacFx `DropObjectsNotInSource` stays false — dropping extras by default is unsafe | @@ -110,6 +111,18 @@ Integration tests skip unless `ARTSYNC_INTEGRATION=true`. That starts SQL Server 2022, seeds `artsync_src` / `artsync_tgt`, and runs the schema + data suite (FK graphs, data types, reports, apply). See [tests/fixtures/README.md](tests/fixtures/README.md). +### MARS lab (Azure SQL + local restore) + +Disposable stand-ins for the captured Azure → on-prem job. Not production. See [infra/README.md](infra/README.md). + +```bash +./scripts/lab-azure.sh # Azure SQL MARS (westus2, serverless) +./scripts/lab-up.sh # Docker SQL + restore _data/*.bak as [MARS] +./scripts/lab-smoke.sh # both sides reachable +``` + +`.env` is gitignored. Tear down with `./scripts/lab-down.sh`. + ### Publish the three executable names **Windows (PowerShell):** @@ -134,7 +147,7 @@ CI publishes `linux-x64`, `win-x64`, `osx-x64`, and `osx-arm64` on every green m ### Drop-in on Windows scheduler -Replace the Devart executable on `PATH`, or update the scheduled task program from `….com` to `….exe` of the same basename. Confirm the job uses live `/source` and `/target`, not `/compfile` or `backup:` endpoints. +Replace the Devart executable on `PATH`, or update the scheduled task program from `….com` to `….exe` of the same basename. Live `/source` and `/target` still override `/compfile` connections. `.scomp` and `backup:` endpoints are not supported. Recommended first production cutover: @@ -223,7 +236,8 @@ All engine code sits behind thin interfaces. `ArtSync.Cli` depends only on `IArg | `/sync` / `/sync:` | Complete | | `/report` + `/reportformat:HTML\|XML\|CSV` | Complete (XLS → exit 10) | | `/log` | Complete (passwords redacted) | -| `/compfile` (`.scomp` / `.dcomp`) | Exit 10 until a real file is in `tests/fixtures/` | +| `/compfile` (`.dcomp`) | Complete — connections, compare/sync options, exclude masks, `Included="False"` objects. CLI wins. | +| `/compfile` (`.scomp`) | Exit 10 — schema projects not loaded in v1 | | `/source backup:…` | Exit 10 — out of scope in v1 | | Extra objects on the target (schema drop) | Not dropped (`DropObjectsNotInSource` stays false) | | Azure SQL Database | Supported | diff --git a/docker-compose.lab.yml b/docker-compose.lab.yml new file mode 100644 index 0000000..cd306d6 --- /dev/null +++ b/docker-compose.lab.yml @@ -0,0 +1,11 @@ +# Lab overlay: mount captured backups and give SQL Server more RAM. +# Usage: docker compose -f docker-compose.yml -f docker-compose.lab.yml up -d +# +# CI keeps using docker-compose.yml only so GitHub Actions does not create an empty _data/. + +services: + sqlserver: + environment: + MSSQL_MEMORY_LIMIT_MB: "4096" + volumes: + - ./_data:/bak:ro diff --git a/infra/README.md b/infra/README.md new file mode 100644 index 0000000..7c9300f --- /dev/null +++ b/infra/README.md @@ -0,0 +1,49 @@ +# ArtSync lab + +Disposable **source** (Azure SQL) and **target** (Docker SQL Server) for the captured MARS job. Not production. + +| Side | What | How | +|---|---|---| +| Target (on-prem stand-in) | SQL Server 2022 in Docker, database `MARS` | `./scripts/lab-up.sh` restores `_data/*.bak` | +| Source (Azure stand-in) | Azure SQL Database `MARS`, serverless | `./scripts/lab-azure.sh` | + +`.env` at the repo root is gitignored. `_data/` stays gitignored. + +## One-time + +```bash +# 1. Azure SQL server + database + firewall for this IP +./scripts/lab-azure.sh + +# 2. Local SQL + restore the captured backup (several minutes) +./scripts/lab-up.sh +``` + +Resource group: `rg-artsync`. Azure SQL defaults to **westus2** (this Microsoft Partner Network subscription blocks SQL in eastus / eastus2). SKU: serverless `GP_S_Gen5_1` (auto-pause after 60 minutes). + +## Compare without applying + +```bash +set +H +set -a && source .env && set +a +dotnet run --project src/ArtSync.Cli -- \ + /datacompare \ + /compfile:"$PWD/_data/TheSync!.dcomp" \ + /source connection:"$ARTSYNC_SRC_CS" \ + /target connection:"$ARTSYNC_TGT_CS" +``` + +Empty Azure vs restored local will show many source-only / different rows until you seed Azure (schema then data `/sync` local → Azure, or import a bacpac). Do not `/sync` Azure → local until you mean to overwrite the restored copy. + +## Tear down Azure + +```bash +./scripts/lab-down.sh +# also drop local SQL volume: ARTSYNC_LAB_DOWN_DOCKER=1 ./scripts/lab-down.sh +``` + +## Smoke test + +```bash +./scripts/lab-smoke.sh +``` diff --git a/infra/lab-tables-only.scflt b/infra/lab-tables-only.scflt new file mode 100644 index 0000000..8b50492 --- /dev/null +++ b/infra/lab-tables-only.scflt @@ -0,0 +1,37 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/infra/lab.env.example b/infra/lab.env.example new file mode 100644 index 0000000..06afceb --- /dev/null +++ b/infra/lab.env.example @@ -0,0 +1,22 @@ +# ArtSync lab (do not commit the real .env) +# +# Copy to ../.env and fill the Azure password after ./scripts/lab-azure.sh + +ARTSYNC_RESOURCE_GROUP=rg-artsync +ARTSYNC_LOCATION=westus2 +ARTSYNC_SQL_ADMIN=artsyncadmin +ARTSYNC_SQL_PASSWORD= + +# Filled by lab-azure.sh +ARTSYNC_AZURE_SERVER= +ARTSYNC_AZURE_DB=MARS + +# Local Docker SQL (on-prem stand-in) +ARTSYNC_LOCAL_SERVER=localhost,1433 +ARTSYNC_LOCAL_DB=MARS +ARTSYNC_LOCAL_USER=sa +ARTSYNC_LOCAL_PASSWORD=ArtSync_Test@2026 + +# Filled by lab-azure.sh (quoted; values contain spaces/semicolons) +# ARTSYNC_SRC_CS="Server=….database.windows.net,1433;Database=MARS;User ID=artsyncadmin;Password=…;Encrypt=True;TrustServerCertificate=False;Connection Timeout=60" +# ARTSYNC_TGT_CS="Server=localhost,1433;Database=MARS;User ID=sa;Password=ArtSync_Test@2026;TrustServerCertificate=True;Connection Timeout=30" diff --git a/infra/main.bicep b/infra/main.bicep new file mode 100644 index 0000000..920edea --- /dev/null +++ b/infra/main.bicep @@ -0,0 +1,65 @@ +@description('ArtSync lab: Azure SQL as the cloud source stand-in. Not production.') +param location string = resourceGroup().location +param administratorLogin string +@secure() +param administratorLoginPassword string +param clientIpv4 string +param databaseName string = 'MARS' +param skuName string = 'GP_S_Gen5_1' +param maxSizeBytes int = 34359738368 + +var serverPrefix = 'artsyncdev' + +resource sqlServer 'Microsoft.Sql/servers@2023-08-01-preview' = { + name: '${serverPrefix}${uniqueString(resourceGroup().id)}' + location: location + properties: { + administratorLogin: administratorLogin + administratorLoginPassword: administratorLoginPassword + version: '12.0' + publicNetworkAccess: 'Enabled' + minimalTlsVersion: '1.2' + } +} + +resource allowAzure 'Microsoft.Sql/servers/firewallRules@2023-08-01-preview' = { + parent: sqlServer + name: 'AllowAzureServices' + properties: { + startIpAddress: '0.0.0.0' + endIpAddress: '0.0.0.0' + } +} + +resource allowClient 'Microsoft.Sql/servers/firewallRules@2023-08-01-preview' = { + parent: sqlServer + name: 'AllowLabClient' + properties: { + startIpAddress: clientIpv4 + endIpAddress: clientIpv4 + } +} + +resource marsDb 'Microsoft.Sql/servers/databases@2023-08-01-preview' = { + parent: sqlServer + name: databaseName + location: location + sku: { + name: skuName + tier: 'GeneralPurpose' + family: 'Gen5' + capacity: 1 + } + properties: { + collation: 'SQL_Latin1_General_CP1_CI_AS' + maxSizeBytes: maxSizeBytes + autoPauseDelay: 60 + minCapacity: json('0.5') + requestedBackupStorageRedundancy: 'Local' + } +} + +output sqlServerName string = sqlServer.name +output sqlServerFqdn string = sqlServer.properties.fullyQualifiedDomainName +output databaseName string = marsDb.name +output adoNet string = 'Server=tcp:${sqlServer.properties.fullyQualifiedDomainName},1433;Database=${marsDb.name};User ID=${administratorLogin};Encrypt=True;TrustServerCertificate=False;Connection Timeout=60' diff --git a/scripts/lab-azure.sh b/scripts/lab-azure.sh new file mode 100755 index 0000000..4021302 --- /dev/null +++ b/scripts/lab-azure.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# Provision rg-artsync + Azure SQL (serverless) and write repo-root .env +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +ENV_FILE="$REPO_ROOT/.env" + +if [[ -f "$ENV_FILE" ]]; then + set +H + set -a + # shellcheck disable=SC1090 + source "$ENV_FILE" + set +a +fi + +RG="${ARTSYNC_RESOURCE_GROUP:-rg-artsync}" +LOC="${ARTSYNC_LOCATION:-westus2}" +ADMIN="${ARTSYNC_SQL_ADMIN:-artsyncadmin}" +DB_NAME="${ARTSYNC_AZURE_DB:-MARS}" + +sql_location_available() { + local loc="$1" + az rest --method get \ + --url "https://management.azure.com/subscriptions/$(az account show --query id -o tsv)/providers/Microsoft.Sql/locations/${loc}/capabilities?api-version=2023-08-01-preview" \ + --query status -o tsv 2>/dev/null | grep -qx Available +} + +if ! sql_location_available "$LOC"; then + echo "==> SQL provisioning not available in $LOC; trying westus2 / centralus / southcentralus…" + LOC="" + for cand in westus2 centralus southcentralus westus westus3 westeurope uksouth; do + if sql_location_available "$cand"; then + LOC="$cand" + break + fi + done + if [[ -z "$LOC" ]]; then + echo "ERROR: no Azure SQL region available on this subscription." >&2 + exit 1 + fi +fi + +PASS="${ARTSYNC_SQL_PASSWORD:-}" +if [[ -z "$PASS" ]]; then + PASS="ArtSync_Lab_$(openssl rand -hex 10)!" +fi + +IP="$(curl -sS --max-time 15 https://api.ipify.org)" +if [[ ! "$IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "ERROR: could not detect public IPv4 (got: $IP)" >&2 + exit 1 +fi + +echo "==> Subscription: $(az account show --query name -o tsv)" +echo "==> Resource group: $RG" +echo "==> SQL location: $LOC" +echo "==> Client IP: $IP" + +if [[ "$(az group exists --name "$RG")" == "true" ]]; then + echo "==> Resource group $RG already exists ($(az group show -n "$RG" --query location -o tsv))" +else + az group create --name "$RG" --location "$LOC" --tags project=artsync purpose=lab >/dev/null +fi + +echo "==> Deploying Azure SQL (serverless GP_S_Gen5_1)…" +OUT_JSON="$(az deployment group create \ + --resource-group "$RG" \ + --template-file "$REPO_ROOT/infra/main.bicep" \ + --parameters \ + location="$LOC" \ + administratorLogin="$ADMIN" \ + administratorLoginPassword="$PASS" \ + clientIpv4="$IP" \ + databaseName="$DB_NAME" \ + --query properties.outputs \ + --output json)" + +FQDN="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["sqlServerFqdn"]["value"])' <<<"$OUT_JSON")" +SERVER="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["sqlServerName"]["value"])' <<<"$OUT_JSON")" + +SRC_CS="Server=$FQDN,1433;Database=$DB_NAME;User ID=$ADMIN;Password=$PASS;Encrypt=True;TrustServerCertificate=False;Connection Timeout=60" +TGT_CS="Server=localhost,1433;Database=MARS;User ID=sa;Password=ArtSync_Test@2026;TrustServerCertificate=True;Connection Timeout=30" + +export RG LOC ADMIN PASS FQDN DB_NAME SRC_CS TGT_CS +umask 077 +python3 - "$ENV_FILE" <<'PY' +import json, os, sys +path = sys.argv[1] +pairs = [ + ("ARTSYNC_RESOURCE_GROUP", os.environ["RG"]), + ("ARTSYNC_LOCATION", os.environ["LOC"]), + ("ARTSYNC_SQL_ADMIN", os.environ["ADMIN"]), + ("ARTSYNC_SQL_PASSWORD", os.environ["PASS"]), + ("ARTSYNC_AZURE_SERVER", os.environ["FQDN"]), + ("ARTSYNC_AZURE_DB", os.environ["DB_NAME"]), + ("ARTSYNC_LOCAL_SERVER", "localhost,1433"), + ("ARTSYNC_LOCAL_DB", "MARS"), + ("ARTSYNC_LOCAL_USER", "sa"), + ("ARTSYNC_LOCAL_PASSWORD", "ArtSync_Test@2026"), + ("ARTSYNC_SRC_CS", os.environ["SRC_CS"]), + ("ARTSYNC_TGT_CS", os.environ["TGT_CS"]), +] +with open(path, "w") as f: + f.write("# Generated by scripts/lab-azure.sh — do not commit\n") + for k, v in pairs: + f.write(f"{k}={json.dumps(v)}\n") +os.chmod(path, 0o600) +PY + +echo "==> Wrote $ENV_FILE" +echo "==> Azure SQL: $SERVER / $FQDN / $DB_NAME" +echo "==> Local target: localhost,1433 / MARS (after ./scripts/lab-up.sh)" +echo +echo "Next: ./scripts/lab-up.sh # restore the .bak locally" +echo "Then: source .env and run ArtSync Azure → local (compare-only first)." diff --git a/scripts/lab-down.sh b/scripts/lab-down.sh new file mode 100755 index 0000000..209dabb --- /dev/null +++ b/scripts/lab-down.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Tear down the disposable lab (Azure RG + optional local Docker SQL). +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +ENV_FILE="$REPO_ROOT/.env" +RG="${ARTSYNC_RESOURCE_GROUP:-rg-artsync}" +DROP_DOCKER="${ARTSYNC_LAB_DOWN_DOCKER:-0}" + +if [[ -f "$ENV_FILE" ]]; then + set +H + set -a + # shellcheck disable=SC1090 + source "$ENV_FILE" + set +a + RG="${ARTSYNC_RESOURCE_GROUP:-$RG}" +fi + +echo "==> Deleting Azure resource group: $RG" +az group delete --name "$RG" --yes --no-wait +echo " deletion started (async)." + +if [[ "$DROP_DOCKER" == "1" ]]; then + echo "==> Stopping local SQL (docker compose down -v)…" + docker compose -f "$REPO_ROOT/docker-compose.yml" -f "$REPO_ROOT/docker-compose.lab.yml" down -v +fi + +echo "Done. Local [MARS] is kept unless ARTSYNC_LAB_DOWN_DOCKER=1." diff --git a/scripts/lab-smoke.sh b/scripts/lab-smoke.sh new file mode 100755 index 0000000..0ebca46 --- /dev/null +++ b/scripts/lab-smoke.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Verify local MARS and Azure SQL lab connections (no sync). +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +ENV_FILE="$REPO_ROOT/.env" + +if [[ ! -f "$ENV_FILE" ]]; then + echo "ERROR: $ENV_FILE missing. Run ./scripts/lab-azure.sh first." >&2 + exit 1 +fi +set +H +set -a +# shellcheck disable=SC1090 +source "$ENV_FILE" +set +a + +SA_PASS="${ARTSYNC_LOCAL_PASSWORD:-ArtSync_Test@2026}" +COMPOSE=(docker compose -f "$REPO_ROOT/docker-compose.yml" -f "$REPO_ROOT/docker-compose.lab.yml") + +echo "==> Local [MARS]" +"${COMPOSE[@]}" exec -T sqlserver \ + /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P "$SA_PASS" -C -d MARS -h -1 -W \ + -Q "SET NOCOUNT ON; SELECT CONCAT(DB_NAME(), ' tables=', CONVERT(varchar(12), (SELECT COUNT(*) FROM sys.tables WHERE is_ms_shipped = 0)));" + +echo "==> Azure [$ARTSYNC_AZURE_DB] @ $ARTSYNC_AZURE_SERVER" +"${COMPOSE[@]}" exec -T sqlserver \ + /opt/mssql-tools18/bin/sqlcmd \ + -S "$ARTSYNC_AZURE_SERVER" \ + -U "$ARTSYNC_SQL_ADMIN" \ + -P "$ARTSYNC_SQL_PASSWORD" \ + -d "$ARTSYNC_AZURE_DB" \ + -C \ + -h -1 -W \ + -Q "SET NOCOUNT ON; SELECT CONCAT(DB_NAME(), ' tables=', CONVERT(varchar(12), (SELECT COUNT(*) FROM sys.tables WHERE is_ms_shipped = 0)));" + +echo "==> Both reachable." diff --git a/scripts/lab-up.sh b/scripts/lab-up.sh new file mode 100755 index 0000000..c0bb233 --- /dev/null +++ b/scripts/lab-up.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# Start local SQL Server (lab overlay) and restore _data/*.bak as database MARS. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +SA_PASS="${ARTSYNC_LOCAL_PASSWORD:-ArtSync_Test@2026}" +DB_NAME="${ARTSYNC_LOCAL_DB:-MARS}" +COMPOSE=(docker compose -f "$REPO_ROOT/docker-compose.yml" -f "$REPO_ROOT/docker-compose.lab.yml") + +sqlcmd() { + "${COMPOSE[@]}" exec -T sqlserver \ + /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P "$SA_PASS" -C -b "$@" +} + +wait_for_sql() { + echo "==> Waiting for SQL Server…" + local i + for i in $(seq 1 30); do + if sqlcmd -Q "SELECT 1" >/dev/null 2>&1; then + echo " ready." + return 0 + fi + sleep 5 + done + echo "ERROR: SQL Server did not start." >&2 + "${COMPOSE[@]}" logs --tail 80 sqlserver || true + exit 1 +} + +echo "==> Starting Docker SQL Server (lab)…" +"${COMPOSE[@]}" up -d --wait --wait-timeout 180 +wait_for_sql + +BAK_HOST="$(ls -1t "$REPO_ROOT"/_data/*.bak 2>/dev/null | head -1 || true)" +if [[ -z "$BAK_HOST" ]]; then + echo "ERROR: no .bak under _data/. Put the MARS backup there first." >&2 + exit 1 +fi +BAK_NAME="$(basename "$BAK_HOST")" +BAK_CT="/bak/$BAK_NAME" +echo "==> Backup: $BAK_NAME" + +if sqlcmd -Q "IF DB_ID('$DB_NAME') IS NOT NULL SELECT 1 ELSE SELECT 0" -h -1 -W | grep -qx 1; then + echo "==> Database [$DB_NAME] already exists. Skipping restore (drop it to re-restore)." + sqlcmd -Q "SELECT name, state_desc, compatibility_level FROM sys.databases WHERE name = '$DB_NAME'" + exit 0 +fi + +echo "==> Restoring [$DB_NAME] (this can take several minutes)…" +# Captured MARS.bak (SQL 2014 SIMGP) uses these logical names; FILELISTONLY if they change. +sqlcmd -Q "RESTORE DATABASE [$DB_NAME] FROM DISK = N'$BAK_CT' WITH REPLACE, STATS = 10, MOVE N'MARS_Data' TO N'/var/opt/mssql/data/${DB_NAME}.mdf', MOVE N'MARS_Log' TO N'/var/opt/mssql/data/${DB_NAME}_log.ldf'" + +echo "==> Restored [$DB_NAME]." +sqlcmd -d "$DB_NAME" -Q "SELECT DB_NAME() AS db, DATABASEPROPERTYEX(DB_NAME(), 'Collation') AS collation, (SELECT COUNT(*) FROM sys.tables WHERE is_ms_shipped = 0) AS user_tables" diff --git a/src/ArtSync.Abstractions/DevartConnectionString.cs b/src/ArtSync.Abstractions/DevartConnectionString.cs new file mode 100644 index 0000000..1dac1bb --- /dev/null +++ b/src/ArtSync.Abstractions/DevartConnectionString.cs @@ -0,0 +1,71 @@ +namespace ArtSync.Abstractions; + +/// +/// Devart connection:"…" strings include keywords Microsoft.Data.SqlClient +/// rejects (notably Transaction Scope Local). Azure SQL also refuses +/// Encrypt=False, which Devart still emits. +/// +public static class DevartConnectionString +{ + private static readonly HashSet SqlClientUnsupported = new(StringComparer.OrdinalIgnoreCase) + { + "Transaction Scope Local", + }; + + public static string ForSqlClient(string raw) + { + if (string.IsNullOrWhiteSpace(raw)) + throw new ArgumentException("Connection string is empty."); + + var pairs = new List<(string Key, string Value)>(); + string? dataSource = null; + + foreach (var (key, value) in SplitPairs(Unquote(raw.Trim()))) + { + if (SqlClientUnsupported.Contains(key)) + continue; + + if (IsDataSourceKey(key)) + dataSource = value; + + pairs.Add((key, value)); + } + + if (IsAzureSql(dataSource)) + { + pairs.RemoveAll(p => p.Key.Equals("Encrypt", StringComparison.OrdinalIgnoreCase)); + pairs.Add(("Encrypt", "True")); + } + + return string.Join(";", pairs.Select(p => $"{p.Key}={p.Value}")); + } + + internal static IEnumerable<(string Key, string Value)> SplitPairs(string connectionString) + { + foreach (var segment in connectionString.Split(';', StringSplitOptions.RemoveEmptyEntries)) + { + var eq = segment.IndexOf('='); + if (eq <= 0) + continue; + yield return (segment[..eq].Trim(), segment[(eq + 1)..].Trim()); + } + } + + internal static string Unquote(string value) + { + if (value.Length >= 2 && value[0] == '"' && value[^1] == '"') + return value[1..^1]; + return value; + } + + private static bool IsDataSourceKey(string key) + => key.Equals("Data Source", StringComparison.OrdinalIgnoreCase) + || key.Equals("Server", StringComparison.OrdinalIgnoreCase) + || key.Equals("Addr", StringComparison.OrdinalIgnoreCase) + || key.Equals("Address", StringComparison.OrdinalIgnoreCase) + || key.Equals("Network Address", StringComparison.OrdinalIgnoreCase); + + private static bool IsAzureSql(string? dataSource) + => dataSource is not null + && dataSource.Contains(".database.windows.net", StringComparison.OrdinalIgnoreCase); +} diff --git a/src/ArtSync.Cli/CliApp.cs b/src/ArtSync.Cli/CliApp.cs index 23e5300..9d801f4 100644 --- a/src/ArtSync.Cli/CliApp.cs +++ b/src/ArtSync.Cli/CliApp.cs @@ -70,6 +70,16 @@ public int Run(IReadOnlyList argv, string argv0) var req = ((ParseResult.Success)result).Request; + try + { + req = CompFileMerger.Apply(req); + } + catch (CompFileException ex) + { + _err.WriteLine($"Error: {ex.Message}"); + return ex.ExitCode; + } + if (!req.Quiet) { foreach (var w in req.Warnings) diff --git a/src/ArtSync.Cli/HelpText.cs b/src/ArtSync.Cli/HelpText.cs index d76a02c..250c587 100644 --- a/src/ArtSync.Cli/HelpText.cs +++ b/src/ArtSync.Cli/HelpText.cs @@ -22,7 +22,7 @@ datacompare [/datacompare] [options] /reportformat:HTML|XML|CSV /log: Write execution log /argfile: Load switches from file (CLI switches win) - /compfile: .scomp/.dcomp project file + /compfile: .dcomp data project (CLI /source /target win). .scomp not yet. /filter: .scflt object filter (schema only) /q Quiet mode diff --git a/src/ArtSync.Compat/ArgvParser.cs b/src/ArtSync.Compat/ArgvParser.cs index 599268a..9052d96 100644 --- a/src/ArtSync.Compat/ArgvParser.cs +++ b/src/ArtSync.Compat/ArgvParser.cs @@ -288,6 +288,12 @@ private static ParseResult ParseTokens( { if (KnownOptions.IsUnsupportedInV1(canonical)) { + if (KnownOptions.IsExplicitlyOff(inlineValue)) + { + i++; + return null; + } + return ParseResult.Fail(10, $"Option /{switchName} ({canonical}) is not implemented in ArtSync v1. " + $"Remove it from the command line."); @@ -346,11 +352,19 @@ private static bool TrySplitSwitch(string token, out string name, out string? va else { name = body[..colon]; - value = body[(colon + 1)..]; + value = Unquote(body[(colon + 1)..]); } return true; } + private static string Unquote(string value) + { + value = value.Trim(); + if (value.Length >= 2 && value[0] == '"' && value[^1] == '"') + return value[1..^1]; + return value; + } + private static bool IsSwitch(string token, string switchNameLower) { if (!token.StartsWith('/')) return false; @@ -417,7 +431,7 @@ private sealed class EndpointBuilder(bool isSource) "(valid keys: connection, server, database, user, password)."); var key = token[..colon].ToLowerInvariant(); - var value = token[(colon + 1)..]; + var value = Unquote(token[(colon + 1)..]); return key switch { diff --git a/src/ArtSync.Compat/CompFileLoader.cs b/src/ArtSync.Compat/CompFileLoader.cs new file mode 100644 index 0000000..7336197 --- /dev/null +++ b/src/ArtSync.Compat/CompFileLoader.cs @@ -0,0 +1,330 @@ +using ArtSync.Abstractions; + +namespace ArtSync.Compat; + +/// +/// Thrown while loading or merging a Devart /compfile. +/// is 10 (missing/unsupported) or 30 (corrupt). +/// +public sealed class CompFileException : Exception +{ + public int ExitCode { get; } + + public CompFileException(int exitCode, string message, Exception? inner = null) + : base(message, inner) + => ExitCode = exitCode; +} + +/// Parsed Devart data-compare project (.dcomp). +public sealed class CompFileDocument +{ + public Endpoint? Source { get; init; } + public Endpoint? Target { get; init; } + public IReadOnlyDictionary Options { get; init; } = + new Dictionary(StringComparer.OrdinalIgnoreCase); + public IReadOnlyList ExcludedObjectNames { get; init; } = []; + /// + /// Mapping objects with Included true (or omitted). When non-empty, + /// these are the Devart checkbox allowlist — compare only these tables. + /// + public IReadOnlyList IncludedObjectNames { get; init; } = []; +} + +/// +/// Loads Devart .dcomp XML captured from dbForge Data Compare +/// (provider 5.x ConnectionString element and 6.x ConnectionModel). +/// +public static class CompFileParser +{ + public static CompFileDocument Load(string path) + { + if (string.IsNullOrWhiteSpace(path)) + throw new CompFileException(10, "/compfile requires a path: /compfile:"); + + if (!File.Exists(path)) + throw new CompFileException(10, $"/compfile: file not found: {path}"); + + string xml; + try { xml = File.ReadAllText(path); } + catch (Exception ex) + { + throw new CompFileException(30, $"/compfile: cannot read '{path}': {ex.Message}", ex); + } + + try + { + return ParseXml(xml, path); + } + catch (CompFileException) + { + throw; + } + catch (Exception ex) + { + throw new CompFileException(30, + $"/compfile '{path}' is not a valid Devart data-compare project: {ex.Message}", ex); + } + } + + internal static CompFileDocument ParseXml(string xml, string pathForErrors) + { + var doc = System.Xml.Linq.XDocument.Parse(xml); + var root = doc.Root ?? throw new CompFileException(30, + $"/compfile '{pathForErrors}' has no root element."); + + if (!root.Name.LocalName.Equals("CompareDocument", StringComparison.OrdinalIgnoreCase)) + throw new CompFileException(30, + $"/compfile '{pathForErrors}' is not a Devart CompareDocument (root is <{root.Name.LocalName}>)."); + + var options = new Dictionary(StringComparer.OrdinalIgnoreCase); + var backupDatabase = false; + ReadOptionPairs(root.Element("Options"), options, ref backupDatabase); + ReadOptionPairs(root.Element("SynchronizationOptions"), options, ref backupDatabase); + + var (included, excluded) = ReadMappedObjects(root); + return new CompFileDocument + { + Source = ReadEndpoint(root.Element("Source")), + Target = ReadEndpoint(root.Element("Target")), + Options = options, + IncludedObjectNames = included, + ExcludedObjectNames = excluded, + }; + } + + private static Endpoint? ReadEndpoint(System.Xml.Linq.XElement? el) + { + if (el is null) return null; + var cs = ReadConnectionString(el); + if (string.IsNullOrWhiteSpace(cs)) return null; + return new Endpoint(EndpointKind.ConnectionString, ConnectionString: cs); + } + + internal static string? ReadConnectionString(System.Xml.Linq.XElement endpoint) + { + var direct = endpoint.Element("ConnectionString")?.Value; + if (!string.IsNullOrWhiteSpace(direct)) + return direct.Trim(); + + foreach (var p in endpoint.Descendants("PropertyValue")) + { + var name = (string?)p.Attribute("Name"); + if (name is not null && + name.Equals("ConnectionString", StringComparison.OrdinalIgnoreCase) && + !string.IsNullOrWhiteSpace(p.Value)) + return p.Value.Trim(); + } + + return null; + } + + private static (IReadOnlyList Included, IReadOnlyList Excluded) + ReadMappedObjects(System.Xml.Linq.XElement root) + { + var included = new List(); + var excluded = new List(); + var compared = root.Element("Mapping")?.Element("ComparedObjects"); + if (compared is null) return (included, excluded); + + foreach (var obj in compared.Elements("Object")) + { + var name = (string?)obj.Attribute("SourceName"); + if (string.IsNullOrWhiteSpace(name)) continue; + var flag = (string?)obj.Attribute("Included") ?? "True"; + if (IsXmlTrue(flag)) + included.Add(name.Trim()); + else + excluded.Add(name.Trim()); + } + + return (included, excluded); + } + + private static void ReadOptionPairs( + System.Xml.Linq.XElement? section, + Dictionary options, + ref bool backupDatabase) + { + if (section is null) return; + + var raw = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var p in section.Elements("PropertyValue")) + { + var name = (string?)p.Attribute("Name"); + if (string.IsNullOrWhiteSpace(name)) continue; + raw[name] = p.Value ?? ""; + } + + if (raw.TryGetValue("BackupDatabase", out var bd) && IsXmlTrue(bd)) + backupDatabase = true; + + ApplyMaskPair(raw, options, "ExcludeObjectsByMask", "ExcludeObjectsMask"); + ApplyMaskPair(raw, options, "IncludeObjectsByMask", "IncludeObjectsMask"); + ApplyMaskPair(raw, options, "IgnoreColumnsByMask", "MappingColumnsMask"); + + foreach (var (name, value) in raw) + { + if (IsPairOrChrome(name)) continue; + + if (!KnownOptions.TryGetCanonical(name, out var canonical)) + continue; + + if (KnownOptions.IsUnsupportedInV1(canonical)) + { + if (!IsXmlTrue(value)) continue; + if (IsBackupChrome(canonical) && !backupDatabase) continue; + throw new CompFileException(10, + $"Option {canonical} from /compfile is not implemented in ArtSync v1."); + } + + options[canonical] = ToCliBool(value); + } + } + + private static void ApplyMaskPair( + Dictionary raw, + Dictionary options, + string flagName, + string maskName) + { + if (!raw.TryGetValue(flagName, out var flag) || !IsXmlTrue(flag)) + return; + if (!raw.TryGetValue(maskName, out var mask) || string.IsNullOrWhiteSpace(mask)) + return; + if (!KnownOptions.TryGetCanonical(flagName, out var canonical)) + canonical = flagName; + options[canonical] = mask.Trim(); + } + + private static bool IsPairOrChrome(string name) + => name.Equals("ExcludeObjectsByMask", StringComparison.OrdinalIgnoreCase) + || name.Equals("ExcludeObjectsMask", StringComparison.OrdinalIgnoreCase) + || name.Equals("IncludeObjectsByMask", StringComparison.OrdinalIgnoreCase) + || name.Equals("IncludeObjectsMask", StringComparison.OrdinalIgnoreCase) + || name.Equals("IgnoreColumnsByMask", StringComparison.OrdinalIgnoreCase) + || name.Equals("MappingColumnsMask", StringComparison.OrdinalIgnoreCase) + || name.Equals("BackupDatabase", StringComparison.OrdinalIgnoreCase) + || name.Equals("BackupType", StringComparison.OrdinalIgnoreCase) + || name.Equals("BackupPathName", StringComparison.OrdinalIgnoreCase) + || name.Equals("BackupExtensionName", StringComparison.OrdinalIgnoreCase) + || name.Equals("CreateBackupFolder", StringComparison.OrdinalIgnoreCase) + || name.Equals("NeedCompressBackup", StringComparison.OrdinalIgnoreCase) + || name.Equals("AddBackupType", StringComparison.OrdinalIgnoreCase) + || name.Equals("BackupExtension", StringComparison.OrdinalIgnoreCase) + || name.Equals("BackupPath", StringComparison.OrdinalIgnoreCase); + + private static bool IsBackupChrome(string canonical) + => canonical.Equals("CreateBackupFolder", StringComparison.OrdinalIgnoreCase) + || canonical.Equals("NeedCompressBackup", StringComparison.OrdinalIgnoreCase) + || canonical.Equals("AddBackupType", StringComparison.OrdinalIgnoreCase) + || canonical.Equals("BackupExtension", StringComparison.OrdinalIgnoreCase) + || canonical.Equals("BackupPath", StringComparison.OrdinalIgnoreCase); + + internal static bool IsXmlTrue(string? value) + { + if (string.IsNullOrWhiteSpace(value)) return false; + return value.Equals("true", StringComparison.OrdinalIgnoreCase) + || value.Equals("yes", StringComparison.OrdinalIgnoreCase) + || value.Equals("y", StringComparison.OrdinalIgnoreCase) + || value.Equals("on", StringComparison.OrdinalIgnoreCase) + || value.Equals("1", StringComparison.OrdinalIgnoreCase); + } + + private static string ToCliBool(string value) + => IsXmlTrue(value) ? "yes" : "no"; +} + +/// +/// Merges a .dcomp into a parsed . +/// Precedence matches SPEC §3.3: command line (already on the request) wins, +/// then argfile (already merged), then the project file. +/// +public static class CompFileMerger +{ + public static CommandRequest Apply(CommandRequest request) + { + if (string.IsNullOrWhiteSpace(request.CompFilePath)) + return request; + + var ext = Path.GetExtension(request.CompFilePath); + if (ext.Equals(".scomp", StringComparison.OrdinalIgnoreCase) || + request.Operation == OperationType.SchemaCompare) + { + throw new CompFileException(10, + "/compfile (.scomp) is not yet supported. " + + "Provide /source and /target endpoints directly."); + } + + if (!ext.Equals(".dcomp", StringComparison.OrdinalIgnoreCase) && + ext.Length > 0) + { + throw new CompFileException(10, + "/compfile for /datacompare must be a .dcomp project file."); + } + + var doc = CompFileParser.Load(request.CompFilePath); + var options = new Dictionary(request.Options, StringComparer.OrdinalIgnoreCase); + + foreach (var (key, value) in doc.Options) + { + if (!options.ContainsKey(key)) + options[key] = value; + } + + var exclude = UnionMasks( + options.GetValueOrDefault("ExcludeObjectsByMask"), + doc.ExcludedObjectNames); + if (!string.IsNullOrWhiteSpace(exclude)) + options["ExcludeObjectsByMask"] = exclude; + + // Devart mapping checkboxes are an allowlist. Without this, ArtSync + // compares every non-excluded table in the database. + if (doc.IncludedObjectNames.Count > 0 + && !request.Options.ContainsKey("IncludeObjectsByMask")) + { + options["IncludeObjectsByMask"] = string.Join(",", doc.IncludedObjectNames); + } + + var warnings = request.Warnings.ToList(); + var source = request.Source ?? doc.Source; + var target = request.Target ?? doc.Target; + if (request.Source is null && LooksPasswordless(doc.Source)) + warnings.Add("/compfile source has no Password=; pass /source connection:\"…\" to override."); + if (request.Target is null && LooksPasswordless(doc.Target)) + warnings.Add("/compfile target has no Password=; pass /target connection:\"…\" to override."); + + return request with + { + Source = source, + Target = target, + Options = options, + Warnings = warnings, + }; + } + + private static string UnionMasks(string? existing, IReadOnlyList extra) + { + var parts = new List(); + if (!string.IsNullOrWhiteSpace(existing)) + { + foreach (var p in existing.Split([',', ';'], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)) + parts.Add(p); + } + + foreach (var name in extra) + { + if (parts.Exists(p => p.Equals(name, StringComparison.OrdinalIgnoreCase))) + continue; + parts.Add(name); + } + + return string.Join(",", parts); + } + + private static bool LooksPasswordless(Endpoint? ep) + { + var cs = ep?.ConnectionString; + if (string.IsNullOrWhiteSpace(cs)) return false; + return cs.IndexOf("Password=", StringComparison.OrdinalIgnoreCase) < 0; + } +} diff --git a/src/ArtSync.Compat/KnownOptions.cs b/src/ArtSync.Compat/KnownOptions.cs index d2b41e1..76a4875 100644 --- a/src/ArtSync.Compat/KnownOptions.cs +++ b/src/ArtSync.Compat/KnownOptions.cs @@ -114,6 +114,7 @@ public static class KnownOptions ["micomput"] = "IgnoreComputedColumns", ["ignorelobcolumns"] = "IgnoreLobColumns", ["milob"] = "IgnoreLobColumns", + ["ignoreblobcolumns"] = "IgnoreLobColumns", ["ignorerowguidcolumns"] = "IgnoreRowguidColumns", ["mirowguid"] = "IgnoreRowguidColumns", ["ignoretemporaltablesyscolumns"] = "IgnoreTemporalTableSysColumns", @@ -137,6 +138,7 @@ public static class KnownOptions ["bulkinsert"] = "BulkInsert", ["bi"] = "BulkInsert", ["useschemaprefix"] = "UseSchemaNamePrefix", + ["useschemanameprefix"] = "UseSchemaNamePrefix", ["useschananameprefix"] = "UseSchemaNamePrefix", ["fullnames"] = "UseSchemaNamePrefix", ["reseedidentitycolumns"] = "ReseedIdentityColumns", @@ -251,6 +253,22 @@ public static bool TryGetCanonical(string name, out string canonical) public static bool IsUnsupportedInV1(string canonicalName) => _unsupportedInV1.Contains(canonicalName); + /// + /// Devart jobs often pass unsupported backup switches as :No. + /// That is a no-op, not a request to create backups. + /// + public static bool IsExplicitlyOff(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + return false; + return value.Equals("no", StringComparison.OrdinalIgnoreCase) + || value.Equals("n", StringComparison.OrdinalIgnoreCase) + || value.Equals("false", StringComparison.OrdinalIgnoreCase) + || value.Equals("f", StringComparison.OrdinalIgnoreCase) + || value.Equals("off", StringComparison.OrdinalIgnoreCase) + || value.Equals("0", StringComparison.OrdinalIgnoreCase); + } + public static bool IsWarningOnly(string rawSwitchName) => _warningOnlyNames.Contains(rawSwitchName); diff --git a/src/ArtSync.Data/CanonicalPayload.cs b/src/ArtSync.Data/CanonicalPayload.cs index 09781e8..4a39462 100644 --- a/src/ArtSync.Data/CanonicalPayload.cs +++ b/src/ArtSync.Data/CanonicalPayload.cs @@ -57,8 +57,12 @@ when IsIgnoreTime(options) => "float" or "real" when IsRoundFloat(options) => $"CAST(ROUND({quotedColumnName}, 10) AS NVARCHAR(50))", - "float" or "real" => - $"CAST({quotedColumnName} AS NVARCHAR(50))", + // IEEE bits — CAST to NVARCHAR is not stable across all engines/values. + "float" => + $"CONVERT(NVARCHAR(32), CONVERT(VARBINARY(8), {quotedColumnName}), 2)", + + "real" => + $"CONVERT(NVARCHAR(32), CONVERT(VARBINARY(4), {quotedColumnName}), 2)", // Binary types: hex-encode so HASHBYTES sees printable chars "binary" or "varbinary" or "rowversion" or "timestamp" => @@ -84,6 +88,11 @@ when IsIgnoreTime(options) => "int" or "bigint" or "smallint" or "tinyint" => $"CAST({quotedColumnName} AS NVARCHAR(20))", + // sql_variant: hash the native binary (includes base type + value). + // CAST(... AS NVARCHAR) collapses int 42 and N'42' to the same string. + "sql_variant" => + $"CONVERT(NVARCHAR(MAX), CONVERT(VARBINARY(8000), {quotedColumnName}), 2)", + // XML: stringify (bounded via LOB path above when isLob=true) "xml" => $"CAST({quotedColumnName} AS NVARCHAR(MAX))", diff --git a/src/ArtSync.Data/DataOperationHandler.cs b/src/ArtSync.Data/DataOperationHandler.cs index ada6826..13174d9 100644 --- a/src/ArtSync.Data/DataOperationHandler.cs +++ b/src/ArtSync.Data/DataOperationHandler.cs @@ -37,13 +37,6 @@ public OperationResult Run(CommandRequest request) logger.LogLine($" target={DescribeEndpoint(request.Target)}"); logger.LogLine($" sync={request.SyncMode}"); - // ── Input validation ────────────────────────────────────────────── - - if (request.CompFilePath is not null) - return Finish(logger, new(10, - "/compfile (.dcomp) is not yet supported. " + - "Provide /source and /target endpoints directly.")); - if (request.Source is null) return Finish(logger, new(10, "No /source specified. " + @@ -61,6 +54,16 @@ public OperationResult Run(CommandRequest request) $"srcOnly={info.OnlyInSourceRows} tgtOnly={info.OnlyInTargetRows} " + $"diff={info.DifferentRows}"); + foreach (var g in _engine.LastDiffs + .GroupBy(d => d.TableName, StringComparer.OrdinalIgnoreCase) + .OrderBy(g => g.Key, StringComparer.OrdinalIgnoreCase)) + { + int src = g.Count(d => d.Kind == RowDiffKind.OnlyInSource); + int tgt = g.Count(d => d.Kind == RowDiffKind.OnlyInTarget); + int chg = g.Count(d => d.Kind == RowDiffKind.Different); + logger.LogLine($" table {g.Key}: {g.Count()} row(s) (+{src} src-only, -{tgt} tgt-only, ~{chg} changed)"); + } + if (info.SkippedTables.Count > 0) logger.LogLine($" skipped tables: {string.Join(", ", info.SkippedTables)}"); diff --git a/src/ArtSync.Data/SqlDataApplier.cs b/src/ArtSync.Data/SqlDataApplier.cs index 34c0c6e..c636eec 100644 --- a/src/ArtSync.Data/SqlDataApplier.cs +++ b/src/ArtSync.Data/SqlDataApplier.cs @@ -57,7 +57,7 @@ private static void ExecuteBatches(string cs, IReadOnlyList batches, boo using var cmd = tx is null ? new SqlCommand(batch, conn) : new SqlCommand(batch, conn, tx); - cmd.CommandTimeout = 300; + cmd.CommandTimeout = 0; cmd.ExecuteNonQuery(); } tx?.Commit(); diff --git a/src/ArtSync.Data/SqlDataCompare.cs b/src/ArtSync.Data/SqlDataCompare.cs index ecf1fc3..58531bc 100644 --- a/src/ArtSync.Data/SqlDataCompare.cs +++ b/src/ArtSync.Data/SqlDataCompare.cs @@ -167,7 +167,7 @@ internal static string BuildCs(Endpoint ep) { if (string.IsNullOrWhiteSpace(ep.ConnectionString)) throw new ArgumentException("Connection string is empty."); - return ep.ConnectionString!; + return DevartConnectionString.ForSqlClient(ep.ConnectionString); } if (string.IsNullOrWhiteSpace(ep.Server)) throw new ArgumentException("Missing server."); diff --git a/src/ArtSync.Data/SqlDataScripter.cs b/src/ArtSync.Data/SqlDataScripter.cs index e588e3c..98c4586 100644 --- a/src/ArtSync.Data/SqlDataScripter.cs +++ b/src/ArtSync.Data/SqlDataScripter.cs @@ -217,9 +217,10 @@ string ResolveTgt(string sourceName) // DBCC CHECKIDENT requires a literal value; use a variable. var varName = $"@__reseed_{reseedIdx++}"; var applyName = TableRef(ti with { TargetQualifiedName = ResolveTgt(t) }, options); - var bareTable = applyName.Replace("[", "").Replace("]", ""); sb.AppendLine($"DECLARE {varName} BIGINT = (SELECT ISNULL(MAX({idCol.QuotedName}), 0) FROM {applyName});"); - sb.AppendLine($"DBCC CHECKIDENT ('{bareTable}', RESEED, {varName});"); + // DBCC CHECKIDENT needs a schema-qualified name; stripping + // [schema].[table] down to table breaks non-dbo objects. + sb.AppendLine($"DBCC CHECKIDENT ('{applyName.Replace("'", "''")}', RESEED, {varName});"); } } } @@ -359,6 +360,8 @@ private static string SelectExpression(SqlColumnInfo c) { "geography" or "geometry" => $"{c.QuotedName}.Serialize() AS {c.QuotedName}", "hierarchyid" => $"CAST({c.QuotedName} AS NVARCHAR(900)) AS {c.QuotedName}", + "sql_variant" => + $"SQL_VARIANT_PROPERTY({c.QuotedName}, 'BaseType') AS {QuoteIdent(c.Name + "__svt")}, {c.QuotedName}", _ => c.QuotedName, }; } @@ -384,6 +387,7 @@ private static void AppendBulkInserts( var colNames = string.Join(", ", allCols.Select(c => c.QuotedName)); var tableRef = TableRef(tableInfo, options); + int batches = 0; for (int i = 0; i < rows.Count; i += BulkBatchSize) { var batch = rows.Skip(i).Take(BulkBatchSize).ToList(); @@ -394,7 +398,14 @@ private static void AppendBulkInserts( var comma = j < batch.Count - 1 ? "," : ";"; sb.AppendLine($" ({values}){comma}"); } + + batches++; + // New batch every 20 inserts (~2k rows) so Azure SQL is not one giant command. + if (batches % 20 == 0) + sb.AppendLine("GO"); } + if (batches % 20 != 0) + sb.AppendLine("GO"); } private static string BuildInsert( @@ -455,6 +466,11 @@ private static string FormatCol(SqlColumnInfo c, IReadOnlyDictionary FormatSqlVariant(value, baseType: null), + // ── String / XML — N'...' with escaped single quotes ────────────── _ => $"N'{EscapeString(Convert.ToString(value, CultureInfo.InvariantCulture) ?? "")}'", }; } + /// + /// Emits a T-SQL literal that preserves the sql_variant base type. + /// INSERT ... VALUES (N'42') stores nvarchar, not int. + /// + internal static string FormatSqlVariant(object value, string? baseType) + { + var bt = string.IsNullOrWhiteSpace(baseType) + ? InferSqlVariantBaseType(value) + : baseType.Trim().ToLowerInvariant(); + + return bt switch + { + "bit" => value is bool b ? (b ? "CONVERT(bit, 1)" : "CONVERT(bit, 0)") + : $"CONVERT(bit, {Convert.ToInt32(value, CultureInfo.InvariantCulture)})", + "tinyint" => $"CONVERT(tinyint, {Convert.ToByte(value, CultureInfo.InvariantCulture)})", + "smallint" => $"CONVERT(smallint, {Convert.ToInt16(value, CultureInfo.InvariantCulture)})", + "int" => $"CONVERT(int, {Convert.ToInt32(value, CultureInfo.InvariantCulture)})", + "bigint" => $"CONVERT(bigint, {Convert.ToInt64(value, CultureInfo.InvariantCulture)})", + "decimal" or "numeric" => + $"CONVERT(decimal(38, 18), {Convert.ToDecimal(value, CultureInfo.InvariantCulture).ToString(CultureInfo.InvariantCulture)})", + "money" or "smallmoney" => + $"CONVERT({bt}, {Convert.ToDecimal(value, CultureInfo.InvariantCulture).ToString(CultureInfo.InvariantCulture)})", + "float" => $"CONVERT(float, {Convert.ToDouble(value, CultureInfo.InvariantCulture).ToString("R", CultureInfo.InvariantCulture)})", + "real" => $"CONVERT(real, {Convert.ToSingle(value, CultureInfo.InvariantCulture).ToString("R", CultureInfo.InvariantCulture)})", + "datetime" or "smalldatetime" or "datetime2" or "date" or "time" or "datetimeoffset" => + $"CONVERT({bt}, {Format(value, bt)})", + "uniqueidentifier" => $"CONVERT(uniqueidentifier, {Format(value, "uniqueidentifier")})", + "binary" or "varbinary" => + $"CONVERT(varbinary(8000), {Format(value, "varbinary")})", + "varchar" or "char" => + $"CONVERT({bt}(8000), N'{EscapeString(Convert.ToString(value, CultureInfo.InvariantCulture) ?? "")}')", + _ => + $"CONVERT(nvarchar(4000), N'{EscapeString(Convert.ToString(value, CultureInfo.InvariantCulture) ?? "")}')", + }; + } + + private static string InferSqlVariantBaseType(object value) => value switch + { + bool => "bit", + byte => "tinyint", + short => "smallint", + int => "int", + long => "bigint", + decimal => "decimal", + double => "float", + float => "real", + DateTime => "datetime", + DateTimeOffset => "datetimeoffset", + Guid => "uniqueidentifier", + TimeSpan => "time", + byte[] => "varbinary", + _ => "nvarchar", + }; + // ── Helpers ─────────────────────────────────────────────────────────────── private static string Numeric(object value) diff --git a/src/ArtSync.Schema/ArtSync.Schema.csproj b/src/ArtSync.Schema/ArtSync.Schema.csproj index 3e2acff..7ed1f05 100644 --- a/src/ArtSync.Schema/ArtSync.Schema.csproj +++ b/src/ArtSync.Schema/ArtSync.Schema.csproj @@ -7,6 +7,7 @@ + diff --git a/src/ArtSync.Schema/DacFxSchemaCompare.cs b/src/ArtSync.Schema/DacFxSchemaCompare.cs index f671b4d..d363635 100644 --- a/src/ArtSync.Schema/DacFxSchemaCompare.cs +++ b/src/ArtSync.Schema/DacFxSchemaCompare.cs @@ -33,6 +33,7 @@ public ISchemaSession OpenSession( var comparison = new SchemaComparison(srcEp, tgtEp); DacFxOptionMap.Apply(comparison.Options, options); + ApplyAzureSqlPublishDefaults(comparison.Options, tgtCs); return new DacFxSchemaSession(comparison, filter, options); } @@ -51,7 +52,7 @@ internal static string BuildConnectionString(Endpoint ep, string role) { if (string.IsNullOrWhiteSpace(ep.ConnectionString)) throw new ArgumentException($"Connection string for {role} endpoint is empty."); - return ep.ConnectionString!; + return DevartConnectionString.ForSqlClient(ep.ConnectionString); } // LiveSplit → build an ADO.NET connection string. @@ -90,6 +91,52 @@ private static bool IsConnectionException(Exception ex) || msg.Contains("server", StringComparison.OrdinalIgnoreCase) || ex is Microsoft.Data.SqlClient.SqlException; } + + /// + /// Azure SQL cannot host on-prem logins/users/filegroups. DacFx also + /// treats SQL Server vs Azure SQL as an incompatible platform. + /// + internal static void ApplyAzureSqlPublishDefaults(DacDeployOptions deployOptions, string targetCs) + { + if (targetCs.IndexOf(".database.windows.net", StringComparison.OrdinalIgnoreCase) < 0) + return; + + deployOptions.AllowIncompatiblePlatform = true; + deployOptions.ScriptDatabaseOptions = false; + deployOptions.BlockOnPossibleDataLoss = false; + deployOptions.ExcludeObjectTypes = + [ + ObjectType.Users, + ObjectType.Logins, + ObjectType.RoleMembership, + ObjectType.Permissions, + ObjectType.DatabaseRoles, + ObjectType.ApplicationRoles, + ObjectType.DatabaseOptions, + ObjectType.Filegroups, + ObjectType.Assemblies, + ObjectType.Views, + ObjectType.StoredProcedures, + ObjectType.ScalarValuedFunctions, + ObjectType.TableValuedFunctions, + ObjectType.DatabaseTriggers, + ObjectType.ServerTriggers, + ObjectType.Synonyms, + ObjectType.ExtendedProperties, + ObjectType.XmlSchemaCollections, + ObjectType.FullTextCatalogs, + ObjectType.Certificates, + ObjectType.PartitionFunctions, + ObjectType.PartitionSchemes, + ObjectType.EventNotifications, + ObjectType.Endpoints, + ObjectType.LinkedServers, + ObjectType.LinkedServerLogins, + ObjectType.ServerRoles, + ObjectType.ServerRoleMembership, + ObjectType.Audits, + ]; + } } /// @@ -194,7 +241,13 @@ public SchemaCompareInfo Compare() var targetName = (_comparison.Target as SchemaCompareDatabaseEndpoint)?.DatabaseName ?? "Target"; var scriptResult = _result!.GenerateScript(targetName); - return scriptResult.Success ? scriptResult.Script : null; + if (scriptResult.Success) + return scriptResult.Script; + + var detail = scriptResult.Message + ?? scriptResult.Exception?.Message + ?? "DacFx returned Success=false with no message."; + throw new SchemaIoException($"Script generation failed: {detail}", scriptResult.Exception); } catch (Exception ex) { @@ -211,10 +264,13 @@ public void Publish() if (!publishResult.Success) { var msgs = publishResult.Errors - .Select(e => e.Message) - .Take(5); - throw new SchemaConnectionException( - $"Publish failed: {string.Join("; ", msgs)}"); + .Select(e => string.IsNullOrWhiteSpace(e.Message) + ? e.Exception?.Message ?? $"error {e.Number}" + : e.Message) + .Take(8) + .ToList(); + var detail = msgs.Count > 0 ? string.Join("; ", msgs) : "no DacFx error details"; + throw new SchemaConnectionException($"Publish failed: {detail}"); } } catch (SchemaConnectionException) { throw; } diff --git a/src/ArtSync.Schema/SchemaOperationHandler.cs b/src/ArtSync.Schema/SchemaOperationHandler.cs index 9d5b650..ac90477 100644 --- a/src/ArtSync.Schema/SchemaOperationHandler.cs +++ b/src/ArtSync.Schema/SchemaOperationHandler.cs @@ -37,13 +37,6 @@ public OperationResult Run(CommandRequest request) logger.LogLine($" target={DescribeEndpoint(request.Target)}"); logger.LogLine($" sync={request.SyncMode}"); - // ── Input validation ────────────────────────────────────────────── - - if (request.CompFilePath is not null) - return Finish(logger, new(10, - "/compfile (.scomp) is not yet supported. " + - "Provide /source and /target endpoints directly.")); - if (request.Source is null) return Finish(logger, new(10, "No /source specified. " + diff --git a/tests/ArtSync.Cli.Tests/ArtSync.Cli.Tests.csproj b/tests/ArtSync.Cli.Tests/ArtSync.Cli.Tests.csproj index c2733fe..8846975 100644 --- a/tests/ArtSync.Cli.Tests/ArtSync.Cli.Tests.csproj +++ b/tests/ArtSync.Cli.Tests/ArtSync.Cli.Tests.csproj @@ -25,4 +25,8 @@ + + + + \ No newline at end of file diff --git a/tests/ArtSync.Cli.Tests/CliAppTests.cs b/tests/ArtSync.Cli.Tests/CliAppTests.cs index 6c109d1..f7243ca 100644 --- a/tests/ArtSync.Cli.Tests/CliAppTests.cs +++ b/tests/ArtSync.Cli.Tests/CliAppTests.cs @@ -197,6 +197,54 @@ public void CustomHandler_ReceivesCorrectRequest() captured.Source!.Server.Should().Be("SrcServer"); captured.Target!.Server.Should().Be("TgtServer"); } + + [Fact] + public void CompFile_MissingDcomp_Returns10() + { + var code = BuildApp().Run( + new[] { "/datacompare", "/compfile:" + Path.Combine(Path.GetTempPath(), "missing-artsync.dcomp") }, + "datacompare"); + code.Should().Be(10); + } + + [Fact] + public void CompFile_CorruptDcomp_Returns30() + { + var tmp = Path.Combine(Path.GetTempPath(), $"artsync-cli-bad-{Guid.NewGuid():N}.dcomp"); + File.WriteAllText(tmp, ""); + try + { + BuildApp().Run(new[] { "/datacompare", "/compfile:" + tmp }, "datacompare") + .Should().Be(30); + } + finally { File.Delete(tmp); } + } + + [Fact] + public void CompFile_SampleDcomp_MergesIntoHandlerRequest() + { + CommandRequest? captured = null; + var handler = new StubHandler(100, req => captured = req); + var fixture = Path.Combine(AppContext.BaseDirectory, "sample.dcomp"); + + var code = BuildApp(handler).Run( + new[] + { + "/datacompare", + "/compfile:" + fixture, + "/source", "connection:Data Source=cli-src;Initial Catalog=X;Password=secret", + "/target", "connection:Data Source=cli-tgt;Initial Catalog=X;Password=secret", + "/sync", + }, + "datacompare"); + + code.Should().Be(100); + captured.Should().NotBeNull(); + captured!.Source!.ConnectionString.Should().Contain("cli-src"); + captured.Options["ExcludeObjectsByMask"].Should().Contain("pr.tbl_BankAccount"); + captured.Options["IncludeObjectsByMask"].Should().Contain("dbo.tbl_Loans"); + captured.Options["DisableForeignKeys"].Should().Be("yes"); + } } // ─── Test doubles ───────────────────────────────────────────────────────────── diff --git a/tests/ArtSync.Compat.Tests/ArgvParserGoldenTests.cs b/tests/ArtSync.Compat.Tests/ArgvParserGoldenTests.cs index 1a1cfec..737cc99 100644 --- a/tests/ArtSync.Compat.Tests/ArgvParserGoldenTests.cs +++ b/tests/ArtSync.Compat.Tests/ArgvParserGoldenTests.cs @@ -552,6 +552,111 @@ public void Schema_MixedConnectionAndSplitParams_ReturnsExit11() .Which.ExitCode.Should().Be(11); } + [Fact] + public void Data_SchedulerJob_CompfilePlusLiveConnections_Parses() + { + // Shape of a captured Windows scheduler job: /compfile + /source+/target + // override + /sync + /log. Passwords and hosts are placeholders. + const string src = + @"connection:Data Source=example.database.windows.net;Encrypt=False;Enlist=False;Initial Catalog=ExampleDb;Integrated Security=False;Password=P@ss^#$word;User ID=example;Pooling=False;Transaction Scope Local=True"; + const string tgt = + @"connection:Data Source=SQLHOST\INSTANCE;Encrypt=False;Enlist=False;Initial Catalog=ExampleDb;Integrated Security=False;Password=placeholder;User ID=example;Pooling=False;Transaction Scope Local=True"; + + var argv = new[] + { + "/datacompare", + @"/compfile:C:\Users\Worker\Desktop\TheSync!.dcomp", + "/sync", + "/source", src, + "/target", tgt, + @"/log:D:\Apps\Jobs\sync_20260101.log", + }; + + var req = AssertSuccess(Parser().Parse(argv, "datacompare.com")); + req.Operation.Should().Be(OperationType.DataCompare); + req.SyncMode.Should().Be(SyncMode.Apply); + req.CompFilePath.Should().Be(@"C:\Users\Worker\Desktop\TheSync!.dcomp"); + req.LogPath.Should().Be(@"D:\Apps\Jobs\sync_20260101.log"); + req.Source!.ConnectionString.Should().Contain("Transaction Scope Local=True"); + req.Source.ConnectionString.Should().Contain("P@ss^#$word"); + } + + [Fact] + public void Data_QuotedSwitchValues_AreUnquoted() + { + var argv = new[] + { + "/datacompare", + @"/compfile:""C:\Users\Worker\Desktop\TheSync!.dcomp""", + "/source", @"connection:""Data Source=example;Initial Catalog=Db""", + "/target", @"connection:""Data Source=other;Initial Catalog=Db""", + @"/log:""D:\Apps\Jobs\sync.log""", + }; + + var req = AssertSuccess(Parser().Parse(argv, "datacompare")); + req.CompFilePath.Should().Be(@"C:\Users\Worker\Desktop\TheSync!.dcomp"); + req.LogPath.Should().Be(@"D:\Apps\Jobs\sync.log"); + req.Source!.ConnectionString.Should().Be("Data Source=example;Initial Catalog=Db"); + } + + [Fact] + public void Data_BackupSwitchesExplicitlyOff_AreNoOps() + { + var argv = new[] + { + "/datacompare", + "/source", "connection:Data Source=S;Initial Catalog=D", + "/target", "connection:Data Source=T;Initial Catalog=D", + "/CreateBackupFolder:No", + "/NeedCompressBackup:No", + "/UseSchemaNamePrefix:No", + "/DisableForeignKeys:No", + "/DisableDmlTriggers:No", + "/ExcludeObjectsByMask:*tbl_Audit*,util.log", + }; + + var req = AssertSuccess(Parser().Parse(argv, "datacompare.com")); + req.Options.Should().NotContainKey("CreateBackupFolder"); + req.Options.Should().NotContainKey("NeedCompressBackup"); + req.Options.Should().ContainKey("UseSchemaNamePrefix"); + req.Options["UseSchemaNamePrefix"].Should().Be("No"); + req.Options.Should().ContainKey("ExcludeObjectsByMask"); + req.Options["ExcludeObjectsByMask"].Should().Contain("tbl_Audit"); + } + + [Fact] + public void Data_BackupSwitchOn_Returns10() + { + var argv = new[] + { + "/datacompare", + "/source", "connection:Data Source=S;Initial Catalog=D", + "/target", "connection:Data Source=T;Initial Catalog=D", + "/CreateBackupFolder:Yes", + }; + + Parser().Parse(argv, "datacompare") + .Should().BeOfType() + .Which.ExitCode.Should().Be(10); + } + + [Fact] + public void Data_ReportFormatXls_Returns10() + { + var argv = new[] + { + "/datacompare", + "/source", "connection:Data Source=S;Initial Catalog=D", + "/target", "connection:Data Source=T;Initial Catalog=D", + "/report:out.xls", + "/reportformat:xls", + }; + + Parser().Parse(argv, "datacompare") + .Should().BeOfType() + .Which.ExitCode.Should().Be(10); + } + // ─── Helper ─────────────────────────────────────────────────────────────── private static CommandRequest AssertSuccess(ParseResult result) diff --git a/tests/ArtSync.Compat.Tests/ArtSync.Compat.Tests.csproj b/tests/ArtSync.Compat.Tests/ArtSync.Compat.Tests.csproj index 6739fc5..5dcf51a 100644 --- a/tests/ArtSync.Compat.Tests/ArtSync.Compat.Tests.csproj +++ b/tests/ArtSync.Compat.Tests/ArtSync.Compat.Tests.csproj @@ -24,4 +24,8 @@ + + + + \ No newline at end of file diff --git a/tests/ArtSync.Compat.Tests/CompFileParserTests.cs b/tests/ArtSync.Compat.Tests/CompFileParserTests.cs new file mode 100644 index 0000000..91967b4 --- /dev/null +++ b/tests/ArtSync.Compat.Tests/CompFileParserTests.cs @@ -0,0 +1,199 @@ +using ArtSync.Abstractions; +using ArtSync.Compat; +using FluentAssertions; + +namespace ArtSync.Compat.Tests; + +public sealed class CompFileParserTests +{ + private static string FixturePath => + Path.Combine(AppContext.BaseDirectory, "sample.dcomp"); + + [Fact] + public void SampleDcomp_LoadsConnectionsOptionsAndExclusions() + { + var doc = CompFileParser.Load(FixturePath); + + doc.Source!.Kind.Should().Be(EndpointKind.ConnectionString); + doc.Source.ConnectionString.Should().Contain("example.database.windows.net"); + doc.Target!.ConnectionString.Should().Contain(@"SQLHOST\INSTANCE"); + + doc.Options["ExcludeObjectsByMask"].Should().Contain("*tbl_History_*"); + doc.Options["MappingIgnoreSpaces"].Should().Be("yes"); + doc.Options["IgnoreLobColumns"].Should().Be("no"); + doc.Options["IgnoreIdentityColumns"].Should().Be("no"); + doc.Options["DisableForeignKeys"].Should().Be("yes"); + doc.Options["DisableDmlTriggers"].Should().Be("yes"); + doc.Options["UseSchemaNamePrefix"].Should().Be("no"); + doc.Options.Should().NotContainKey("CreateBackupFolder"); + doc.Options.Should().NotContainKey("NeedCompressBackup"); + + doc.ExcludedObjectNames.Should().Contain("pr.tbl_BankAccount"); + doc.ExcludedObjectNames.Should().Contain("pr.tbl_PreliminaryReceipts"); + doc.ExcludedObjectNames.Should().NotContain("dbo.tbl_Loans"); + doc.IncludedObjectNames.Should().Contain("dbo.tbl_Loans"); + doc.IncludedObjectNames.Should().Contain("dbo.tbl_Address"); + doc.IncludedObjectNames.Should().NotContain("pr.tbl_BankAccount"); + } + + [Fact] + public void V6ConnectionModel_ReadsConnectionString() + { + var xml = """ + + + + + Data Source=example.database.windows.net;Initial Catalog=ExampleDb;User ID=example + + + + + Data Source=SQLHOST;Initial Catalog=ExampleDb;User ID=example + + + + + + + + + """; + var tmp = Path.Combine(Path.GetTempPath(), $"artsync-v6-{Guid.NewGuid():N}.dcomp"); + File.WriteAllText(tmp, xml); + try + { + var doc = CompFileParser.Load(tmp); + doc.Source!.ConnectionString.Should().Contain("example.database.windows.net"); + doc.Target!.ConnectionString.Should().Contain("Data Source=SQLHOST"); + } + finally { File.Delete(tmp); } + } + + [Fact] + public void MissingFile_Throws10() + { + var act = () => CompFileParser.Load(Path.Combine(Path.GetTempPath(), "no-such-artsync.dcomp")); + act.Should().Throw().Which.ExitCode.Should().Be(10); + } + + [Fact] + public void CorruptXml_Throws30() + { + var tmp = Path.Combine(Path.GetTempPath(), $"artsync-bad-{Guid.NewGuid():N}.dcomp"); + File.WriteAllText(tmp, ""); + try + { + var act = () => CompFileParser.Load(tmp); + act.Should().Throw().Which.ExitCode.Should().Be(30); + } + finally { File.Delete(tmp); } + } + + [Fact] + public void Merge_CliConnectionsWin_AndUnionsCheckboxExcludes() + { + var req = new CommandRequest( + Operation: OperationType.DataCompare, + Source: new Endpoint(EndpointKind.ConnectionString, ConnectionString: "Data Source=cli-src;Initial Catalog=X;Password=p"), + Target: new Endpoint(EndpointKind.ConnectionString, ConnectionString: "Data Source=cli-tgt;Initial Catalog=X;Password=p"), + SyncMode: SyncMode.Apply, + SyncFilePath: null, + ArgFilePath: null, + CompFilePath: FixturePath, + FilterFilePath: null, + ReportPath: null, + LogPath: null, + ReportFormat: null, + Quiet: false, + Argv0: "datacompare", + Options: new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["ExcludeObjectsByMask"] = "*tbl_Reminder*", + }, + Warnings: []); + + var merged = CompFileMerger.Apply(req); + merged.Source!.ConnectionString.Should().Contain("cli-src"); + merged.Target!.ConnectionString.Should().Contain("cli-tgt"); + merged.Options["ExcludeObjectsByMask"].Should().Contain("*tbl_Reminder*"); + merged.Options["ExcludeObjectsByMask"].Should().Contain("pr.tbl_BankAccount"); + merged.Options["ExcludeObjectsByMask"].Should().NotContain("*tbl_History_*"); + merged.Options["IncludeObjectsByMask"].Should().Contain("dbo.tbl_Loans"); + merged.Options["IncludeObjectsByMask"].Should().Contain("dbo.tbl_Address"); + merged.Options["IncludeObjectsByMask"].Should().NotContain("pr.tbl_BankAccount"); + merged.Options["DisableDmlTriggers"].Should().Be("yes"); + } + + [Fact] + public void Merge_FillsEndpointsFromProject_WhenCliOmitsThem() + { + var req = new CommandRequest( + Operation: OperationType.DataCompare, + Source: null, + Target: null, + SyncMode: SyncMode.None, + SyncFilePath: null, + ArgFilePath: null, + CompFilePath: FixturePath, + FilterFilePath: null, + ReportPath: null, + LogPath: null, + ReportFormat: null, + Quiet: false, + Argv0: "datacompare", + Options: new Dictionary(StringComparer.OrdinalIgnoreCase), + Warnings: []); + + var merged = CompFileMerger.Apply(req); + merged.Source.Should().NotBeNull(); + merged.Target.Should().NotBeNull(); + merged.Warnings.Should().Contain(w => w.Contains("Password=", StringComparison.Ordinal)); + } + + [Fact] + public void Merge_Scomp_Throws10() + { + var req = new CommandRequest( + Operation: OperationType.SchemaCompare, + Source: null, Target: null, + SyncMode: SyncMode.None, SyncFilePath: null, ArgFilePath: null, + CompFilePath: "schema.scomp", + FilterFilePath: null, ReportPath: null, LogPath: null, ReportFormat: null, + Quiet: false, Argv0: "schemacompare", + Options: new Dictionary(), + Warnings: []); + + var act = () => CompFileMerger.Apply(req); + act.Should().Throw().Which.ExitCode.Should().Be(10); + } + + [Fact] + public void LocalCapture_TheSyncDcomp_ParsesIfPresent() + { + var path = FindUp("_data/TheSync!.dcomp"); + if (path is null) return; + + var doc = CompFileParser.Load(path); + doc.Source!.ConnectionString.Should().Contain("database.windows.net"); + doc.ExcludedObjectNames.Should().Contain("pr.tbl_BankAccount"); + doc.IncludedObjectNames.Should().HaveCount(123); + doc.IncludedObjectNames.Should().Contain("dbo.tbl_Loans"); + doc.IncludedObjectNames.Should().NotContain("UW.tbl_Loans"); + doc.Options["ExcludeObjectsByMask"].Should().Contain("tbl_EmailNotificationQueue"); + doc.Options["DisableForeignKeys"].Should().Be("yes"); + doc.Options.Should().NotContainKey("CreateBackupFolder"); + } + + private static string? FindUp(string relative) + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null) + { + var candidate = Path.Combine(dir.FullName, relative); + if (File.Exists(candidate)) return candidate; + dir = dir.Parent; + } + return null; + } +} diff --git a/tests/ArtSync.Compat.Tests/DevartConnectionStringTests.cs b/tests/ArtSync.Compat.Tests/DevartConnectionStringTests.cs new file mode 100644 index 0000000..732f696 --- /dev/null +++ b/tests/ArtSync.Compat.Tests/DevartConnectionStringTests.cs @@ -0,0 +1,46 @@ +using ArtSync.Abstractions; +using FluentAssertions; + +namespace ArtSync.Compat.Tests; + +public sealed class DevartConnectionStringTests +{ + private const string SchedulerShape = + "Data Source=example.database.windows.net;Encrypt=False;Enlist=False;" + + "Initial Catalog=ExampleDb;Integrated Security=False;Password=P@ss^#$word;" + + "User ID=example;Pooling=False;Transaction Scope Local=True"; + + [Fact] + public void ForSqlClient_DropsTransactionScopeLocal() + { + var cs = DevartConnectionString.ForSqlClient(SchedulerShape); + cs.Should().NotContain("Transaction Scope Local"); + cs.Should().Contain("Pooling=False"); + cs.Should().Contain("P@ss^#$word"); + } + + [Fact] + public void ForSqlClient_Azure_ForcesEncrypt() + { + var cs = DevartConnectionString.ForSqlClient(SchedulerShape); + cs.Should().Contain("Encrypt=True"); + cs.Should().NotContain("Encrypt=False"); + } + + [Fact] + public void ForSqlClient_OnPrem_KeepsEncryptFalse() + { + var cs = DevartConnectionString.ForSqlClient( + "Data Source=SQLHOST\\INSTANCE;Encrypt=False;Initial Catalog=ExampleDb;" + + "User ID=u;Password=p;Transaction Scope Local=True"); + cs.Should().Contain("Encrypt=False"); + cs.Should().NotContain("Transaction Scope Local"); + } + + [Fact] + public void ForSqlClient_StripsWrappingQuotes() + { + var cs = DevartConnectionString.ForSqlClient("\"Data Source=local;Initial Catalog=Db\""); + cs.Should().StartWith("Data Source=local"); + } +} diff --git a/tests/ArtSync.Data.Tests/CanonicalPayloadTests.cs b/tests/ArtSync.Data.Tests/CanonicalPayloadTests.cs index 2912427..509585a 100644 --- a/tests/ArtSync.Data.Tests/CanonicalPayloadTests.cs +++ b/tests/ArtSync.Data.Tests/CanonicalPayloadTests.cs @@ -112,14 +112,20 @@ public void DateTimeWithIgnoreTime_CastsToDate(string sqlType) // ── Float types ─────────────────────────────────────────────────────────── [Fact] - public void FloatType_NoRound_CastToNVarchar() + public void FloatType_NoRound_UsesIeeeBits() { var expr = CanonicalPayload.BuildExpression("[Price]", "float", NoOpts); - expr.Should().Contain("CAST("); - expr.Should().Contain("NVARCHAR(50)"); + expr.Should().Contain("VARBINARY(8)"); expr.Should().NotContain("ROUND"); } + [Fact] + public void RealType_NoRound_UsesIeeeBits() + { + var expr = CanonicalPayload.BuildExpression("[Price]", "real", NoOpts); + expr.Should().Contain("VARBINARY(4)"); + } + [Fact] public void FloatType_WithRound_UsesRound() { @@ -184,10 +190,18 @@ public void XmlType_CastToNVarcharMax() // ── Unknown type ────────────────────────────────────────────────────────── + [Fact] + public void SqlVariant_UsesBinaryConvert() + { + var expr = CanonicalPayload.BuildExpression("[Val]", "sql_variant", NoOpts); + expr.Should().Contain("VARBINARY(8000)"); + expr.Should().Contain(", 2)"); + } + [Fact] public void UnknownType_FallsBackToNVarcharMax() { - var expr = CanonicalPayload.BuildExpression("[Col]", "sql_variant", NoOpts); + var expr = CanonicalPayload.BuildExpression("[Col]", "madeuptype", NoOpts); expr.Should().Contain("NVARCHAR(MAX)"); } diff --git a/tests/ArtSync.Data.Tests/DataHandlerUnitTests.cs b/tests/ArtSync.Data.Tests/DataHandlerUnitTests.cs index 8b36e3d..96bf997 100644 --- a/tests/ArtSync.Data.Tests/DataHandlerUnitTests.cs +++ b/tests/ArtSync.Data.Tests/DataHandlerUnitTests.cs @@ -161,12 +161,12 @@ public void MissingTarget_Returns10() } [Fact] - public void CompFile_Returns10_UntilImplemented() + public void CompFilePath_DoesNotBlockWhenEndpointsPresent() { var fake = new FakeDataCompare( - new DataCompareInfo(DataCompareStatus.Identical, 0, 0, 0, 0, [], [])); + new DataCompareInfo(DataCompareStatus.Identical, 0, 0, 0, 0, ["dbo.Orders"], [])); - DataHandler(fake).Run(MakeRequest(compFile: "data.dcomp")).ExitCode.Should().Be(10); + DataHandler(fake).Run(MakeRequest(compFile: "data.dcomp")).ExitCode.Should().Be(100); } // ─── Connection failure → exit 40 ───────────────────────────────────────── diff --git a/tests/ArtSync.Data.Tests/DevartConnectionStringBuildTests.cs b/tests/ArtSync.Data.Tests/DevartConnectionStringBuildTests.cs new file mode 100644 index 0000000..092f587 --- /dev/null +++ b/tests/ArtSync.Data.Tests/DevartConnectionStringBuildTests.cs @@ -0,0 +1,24 @@ +using ArtSync.Abstractions; +using ArtSync.Data; +using FluentAssertions; + +namespace ArtSync.Data.Tests; + +public sealed class DevartConnectionStringBuildTests +{ + [Fact] + public void BuildCs_SchedulerAzureString_IsSqlClientSafe() + { + var ep = new Endpoint( + EndpointKind.ConnectionString, + ConnectionString: + "Data Source=example.database.windows.net;Encrypt=False;Enlist=False;" + + "Initial Catalog=ExampleDb;Integrated Security=False;Password=P@ss^#$word;" + + "User ID=example;Pooling=False;Transaction Scope Local=True"); + + var cs = SqlDataCompare.BuildCs(ep); + cs.Should().NotContain("Transaction Scope Local"); + cs.Should().Contain("Encrypt=True"); + cs.Should().Contain("P@ss^#$word"); + } +} diff --git a/tests/ArtSync.Data.Tests/SqlFkGraphTests.cs b/tests/ArtSync.Data.Tests/SqlFkGraphTests.cs index 9bb30b3..418d8d8 100644 --- a/tests/ArtSync.Data.Tests/SqlFkGraphTests.cs +++ b/tests/ArtSync.Data.Tests/SqlFkGraphTests.cs @@ -68,6 +68,27 @@ public void ExcludeMask_DropsMatchingTables() SqlNameMask.IsTableIncluded("[dbo].[Customers]", opts).Should().BeTrue(); } + [Fact] + public void ExcludeMask_SchedulerList_DropsHistoryAndQueueTables() + { + var opts = new Dictionary + { + ["ExcludeObjectsByMask"] = + "*tbl_PreliminaryReceipts*,*tbl_PRLoan*,*tbl_MARSVal*,*tbl_SvcPeriodBal*," + + "*tbl_MARSSvcCompare*,*tbl_Reminder*,*tbl_EmailNotificationQueue*," + + "*relationshipcashflowhistory*,*tbl_ReportPVHistory*,*tbl_reporthistory*," + + "*tbl_History_IRR*,*tbl_History_PV*,*ReleasePrice*,*OccupancyStatus*," + + "*ReleasedLog*,*ReleasePriceLog*,*InterestCalcMethod*,*PrincipalCalculation*," + + "util.log,*BankAccount*,*IncomeType*,*ReportCount*,*ReportingMonth*", + }; + + SqlNameMask.IsTableIncluded("[dbo].[tbl_EmailNotificationQueue]", opts).Should().BeFalse(); + SqlNameMask.IsTableIncluded("[dbo].[tbl_History_PV]", opts).Should().BeFalse(); + SqlNameMask.IsTableIncluded("[dbo].[BankAccount]", opts).Should().BeFalse(); + SqlNameMask.IsTableIncluded("[dbo].[Customers]", opts).Should().BeTrue(); + SqlNameMask.IsTableIncluded("[dbo].[Loan]", opts).Should().BeTrue(); + } + [Fact] public void IncludeMask_KeepsOnlyMatches() { diff --git a/tests/ArtSync.Data.Tests/SqlValueFormatterTests.cs b/tests/ArtSync.Data.Tests/SqlValueFormatterTests.cs index a2d1ce7..9ad03db 100644 --- a/tests/ArtSync.Data.Tests/SqlValueFormatterTests.cs +++ b/tests/ArtSync.Data.Tests/SqlValueFormatterTests.cs @@ -221,6 +221,24 @@ public void StringTypes_ProduceNPrefixedQuotedLiteral(string typeName) result.Should().Be("N'Hello World'"); } + [Fact] + public void SqlVariant_Int_UsesTypedConvert() + => SqlValueFormatter.Format(42, "sql_variant").Should().Be("CONVERT(int, 42)"); + + [Fact] + public void SqlVariant_String_UsesNVarcharConvert() + => SqlValueFormatter.Format("Active", "sql_variant") + .Should().Be("CONVERT(nvarchar(4000), N'Active')"); + + [Fact] + public void SqlVariant_VarcharBaseType_UsesVarcharConvert() + => SqlValueFormatter.FormatSqlVariant("D", "varchar") + .Should().Be("CONVERT(varchar(8000), N'D')"); + + [Fact] + public void SqlVariant_Int_DoesNotQuoteAsString() + => SqlValueFormatter.Format(42, "sql_variant").Should().NotContain("N'"); + [Fact] public void String_SingleQuoteEscaped() => SqlValueFormatter.Format("O'Brien", "nvarchar").Should().Be("N'O''Brien'"); diff --git a/tests/ArtSync.Schema.Tests/SchemaHandlerUnitTests.cs b/tests/ArtSync.Schema.Tests/SchemaHandlerUnitTests.cs index c29cfff..37a02f6 100644 --- a/tests/ArtSync.Schema.Tests/SchemaHandlerUnitTests.cs +++ b/tests/ArtSync.Schema.Tests/SchemaHandlerUnitTests.cs @@ -197,13 +197,13 @@ public void MissingTarget_Returns10() } [Fact] - public void CompFile_Returns10_UntilImplemented() + public void CompFilePath_DoesNotBlockWhenEndpointsPresent() { var fake = new FakeSchemaCompare( info: new SchemaCompareInfo(true, false, 0, [], [])); Handler(fake).Run(MakeRequest(compFile: "schema.scomp")) - .ExitCode.Should().Be(10); + .ExitCode.Should().Be(100); } // ─── Filter file: missing .scflt → exit 114 ─────────────────────────────── diff --git a/tests/fixtures/README.md b/tests/fixtures/README.md index 57ebeb6..31333db 100644 --- a/tests/fixtures/README.md +++ b/tests/fixtures/README.md @@ -67,23 +67,19 @@ docker exec -i $(docker compose ps -q sqlserver) \ < tests/fixtures/setup.sql ``` -## `/compfile` (`.scomp` / `.dcomp`) — waiting for a real fixture - -`/compfile` parsing is intentionally **not implemented**. Both `SchemaOperationHandler` and -`DataOperationHandler` return exit **10** with a clear message when `/compfile` is supplied. - -To unblock a parser, commit a captured `.scomp` or `.dcomp` file here in -`tests/fixtures/` (e.g. `example.scomp`). The file must come from a real Devart -dbForge run — do **not** invent XML structure, as the exact element names and -attribute layout are proprietary. - -Once a sample exists in this directory: -1. Create `src/ArtSync.Compat/CompFileParser.cs` that reads the XML and populates - a `CommandRequest` (connections, options, skipped objects list). -2. Wire it into `SchemaOperationHandler` / `DataOperationHandler` as a pre-step - before the `/source`+`/target` validation. -3. Add an integration test that passes the fixture via `/compfile` and expects - the same exit code as when passing source/target directly. +## `/compfile` (`.dcomp`) + +Data-compare project files are loaded by `ArtSync.Compat.CompFileParser`. A sanitized +capture of the Devart `CompareDocument` schema is in [`sample.dcomp`](sample.dcomp). + +- Command line `/source` and `/target` override project connections (SPEC §3.3). +- Compare and synchronization options fill in only when the CLI did not set them. +- `CreateBackupFolder` / `NeedCompressBackup` in the project are ignored unless + `BackupDatabase` is on (then exit 10). +- Objects with `Included="False"` are appended to `ExcludeObjectsByMask`. +- `.scomp` (schema projects) still exit **10**. + +Corrupt XML returns exit **30**. Missing file returns **10**. ## Local SQL Server (no Docker) diff --git a/tests/fixtures/sample.dcomp b/tests/fixtures/sample.dcomp new file mode 100644 index 0000000..38acbf9 --- /dev/null +++ b/tests/fixtures/sample.dcomp @@ -0,0 +1,64 @@ + + + + + aa8afe9c-7d62-4564-b477-c12e62e0f440 + 5.8.106.0 + 2.0 + + + ExampleDb.example.database.windows.net + Data Source=example.database.windows.net;Encrypt=False;Initial Catalog=ExampleDb;Integrated Security=False;User ID=example + ExampleDb + + + ExampleDb.SQLHOST\INSTANCE + Data Source=SQLHOST\INSTANCE;Encrypt=False;Initial Catalog=ExampleDb;Integrated Security=False;User ID=example + ExampleDb + + + true + false + false + false + false + false + *ModifiedDate*,*CreatedOn*,*UpdatedBy* + true + *tbl_History_*,util.log + false + * + true + true + false + true + true + true + true + + + false + true + true + false + true + true + false + true + false + false + true + + + + + + + + + + + + + +