feat: refactor conformance suite into vertical-agnostic architecture (RFC #520)

.github/workflows/conformance-tests.yml

@@ -38,24 +38,24 @@
     runs-on: ubuntu-latest
     steps:
       - name: Check out conformance repo
         uses: actions/checkout@v5
         with:
           path: conformance
 
       - name: Check out samples repo
         uses: actions/checkout@v5
         with:
           repository: Universal-Commerce-Protocol/samples
           path: samples
 
       - name: Check out SDK repo
         uses: actions/checkout@v5
         with:
           repository: Universal-Commerce-Protocol/python-sdk
           path: python-sdk
 
       - name: Install uv
         uses: astral-sh/setup-uv@v8.2.0
         with:
           prune-cache: false
 
@@ -75,7 +75,7 @@
           uv run --directory samples/rest/python/server import_csv.py \
             --products_db_path=${DATABASE_PATH}/products.db \
             --transactions_db_path=${DATABASE_PATH}/transactions.db \
-            --data_dir=../../../../conformance/test_data/flower_shop
+            --data_dir=../../../../conformance/shopping/fixtures/flower_shop
 
       - name: Start Flower Shop server
         run: |
@@ -101,15 +101,9 @@
         continue-on-error: ${{ github.event_name == 'pull_request' }}
         working-directory: conformance
         run: |
-          EXIT_CODE=0
-          for test_file in *_test.py; do
-            echo "::group::${test_file}"
-            if ! uv run "${test_file}" \
-              --server_url=http://localhost:${MERCHANT_SERVER_PORT} \
-              --simulation_secret=${SIMULATION_SECRET} \
-              --conformance_input=test_data/flower_shop/conformance_input.json; then
-              EXIT_CODE=1
-            fi
-            echo "::endgroup::"
-          done
-          exit ${EXIT_CODE}
+          uv run ucp-conformance \
+            --server_url=http://localhost:${MERCHANT_SERVER_PORT} \
+            --simulation_secret=${SIMULATION_SECRET} \
+            --platform=google \
+            --suite=all \
+            -v

README.md

@@ -14,21 +14,35 @@
    limitations under the License.
 -->
 
-# UCP SDK Integration Tests
-
-This directory contains integration tests that run against a running UCP
-Merchant Server instance. These tests are language-agnostic regarding the server
-implementation (Python, Node.js, etc.) and verify adherence to the UCP
-specification.
+# UCP Conformance Test Suite
+
+This repository contains the official Universal Commerce Protocol (UCP)
+conformance test suite. The suite validates merchant server implementations
+against the UCP specification across core protocol requirements, common extensions
+(payments, webhooks), and vertical-specific domains (such as retail shopping).
+
+## Architecture
+
+The test suite follows a 4-tier modular architecture per UCP RFC #520:
+
+- **`framework/`**: Shared testing harness decoupled from any business vertical.
+  Provides capability discovery, platform profile evaluation, mock webhook and
+  agent servers, base test cases, and capability decorators.
+- **`core/`**: Agnostic protocol tests (`protocol_test`, `binding_test`,
+  `security_test`, generic `idempotency_test`). Can be run against any UCP server
+  regardless of business vertical.
+- **`common/`**: Cross-cutting protocol capabilities such as webhooks
+  (`common/webhooks/`) and payment handlers (`common/payments/`).
+- **`shopping/`**: Retail shopping domain tests organized by functional domain:
+  `checkout/`, `discount/`, `fulfillment/`, `order/`, and `validation/`.
+- **`platforms/`**: Platform certification profiles (e.g. `google.yaml`) that
+  mandate required capabilities and payment handlers.
 
 ## Prerequisites
 
-The tests assume a UCP Merchant Server is running and accessible via HTTP. The
-server must be started with databases initialized using data from
-`test_data/flower_shop` directory. Instructions to start the servers follow.
-
-NOTE: These instructions assume the commands are executed from the directory
-containing this README.
+The tests assume a UCP Merchant Server is running and accessible via HTTP.
+For testing the shopping vertical, the server must be started with databases
+initialized using data from `shopping/fixtures/flower_shop` (or `test_data/flower_shop`).
 
 ### Updating dependencies
 
@@ -37,24 +51,24 @@ uv sync
 
 uv sync --directory ../samples/rest/python/server/
 
-uv sync --directory ../sdk/python/
+uv sync --directory ../python-sdk/
 ```
 
-### Initializing the database
+### Initializing the test database
 
 ```bash
 DATABASE_PATH=/tmp/ucp_test
 
 rm -rf ${DATABASE_PATH}
-mkdir ${DATABASE_PATH}
+mkdir -p ${DATABASE_PATH}
 
 uv run --directory ../samples/rest/python/server import_csv.py \
     --products_db_path=${DATABASE_PATH}/products.db \
     --transactions_db_path=${DATABASE_PATH}/transactions.db \
-    --data_dir=../../../../conformance/test_data/flower_shop
+    --data_dir=../../../../conformance/shopping/fixtures/flower_shop
 ```
 
-Starting the server:
+### Starting the server
 
 ```bash
 SIMULATION_SECRET=super-secret-sim-key
@@ -68,17 +82,51 @@ uv run --directory ../samples/rest/python/server server.py \
 MERCHANT_SERVER_PID=$!
 ```
 
-## Running the Tests
+## Running the Conformance Tests
+
+Use the `ucp-conformance` CLI orchestrator (or `uv run runner.py`):
 
 ```bash
-for test_file in *_test.py; do
-uv run ${test_file} \
+# Run all discovered tests
+uv run ucp-conformance \
+    --server_url=http://localhost:${MERCHANT_SERVER_PORT} \
+    --simulation_secret=${SIMULATION_SECRET}
+
+# Run only agnostic core protocol tests (zero retail dependencies)
+uv run ucp-conformance --suite=core --server_url=http://localhost:${MERCHANT_SERVER_PORT}
+
+# Run retail shopping tests
+uv run ucp-conformance --suite=shopping --server_url=http://localhost:${MERCHANT_SERVER_PORT}
+
+# Certify server compliance against a platform profile (e.g., Google)
+uv run ucp-conformance \
+    --platform=google \
     --server_url=http://localhost:${MERCHANT_SERVER_PORT} \
-    --simulation_secret=${SIMULATION_SECRET} \
-    --conformance_input=test_data/flower_shop/conformance_input.json
-done
+    --simulation_secret=${SIMULATION_SECRET}
+
+# Dry run to preview test execution list
+uv run ucp-conformance --suite=all --dry-run
 ```
 
+### CLI Options
+
+| Option                | Default                 | Description                                                    |
+| --------------------- | ----------------------- | -------------------------------------------------------------- |
+| `--server_url`        | `http://localhost:8182` | Base URL of the target UCP server.                             |
+| `--platform`          | `None`                  | Platform profile to validate compliance against (`google`).    |
+| `--suite`             | `all`                   | Test suite(s) to execute: `all`, `core`, `common`, `shopping`. |
+| `--simulation_secret` | `""`                    | Secret for simulation endpoints.                               |
+| `--conformance_input` | `None`                  | Optional path to custom conformance input JSON.                |
+| `-k`, `--filter`      | `None`                  | Filter test cases matching a pattern.                          |
+| `-v`, `--verbose`     | `False`                 | Enable verbose execution logs.                                 |
+| `--dry-run`           | `False`                 | List matching tests without executing.                         |
+
+### Customizing Test Fixtures
+
+Shopping test fixtures (SKU, expected pricing, discount codes, shipping destinations)
+can be customized in `shopping/fixtures/flower_shop/test_fixtures.json` or by passing
+a custom configuration file using `--fixture_config`.
+
 ## Cleaning Up
 
 Terminate the server using: