test(ministack): add webhook to scale-up smoke chain

.github/workflows/ministack.yml

@@ -11,6 +11,7 @@ on:
       - "policies/**"
       - "examples/**"
       - "modules/**"
+      - "lambdas/**"
   pull_request:
     paths:
       - ".github/workflows/ministack.yml"
@@ -19,6 +20,7 @@ on:
       - "policies/**"
       - "examples/**"
       - "modules/**"
+      - "lambdas/**"
   workflow_dispatch:
 
 concurrency:
@@ -103,3 +105,64 @@ jobs:
         env:
           EXAMPLE: ${{ matrix.example }}
         run: tests/ministack/run-example.sh destroy "$EXAMPLE"
+
+  integration_smoke:
+    name: Run webhook and pool lifecycle smoke test against MiniStack
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    services:
+      ministack:
+        image: ghcr.io/ministackorg/ministack:1.5.7@sha256:37361b9ef886463d5632d5a4b2d114da4b7a5c5793f52f07dbc72579f2fd9207
+        ports:
+          - 4566:4566
+        options: --add-host=host.docker.internal:host-gateway
+        env:
+          MINISTACK_ACCOUNT_ID: "000000000000"
+          MINISTACK_REGION: eu-west-1
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1
+        with:
+          egress-policy: audit
+
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Setup Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version-file: lambdas/.nvmrc
+          package-manager-cache: false
+
+      - name: Setup Terraform
+        uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
+        with:
+          terraform_version: latest
+          terraform_wrapper: false
+
+      - name: Install Lambda dependencies
+        working-directory: lambdas
+        run: yarn install --frozen-lockfile
+
+      - name: Build smoke-test Lambda distributions
+        working-directory: lambdas
+        run: |
+          yarn workspace @aws-github-runner/webhook dist
+          yarn workspace @aws-github-runner/control-plane dist
+
+      - name: Start MockServer
+        id: mockserver
+        uses: mock-server/setup-mockserver@24612c2ccef1f83d587f331ed77cc5cef441e0b1 # v1.0.0
+        with:
+          version: '7.6.0@sha256:80b3b1a26f3553d0c81a3f3896b5b7274c17b2a2e52f0fd2b28e246bc9efa290'
+          port: '1080'
+          startup-timeout: '60'
+
+      - name: Run webhook and pool lifecycle smoke test
+        env:
+          MINISTACK_GITHUB_MOCK_HOST: host.docker.internal
+          MINISTACK_GITHUB_MOCK_PORT: "1080"
+          MINISTACK_GITHUB_MOCK_URL: ${{ steps.mockserver.outputs.url }}
+        run: sh tests/ministack/run-smoke.sh

tests/ministack/README.md

@@ -33,3 +33,54 @@ MiniStack's AWS-compatible EC2 API, then removes only the resources it created
 during cleanup. MiniStack v1.5.7 provides the EC2 image behavior needed by the
 `default`, `ephemeral`, and `multi-runner` examples, so they are included in
 the same lifecycle matrix.
+
+## Webhook and runner lifecycle smoke test
+
+The smoke test covers two independent lifecycle chains. The webhook chain
+sends signed `workflow_job` webhooks through the API Gateway endpoint and
+verifies the asynchronous path through EventBridge, the dispatcher Lambda, SQS,
+and the scale-up Lambda. It runs scale-up once without a dynamic label and once
+with `ghr-ec2-instance-type:m5.large`, checking that the first launch uses a
+configured default instance type and the second launch uses exactly `m5.large`.
+The scale-up Lambda calls a pinned `mockserver/mockserver` container initialized
+from `github-api-expectations.json`; the test uses MockServer's verification API
+to confirm the expected GitHub API calls for both jobs. It also checks the
+webhook, dispatcher, and scale-up Lambda log groups for each smoke job ID, then
+confirms that both MiniStack EC2 runner instances are removed and terminated.
+
+The second, pool chain then invokes the pool Lambda with a pool size of one and verifies every
+expected GitHub API route for pool reconciliation, including the installation,
+token, runner-list, and registration-token calls, before confirming that it
+creates a second EC2 runner. Installation lookup is mocked for configurations
+that do not provide a stored installation ID, but is conditional and is not a
+required assertion. The test also verifies the `ghr:Application`,
+`ghr:created_by`, `ghr:Type`, and `ghr:Owner` tags used to discover managed
+instances. MiniStack does not currently propagate the Terraform launch-template
+`ghr:environment` tag to instances, so that tag is not asserted by this smoke
+test. For both the scale-up and pool runners, it invokes the
+scale-down Lambda and verifies every required GitHub API route, including token
+creation, runner listing, runner-state lookup, and runner deletion. It then
+verifies the GitHub runner `404`, checks the
+scale-down log entry as supplementary evidence, and verifies EC2 termination.
+The pool schedule is configured for a far-future date because the test invokes
+the Lambda directly.
+
+Build the two real Lambda distributions, start MiniStack, and run:
+
+```sh
+(cd lambdas && yarn install --frozen-lockfile)
+(cd lambdas && yarn workspace @aws-github-runner/webhook dist)
+(cd lambdas && yarn workspace @aws-github-runner/control-plane dist)
+sh tests/ministack/run-smoke.sh
+```
+
+The smoke script generates a temporary RSA key and Terraform variables file,
+starts the MockServer container on a temporary port, and removes all temporary
+state during cleanup. In CI, the pinned MockServer setup action starts the
+server and waits for readiness; the expectations are loaded after checkout.
+MiniStack must be able to reach
+`host.docker.internal`;
+override the hostname with `MINISTACK_GITHUB_MOCK_HOST` when using a different
+container runtime. When MiniStack is exposed on a non-default local port, use a
+host address reachable from its container for `AWS_ENDPOINT_URL`, for example
+`AWS_ENDPOINT_URL=http://<host-ip>:14568`, instead of `127.0.0.1`.

tests/ministack/github-api-expectations.json

@@ -0,0 +1,77 @@
+[
+  {
+    "httpRequest": {
+      "method": "GET",
+      "path": "/api/v3/repos/test-owner/test-repo/actions/jobs/123456"
+    },
+    "httpResponse": {
+      "statusCode": 200,
+      "headers": {
+        "Content-Type": ["application/json"],
+        "X-RateLimit-Limit": ["5000"],
+        "X-RateLimit-Remaining": ["4999"]
+      },
+      "body": "{\"id\":123456,\"status\":\"queued\",\"name\":\"ministack-smoke\"}"
+    }
+  },
+  {
+    "httpRequest": {
+      "method": "GET",
+      "path": "/api/v3/repos/test-owner/test-repo/actions/jobs/123457"
+    },
+    "httpResponse": {
+      "statusCode": 200,
+      "headers": {
+        "Content-Type": ["application/json"],
+        "X-RateLimit-Limit": ["5000"],
+        "X-RateLimit-Remaining": ["4999"]
+      },
+      "body": "{\"id\":123457,\"status\":\"queued\",\"name\":\"ministack-smoke-dynamic\"}"
+    }
+  },
+  {
+    "httpRequest": {
+      "method": "POST",
+      "path": "/api/v3/app/installations/123/access_tokens"
+    },
+    "httpResponse": {
+      "statusCode": 201,
+      "headers": {
+        "Content-Type": ["application/json"],
+        "X-RateLimit-Limit": ["5000"],
+        "X-RateLimit-Remaining": ["4999"]
+      },
+      "body": "{\"token\":\"ministack-installation-token\",\"expires_at\":\"2099-01-01T00:00:00Z\"}"
+    }
+  },
+  {
+    "httpRequest": {
+      "method": "GET",
+      "path": "/api/v3/orgs/test-owner/installation"
+    },
+    "httpResponse": {
+      "statusCode": 200,
+      "headers": {
+        "Content-Type": ["application/json"],
+        "X-RateLimit-Limit": ["5000"],
+        "X-RateLimit-Remaining": ["4999"]
+      },
+      "body": "{\"id\":123}"
+    }
+  },
+  {
+    "httpRequest": {
+      "method": "POST",
+      "path": "/api/v3/orgs/test-owner/actions/runners/registration-token"
+    },
+    "httpResponse": {
+      "statusCode": 201,
+      "headers": {
+        "Content-Type": ["application/json"],
+        "X-RateLimit-Limit": ["5000"],
+        "X-RateLimit-Remaining": ["4999"]
+      },
+      "body": "{\"token\":\"ministack-registration-token\",\"expires_at\":\"2099-01-01T00:00:00Z\"}"
+    }
+  }
+]