diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6479819b4..0a76978be 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -202,8 +202,10 @@ MongoDB is CosmosDB-compatible — avoid MongoDB features that CosmosDB's MongoD (`.agents/skills/rust-best-practices/SKILL.md`). - **Databases**: keep MongoDB usage CosmosDB-compatible (see migrations above). - **Documentation is not optional**: if you add or change a component, API, data model, pattern, or - deployment behavior, update the relevant doc in [`docs/`](./docs/) (or add one and link it from the - README) as the last step before opening your PR. + deployment behavior, update the relevant technical doc in [`docs/`](./docs/) and user-facing + guide in [`website/src/content/docs/`](./website/src/content/docs/) as appropriate. Add new + website pages to the sidebar in [`website/astro.config.mjs`](./website/astro.config.mjs) + as the last step before opening your PR. ## Submitting a pull request @@ -424,6 +426,9 @@ installed dependencies. - Start with the [system architecture](./docs/architecture/system-architecture.md) and the [app design](./docs/architecture/app-design.md) docs for the big picture. -- The [README](./README.md) introduces the platform, provides a local quick start, and links the full documentation index. +- The [official documentation website](https://microsoft.github.io/scope/) includes + [local setup](https://microsoft.github.io/scope/getting-started/local-development/), + the [development guide](https://microsoft.github.io/scope/resources/development/), and + [contribution guidance](https://microsoft.github.io/scope/resources/contributing/). - Browse [open issues](https://github.com/microsoft/scope/issues) for bugs and proposed improvements. Discuss larger changes in an issue before starting work. diff --git a/README.md b/README.md index a449d61a5..2498cfd61 100644 --- a/README.md +++ b/README.md @@ -5,320 +5,22 @@

An open-source agentic experience evaluation platform.

- Evaluate how AI agents use your product, approach real tasks and respond to feedback. Compare accross surfaces (CLI, MCP, Skills, doc, ...), context, tasks, operating systems, ... + Evaluate how AI agents use your product, approach real tasks, and respond to feedback. Define success with reusable criteria, follow each run live, and inspect the evidence behind every result through the Portal or CLI.

- Node.js 22 - pnpm 10.29.1 - TypeScript MIT License

- -

- Why Scope | - How it works | - Get started | - Documentation | - Contributing -

-## Why Scope? - -A working result is only part of the agentic experience. Scope helps you -evaluate the output, the steps an agent took, and how it responded to feedback. -Use repeatable evaluations to understand both successes and failures. - -- **Define what success means.** Organize evaluation criteria into a directed - acyclic graph (DAG), with dependencies between checks. -- **Inspect the evidence.** Follow live logs and review generated files, - workspace snapshots, criteria results, and captured agent activity. -- **Evaluate changes.** Reuse tasks and saved profiles to understand how the - agent, model, skills, tools, or starting codebase affect the experience. -- **Automate evaluations.** Submit and manage runs from the CLI or REST API, - and explore results, reports, and insights in the Portal. - -Scope is for product managers, developers, researchers, and teams evaluating how their software is being used by coding agents. -Results describe the tasks and configurations you tested, not a universal agent -ranking. The automated Judge can make mistakes; important conclusions need human -review. - -## How it works - -```mermaid -flowchart LR - Client["Portal or CLI"] --> API["API"] - API --> DB[("MongoDB")] - Scheduler["Scheduler"] -->|claim pending runs| DB - Scheduler --> Queue["Storage Queues"] - Queue --> Worker["Coding agent worker"] - Worker --> Judge["Judge"] - Judge -->|criteria results| DB - Worker -->|logs| Redis["Redis"] - Redis -->|live events| API - Worker -->|artifacts| Blob["Blob Storage"] - Worker -->|run status| DB -``` - -1. **Define** a task, its evaluation criteria, and the agent configuration. -2. **Submit** a request through the Portal or CLI. The scheduler dispatches - pending runs to the appropriate worker queue. -3. **Execute and evaluate.** The worker runs the agent and asks the Judge to - evaluate its output. Runs can include multiple feedback iterations. -4. **Inspect and compare.** Review logs, snapshots, and criteria results. - Post-processing and report workers produce additional analysis when enabled. - -MongoDB holds evaluation configuration and run records. Blob Storage holds larger -artifacts, and Redis relays live events. Local development uses MongoDB, Redis, -Azurite (the Azure Storage emulator), and Lowkey Vault. See the -[system architecture](./docs/architecture/system-architecture.md) for service -details and production deployment considerations. - -### Coding agents - -| Worker | Integration | Getting started | -| --- | --- | --- | -| GitHub Copilot | Agent Client Protocol (ACP) | `pnpm docker:dev:copilot` | -| Claude Code | ACP | `pnpm docker:dev:claude-code` | -| GitHub Copilot on Windows | Windows ACP worker | Deployment-specific; see [system architecture](./docs/architecture/system-architecture.md) | - -The default local workflow below uses the Copilot worker. The -`pnpm docker:dev:all` command enables both local ACP workers, not every -deployment-specific integration. Each provider requires its own credentials -and access to the selected models. - -## Getting started - -### Prerequisites - -| Tool | Requirement | -| --- | --- | -| Git | Clone the repository; fork it first if you plan to contribute. | -| Node.js | Version 22, matching CI. | -| pnpm | Version 10.29.1, pinned in `package.json`. | -| Docker with Compose v2 | Run the local stack. Use a current version with Compose Watch support. | -| [mkcert](https://github.com/FiloSottile/mkcert#installation) | Create trusted HTTPS certificates for the local sign-in emulator. | -| [GitHub CLI](https://cli.github.com/) | Obtain a token for the Copilot quick start with `gh auth login`. | - -The Copilot worker requires an **active GitHub Copilot entitlement** on the -account supplying its token. Authenticating with `gh auth login` alone does not -grant Copilot access. You also need credentials with access to the models used -by the Judge and other AI features. Provider usage may incur charges or consume -quotas. The local backing services don't require an Azure subscription. - -The commands below use a Bash-compatible shell. Rust is only required on the -host if you build or modify the gateway outside Docker. - -### 1. Clone and install - -```bash -git clone https://github.com/microsoft/scope.git -cd scope -corepack enable -pnpm install --frozen-lockfile -``` - -If you cloned a fork, run the remaining commands from that checkout instead. - -### 2. Configure and start the stack - -Authenticate with an account that has an active Copilot entitlement: - -```bash -gh auth login -GITHUB_TOKEN="$(gh auth token)" pnpm docker:dev:copilot -``` - -This builds and starts the Copilot worker, Portal, API, scheduler, Judge, -token manager, gateway, post-processing and reporting services, and their local -dependencies. Database migrations and development agent registration run -automatically. The first build downloads several images and can take some time. - -The startup scripts also generate `.env` from [`.env.base`](./.env.base) and -configure local sign-in over HTTPS. Local authentication requires your browser -to trust the development certificate. The scripts run `mkcert -install` to add -a local certificate authority to the OS/browser trust store and generate the -emulator's `localhost` certificate. On first use, you may be prompted to approve -this trust-store change. See the -[local authentication instructions](./ENV_VARIABLES.md#local-dev-setup-entra-local). - -For persistent overrides, copy [`.env.local.example`](./.env.local.example) to -`.env.local` and edit it locally. Don't put credentials in `.env.base` or commit -them. Avoid editing the generated `.env`, which is regenerated by the scripts. -See the [environment reference](./ENV_VARIABLES.md) for provider credentials, -Judge models, and optional Azure AI Foundry configuration. - -### 3. Open the Portal - -In a second terminal, from the repository root: - -```bash -pnpm open:portal -``` - -The default address is `http://localhost:5100`. Git worktrees get their own port -assignments; `pnpm open:portal` resolves the correct address automatically. -Local sign-in uses the seeded emulator users, such as `alice@entralocal.dev`, -not a production Microsoft Entra tenant. See -[local authentication setup](./ENV_VARIABLES.md#local-dev-setup-entra-local). - -> **Local development is not a security sandbox.** The ACP worker configuration -> mounts the Docker socket so agents can run containers. Use a dedicated -> environment for untrusted tasks, and don't expose this development stack to -> the internet. - -## Run your first evaluation - -### From the Portal - -1. Select or create a project to keep your evaluation data together. -2. Create a task and at least one observable evaluation criterion. For example, - ask the agent to create a Node.js HTTP server and evaluate whether its source - implements a `GET /health` route returning JSON. -3. Open the run submission page, select the running Copilot agent and an - available model, and attach your task and criteria. You can save the agent - configuration as a reusable profile. -4. Submit the run, follow its live logs, and inspect the Judge's results and - workspace artifacts. - -If no models are available, check your provider access and the model-scanner -logs before submitting. A registered agent isn't necessarily running; choose -the worker enabled by your Compose command. - -### From the CLI - -Build the CLI and its shared dependency, then explore the available commands: - -```bash -pnpm build:cli -pnpm cli --help -pnpm cli project list -``` - -Select a project using the ID returned by `project list`: - -```bash -pnpm cli project use -pnpm cli criteria list -pnpm cli run submit --help -pnpm cli run list -``` - -Replace `` with an actual ID. Submission requires a selected project, -a task, and evaluation criteria. Use `--project ` or `SCOPE_PROJECT` -to select a project explicitly in automation. The CLI reads local port settings -from the generated `.env`; set `SCOPE_API_URL` to target another instance. - -### Evaluation building blocks - -| Concept | Purpose | -| --- | --- | -| Tasks and scenarios | Define the work the agent should perform. | -| Criteria | Define observable checks and dependencies for the Judge. | -| Personas | Configure the evaluation perspective and feedback style. | -| Profiles and variations | Save an agent configuration and compare changes against a baseline. | -| Skills and MCP servers | Provide agent instructions and tools through the Model Context Protocol. | -| Codebases | Seed a run with a versioned starting workspace. | - -The YAML files in [`config/`](./config/) are portable examples, not the live -configuration database. MongoDB is the runtime source of truth. Manage -configuration through the Portal or CLI; don't assume that editing an example -file changes an existing evaluation. - -## Development - -The repository is a pnpm workspaces monorepo, primarily TypeScript, with a React -Portal and a Rust AI gateway. - -| Path | Contents | -| --- | --- | -| [`apps/api/`](./apps/api/) | REST API and live event streaming | -| [`apps/portal/`](./apps/portal/) | Web UI and Storybook components | -| [`apps/cli/`](./apps/cli/) | CLI for evaluation management and automation | -| [`apps/scheduler/`](./apps/scheduler/) and [`apps/judge/`](./apps/judge/) | Run dispatch and criteria evaluation | -| [`apps/workers/`](./apps/workers/) | Coding-agent, post-processing, and report workers | -| [`apps/gateway/`](./apps/gateway/) and [`apps/token-manager/`](./apps/token-manager/) | AI traffic capture and credential management | -| [`packages/`](./packages/) | Shared types, storage clients, migrations, and supporting libraries | -| [`evaluations/`](./evaluations/) | Developer-run static prompt quality tooling | -| [`config/`](./config/) and [`docs/`](./docs/) | Evaluation examples and documentation | - -Useful commands from the repository root: - -```bash -pnpm test # Unit tests -pnpm lint # Workspace lint and type checks -pnpm build # Workspace builds -pnpm storybook # Portal component catalog -pnpm test:integration # Integration tests; requires .env and backing services -``` - -Evaluate Scope's own AI prompts with `pnpm eval:prompts -- --mode quality` -(the default mode). See the [prompt evaluation guide](./docs/architecture/prompt-evaluations.md) -for setup, model credentials, and offline validation. - -For service-by-service development, Rust commands, migrations, and code -conventions, read [CONTRIBUTING.md](./CONTRIBUTING.md). - ## Documentation -| Topic | Guide | -| --- | --- | -| Architecture and run lifecycle | [System architecture](./docs/architecture/system-architecture.md) | -| Domain models and API design | [Application design](./docs/architecture/app-design.md) | -| Project organization | [Projects](./docs/architecture/data-organization-projects.md) | -| Evaluation and criteria DAGs | [Criteria provider](./docs/architecture/criteria-provider.md) | -| Static prompt quality | [Prompt evaluations](./docs/architecture/prompt-evaluations.md) | -| Agent context | [Skills](./docs/architecture/skills.md) and [codebases](./docs/architecture/codebases.md) | -| Scheduling and recovery | [Queue scheduler](./docs/architecture/queue-scheduler.md) | -| Configuration and authentication | [Environment variables](./ENV_VARIABLES.md) | -| AI limitations and data handling | [Responsible AI FAQ](./docs/responsible-ai-faq.md) | -| More architecture, operations, and research | [Documentation index](./docs/README.md) | - -## Contributing - -Contributions aren't limited to new workers. Documentation improvements, -reproducible bug reports, evaluation examples, tests, and accessibility fixes -are useful ways to get involved. - -Search [existing issues](https://github.com/microsoft/scope/issues) before -reporting a bug or proposing a feature. For larger changes, open an issue to -discuss the approach before implementation. Include reproduction steps and -relevant versions in bug reports, and remove credentials and sensitive run -content from logs. - -Read the [contribution guide](./CONTRIBUTING.md) for setup, conventions, and the -pull request process. Open pull requests against `microsoft/scope` on `main`, -including when working from a fork. Most contributions require the Microsoft -Contributor License Agreement; the CLA bot will guide you through it. - -Everyone participating in the project is expected to follow the -[Code of Conduct](./CODE_OF_CONDUCT.md). - -## Support and security - -For usage questions, bugs, and feature requests, see [SUPPORT.md](./SUPPORT.md). -Report vulnerabilities privately through [SECURITY.md](./SECURITY.md), never -through a public GitHub issue. - -Evaluation artifacts can contain prompts, source code, tool output, and network -metadata. Only use data and credentials approved for your deployment, configure -appropriate access controls, and review generated code before using it. Scope -doesn't certify that an agent or its output is safe or production-ready. Read -the [Responsible AI FAQ](./docs/responsible-ai-faq.md) before running sensitive -or untrusted workloads. - -## License and trademarks - -Scope is licensed under the [MIT License](./LICENSE). Third-party attributions -are in [`NOTICE`](./NOTICE); contributors changing dependencies should follow -the [notice maintenance instructions](./CONTRIBUTING.md#third-party-notices). +Users and contributors, visit the **[official documentation website](https://microsoft.github.io/scope/)** +for setup instructions, evaluation workflows, architecture, and reference guides. -This project may contain trademarks or logos for projects, products, or -services. Authorized use of Microsoft trademarks or logos must follow -[Microsoft's Trademark and Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). -Use of Microsoft trademarks or logos in modified versions of this project -must not cause confusion or imply Microsoft sponsorship. Any use of -third-party trademarks or logos is subject to those parties' policies. +Start with [Getting started](https://microsoft.github.io/scope/getting-started/access/) +or [Local development](https://microsoft.github.io/scope/getting-started/local-development/). +To help improve Scope, read the [contributor guide](https://microsoft.github.io/scope/resources/contributing/). +For help or vulnerability reporting, see [Support and security](https://microsoft.github.io/scope/resources/support/). diff --git a/apps/cli/src/utils/update-check.test.ts b/apps/cli/src/utils/update-check.test.ts index 2c891c140..32bc5dccf 100644 --- a/apps/cli/src/utils/update-check.test.ts +++ b/apps/cli/src/utils/update-check.test.ts @@ -44,4 +44,22 @@ describe("public CLI release lookup", () => { expect(execSync).toHaveBeenCalledWith(expect.stringContaining("gh release list --repo microsoft/scope"), expect.anything()); expect(fetchMock).not.toHaveBeenCalled(); }); + + it("preserves a custom releases URL without invoking gh", async () => { + vi.stubEnv("SCOPE_RELEASES_URL", "http://localhost:9999/releases"); + const fetchMock = vi.fn().mockResolvedValue(Response.json([{ tag_name: "cli/v2.0.0" }])); + vi.stubGlobal("fetch", fetchMock); + const { fetchLatestVersion } = await import("./update-check.js"); + expect(await fetchLatestVersion()).toBe("2.0.0"); + expect(execSync).not.toHaveBeenCalled(); + expect(fetchMock).toHaveBeenCalledWith("http://localhost:9999/releases", expect.anything()); + }); + + it("returns no version when no CLI release has been published", async () => { + vi.mocked(execSync).mockReturnValue(""); + const fetchMock = vi.fn().mockResolvedValue(Response.json([{ tag_name: "other/v9.0.0" }])); + vi.stubGlobal("fetch", fetchMock); + const { fetchLatestVersion } = await import("./update-check.js"); + expect(await fetchLatestVersion()).toBeUndefined(); + }); }); diff --git a/website/AGENTS.md b/website/AGENTS.md index cb56d354a..b763efcd4 100644 --- a/website/AGENTS.md +++ b/website/AGENTS.md @@ -3,17 +3,20 @@ Project context for AI coding agents working on **scope-doc**, the Starlight-based documentation site for **Scope**. -## What this repo is +## What this directory is -A static documentation site published to GitHub Pages. +A static documentation site published to GitHub Pages, within the +[microsoft/scope](https://github.com/microsoft/scope) monorepo. - Framework: **Astro 6.x** + **@astrojs/starlight** - Package manager: **pnpm** (pinned via `packageManager` in `package.json`) - TypeScript strict - Lives under the `website/` directory of the repo (all site sources, config, and `package.json` are rooted here; run every command from - `website/`) -- Deployed by `.github/workflows/static.yml` (build + deploy jobs), + `website/` unless explicitly directed to the repository root). + The site has its own lockfile and is not part of the root pnpm workspace. +- Deployed by [../.github/workflows/static.yml](../.github/workflows/static.yml) + (build + deploy jobs), which builds from `website/` via a `working-directory` default and `website/**` path filters; both PR and production builds explicitly use `SITE=https://microsoft.github.io` and `BASE_PATH=/scope`. @@ -46,41 +49,40 @@ A static documentation site published to GitHub Pages. `starlight-openapi` config - `dist/` — build output (gitignored) -## The source of truth: scope-core +## The source of truth: microsoft/scope -The product itself lives in the -[scope-core](https://github.com/growth-ecosystems/scope-core) -repository. When writing docs, **read scope-core before writing any -factual claim**. Everything in the documentation \u2014 endpoints, field -names, statuses, worker IDs, behaviors, defaults, error messages, -anything \u2014 MUST be grounded in the source code. Do not invent. If -the source doesn't say it, it doesn't go in the docs; ask the user -or leave it out. +The product and this documentation site live in this same +[microsoft/scope](https://github.com/microsoft/scope) checkout. When writing +docs, **read the local product source before writing any factual claim**. +Everything in the documentation, including endpoints, field names, +statuses, worker IDs, behaviors, defaults, and error messages, MUST be +grounded in the source code. Do not invent. If the source doesn't say it, +it doesn't go in the docs; ask the user or leave it out. -## Where to look in scope-core +## Where to look in this monorepo -Anything that comes from the source — endpoints, field names, -status enums, defaults, validation rules — must be read from -scope-core at the time you write it, not copied from this file. +Read endpoints, field names, status enums, defaults, and validation rules +from this checkout at the time you write, not from this file. Use this map as a starting point; do not treat it as a substitute -for opening the file. +for opening the file. Paths in the table are relative to the repository +root, one directory above `website/`; site paths elsewhere in this guide +are relative to `website/`. -| Topic | File(s) in scope-core | +| Topic | File(s) from the repository root | | --- | --- | -| Workers (allowed IDs, validation) | `packages/shared/src/schemas/request.ts` (`VALID_WORKERS`) | -| Worker display names / labels | the `"name"` field in each worker's agent registration (upsert) payload (e.g. "GitHub Copilot CLI", "Claude Code CLI", "VS Code Copilot") | -| Worker software stacks (pre-installed tools) | `apps/workers/*/src/test-worker.ts` — the `checkTools([...])` array lists every runtime and build tool baked into the container image | -| Request payload, scenario shape | `packages/shared/src/schemas/request.ts` (`CreateRequestInputSchema`, `ScenarioSchema`) | -| Request status / outcome enums | `packages/shared/src/schemas/request.ts` (`RequestStatusSchema`, `RequestOutcomeSchema`) | -| Request lifecycle / scheduler | `apps/api/src/index.ts`, `docs/architecture/queue-scheduler.md` | -| Profile + version schemas | `packages/shared/src/schemas/profile.ts` | -| Criterion schema, DAG rules | `packages/shared/src/schemas/criteria.ts` | -| Route handlers, validation, error codes | `apps/api/src/routes/*.ts` | -| VS Code worker behavior | `docs/architecture/vscode-electron-worker.md`, `vscode-web-worker.md`, `worker-requirements.md` | -| OpenAPI source | `apps/api/src/openapi/registry.ts`; generated snapshot at `src/openapi/scope-openapi.json` | -| Swagger UI | served by the API; check `apps/api/src/index.ts` for the route | - -When in doubt, `grep` scope-core for the symbol or string before +| Agent IDs, names, and registration validation | [packages/shared/src/schemas/agent.ts](../packages/shared/src/schemas/agent.ts), [apps/api/src/routes/agents.ts](../apps/api/src/routes/agents.ts) | +| Worker software stacks (pre-installed tools) | `apps/workers/*/src/test-worker.ts`: the `checkTools([...])` arrays list the tools checked by each worker's smoke test | +| Request payload, scenario shape | [packages/shared/src/schemas/request.ts](../packages/shared/src/schemas/request.ts) (`CreateRequestInputSchema`), [packages/shared/src/schemas/scenario.ts](../packages/shared/src/schemas/scenario.ts) (`ScenarioSchema`) | +| Request status / outcome enums | [packages/shared/src/schemas/request.ts](../packages/shared/src/schemas/request.ts) (`RequestStatusSchema`, `RequestOutcomeSchema`) | +| Request lifecycle / scheduler | [apps/api/src/index.ts](../apps/api/src/index.ts), [apps/scheduler/src/request-scheduler.ts](../apps/scheduler/src/request-scheduler.ts), [docs/architecture/queue-scheduler.md](../docs/architecture/queue-scheduler.md) | +| Profile + version schemas | [packages/shared/src/schemas/profile.ts](../packages/shared/src/schemas/profile.ts) | +| Criterion schema, DAG rules | [packages/shared/src/schemas/criteria.ts](../packages/shared/src/schemas/criteria.ts) | +| Route handlers, validation, error codes | [apps/api/src/routes/](../apps/api/src/routes/) | +| Coding worker behavior | [apps/workers/](../apps/workers/), [docs/architecture/worker-requirements.md](../docs/architecture/worker-requirements.md) | +| OpenAPI source | [apps/api/src/openapi/registry.ts](../apps/api/src/openapi/registry.ts); generated snapshot at [website/src/openapi/scope-openapi.json](src/openapi/scope-openapi.json) | +| Swagger UI | Served by the API; check [apps/api/src/index.ts](../apps/api/src/index.ts) for the route | + +When in doubt, search this checkout for the symbol or string before writing anything in the docs. ## Standing user rulings @@ -223,9 +225,14 @@ pnpm install pnpm test # plugin regressions, using Node's built-in test runner pnpm run build # writes dist/ pnpm run dev # local preview at http://localhost:4321 -pnpm run refresh:openapi # generate the OpenAPI snapshot from scope-core +pnpm run refresh:openapi # generate the OpenAPI snapshot from this monorepo's API ``` +For `refresh:openapi`, install the root workspace dependencies first. +The script runs `pnpm --filter api generate:openapi` from the repository +root, using [apps/api/src/openapi/generate.ts](../apps/api/src/openapi/generate.ts) +to update [src/openapi/scope-openapi.json](src/openapi/scope-openapi.json). + Both `pnpm test` and `pnpm run build` must pass. The public build uses `SITE=https://microsoft.github.io BASE_PATH=/scope pnpm run build`; exercise that configuration when changing links or deployment settings, @@ -236,12 +243,12 @@ page count can indicate a content collection file failed to parse. ## Workflow - Each logical change is its own commit. Push to refresh the PR; CI - rebuilds and redeploys to GH Pages. + tests and builds the site. Deployment to GitHub Pages runs on `main`. - `main` is protected — open a PR, don't push directly. ## When in doubt -1. Read the relevant file in scope-core (schemas first, then route +1. Read the relevant file in this checkout (schemas first, then route handlers). 2. Generate the OpenAPI snapshot with `pnpm run refresh:openapi` and inspect `src/openapi/scope-openapi.json`. diff --git a/website/README.md b/website/README.md index c82223ff7..9ed792662 100644 --- a/website/README.md +++ b/website/README.md @@ -7,7 +7,16 @@ End-user documentation site for **Scope**, built with and published to GitHub Pages. The product and this documentation site live in -[microsoft/scope](https://github.com/microsoft/scope). +[microsoft/scope](https://github.com/microsoft/scope). Users and contributors +should start at the [official documentation website](https://microsoft.github.io/scope/). + +The root README is a short introduction and entry point to the website. +Keep detailed setup, usage, and contribution guidance in +[src/content/docs/](src/content/docs/) and register new pages in the sidebar. +Local setup lives in +[getting-started/local-development.md](src/content/docs/getting-started/local-development.md); +development, contribution, and support guidance live under +[resources/](src/content/docs/resources/). ## Project structure @@ -45,6 +54,10 @@ Sidebar order is defined in `astro.config.mjs`, not by directory order. ## Commands +Run these commands from `website/`. The site has its own +[package.json](package.json) and [pnpm-lock.yaml](pnpm-lock.yaml), separate +from the root pnpm workspace. + | Command | Action | | :--------------------- | :--------------------------------------------------------- | | `pnpm install` | Install dependencies | @@ -52,7 +65,7 @@ Sidebar order is defined in `astro.config.mjs`, not by directory order. | `pnpm build` | Build the production site to `./dist/` | | `pnpm preview` | Preview the production build locally | | `pnpm test` | Test site plugins with Node's built-in test runner | -| `pnpm refresh:openapi` | Generate the OpenAPI snapshot from `scope-core` | +| `pnpm refresh:openapi` | Generate the OpenAPI snapshot from this monorepo's API | ## Authoring docs @@ -76,17 +89,19 @@ Sidebar order is defined in `astro.config.mjs`, not by directory order. [`starlight-openapi`](https://starlight-openapi.vercel.app/) — do not edit them by hand. - Run `pnpm refresh:openapi` from this directory after changing API - routes or schemas. It runs `apps/api`'s generator from the same - `scope-core` checkout, so root workspace dependencies must be - installed first. + routes or schemas. It runs `pnpm --filter api generate:openapi` from the + repository root, using [apps/api/src/openapi/generate.ts](../apps/api/src/openapi/generate.ts) + in this same `microsoft/scope` checkout. Install the root workspace + dependencies first; the generator updates + [src/openapi/scope-openapi.json](src/openapi/scope-openapi.json). - To list a new article or talk on the Community page, add one YAML file under `src/content/articles/` or `src/content/talks/`. See "Articles & talks" in [AGENTS.md](AGENTS.md) for the fields. See [AGENTS.md](AGENTS.md) for conventions, the source-of-truth -policy (everything factual must be grounded in scope-core), and -where to look in scope-core for any given topic. +policy (everything factual must be grounded in this checkout's source), +and where to look in the monorepo for any given topic. ## Deployment diff --git a/website/astro.config.mjs b/website/astro.config.mjs index df3d89cb4..15dc99071 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -105,6 +105,7 @@ export default defineConfig({ label: 'Getting Started', items: [ { label: 'Access', slug: 'getting-started/access' }, + { label: 'Local development', slug: 'getting-started/local-development' }, { label: 'Install the CLI', slug: 'getting-started/install-cli' }, { label: 'Your first run', slug: 'getting-started/first-run' }, ], @@ -158,10 +159,18 @@ export default defineConfig({ ], }, ...openAPISidebarGroups, + { + label: 'Contributing', + items: [ + { label: 'Contributor guide', slug: 'resources/contributing' }, + { label: 'Development guide', slug: 'resources/development' }, + ], + }, { label: 'Resources', collapsed: true, items: [ + { label: 'Support and security', slug: 'resources/support' }, { label: 'FAQ', slug: 'resources/faq' }, { label: 'Data collection and privacy', slug: 'resources/data-collection' }, { label: 'Troubleshooting', slug: 'resources/troubleshooting' }, diff --git a/website/src/components/Home.astro b/website/src/components/Home.astro index c99371ce7..f844ab69c 100644 --- a/website/src/components/Home.astro +++ b/website/src/components/Home.astro @@ -129,6 +129,7 @@ const paths = [ ))}
Already running experiments?TroubleshootingCommon questionsData & privacy
+
Running your own stack or contributing?Local developmentContributingSupport & security
diff --git a/website/src/content/docs/getting-started/access.md b/website/src/content/docs/getting-started/access.md index cf2a0f961..d0364f9cb 100644 --- a/website/src/content/docs/getting-started/access.md +++ b/website/src/content/docs/getting-started/access.md @@ -6,6 +6,10 @@ description: How to reach the Scope Portal. You can access Scope through the web Portal, the REST API, or the `scope` CLI. +This page assumes you have access to an existing deployment. To run Scope +yourself, follow [Local development](/getting-started/local-development/) for +prerequisites, setup, authentication, and a first evaluation. + ## Requirements Your deployment administrator provides the Portal URL and any required diff --git a/website/src/content/docs/getting-started/first-run.md b/website/src/content/docs/getting-started/first-run.md index 714c83335..4bf233d00 100644 --- a/website/src/content/docs/getting-started/first-run.md +++ b/website/src/content/docs/getting-started/first-run.md @@ -14,18 +14,22 @@ profile). Scope creates one **run** per execution attempt — see ## Before you start - You can reach the [Portal](/getting-started/access/) for your - deployment. -- You don't need to set anything up — the catalog ships with example - task prompts, criteria sets, and a default Copilot setup. + deployment, or have followed + [Local development](/getting-started/local-development/). +- Your deployment has a running Copilot worker and access to an available + model. If the catalog doesn't contain suitable task prompts, criteria, or + profiles, create them as part of the steps below. ## 1. Open the Portal Navigate to the Portal URL provided for your deployment, for example **https://your-scope.example.com**. -The home view lists recent runs across all users. The left navigation -takes you to runs, tasks, criteria, profiles, prompt features, and -more. +Select or create a project to keep your evaluation data together. + +The Portal then opens **Statistics** (`/statistics`), which summarizes +benchmark results for the selected project. The left navigation takes you +to runs, tasks, criteria, profiles, prompt features, and more. ## 2. Start a new request diff --git a/website/src/content/docs/getting-started/install-cli.md b/website/src/content/docs/getting-started/install-cli.md index 2d05af5a7..af42cfb60 100644 --- a/website/src/content/docs/getting-started/install-cli.md +++ b/website/src/content/docs/getting-started/install-cli.md @@ -34,7 +34,9 @@ curl --fail --location https://raw.githubusercontent.com/microsoft/scope/main/in The installer checks the downloaded bundle's version before replacing an existing installation. Missing releases, download errors, and version mismatches fail without replacing your installed CLI. If no public CLI -release has been published yet, the installer reports that explicitly. +release has been published yet, the installer reports that explicitly; +follow [Local development](/getting-started/local-development/) to build +the CLI from source instead. ## Add to PATH diff --git a/website/src/content/docs/getting-started/local-development.md b/website/src/content/docs/getting-started/local-development.md new file mode 100644 index 000000000..2565dd83c --- /dev/null +++ b/website/src/content/docs/getting-started/local-development.md @@ -0,0 +1,162 @@ +--- +title: Local development +description: Set up Scope locally and run your first evaluation with the Portal or CLI. +--- + +Use this guide to run your own local Scope stack. If your organization already +provides a deployment, start with [Access](/getting-started/access/) instead. +The default local workflow uses the GitHub Copilot worker. + +## Prerequisites + +| Tool | Requirement | +| --- | --- | +| Git | Clone the repository; fork it first if you plan to contribute. | +| [Node.js](https://nodejs.org/) | Version 22, matching CI. | +| [pnpm](https://pnpm.io/) | Version 10.29.1, pinned in [package.json](https://github.com/microsoft/scope/blob/main/package.json). | +| Docker with Compose v2 | Run the local stack. Use a current version with Compose Watch support. | +| [mkcert](https://github.com/FiloSottile/mkcert#installation) | Create trusted HTTPS certificates for the local sign-in emulator. | +| [GitHub CLI](https://cli.github.com/) | Obtain a token for the Copilot quick start with `gh auth login`. | + +The Copilot worker requires an **active GitHub Copilot entitlement** on the +account supplying its token. Authenticating with `gh auth login` alone does not +grant Copilot access. You also need credentials with access to the models used +by the Judge and other AI features. Provider usage may incur charges or consume +quotas. The local backing services don't require an Azure subscription. + +The commands below use a Bash-compatible shell. Rust is only required on the +host if you build or modify the gateway outside Docker. + +## 1. Clone and install + +```bash +git clone https://github.com/microsoft/scope.git +cd scope +corepack enable +pnpm install --frozen-lockfile +``` + +If you cloned a fork, run the remaining commands from that checkout instead. + +## 2. Configure and start the stack + +Authenticate with an account that has an active Copilot entitlement: + +```bash +gh auth login +GITHUB_TOKEN="$(gh auth token)" pnpm docker:dev:copilot +``` + +This builds and starts the Copilot worker, Portal, API, scheduler, Judge, +token manager, gateway, post-processing and reporting services, and their local +dependencies. Database migrations and development agent registration run +automatically. The first build downloads several images and can take some time. + +The startup scripts generate the local environment file from +[.env.base](https://github.com/microsoft/scope/blob/main/.env.base) and configure +local sign-in over HTTPS. Local authentication requires your browser to trust +the development certificate. The scripts run `mkcert -install` to add a local +certificate authority to the OS/browser trust store and generate the +emulator's `localhost` certificate. On first use, you may be prompted to approve +this trust-store change. See the +[local authentication instructions](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md#local-dev-setup-entra-local). + +For persistent overrides, copy +[.env.local.example](https://github.com/microsoft/scope/blob/main/.env.local.example) +to a file named `.env.local` and edit it locally. Don't put credentials in +[.env.base](https://github.com/microsoft/scope/blob/main/.env.base) or commit +them. Avoid editing the generated `.env`, which is regenerated by the scripts. +See the +[environment reference](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md) +for provider credentials, Judge models, and optional Azure AI Foundry +configuration. + +## 3. Open the Portal + +In a second terminal, from the repository root: + +```bash +pnpm open:portal +``` + +The default address is `http://localhost:5100`. Git worktrees get their own port +assignments; `pnpm open:portal` resolves the correct address automatically. +Local sign-in uses the seeded emulator users, such as `alice@entralocal.dev`, +not a production Microsoft Entra tenant. See +[local authentication setup](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md#local-dev-setup-entra-local). + +:::caution[Local development is not a security sandbox] +The ACP worker configuration mounts the Docker socket so agents can run +containers. Use a dedicated environment for untrusted tasks, and don't expose +this development stack to the internet. Read +[Support and security](/resources/support/) before running sensitive workloads. +::: + +## Run your first evaluation + +### From the Portal + +1. Select or create a project to keep your evaluation data together. +2. Create a task and at least one observable evaluation criterion. For example, + ask the agent to create a Node.js HTTP server and evaluate whether its source + implements a `GET /health` route returning JSON. +3. Open the run submission page, select the running Copilot agent and an + available model, and attach your task and criteria. You can save the agent + configuration as a reusable profile. +4. Submit the request, follow the run's live logs, and inspect the Judge's + results and workspace artifacts. + +If no models are available, check your provider access and the model-scanner +logs before submitting. A registered agent isn't necessarily running; choose +the worker enabled by your Compose command. See +[Submitting requests (Portal)](/guides/submitting-requests-portal/) for the +full workflow. + +### From the CLI + +Build the CLI and its shared dependency, then point it at your local API. The +CLI has no default API URL, so set `SCOPE_API_URL` explicitly using the +`API_PORT` value from the generated `.env` (`3100` unless your worktree +offsets it): + +```bash +pnpm build:cli +export SCOPE_API_URL=http://localhost:3100 +pnpm cli --help +pnpm cli project list +``` + +Select a project using the ID returned by `project list`: + +```bash +pnpm cli project use +pnpm cli criteria list +pnpm cli run submit --help +pnpm cli run list +``` + +Replace `` with an actual ID. Submission requires a selected +project, a task, and evaluation criteria. Use `--project ` or +`SCOPE_PROJECT` to select a project explicitly in automation. Set +`SCOPE_API_URL` or pass `-u ` to target another instance. See +[Submitting requests (CLI)](/guides/submitting-requests-cli/) for submission +examples. + +## Coding agents + +| Worker | Integration | Getting started | +| --- | --- | --- | +| GitHub Copilot | Agent Client Protocol (ACP) | `pnpm docker:dev:copilot` | +| Claude Code | ACP | `pnpm docker:dev:claude-code` | +| GitHub Copilot on Windows | Windows ACP worker | Deployment-specific; see [system architecture](https://github.com/microsoft/scope/blob/main/docs/architecture/system-architecture.md). | + +The `pnpm docker:dev:all` command enables both local ACP workers, not every +deployment-specific integration. Each provider requires its own credentials +and access to the selected models. + +## Next steps + +Read [Concepts](/introduction/concepts/) for the evaluation building blocks, +[Development guide](/resources/development/) for repository layout and +commands, and [Contributing](/resources/contributing/) for how to propose +changes. diff --git a/website/src/content/docs/introduction/concepts.md b/website/src/content/docs/introduction/concepts.md index be2397235..f314943c0 100644 --- a/website/src/content/docs/introduction/concepts.md +++ b/website/src/content/docs/introduction/concepts.md @@ -116,3 +116,20 @@ Optional capabilities that extend what an agent can do during a run: All three are properties of a profile. See [Using MCP servers, skills & extensions](/guides/mcp-skills-extensions/). + +## Evaluation building blocks + +| Concept | Purpose | +| --- | --- | +| Tasks and scenarios | Define the work the agent should perform. | +| Criteria | Define observable checks and dependencies in the Judge's criteria DAG. | +| Personas | Configure the evaluation perspective and feedback style. | +| Profiles and variations | Save an agent configuration and compare changes against a baseline. | +| Skills and MCP servers | Provide agent instructions and tools through the Model Context Protocol. | +| Codebases | Seed a run with a versioned starting workspace. | + +The YAML files in +[config/](https://github.com/microsoft/scope/tree/main/config) are portable +examples, not the live configuration database. MongoDB is the runtime source +of truth. Manage configuration through the Portal or CLI; don't assume that +editing an example file changes an existing evaluation. diff --git a/website/src/content/docs/introduction/what-is-scope.md b/website/src/content/docs/introduction/what-is-scope.md index f79530253..e1b6f0b0e 100644 --- a/website/src/content/docs/introduction/what-is-scope.md +++ b/website/src/content/docs/introduction/what-is-scope.md @@ -28,6 +28,11 @@ swap the agent, the model, or the surrounding tools. their parents passing — use it for multi-step evaluation, or just leave dependencies off and every criterion becomes a root. - **Watch runs in real time** as logs stream from the worker. +- **Inspect the evidence.** Review generated files, workspace snapshots, + criteria results, captured agent activity, reports, and insights. +- **Evaluate changes.** Reuse tasks, profiles, and versioned starting codebases + to compare surfaces such as CLIs, MCP servers, skills, and documentation, + as well as models, context, tasks, and operating systems. - **Compare across heterogeneous tasks** using prompt features — characteristics Scope detects on your task prompt (e.g. "asks for an API", "asks for TypeScript") so you can ask questions like *"how @@ -36,6 +41,31 @@ swap the agent, the model, or the surrounding tools. - **Automate everything** through the REST API or the `scope` CLI — submit runs, manage profiles and criteria, fetch results. +## How it works + +1. **Define** a task, its evaluation criteria, and the agent configuration. +2. **Submit** a request through the Portal or CLI. The API stores the request + in MongoDB. The scheduler claims pending runs and dispatches them to the + appropriate storage queue. +3. **Execute and evaluate.** The worker runs the agent and asks the Judge to + evaluate its output. Runs can include multiple feedback iterations. +4. **Inspect and compare.** Review logs, snapshots, and criteria results. + Post-processing and report workers produce additional analysis when enabled. + +| Service | Role in a run | +| --- | --- | +| MongoDB | Stores evaluation configuration, run records, worker status updates, and the Judge's criteria results. | +| Storage Queues | Deliver work from the scheduler to coding-agent workers. | +| Redis | Relays worker logs and live events to the API for Portal and CLI clients. | +| Blob Storage | Holds larger artifacts, including workspace snapshots written by workers. | + +Local development uses MongoDB, Redis, Azurite (the Azure Storage emulator), +and Lowkey Vault. See +[Local development](/getting-started/local-development/) to run the stack, or +the +[system architecture](https://github.com/microsoft/scope/blob/main/docs/architecture/system-architecture.md) +for service details and production deployment considerations. + ## When Scope is the right tool Use Scope when you want to: @@ -58,15 +88,31 @@ against tasks you control. Scope is for teams that need to measure and compare AI coding agents: +- **Product managers** evaluating how coding agents use their software and + respond to feedback. - **Engineers** designing task prompts and criteria to characterize the agentic experience. - **Researchers** comparing agent trajectories across diverse tasks. - **Pipelines and tooling** that submit runs programmatically via the REST API. +## Interpreting results + +A working result is only part of the agentic experience. Use repeatable +evaluations to understand both successes and failures, including the steps an +agent took and how it responded to feedback. + +Results describe the tasks and configurations you tested, not a universal +agent ranking. The automated Judge can make mistakes; important conclusions +need human review. See [Support and security](/resources/support/) for +responsible-use guidance. + ## Where to next - New here? Read [Concepts](/introduction/concepts/) to get familiar with the vocabulary. - Ready to submit your first run? Jump to [Access](/getting-started/access/). +- Want to run Scope locally? Follow + [Local development](/getting-started/local-development/). +- Interested in improving Scope? Read [Contributing](/resources/contributing/). diff --git a/website/src/content/docs/resources/contributing.md b/website/src/content/docs/resources/contributing.md new file mode 100644 index 000000000..4662aab1d --- /dev/null +++ b/website/src/content/docs/resources/contributing.md @@ -0,0 +1,58 @@ +--- +title: Contributing +description: How to contribute to Scope, propose changes, and follow the project's contribution and licensing policies. +--- + +Contributions aren't limited to new workers. Documentation improvements, +reproducible bug reports, evaluation examples, tests, and accessibility fixes +are useful ways to get involved. + +## Before you start + +Search [existing issues](https://github.com/microsoft/scope/issues) before +reporting a bug or proposing a feature. For larger changes, open an issue to +discuss the approach before implementation. Include reproduction steps and +relevant versions in bug reports, and remove credentials and sensitive run +content from logs. + +Set up your environment with +[Local development](/getting-started/local-development/), then use the +[Development guide](/resources/development/) to find the relevant components +and architecture references. Read +[CONTRIBUTING.md](https://github.com/microsoft/scope/blob/main/CONTRIBUTING.md) +for coding conventions, testing requirements, and the pull request process. + +## Pull requests + +Open pull requests against `microsoft/scope` on `main`, including when working +from a fork. Keep changes focused, include appropriate tests, and update +affected documentation. See +[Reviewing and merging community contributions](https://github.com/microsoft/scope/blob/main/CONTRIBUTING.md#reviewing-and-merging-community-contributions) +for what to expect during review, including acknowledgement and follow-up +timelines. + +Most contributions require the +[Microsoft Contributor License Agreement](https://cla.opensource.microsoft.com). +The CLA bot will determine whether you need to sign and guide you through it. + +Everyone participating in the project is expected to follow the +[Code of Conduct](https://github.com/microsoft/scope/blob/main/CODE_OF_CONDUCT.md). + +For usage questions and private vulnerability reporting, see +[Support and security](/resources/support/). + +## License and trademarks + +Scope is licensed under the +[MIT License](https://github.com/microsoft/scope/blob/main/LICENSE). +Third-party attributions are in +[NOTICE](https://github.com/microsoft/scope/blob/main/NOTICE); contributors +changing dependencies should follow the +[notice maintenance instructions](https://github.com/microsoft/scope/blob/main/CONTRIBUTING.md#third-party-notices). + +This project may contain trademarks or logos for projects, products, or +services. Authorized use of Microsoft trademarks or logos must follow +[Microsoft's Trademark and Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). +Use of Microsoft trademarks or logos in modified versions of this project +must not cause confusion or imply Microsoft sponsorship. Any use of +third-party trademarks or logos is subject to those parties' policies. diff --git a/website/src/content/docs/resources/development.md b/website/src/content/docs/resources/development.md new file mode 100644 index 000000000..c20c1e64d --- /dev/null +++ b/website/src/content/docs/resources/development.md @@ -0,0 +1,65 @@ +--- +title: Development guide +description: Repository layout, development commands, and architecture references for Scope contributors. +--- + +The repository is a pnpm workspaces monorepo, primarily TypeScript 5, with a +React Portal and a Rust AI gateway. Start with +[Local development](/getting-started/local-development/) to install +dependencies and run the stack, then read +[Contributing](/resources/contributing/) before opening a pull request. + +## Repository structure + +| Path | Contents | +| --- | --- | +| [apps/api/](https://github.com/microsoft/scope/tree/main/apps/api) | REST API and live event streaming | +| [apps/portal/](https://github.com/microsoft/scope/tree/main/apps/portal) | Web UI and Storybook components | +| [apps/cli/](https://github.com/microsoft/scope/tree/main/apps/cli) | CLI for evaluation management and automation | +| [apps/scheduler/](https://github.com/microsoft/scope/tree/main/apps/scheduler) and [apps/judge/](https://github.com/microsoft/scope/tree/main/apps/judge) | Run dispatch and criteria evaluation | +| [apps/workers/](https://github.com/microsoft/scope/tree/main/apps/workers) | Coding-agent, post-processing, and report workers | +| [apps/gateway/](https://github.com/microsoft/scope/tree/main/apps/gateway) and [apps/token-manager/](https://github.com/microsoft/scope/tree/main/apps/token-manager) | AI traffic capture and credential management | +| [packages/](https://github.com/microsoft/scope/tree/main/packages) | Shared types, storage clients, migrations, and supporting libraries | +| [evaluations/](https://github.com/microsoft/scope/tree/main/evaluations) | Developer-run static prompt quality tooling | +| [config/](https://github.com/microsoft/scope/tree/main/config) and [docs/](https://github.com/microsoft/scope/tree/main/docs) | Evaluation examples and technical documentation | +| [website/](https://github.com/microsoft/scope/tree/main/website) | This documentation website | + +## Development commands + +Run these commands from the repository root: + +```bash +pnpm test # Unit tests +pnpm lint # Workspace lint and type checks +pnpm build # Workspace builds +pnpm storybook # Portal component catalog +pnpm test:integration # Integration tests; requires .env and backing services +``` + +Evaluate Scope's own AI prompts with `pnpm eval:prompts -- --mode quality` +(the default mode). See the +[prompt evaluation guide](https://github.com/microsoft/scope/blob/main/docs/architecture/prompt-evaluations.md) +for setup, model credentials, and offline validation. + +For service-by-service development, Rust commands, migrations, and code +conventions, read +[CONTRIBUTING.md](https://github.com/microsoft/scope/blob/main/CONTRIBUTING.md). +For documentation authoring and preview commands, see +[website/README.md](https://github.com/microsoft/scope/blob/main/website/README.md). + +## Architecture and configuration references + +These detailed engineering references live alongside the source code: + +| Topic | Guide | +| --- | --- | +| Architecture and run lifecycle | [System architecture](https://github.com/microsoft/scope/blob/main/docs/architecture/system-architecture.md) | +| Domain models and API design | [Application design](https://github.com/microsoft/scope/blob/main/docs/architecture/app-design.md) | +| Project organization | [Projects](https://github.com/microsoft/scope/blob/main/docs/architecture/data-organization-projects.md) | +| Evaluation and criteria DAGs | [Criteria provider](https://github.com/microsoft/scope/blob/main/docs/architecture/criteria-provider.md) | +| Static prompt quality | [Prompt evaluations](https://github.com/microsoft/scope/blob/main/docs/architecture/prompt-evaluations.md) | +| Agent context | [Skills](https://github.com/microsoft/scope/blob/main/docs/architecture/skills.md) and [codebases](https://github.com/microsoft/scope/blob/main/docs/architecture/codebases.md) | +| Scheduling and recovery | [Queue scheduler](https://github.com/microsoft/scope/blob/main/docs/architecture/queue-scheduler.md) | +| Configuration and authentication | [Environment variables](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md) | +| AI limitations and data handling | [Responsible AI FAQ](https://github.com/microsoft/scope/blob/main/docs/responsible-ai-faq.md) | +| More architecture, operations, and research | [Documentation index](https://github.com/microsoft/scope/blob/main/docs/README.md) | diff --git a/website/src/content/docs/resources/support.md b/website/src/content/docs/resources/support.md new file mode 100644 index 000000000..8ba16fef2 --- /dev/null +++ b/website/src/content/docs/resources/support.md @@ -0,0 +1,42 @@ +--- +title: Support and security +description: Get help, report vulnerabilities privately, and understand Scope's responsible-use requirements. +--- + +## Getting help + +For usage questions, bugs, and feature requests, see +[SUPPORT.md](https://github.com/microsoft/scope/blob/main/SUPPORT.md). +Search [existing issues](https://github.com/microsoft/scope/issues) before +opening a new issue. Include reproduction steps and relevant versions, but +remove credentials and sensitive run content from logs. + +## Reporting vulnerabilities + +Report vulnerabilities privately through +[SECURITY.md](https://github.com/microsoft/scope/blob/main/SECURITY.md), which +links to the current Microsoft security reporting guidance. Never report a +vulnerability through a public GitHub issue. + +## Responsible use + +Evaluation artifacts can contain prompts, source code, tool output, and +network metadata. Only use data and credentials approved for your deployment, +configure appropriate access controls, and review generated code before +using it. + +Scope doesn't certify that an agent or its output is safe or production-ready. +Results describe the tasks and configurations you tested, not a universal +agent ranking. The automated Judge can make mistakes; important conclusions +need human review. + +Local development is not a security sandbox. The ACP worker configuration +mounts the Docker socket so agents can run containers. Use a dedicated +environment for untrusted tasks, and don't expose the development stack to +the internet. + +Read the +[Responsible AI FAQ](https://github.com/microsoft/scope/blob/main/docs/responsible-ai-faq.md) +before running sensitive or untrusted workloads. See +[Data collection and privacy](/resources/data-collection/) for the data Scope +handles and the deployment operator's responsibilities.