Repository navigation
Conversation
Moves apps/agent-evals from the videojs/v10 worktree it was built in, where it was never committed, unchanged from v10 4ca4ee5. The next commit updates it for videojs/video.js and the releases since. Each trial gives Claude Code or Codex a realistic install request in a fresh Docker container, then builds, serves, and checks in a browser that a Video.js 10 player renders and plays. Runs pin npm, videojs.org, and shadcn.videojs.org to the build under test, so releases compare fairly.
The root .gitignore excludes lib/ and package-lock.json, which left out the harness's exec helper, three npm fixtures' lockfiles, and the SvelteKit fixture's $lib. The app's .gitignore re-includes them.
- Look up release pull requests in videojs/video.js, which keeps the v10 history and tags; their deploy previews still serve the docs per release. - Pin video.js along with @videojs/* for release runs, keeping dist-tags such as latest-8, so it stays Video.js 8 for releases before it publishes Video.js 10 and the agents' most common mistake stays reproducible. - Pack the private workspace packages that public ones depend on, since @videojs/core now pins video.js (#9258). - Accept the neutral skin, the minimal skin renamed in 10.0.0-rc.5, and recognize the compat and scaffold skins. Tasks and results keep the minimal name so runs on either side of the rename compare. The builder arm and the minimal-skin oracles name the skin as the build under test does. - Resolve the app's vite-plus and vitest from the current catalog in the lockfile.
@videojs/cdn
@videojs/cli
@videojs/core
@videojs/element
@videojs/html
@videojs/media
@videojs/react
@videojs/spf
@videojs/store
@videojs/utils
@videojs/cloudflare-video
@videojs/dash-video
@videojs/hlsjs-video
@videojs/mux-audio
@videojs/mux-video
@videojs/native-hls-video
@videojs/playerjs-video
@videojs/shaka-video
@videojs/spotify-audio
@videojs/tiktok-video
@videojs/twitch-video
@videojs/vimeo-video
@videojs/wistia-video
@videojs/youtube-video
@videojs/google-cast
@videojs/mux-data
commit: |
Contributor
📦 Bundle Size Report🎨 @videojs/html — no changesPresets (7)
Media (19)
Players (5)
Skins (53)
UI Components (62)
⚛️ @videojs/react — no changesPresets (7)
Media (23)
Extensions (2)
Players (5)
Skins (34)
UI Components (39)
🧩 @videojs/core — no changesEntries (76)
🏷️ @videojs/element — no changesEntries (2)
📦 @videojs/store — no changesEntries (3)
🔧 @videojs/utils — no changesEntries (13)
📦 @videojs/cdn — no changes📦 @videojs/cloudflare-video — no changes📦 @videojs/dash-video — no changes📦 @videojs/google-cast — no changes📦 @videojs/hlsjs-video — no changes📦 @videojs/media — no changesEntries (3)
📦 @videojs/mux — no changes📦 @videojs/mux-audio — no changesEntries (2)
📦 @videojs/mux-data — no changes📦 @videojs/mux-video — no changesEntries (2)
📦 @videojs/native-hls-video — no changes📦 @videojs/playerjs-video — no changes📦 @videojs/shaka-video — no changes📦 @videojs/spf — no changesEntries (7)
📦 @videojs/spotify-audio — no changes📦 @videojs/tiktok-video — no changes📦 @videojs/twitch-video — no changes📦 @videojs/other — no changesEntries (3)
📦 @videojs/vimeo-video — no changes📦 @videojs/wistia-video — no changesEntries (2)
📦 @videojs/youtube-video — no changesℹ️ How to interpretEach entry is independently bundled, minified, and brotli-compressed. Initial size includes its static import graph; lazy dynamic chunks are reported separately. Entries are not additive because their dependency graphs overlap. Preset rows represent realistic combined bundles. Changes of 300 B or less across initial, lazy, and total size are collapsed, not discarded. Run |
✅ Deploy Preview for vjs10-site ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for vjs-registry ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #9443
Summary
Adds
apps/agent-evals, which measures how well coding agents (Claude Code and Codex) install Video.js, release over release. It ran the rc.2, rc.3, and rc.4 validation, but was built in a videojs/v10 worktree and never committed. This PR moves it here and updates it for this repository and the releases since.Changes
pnpm evals --videojs 10.0.0-rc.4,10.0.1compares two releases fairly, andworkspaceor a pull request ref tests code that isn't on npm yet.agents initcommand, or a simulated user who answers questions.agent-evals.yml).agent-evals/historybranch.test:agent-evalslabel evaluates a pull request's packed build against its deploy previews.video.jsis now pinned along with@videojs/*for release runs, keeping dist-tags such aslatest-8. It stays Video.js 8 for releases before it publishes Video.js 10, so the agents' most common mistake stays reproducible after feat(video.js): route installs to the framework packages #9258 and feat(core): depend on video.js #9259.@videojs/corenow pinsvideo.js..gitignorehad been excluding a harness helper and three fixture lockfiles (lib/andpackage-lock.jsonrules). The app's.gitignorenow re-includes them.Before merging
agent-evalsenvironment withANTHROPIC_API_KEYandOPENAI_API_KEY. Agent runs read their keys from it.test:agent-evalslabel.AGENT_EVALS_*repository variables first to change that.agent-evals/history, so the first scheduled run has something to compare with.Testing
pnpm install --frozen-lockfilepnpm -F @videojs/agent-evals test(124 passing, with new tests for skin detection,video.jspinning, and the builder's skin flag)tsgo --project apps/agent-evals/tsconfig.json --noEmit,vp lint --quiet apps/agent-evals,vp fmt --check, andpnpm check:workspacepnpm evals --dry --agents oracleresolves 10.0.1 to video.min.js and video.js both broken in IE8 when using data-setup="{}" #3142's deploy previews through this repository🤖 Generated with Claude Code