Skip to content

[Bug]: Concurrent plugin requests corrupt .installed.json, so a just-installed plugin is recorded as source "local" and Update/launch then fail #5496

Description

@enyst

I'm an AI agent (Claude Code) helping Engel Nyst (@enyst) with project work. I found this while mapping every user-facing Agent Canvas feature with a new verification skill, and a second, independent agent reproduced it on a fresh stack before I filed it.

Is there an existing issue for the same bug?

  • I have searched existing issues and this is not a duplicate.

Bug Description

Installing a plugin through the Agent Server sometimes persists it with "source": "local" instead of its real source. After that, Update fails with HTTP 500 and Start Conversation builds a launch link the agent cannot load. In Agent Canvas this happened on 3 of 6 install cycles of the same catalog plugin.

The metadata file is written non-atomically and read/written by overlapping requests:

  • openhands/sdk/extensions/installation/metadata.py: save_to_dir uses Path.write_text, which truncates and then writes; load_from_dir swallows the resulting JSON error and returns empty metadata.
  • MetadataSession.__exit__ saves on every clean exit, including read-only sessions; manager.py list_installed opens such a session on every GET /api/plugins/installed.
  • With empty metadata, discover_untracked re-adds the installed directory with source="local" ("Discovered untracked extension: magic-test"), and the GET saves that over the real entry. A GET that loaded metadata before the install saved it can also overwrite it (lost update).
  • agent_server/plugins_router.py declares the endpoints as sync defs, so FastAPI runs them in parallel threads with no lock. Agent Canvas fires two or three GET /api/plugins/installed right after an install, which triggers the race.

Expected Behavior

After a successful POST /api/plugins/install, the persisted metadata keeps the real source (the catalog path, or github:... plus repo_path) regardless of concurrent list calls. Update re-fetches the plugin, and the launch link loads.

Actual Behavior

stack.log shows 16 lines like:

Failed to load installed extension metadata: ... Invalid JSON: EOF while parsing a value at line 1 column 0 [input_value='']
Discovered untracked extension: magic-test

Then .installed.json holds "source": "local". Update returns 500 Unable to parse extension source: local. Expected formats: 'github:owner/repo', git URL, or local path, and Start Conversation produces /launch?plugins=... whose modal reads "Launch local".

Plugin detail after install shows source local

Steps to Reproduce

  1. Start Agent Canvas 1.24.0 (Agent Server 1.50.1): npx @openhands/agent-canvas.
  2. Open Customize > Plugins and install a catalog plugin (for example magic-test).
  3. Uninstall and install it again a few times; after each install read ~/.openhands/.../plugins/installed/.installed.json (in our harness: <run>/private/plugins/installed/.installed.json).
  4. When the entry shows "source": "local", open the plugin and click Update, or Start Conversation.

Without the UI, overlapping GET /api/plugins/installed calls during POST /api/plugins/install (for example a small loop of concurrent GETs) should reproduce the empty read and the local rediscovery.

Acceptance Criteria

  • Installing a plugin always persists its real source, even while GET /api/plugins/installed runs concurrently.
  • Concurrent readers never see an empty or truncated .installed.json (atomic write: temp file + os.replace).
  • A read-only list call never overwrites a tracked entry with a source="local" rediscovery (no save from read-only sessions, or a lock around load-modify-save).
  • Update and Start Conversation work right after install.

Installation Method

npx @openhands/agent-canvas (Agent Server via uvx)

SDK Version

1.50.1 (the version Agent Canvas 1.24.0 pins)

Operating System

Linux

Screenshots and Additional Context

The duplicated error toast on Update is a separate Canvas issue. No duplicate found for .installed.json, untracked extension, or source local.

Activity

  1. added
    bugSomething isn't working
    on Oct 5, 2026
  2. added theissue type on Oct 5, 2026
  3. vnktadithya commented on Oct 5, 2026

    @vnktadithya
    Contributor

    Hi @enyst, I'd like to work on this. I'll reproduce both the empty-read and the lost-update paths locally first, then follow up here with my proposed approach before opening a PR.

  4. vnktadithya commented on Oct 7, 2026

    @vnktadithya
    Contributor

    Hello @enyst. Looked into the issue. Reproduced both paths with a forced thread interleaving. They're two separate races: the torn read from the truncating write_text, and a lost update where aGET /installed loads metadata before the install saves, then writes its stale copy plus a source="local" rediscovery over it. An atomic write alone only fixes the first.

    Proposed fix, all in metadata.py: save through the existing atomic_write_text, and have InstallationMetadata.open() take a FileLock before loading and release it in MetadataSession.__exit__, with a 30s timeout like the agent/LLM profile stores. list_installed keeps its self-healing save (existing tests pin it); it just can no longer interleave with another session. This covers plugins, skills and canvas extensions, and I have regression tests for each race that fail on main and pass with the fix.

    Two questions before I open the PR:

    1) get() and canvas _tracked_names read without the lock. On Windows, the atomic replace fails with WinError 5 if one of those readers has the file open at that moment, where the old in-place write_text succeeded (checked both on Windows). I'd have them read under the same lock without saving, the way ProviderConnectionStore.get() does. Is it ok to include that in this PR?

    2) install() removes and re-copies the extension directory outside the lock, so a list call during an Update can prune the entry, and the reinstall then loses its enabled state (I reproduced a disabled plugin coming back enabled). Hold the lock across the directory swap in this PR, or leave it as a follow-up?

  5. smolpaws commented on Oct 8, 2026

    @smolpaws
    Contributor

    I am an AI agent helping Engel Nyst (@enyst) with OpenHands project maintenance.

    Fresh live reproduction during the 2026-10-09 Agent Canvas verify-openhands sweep, with Agent Canvas main@3500d9e5c7cbdfb5a2cbfe9499ce4448eb10c0a2 and Agent Server/SDK 1.53.0:

    1. In /plugins, install the magic-test catalog plugin. The installed API response records the catalog path as source, enabled: true, and installed_at: 23:08:22.154164 UTC.
    2. Reload, turn its switch off, and check /api/plugins/installed. The API now reports enabled: false with the original source and timestamp.
    3. Reload again. The next GET reports enabled: true, source: "local", and a new installed_at: 23:08:51.827089 UTC. The disabled plugin was silently re-enabled and lost its catalog identity.
    4. Click Update. Canvas shows Unable to parse extension source: local....

    The Agent Server log at the same transition reports Failed to load installed extension metadata ... Invalid JSON: EOF ... input_value='', followed by Discovered untracked extension: magic-test. That directly matches this issue's metadata truncation/read race. The failure was reproduced through the real UI and API; the local run's F19.enable-toggle and F19.refresh checks are failures. A reviewed screenshot of the Update error is retained with the private sweep evidence.

  6. vnktadithya commented on Oct 9, 2026

    @vnktadithya
    Contributor

    @enyst, opened #5690 for this: atomic write plus a lock around the metadata load-save cycle, with before/after runs through the Agent Server API in the description. The PR description check requires the linked issue to carry the ready-for-dev label. Could you please add the label and check if my approach aligns well?

  7. vnktadithya commented on Oct 18, 2026

    @vnktadithya
    Contributor

    @enyst, a quick update on this one. #5690 has the fix (an atomic write plus a lock around the metadata load-save cycle), with before/after runs through the Agent Server API in the description, and the repo's review agent has approved the current head. Its PR description check needs ready-for-dev on this issue, since the issue was opened after the readiness rollout. Could you add it when you get a chance, if the approach looks right to you?

    I've also filed the remaining directory-swap window in install() as #5985 (now ready-for-dev). It builds on the lock from #5690, so I'll pick it up once #5690 is in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions