Skip to content

Test Qwen3.8 27B and audit the native Forge handoff against a real Homebase task #111

Closed

Description

@kudama

Completed — 2026-11-18

Qwen3.8 27B Q4_K_M is installed with verified official layer digests. Evaluation complete; no production model switch, controller deployment or Homebase implementation was performed.

Native/generic trials under reasoning-off failed during source retrieval or model response. A separately labeled direct source-packet/structured-output diagnostic returned exact-source changes, but unchanged generated Python fails compilation. The final native low-reasoning trial239c2fd3df4b4f88bca5d465d9d45886 completed requested source reads, then hit the240-second deadline without a proposal. No usable model implementation accepted. These are bounded single-task trials, not a general coding benchmark.

The handoff audit found repeated JSON keys silently accepted in native results. Draft PR #113 fixes this;95targeted tests and both GitHub workflows pass, and restoring legacy parsing makes both new native regression cases fail. Issue #112 remains open for review/release. No fix deployed.

Foundry access returned after the earlier interruption. Host uptime exceeds4days; cause remains unknown. Final checks:SSH works,HomebaseHTTP200,forge_local ready,no models loaded,no unfinished live Forge tasks,deployed controller hashes and selected sources unchanged. Raw evidence recovered; no test remains blocked.

Recommendation:retain the current default. ReviewPR113 and simplify source delivery/task sizing/output contracts before further model promotion. Detailed results and all attempts are in comments and output/qwen38-handoff-20261117/REPORT.md.

Original scope

Requested by Mike on2026-11-17 after #110:install/testQwen3.8,audit the real MCP/native handoff, distinguish delivery/schema/source gates from independent correctness, and keep production/default routing unchanged. Real task:Homebase host-selection settings (kudama/homelab#14). No worker writes,command tools or credential grants.

Activity

  1. kudama commented on Nov 18, 2026

    @kudama
    OwnerAuthor

    2026-11-17 interim audit:86 existing handoff/native/receipt tests pass. Actual MCP bridge smoke against an isolated native controller rejects incorrect source hashes before inference, rejects duplicate orders and returns immutable snapshot provenance with review_required. Existing Qwen3-Coder native task65001ace8cfb41fe868dd9c9f80d2be2 failed tool_denied after requesting100 lines from the1-line test seed; no extra source returned. This is a protocol failure, not a reviewed code-quality result. Native duplicate-key acceptance was reproduced and a focused fix is now draft PR #113 / issue #112:95 checks pass, both new native cases fail under restored legacy parsing, and both GitHub workflows passed. Deployed controller/defaults remain unchanged. Qwen3.8 verified-layer download and model trials are still pending.

  2. kudama commented on Nov 18, 2026

    @kudama
    OwnerAuthor

    2026-11-17 model/handoff results saved; final diagnostic blocked by Foundry reachability.

    Installed Qwen3.8 27B Q4_K_M with verified official model/projector and metadata digests. Production routing and Homebase deployment unchanged.

    Native MCP trials on isolated controller code:

    • Current coder65001ace8cfb41fe868dd9c9f80d2be2: tool_denied, requested100 lines from1-line seed.
    • Qwen3.8 task92dde56c9c5c491aba4141df1cc18602: tool_limit_exceeded at10 calls after continued selected-source inspection.
    • Generic guided Qwen3.8 e819017b7a7248f8b4565c9ef00155ea: tool_denied on settings.py211..320 (>80lines).
      -12-call native follow-up9d2d2a32b5714111a1d93c0bdfb1d4c1: invalid_model_response before usable tool output.

    A separately labeled direct source-packet + structured-output coding diagnostic returned valid shape and exact-source replacements in106.06s, but unchanged generated Python fails compilation (unmatched parentheses in public_config). Static review also finds its host-selection expression always returns None. Both generated and independent tests fail at import; no model implementation accepted. This direct diagnostic is not a native handoff pass.

    Handoff fix: draft PR #113 / issue #112;95 targeted checks pass, both new native cases fail with legacy parsing, and both GitHub workflows pass. No fix deployed.

    Current adapter forces reasoning off. Installed metadata supports false/low/medium/xhigh, default medium. Final low-profile trial was not submitted: a runner-label guard prevented overwriting an existing record; label corrected locally, then Foundry stopped accepting SSH/Homebase connections and forge_local became unavailable. This harness/availability failure is not scored as a model failure. User was asked whether Foundry is restarting.

    Recommendation so far: no model promotion; fix the verified handoff defect and improve source delivery/configuration before further model swaps. Remaining: restore host reachability, retrieve raw remote records, run corrected low-reasoning native check and verify host health. Evidence retained in output/qwen38-handoff-20261117/REPORT.md and review-summary.json; raw native records remain in the private Foundry evaluation directory.

  3. added
    work:blockedWaiting for a stated prerequisite
    worker:forgeForge architecture, security, integration, or operations
    and removed
    work:blockedWaiting for a stated prerequisite
    on Nov 18, 2026
  4. kudama commented on Nov 18, 2026

    @kudama
    OwnerAuthor

    2026-11-18: user confirmed Foundry should be online. Rechecked: authenticated Forge health and SSH now respond; model inventory retains Qwen3.8. Reconciled evaluation state: no existing low-profile attempt record, no unfinished live Forge tasks, no loaded model. Uploaded the corrected isolated runner and resumed the final low-reasoning native handoff check with unchanged source permissions/default routing. Prior outage cause remains unconfirmed.

  5. kudama commented on Nov 18, 2026

    @kudama
    OwnerAuthor

    2026-11-18 evaluation completed after connectivity returned. Foundry uptime exceeds4days; no recent reboot was observed and the earlier access-failure cause remains unconfirmed.

    Final isolated low-reasoning native MCP trial239c2fd3df4b4f88bca5d465d9d45886 used the same pinned Homebase source,12read-only calls,16K context,4096output and240-second deadline. It completed the five requested source reads, then failed deadline_exceeded at240.57s without a final proposal. Bad-source admission was rejected before inference, duplicate order rejected, and source hashes remained unchanged. The profile changed think to the model-supported low setting in a private controller copy only.

    Conclusion: Qwen3.8 is installed, but no usable uncorrected implementation was accepted across these trials. Do not promote it on this evidence. Current handoff/tool budgeting/configuration and independent code validation need improvement; results are limited single-task trials, not proof about every possible model profile.

    The verified native duplicate-JSON-key defect has a concrete draft fix in PR #113 / issue #112:95targeted checks and both GitHub workflows pass, with legacy-mutation reproduction. No fix or model routing change deployed.

    Final health:SSH reachable,HomebaseHTTP200,forge_local ready,no models left loaded,no unfinished live Forge tasks,deployed app.py/work_orders.py hashes unchanged. Raw remote evidence recovered to the private Forge workspace output/qwen38-handoff-20261117; REPORT.md and review-summary.json updated. No remaining test is blocked. Next action:reviewPR113 and improve/simplify the handoff before another default-model decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    worker:forgeForge architecture, security, integration, or operations

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions