Skip to content

Add reusable single-entry Obsidian upsert helper, Add "Reset AI Breaker" button, Fix AI Studio response parsing for nested JSON payloads, Add Polyceph compatability hooks#28

Open
Ryah wants to merge 3 commits into
pixelnull:stagingfrom
Ryah:main
Open

Add reusable single-entry Obsidian upsert helper, Add "Reset AI Breaker" button, Fix AI Studio response parsing for nested JSON payloads, Add Polyceph compatability hooks#28
Ryah wants to merge 3 commits into
pixelnull:stagingfrom
Ryah:main

Conversation

@Ryah
Copy link
Copy Markdown

@Ryah Ryah commented May 6, 2026

I had my own personal fork to work on my companion extension, and genuinely forgot pull requests were a thing. So I decided to submit one. Sorry if it's a lot in one PR lol.

Summary

  • Add reusable upsertConvertedEntry(...) helper in the import bridge for single-entry conversion/upsert workflows.
    • Future lorebook automation support, added example for MemoryBooks in my companion extension.
    • My example is pretty hardcoded for specifically MemoryBooks, so I didn't want to include it in this PR to avoid any future conflicts.
  • Add Polyceph compatibility hooks for retrieval/publish integration:
    • deepLoreEnhanced_polycephRetrieve
    • deepLoreEnhanced_polycephPublish
  • Improve AI response parsing to handle additional provider envelope shapes and nested response payloads.
  • Fix AI Studio JSON API parsing fallback so nested JSON response envelopes resolve to usable model text/results.
  • Fix extractAiResponseClient object-wrapper guard to only unwrap objects with known array-valued keys, preventing silent mis-parse on arbitrary object shapes.
  • Add AI search trace logging ([DLE][TRACE] aiSearch_call_result / aiSearch_parse_failure_context) to surface raw result shape and parse failures without changing production behavior.
  • Add a drawer footer Reset AI Breaker control for manual breaker recovery
  • Improve AI breaker UX with countdown visibility and explicit manual reset control.
    • Breaker kept firing when I was testing the auto summary + keyword part of my MemoryBooks -> Obsidian pipeline failing (which led me to figure out the consistent JSON enveloping bug in AI Studio) so I added that in as a fallback to not have to refresh the page).

Why

  • Enables cleaner extension-to-extension integration without duplicating import/write logic.
  • Improves resilience and usability around AI service degradation/recovery.
  • Increases compatibility with provider-specific response payload formats.

Ryah and others added 3 commits May 3, 2026 17:33
…gle entry lorebook import helper function

Co-authored-by: Copilot <copilot@github.com>
- Restore Google AI Studio envelope unwrapping in cmrsResultToText
  (candidates[].content.parts[].text extraction)
- Restore usageMetadata token count mapping
  (promptTokenCount → input_tokens, candidatesTokenCount → output_tokens)
- Fix extractAiResponseClient object-wrapper detection to only accept
  known keys (selected, results, entries, titles), preventing false
  positives on arbitrary objects
- Add [DLE][TRACE] aiSearch_call_result log before extractAiResponseClient
  to surface raw result shape during AI search
- Add [DLE][TRACE] aiSearch_parse_failure_context on parse failure
- Add debug metadata to cmrsResultToText return value
  (rawKeys, contentType, textSource, candidatesFound, partsFound,
  usageKeys, textPreview)
@pixelnull
Copy link
Copy Markdown
Owner

pixelnull commented May 14, 2026

Hey! @Ryah

Don't apologize for doing a PR or the bundle. You said you forgot PRs were a thing while working on your fork; that's how most useful contributions actually happen, and the volume of useful stuff in this PR is real. Thanks for sending it. The Reset AI Breaker UX, the upsertConvertedEntry helper, the AI Studio response-parsing work, and the Polyceph integration are all things I want to land... BUUUT, you're right, just not in one bundle.

The reviewability blockers, concretely:

Bundled scope - Four genuinely independent changes (breaker UX, import helper, AI parser, Polyceph pipeline-replay). Reviewing them together means a problem in any one blocks the others, and rolling back later means surgical reverts across 14 files. I want to merge the breaker button this week and still be asking questions about Polyceph in two weeks without holding the breaker hostage.

Reformatter pollution. I diffed the base and head of each large file with whitespace stripped:

  • settings-popup.html: 1163 add / 927 del at the line level - but identical content character-for-character once whitespace is removed. 100% reformatter pass, zero behavior change.
  • style.css: 1827 add / 377 del at the line level - but only a few hundred bytes of real content change (the new .dle-breaker-reset-btn, .dle-footer-breaker-row, .dle-ai-breaker-countdown rules). The other ~99% is rule-body expansion / indentation.
  • drawer.html: 151 add / 80 del - ~380 bytes (Reset AI Breaker button markup).

That's an enormous amount of noise for a small amount of real change, and it's hiding the real change. I'd guess your editor has format-on-save with different rules than this repo uses. Maybe a attribute-per-line HTML wrapping at ~120 cols, and one-property-per-line CSS expansion. If you can either turn that off for these files, or revert the formatter pass before opening the smaller PRs, the actual review goes from "scan 3000 lines for behavior changes" to "review 30 lines".

Polyceph surface is bigger than the PR body implies. retrieveLoreForPolyceph in index.js is basically a full parallel entry point that re-runs the pipeline (matchEntries → runPipeline → applyPinBlock → contextualGating → reinjectionCooldown → requiresExcludes → stripDedup → formatAndGroup) and publishes injection state to the drawer.

That's ~109 new lines plus two globalThis.deepLoreEnhanced_polyceph* surfaces. Adding a hook is a soft API commitment; this is a parallel pipeline entry point, which is a much bigger commitment because the drawer/sidebar state has to stay consistent across both call paths forever. I want to evaluate that in isolation, not bundled with three other things.

What I'd like

Four PRs against staging:

  1. feat: Reset AI Breaker drawer button + cooldown countdown - src/state.js (resetAiCircuitBreaker, getAiCircuitCooldownRemainingMs), src/drawer/drawer-render-footer.js, src/drawer/drawer-events.js, the ~3 new selectors in style.css, the breaker button markup in drawer.html, the test/unit.mjs cases that cover it. Drop the formatter-only chunks from the CSS and HTML. Smallest PR, lowest risk, highest immediate utility. I'll merge this first.

  2. feat: upsertConvertedEntry helper for single-entry Obsidian writes - src/vault/import.js only, plus tests. Clean addition, no behavior change to existing import flow.

  3. fix: handle Google AI Studio nested response envelopes - src/ai/ai.js, src/helpers.js (cmrsResultToText + extractAiResponseClient changes), trace logging, tests. Looks good. One question I want to dig into before merge: extractInspectStyleEnvelopeText is regex-parsing util.inspect-formatted output (single-quoted strings concatenated with +). That's weird, it implies the AI Studio response is reaching cmrsResultToText already converted to inspect-string form rather than as an object. Could you point me at where in the call stack the inspect-stringification happens? I'd rather fix the upstream stringify than parse the symptom downstream, if we can. The Gemini candidates[].content.parts[].text extraction itself is clearly correct and welcome.

  4. feat: Polyceph pipeline-replay surface (deepLoreEnhanced_polycephRetrieve / Publish) - the index.js additions and the two globals. I want to look at this one with fresh eyes because it's effectively a second public entry point to the retrieval pipeline, and the drawer/sidebar injection-state contract has to hold across both. Things I'll want to verify: does publishPolycephTrace racing with a real onGenerate cause the injection panel to show wrong data? does setLastInjectionEpoch(chatEpoch) after Polyceph desync the swipe-rollback tracking? Worth slowing down on alone.

Workflow suggestion:

git checkout staging
git pull
git checkout -b breaker-reset
git cherry-pick <commits relevant to breaker>
# remove formatter-only hunks from style.css / drawer.html / settings-popup.html
git push
gh pr create --base staging
# repeat for the other three branches

If you'd rather not split it yourself, fair. Say the word and I'll cherry-pick the breaker UX and upsertConvertedEntry directly off your branch into focused commits on my end, credit you in the commit messages, and close this PR with thanks. The AI Studio parser and Polyceph pieces I'd really rather you author standalone, since I want to ask the upstream-stringify question on one and the pipeline-state-contract questions on the other, and you're best positioned to answer.

Thanks again: the forgot-PRs-existed energy is exactly why I left issue #2 open. Rather get a 14-file dump and ask for a split than not see the contribution.

🖤

@Ryah
Copy link
Copy Markdown
Author

Ryah commented May 19, 2026

Sorry, I just saw this. Honestly my plate has gotten very full now that I've been brought onto the Freaky Frankenstein team and with personal things in life, so I don't have much time to pick the commits apart. Instead, if you would like, you are free to do whatever with my fork. I probably won't be updating it for a while, so absolutely feel free to take whatever you want from it! I still use DeepLore all the time so I'm happy to contribute in any way I can, I just won't have the mental energy to look at a line of code for a while lol.

There's technically 2 branches. The main one (this PR) and another branch called "fork" where I threw more things into. Feel free to search through the commits and cherrypick whatever you would like added.

pixelnull added a commit that referenced this pull request May 22, 2026
…frontmatter surgery

Four Wave 2 features land together; all touched files share helpers.js

or settings, so a per-feature split would force git add -p churn for

minimal benefit. Post-audit regressions caught by 3 parallel Opus

audit agents are folded in (no separate fixup commits).

#16 Reverse Priority -- new settings.priorityReversed (default false)

drives a comparePriority(a, b, reversed) helper used by pipeline budget

allocation (match.js, pipeline.js, stages.js), Librarian view, Cartographer,

and default Browse/drawer sort. Explicit priority_asc/priority_desc dropdown

choices stay literal -- only the default cases flip. /dle-simulate now

honors the toggle too (audit Agent A finding).

PR #28.2 upsertConvertedEntry -- single-entry convert-and-upsert for

companion-extension integration. Extracts existing rename loop into

private _findUniquePath helper (importEntries refactored to use it).

Collision policy: rename | replace | skip. Defaults to

resolveWriteVault('autoSuggest'). Guard rejects vaults missing

host/port/apiKey (audit Agent B finding).

#18 Caveman-compress import -- new src/caveman.js applies article /

filler / pleasantry stripping at import time, annotates frontmatter

with compress: caveman. Setting importCompressByDefault (default false).

Forward-compat: unknown modes (e.g. ai-summary) log warning, do not

annotate (audit Agent B finding -- previous version lied about state).

Word-boundary regex uses lookaround instead of \\b so A-frame / A-list

survive (audit Agent B finding -- previous \\b stripped the leading A).

Mask uses per-call random alphanumeric tag (no NUL bytes, no

whitespace-cleanup fragility -- audit Agent B finding).

Update Existing Entries -- new updateFrontmatterFields() (pure) +

updateEntryFields() (vault wrapper) for surgical scalar-field patches.

Preserves quoting style, body, untouched fields. Refuses block scalars,

inline-flow arrays, and NaN/Infinity (audit Agent C findings -- silent

corruption otherwise). Handles CRLF round-trip and hyphenated/dotted

keys (audit Agent C findings -- silent dup-key files otherwise).

Tests: +101 new unit assertions (caveman + resolveCompressMode +

updateFrontmatterFields + 13 audit-driven regression guards). Total

3068 passing, 0 failing. Zero broken imports.
pixelnull added a commit that referenced this pull request May 22, 2026
New state.resetAiCircuitBreaker() — operator-initiated path that discards

pending cooldown without claiming the underlying service recovered. Same

end state as recordAiSuccess() but emits ai_circuit event tagged with

manualReset:true so analytics/debug can distinguish from organic recovery.

Returns {wasOpen, hadPendingCooldown} so the toast text can match what

actually happened.

Drawer footer button (dle-reset-ai-breaker) — conditional, only visible

while breaker open; renderFooter() toggles dle-hidden each tick and the

tooltip shows the live countdown (e.g. 'retry in ~12s'). Click fires a

confirmation popup before resetting.

Settings Danger Zone button (dle-sp-reset-ai-breaker) — always

accessible, sits next to Reset All Settings. Same confirmation popup +

toast as the footer path.
pixelnull added a commit that referenced this pull request May 22, 2026
…iCircuitBreaker (PR #28.1)

Two shipped features had no direct unit/integration coverage. Both small pure-ish surfaces — easy to backfill.

comparePriority(a, b, reversed) — 4 tests, 8 assertions:

  * normal order (lower priority wins)

  * reversed order (higher priority wins)

  * missing priority defaults to 50

  * equal priorities return 0

resetAiCircuitBreaker() — 4 integration tests, 9 assertions:

  * closed circuit: wasOpen=false, no state change

  * tripped circuit: wasOpen=true, hadPendingCooldown=true, fully cleared

  * observer fires on open→closed transition

  * observer does NOT fire when already closed (no spurious notifications)

Totals: 1506 unit (+8), 282 integration (+9), 266 regression unchanged, 0 broken imports.
pixelnull added a commit that referenced this pull request May 22, 2026
Partial port of PR #28 (Ryah). Code review caught an overclaim in my first draft theory: the dangling-fence strip does NOT close the json_schema breaker-trip bug. Reframed honestly.

## The real fix

cmrsResultToText now reads Gemini's usageMetadata.{promptTokenCount, candidatesTokenCount} as alternates for OAI's usage.{prompt_tokens, completion_tokens}. Without these aliases every Gemini call reported 0/0 input/output tokens in DLE stats.

## Defensive additions (no current bug closed)

- extractAiResponseClient strips dangling trailing triple-backtick fences before the direct JSON.parse attempt. Pre-fix code already rescues '[...]\\n`' via bracket-balanced extraction; the strip just makes the first parse attempt succeed without walking the whole string. Belt-and-suspenders.

- extractAiResponseClient accepts pre-parsed array input directly. Forward-compatible with callers that skip the stringify step. Non-array objects still return null (parity with prior behavior — caller's BUG-383 wrapper-key unwrap stays in charge).

## Open bug NOT closed by this commit

Gemini json_schema-flow responses with trailing fence still trip the breaker. The failure happens at ST layer: ST.extractJsonFromData runs tryParse(text), fence breaks parse, ST returns '{}' to DLE. By the time text reaches extractAiResponseClient it is already '{}' — no fence left to strip. Real fix needs upstream ST PR (harden tryParse) or larger DLE workaround that bypasses ST's json_schema processing. Documented in audit/v2.5-prep/PR-28.3-VERDICT.md. Defer to 3.0 unless user-reported.

## Out of scope (deferred to 3.0)

- Gemini candidates[]/responseContent.parts envelope extraction in cmrsResultToText. ST's chat-completions.js:726 already unwraps to OAI format; CMRS strips responseContent before DLE sees it. Ships dead code paths.

- inspect-string envelope parser. ST's util.inspect call at chat-completions.js:716 is a console.debug — not in HTTP response. No upstream source for the pattern found.

- debug field on every cmrsResultToText return. Production code shouldn't carry per-call diagnostic payloads.

- Trace logging additions. Separate concern from parser fixes.

## Tests

+6 unit tests, +12 assertions. Test comments accurately describe what the assertions prove (bracket-balance is the actual win path on object-wrap; strip is defense-in-depth on direct-array path).

1550 unit + 282 integration + 275 fields + 154 stages + 204 contracts + 223 diagnostics + 243 vault + 266 regression. 0 broken imports.

PR #28 stays open; mark as partially merged in v2.5.

Co-Authored-By: Ryah <14987609+Ryah@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants