Add reusable single-entry Obsidian upsert helper, Add "Reset AI Breaker" button, Fix AI Studio response parsing for nested JSON payloads, Add Polyceph compatability hooks#28
Conversation
…gle entry lorebook import helper function Co-authored-by: Copilot <copilot@github.com>
- Restore Google AI Studio envelope unwrapping in cmrsResultToText (candidates[].content.parts[].text extraction) - Restore usageMetadata token count mapping (promptTokenCount → input_tokens, candidatesTokenCount → output_tokens) - Fix extractAiResponseClient object-wrapper detection to only accept known keys (selected, results, entries, titles), preventing false positives on arbitrary objects - Add [DLE][TRACE] aiSearch_call_result log before extractAiResponseClient to surface raw result shape during AI search - Add [DLE][TRACE] aiSearch_parse_failure_context on parse failure - Add debug metadata to cmrsResultToText return value (rawKeys, contentType, textSource, candidatesFound, partsFound, usageKeys, textPreview)
|
Hey! @Ryah Don't apologize for doing a PR or the bundle. You said you forgot PRs were a thing while working on your fork; that's how most useful contributions actually happen, and the volume of useful stuff in this PR is real. Thanks for sending it. The Reset AI Breaker UX, the The reviewability blockers, concretely: Bundled scope - Four genuinely independent changes (breaker UX, import helper, AI parser, Polyceph pipeline-replay). Reviewing them together means a problem in any one blocks the others, and rolling back later means surgical reverts across 14 files. I want to merge the breaker button this week and still be asking questions about Polyceph in two weeks without holding the breaker hostage. Reformatter pollution. I diffed the base and head of each large file with whitespace stripped:
That's an enormous amount of noise for a small amount of real change, and it's hiding the real change. I'd guess your editor has format-on-save with different rules than this repo uses. Maybe a attribute-per-line HTML wrapping at ~120 cols, and one-property-per-line CSS expansion. If you can either turn that off for these files, or revert the formatter pass before opening the smaller PRs, the actual review goes from "scan 3000 lines for behavior changes" to "review 30 lines". Polyceph surface is bigger than the PR body implies. That's ~109 new lines plus two What I'd like Four PRs against
Workflow suggestion: If you'd rather not split it yourself, fair. Say the word and I'll cherry-pick the breaker UX and Thanks again: the forgot-PRs-existed energy is exactly why I left issue #2 open. Rather get a 14-file dump and ask for a split than not see the contribution. 🖤 |
|
Sorry, I just saw this. Honestly my plate has gotten very full now that I've been brought onto the Freaky Frankenstein team and with personal things in life, so I don't have much time to pick the commits apart. Instead, if you would like, you are free to do whatever with my fork. I probably won't be updating it for a while, so absolutely feel free to take whatever you want from it! I still use DeepLore all the time so I'm happy to contribute in any way I can, I just won't have the mental energy to look at a line of code for a while lol. There's technically 2 branches. The main one (this PR) and another branch called "fork" where I threw more things into. Feel free to search through the commits and cherrypick whatever you would like added. |
…frontmatter surgery Four Wave 2 features land together; all touched files share helpers.js or settings, so a per-feature split would force git add -p churn for minimal benefit. Post-audit regressions caught by 3 parallel Opus audit agents are folded in (no separate fixup commits). #16 Reverse Priority -- new settings.priorityReversed (default false) drives a comparePriority(a, b, reversed) helper used by pipeline budget allocation (match.js, pipeline.js, stages.js), Librarian view, Cartographer, and default Browse/drawer sort. Explicit priority_asc/priority_desc dropdown choices stay literal -- only the default cases flip. /dle-simulate now honors the toggle too (audit Agent A finding). PR #28.2 upsertConvertedEntry -- single-entry convert-and-upsert for companion-extension integration. Extracts existing rename loop into private _findUniquePath helper (importEntries refactored to use it). Collision policy: rename | replace | skip. Defaults to resolveWriteVault('autoSuggest'). Guard rejects vaults missing host/port/apiKey (audit Agent B finding). #18 Caveman-compress import -- new src/caveman.js applies article / filler / pleasantry stripping at import time, annotates frontmatter with compress: caveman. Setting importCompressByDefault (default false). Forward-compat: unknown modes (e.g. ai-summary) log warning, do not annotate (audit Agent B finding -- previous version lied about state). Word-boundary regex uses lookaround instead of \\b so A-frame / A-list survive (audit Agent B finding -- previous \\b stripped the leading A). Mask uses per-call random alphanumeric tag (no NUL bytes, no whitespace-cleanup fragility -- audit Agent B finding). Update Existing Entries -- new updateFrontmatterFields() (pure) + updateEntryFields() (vault wrapper) for surgical scalar-field patches. Preserves quoting style, body, untouched fields. Refuses block scalars, inline-flow arrays, and NaN/Infinity (audit Agent C findings -- silent corruption otherwise). Handles CRLF round-trip and hyphenated/dotted keys (audit Agent C findings -- silent dup-key files otherwise). Tests: +101 new unit assertions (caveman + resolveCompressMode + updateFrontmatterFields + 13 audit-driven regression guards). Total 3068 passing, 0 failing. Zero broken imports.
New state.resetAiCircuitBreaker() — operator-initiated path that discards
pending cooldown without claiming the underlying service recovered. Same
end state as recordAiSuccess() but emits ai_circuit event tagged with
manualReset:true so analytics/debug can distinguish from organic recovery.
Returns {wasOpen, hadPendingCooldown} so the toast text can match what
actually happened.
Drawer footer button (dle-reset-ai-breaker) — conditional, only visible
while breaker open; renderFooter() toggles dle-hidden each tick and the
tooltip shows the live countdown (e.g. 'retry in ~12s'). Click fires a
confirmation popup before resetting.
Settings Danger Zone button (dle-sp-reset-ai-breaker) — always
accessible, sits next to Reset All Settings. Same confirmation popup +
toast as the footer path.
…iCircuitBreaker (PR #28.1) Two shipped features had no direct unit/integration coverage. Both small pure-ish surfaces — easy to backfill. comparePriority(a, b, reversed) — 4 tests, 8 assertions: * normal order (lower priority wins) * reversed order (higher priority wins) * missing priority defaults to 50 * equal priorities return 0 resetAiCircuitBreaker() — 4 integration tests, 9 assertions: * closed circuit: wasOpen=false, no state change * tripped circuit: wasOpen=true, hadPendingCooldown=true, fully cleared * observer fires on open→closed transition * observer does NOT fire when already closed (no spurious notifications) Totals: 1506 unit (+8), 282 integration (+9), 266 regression unchanged, 0 broken imports.
Partial port of PR #28 (Ryah). Code review caught an overclaim in my first draft theory: the dangling-fence strip does NOT close the json_schema breaker-trip bug. Reframed honestly. ## The real fix cmrsResultToText now reads Gemini's usageMetadata.{promptTokenCount, candidatesTokenCount} as alternates for OAI's usage.{prompt_tokens, completion_tokens}. Without these aliases every Gemini call reported 0/0 input/output tokens in DLE stats. ## Defensive additions (no current bug closed) - extractAiResponseClient strips dangling trailing triple-backtick fences before the direct JSON.parse attempt. Pre-fix code already rescues '[...]\\n`' via bracket-balanced extraction; the strip just makes the first parse attempt succeed without walking the whole string. Belt-and-suspenders. - extractAiResponseClient accepts pre-parsed array input directly. Forward-compatible with callers that skip the stringify step. Non-array objects still return null (parity with prior behavior — caller's BUG-383 wrapper-key unwrap stays in charge). ## Open bug NOT closed by this commit Gemini json_schema-flow responses with trailing fence still trip the breaker. The failure happens at ST layer: ST.extractJsonFromData runs tryParse(text), fence breaks parse, ST returns '{}' to DLE. By the time text reaches extractAiResponseClient it is already '{}' — no fence left to strip. Real fix needs upstream ST PR (harden tryParse) or larger DLE workaround that bypasses ST's json_schema processing. Documented in audit/v2.5-prep/PR-28.3-VERDICT.md. Defer to 3.0 unless user-reported. ## Out of scope (deferred to 3.0) - Gemini candidates[]/responseContent.parts envelope extraction in cmrsResultToText. ST's chat-completions.js:726 already unwraps to OAI format; CMRS strips responseContent before DLE sees it. Ships dead code paths. - inspect-string envelope parser. ST's util.inspect call at chat-completions.js:716 is a console.debug — not in HTTP response. No upstream source for the pattern found. - debug field on every cmrsResultToText return. Production code shouldn't carry per-call diagnostic payloads. - Trace logging additions. Separate concern from parser fixes. ## Tests +6 unit tests, +12 assertions. Test comments accurately describe what the assertions prove (bracket-balance is the actual win path on object-wrap; strip is defense-in-depth on direct-array path). 1550 unit + 282 integration + 275 fields + 154 stages + 204 contracts + 223 diagnostics + 243 vault + 266 regression. 0 broken imports. PR #28 stays open; mark as partially merged in v2.5. Co-Authored-By: Ryah <14987609+Ryah@users.noreply.github.com>
I had my own personal fork to work on my companion extension, and genuinely forgot pull requests were a thing. So I decided to submit one. Sorry if it's a lot in one PR lol.
Summary
upsertConvertedEntry(...)helper in the import bridge for single-entry conversion/upsert workflows.deepLoreEnhanced_polycephRetrievedeepLoreEnhanced_polycephPublishextractAiResponseClientobject-wrapper guard to only unwrap objects with known array-valued keys, preventing silent mis-parse on arbitrary object shapes.[DLE][TRACE] aiSearch_call_result/aiSearch_parse_failure_context) to surface raw result shape and parse failures without changing production behavior.Reset AI Breakercontrol for manual breaker recoveryWhy