Skip to content

Latest commit

 

History

History
411 lines (306 loc) · 20.4 KB

data-flow.md

File metadata and controls

411 lines (306 loc) · 20.4 KB

Feed Data Lifecycle: Fetch, Merge, Retention, and Persistence

Traces the full journey of a feed item — from the first remote fetch through local merge, retention pruning, disk persistence, and dashboard display. Covers what is actually stored, when items are permanently deleted vs. visually filtered, and how retention setting changes affect stored data.


1. The Full Lifecycle at a Glance

Remote RSS/Atom feed (XML)
        │
        ▼
  ┌─────────────┐
  │    FETCH    │  HTTP request to the feed URL
  └──────┬──────┘
         │  raw XML text
         ▼
  ┌─────────────┐
  │    PARSE    │  Extract title, guid, pubDate, content, media type…
  └──────┬──────┘
         │  new FeedItem[]
         ▼
  ┌──────────────────────────────────────────────────┐
  │                    MERGE                         │
  │                                                  │
  │  new items  ──────────────────────────► merged   │
  │                                          list    │
  │  existing items not in new fetch                 │
  │  (carry-forward) ──────────────────────►         │
  └──────────────────────┬───────────────────────────┘
                         │  combined FeedItem[]
                         ▼
  ┌──────────────────────────────────────────────────┐
  │            RETENTION PASS 1 (inline)             │
  │                                                  │
  │  Carry-forward candidates are pre-filtered:      │
  │  drop unprotected items older than cutoff        │
  │  before they enter the merged list               │
  └──────────────────────┬───────────────────────────┘
                         │
                         ▼
  ┌──────────────────────────────────────────────────┐
  │            RETENTION PASS 2 (applyFeedLimits)    │
  │                                                  │
  │  Time-based:  drop read items older than cutoff  │
  │  Count-based: drop oldest non-protected items    │
  │               beyond the max-items limit         │
  │                                                  │
  │  Protected (starred / saved) → always kept       │
  └──────────────────────┬───────────────────────────┘
                         │  trimmed FeedItem[]
                         ▼
  ┌─────────────┐
  │   PERSIST   │  saveSettings() → data.json
  └──────┬──────┘
         │
         ▼
  ┌─────────────┐
  │   DISPLAY   │  Dashboard reads from in-memory settings;
  │             │  view filters (unread / starred / folder…)
  │             │  are applied on top — no additional items
  │             │  are deleted at display time
  └─────────────┘

Key point: retention happens before the write to disk, not at display time. Items removed by retention are gone from data.json after the next save.


2. Storage

The plugin saves its entire settings object to .obsidian/plugins/obsidian-rss-dashboard/data.json. This includes:

  • All feed configuration (URL, folder, per-feed retention settings, scan interval, keyword rules, etc.)
  • All article items (FeedItem[]) for every feed
  • Global plugin settings (display preferences, tag definitions, highlight words, sidebar state, icon ordering, etc.)

Not stored: raw RSS/XML documents. There is no hidden local XML archive.

  • On startup: the entire settings object is loaded into memory.
  • On every save: saveSettings() overwrites data.json with the full current in-memory state.
  • Single-feed refresh: saves immediately after that feed's merge + retention step.
  • Multi-feed batch refresh: one save at the end of the entire batch, not per-feed.
  • While a multi-feed batch is running, completed feeds update the existing global progress label in place. The sidebar and article list are not rebuilt between feed completions, and image-cache completions produced by the batch do not schedule extra dashboard renders. The batch performs one final dashboard refresh after all feed attempts settle.

SQLite was trialled in 2.2.0-beta.2 and reverted in 2.2.0-beta.4 for cross-platform stability. JSON has been the sole persistence mechanism since then.


3. Fetch and Parse

Each refresh cycle:

  1. Sends an HTTP request to the feed URL.
  2. Parses the raw XML into structured FeedItem objects (title, guid, pubDate, content, media type, cover image, etc.).
  3. Passes the parsed items into the merge step.

Feeds with excludeFromRefresh: true are skipped by bulk and automatic refresh. They can still be refreshed directly from a feed view.

Automatic refresh scheduling

Automatic refresh uses one rearmable timeout. When at least one non-excluded feed has scanInterval: 0 or no value, the global interval schedules one all-eligible-feed refresh batch; its next run is anchored to lastGlobalRefreshCompletedAt. scanInterval: -1 turns automatic scheduling off for that feed, while a positive value uses that feed's own completion time and interval. This means a feed with a positive custom value can still refresh when the global interval is Off.

lastRefreshAttemptCompletedAt is persisted per feed after every completed attempt, including parser-reported failure, thrown error, and timeout. lastUpdated remains the timestamp of the most recent successful parse. The next due time is derived in memory from the completion timestamp and effective interval; it is never persisted. A changed feed URL starts a new refresh identity and clears completion and error history.

Global refresh completion

lastGlobalRefreshCompletedAt is a separate persisted settings value. It advances only after an explicit refresh of the complete eligible feed set has settled, including batches with individual failures. It does not advance for single-feed, folder, selection, due-subset, failed-only, empty, or excluded-only refreshes. The legacy lastRefreshTimestamp remains readable for older data but is not used or updated by refresh behavior or status UI.

When the user stops an automatic global batch, its completion timestamp remains unchanged. The in-memory scheduler defers the next automatic global attempt by one configured global interval, preventing an immediate retry of the stopped batch while preserving the recorded completion time.


4. Merge and Carry-Forward

The merge step combines two sources:

Source What it contains
Freshly fetched items Whatever the remote feed returned this cycle
Carried-forward items Existing locally-stored items that were absent from this fetch

Carry-forward exists so local history is not wiped every time a feed server stops returning older items. The local list accumulates over time — up to the retention limits.

The merge actually has three distinct paths:

For each item guid seen in this refresh:
        │
        ├── exists locally? ─────────────────────────────────────────────────┐
        │                                                                     │
        │   YES: build updatedItem as { ...existingItem, ...freshFields }    │
        │        Fresh content fields (title, description, content,          │
        │        pubDate, coverImage, etc.) are taken from the remote.       │
        │        Local state fields (read, starred, saved, tags,             │
        │        savedFilePath) are preserved from existingItem.             │
        │        → goes into updatedItems[]                                  │
        │                                                                     │
        │   NO:  brand-new item — read:false, starred:false, tags:[]         │
        │        → goes into newItems[]                              ◄───────┘
        │
        └── (after iterating all fetched items)

For each existing local item NOT seen in this refresh:
        │
        ├── older than autoDeleteCutoff AND unprotected? ──► skip (pre-filter)
        │
        └── otherwise ──► carry forward unchanged into carriedForward[]

Final merged list = [...carriedForward, ...updatedItems, ...newItems]

Key implication: refreshing a feed does not reset read state, tags, or starred status. Those fields are always taken from the locally-stored version when the item matches by guid.


5. Retention Rules

Retention runs in two passes during parseFeed, both before the write to disk.

Pass 1 (inline, during carry-forward assembly): Unprotected carry-forward candidates older than autoDeleteCutoffMs are dropped before they enter the merged list at all. This prevents old items from reappearing as unread when a duration is re-enabled after being disabled.

Pass 2 (applyFeedLimits): Runs applyFeedRetentionLimits() on the fully merged list, applying both time-based and count-based rules.

The two passes are separate functions with independent logic. Pass 1 only applies the time-based cutoff to carry-forward items. Pass 2 applies both rules to all items.

5a. Time-Based Cleanup (Auto-Delete Duration)

Removes old, unprotected articles so the feed does not grow indefinitely.

  • When it runs: every feed refresh; also the "Apply feed limits to all feeds" command.
  • What it removes: articles older than the cutoff that do not match an enabled retention protection.
  • Disabled duration: an autoDeleteDuration of 0 disables the time-based pass.
  • Protections: starred and saved articles are protected by default; tagged and unread articles are protected only when their respective global setting is enabled.
For each article in merged list:
        │
        ├── matches an enabled retention protection? ► KEEP
        │
        ├── pubDate within cutoff window? ──────────► KEEP
        │
        └── older than cutoff + unprotected ─────────► REMOVE

5b. Count-Based Cleanup (Max Items Limit)

Enforces a hard cap on the number of non-protected articles per feed. This is to maintain optimal plugin performance by automatically removing unwanted feeds.

  • Keeps the newest N non-protected items (N = maxItemsLimit).
  • Protected items are excluded from the count and never removed.
  • Any article state can be protected when its corresponding global protection setting is enabled.
Non-protected items sorted newest → oldest:
  [1] [2] [3] … [N]  ← kept
  [N+1] [N+2] …      ← removed

Protected items: always appended, never counted

5c. What Counts as "Protected"?

Protection is determined solely by isProtectedItem() and the current global protection settings:

function isProtectedItem(
  item: FeedItem,
  protections: FeedRetentionProtections,
): boolean {
  return (
    (protections.protectStarred && item.starred) ||
    (protections.protectSaved && item.saved) ||
    (protections.protectTagged && item.tags?.length > 0) ||
    (protections.protectUnread && !item.read)
  );
}

The default policy protects starred and saved articles. Tagged and unread articles are unprotected by default, but users can enable either protection in General Settings. Matching any enabled protection shields an article from both retention passes.

5d. Retention Summary

Article state Default protection When its protection is enabled
Starred 🟢 Protected 🟢 Protected
Saved 🟢 Protected 🟢 Protected
Tagged 🔴 Unprotected 🟢 Protected
Unread 🔴 Unprotected 🟢 Protected
Other 🔴 Unprotected 🔴 Unprotected

An unprotected article is removed when it is past the active age cutoff or falls beyond the newest unprotected maxItemsLimit items. An article matching any enabled protection is retained by both passes.


6. Display and View Filters

The dashboard reads articles from the in-memory settings object (which mirrors data.json). View filters — unread, read, starred, saved, folder, tags — are applied at render time to decide which articles to show.

View filters are not retention. They do not remove articles from storage. An article hidden by the unread filter is still in data.json; switching the filter back reveals it.

data.json (all stored articles)
        │
        ▼
  in-memory settings
        │
        ├── view filter: unread  ──► show only !item.read
        ├── view filter: starred ──► show only item.starred
        ├── folder scope         ──► show only feeds in folder
        └── tag filters          ──► show only matching tags

Articles not matching the filter are hidden, not deleted.

7. If I Change the Auto-Delete Setting, What Happens?

The answer depends on where you change the setting.

Per-Feed Auto-Delete Duration (Edit Feed modal)

When you change autoDeleteDuration in the Edit Feed modal and click Save, the plugin:

  1. Writes the new value to the in-memory feed object and saves to data.json.
  2. Immediately triggers a single-feed refresh (refreshSelectedFeed) — the full fetch + merge + retention pipeline runs right away.

The new cutoff is enforced before you close the modal. You do not need to wait for the next scheduled refresh.

Global Default Auto-Delete Duration (Settings tab)

Changing defaultAutoDeleteDuration in the plugin settings tab saves the new value to data.json. A longer duration or disabled auto-deletion applies immediately and affects future refreshes. Shortening an active duration, or changing from disabled to an active duration, opens a confirmation dialog.

Tightening (e.g. 90 days → 30 days) or Re-enabling

For a global setting, users can choose Apply now, Apply on next refresh, or Cancel. Apply now saves the setting and applies retention to all existing feeds, including refreshing the active dashboard view. Apply on next refresh saves the setting without pruning cached articles immediately. Cancel restores the prior control value and does not save.

Loosening (e.g. 30 days → 90 days) or Disabling

The setting change alone does not restore any articles. Articles that were already pruned from data.json are gone. On the next refresh, the plugin will carry forward whatever is still in local storage and will no longer remove items that are now within the wider window — but nothing is un-deleted.

Global Retention Protections

Turning an enabled retention protection off follows the same confirmation flow as shortening the duration. Turning a protection on saves immediately. The protection choices apply globally to starred, saved, tagged, and unread article states, and to both age- and count-based retention.

Old articles only reappear if:

  • The remote feed still publishes them and returns them on the next fetch, or
  • They were still present in data.json when the setting was changed (i.e. they had not yet been pruned).

8. Implementation Reference

Concern Location
Merge (all three paths) parseFeed() — src/services/feed-parser.ts
Carry-forward pre-filter (pass 1) inline inside parseFeed() — src/services/feed-parser.ts
Final dedup + concat mergeFeedHistoryItems() — src/services/feed-parser.ts
Retention pass 2 applyFeedRetentionLimits() via applyFeedLimits() — src/services/feed-parser.ts
Refresh orchestration (single feed) refreshSingleFeed() — main.ts
Refresh orchestration (batch) refreshFeedBatch() — main.ts
Exclude from refresh getRefreshableFeeds() — main.ts
Persist to disk saveSettings() → this.saveData(this.settings) — main.ts
Metadata storage (vault location) loadMetadata(), ensureMetadataFolderExists() — main.ts
Metadata migration migrateMetadataToVaultLocation(), revertMetadataToPluginDefault() — main.ts
View filter application getFilteredArticles() / getUnfilteredArticles() — src/views/dashboard-view.ts

9. Metadata Storage (Dual-Mode Persistence)

Overview

In addition to feed item storage, the plugin can now store metadata (settings/data.json) in two locations:

  • Plugin Default: <vault>/.obsidian/plugins/rss-dashboard/data.json (isolated, no sync)
  • Vault Location: User-defined path, e.g. <vault>/.rss-dashboard-data/data.json (synced with vault)

This allows users to sync metadata across devices while keeping it separate from feed shards.

Load and Save Flow

On plugin load:

  1. Check settings.metadataStorageMode
  2. If "vault-location": Try to load from vault location first
  3. Fall back to plugin-default if vault location unavailable
  4. Merge with default settings using Object.assign()

On settings save:

  1. Determine save target based on current metadataStorageMode
  2. If "vault-location": - Call ensureMetadataFolderExists() to create folder if needed (idempotent) - Write to vault location using vault.adapter.write()
  3. If "plugin-default": - Use Plugin.saveData() (Obsidian's default)

Migration and Revert

Migrate to Vault Location:

  1. Ensure target folder exists (create if needed)
  2. Load current metadata from plugin-default
  3. Write to vault location
  4. Update settings.metadataStorageMode to "vault-location"
  5. Save settings (which now writes to vault location)
  6. On error: revert mode to "plugin-default" (no partial state)

Revert to Plugin Default:

  1. Load current metadata from vault location
  2. Update settings.metadataStorageMode to "plugin-default"
  3. Save settings using Plugin.saveData()
  4. Optionally clean up vault location file
  5. On error: keep metadata where it is (no partial state)

Backup Integration

The BackupService follows metadata storage location:

  • Async backups check metadataStorageMode and back up from the active location
  • Sync backups (desktop) resolve the absolute path and back up accordingly
  • Portable data bundles include metadataStorageMode and metadataStorageFolder fields

Import/Export

When importing a portable data bundle:

  1. Extract metadataStorageMode and metadataStorageFolder from bundle
  2. Back up current metadata before import
  3. Restore metadata according to bundle's configuration
  4. On import failure: restore backup (no partial state)

Implementation Details

  • File I/O: Uses vault.adapter for vault-location (mobile-compatible), Plugin.saveData() for plugin-default
  • Error Handling: All state changes are transactional — revert on any error
  • Idempotency: Folder creation handles race conditions and existing folders
  • Type Safety: Uses instanceof TFolder checks (not isFolder() which doesn't exist)