Skip to content

feat(feeds): add curated RSS feed review#25

Open
technowizard wants to merge 18 commits into
mainfrom
feature/rss-feed-support
Open

feat(feeds): add curated RSS feed review#25
technowizard wants to merge 18 commits into
mainfrom
feature/rss-feed-support

Conversation

@technowizard

Copy link
Copy Markdown
Owner

Summary

Adds user-curated RSS and Atom subscriptions to Loreo, with a calm review workflow that keeps discovered articles separate from the saved library until the reader acts.

Feeds are a recurring subscription domain rather than one-time imports. New entries stage in Review by default, while optional per-feed Auto-save applies only to entries first discovered after it is enabled. Saving creates or reuses the reader's existing Loreo article and queues extraction without duplicating links.

Architecture

flowchart TB
  UI[Feeds and Manage Feeds UI] --> API[Authenticated feed API]
  API --> Ingestion[Feed ingestion service]
  Scheduler[BullMQ recurring scheduler] --> Worker[Per-feed poll worker]
  Worker --> Ingestion
  Ingestion --> Fetch[DNS-pinned fetch and parser]
  Fetch --> Publisher[Public RSS or Atom publisher]
  Ingestion --> Items[(feed_subscriptions and feed_items)]
  Ingestion --> Save[Duplicate-aware link save]
  Save --> Links[(saved links)]
  Save --> Extraction[Content extraction queue]
Loading

What changed

Area End state
Review URL-backed Review, Saved, and Dismissed collections with source filtering, server-side sorting, cursor infinite loading, and focus-safe row virtualization
Management Responsive Manage Feeds dialog with search, health filters, immediate pause/Auto-save controls, source summaries, and confirmed deletion that preserves saved articles
Ingestion RSS 1.0/2.0 and Atom parsing, newest-50 initial staging, GUID/URL deduplication, conditional requests, failure backoff, and 90-day/500-item retention
Polling Redis-backed recurring due-feed scans and lifecycle-deduplicated per-feed jobs, without delaying HTTP startup when Redis is unavailable
Safety Redirect-safe DNS pinning, public-address enforcement, decompression and size limits, parser/DNS work budgets, user-scoped APIs, and database ownership constraints
Navigation Feeds added to desktop and mobile navigation with English and Indonesian copy

Key decisions

  • Review remains a discovery surface, not a full RSS reader. There is no read/unread or folder state.
  • Auto-save is future-only. Enabling it never saves the existing Review backlog, and dismissed identities remain dismissed.
  • Feed deletion removes subscription history but leaves independently saved Loreo articles intact.
  • Feed item covers and source logos load directly from publishers by product choice, with local fallbacks for missing or failed images.
  • Migration 0005 installs ownership and state constraints as NOT VALID, enforcing new writes while allowing operators to audit legacy inconsistencies before validation.

Verification

Gate Result
Server suite passed, 36 files / 275 tests
Web suite passed, 28 files / 135 tests
Workspace typecheck passed
Production build passed
Lint and formatting passed
Frozen lockfile install passed
Drizzle schema drift no changes
Fresh correctness, security, data, and standards reviews no actionable findings

Authenticated disposable-account QA covered valid and blocked additions, mixed-source review, 78-79 item pagination, keyboard focus under virtualization, save/dismiss/reconciliation, recurring scheduler execution, failure warnings, desktop/mobile manager history, pending-add locking, publisher images, toggles, and destructive deletion with saved-article preservation.

Operations

Database migrations run at server startup. Migrations 0004 and 0005 use standard index creation and should be applied during a low-traffic maintenance window for large installations. docs/SELF_HOSTING.md includes backup, audit, monitoring, validation, and rollback guidance for migration 0005.

Add the throwaway RSS review prototype used to settle the source-grouped
shelves direction before production implementation.
Add feed subscription and feed item tables, repositories, migrations, and
repository tests for the first RSS support slice.
Add guarded feed fetching and RSS/Atom normalization so future ingestion can
accept user-provided feed URLs without bypassing Loreo's public HTTP(S) URL
safety posture.
Route manual article saves through a reusable link-save service so RSS ingestion
can reuse saved links, enqueue extraction only for new links, and reconcile
matching staged feed items without changing duplicate URL responses.
Add the service layer for subscribing to feeds and polling existing feeds,
including initial staging, auto-save, conditional metadata updates, failure
backoff, and feed item retention pruning.
Add the feed poll queue and worker for scheduled and manual feed refreshes,
including due-subscription enqueueing, not-yet-due skips, ingestion delegation,
and graceful server shutdown registration.
Expose protected feed subscription and item endpoints for adding feeds,
refreshing subscriptions, reviewing staged items, and saving or dismissing
feed items through the existing ingestion and link-save seams.
Add the typed web API layer, feed query keys, protected route skeleton,
and localized copy for the Feeds review surface.
Replace the feeds skeleton with the production review surface: add-feed
form, source-grouped shelves, responsive masonry layout, and feed item
actions for save, dismiss, and refresh.
Add cursor pagination and per-subscription summaries for scalable feed review,
including forward migrations for existing installations and pagination indexes.
Integrate feeds into navigation, add scalable mixed-source review collections,
and provide responsive Manage Feeds and Add Feed workflows.
Describe feed review, polling, retention, and self-hosting behavior for the new
subscription workflow.
Schedule due-feed scans autonomously, pin validated DNS addresses across
redirects, and preserve exact cursor ordering. Keep migration and retention
behavior safe for existing installations.
Add confirmed user-scoped feed deletion while preserving saved articles.
Simplify management hierarchy, recovery states, image fallbacks, and
focus-safe collection virtualization across desktop and mobile.
@technowizard technowizard self-assigned this Jul 18, 2026
Rename unpublished migration and journal tag for readable deployment history.
Preserve migration ordering and SQL behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant