Add X like velocity to the paper trending score

.agents/skills/pwc-cron-health/references/job-probes.md

@@ -220,6 +220,13 @@ For every managed leaderboard sync, fetch the current upstream source read-only
 - **Independent proof:** Open one matched post from the JSON and confirm it is a paper author or the paper's own organization (business-verified account) introducing that paper, never an outside account; with apply on, confirm the paper page's X threads card shows it.
 - **Correctness:** `X search HTTP 401/403` means a bad or unentitled token and `429` an exhausted quota (the run stops at the first one). A zero `papers_scanned` means every recent candidate already has threads or used its three scans.
 
+## `refresh_x_post_likes`
+
+- **Effect:** Hourly at :35, adds an `x_post_metrics` row for every `paper_x_threads` post from the last 7 days, then reads each due post (about 1, 2, 4, 8, 12, 24, 48, 72, 120 and 144 h after posting) and updates its likes-per-hour EMA (`like_velocity`). Syndication is tried first; posts it cannot serve use the paid X API (`X_API_BEARER_TOKEN`), capped at `PWC_X_LIKES_API_DAILY_CAP` (default 300) posts per UTC day, tracked in `x_post_likes_api_usage.json` under `PWC_CRON_LOCK_DIR`. Posts the API reports unavailable get `gone_at`. Report-only (reads through syndication and rolls back) unless `PWC_X_LIKES_APPLY=true`.
+- **Freshness metrics** (dashboard Database Metrics): `last_write` = latest `x_post_metrics.last_read_at`, `tracked_posts` = posts with a pending reading. Report-only runs write nothing, so both stay empty until apply is on.
+- **Independent proof:** Pick a recently read post (`ORDER BY last_read_at DESC`), open it on X, and confirm its like count is close to `like_count`; then confirm the paper's `x_trending_component` is set after the next `calculate_paper_trending_scores` run.
+- **Correctness:** The JSON summary lists `syndication_reads`, `api_reads`, `api_reads_today`, `unread` and `api_error`. `api_error` with HTTP 401/402 means a bad token or exhausted X credits and exits 1; many `unread` with `api_reads_today` at the cap means syndication is failing and the cap is doing its job. The X part only affects rankings when `PWC_TRENDING_X_WEIGHT` in `/etc/paperswithcode/trending-cron.env` is above 0.
+
 ## `link_google_scholar_profiles`
 
 - **Effect:** Checks Wikidata Scholar-ID candidates (P1960) for authors with 3+ papers: each new author once on Semantic Scholar (ORCID / S2 author ID on 15 sampled papers) into `author_google_scholar_candidates`, then spends the SerpAPI searches left this month (minus 10) reading Scholar profiles of S2-matched candidates, largest authors first, and upserts confirmed links into `author_google_scholar_profiles`; never deletes links.

.github/workflows/scripts/README.md

@@ -36,6 +36,7 @@ Generated from `deploy/cron_schedules.py`; update this table when you change it.
 | `pwc-cron-link-google-scholar-profiles.timer` | `Wed *-*-* 05:20:00` | `link_google_scholar_profiles` | `backend/scripts/link_google_scholar_profiles.py` |
 | `pwc-cron-refresh-conference-deadlines.timer` | `*-*-* 07:40:00`<br>`*-*-* 19:40:00` | `refresh_conference_deadlines` | `backend/scripts/refresh_conference_deadlines.py` |
 | `pwc-cron-discover-x-threads.timer` | `*-*-* 08,20:22:00` | `discover_x_threads` | `backend/scripts/discover_x_threads.py` |
+| `pwc-cron-refresh-x-post-likes.timer` | `*-*-* *:35:00` | `refresh_x_post_likes` | `backend/scripts/refresh_x_post_likes.py` |
 | `pwc-cron-post-trending-paper-to-x.timer` | `*-*-* 00/4:13:00` | `post_trending_paper_to_x` | `backend/scripts/post_trending_paper_to_x.py` |
 | `pwc-cron-post-x-insight.timer` | `*-*-* 09,17:43:00` | `post_x_insight` | `backend/scripts/post_x_insight.py` |
 | `pwc-cron-warm-paper-suggested-questions.timer` | `*-*-* *:33:00` | `warm_paper_suggested_questions` | `backend/scripts/warm_paper_suggested_questions.py` |
@@ -182,6 +183,24 @@ history and stores them on repositories for the trending API/view pipeline.
 `update_citations.py` pulls citation counts from Semantic Scholar for ArXiv
 papers. Runs are bounded so hourly runs do not pile up under API throttling.
 
+`refresh_x_post_likes.py` tracks how fast linked X posts gain likes, for the
+X part of the paper trending score. Hourly at :35 it adds an `x_post_metrics`
+row for each `paper_x_threads` post from the last week, then reads each post
+about 1, 2, 4, 8, 12, 24, 48, 72, 120 and 144 hours after posting and keeps an
+EMA of likes per hour. Reads use the free embed syndication endpoint first and
+fall back to the paid X API (`X_API_BEARER_TOKEN`) for at most
+`PWC_X_LIKES_API_DAILY_CAP` posts per UTC day (default 300). It only reports,
+reading through syndication and rolling back, unless `PWC_X_LIKES_APPLY=true`.
+`calculate_paper_trending_scores` ranks each paper's fastest post from the last
+week against the others (posts under `PWC_TRENDING_X_MIN_LIKES_PER_HOUR`,
+default 1, count for nothing) and weighs it by `PWC_TRENDING_X_WEIGHT` from
+`/etc/paperswithcode/trending-cron.env` (default 0; GitHub and Hugging Face
+share the rest equally).
+
+```bash
+uv run --env-file backend/keys.env python -m backend.scripts.refresh_x_post_likes --max-posts 20
+```
+
 `discover_x_threads.py` finds X posts in which authors introduce recent
 papers and links them to the paper page's "X threads" card. Twice a day it
 takes up to 40 arXiv papers from the last 10 days that are trending or featured

.github/workflows/scripts/hourly/calculate_paper_trending_scores.py

@@ -3,14 +3,17 @@
 
 Computes a paper-level trending score that blends the existing GitHub star
 velocity EMA (``repositories.trending_score``, stars/hour) with Hugging Face
-artifact Hub trending scores (``hf_artifacts.trending_score``). The formula
-lives in ``backend/utils/trending.py`` so it is unit-testable without a DB.
+artifact Hub trending scores (``hf_artifacts.trending_score``) and, when
+``PWC_TRENDING_X_WEIGHT`` is above 0, the like velocity of the paper's fastest
+linked X post (``x_post_metrics.like_velocity``). The formula lives in
+``backend/utils/trending.py`` so it is unit-testable without a DB.
 
 Pipeline:
   1. Load the scoring population: latest-version papers published in the last
-     year, with their best official-repo velocity and linked HF artifact scores.
-  2. Derive corpus normalization scales (95th percentiles) for GitHub velocity
-     and per-kind HF trending scores.
+     year, with their best official-repo velocity, linked HF artifact scores,
+     and fastest linked X post from the last week.
+  2. Derive the quantile-rank corpora for GitHub velocity, per-kind HF
+     trending scores, and X like velocity.
   3. Compute the composite score per paper and persist it to
      ``papers.trending_score`` (+ debug components). Papers with no live signal
      are cleared to NULL so they drop out of the composite feed.
@@ -32,14 +35,18 @@
 
 from backend.database.connection import db_manager
 from backend.utils.trending import (
+    DEFAULT_X_MIN_LIKES_PER_HOUR,
+    DEFAULT_X_WEIGHT,
     KIND_WEIGHTS,
+    X_POST_WINDOW_DAYS,
     apply_decaying_pin,
     compute_paper_score,
     decaying_pin_floor,
     is_hf_aggregator,
     parse_trending_paper_id_pin_specs,
     parse_trending_pin_specs,
     sorted_positive,
+    x_corpus,
 )
 
 # Score papers published within this window (bounds the work; older papers fall
@@ -64,13 +71,25 @@ def log(message: str):
     print(f"{timestamp} - {message}", flush=True)
 
 
+def x_settings() -> tuple[float, float]:
+    """Return the configured ``(x_weight, x_min_likes_per_hour)``."""
+    weight = float(os.getenv("PWC_TRENDING_X_WEIGHT") or DEFAULT_X_WEIGHT)
+    if not 0.0 <= weight < 1.0:
+        raise ValueError("PWC_TRENDING_X_WEIGHT must be in [0, 1)")
+    min_likes = float(
+        os.getenv("PWC_TRENDING_X_MIN_LIKES_PER_HOUR") or DEFAULT_X_MIN_LIKES_PER_HOUR
+    )
+    return weight, min_likes
+
+
 def load_population(session, cutoff):
-    """Return ``{paper_id: {"github_raw", "existing", "artifacts"}}``.
+    """Return ``{paper_id: {"github_raw", "existing", "artifacts", "x_raw"}}``.
 
     - ``github_raw``: best fresh official-repo stars/hour (0.0 if none).
     - ``existing``: whether the paper currently has a non-null trending_score.
     - ``artifacts``: list of ``(kind, trending_score)`` for linked HF artifacts,
       excluding aggregators (see ``is_hf_aggregator``).
+    - ``x_raw``: likes/hour of the fastest linked X post from the last week.
     """
     log("Loading paper population (GitHub component)...")
     github_rows = session.execute(
@@ -102,6 +121,7 @@ def load_population(session, cutoff):
             "github_raw": float(row.github_raw) if row.github_raw is not None else 0.0,
             "existing": bool(row.existing),
             "artifacts": [],
+            "x_raw": 0.0,
         }
     log(f"Population size: {len(population)} papers")
 
@@ -143,6 +163,31 @@ def load_population(session, cutoff):
     log(f"HF artifact score links: {hf_links}")
     log(f"HF aggregator links ignored: {aggregator_links}")
 
+    log("Loading X like velocity component...")
+    x_rows = session.execute(
+        text("""
+            SELECT t.paper_id, MAX(m.like_velocity) AS x_raw
+            FROM paper_x_threads t
+            JOIN x_post_metrics m ON m.tweet_id = t.tweet_id
+            JOIN papers p ON p.id = t.paper_id
+            WHERE p.is_latest_version = TRUE
+              AND p.date_published >= :cutoff
+              AND m.gone_at IS NULL
+              AND m.like_velocity > 0
+              AND m.posted_at >= NOW() - (:window_days || ' days')::interval
+            GROUP BY t.paper_id
+        """),
+        {"cutoff": cutoff, "window_days": str(X_POST_WINDOW_DAYS)},
+    ).fetchall()
+    x_papers = 0
+    for row in x_rows:
+        entry = population.get(row.paper_id)
+        if entry is None:
+            continue
+        entry["x_raw"] = float(row.x_raw)
+        x_papers += 1
+    log(f"Papers with a live X post: {x_papers}")
+
     return population
 
 
@@ -168,7 +213,16 @@ def _summ(values):
     return github_sorted, sorted_by_kind
 
 
-def build_score_changes(population, github_sorted, sorted_by_kind, paper_ids=None):
+def build_score_changes(
+    population,
+    github_sorted,
+    sorted_by_kind,
+    paper_ids=None,
+    *,
+    x_sorted=(),
+    x_weight=DEFAULT_X_WEIGHT,
+    x_min_likes_per_hour=DEFAULT_X_MIN_LIKES_PER_HOUR,
+):
     """Compute updates/clears for all papers or an explicit target set."""
     selected = sorted(population) if paper_ids is None else sorted(set(paper_ids))
     updates = []
@@ -177,19 +231,25 @@ def build_score_changes(population, github_sorted, sorted_by_kind, paper_ids=Non
         entry = population.get(paper_id)
         if entry is None:
             continue
-        score, github_component, hf_component = compute_paper_score(
+        score, github_component, hf_component, x_component = compute_paper_score(
             entry["github_raw"],
             github_sorted,
             entry["artifacts"],
             sorted_by_kind,
+            x_raw=entry["x_raw"],
+            x_sorted=x_sorted,
+            x_weight=x_weight,
+            x_min_likes_per_hour=x_min_likes_per_hour,
         )
-        if github_component > 0 or hf_component > 0:
+        # With X weighted at 0, an X-only paper scores 0 and stays unscored.
+        if score > 0:
             updates.append(
                 {
                     "id": paper_id,
                     "score": round(score, 4),
                     "g": round(github_component, 4),
                     "h": round(hf_component, 4),
+                    "x": round(x_component, 4),
                 }
             )
         elif entry["existing"]:
@@ -239,11 +299,19 @@ def calculate_paper_scores(session, paper_ids=None) -> dict:
         return {"population": 0, "selected": 0, "written": 0, "cleared": 0}
 
     github_sorted, sorted_by_kind = derive_corpora(population)
+    x_weight, x_min_likes_per_hour = x_settings()
+    x_sorted = x_corpus(
+        (entry["x_raw"] for entry in population.values()), x_min_likes_per_hour
+    )
+    log(f"X like velocity corpus: n={len(x_sorted)}, weight={x_weight}")
     updates, clears = build_score_changes(
         population,
         github_sorted,
         sorted_by_kind,
         paper_ids=paper_ids,
+        x_sorted=x_sorted,
+        x_weight=x_weight,
+        x_min_likes_per_hour=x_min_likes_per_hour,
     )
     if paper_ids is None:
         # The population only covers the window, so aged-out scores would
@@ -270,6 +338,7 @@ def write_scores(session, updates, clears):
                 SET trending_score = :score,
                     github_trending_component = :g,
                     hf_trending_component = :h,
+                    x_trending_component = :x,
                     trending_score_updated_at = NOW(),
                     updated_at = NOW()
                 WHERE id = :id
@@ -289,6 +358,7 @@ def write_scores(session, updates, clears):
                 SET trending_score = NULL,
                     github_trending_component = NULL,
                     hf_trending_component = NULL,
+                    x_trending_component = NULL,
                     trending_score_updated_at = NOW(),
                     updated_at = NOW()
                 WHERE id = ANY(:ids)

backend/database/migrations/add_x_post_metrics.sql

@@ -0,0 +1,60 @@
+-- Migration: X post like velocity for the paper trending score
+--
+-- `x_post_metrics` keeps one row per X post linked to a paper (a post can be
+-- linked to several papers), with its latest like count and smoothed likes per
+-- hour. The `refresh_x_post_likes` CRON reads each post on a decaying schedule
+-- during its first week; `calculate_paper_trending_scores` reads the velocity.
+-- `papers.x_trending_component` stores the normalized X part of the score.
+--
+-- Run as the schema owner (postgres) for `public` before merging.
+-- Idempotent: safe to re-run.
+
+BEGIN;
+
+-- Adding a nullable column is instant, but it needs a brief exclusive lock on
+-- the busy papers table; fail fast instead of queueing behind long readers.
+SET LOCAL lock_timeout = '10s';
+
+CREATE TABLE IF NOT EXISTS public.x_post_metrics (
+    tweet_id VARCHAR(32) PRIMARY KEY,
+    posted_at TIMESTAMPTZ NOT NULL,
+    like_count INTEGER,
+    like_velocity DOUBLE PRECISION,
+    last_read_at TIMESTAMPTZ,
+    last_source VARCHAR(16),
+    read_count INTEGER NOT NULL DEFAULT 0,
+    next_read_at TIMESTAMPTZ,
+    gone_at TIMESTAMPTZ,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS ix_x_post_metrics_next_read_at
+    ON public.x_post_metrics (next_read_at)
+    WHERE next_read_at IS NOT NULL;
+
+ALTER TABLE public.papers
+    ADD COLUMN IF NOT EXISTS x_trending_component NUMERIC(12, 4);
+
+GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE public.x_post_metrics
+    TO paperswithcode_user;
+
+DO $$
+BEGIN
+    IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'pwc_staging') THEN
+        GRANT SELECT ON TABLE public.x_post_metrics TO pwc_staging;
+    END IF;
+    -- Same grants deploy/configure_db_roles.py gives the isolated scoring role.
+    IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'pwc_cron_trending') THEN
+        GRANT SELECT ON TABLE public.paper_x_threads, public.x_post_metrics
+            TO pwc_cron_trending;
+        GRANT UPDATE (x_trending_component) ON public.papers TO pwc_cron_trending;
+    END IF;
+END
+$$;
+
+COMMENT ON TABLE public.x_post_metrics IS
+'Like counts and smoothed likes per hour of linked X posts, from the refresh_x_post_likes CRON.';
+COMMENT ON COLUMN public.papers.x_trending_component IS
+'Quantile rank (0-1) of the paper''s fastest linked X post in the trending score.';
+
+COMMIT;

backend/models/database/__init__.py

@@ -175,6 +175,7 @@
     PaperXTrending,
     XInsightPost,
     PaperXThreadScan,
+    XPostMetrics,
     PaperRepository,
 )
 from backend.models.database.taxonomy import (
@@ -332,6 +333,7 @@
     "PaperXTrending",
     "XInsightPost",
     "PaperXThreadScan",
+    "XPostMetrics",
     "PaperRepository",
     "Area",
     "Task",

backend/models/database/papers.py

@@ -149,6 +149,9 @@ class Paper(Base):
     hf_trending_component: Mapped[Optional[float]] = mapped_column(
         Numeric(12, 4), nullable=True
     )
+    x_trending_component: Mapped[Optional[float]] = mapped_column(
+        Numeric(12, 4), nullable=True
+    )
     # When this paper was last scanned for linked Hugging Face artifacts.
     hf_artifacts_scanned_at: Mapped[Optional[datetime]] = mapped_column(
         DateTime(timezone=True), nullable=True

backend/models/database/resources.py

@@ -9,6 +9,7 @@
     CheckConstraint,
     Date,
     DateTime,
+    Float,
     ForeignKey,
     Index,
     Integer,
@@ -439,6 +440,28 @@ class PaperXThreadScan(Base):
     linked_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True))
 
 
+class XPostMetrics(Base):
+    """Like velocity of one linked X post, refreshed during its first week."""
+
+    __tablename__ = "x_post_metrics"
+
+    tweet_id: Mapped[str] = mapped_column(String(32), primary_key=True)
+    posted_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+    like_count: Mapped[Optional[int]] = mapped_column(Integer)
+    # Smoothed likes per hour (EMA over readings).
+    like_velocity: Mapped[Optional[float]] = mapped_column(Float)
+    last_read_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True))
+    # "syndication" or "api".
+    last_source: Mapped[Optional[str]] = mapped_column(String(16))
+    read_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    # NULL once the reading schedule is finished or the post is gone.
+    next_read_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True))
+    gone_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True))
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), server_default=func.now(), nullable=False
+    )
+
+
 class PaperXPost(Base):
     """A post on the Papers with Code X account featuring one paper."""