Skip to content

fix: dynamic queryId & features resolution from X search page - #83

Open
gnuhpc wants to merge 1 commit into
public-clis:mainfrom
gnuhpc:fix/dynamic-queryid-features-from-search-page
Open

fix: dynamic queryId & features resolution from X search page#83
gnuhpc wants to merge 1 commit into
public-clis:mainfrom
gnuhpc:fix/dynamic-queryid-features-from-search-page

Conversation

@gnuhpc

@gnuhpc gnuhpc commented Aug 10, 2026

Copy link
Copy Markdown

Problem

After X rotated all GraphQL queryIds and redesigned its homepage, twitter search returns HTTP 404, user-posts returns empty results, and feed fails intermittently.

Related issues: #78, #73, #69

This PR complements #79 (cookie fix) and #82 (fallback ID refresh) — but addresses the root cause more comprehensively.

Root Causes

1. Homepage no longer contains JS bundles with queryId definitions

X redesigned its homepage to use React Router (TSR). The new homepage returns a minimal HTML with entry-client-*.js (a Vite bootstrap loader) instead of the legacy main.*.js that contains queryId/operationName mappings.

2. _scan_bundles fetches the wrong page

The function fetches https://x.com (homepage) to find JS bundles, but the new homepage only has x-web/x-web/ URLs, which the existing regex does not match.

3. GitHub twitter-openapi queryIds are also stale

_resolve_query_id falls back to _fetch_from_github(), but the community-maintained placeholder.json is also frequently behind X's rotation schedule.

4. Features dict is incomplete

Default FEATURES has only 20 keys, but SearchTimeline requires 39 specific feature flags. Missing flags cause silent API failures.

Fix

Scan X search page instead of homepage

The search page has NOT been redesigned — it still uses responsive-web/client-web/main.*.js with all 106 queryId definitions.

QueryId resolution priority

  1. Live bundle scan (from search page main.js) — most reliable
  2. GitHub twitter-openapi — community fallback
  3. Hardcoded FALLBACK_QUERY_IDS — last resort

Dynamic features from GitHub

Merge all feature flags from twitter-openapi placeholder.json at import time (non-blocking on failure). This brings FEATURES from 20 to 65+ keys.

Match both URL patterns

Added x-web/x-web to the regex alongside responsive-web/client-web.

Recursive chunk scanning

New X frontend uses __vite__mapDeps for dynamic chunk loading. Extract chunk filenames from the entry bundle and scan them too.

Verification

After applying this PR:

  • twitter search "AI" returns results (was 404)
  • twitter user-posts karpathy returns tweets (was empty)
  • twitter feed returns timeline
  • twitter post and twitter quote were unaffected and still work

X rotated all GraphQL queryIds and the homepage no longer contains
the ondemand.s file reference. This PR fixes multiple root causes:

1. Scan X search page instead of homepage for JS bundles
   - Search page (x.com/search?q=AI&f=live) still uses legacy frontend
   - Contains main.*.js with all 106 queryId definitions
   - Homepage uses new React Router (TSR) with no queryId refs

2. Resolve queryId: scan bundles first, then GitHub, then fallback
   - GitHub twitter-openapi is also frequently stale
   - Live bundle scanning is the most reliable source

3. Dynamically merge features from GitHub twitter-openapi
   - Default FEATURES dict has only 20 keys
   - GitHub has 65+ feature flags needed by SearchTimeline etc.
   - Merged at import time, non-blocking on failure

4. Match both responsive-web and x-web JS URL patterns
   - Old: abs.twimg.com/responsive-web/client-web/*.js
   - New: abs.twimg.com/x-web/x-web/*.js

5. Recursively scan entry-client JS for chunk file references
   - New X frontend uses __vite__mapDeps for dynamic chunks
   - Extract chunk filenames from entry bundle and scan them too

Fixes public-clis#78, public-clis#73, public-clis#69
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant