Skip to content

Reduce Apify to Facebook trail status only#559

Open
fatherlinux wants to merge 3 commits into
masterfrom
feature/551-reduce-apify-to-trail-status
Open

Reduce Apify to Facebook trail status only#559
fatherlinux wants to merge 3 commits into
masterfrom
feature/551-reduce-apify-to-trail-status

Conversation

@fatherlinux

Copy link
Copy Markdown
Member

Summary

  • Remove Apify from the news collection pipeline (renderPage.js) — Facebook/Instagram URLs from Serper results now fall through to Playwright/Readability or snippet recovery like any other URL
  • Strip Instagram scraper, isSocialUrl, fetchSocialPosts from apifyService.js
  • Remove social_apify_collection_enabled admin setting
  • Update frontend help text to reflect reduced scope
  • Trim unit tests to Facebook-only

What stays

  • fetchFacebookPosts + isFacebookUrl — used by trailStatusService.js for Reagan-Huffman MTB (POI 5680, medinaTRAILS Facebook page)
  • apify_api_token admin setting + test endpoint
  • Apify token config UI in Settings > Data Collection

Impact

  • Eliminates Facebook Posts Scraper ($16/mo) and Instagram Scraper usage from the daily 19-POI news collection job
  • Only remaining Apify usage: 1 trail status POI checking medinaTRAILS Facebook page
  • Expected savings: ~$25/month

Closes #551 (partially — Apify retained for trail status)

🤖 Generated with Claude Code

Remove Apify from the news collection pipeline (renderPage.js) and strip
Instagram support. Only Reagan-Huffman MTB trail status still uses
Apify's Facebook scraper via trailStatusService. Saves ~$25/month by
eliminating Facebook Posts Scraper and Instagram Scraper usage from the
daily 19-POI news collection job.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes Instagram scraping support and simplifies the Apify service to focus solely on Facebook trail status scraping. This involves removing Instagram-related constants, helper functions, tests, and the associated admin setting, as well as removing the Apify integration from the page rendering service. The review feedback recommends adding defensive checks in fetchFacebookPosts to prevent potential TypeError crashes: specifically, verifying that statusUrl is a string before processing, and ensuring that the result from runActorSync is an array before calling .map() on it.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +70 to 74
const target = extractFacebookPageUrl(statusUrl);
if (!target) {
console.log(`[Apify] Could not extract Facebook page from: ${statusUrl}`);
return { markdown: null, reachable: false, reason: 'invalid Facebook URL' };
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive check: statusUrl is passed to extractFacebookPageUrl(statusUrl) which calls url.match(...). If statusUrl is null, undefined, or not a string, this will throw an unhandled TypeError because it is outside the try-catch block. Adding a type check ensures the function fails gracefully.

  if (typeof statusUrl !== 'string') {
    console.log(`[Apify] Invalid statusUrl: ${statusUrl}`);
    return { markdown: null, reachable: false, reason: 'invalid Facebook URL' };
  }

  const target = extractFacebookPageUrl(statusUrl);
  if (!target) {
    console.log(`[Apify] Could not extract Facebook page from: ${statusUrl}`);
    return { markdown: null, reachable: false, reason: 'invalid Facebook URL' };
  }

Comment on lines +79 to 83
const items = await runActorSync(FACEBOOK_ACTOR_ID, { startUrls: [{ url: target }], maxPosts: maxItems }, token);
if (!items || items.length === 0) {
console.log(`[Apify] No posts found for ${target}`);
return { markdown: null, rawText: null, ogDates: {}, ogImage: null, links: [], reachable: true, reason: 'no posts found' };
return { markdown: null, reachable: true, reason: 'no posts found' };
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive check: runActorSync returns the parsed JSON response. If the API returns a non-array JSON object (e.g., an error payload or metadata), calling items.map will throw a TypeError. Checking Array.isArray(items) prevents this potential crash.

Suggested change
const items = await runActorSync(FACEBOOK_ACTOR_ID, { startUrls: [{ url: target }], maxPosts: maxItems }, token);
if (!items || items.length === 0) {
console.log(`[Apify] No posts found for ${target}`);
return { markdown: null, rawText: null, ogDates: {}, ogImage: null, links: [], reachable: true, reason: 'no posts found' };
return { markdown: null, reachable: true, reason: 'no posts found' };
}
const items = await runActorSync(FACEBOOK_ACTOR_ID, { startUrls: [{ url: target }], maxPosts: maxItems }, token);
if (!Array.isArray(items) || items.length === 0) {
console.log(`[Apify] No posts found or invalid response format for ${target}`);
return { markdown: null, reachable: true, reason: 'no posts found' };
}

fatherlinux and others added 2 commits July 3, 2026 22:51
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove Apify integration — Serper already provides what we need

1 participant