feature: fetch links from paginated hubs add add it to references

.github/workflows/scp-items.yml

@@ -37,6 +37,9 @@ jobs:
       - name: "Crawl Hubs"
         run: make data/scp_hubs.json
 
+      - name: "Process Hubs"
+        run: make data/processed/hubs
+
       - name: "Crawl Items"
         run: make data/scp_items.json
 

.gitignore

@@ -105,3 +105,4 @@ venv.bak/
 
 
 *.json
+.DS_Store

PAGINATION.md

@@ -0,0 +1,158 @@
+# Hub Pagination Support
+
+## Overview
+
+Some hub pages on the SCP Wiki have paginated versions that load additional content. These pages have URLs in the format `/hub-name/p/1`, `/hub-name/p/2`, etc.
+
+Examples:
+- https://scp-wiki.wikidot.com/church-of-the-broken-god-hub/p/2
+- https://scp-wiki.wikidot.com/chaos-insurgency-hub/p/5
+
+The paginated pages typically only contain lists of links to additional SCP items, tales, or other content that didn't fit on the main hub page.
+
+## Implementation
+
+### 1. Approach
+
+Since paginated pages only contain **lists of links** (not full content), we use a lightweight approach:
+
+1. **Main Hub Crawl**: The `ScpHubSpider` crawls only the main hub pages
+2. **Link Extraction**: `fetch_paginated_links.py` extracts pagination URLs from hub content and fetches the links from each paginated page using simple HTTP requests
+3. **Link Integration**: During postprocessing, the extracted links are merged into the hub's `references` array
+
+This is much more reliable and efficient than trying to crawl paginated pages with Scrapy, as it avoids rate limiting issues.
+
+### 2. Link Extraction Script
+
+The `fetch_paginated_links.py` script:
+
+```python
+def extract_links_from_page(url, timeout=30):
+    """Fetch a paginated page and extract all links from the content area."""
+    with urlopen(url, timeout=timeout) as response:
+        html = response.read().decode('utf-8')
+
+    soup = BeautifulSoup(html, 'lxml')
+    content_div = soup.find('div', id='page-content')
+
+    links = []
+    for a_tag in content_div.find_all('a', href=True):
+        href = a_tag['href']
+        if not (href.startswith('http') or href.startswith('#') or '/p/' in href):
+            links.append(href.lstrip('/'))
+
+    return links
+```
+
+The script:
+- Scans all hub `raw_content` for pagination links (pattern: `/hub-name/p/digit`)
+- Fetches each paginated URL via HTTP
+- Extracts all content links from the page
+- Saves results to `data/paginated_links.json`
+
+### 3. Data Structure
+
+The `paginated_links.json` file has this format:
+
+```json
+{
+  "chaos-insurgency-hub": {
+    "1": {
+      "url": "https://scp-wiki.wikidot.com/chaos-insurgency-hub/p/1",
+      "links": ["scp-xxxx", "tale-yyyy", ...],
+      "link_count": 50
+    },
+    "2": {
+      "url": "https://scp-wiki.wikidot.com/chaos-insurgency-hub/p/2",
+      "links": [...],
+      "link_count": 48
+    }
+  }
+}
+```
+
+### 4. Postprocessing
+
+The postprocessing script (`scp_crawler/postprocessing.py`):
+
+1. Loads `data/paginated_links.json` if available
+2. For each hub with paginated links:
+   - Collects all links from all paginated pages
+   - Merges them into the hub's existing `references` array
+   - Removes duplicates
+3. Outputs the enriched hubs to `data/processed/hubs/index.json`
+
+```python
+if link in paginated_links_data:
+    all_paginated_links = []
+    for page_num, page_data in paginated_links_data[link].items():
+        all_paginated_links.extend(page_data.get("links", []))
+
+    existing_refs = set(hub.get("references", []))
+    new_refs = existing_refs.union(set(all_paginated_links))
+    hub["references"] = list(new_refs)
+```
+
+### 5. Makefile Integration
+
+The makefile automatically handles pagination:
+
+```makefile
+data/scp_hubs.json: .venv
+	@echo "==> Crawling main hubs..."
+	@rm -f data/scp_hubs.json data/paginated_links.json
+	@$(PYTHON_VENV) python -m scrapy crawl scp_hubs -o data/scp_hubs.json
+	@echo "==> Fetching links from paginated pages..."
+	@$(PYTHON_VENV) python fetch_paginated_links.py
+```
+
+## Usage
+
+### Full Pipeline
+
+```bash
+make data/scp_hubs.json      # Crawl main hubs + fetch paginated links
+make data/processed/hubs     # Merge paginated links into references
+```
+
+### Manual Steps
+
+```bash
+# 1. Crawl main hubs
+scrapy crawl scp_hubs -o data/scp_hubs.json
+
+# 2. Extract paginated links
+source .venv/bin/activate
+python fetch_paginated_links.py
+
+# 3. Process and merge
+make data/processed/hubs
+```
+
+## Benefits
+
+✅ **Reliable**: Simple HTTP requests avoid Scrapy rate limiting  
+✅ **Efficient**: Only fetches what's needed (links, not full pages)  
+✅ **Complete**: All paginated URLs are successfully fetched  
+✅ **Clean**: No duplicate content, all links merged into `references`
+
+## Example Result
+
+A hub with pagination will have an enriched `references` array:
+
+```json
+{
+  "chaos-insurgency-hub": {
+    "title": "Chaos Insurgency Hub",
+    "references": [
+      "scp-001",
+      "scp-002",
+      "tale-insurgency-rising",
+      // ... links from main page
+      "scp-9999",
+      "scp-10000",
+      // ... links from paginated pages
+    ]
+  }
+}
+```

README.md

@@ -32,9 +32,15 @@ To crawl the International Hub for SCP Items and save to a custom location:
 scrapy crawl scp_int -o scp_international_items.json
 ```
 
+To crawl pages tagged as `supplement` and save to a custom location:
+
+```bash
+scrapy crawl scp_supplement -o scp_supplement.json
+```
+
 ## Raw Content Structure
 
-There are two types of content downloaded- SCP Items and SCP Tales.
+There are multiple types of content downloaded (Items, Tales, GOI formats, and Supplements).
 
 All content (both SCP Items and Tales) contain the following:
 
@@ -66,6 +72,7 @@ The crawler generates a series of json files containing an array of objects repr
 | scp_titles.json     | Main          | Title | scp     |
 | scp_hubs.json       | Main          | Hub   | scp     |
 | scp_tales.json      | Main          | Tale  | scp     |
+| scp_supplement.json | Main          | Supplement | scp |
 | scp_int.json        | International | Item  | scp_int |
 | scp_int_titles.json | International | Title | scp_int |
 | scp_int_tales.json  | International | Tale  | scp_int |
@@ -76,7 +83,27 @@ To regenerate all files run `make fresh`.
 
 ## Post Processed Data
 
-The postproc system takes the Titles, Hubs, Items, and Tales and uses them to generate a comprehensive set of objects. It combines and cross references data and expands on the data already there.
+The postproc system takes Titles, Hubs, Items, Tales, GOI, and Supplements and uses them to generate a comprehensive set of objects. It combines and cross references data and expands on the data already there.
+
+### Hub Pagination
+
+Many hub pages have paginated sections that load additional content. The crawler automatically handles this by:
+
+1. Extracting pagination URLs from hub content (e.g., `/chaos-insurgency-hub/p/2`)
+2. Fetching links from all paginated pages via HTTP
+3. Merging these links into the hub's `references` array during postprocessing
+
+This ensures that all links from paginated pages are included in the hub data without needing to crawl full pages. See [PAGINATION.md](PAGINATION.md) for detailed documentation.
+
+**Generated pagination data:**
+- `data/paginated_links.json` - Intermediate file containing links from all paginated hub pages
+- `data/processed/hubs/index.json` - Final hub data with merged pagination links in `references`
+
+### Output Structure
+
+Supplements are written to `data/processed/supplement/` and include additional fields like `parent_scp` and `parent_tale`.
+
+Hubs are written to `data/processed/hubs/` and include enriched `references` arrays with links from both the main hub page and any paginated sections.
 
 
 ## Content Licensing

fetch_paginated_links.py

@@ -0,0 +1,125 @@
+"""Fetch links from paginated hub pages without full crawling.
+
+This is much more reliable than crawling with Scrapy since we only need
+the links, not the full page content.
+"""
+import json
+import re
+import time
+from urllib.parse import urljoin
+from urllib.request import urlopen
+from urllib.error import HTTPError, URLError
+
+from bs4 import BeautifulSoup
+from tqdm import tqdm
+
+
+def extract_links_from_page(url, timeout=30):
+    """Fetch a paginated page and extract all links from the content area."""
+    try:
+        with urlopen(url, timeout=timeout) as response:
+            html = response.read().decode('utf-8')
+
+        soup = BeautifulSoup(html, 'lxml')
+
+        # Find the main content area (same as in WikiMixin)
+        content_div = soup.find('div', id='page-content')
+        if not content_div:
+            return []
+
+        # Extract all links from the content
+        links = []
+        for a_tag in content_div.find_all('a', href=True):
+            href = a_tag['href']
+
+            # Skip external links, anchors, and special pages
+            if href.startswith('http') or href.startswith('#') or href.startswith('javascript:'):
+                continue
+
+            # Clean the link (remove leading /)
+            clean_link = href.lstrip('/')
+
+            # Skip pagination links themselves
+            if '/p/' in clean_link:
+                continue
+
+            links.append(clean_link)
+
+        return links
+
+    except (HTTPError, URLError) as e:
+        print(f"  ✗ Failed to fetch {url}: {e}")
+        return []
+    except Exception as e:
+        print(f"  ✗ Error processing {url}: {e}")
+        return []
+
+
+def fetch_paginated_links(hub_file="data/scp_hubs.json", output_file="data/paginated_links.json"):
+    """Extract pagination URLs and fetch links from each page."""
+
+    try:
+        with open(hub_file) as f:
+            hubs = json.load(f)
+    except FileNotFoundError:
+        print(f"❌ {hub_file} not found")
+        return
+
+    # Extract pagination URLs from hub raw_content
+    pagination_pattern = re.compile(r'href="(/[^"]*-hub/p/\d+)"')
+    base_domain = "https://scp-wiki.wikidot.com"
+
+    all_pagination_urls = set()
+    for hub in hubs:
+        raw_content = hub.get("raw_content", "")
+        if not raw_content:
+            continue
+
+        matches = pagination_pattern.findall(raw_content)
+        for match in matches:
+            full_url = urljoin(base_domain, match)
+            all_pagination_urls.add(full_url)
+
+    print(f"Found {len(all_pagination_urls)} paginated URLs to fetch")
+
+    # Fetch links from each paginated page
+    results = {}
+
+    for url in tqdm(sorted(all_pagination_urls), desc="Fetching paginated links"):
+        # Extract hub name and page number from URL
+        match = re.search(r'/([^/]+-hub)/p/(\d+)', url)
+        if not match:
+            continue
+
+        hub_name = match.group(1)
+        page_num = int(match.group(2))
+
+        # Fetch the links
+        links = extract_links_from_page(url)
+
+        if links:
+            if hub_name not in results:
+                results[hub_name] = {}
+
+            results[hub_name][page_num] = {
+                "url": url,
+                "links": links,
+                "link_count": len(links)
+            }
+            print(f"  ✓ {hub_name}/p/{page_num}: {len(links)} links")
+
+        # Be nice to the server
+        time.sleep(0.5)
+
+    # Save results
+    with open(output_file, 'w') as f:
+        json.dump(results, f, indent=2, sort_keys=True)
+
+    print(f"\n✅ Saved paginated links to {output_file}")
+    print(f"   Total hubs with pagination: {len(results)}")
+    total_pages = sum(len(pages) for pages in results.values())
+    print(f"   Total pages fetched: {total_pages}")
+
+
+if __name__ == "__main__":
+    fetch_paginated_links()