UW Math Prerequisite Graph Scraper

A small Python scraper that builds a directed prerequisite graph for the University of Waterloo Math Faculty by querying UWFlow's public GraphQL API (https://uwflow.com/graphql).

It starts from a set of "root" courses (intro courses with no math-faculty prerequisites) and walks forward through the leads to (postrequisite) relationships, growing the graph until everything reachable has been visited. For every visited course it also records the in-graph prerequisite edges, giving a complete view of dependencies between the nodes that ended up in the graph.

Output

A single JSON file at data/course_graph.json:

{
  "metadata": { "generated_at": "...", "roots": [...], "node_count": N, "edge_count": M, ... },
  "nodes": {
    "math135": {
      "code": "math135",
      "name": "Algebra for Honours Mathematics",
      "description": "...",
      "prereqs_text": "...",
      "coreqs_text": null,
      "antireqs_text": "MATH145",
      "subject": "MATH",
      "is_root": true,
      "depth": 0
    }
  },
  "edges": [
    { "from": "math135", "to": "math136", "type": "prerequisite_of", "is_corequisite": false }
  ]
}

Edge direction is "must take before". math135 -> math136 means MATH 135 is a prerequisite of MATH 136.

Allowed subjects

By default, only courses whose subject prefix is one of the following are kept in the graph (everything else is dropped during traversal):

MATH, AMATH, PMATH, CO, CS, STAT, ACTSC, CM, CFM, SE

Default root courses

math135, math136, math137, math138, math145, math146, math147, math148,
cs115, cs135, cs136, cs145, cs146

Usage

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Run with defaults
python3 scraper.py

# Override roots
python3 scraper.py --roots math135,math137,cs115

# Override allowed subjects
python3 scraper.py --subjects MATH,CS,STAT

# Verbose logs
python3 scraper.py --log-level DEBUG

CLI flags:

• --roots – comma-separated list of root course codes (lowercase, no space).
• --subjects – comma-separated list of subject prefixes to keep.
• --output – output JSON path (default data/course_graph.json).
• --log-level – console log level (default INFO).
• --delay – seconds to sleep between GraphQL requests (default 0.1).

Logging

Two log handlers are attached:

• Console: human-readable, level controlled by --log-level.
• File: logs/scraper.log, always at DEBUG level with timestamps.

Visualizer

A minimal web viewer is provided at index.html. It uses vis-network via CDN and loads data/course_graph.json directly.

Because browsers block fetch from file://, serve the directory over HTTP:

python3 -m http.server 8000
# then open http://localhost:8000/

Features:

• Force-directed layout that auto-freezes once it settles (drag nodes to rearrange).
• Nodes are colored by subject; root courses have a white border.
• Solid arrows = prerequisites; dashed arrows = corequisites.
• Click any node (or search by code, e.g. cs245) to see its name, prereq text, in-graph prereqs, courses it leads to, and a link to its UWFlow page.
• Prereq chain / Leads-to focus: open a separate overlay subgraph showing only the transitive ancestors or descendants of the selected course (strictly directional walk).
• Filters: hide advanced/enriched variants (145, 146, 249, 240E, etc.), and toggle subjects on/off via the dropdown.
• "Re-arrange" re-runs the physics solver briefly if you've dragged things into a tangle.

Deploying to Cloudflare Pages

The site is fully static (HTML + JS + JSON), no build step required.

Git-connected (recommended)

1. Push this repo to GitHub/GitLab.
2. In the Cloudflare dashboard: Workers & Pages → Create → Pages → Connect to Git → pick this repo.
3. Build settings:
   • Framework preset: None
   • Build command: (empty)
   • Build output directory: /
4. Save and deploy. Subsequent pushes auto-deploy.

wrangler.toml has pages_build_output_dir = "." so the Pages CLI deploys also work. The _headers file sets caching (/data/* cached 5 min, HTML always revalidated) and basic security headers.

One-off deploy via CLI

npx wrangler@latest pages deploy . --project-name uw-course-graph

To refresh the graph data later: re-run python3 scraper.py, commit the updated data/course_graph.json, and push.