Skip to content

README.md

Latest commit

 

History

History
243 lines (181 loc) · 9.2 KB

README.md

File metadata and controls

243 lines (181 loc) · 9.2 KB
Raw

7 Cups Forum — Reply Graph & Interaction Network

Structural analysis of public peer-support threads on 7 Cups. A thread is scraped into a post-level record set, then lifted into two graphs: a reply forest over posts and a weighted interaction network over participants. Output is GraphML plus a printed metrics summary.

Pipeline

7 Cups thread (HTML)
        │
        │  webscraper.py        saved file  |  requests  |  Selenium
        ▼
   posts.json                   one record per post, explicit parent edges
        │
        │  buildGraphs.py       igraph
        ▼
reply_graph.graphml  +  user_graph.graphml  +  metrics summary

Reply edges are recovered, not inferred. Each reply footer carries a ?post=<id> link naming the exact post it answers, so the parent relation is read off the DOM rather than reconstructed from indentation or timestamps.

Graph formalism

Reply graph — G_R = (P, E_R), directed, unweighted
Nodes P posts
Edges E_R {(p, parent(p)) : p ∈ P, parent(p) ∈ P}
Node attributes author, depth, is_op, hearts

Every post replies to at most one parent, so out-degree ≤ 1 for all p and G_R is an in-forest — one in-tree per thread, rooted at the thread starter. Roots are the out-degree-0 vertices. In-degree is the direct-reply count and is the natural local measure of which post pulled the conversation toward it. is_dag() is reported as an integrity check, not a finding: a cycle would mean the parser is wrong, not that the forum is.

User graph — G_U = (U, E_U, w), directed, weighted
Nodes U participants
Edges E_U (a, b) where a replied to b
Weight w(a,b) = |{p : author(p) = a, reply_target(p) = b}|
Node attributes posts (post count)

Self-loops are retained (a user replying to their own post) and count toward the weight totals in the summary. Note that igraph's reciprocity() ignores loops by default, so the reciprocity figure and the per-user reply tallies are computed over slightly different edge sets — read them accordingly.

Record schema (posts.json)

Field Type Notes
post_id str Site's numeric id. Primary key.
depth int From data-depth; 0 for the thread starter.
author / author_id str Handle, or u_ + 8 hex chars of SHA-256 when ANONYMIZE=True.
is_op bool Thread starter, or carries the Original Poster badge.
timestamp str | null Raw display string. Not normalised — no temporal analysis yet.
parent_post_id str | null Reply-graph edge target.
reply_to_user / reply_to_user_id str | null @handle from the footer. User-graph edge target.
hearts int | null Reaction count.
text str Post body, whitespace-collapsed.

author / reply_to_user and author_id / reply_to_user_id are mutually exclusive in a given file. buildGraphs.py detects which pair is present and adapts, so the same analysis runs over identified and pseudonymised exports.

Setup

pip3 install -r requirements.txt

requests + beautifulsoup4 cover file and requests modes. selenium + webdriver-manager are needed only for live browser fetches and require Chrome. igraph is needed only for buildGraphs.py.

Run

Parse a page you already saved — fastest, and sidesteps the bot check entirely:

python3 webscraper.py thread.html

Fetch live through a real browser — clears the Cloudflare interstitial. A window opens; solve any challenge, then press Enter in the terminal to capture:

python3 webscraper.py --selenium "https://www.7cups.com/forum/<community>/<Sub_ID>/<Thread_ID>/"

Fetch with plain requests — light, frequently blocked:

python3 webscraper.py --requests "https://www.7cups.com/forum/..."

Build the graphs:

python3 buildGraphs.py posts.json

Writes reply_graph.graphml and user_graph.graphml (open in Gephi, Cytoscape, igraph, or NetworkX) and prints node/edge counts, root count, weak component count, max depth, max in-degree, DAG check, reciprocity, and per-user reply tallies.

Multi-page threads: raise MAX_PAGES near the top of webscraper.py.

Repository layout

webscraper.py        fetch (file / requests / Selenium) + parse -> posts.json
buildGraphs.py       posts.json -> igraph graphs, metrics, GraphML
requirements.txt     dependencies
data/                scraped data (raw pages and identified exports gitignored)
docs/                report outline and notes
viz/                 placeholder — reply-tree viewer not yet implemented

Data handling

The corpus is unsolicited personal disclosure by people in distress on a mental-health platform. Public visibility is not consent to be a dataset.

This repository is public. Therefore:

  • No scraped content is committed — not raw pages, not posts.json, not GraphML. .gitignore covers *.html, posts.json, data/posts.json, *.graphml.
  • Hashing handles is not sufficient anonymization. Verbatim text is a search key that recovers the thread and therefore the handle, which makes the pseudonym reversible in one query. Any published export must be structure-only: post_id, depth, parent_post_id, author_id, hearts, and derived features. Bodies stay local.
  • Set ANONYMIZE = True in webscraper.py before generating anything that leaves your machine.
  • REQUEST_DELAY throttles requests mode. Do not remove it. Selenium mode is human-paced by construction.
  • Use is limited to topology and aggregate statistics. No profiling, no re-identification, no per-user case studies in the report.

Check 7 Cups' terms of service and your institution's ethics requirements before scaling the collection beyond the current sample.

Status

  • Platform recon — posts are server-rendered in raw HTML
  • Scraper built and validated (webscraper.py)
  • Sample scraped — 1 penpals thread, 111 posts, 2 participants, max reply depth 23
  • Graph construction and metrics (buildGraphs.py)
  • Multi-thread scrape — required for a non-degenerate user network
  • Interactive reply-tree viewer (viz/ is currently a placeholder)
  • Report (docs/)
  • NLP / emotion layer

Known limitations

These are real and affect interpretation. Do not report metrics without them.

  1. The sample is a dyad. 111 posts across 2 participants. The reply forest is substantive — depth 23 is a genuinely long support exchange — but the user graph is K₂ with loops. Reciprocity, centrality, assortativity, and community detection are all degenerate at |U| = 2. Multi-thread collection is the critical path, not an enhancement.

  2. Cross-page parents are dropped silently. buildGraphs.py filters edges to parent_post_id ∈ idx. If a reply's parent lives on a page outside the scrape window, the edge vanishes with no warning, the forest fragments, and the root count is inflated. Any multi-page result needs an explicit dropped-edge count before it is trusted.

  3. User edges come from the @mention, not the parent's author. reply_to_user is the rendered footer text; the authoritative relation is author(parent_post_id). These diverge on edited, deleted, or renamed accounts. Resolving user edges through parent_post_id would be strictly more robust and would also make the two graphs consistent by construction.

  4. ?p= pagination is assumed, not verified against the live site (flagged in-source).

  5. Timestamps are unparsed strings, so response latency, burstiness, and temporal motifs are currently out of reach.

  6. summarize() assumes a non-empty graph. max(reply.vs['depth']) and indeg.index(max(indeg)) raise on an empty parse.

  7. Weight assignment is order-coupled. g.es["weight"] = [w[e] for e in w] is correct only because add_edges iterated the same Counter in the same insertion order. It holds on CPython 3.7+ but is fragile to any refactor; building (edge, weight) pairs together removes the coupling.

Roadmap

  • Scrape n threads across multiple sub-communities; union the per-thread forests into one corpus and build a single user graph over the union.
  • Resolve user edges via parent_post_id → author and drop the @mention path.
  • Parse timestamps; add inter-arrival and response-latency features.
  • Structural characterisation: degree distribution, depth/branching profile, component sizes, listener-vs-member role separation.
  • Emotion / sentiment layer over post bodies as node features.

Team

Name Roll GitHub
Arnav Kothari SE25UCSE014 @usuallyarnav
Advik Prasad SE25UARI118 @LeafyChan
Anvitha Reddy SE25UDSC041 @anvi-ar
TVNS SaiCharan SE25UCSE071 @thesungod07

Workflow

git pull before starting. Small commits, pushed often. Tasks tracked in Issues and the Projects board. Report drafted wherever co-editing is easiest, exported into docs/ when final.