Add study: study_016

studies/study_016/README.md

@@ -0,0 +1,7 @@
+# The "False Consensus Effect": An Egocentric Bias in Social Perception and Attribution Processes
+
+This HumanStudy-Hub package was reconstructed from a published paper.
+
+- Review `study.json` for the study overview and readiness status.
+- Review `audit/missing_information.json` before running the study.
+- Run `python task/adapter.py --smoke-test` to check the package entry point.

studies/study_016/audit/missing_information.json

@@ -0,0 +1,68 @@
+{
+  "_purpose": "Authoritative researcher checklist of missing verbatim content and researcher decisions still required before this package can be treated as a full, faithful reconstruction of Ross, Greene & House (1977). Every entry has study/field/reason/impact/suggested_action.",
+  "entries": [
+    {
+      "study": "study2",
+      "field": "materials.study2.items[*] full item wording",
+      "reason": "The paper states explicitly (p.286, evidence.json id s2-questionnaire-not-reproduced) that the full 35-item questionnaire text is 'too lengthy to be reproduced' and was available only by request from the first author. Only the short category-pair labels from Table 3 survive in this source.",
+      "impact": "task/adapter.py administers Study 2 using paraphrased category labels (e.g. 'Shy / not shy') rather than the original item stems. Participant responses may not match how the original wording framed the choice, especially for ambiguous items (e.g. what counted as 'often' thinking about dying).",
+      "suggested_action": "Obtain the original questionnaire from the paper's open materials (if the contributor supplies an authorized OSF/author-provided URL in a future job) or transcribe it from an institutional archive, then replace materials.json study2.items[*] with verbatim item text and relabel evidence_label to 'verbatim'."
+    },
+    {
+      "study": "study2",
+      "field": "materials.study2.items[*].category_1 for 'Subscribe to magazines on list provided'",
+      "reason": "The paper references 'magazines on list provided' but the actual magazine list is not reproduced anywhere in the extracted text.",
+      "impact": "This item cannot be administered exactly as originally presented; adapter.py presents only the generic category label.",
+      "suggested_action": "If the original magazine list can be located (author request), add it verbatim; otherwise mark this item as excluded from faithful replication rather than guessing a list."
+    },
+    {
+      "study": "study2",
+      "field": "Table 3 numeric cells (mean percentage estimates, t-values)",
+      "reason": "pdftotext OCR of the Table 3 columns on pages 287-288 is degraded in places (garbled digits/characters, e.g. 'h0.24' style artifacts in the political-expectations rows).",
+      "impact": "source/evidence.json id s2-table3-items-and-results transcribes the table as extracted, but a handful of individual cell values should be treated as approximate; they are not used programmatically anywhere in task/adapter.py or evaluation/evaluation.py, only as historical-comparison context.",
+      "suggested_action": "Re-OCR page 287-288 at higher resolution or against a cleaner PDF scan, or manually verify against the published journal issue, before citing exact per-item t-values from this package."
+    },
+    {
+      "study": "study1",
+      "field": "materials.trait_scale_template anchors for traits other than cooperativeness",
+      "reason": "The paper reproduces full verbatim 100-point anchor wording for exactly one trait (cooperativeness); it states all other traits followed 'the same' structure but never reprints their specific anchor sentences.",
+      "impact": "materials.json renders anchors for all other traits (shyness, trust, adventurousness, gregariousness, laziness, competitiveness, generosity, self-confidence, legal knowledge, argumentativeness, miserliness, rationality, social concern, idealism, selfishness, and the Study 3/4 trait set) via a paraphrase template, explicitly labeled 'derived'. Exact original wording for these anchors is unrecoverable from this source.",
+      "suggested_action": "If open materials become available (with user authorization), replace the derived anchors with verbatim wording for each trait."
+    },
+    {
+      "study": "study1",
+      "field": "Table 2 / Table 3 per-trait numeric breakdown (Study 1 trait ratings)",
+      "reason": "The printed paper appears to mislabel the Study 1 trait-ratings table as 'Table 3' (duplicating the Study 2 table number) and the OCR of its numeric cells (pages 283-284, e.g. 'x9 5', '54.9') is degraded.",
+      "impact": "source/evidence.json id s1-table2-trait-note relies on the narrative F-ratios/p-values in body text rather than the raw table cells, which are not independently re-derivable from this PDF extraction.",
+      "suggested_action": "Obtain a cleaner scan or the publisher's typeset PDF to recover exact per-trait numeric means for Study 1's Table 2/3."
+    },
+    {
+      "study": "study4",
+      "field": "trait list used for Study 4 ratings",
+      "reason": "Study 1 and Study 3 explicitly name the traits rated; Study 4's methods section (p.291) does not restate which traits were used, only that ratings were made about two supposedly real target individuals.",
+      "impact": "materials.json study4.traits carries over Study 3's 8-trait list as a labeled assumption ('missing' evidence label). If the original Study 4 trait list differed, task/adapter.py's Study 4 condition would be administering the wrong traits.",
+      "suggested_action": "Confirm the Study 4 trait list against the original materials (author request) before treating Study 4 trait-extremity results as more than exploratory."
+    },
+    {
+      "study": "study4",
+      "field": "faithful behavioral replication of the real-choice manipulation",
+      "reason": "The original Study 4 depended on a live experimenter, a real 30-minute campus walk with a genuine social-compliance cost, and post-hoc debriefing about deception. A text-only agent task cannot reproduce the lived behavioral stakes, only the textual cover story and a stated hypothetical choice.",
+      "impact": "task/task.json marks study4 readiness as 'blocked_partial': results from this condition should be interpreted as a text-based reconstruction, not a behavioral replication, and must not be pooled with real Study 4 data from the original paper.",
+      "suggested_action": "If a researcher wants a truer behavioral analogue, this would require a different experimental platform (e.g. an actual field task or an incentivized decision with real consequences for the agent/participant) rather than a text-only conversational task."
+    },
+    {
+      "study": "study1",
+      "field": "space_program story option-label consistency",
+      "reason": "The story text asks about voting 'for'/'against' the funding allocation, but Table 1 in the paper labels the same two options 'vote for cutback' / 'vote against cutback' -- an apparent inconsistency in the original publication (possibly a sign flip between the story framing and the table labeling).",
+      "impact": "materials.json preserves the story-text framing ('for the space program allocation' / 'against') for participant-facing text, since that is what a participant actually reads, and flags the Table 1 labeling discrepancy in source/evidence.json (s1-story-spaceprogram). A researcher should decide which framing to trust when comparing new data to the original Table 1 percentages.",
+      "suggested_action": "Consult original materials (if available) to determine whether Table 1's 'cutback' labels reflect a different/earlier version of the story, or a labeling error, before directly comparing new consensus estimates to the published Table 1 numbers for this story."
+    },
+    {
+      "study": "paper",
+      "field": "Fields & Schuman unpublished manuscript citation (footnote 5)",
+      "reason": "The paper cites an unpublished manuscript ('Public Beliefs about the Beliefs of the Public') as corroborating evidence; this could not be independently retrieved or verified because network research was not authorized for this job.",
+      "impact": "This citation is recorded as-is in source/evidence.json (item refs-1 context) but not verified.",
+      "suggested_action": "If a future job for this paper supplies an authorized external-materials URL, this citation could be checked, but it is not required for running the core task/evaluation package."
+    }
+  ]
+}

studies/study_016/evaluation/evaluation.py

@@ -0,0 +1,268 @@
+#!/usr/bin/env python3
+"""
+Evaluation for the Ross, Greene, & House (1977) "False Consensus Effect" package.
+
+Consumes a JSON-lines file of participant response records produced by
+task/adapter.py (one record per line, one simulated participant each) and
+tests the two hypotheses common to Study 1, Study 3, and Study 4:
+
+  H1 (perceived consensus): participants who personally choose option X
+     estimate a higher percentage of peers choosing X than participants who
+     chose the alternative.
+  H2 (trait-rating extremity): participants give less extreme trait ratings
+     to a "typical" actor who shares their own choice than to one who does not.
+
+Study 2 is evaluated separately (per-item direction/count check, matching
+the paper's "32 of 34 items in predicted direction" summary statistic).
+
+No network access is used or required. Where too few records are supplied to
+run a given check, that check reports status "not_ready" with the specific
+missing requirement rather than fabricating a result.
+"""
+import argparse
+import json
+import math
+import statistics
+import sys
+from collections import defaultdict
+
+
+def load_records(path):
+    records = []
+    with open(path, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                records.append(json.loads(line))
+    return records
+
+
+def normal_two_tailed_p(z):
+    """Two-tailed p-value from the standard normal CDF (erf-based)."""
+    return 2.0 * (1.0 - 0.5 * (1.0 + math.erf(abs(z) / math.sqrt(2))))
+
+
+def welch_t_test(sample1, sample2):
+    """Welch's t-test for unequal variances. Returns dict with t, df, p (normal
+    approximation to the t-distribution -- adequate for the n>~20 per-group
+    sizes this package targets; not an exact Student-t CDF, to avoid a SciPy
+    dependency in an offline-only adapter)."""
+    n1, n2 = len(sample1), len(sample2)
+    if n1 < 2 or n2 < 2:
+        return None
+    m1, m2 = statistics.mean(sample1), statistics.mean(sample2)
+    v1, v2 = statistics.variance(sample1), statistics.variance(sample2)
+    se = math.sqrt(v1 / n1 + v2 / n2)
+    if se == 0:
+        return {"t": 0.0, "df": n1 + n2 - 2, "p_two_tailed_normal_approx": 1.0, "mean1": m1, "mean2": m2}
+    t = (m1 - m2) / se
+    df_num = (v1 / n1 + v2 / n2) ** 2
+    df_den = ((v1 / n1) ** 2) / (n1 - 1) + ((v2 / n2) ** 2) / (n2 - 1)
+    df = df_num / df_den if df_den > 0 else n1 + n2 - 2
+    p = normal_two_tailed_p(t)
+    return {"t": t, "df": df, "p_two_tailed_normal_approx": p, "mean1": m1, "mean2": m2}
+
+
+def one_sample_t_test(diffs):
+    n = len(diffs)
+    if n < 2:
+        return None
+    m = statistics.mean(diffs)
+    sd = statistics.stdev(diffs)
+    if sd == 0:
+        return {"t": float("inf") if m != 0 else 0.0, "df": n - 1, "p_two_tailed_normal_approx": 0.0 if m != 0 else 1.0, "mean_diff": m, "n": n}
+    se = sd / math.sqrt(n)
+    t = m / se
+    p = normal_two_tailed_p(t)
+    return {"t": t, "df": n - 1, "p_two_tailed_normal_approx": p, "mean_diff": m, "n": n}
+
+
+def evaluate_consensus_hypothesis(records, study_id, condition_id):
+    """H1 for study1/study3/study4: own-choice group's estimate of their own
+    option should exceed the other group's estimate of that same option."""
+    subset = [r for r in records if r.get("study_id") == study_id and r.get("condition_id") == condition_id]
+    if not subset:
+        return {"status": "not_ready", "missing_requirement": f"no records for {study_id}/{condition_id}"}
+
+    options = sorted({opt for r in subset for opt in r.get("consensus_estimate", {})})
+    if len(options) != 2:
+        return {"status": "not_ready", "missing_requirement": "records do not agree on a two-option consensus_estimate schema"}
+    optA, optB = options
+
+    group_est_for_A = defaultdict(list)  # own_choice -> list of estimate(%A)
+    for r in subset:
+        est = r.get("consensus_estimate", {})
+        if optA not in est:
+            continue
+        choice = r.get("own_choice")
+        if choice not in (optA, optB):
+            continue
+        group_est_for_A[choice].append(float(est[optA]))
+
+    if len(group_est_for_A.get(optA, [])) < 2 or len(group_est_for_A.get(optB, [])) < 2:
+        return {
+            "status": "not_ready",
+            "missing_requirement": (
+                f"need >=2 participants choosing '{optA}' and >=2 choosing '{optB}' in {study_id}/{condition_id}; "
+                f"have {len(group_est_for_A.get(optA, []))} and {len(group_est_for_A.get(optB, []))}"
+            ),
+        }
+
+    test = welch_t_test(group_est_for_A[optA], group_est_for_A[optB])
+    predicted_direction = test["mean1"] > test["mean2"]
+    return {
+        "status": "ok",
+        "study_id": study_id,
+        "condition_id": condition_id,
+        "option_compared": optA,
+        "n_chose_optA": len(group_est_for_A[optA]),
+        "n_chose_optB": len(group_est_for_A[optB]),
+        "mean_estimate_pctA_by_optA_choosers": test["mean1"],
+        "mean_estimate_pctA_by_optB_choosers": test["mean2"],
+        "predicted_direction_confirmed": predicted_direction,
+        "t": test["t"],
+        "df": test["df"],
+        "p_two_tailed_normal_approx": test["p_two_tailed_normal_approx"],
+        "note": "p-value uses a normal approximation to the t-distribution; see welch_t_test docstring.",
+    }
+
+
+def evaluate_trait_extremity_hypothesis(records, study_id, condition_id):
+    """H2 for study1/study3/study4: within each participant, the trait-rating
+    extremity (|rating|, averaged over traits) for the actor who differs from
+    the rater's own choice should exceed the extremity for the actor who
+    matches the rater's own choice."""
+    subset = [r for r in records if r.get("study_id") == study_id and r.get("condition_id") == condition_id and r.get("trait_ratings")]
+    if len(subset) < 2:
+        return {"status": "not_ready", "missing_requirement": f"need >=2 records with trait_ratings for {study_id}/{condition_id}; have {len(subset)}"}
+
+    diffs = []
+    for r in subset:
+        choice = r.get("own_choice")
+        traits = r.get("trait_ratings", {})
+        matched_vals, mismatched_vals = [], []
+        for trait, ratings in traits.items():
+            for target, val in ratings.items():
+                if target == choice:
+                    matched_vals.append(abs(float(val)))
+                else:
+                    mismatched_vals.append(abs(float(val)))
+        if not matched_vals or not mismatched_vals:
+            continue
+        diffs.append(statistics.mean(mismatched_vals) - statistics.mean(matched_vals))
+
+    if len(diffs) < 2:
+        return {"status": "not_ready", "missing_requirement": "could not compute matched/mismatched extremity pairs from trait_ratings"}
+
+    test = one_sample_t_test(diffs)
+    return {
+        "status": "ok",
+        "study_id": study_id,
+        "condition_id": condition_id,
+        "n": test["n"],
+        "mean_extremity_difference_mismatched_minus_matched": test["mean_diff"],
+        "predicted_direction_confirmed": test["mean_diff"] > 0,
+        "t": test["t"],
+        "df": test["df"],
+        "p_two_tailed_normal_approx": test["p_two_tailed_normal_approx"],
+        "note": "p-value uses a normal approximation to the t-distribution; see welch_t_test docstring.",
+    }
+
+
+def evaluate_study2(records):
+    subset = [r for r in records if r.get("study_id") == "study2"]
+    if not subset:
+        return {"status": "not_ready", "missing_requirement": "no study2 records supplied"}
+
+    per_item = defaultdict(lambda: {"cat1": [], "cat2": []})
+    for r in subset:
+        self_cat = {row["item_index"]: row["category_chosen"] for row in r.get("self_category", [])}
+        peer_est = {row["item_index"]: row["pct_category_1"] for row in r.get("peer_estimate_pct", [])}
+        for idx, cat in self_cat.items():
+            if idx not in peer_est:
+                continue
+            bucket = "cat1" if cat == "category_1" else "cat2"
+            per_item[idx][bucket].append(float(peer_est[idx]))
+
+    item_results = {}
+    predicted_direction_count = 0
+    tested_count = 0
+    for idx, groups in per_item.items():
+        if len(groups["cat1"]) < 2 or len(groups["cat2"]) < 2:
+            item_results[idx] = {"status": "not_ready", "missing_requirement": "need >=2 self-raters in each category for this item"}
+            continue
+        test = welch_t_test(groups["cat1"], groups["cat2"])
+        predicted = test["mean1"] > test["mean2"]
+        tested_count += 1
+        if predicted:
+            predicted_direction_count += 1
+        item_results[idx] = {
+            "status": "ok",
+            "n_cat1": len(groups["cat1"]),
+            "n_cat2": len(groups["cat2"]),
+            "mean_estimate_by_cat1_raters": test["mean1"],
+            "mean_estimate_by_cat2_raters": test["mean2"],
+            "predicted_direction_confirmed": predicted,
+            "t": test["t"],
+            "p_two_tailed_normal_approx": test["p_two_tailed_normal_approx"],
+        }
+
+    if tested_count == 0:
+        return {"status": "not_ready", "missing_requirement": "no item had >=2 respondents in both self-categories yet"}
+
+    return {
+        "status": "ok",
+        "items_tested": tested_count,
+        "items_total_available": len(per_item),
+        "items_in_predicted_direction": predicted_direction_count,
+        "paper_reported_baseline": "32 of 34 items in predicted direction (Ross, Greene & House, 1977, p.286)",
+        "per_item": item_results,
+    }
+
+
+STUDY_CONDITIONS = {
+    "study1": ["supermarket", "term_paper", "traffic_ticket", "space_program"],
+    "study3": ["eat_at_joes", "repent"],
+    "study4": ["generic"],
+}
+
+
+def run_evaluation(records, only_study=None):
+    result = {"consensus_hypothesis": {}, "trait_extremity_hypothesis": {}, "study2": None}
+
+    for study_id, conditions in STUDY_CONDITIONS.items():
+        if only_study and study_id != only_study:
+            continue
+        for condition_id in conditions:
+            key = f"{study_id}/{condition_id}"
+            result["consensus_hypothesis"][key] = evaluate_consensus_hypothesis(records, study_id, condition_id)
+            result["trait_extremity_hypothesis"][key] = evaluate_trait_extremity_hypothesis(records, study_id, condition_id)
+
+    if only_study is None or only_study == "study2":
+        result["study2"] = evaluate_study2(records)
+
+    return result
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--records", required=True, help="path to a JSON-lines file of adapter.py response records")
+    parser.add_argument("--study", choices=["study1", "study2", "study3", "study4"], help="restrict evaluation to one study")
+    args = parser.parse_args()
+
+    try:
+        records = load_records(args.records)
+    except FileNotFoundError:
+        print(json.dumps({"status": "not_ready", "missing_requirement": f"records file not found: {args.records}"}, indent=2))
+        sys.exit(0)
+
+    if not records:
+        print(json.dumps({"status": "not_ready", "missing_requirement": "records file is empty"}, indent=2))
+        sys.exit(0)
+
+    result = run_evaluation(records, only_study=args.study)
+    print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+    main()