Temporal Narrative Alignment for Movie-Remake Retrieval

This repository contains a reproducible pilot extension of Chaturvedi, Srivastava, and Roth's NAACL 2018 paper, Where Have I Heard This Story Before? Identifying Narrative Similarity in Movie Remakes.

The original work models events, entities, and character correspondences in movie plots. This extension asks whether explicitly matching the beginning, middle, and end of two narratives improves remake retrieval, and whether the signal survives when character proper names are masked.

This is an independent research extension. It is not affiliated with or endorsed by the original authors, UNC–Chapel Hill, or the authors' institutions at the time of publication.

Main finding

On a held-out set of 371 queries from 207 cluster-disjoint remake families, a coarse position-aware three-segment model outperforms a content-matched orderless control by 4.0 P@1 points:

Comparison	P@1 difference	Cluster-bootstrap 95% CI	Raw p	Holm-adjusted p
Three-act alignment − orderless control	+4.0 pp	[+1.8, +6.5] pp	0.0013	0.0051
Hybrid − global TF-IDF	+1.6 pp	[0.0, +3.5] pp	0.1462	0.2924
Name-masked global − unmasked lemma global	−5.7 pp	[−10.1, −1.3] pp	0.0177	0.0532
Name-masked alignment − name-masked orderless	−2.2 pp	[−6.4, +2.0] pp	0.3771	0.3771

The position-sensitive association is statistically detectable in the unmasked representation on this exploratory split, but the pattern is not observed under the character-name masking condition. The result does not establish name-invariant narrative understanding.

Method

Each released BookNLP-processed summary is split into three equal-count sentence segments. The repository uses “three-act” as a compact method label; the boundaries are not inferred narratological acts and the model does not explicitly recover temporal relations.

1. Global lexical baseline: word and bigram TF-IDF over the complete plot summary.
2. Three-act alignment: average cosine similarity between corresponding beginning, middle, and ending blocks.
3. Orderless control: average all nine cross-block similarities. This retains the segment content while removing positional alignment.
4. Temporal residual: aligned similarity minus orderless similarity.
5. Hybrid: global TF-IDF plus a development-tuned weight on the temporal residual.
6. Name-masked stress test: tokens tagged by the released BookNLP annotations as animate NNP/NNPS mentions are replaced with character before fitting the lexical representation.

The hybrid weight is selected only on development clusters. TF-IDF vocabulary and inverse-document frequencies are fit transductively, without labels, on all 577 released summaries. Precision at 1 is the primary metric; mean reciprocal rank and Hit@5 are secondary. Hit@5 records whether at least one same-cluster candidate appears in the top five. Confidence intervals use 10,000 cluster bootstrap resamples. Four comparisons specified in the analysis script use 100,000 cluster sign-flip randomizations followed by Holm correction.

Data split

The official release contains 577 movies in 266 remake clusters, processed plot summaries, and 466 released Story Kernel predictions.

The released prediction/test files do not reconstruct the paper's stated cluster-level split. The 111 omitted movies belong to 101 clusters, and 93 of those clusters also occur among the 466 released queries; only 8 clusters containing 16 movies are wholly omitted. This pilot therefore creates a deterministic group split over the released query clusters:

Partition	Queries	Remake clusters
Development	95	51
Test	371	207

Development/test cluster overlap is zero. The random seed is 20260807. Retrieval considers all 577 released movies except the query itself.

Results

Method	P@1	MRR	Hit@5	Correct / 371
Global TF-IDF + temporal residual	84.9%	87.6%	90.3%	315
Global word/bigram TF-IDF	83.3%	86.8%	90.0%	309
Global lemma TF-IDF, unmasked	79.8%	84.2%	90.0%	296
Global lemma TF-IDF, names masked	74.1%	79.5%	85.2%	275
Three-act temporal alignment	70.9%	76.8%	83.6%	263
Orderless segment control	66.8%	74.1%	83.6%	248
Name-masked three-act alignment	66.6%	72.6%	79.2%	247
Name-masked orderless control	68.7%	75.4%	83.6%	255
Released Story Kernel predictions, same held-out subset	65.2%	—	—	242

The last row evaluates the authors' released predictions only on this pilot's held-out subset. It is not a new implementation of the original Story Kernel. The paper reports P@1 of 0.637 on its full released prediction set; that number and the subset result above should not be treated as a strict head-to-head comparison with the new split.

Repository layout

.
├── src/
│   └── temporal_narrative_pilot.py
├── notebooks/
│   └── temporal_narrative_alignment.ipynb
├── results/
│   ├── pilot_accuracy.png
│   ├── pilot_cases.csv
│   ├── pilot_comparisons.csv
│   ├── pilot_results.csv
│   └── pilot_summary.json
├── tests/
│   └── test_pilot.py
├── CITATION.cff
├── THIRD_PARTY_NOTICE.md
├── requirements.txt
└── requirements-notebook.txt

The repository does not redistribute plot summaries or the downloaded archive.

Reproduce the pilot

Python 3.12 was used for the reported run and is required by the pinned direct dependencies in this repository.

git clone https://github.com/henry653/temporal-narrative-alignment.git
cd temporal-narrative-alignment

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

python src/temporal_narrative_pilot.py

On first use, the script downloads the official 7.8 MB archive to .cache/movie-remake-dataset/, verifies the pinned SHA-256 checksum fd8ee883334ccb9964ba96a56d4fabdf5423cf37e88de7e3a2418435773f77b7, and extracts it locally. A normal run writes aggregate artifacts to results/. Download time depends on the network; the analysis itself is designed to run on CPU.

To use an already extracted copy of the dataset:

python src/temporal_narrative_pilot.py \
  --data-root /path/to/MovieRemakeDataset_NAACL2018

For a fast pipeline check with reduced resampling:

python src/temporal_narrative_pilot.py \
  --bootstrap 200 \
  --randomizations 1000 \
  --output-dir scratch-results

The default reported configuration is:

python src/temporal_narrative_pilot.py \
  --seed 20260807 \
  --bootstrap 10000 \
  --randomizations 100000 \
  --output-dir results

Notebook

Install the notebook environment and launch JupyterLab from the repository root:

python -m pip install -r requirements-notebook.txt
jupyter lab notebooks/temporal_narrative_alignment.ipynb

The committed notebook is executed and contains the aggregate tables and figure. Running it again invokes the same source script, refreshes results/, and keeps the downloaded corpus in the ignored local cache.

Tests

The unit tests do not download the dataset:

python -m unittest discover -s tests -v
python -m py_compile src/temporal_narrative_pilot.py

Continuous integration runs these checks on Python 3.12.

Output files

• pilot_results.csv: P@1, MRR, Hit@5, confidence intervals, and query counts by method.
• pilot_comparisons.csv: paired differences, cluster-bootstrap intervals, raw randomization p-values, and Holm-adjusted p-values.
• pilot_cases.csv: bounded title-only examples for qualitative inspection.
• pilot_summary.json: machine-readable configuration, data checks, metrics, and limitations.
• pilot_accuracy.png: the main comparison figure.

Troubleshooting

• Checksum mismatch: remove .cache/movie-remake-dataset/MovieRemakeDataset_NAACL2018.zip, confirm the official resource page is available, and retry. Do not bypass the checksum without reviewing a changed upstream release.
• Offline execution: download and extract the official archive separately, then pass the leaf directory containing movieRemakesManuallyCleaned.tsv, predictionsStoryKernel.csv, and processedSummaries/ with --data-root.
• Incomplete local dataset: --data-root must point directly to the extracted MovieRemakeDataset_NAACL2018 directory, not its parent.
• Wrong output location: run commands from the repository root or set an explicit --output-dir.
• Refresh from scratch: remove the ignored .cache/movie-remake-dataset/ directory and rerun the script. Tracked aggregate results are overwritten only when --output-dir results is used.

Limitations

• This is a lexical pilot, not an exact reimplementation of the paper's character-alignment kernel.
• Three equal sentence blocks are a coarse proxy for narrative phases.
• The BookNLP mask can miss names or mask the wrong mentions.
• Shared Wikipedia writing patterns, titles, and names may create retrieval shortcuts.
• The dataset contains known remakes rather than graded human judgments of broader narrative similarity.
• TF-IDF feature fitting is transductive: all 577 texts contribute vocabulary and IDF statistics, although test labels are not used.
• Remake-cluster membership is a proxy relevance label; cross-cluster films are treated as negatives even though some may share substantial narrative structure.
• The design and hyperparameters are exploratory. A confirmatory study should freeze the protocol before evaluation on an external corpus.

A natural next step is name-invariant event-role alignment: map characters to anonymous within-story roles, embed structured events, and use monotonic or soft-DTW alignment across the resulting event sequences.

Data and licensing

The code and repository-authored documentation are released under the MIT License. The Movie Remake Dataset is a separate third-party resource. Its downloaded bundle does not include an explicit license grant, although it draws on the CMU Movie Summary Corpus, whose official page labels that upstream corpus CC Attribution-ShareAlike and requests citation of Bamman, O'Connor, and Smith (2013). The processed annotations are associated with Bamman, Underwood, and Smith (2014). This repository therefore contains only code, aggregate metrics, a figure, and title-level cases; it does not redistribute plot summaries or BookNLP token files. See THIRD_PARTY_NOTICE.md for the scope of each attribution. Users are responsible for reviewing the official resource pages and applicable terms before using the data.

Citation

If this repository is useful, cite the original paper:

@inproceedings{chaturvedi-etal-2018-heard,
  title = {Where Have {I} Heard This Story Before? Identifying Narrative Similarity in Movie Remakes},
  author = {Chaturvedi, Snigdha and Srivastava, Shashank and Roth, Dan},
  booktitle = {Proceedings of NAACL-HLT 2018, Volume 2 (Short Papers)},
  year = {2018},
  pages = {673--678},
  doi = {10.18653/v1/N18-2106},
  url = {https://aclanthology.org/N18-2106/}
}

The complete ACL Anthology record is also available in REFERENCES.bib.