Modernize + verify 5 tutorial notebooks (Longformer/IMDB, node2vec, bikeshare, jigsaw) - #2
Open
jlealtru wants to merge 18 commits into
Open
Modernize + verify 5 tutorial notebooks (Longformer/IMDB, node2vec, bikeshare, jigsaw)#2jlealtru wants to merge 18 commits into
jlealtru wants to merge 18 commits into
Conversation
The notebooks were authored ~2020 against CUDA-only PyTorch with
unpinned/legacy deps. This commit introduces the reproducible
environment plus the first device-portable notebook; the remaining
notebooks will follow in their own commits.
Infrastructure
- pyproject.toml: every direct dep exact-pinned (torch 2.5.1,
transformers 4.46.3, datasets 3.1.0, gensim 4.3.3, spacy 3.8.2,
pecanpy 2.0.9, tensorboard 2.18.0, ...). Dev group has
jupyterlab/ipykernel/nbconvert/tqdm.
- uv.lock: hash-verified resolved graph. uv sync --frozen refuses
any dep whose hash doesn't match, so installs are reproducible.
- .python-version = 3.11 (gensim 4.3.3 requires scipy<1.14, which
only has wheels through 3.11).
- .gitignore excludes the venv, HF cache (data/.hf_cache/),
nbconvert reruns, Trainer checkpoints, training logs, wandb
dirs, macOS noise.
Scaffolding (notebooks/_utils.py)
- pick_device(): CUDA -> MPS -> CPU fallback.
- set_seed(): torch/numpy/random seeding.
- On import: sets PYTORCH_ENABLE_MPS_FALLBACK=1 and points HF_HOME /
HF_DATASETS_CACHE / HF_HUB_CACHE at data/.hf_cache/ so the repo
is self-contained (no surprise gigabytes under ~/.cache).
One-off patchers (scripts/)
- patch_notebooks.py, patch_training_args.py, patch_node2vec.py:
used to migrate the legacy notebooks to current library APIs
(transformers 4.46 renames, gensim 4 keyword renames,
stellargraph -> pecanpy). Run once each; kept in git so the
migration is reproducible/extendable.
- resume_roberta_imdb.py: restarts RoBERTa+IMDB from the latest
results/checkpoint-N/ if a long-running nbconvert is killed.
notebooks/RoBERTA with IMDB.ipynb
- Auto-injected setup cell uses pick_device() instead of the
original hard-coded 'cuda'.
- Drops the 2020-era /media/data_files/... Linux cache_dir; HF
cache goes through HF_HOME (set by _utils).
- Tokenization is now truncation-only at .map() time. Padding is
chosen by the data collator at training time based on device:
MPS: padding='max_length', max_length=512. Dynamic per-batch
shapes thrash MPSGraph's per-shape cache on Apple Silicon
-- observed 95 min wall, 0 optimizer steps, with 100% of
Python frames in MPSGraphSpecializationCache before this
fix.
CUDA/CPU: padding='longest'. CUDA tolerates variable shapes
cheaply and benefits from skipping pad FLOPs on
short examples.
- TrainingArguments: bf16=True replaces fp16=True (MPS doesn't
support fp16 well; CUDA/CPU ignore bf16 gracefully). Bigger
micro-batch (4 -> 32) with halved gradient_accumulation_steps
(16 -> 2) for the same effective batch 64 in fewer optimizer
steps. 4 dataloader workers (was 0). report_to='tensorboard'
with logging_dir='../results/runs' -- no external wandb account
needed.
…ikeshare, jigsaw) Bring five notebooks to current library versions and verify each runs end-to-end on the uv/Python 3.11 env (Apple Silicon MPS). Re-executed copies confirm 0 error cells. - Longformer+IMDB: add SMOKE_TEST toggle (subsample/short-seq/1-epoch); fix dead /media/... save path. Smoke-verified on MPS. - processing_capital_bikeshare: full run. Handle the mid-2020 trip-CSV schema change and the ArcGIS station-locations layer's new NAME/LATITUDE/LONGITUDE columns (was ADDRESS/ID). Produces graph_data_full.csv (117k edges). - node2vec: full run (721-node graph). Fix gensim Word2Vec iter -> epochs. - Multi_label roberta + longformer (jigsaw): fix pre-existing SyntaxErrors in from_pretrained(...) (missing commas), repoint dead /media checkpoint to roberta-base, DataLoader num_workers=0 (macOS spawn can't pickle notebook classes), add SMOKE_TEST toggle. Smoke-verified on synthetic schema-accurate data (real Kaggle data still required for a full run). - Normalize all 5 kernelspecs conda-env-torch-py -> python3. - Add kaggle to dev deps; gitignore large data dirs + model output dirs; document the SMOKE_TEST workflow and data prerequisites in README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- scripts/fetch_bikeshare.sh: download the 24 monthly Capital Bikeshare trip zips (2019+2020) from the public S3 bucket into data/capital_bikes/. - scripts/fetch_jigsaw.sh: download + unzip the Jigsaw Toxic Comment competition data into data/jigsaw/ via the kaggle CLI (requires token). - README: point the data-prerequisites table back at the two scripts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both IMDB transformer notebooks now execute end-to-end (0 error cells) under SMOKE_TEST=1 on the uv/Py3.11/MPS env. BigBird text classification.ipynb: - Add SMOKE_TEST env toggle (subsample + short seq + 1 epoch); full config is the default. Threads MAX_LENGTH/N_SAMPLE/NUM_EPOCHS/WARMUP/GRAD_ACCUM through the tokenizer, tokenization, training args, collator, and dataset subsample. - Fix the leftover hard-coded /media/... save path in the save-model cell (was OSError: Read-only file system) -> ../results/bigbird_base_imdb. - Normalize dead conda-env-torch-py kernelspec -> python3. RoBERTA with IMDB.ipynb: - Add the same SMOKE_TEST toggle (it had none) so it can be re-smoke-verified without the ~24h full fine-tune. Normalize kernelspec -> python3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both ETM notebooks now execute end-to-end (0 error cells) under SMOKE_TEST=1 on the uv/Py3.11/MPS env. Data plumbing: - scripts/fetch_pitchfork.py: pulls reviews.csv from the HF mattismegevand/pitchfork dataset, remaps to the notebooks' schema (rating->score, synthetic unique `link`), and generates the missing stop.txt (spaCy English stopwords) into data/pitchfork/. - pyproject: new `etm` dependency-group pinning the spaCy models the notebooks load (en_core_web_lg + en_core_web_md); install with `uv sync --frozen --group etm`. Notebook changes (both): - SMOKE_TEST toggle: subsample documents, relax vocab pruning (no_below), and cut epochs so the full pipeline runs in minutes; default reproduces the full config. - WANDB_MODE=disabled by default (was hardcoded to the private jlealtru/ETM_runs_p entity, which no one else can write to). - spaCy nlp.pipe n_process=1 and DataLoader num_workers=0: spaCy multiprocessing and notebook-defined Dataset/collate_fn deadlock/fail under nbconvert on macOS (spawn). - Corpus/dictionary caches tagged by mode so smoke and full runs don't clobber. - Normalize dead conda-env-torch-py kernelspec -> python3. - etm_preprocessed_data: clamp a hard-coded inspection index (docs[9984]) so it also works on the smaller smoke subsample. Removed trailing scratch cells that were already broken before modernization (never executed in the original: truncated pyLDAvis call, `.to.` typo, undefined vocab/model/gammas, an ETM-API-mismatched EVAE experiment), so each notebook now runs clean top to bottom. README: document scripts/fetch_pitchfork.py + `uv sync --group etm`, and add BigBird and the two ETM notebooks to the SMOKE_TEST list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e sampling Fixes three P1 findings from code review. All 7 affected notebooks re-verified end-to-end under SMOKE_TEST=1 (0 error cells each). P1a — _utils import failed from the documented repo-root Jupyter launch. The setup cell did `sys.path.insert(0, '.')`, but _utils.py lives in notebooks/, so a kernel started at the repo root raised ModuleNotFoundError. Replaced with a small loop that locates notebooks/_utils.py from the repo root, notebooks/, or a subdir. Applied to all 7 notebooks that import _utils. Verified the import works from both the repo root and notebooks/. P1b — BigBird no longer reliably used block-sparse attention. Transformers 4.46.3 silently switches BigBird to dense original_full attention when seq_len <= 704 (2*block_size + 3*block_size + num_random_blocks*block_size, block_size 64). The prior dynamic 'longest' padding on CUDA/CPU let batches fall below that and go dense (defeating the tutorial's point; OOM risk). Now pad every batch to a fixed MAX_LENGTH on all backends. Bumped the smoke seq from 128 to 768 (smallest multiple of 64 above 704) so the smoke run still exercises block-sparse. Verified: no "Changing attention type" warning at 768; the full run at 1024 stays block-sparse. P1c — Jigsaw smoke runs were neither short nor sampled. The custom Dataset classes hard-coded max_length (3048/2048 for Longformer, 512 for RoBERTa) instead of MAX_LENGTH, and prediction ran over the entire Kaggle test set. Threaded MAX_LENGTH into every encode_plus call and slice the test set to N_SAMPLE under SMOKE_TEST. Verified on synthetic data: test sampled 300 -> 64 rows, seq 128 (RoBERTa) / 512 (Longformer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
etm_preprocessed_data.ipynb and etm_spacy_pipeline.ipynb both cached
corpus{SUFFIX}.mm / dict{SUFFIX}.pkl / docs{SUFFIX}.pkl under
data/pitchfork/ despite using different preprocessing (en_core_web_lg
lemmas vs en_core_web_md lowercase tokens), and only the former writes
doc_ids{SUFFIX}.pkl. Its "skip tokenization if corpus exists" check then
trips over the other notebook's cache: FileNotFoundError on doc_ids if
the spacy notebook ran first, or a silent corpus/doc_ids mismatch in the
opposite order. Namespace this notebook's cache files with _lg.
Found by running both notebooks in sequence on CUDA (RTX 3090).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified on CUDA (RTX 3090, torch 2.5.1+cu124): all smoke tests pass. The MPS-tuned compromises now switch on device.type so one notebook serves both machines: - IMDB notebooks (RoBERTa/Longformer/BigBird): dataloader_pin_memory was hard-coded False for MPS; pinned host memory is a straight win on CUDA. - Multilabel notebooks (RoBERTa/Longformer): fp16=False becomes bf16=(device.type=='cuda') — matching the bf16 choice of the IMDB notebooks (Ampere+) while keeping MPS/CPU in fp32 — and dataloader_num_workers 0 -> 4 with pinned memory on CUDA (the per-item Python tokenization in Data_Processing benefits directly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ETM notebooks already call spacy.prefer_gpu(), which silently
stays on CPU unless cupy is installed. cupy is CUDA-only, so it cannot
live in the base dependencies (the env must stay installable on
macOS); expose it as an opt-in group instead:
uv sync --frozen --group spacy-cuda
Pinned to the same era as the rest of the stack. The sys_platform
marker makes the group a no-op on macOS.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both Pitchfork ETM notebooks rebuilt every document's dense BOW in
python inside Data_Processing.__getitem__, once per document per
epoch, pinning one CPU core while the GPU idled at ~4%. Expand the
sparse corpus once into a float32 (num_docs x vocab_size) matrix
(1.1-1.7 GB, fits comfortably on the GPU), move it to the device, and
train on shuffled index batches sliced from it. Loss computation,
hyperparameters and epoch counts are unchanged; the loops now also
track per-epoch losses and plot NELBO/KL curves after training.
Verified on the RTX 3090 (cold caches, full config; executed copies
kept locally in results/_nbruns/speedup_{smoke,full}/):
- etm_preprocessed_data: 53m07s -> 18m32s wall, training cell
43m53s -> 9m03s; final NELBO 1913.01 vs 1913.17 baseline; topic
diversity 0.352 vs 0.351. GPU util ~4% -> ~96% during optimization;
most of the remaining cell time is the every-40-epoch topic
metrics (CPU gensim coherence), unchanged from baseline.
- etm_spacy_pipeline: 1h44m57s -> 13m32s wall, training cell
1h36m05s -> 4m39s; final loss/doc 2289.9 vs 2345.9; GPU util
sustained 97-98%.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
etm_spacy_pipeline only reads lexical attributes (lower_, is_stop, is_punct, is_digit), which the tokenizer alone provides, yet the pipeline still ran tok2vec/tagger/attribute_ruler/lemmatizer on all 20,869 reviews. Disable every component: benchmarked byte-identical output on 1,000 docs, 88.8s -> 7.1s CPU (projected ~2.5 min full corpus vs the 7m58s the cell took with GPU tokenization). The other ETM notebook keeps its full pipeline because it lemmatizes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrite the comments added during modernization (SMOKE_TEST block, setup shim, dense-matrix and training-loop notes) in the notebooks' own short style. Comment-only edits were patched via nbformat so existing cell outputs are preserved. etm_spacy_pipeline defined get_topics/get_topic_diversity/ get_most_similar_words but never called them, ending at the training cell. Add the post-training exploration the other ETM notebook has: beta from the trained model, topic words, topic diversity, and nearest neighbors. nearest_neighbors called vocab.index(word), which gensim's Dictionary does not have (latent bug, function was never exercised); use vocab.token2id[word]. Both notebooks pass SMOKE_TEST=1 nbconvert end to end including the new cells. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dieng et al.'s training script feeds the encoder length-normalized bows while keeping the reconstruction loss on raw counts; the notebook passed raw counts to both. Normalize at the three input sites (training loop, document_inference, and the single-doc demo) so inference sees the same input scale as training. Full 800-epoch run (warm caches, executed copy kept in results/_nbruns/speedup_full/etm_preprocessed_data__bownorm.ipynb): final NELBO 1912.54 vs 1913.01 unnormalized, topic diversity 0.352 -> 0.367, and much sharper document-topic assignments on the test set (a country record at 0.93 on the country/folk topic where nothing cleared 0.68 before). Topics remain clean genre topics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Compares google/embeddinggemma-300m against MiniLM and MPNet on the same train split, reduced to the ETM's 20 topics. The section self-skips when sentence-transformers/transformers are too old or HF_TOKEN is missing, since the model is gated. Bumps transformers, tokenizers and sentence-transformers to the versions that carry the Gemma3 encoder, adds python-dotenv to read HF_TOKEN from .env, and pins default-groups so uv sync stops pruning the notebook deps. Also fixes what the rerun uncovered: the stopword list now unions in sklearn's list plus the contraction stems its tokenizer leaves behind, the embedding cells force a re-encode instead of loading a stale cache, and the split-size assert is dropped now that pitchfork.csv has grown past the counts the ETM notebooks were written against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Trainer renamed evaluation_strategy to eval_strategy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The saved outputs predated the corpus growing to 25,705 reviews, so the topic lists, inference tables and timings below the split cell still described the old 17,738/3,131 run. Re-executed top to bottom against the current corpus: 21,849/3,856, minilm 27 topics reduced to 20 at 0.282 diversity, mpnet 29 reduced to 20 at 0.305, the etm reference is 0.367. Also moves the mpnet encode up beside the minilm one. Encoding after the first pipeline has been fitted leaves umap's knn graph, hdbscan's prediction data and the soft membership matrix resident, and on unified memory the encoder competes with all of it: 59.6 min against 8.7 for the same work in an empty kernel. Moving it cut the encode to 10.0 min and, unexpectedly, the mpnet fit from 16.4 min to 0.4 -- the fit was being starved too. Whole notebook is ~12 min against ~77, with identical clusters, diversity and per-document probabilities. The comment claiming mpnet is five times slower to encode was wrong on both counts and now records the measurement instead. Fixes the EmbeddingGemma requirements note, which told readers to build a separate env after pyproject had already been bumped to the versions that carry the Gemma3 encoder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings five tutorial notebooks to current library versions and verifies each runs end-to-end on the
uv/ Python 3.11 environment (Apple-Silicon MPS). Re-executed copies confirm 0 error cells.Notebooks
Longformer with IMDB.ipynbSMOKE_TESTtoggle; fixed dead/media/...save pathprocessing_capital_bikeshare_data.ipynbNAME/LATITUDE/LONGITUDE; emitsgraph_data_full.csv(117k edges)node2vec with capitol bikeshare data.ipynbWord2Vec(iter=)→epochs=; 721-node graphMulti_label_classification_roberta.ipynbSyntaxErrorfix; dead checkpoint →roberta-base;num_workers=0;SMOKE_TESTMulti_label_classification_longformer_tutorial.ipynbAlso
conda-env-torch-py→python3.kaggleto dev deps; gitignore large data dirs + model output dirs.SMOKE_TESTworkflow and data prerequisites.Notes for reviewer
~/.kaggle/kaggle.json+ accepted competition rules).scripts/fetch_*.sh), regeneratednotebooks/images/*, and unrelated prior-session edits are intentionally excluded from this PR.🤖 Generated with Claude Code