Skip unreferenced static files when ingesting edX course archives

[mbertrand]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/13350

Description (What does it do?)

• documents_from_olx now skips static files nothing in the course refers to, matching on filename against the text of every block so that asset-v1:...+type@asset+block/<name> links count as well as /static/<name>, and percent-encoded and entity-escaped spellings do too.
• Legacy transcripts (subs_<id>.srt.sjson) are kept only when a video block declares their id via sub / youtube / youtube_id_1_0. Those are read by parsing the video elements, not by matching attribute text, so sub='x' and sub = "x" count.
• The asset manifests and info/updates.items.json are no longer ingested and no longer count as references. Announcements the course team deleted don't keep an asset alive; live ones still do.
• Staff-only files are excluded as reference sources too, so an asset only a hidden block links is unreferenced. A file a visible block still links stays, though, and a block hanging under both a visible and a staff-only parent is visible: courses keep staff-only duplicates of live video blocks, and the two copies share one set of transcripts.
• unpublish_staff_only_files becomes unpublish_excluded_files and walks the same exclusion function, so what ingestion skips and what the cleanup removes can't drift. It gains --dry-run, and reports what each run excludes out of the content files it has, optionally as a CSV.
• audit_olx_references reports the same exclusions against an extracted archive without touching the DB.

How can this be tested?

Uses 15.671.1x, the course from the issue.

1. Pull and extract the 3T2026 archive the counts below come from (560MB, so this is the slow step):

   aws s3 cp "s3://ol-data-lake-landing-zone-production/mitxonline/openedx/raw_data/course_xml/course-v1:MITxT+15.671.1x+3T2026/fc8120bd39cbdc0a21a8343f359b89f74a066647f0c7b49ca28e40d86d54477b.xml.tar.gz" /tmp/ulab.tar.gz
   mkdir -p /tmp/archives/ulab && tar xzf /tmp/ulab.tar.gz -C /tmp/archives/ulab

   A newer nightly archive works too, it just won't match the numbers exactly.

2. See what the filter excludes, no DB needed:

   docker compose run --rm -v /tmp/archives:/archives web \
     ./manage.py audit_olx_references /archives/ulab/course

   === /archives/ulab/course
     ingestable files      : 5430
     excluded              : 4184 (77%)
       under static/       : 2880
         transcripts       : 2748
         documents         : 132
       elsewhere           : 1304  (staff-only blocks, manifests, announcements)
   

3. To exercise the command itself, create the run and the content files ingestion would produce from that archive, without putting 5430 files through tika:

   seed.py

   cat > seed.py <<'PY'
   from pathlib import Path
   from learning_resources.constants import LearningResourceType, PlatformType, VALID_TEXT_FILE_TYPES
   from learning_resources.etl.constants import ETLSource
   from learning_resources.etl.utils import get_edx_module_id
   from learning_resources.models import ContentFile, Course, LearningResource, LearningResourceRun
   
   OLX = Path("/archives/ulab/course")
   resource, _ = LearningResource.objects.get_or_create(
       readable_id="course-v1:MITxT+15.671.1x",
       defaults=dict(title="u.lab", resource_type=LearningResourceType.course.name,
                     platform_id=PlatformType.mitxonline.name,
                     etl_source=ETLSource.mitxonline.name, published=True))
   Course.objects.get_or_create(learning_resource=resource)
   run, _ = LearningResourceRun.objects.get_or_create(
       learning_resource=resource, run_id="course-v1:MITxT+15.671.1x+3T2026",
       defaults=dict(published=True))
   keys = {get_edx_module_id(str(p.relative_to(OLX.parent)), run): p
           for p in sorted(OLX.rglob("*")) if p.is_file()
           and p.suffix.lower() in VALID_TEXT_FILE_TYPES
           and not any("draft" in d for d in p.relative_to(OLX).parts[:-1])}
   ContentFile.objects.filter(run=run).delete()
   ContentFile.objects.bulk_create([
       ContentFile(run=run, key=k, edx_module_id=k, published=True,
                   source_path=str(p.relative_to(OLX.parent)), file_extension=p.suffix.lower())
       for k, p in keys.items()])
   print("resource", resource.id, "content files", len(keys))
   PY

   docker compose run --rm -v /tmp/archives:/archives web \
     ./manage.py shell -c "exec(open('seed.py').read())"

   resource 2 content files 4932
   

   5430 paths collapse to 4932 keys because get_edx_module_id folds spaces to underscores.

4. The 2014 syllabus the issue names is served, and --dry-run says what would go without touching it:

   curl -s -G "http://localhost:8061/api/v1/contentfiles/" \
     --data-urlencode "edx_module_id=asset-v1:MITxT+15.671.1x+3T2026+type@asset+block@U.Lab_Syllabus_1.7.pdf" \
     | jq -c '{count, paths: [.results[].source_path]}'
   docker compose run --rm web ./manage.py unpublish_excluded_files \
     --source mitxonline --resource-ids 2 --dry-run --report /src/ulab_report.csv

   {"count":1,"paths":["course/static/U.Lab Syllabus 1.7.pdf"]}
   mitxonline run course-v1:MITxT+15.671.1x+3T2026: 3852 out of 4932 content files excluded
   mitxonline summary: 3852 out of 4932 content files excluded across 1 run(s), would unpublish 3852
   Wrote 1 rows to /src/ulab_report.csv
   

   cat ulab_report.csv

   etl_source,run_id,excluded,unpublished,total
   mitxonline,course-v1:MITxT+15.671.1x+3T2026,3852,3852,4932
   

   excluded counts the rows the archive excludes whatever their publish state, unpublished only the ones this call would flip, so a re-run shows the same 3852 excluded and 0 to unpublish.

5. Drop --dry-run to apply it, then re-check the counts and the same curl:

   docker compose run --rm web ./manage.py unpublish_excluded_files --source mitxonline --resource-ids 2
   docker compose run --rm web ./manage.py shell -c "
   from learning_resources.models import ContentFile
   cf = ContentFile.objects.filter(run__run_id='course-v1:MITxT+15.671.1x+3T2026')
   print(cf.filter(published=True).count(), cf.filter(published=False).count())"

   mitxonline run course-v1:MITxT+15.671.1x+3T2026: 3852 out of 4932 content files excluded
   mitxonline summary: 3852 out of 4932 content files excluded across 1 run(s), unpublished 3852
   1080 3852
   

   The curl from step 4 now returns {"count":0,"paths":[]}, and the same run queues deindex_run_content_files and remove_unpublished_run_content_files.

Additional Context

I ran step 2 against the current archive of one course per edX source. mit_edx 6.002.1x is the control: nothing in it is unreferenced, and the 434 transcripts it loses are all ones only staff-only blocks link, so 2162 of 2596 stay. Staff-only, manifests, transcripts and other static sum to the excluded total; videos counts staff-only blocks of all video blocks rather than files, so it sits outside that sum; one video carries a transcript per language.

archive	ingestable	excluded	staff-only	manifests	videos	transcripts	other static
mitxonline 15.671.1x 3T2026	5430	4184	1301	3	143 of 224	2748 of 3308	132 of 137
xpro AMx_Hubbell R2	1479	452	21	3	2 of 78	275 of 344	153 of 162
mit_edx 6.002.1x 2T2019	3843	654	218	2	32 of 189	434 of 2596	0 of 0
oll 11.405x 2T2020	2140	776	31	3	0 of 87	737 of 1357	5 of 6

Across those four, the filter never drops a transcript a visible video block declares: 7605 transcripts on disk, zero violations. 15.671.1x is the hard case, with 143 of its 224 video blocks staff-only; its 81 visible ones declare 574 transcript files in 13 languages, 560 of them in the archive, and all 560 are kept. Nor is the content lost — every distinctive line of the dropped 01_Intro_SPT_ITA.srt survives verbatim in the kept 1dd98469-ed52-4163-b76f-9f3866b10e93-it.srt, so what goes is an older copy of a replaced video.

Worth knowing before this runs on production: after the command, 15.671.1x keeps 5 of its 137 static documents, and none of them is a syllabus. The current run's syllabus link (u-lab_1x_Syllabus_2022.pdf) lives in html/1b8267754ed64889a30cfdf09a120e95.html, which is a visible_to_staff_only block, so #3909 already excludes the block and this PR now excludes the file it links. That's consistent — learners can't see that link either — but if we'd rather a run's syllabus survive regardless, that's a separate rule and I'd rather add it deliberately than by accident.

Two limits I'd rather state than have found in review. An asset mentioned only inside a hidden block that sits inline in a parent XML file is still kept, because there's no separate file to exclude from the scan; all four archives are pointer-style and #3909 found no production course with inline staff-only blocks, so this is unobserved, but the answer-key exclusion is best-effort rather than a guarantee.

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

🟡 Changes recommended

Multi-ID YouTube declarations can cause valid legacy transcripts to be incorrectly excluded.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 2
• Review effort level: Balanced