Skip to content

Implement cleanup of run files upon model session and run deletion - #829

Open
Creylay wants to merge 1 commit into
developfrom
fix/run-path-directory-cleanup
Open

Implement cleanup of run files upon model session and run deletion#829
Creylay wants to merge 1 commit into
developfrom
fix/run-path-directory-cleanup

Conversation

@Creylay

@Creylay Creylay commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes a bug where Run.run_path on disk can be either a file (scikit-learn/torch models) or a directory (Hugging Face models). Since Run.id gets reused by SQLite once rows are deleted, a stale artifact left behind by one run could collide with a new run that lands on the same id, crashing training with IsADirectoryError/PermissionError. This same file-vs-directory assumption also broke DELETE /run/{id}: deleting a run whose run_path was a directory always failed with a 500, since it used os.remove (files only) instead of a helper that handles both. Also fixes a related cleanup gap where deleting a model session never removed its runs' files from disk, letting stale artifacts orphan in the first place.


Type of Change

Check all that apply like this [x]:

  • Backend change
  • Frontend change
  • CI / Workflow change
  • Build / Packaging change
  • Bug fix
  • Documentation

Changes (by file)

  • DashAI/back/job/model_job.py: clear any pre-existing file/directory at run_path before saving a trained model (root-cause fix).
  • DashAI/back/api/api_v1/endpoints/runs.py: delete_run now uses remove_p, so runs whose run_path is a directory (Hugging Face models) can actuallybe deleted.
  • DashAI/back/api/api_v1/endpoints/model_sessions.py: delete_model_session now cleans up the run_path/plot files of cascaded runs on disk instead of leaving them orphaned.
  • DashAI/back/api/utils.py: relocated the shared remove_path helper here el_job.py` can use it.
  • tests/back/api/test_jobs.py, tests/back/api/test_runs_api.py: added regression tests for the three scenarios above.

Testing

  • Manually reproduced the original bug (train a Hugging Face model → delete its model session → train new runs until an id is reused) and confirmed it no longer fails, and that the affected run can be deleted normally.
  • Manually deleted a run whose run_path was a directory (Hugging Face model). Deletion now succeeds instead of returning a 500.

@Creylay Creylay added bug Something isn't working back Backend work labels Aug 18, 2026
@cristian-tamblay cristian-tamblay added the conflict This PR has merge conflicts label Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

back Backend work bug Something isn't working conflict This PR has merge conflicts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants