Skip to content

Save 20~30 minutes on every shared memory run. - #15

Open
ezhang7423 wants to merge 1 commit into
lukashermann:mainfrom
ezhang7423:patch-1
Open

Save 20~30 minutes on every shared memory run.#15
ezhang7423 wants to merge 1 commit into
lukashermann:mainfrom
ezhang7423:patch-1

Conversation

@ezhang7423

Copy link
Copy Markdown

This addition to the code modifies the behavior of the multiprocessing.resource_tracker module in Python. It applies a monkey patch to ensure that the SharedMemory resource type is not tracked by the resource tracker.

The purpose of this modification is to prevent the training and validation language episodes from being removed from memory each time. By allowing these episodes to persist in memory, the code can save a significant amount of time, approximately 20 to 30 minutes, in each run.

The modification achieves this by overriding the register and unregister functions of the resource tracker. When a resource type is "shared_memory," these functions return None, effectively skipping the registration and unregistration process.

Additionally, the modification removes the "shared_memory" entry from the _CLEANUP_FUNCS dictionary of the resource tracker. This ensures that the cleanup function for shared memory is not called.

By making these changes, the code avoids the unnecessary overhead of reloading the training and validation language episodes, resulting in a considerable time-saving benefit during execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant