Migrated from: goodfire-ai/spd-gf#67
Original author: @leesharkey
We’d like to evaluate if the method is stable during training with a subset of the dataset. We think this is important to ensure training at scale actually works. This is because, as we scale, our SPD model will increasingly see less and less of the training dataset in a time-window of a fixed size.
The approach here is to take e.g. a trained TMS model and then try to decompose the network using a dataset that is different from the training distribution. In particular, train it using a dataset where some of the features are always off.
In an ideal world, we would like PD to decompose the network into components that correspond to the active features and ‘the rest’, where the rest includes the features that weren’t active in the PD training dataset. The purpose of this experiment is to test the robustness of PD to distributional shift.
Another way to test this if the above succeeds: First do the above and get your decomposition components corresponding to ‘active’ and ‘the rest’ components. Then, start a new phase of PD training that uses the originalTMS dataset. See if it transitions gracefully. (it might not transition if, for instance, gates have been turned off. This matters for stability of general)
Migrated from: goodfire-ai/spd-gf#67
Original author: @leesharkey
We’d like to evaluate if the method is stable during training with a subset of the dataset. We think this is important to ensure training at scale actually works. This is because, as we scale, our SPD model will increasingly see less and less of the training dataset in a time-window of a fixed size.
The approach here is to take e.g. a trained TMS model and then try to decompose the network using a dataset that is different from the training distribution. In particular, train it using a dataset where some of the features are always off.
In an ideal world, we would like PD to decompose the network into components that correspond to the active features and ‘the rest’, where the rest includes the features that weren’t active in the PD training dataset. The purpose of this experiment is to test the robustness of PD to distributional shift.
Another way to test this if the above succeeds: First do the above and get your decomposition components corresponding to ‘active’ and ‘the rest’ components. Then, start a new phase of PD training that uses the originalTMS dataset. See if it transitions gracefully. (it might not transition if, for instance, gates have been turned off. This matters for stability of general)