Skip to content

fix(simclr): normalise per series and turn off permutation in the default augmentation - #107

Open
aryamaan-roy wants to merge 5 commits into
chronocratic:devfrom
aryamaan-roy:fix-simclr
Open

aryamaan-roy wants to merge 5 commits into
chronocratic:devfrom
aryamaan-roy:fix-simclr

Conversation

@aryamaan-roy

Copy link
Copy Markdown
Contributor

Fixes #99.

Problem

With its default augmentation, a trained SimCLR encoder scores lower than an untrained one in a linear-probe evaluation: 61% against 66% on CricketX, and 40% against 58% on InsectWingbeatSound.

  • Permutation. SimCLR's loss compares the two views timestep by timestep, so permuting segments trains the encoder to ignore where things happen in a series. Turning permutation off alone raises InsectWingbeatSound from 40% to 61%.
  • Per-series scaling. The weak view multiplies each series by one factor, which the encoder's GroupNorm(1, C) cancels.
  • MINMAX(0, 1) data. Without per-series normalisation the loss stays close to chance, as with TS-TCC before fix(tstcc): stop default augmentation from collapsing the encoder to chance #100.

Changes

Same pattern as #100:

  • New default pair: TSTCCScaling (per-timestep) as the weak view and Jitter(0.8) as the strong view, with permutation off. _default_simclr_pair(max_segments=5) restores the reference's strong view.
  • instance_normalize=True: per-series z-normalisation over time, in training and in encode().
  • No chance-loss warning: here the loss ends well below chance, so the warning would not trigger.
  • Old checkpoints: load them with instance_normalize=False, because encode() now normalises its input.

Testing

  • Regression test, tests/integration/test_simclr_learns.py: it fails on the old code and passes with the fix. It checks normalisation only; the probe below covers permutation.
  • Unit tests for the new default pair and for instance_normalize.
  • Full suite: 1,700 tests pass.

Probe accuracy at B=128, mean of 3 seeds. The probe is an RBF SVM on encode() features (TS2Vec protocol), and the old pair was run with normalisation on, so only the augmentation differs:

CricketX InsectWingbeatSound
untrained encoder 66% 58%
old default pair 61% 40%
new default 74% 63%

At B=16 the new default gives 70% and 62%. ECG5000, FordA and TwoPatterns are within about 1 point under every setting.

…m features

Permutation off by default, per-timestep weak view, instance_normalize in training and encode() (chronocratic#99).
if max_segments > 1
else jitter
)
return RolePairProducer(first=TSTCCScaling(), second=strong)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is wrong to use TSTCCScaling here. Please change its name, move it somewhere else, or make a new one for SimCLR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it to chronocratic.models.augmentation.primitives as TimestepScaling, next to Scaling. TS-TCC and SimCLR both import it from there.

from chronocratic.models.convolutional.standard.encoders.resnet import Conv1dResNetEncoder
from chronocratic.models.convolutional.standard.simclr.augmentations import _default_simclr_pair
from chronocratic.models.convolutional.standard.simclr.config import _MIN_SINGLETON_SPLIT_COUNT
from chronocratic.models.convolutional.standard.tstcc.model import _instance_normalize

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is wrong to use private functions outside tstcc. This needs to be restructured so that the function exists where both models can reach it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it to chronocratic.models.utils as the public normalize_per_series, next to zero_fill_padding. Both models import it from there.

TSTCCScaling is now TimestepScaling in augmentation.primitives; _instance_normalize is now utils.normalize_per_series.

@skafai skafai left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @aryamaan-roy ,

See the requested changes.

# --------------------------------------------------------------------------- #


@dataclass

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two should not be primitives, because they are not.

Consider moving them somewhere like

src/chronocratic/models/convolutional/standard/augmentaion/timestep_scaling.py (augmentation can be a package with init to export this method in a barell)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved both out of primitives.py into convolutional/standard/augmentation/timestep_scaling.py, with an init.py that re-exports them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SimCLR: default augmentation may collapse training to chance (same failure as TS-TCC)

2 participants