Skip to content

Implement true CLT: per-source-layer encoders/decoders with causal masking#12

Open
bartbussmann wants to merge 1 commit into
mainfrom
codex/review-clt-implementation-for-correctness
Open

Implement true CLT: per-source-layer encoders/decoders with causal masking#12
bartbussmann wants to merge 1 commit into
mainfrom
codex/review-clt-implementation-for-correctness

Conversation

@bartbussmann

Copy link
Copy Markdown
Owner

Motivation

  • Support a true Cross-Layer Transcoder (CLT) mode where features are per-source-layer and may write to multiple later target layers, and enforce causal source→target writes to match the CLT semantics.
  • Make CLT configuration explicit by allowing multiple input layers (num_input_layers) so concatenated multi-layer inputs are handled correctly.
  • Ensure training utilities (decoder normalization, auxiliary loss, dead-feature tracking, and BatchTopK) correctly handle the 3D activation tensors produced by true CLT mode.

Description

  • Added num_input_layers to EncoderConfig and introduced multi_layer / true_clt flags in SharedEncoder to detect CLT mode via num_output_layers > 1 and num_input_layers > 1 and to compute input_size_per_layer when needed.
  • Reworked encoder weights to support per-source-layer features: W_enc/b_enc become shape (num_input_layers, input_size_per_layer, dict_size) / (num_input_layers, dict_size) in true CLT mode, and added encode_linear to compute per-layer pre-activations.
  • Expanded decoder to per-(source,target)-layer decoders W_dec with shape (num_input_layers, num_output_layers, dict_size, output_size) in true CLT mode and registered a causal_mask buffer so source layer s can only write to target layers t >= s, and applied that mask during decode and make_decoder_weights_and_grad_unit_norm.
  • Updated update_inactive_features, auxiliary-loss path, BatchTopK flattening logic, and forward paths to correctly handle 3D activations ((B, S, D)) produced by CLT encoders and to preserve backward compatibility in non-CLT mode.
  • Added CLT smoke test coverage: changed shape assertions for the concatenated-input CLT test and added test_clt_is_causal to verify perturbations to later-layer inputs do not affect earlier-layer outputs.

Testing

  • Successfully compiled the modified modules with python -m py_compile base.py config.py test_clt.py which reported no syntax errors (success).
  • Attempted to run python test_clt.py but the run failed in this environment with ModuleNotFoundError: No module named 'torch', so the full runtime tests could not be executed here (failure due to missing dependency).

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant