Skip to content

Add SpliceBERT 1024nt results - #1

Open
chenkenbio wants to merge 2 commits into
darlednik:mainfrom
chenkenbio:agent/add-splicebert-1024nt
Open

Add SpliceBERT 1024nt results#1
chenkenbio wants to merge 2 commits into
darlednik:mainfrom
chenkenbio:agent/add-splicebert-1024nt

Conversation

@chenkenbio

Copy link
Copy Markdown

Summary

  • add a GENEB embedding extractor for the original SpliceBERT.1024nt Zenodo checkpoint
  • add the harness-generated submission for all 100 tasks under the full, 10-shot, and 1-shot regimes
  • add a model card documenting training data, pooling, context handling, and overlap limitations

The encoder is frozen. Sequences longer than the native 1,024-nt context are processed with 1,024-nt windows at a 512-nt stride, with an end-anchored final window and equal averaging of window embeddings. Each window uses final-layer mean pooling over non-padding tokens, including [CLS] and [SEP].

Results

Regime Micro MCC Macro MCC
Full 0.5545 0.5680
10-shot 0.3354 0.3402
1-shot 0.1687 0.1545

The mean full-data MCC across the seven splice-site tasks is 0.6334.

Disclosure

  • provenance is marked self-reported
  • zero-shot status is not claimed because sequence-level pretraining overlap has not been established
  • the checkpoint was pretrained on vertebrate primary RNA; potential genomic-locus overlap is documented in the model card
  • GENEB sequences shorter than the 64-nt pretraining minimum are evaluated unchanged, while overlength sequences use the documented sliding-window policy

Validation

  • python tools/validate_submission.py submissions/splicebert-1024nt.json — valid submission with 100 tasks in each regime
  • python tools/build_leaderboard.py — rebuilt successfully with 41 models and 100 tasks across three regimes
  • focused extractor tests — 9 passed
  • extractor Python compilation and git diff --check — passed
  • complete GPU evaluation — 100/100 tasks, exit status 0

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f8a479a007

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread harness/extractors/splicebert.py Outdated
Comment thread harness/extractors/splicebert.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant