Skip to content

Fix dynamic RoPE scaling formula - #476

Merged
jonatanklosko merged 2 commits into
elixir-nx:mainfrom
dkuku:fix/dynamic-rope-scaling
Oct 5, 2026
Merged

jonatanklosko merged 2 commits into
elixir-nx:mainfrom
dkuku:fix/dynamic-rope-scaling

Conversation

@dkuku

@dkuku dkuku commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

In dynamic RoPE scaling, base was inadvertently raised to the power of dim / (dim - 2) along with the context scaling multiplier due to pipeline operator ordering.
According to NTK-aware dynamic scaling:

$$ \text{base}' = \text{base} \times \left( \frac{\alpha \times s}{s_{\text{orig}}} - (\alpha - 1) \right)^{\frac{d}{d-2}} $$

Previously, (base * scale) was raised to the exponent. This PR fixes the calculation so only the scaling factor is exponentiated before multiplying with base:

    # Previous (buggy): (base * scale) ^ (d / (d - 2))
    base
    |> Nx.multiply(scale)
    |> Nx.pow(size / (size - 2))

    # Fixed: base * (scale ^ (d / (d - 2)))
    Nx.multiply(base, Nx.pow(scale, size / (size - 2)))

References

• Matching implementation in Hugging Face modeling_rope_utils.py https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_rope_utils.py (_compute_dynamic_ntk_parameters)
• Mathematical derivation: NTK-aware Scaling Context Extension https://mbrenndoerfer.com/writing/ntk-aware-scaling-context-extension
• Original proposal: Reddit /r/LocalLLaMA discussion https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/

This PR fixes the calculation and adds a unit test in LayersTest.

codex and antigravity was used to find and fix it.

@jonatanklosko

Copy link
Copy Markdown
Member

Good catch, thank you!

@jonatanklosko
jonatanklosko merged commit c29a780 into elixir-nx:main Oct 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants