Initial changes for Recurrent state rollback for nemotron for cuda

[lnigam]

Initial changes for Recurrent state rollback for nemotron for cpu and cuda

Overview

This PR adds support for Recurrent state rollback for nemotron model. It creates separate copies of states for every draft token and then rollback to the last accepted token's state. Also added test case to verify the ssm support

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes, for code review and help with test case creation

[ggml-gh-bot]

Hi @lnigam, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[ggerganov]

The main problem atm is that I missed the fact that when the new operator is not supported, we don't just need to update the compute graph - we also have to communicate back to the app that the requested n_rs_seq was not satisfied. This gets messy as it requires some extra logic outside of libllama and generally it is something that I would like to avoid.

My suggestion is to drop the llm_fused_op_probe and cparams.fused_ssm_scan related stuff and directly implement the new op across all backends. This is what we actually did for the ggml_gated_delta_net op - a very similar case to this one:

#24086

We can do that because the change in the backends is relatively minimal and an agent should be able to do it correctly single-shot. Just make sure the provided tests in test-backend-ops.cpp are good enough in order to catch in the CI if some of the backends failed it (I think the current ones are good).

cc @gaugarg-nv for vis, as you successfully did this for the ggml_gated_delta_net op before.

---

If this turns to be difficult to do for all the backends, then a simpler option is to just do it for the CPU and CUDA backend (i.e. you already did that) and then mark all backends that they do not support the op. This will cause the computation to fallback to CPU on these devices until it gets implemented. But that should be fine and more importantly, the rollback logic would still work - it would just be a bit slower, since it is going to run on the CPU.