grafana fails with rbac errors on startup

[ardentperf]

steps to reproduce

scripts/setup.sh
monitoring/setup.sh

the grafana operator fails to start up the grafana pod itself, and the following error appears in the logs for the operator:

grafana-operator-controller-manager-85c8457c4b-rvgml 2026-01-05T19:14:08.081Z error Reconciler error {"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"grafana","namespace":"grafana"}, "namespace": "grafana", "name": "grafana", "reconcileID": "01afe12c-f995-45a0-92e2-1a1f4632f78a", "error": "updating grafana version in spec: grafanas.grafana.integreatly.org \"grafana\" is forbidden: User \"system:serviceaccount:grafana:grafana-operator-controller-manager\" cannot update resource \"grafanas\" in API group \"grafana.integreatly.org\" in the namespace \"grafana\""}

it looks like the out-of-the-box privilege configuration with the official grafana operator is missing the update verb. will write a quick PR to add a service account and role for grafana, similar to what is done in monitoring/prometheus-instance/deploy_prometheus.yaml

[sxd]

I haven't been able to reproduce this issue on main, there's any additional step to reproduce it? For me the pods are starting without problems
In any case, I think that this should be solved upstream like they already did couple of weeks ago here grafana/grafana-operator#2391 and not by us

[ardentperf]

I wonder if the repro is tied to the specific versions of kubernetes (i'm using the versions pinned via kind in the nix dev environment)? Or maybe a specific version of grafana? That I was using earlier in my testing? I will try to repro this again and get versions of grafana and k8s that I'm using for the repro

@sxd the quick fix in #45 exactly mirrors monitoring/prometheus-instance/deploy_prometheus.yaml ... do you think we should not be doing RBAC for prometheus either, if you're saying we shouldn't do it for grafana?

[ardentperf]

@sxd this is very close to being a repro on main https://github.com/ardentperf/cnpg-playground/actions/runs/20794005106/job/59722747736

i just ran the test from PR #48 without applying this fix first (and added a debug line to the test which prints the actual error from the controller after grafana failed to start) main...ardentperf:cnpg-playground:repro-issue-44

i'm actually a little surprised it works for you. what are your versions? #48 GH actions use the versions that are pinned in the CNPG playground nix devshell