Allow nova_api_audit_map.conf in DefaultConfigOverride

[vyzigold]

Allow providing a nova_api_audit_map.conf file through the
DefaultConfigOverride to NovaAPI.

Conditionally configure the
audit_middleware_notifications.use_oslo_messaging
based on if notification_transport_url is defined or not.

This is being tested in functional tests, kuttl tests and the
nova-operator-tempest-multinode test job.

Jira: OSPRH-29257

[gibizer]

Could you please providing reasoning why this changes is needed?

[SeanMooney]

this might be requried for non default middelware? but we do not support modifying the mideddlware without a supprort exectpion so this probaly needs deicssion about how this will be tested and if it will be supproted in the product at all unless that has alreayd happend

in either case any feature work shoudl be assocated with a jira issue in osprh so at a minimum that context need to be capture before this proceeds.

[SeanMooney]

this appares to be realted to https://opendev.org/openstack/pycadf/src/branch/master/doc/source/audit_maps.rst#L43
but https://opendev.org/openstack/keystonemiddleware/src/branch/master/keystonemiddleware/audit/_api.py

has never been suprpoted in optream nova with any testing and is not supproted in our downstream today.

it might work but there is 0 testing or supprot for that form the nova maintaienr today
which is shy that woudl requrie a supprot expction to enable.

i dont nessiarlly object to supprotign the config file but customer woudl not be able to use this without violdating the supprot for the nova api.

if they have any api perfonce issue or geniss issue we woudl require them to repoduce with the default middelware set before there requrest would be supproted and all issue related to this woudl have to be feilded by the Secrity team or how eve fis the formal maintaienr of that in our downstrema

so can you please provide the jira planing docs and work wirht teh relevent product owners to ensure this is planned properly before we proceed.

[vyzigold]

We have a feature for providing documentation on how to enable and forward audit logs. I added the task to enable the nova_api_audit_map.conf customization in nova-operator in the PR description following an example I saw in some other PR here.

@SeanMooney we have a whole feature based around audit logs https://redhat.atlassian.net/browse/RHOSSTRAT-1255 . Plan is to provide documentation on how to enable audit logs and how to configure openshift-logging to distinguish them and to forward them into a separate log storage. As part of the feature we'll also work on providing test coverage. Feel free to leave feedback if you want.

[SeanMooney]

I have commented on the Jira, but this is not sufficient to enable this feature.

The CADF middleware that this is for uses RabbitMQ by default and falls back to the notification message bus if its own config options are not set:
https://opendev.org/openstack/keystonemiddleware/src/branch/master/keystonemiddleware/audit/__init__.py#L44-L66

It is not correct to set the transport_url via the CustomServiceConfig or the nova-api-audit-map.conf, as that would be in plain text. This means that today there is no protection from enabling the middleware via api-paste.ini when
you have not passed a notification bus.

This patch is also missing Kuttl tests to show that the feature works end-to-end in that environment. Ideally, this would be enabled in one of the EDPM jobs—either the Ceph job or the local storage job, whichever has notifications enabled.

The fact that middleware can use a RabbitMQ connection running in the Nova API process without our knowledge is also concerning from a performance and concurrency point of view, especially given the Eventlet removal efforts and the change to how that works now.

We do not allow long-running connections in the API process, and if it is killed and recreated on every request, that will also slow down every request by at least the TCP handshake time for every notification it sends.

[vyzigold]

We don't plan to use RabbitMQ or a notification bus, since we don't have anything, that'd listen there for these logs. The documentation will have users set the log driver for the audit middleware like this:

          [audit_middleware_notifications]
          driver = log

Logs will then get collected by the openshift-logging and stored.

I'll add kuttl-test coverage 👍

[SeanMooney]

We don't plan to use RabbitMQ or a notification bus, since we don't have anything, that'd listen there for these logs. The documentation will have users set the log driver for the audit middleware like this:

          [audit_middleware_notifications]
          driver = log

Logs will then get collected by the openshift-logging and stored.

I'll add kuttl-test coverage 👍

or we could default the value of use_oslo_messaging

[audit_middleware_notifications]
use_oslo_messaging = < true if notification bus is passed else false>

whey woudl still have to opt into the middlware by modifying the pipeline
but then it would route to the logs if notification were disabled
and send the notification to the notifcation buss
if they want to overreid that they can then also add driver = log
to alwasy send it to the logs.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/5fdad4322b6d4f1582634d34e221b439

✔️ openstack-meta-content-provider SUCCESS in 2h 31m 55s
❌ nova-operator-kuttl FAILURE in 55m 29s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 15m 25s
❌ nova-operator-tempest-multinode-ceph FAILURE in 39m 22s

ill see if i can find time to re reivew but i think most of my feedback has been adressed
but i still want the compute folks to way in on if they are ok wiht this in general

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/202ae4c0635347cea19e10a20918927f

✔️ openstack-meta-content-provider SUCCESS in 3h 55m 35s
❌ nova-operator-kuttl FAILURE in 51m 03s
❌ nova-operator-tempest-multinode FAILURE in 2h 13m 07s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 58m 31s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d2b200289b52491283b1b69f94c8ab0c

❌ openstack-meta-content-provider FAILURE in 5m 23s
⚠️ nova-operator-kuttl SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode-ceph SKIPPED Skipped due to failed job openstack-meta-content-provider

[vyzigold]

recheck download tools issue

[TristanCacqueray]

recheck

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdo/buildset/bb24970215534cecb26784e4431fdf55

❌ openstack-meta-content-provider NODE_FAILURE Node(set) request 100-0000082651 failed in 0s
⚠️ nova-operator-kuttl SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode-ceph SKIPPED Skipped due to failed job openstack-meta-content-provider

[TristanCacqueray]

recheck

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdo/buildset/107c3c8012ac4f15807b47f493ec8188

✔️ openstack-meta-content-provider SUCCESS in 3h 23m 13s
❌ nova-operator-kuttl RETRY_LIMIT in 1m 02s
❌ nova-operator-tempest-multinode RETRY_LIMIT in 33s
❌ nova-operator-tempest-multinode-ceph RETRY_LIMIT in 33s

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdo/buildset/776b5003add34f8bacfea51788fd8f55

✔️ openstack-meta-content-provider SUCCESS in 2h 55m 44s
❌ nova-operator-kuttl FAILURE in 54m 10s
❌ nova-operator-tempest-multinode FAILURE in 35m 09s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 39m 31s

[vyzigold]

recheck no logs

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/66c8395b73754415afb2ce6dee2a85e2

✔️ openstack-meta-content-provider SUCCESS in 3h 16m 24s
❌ nova-operator-kuttl FAILURE in 54m 56s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 30m 24s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 2h 34m 44s

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/5b5c0596c71241dc97ebcade650bf4be

✔️ openstack-meta-content-provider SUCCESS in 2h 57m 21s
✔️ nova-operator-kuttl SUCCESS in 56m 49s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 40m 08s
❌ nova-operator-tempest-multinode-ceph FAILURE in 47m 33s

[vyzigold]

recheck ceph job failed during make openstack while waiting for metallb operator to start

[vyzigold]

Rebased on top of current main and implemented mrkisaolamb's suggestions (see the 3rd commit).

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/16aad2d9e1db4385b7fda0cc581e827a

✔️ openstack-meta-content-provider SUCCESS in 2h 55m 26s
❌ nova-operator-kuttl FAILURE in 1h 00m 50s
✔️ nova-operator-tempest-multinode SUCCESS in 2h 24m 39s
❌ nova-operator-tempest-multinode-ceph FAILURE in 30m 06s

[vyzigold]

I believe kuttl-tests are just blocked on #1108 . And ceph job failed early on deploying nncp it seems. I'll wait until 1108 merges before trying again.

[vyzigold]

just a rebase on top of current main.

[vyzigold]

Resolved mrkisaolamb's comments and squashed the commits. PR description was updated to reflect the added testing and conditional nova.conf config added later.

[gibizer]

Seem OK to me.

I have one follow up request is to add an envtest to the config generation part. E.g. here

nova-operator/test/functional/nova/novaapi_controller_test.go

Lines 284 to 286 in 444a265

	Expect(configDataMap.Data).Should(HaveKey("api-paste.ini"))
	pasteData := string(configDataMap.Data["api-paste.ini"])
	Expect(pasteData).To(Equal("pipeline = cors compute_req_id faultwrap request_log http_proxy_to_wsgi oscomputeversionapp_v2"))

showing that the new conf file is passed. (You don't need full blown example of the file content, just has some data in it a and show that it is passed)

---

Thanks for configuring this in CI as well. I looked at the latest run on this PR and I saw couple of things:

nova-api logs warning at startup that seems related to this change. Is this normal?

2026-05-29 10:24:35.575 10 WARNING keystonemiddleware._common.config [None req-a6ab8632-ab26-48da-bae3-5be4583f9327 - - - - - -] The option "audit_map_file" is not known to keystonemiddleware
2026-05-29 10:24:35.575 10 WARNING keystonemiddleware._common.config [None req-a6ab8632-ab26-48da-bae3-5be4583f9327 - - - - - -] The option "log_name" is not known to keystonemiddleware
2026-05-29 10:24:35.581 11 WARNING keystonemiddleware._common.config [None req-aef3d29a-47ca-40f1-932b-de4ca3f1c350 - - - - - -] The option "audit_map_file" is not known to keystonemiddleware
2026-05-29 10:24:35.581 11 WARNING keystonemiddleware._common.config [None req-aef3d29a-47ca-40f1-932b-de4ca3f1c350 - - - - - -] The option "log_name" is not known to keystonemiddleware
2026-05-29 10:24:35.665 11 WARNING keystonemiddleware._common.config [None req-aef3d29a-47ca-40f1-932b-de4ca3f1c350 - - - - - -] The option "audit_map_file" is not known to keystonemiddleware
2026-05-29 10:24:35.666 11 WARNING keystonemiddleware._common.config [None req-aef3d29a-47ca-40f1-932b-de4ca3f1c350 - - - - - -] The option "log_name" is not known to keystonemiddleware
2026-05-29 10:24:35.669 10 WARNING keystonemiddleware._common.config [None req-a6ab8632-ab26-48da-bae3-5be4583f9327 - - - - - -] The option "audit_map_file" is not known to keystonemiddleware
2026-05-29 10:24:35.670 10 WARNING keystonemiddleware._common.config [None req-a6ab8632-ab26-48da-bae3-5be4583f9327 - - - - - -] The option "log_name" is not known to keystonemiddleware

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/build/78a7d8040b5f431591d51851e9079244/log/controller/ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-70ce4ab587da14dddf23f3816879343650007e91038f935686ab5355e6f2a7a2/namespaces/openstack/pods/nova-api-0/logs/nova-api-log.log.gz

2026-05-29 10:32:25.030 10 WARNING nova.audit [None req-7283328e-b292-477c-96de-a2c4cf6d2620 9e9b8a206be24cb582e8d63b697db17b 9b6dd75ca43e4a05971c47b827ff68e3 - - default default] Skipping service ceilometer as it have no endpoints.

But I also saw the audit events in the logs which is a good sign :)

Anyhow I can approve this if you can confirm that the warnings above are harmless and if you promise to push a follow up PR with the extra envtest.

[vyzigold]

Regarding the warnings

That seems to be an overlook in the audit middleware. The middleware can be configured either inside the api-paste.ini or in .conf under the [audit_middleware_notifications] section. But some of the options (like log_name and audit_map_file) can be configured only in the api-paste.ini. Per the audit middleware docs, we need to set the audit_map_file https://docs.openstack.org/keystonemiddleware/latest/audit.html#configure-audit-middleware . The log_name is just a nice to have and it just changes the string after the WARNING in the audit logs to "nova.audit".

All the config options from api-paste.ini are apparently strings, so the middleware is trying to iterate through them and it's trying to use their oslo.config counterparts to determine what type they should be and then it's trying to convert them to that type. But because these oslo.config doesn't know about some of these options, the lookup fails, the warning message is printed and that particular option stays being string (which is fortunately the correct type in these 2 cases). See https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/_common/config.py#L49

Regarding the envtest

I'll add the envtest as part of this PR today. I usually don't like proposing a bunch of follow-ups and I think we still have enough time to get this into FR6.

[vyzigold]

I added the envtest as requested (see https://github.com/openstack-k8s-operators/nova-operator/compare/5bc3d42125a261788457424f3951260b1dc2bcb9..73cf1161519476e655c7f2164e262fa381335197 for the diff). and rebased on top of current main.

[gibizer]

Regarding the warnings

That seems to be an overlook in the audit middleware. The middleware can be configured either inside the api-paste.ini or in .conf under the [audit_middleware_notifications] section. But some of the options (like log_name and audit_map_file) can be configured only in the api-paste.ini. Per the audit middleware docs, we need to set the audit_map_file https://docs.openstack.org/keystonemiddleware/latest/audit.html#configure-audit-middleware . The log_name is just a nice to have and it just changes the string after the WARNING in the audit logs to "nova.audit".

All the config options from api-paste.ini are apparently strings, so the middleware is trying to iterate through them and it's trying to use their oslo.config counterparts to determine what type they should be and then it's trying to convert them to that type. But because these oslo.config doesn't know about some of these options, the lookup fails, the warning message is printed and that particular option stays being string (which is fortunately the correct type in these 2 cases). See https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/_common/config.py#L49

So you suggest that these are harmless warnings we can ignore. Fine by me.

[gibizer]

Thanks for the additional test. Looks good to me.

[vyzigold]

Had to fix spelling in the new envtest to pass pre-commit.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gibizer, mrkisaolamb, vyzigold

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [gibizer,mrkisaolamb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d323924c889b47ef83f844acfce22915

✔️ openstack-meta-content-provider SUCCESS in 3h 38m 15s
❌ nova-operator-kuttl RETRY_LIMIT in 43m 20s
✔️ nova-operator-tempest-multinode SUCCESS in 3h 00m 04s
✔️ nova-operator-tempest-multinode-ceph SUCCESS in 3h 21m 27s

[vyzigold]

recheck RETRY_LIMIT seems like nmstate operator didn't get ready in time.