fix(config_etcd): keep the previous value when a full reload gets invalid data#13717
Open
AlinsRan wants to merge 5 commits into
Open
fix(config_etcd): keep the previous value when a full reload gets invalid data#13717AlinsRan wants to merge 5 commits into
AlinsRan wants to merge 5 commits into
Conversation
…alid data The full reload path (load_full_data) and the incremental watch path treat an item that fails validation differently: - the watch path only cancels the update of that item and keeps the value already in memory (`goto CONTINUE`, see apache#11268); - the full reload path fires the clean handlers of every old item before reading from etcd and then skips the invalid item, so the item silently disappears from the data plane. A full reload is triggered whenever the watch is cancelled by etcd with `compacted`, which happens on ordinary etcd auto compaction when the global revision moved on outside the watched prefix. An item that has been rejected by the validation for a long time without any impact therefore vanishes at a random point in time. Since global plugins usually all live on the same global rule, one invalid plugin conf takes down every global plugin. Make the full reload behave like the watch path: pass the previous values to load_full_data and, when the new data of a key fails the validation and a previous value for the same key exists, carry the old item over as is - no clean handlers, no filter re-run, no version bump. Keys that are absent from the readdir result are still dropped, and an invalid item without a previous value (first load) is still skipped. The clean handlers of the old items that were replaced or deleted are now fired after the new table has been built. Fixes apache#12834
A plugin conf carrying `_meta.disable: true` is never executed, but its check_schema is still run unconditionally, and a failure invalidates the whole item (route, service, global rule, ...). Some plugins validate the local environment inside check_schema - proxy-cache requires cache_zone to be declared in the config.yaml of this very node - so a conf that is legal elsewhere permanently invalidates the item on this node, even though the plugin is turned off. Keep running check_schema (its side effects, e.g. default injection, are still wanted) but only reject the conf when the plugin is enabled; a disabled plugin logs a warning and is accepted. Re-enabling it goes through a full write and is validated again, so an invalid conf cannot silently become active. Same treatment for the stream subsystem.
The blocks drive a real etcd sync round and sleep past the 5s default block timeout, so they died on a client socket timeout rather than on an assertion.
…y disabled check_disable returns _meta.disable verbatim, so a truthy non-boolean value such as the string "false" would have taken the downgrade path and silently accepted a conf that the schema should have rejected.
The downgrade for _meta.disable plugins piped the check_schema error straight into a warn. That string can echo config values (e.g. a pattern mismatch), so a disabled plugin's bad config could leak into the shared error log. Keep the warn as the signal that the config did not validate, but drop the error detail — the caller can still see it by re-submitting with the plugin enabled. Also assert the admin PUT actually returns passed in the disk.t case, not just the log line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #12834
The full reload path and the incremental watch path in
core/config_etcd.luadisagree on what to do with an item whose data fails validation:sync_data):goto CONTINUE— only the update of that item is cancelled, the value already in memory keeps serving. This is deliberate, see fix: validation fails causing etcd events not to be handled correctly #11268.load_full_data): before calling it,sync_datafired the clean handlers of every old item and dropped the whole table; the invalid item is then simply not inserted. The item disappears from the data plane.So the root cause is not "validation is too strict" — an item can sit there rejected by validation for days with zero impact, because the watch path protects it. What breaks is that a full reload is triggered asynchronously by something completely unrelated: etcd cancels the watch with
compactedwhenever an ordinary auto compaction happened after the global revision moved on outside the watched prefix. At that random moment the item is discarded. Since global plugins normally all live on the same global rule, one invalid plugin conf takes down every global plugin, and the operator sees no cause-effect relation with anything they did.The reporter's case: a
proxy-cacheconf written through the dashboard with the defaultcache_zone: disk_cache_one, on a node whoseconfig.yamldeclares different zones.proxy-cache'scheck_schemavalidatescache_zoneagainst the local node config, so the item is permanently invalid on that node.Commit 1 makes the full reload behave like the watch path.
sync_datano longer fires the old clean handlers up front; it hands the previous values toload_full_data, which:filterre-run, nomodifiedIndexchange, noconf_versionbump. "As is" is exactly whatgoto CONTINUEmeans on the watch path, and re-runningfilteris not safe in general (e.g. the/pluginsfilter triggersplugin.load);readdirresult, so a deleted item is not resurrected;Behaviour change: a full reload hitting invalid data now logs a WARN with the key and the error and keeps serving the last valid config, instead of silently dropping it. That is the semantics the watch path has had since 3.10.0, so no new semantics are introduced — the fix only removes "whether a compaction happens to occur" as a factor.
Commit 2 removes the amplifier.
_meta.disable: truecurrently does not exempt a plugin fromcheck_schema(skip_disabled_pluginonly covers plugins not enabled inconfig.yaml, which is a different thing). A disabled plugin is never executed —_M.filterskips it viacheck_disable— so an environment dependent validation failure has no reason to invalidate the item that carries it.check_schemastill runs (its side effects, such as default injection, are wanted), but the failure is only fatal when the plugin is enabled; otherwise it is logged as a warning and accepted. Re-enabling the conf goes through a normal write and is fully validated, so an invalid conf cannot silently become active.stream_check_schemagets the same treatment.Note this also relaxes the Admin API, which shares
check_single_plugin_schema: a disabled plugin with a locally invalid conf is now accepted there too. That is intentional — with heterogeneousconfig.yamlacross data planes it is the expected behaviour, and the dashboard/direct etcd writes bypassed this check anyway. The two commits are independent: commit 1 alone already removes the outage.Tests
t/core/config_etcd.t, three new cases, all driving the real sync loop against etcd and asserting on the config object of the worker that serves the request. The full reload is entered by settingneed_reload = true— the exact statesync_datasets when etcd answerscompacted— followed by a write that wakes the loop blocked on the watch semaphore.proxy-cachewhosecache_zonedoes not exist on this node. Assert the rule survives the watch path (existing behaviour), then force a full reload and assert it is still there with itsresponse-rewriteplugin intact, plus thekeep the previous configurationWARN. Before the fix the rule is gone at that point and the WARN never appears.keep the previous configurationis logged — the no-previous-value case must still skip.t/plugin/proxy-cache/disk.t: AdminPUTof a disabledproxy-cachewith a non-existent zone now returns 200 and logs the warning; the same conf withdisable: falsestill returns 400.The tests have not been executed locally; relying on CI.
Checklist