Skip to content

OCPBUGS-77839: Fix node degrades due to file and OS update failures #5744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
Mar 7, 2026
Merged

OCPBUGS-77839: Fix node degrades due to file and OS update failures #5744

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions pkg/daemon/update.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1184,7 +1184,7 @@ func (dn *Daemon) update(oldConfig, newConfig *mcfgv1.MachineConfig, skipCertifi
if err := dn.updateFiles(oldIgnConfig, newIgnConfig, addedOrChangedUnits, skipCertificateWrite, forceFilePresent); err != nil {
// When ImageModeStatusReporting is enabled, update the `MachineConfigNodeUpdateFiles` condition to report the experienced error
if imageModeStatusReportingEnabled {
err = upgrademonitor.GenerateAndApplyMachineConfigNodes(
mcnErr := upgrademonitor.GenerateAndApplyMachineConfigNodes(
Copy link
Copy Markdown
Contributor

@pablintino pablintino Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces some changes I'm not sure we want. So, do we really want to ignore errors of GenerateAndApplyMachineConfigNodes? If so, ok, this is fine, but, if we want to fail the call to updateFiles and properly report to the caller without loosing the original error I'd say it's better to:

if mcnErr != nil {
  err = errors.Join(err, fmt.Errorf("eError making MCN for failed file apply: %v", err))
}

Copy link
Copy Markdown
Member Author

@isabella-janssen isabella-janssen Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm misunderstanding your concern here, the change here should better match the existing MCN failure case. Here's an existing similar situation:

machine-config-operator/pkg/daemon/update.go

Lines 994 to 1013 in 9050f6f

if reconcilableError != nil {
Nerr := upgrademonitor.GenerateAndApplyMachineConfigNodes(
&upgrademonitor.Condition{State: mcfgv1.MachineConfigNodeUpdatePrepared, Reason: string(mcfgv1.MachineConfigNodeUpdatePrepared), Message: fmt.Sprintf("Update Failed compatibility validation. MachineConfigs %v and %v are not compatible. Err: %s", oldConfigName, newConfigName, reconcilableError.Error())},
nil,
metav1.ConditionUnknown,
metav1.ConditionFalse,
dn.node,
dn.mcfgClient,
dn.fgHandler,
pool,
)
if Nerr != nil {
klog.Errorf("Error making MCN for Preparing update failed: %v", err)
}
wrappedErr := fmt.Errorf("can't reconcile config %s with %s: %w", oldConfigName, newConfigName, reconcilableError)
if dn.nodeWriter != nil {
dn.nodeWriter.Eventf(corev1.EventTypeWarning, "FailedToReconcile", wrappedErr.Error())
}
return &unreconcilableErr{wrappedErr}
}

The example is logging the MCN apply failure (not treating it as a fatal error) then returning only the reconcilableError that was triggering the need for the MCN update. This new change should match that pattern.

Copy link
Copy Markdown
Contributor

@pablintino pablintino Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, ok, so we area already just printing those errors, ok, I'm fine in that case.

&upgrademonitor.Condition{State: mcfgv1.MachineConfigNodeUpdateExecuted, Reason: string(mcfgv1.MachineConfigNodeUpdateFiles), Message: fmt.Sprintf("Error updating the Files on disk as a part of the in progress phase")},
&upgrademonitor.Condition{State: mcfgv1.MachineConfigNodeUpdateFiles, Reason: fmt.Sprintf("%s%s", string(mcfgv1.MachineConfigNodeUpdateExecuted), string(mcfgv1.MachineConfigNodeUpdateFiles)), Message: fmt.Sprintf("Update failed applying files to node: %s", err.Error())},
metav1.ConditionUnknown,
Expand All @@ -1194,6 +1194,9 @@ func (dn *Daemon) update(oldConfig, newConfig *mcfgv1.MachineConfig, skipCertifi
dn.fgHandler,
pool,
)
if mcnErr != nil {
klog.Errorf("Error making MCN for failed file apply: %v", err)
}
}
return err
}
Expand All @@ -1219,7 +1222,7 @@ func (dn *Daemon) update(oldConfig, newConfig *mcfgv1.MachineConfig, skipCertifi
if err := dn.updateSSHKeys(newIgnConfig.Passwd.Users, oldIgnConfig.Passwd.Users); err != nil {
// When ImageModeStatusReporting is enabled, update the `MachineConfigNodeUpdateFiles` condition to report the experienced error
if imageModeStatusReportingEnabled {
err = upgrademonitor.GenerateAndApplyMachineConfigNodes(
mcnErr := upgrademonitor.GenerateAndApplyMachineConfigNodes(
&upgrademonitor.Condition{State: mcfgv1.MachineConfigNodeUpdateExecuted, Reason: string(mcfgv1.MachineConfigNodeUpdateFiles), Message: fmt.Sprintf("Error updating the Files on disk as a part of the in progress phase")},
&upgrademonitor.Condition{State: mcfgv1.MachineConfigNodeUpdateFiles, Reason: fmt.Sprintf("%s%s", string(mcfgv1.MachineConfigNodeUpdateExecuted), string(mcfgv1.MachineConfigNodeUpdateFiles)), Message: fmt.Sprintf("Update failed applying files to node: %s", err.Error())},
metav1.ConditionUnknown,
Expand All @@ -1229,6 +1232,9 @@ func (dn *Daemon) update(oldConfig, newConfig *mcfgv1.MachineConfig, skipCertifi
dn.fgHandler,
pool,
)
if mcnErr != nil {
klog.Errorf("Error making MCN for failed SSH update: %v", err)
}
}
return err
}
Expand Down Expand Up @@ -1269,7 +1275,7 @@ func (dn *Daemon) update(oldConfig, newConfig *mcfgv1.MachineConfig, skipCertifi
if err := coreOSDaemon.applyOSChanges(*diff, oldConfig, newConfig); err != nil {
// When ImageModeStatusReporting is enabled, update the `MachineConfigNodeUpdateOS` condition to report the experienced error
if imageModeStatusReportingEnabled {
err = upgrademonitor.GenerateAndApplyMachineConfigNodes(
mcnErr := upgrademonitor.GenerateAndApplyMachineConfigNodes(
&upgrademonitor.Condition{State: mcfgv1.MachineConfigNodeUpdateExecuted, Reason: string(mcfgv1.MachineConfigNodeUpdateOS), Message: fmt.Sprintf("Error the OS on disk as a part of the in progress phase")},
&upgrademonitor.Condition{State: mcfgv1.MachineConfigNodeUpdateOS, Reason: fmt.Sprintf("%s%s", string(mcfgv1.MachineConfigNodeUpdateExecuted), string(mcfgv1.MachineConfigNodeUpdateOS)), Message: fmt.Sprintf("Update failed applying new OS config to node: %s", err.Error())},
metav1.ConditionUnknown,
Expand All @@ -1279,6 +1285,9 @@ func (dn *Daemon) update(oldConfig, newConfig *mcfgv1.MachineConfig, skipCertifi
dn.fgHandler,
pool,
)
if mcnErr != nil {
klog.Errorf("Error making MCN for failed OS update: %v", err)
}
}
return err
}
Expand Down