-
Notifications
You must be signed in to change notification settings - Fork 117
Refactor Template Step to use App and AppSource Updates #1829
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
57 commits
Select commit
Hold shift + click to select a range
0269ee0
its looking pretty good
rain-on 765eb80
sort of?
rain-on 86fb93e
DirectoryUpdater looks about right
rain-on c70942b
added empty updaters
rain-on 35e6d26
half way with ref updating
rain-on b07dfd9
Kustomize is in - now for ref
rain-on 1f67816
created updaterHelpers
rain-on 51df92d
ref is compiling
rain-on 8e74fa2
created an intermediate helmupdater its nasty
rain-on b0ef6a3
compiling - wonder what happens with the main convention
rain-on fbe1a5d
moved everything
rain-on d07ff17
added a repositoryAdapter
rain-on fe5dc4a
split out the repo and outputvariables stuff
rain-on bdaaff8
sort out compilation
rain-on ca8752f
fix the null whoopsie
rain-on 690e254
yay
rain-on e10f532
clean up
rain-on 4ed8def
catch the lost convention
rain-on 023f9a8
slow tidy up after review
rain-on 2aa02c3
tidy it up even more
rain-on 535baed
and more
rain-on 54558bd
cleaning
rain-on 233d297
removed updatdfiles and rely on patches
rain-on 766ab76
make file list distinct
rain-on f449e19
added authenticatingrepositoryfactor
rain-on 20cd4c8
moved things around a bit
rain-on ae52c8e
clean up imports
rain-on 96bdae1
move files into a subdirectory under Conventions
rain-on e75926c
fix namespacing
rain-on a55c7f9
its mostly good
rain-on 79a7a6d
Refactored the templating step - yay?
rain-on f2b4d65
Merge remote-tracking branch 'origin/main' into tmm/refactor_other_step
rain-on 638be1f
fix up the whoopsie
rain-on 7e44ec2
yay
rain-on 52065d9
trying to make this nicer
rain-on aa2e5c2
passing
rain-on 4ffe8c5
made the new steop way better
rain-on 3c10736
And the Templating is way nicer
rain-on dbf87e8
add comment
rain-on 5046d60
looks ok
rain-on 1966820
minor tidy up
rain-on bd7df91
fix compilation issues
rain-on f55822d
sorted
rain-on 7a817e3
back it out
rain-on 8cfe54f
remove the slash which wasn't going to work
rain-on d082bfe
I hate windows
rain-on 8103b86
fix compilation
rain-on d5a0e37
tidy up
rain-on 799533e
fixed the pathing issues, wild.
rain-on fb68452
create a test for pathing
rain-on 53b5c3d
make everything posix-ee
rain-on 15f4ecf
normalize to posix deliberately
rain-on 57322e9
force removed files
rain-on c05bc45
fix up the relative path error
rain-on c6b9a72
revert the normalizePath in repositorywrapper
rain-on 92a193b
always force repositoryWrapper to use forward slash
rain-on 526c6e2
fix normalize yet again
rain-on File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
72 changes: 72 additions & 0 deletions
72
source/Calamari/ArgoCD/Conventions/ManifestTemplating/ApplicationSourceUpdater.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| using Calamari.ArgoCD.Domain; | ||
| using Calamari.ArgoCD.Dtos; | ||
| using Calamari.ArgoCD.Git; | ||
| using Calamari.ArgoCD.Models; | ||
| using Calamari.Common.Plumbing.FileSystem; | ||
| using Calamari.Common.Plumbing.Logging; | ||
|
|
||
| namespace Calamari.ArgoCD.Conventions.ManifestTemplating; | ||
|
|
||
| public class ApplicationSourceUpdater | ||
| { | ||
| readonly Application applicationFromYaml; | ||
| readonly DeploymentScope deploymentScope; | ||
| readonly RepositoryAdapter repositoryAdapter; | ||
| readonly ArgoCommitToGitConfig deploymentConfig; | ||
| readonly IPackageRelativeFile[] packageFiles; | ||
| readonly ArgoCDGatewayDto gateway; | ||
| readonly ILog log; | ||
| readonly ICalamariFileSystem fileSystem; | ||
| readonly ArgoCDOutputVariablesWriter outputVariablesWriter; | ||
|
|
||
| public ApplicationSourceUpdater(Application applicationFromYaml, | ||
| ArgoCDGatewayDto gateway, | ||
| DeploymentScope deploymentScope, | ||
| ArgoCommitToGitConfig deploymentConfig, | ||
| IPackageRelativeFile[] packageFiles, | ||
| ILog log, | ||
| ICalamariFileSystem fileSystem, | ||
| ArgoCDOutputVariablesWriter outputVariablesWriter, | ||
| RepositoryAdapter repositoryAdapter) | ||
| { | ||
| this.applicationFromYaml = applicationFromYaml; | ||
| this.deploymentScope = deploymentScope; | ||
| this.deploymentConfig = deploymentConfig; | ||
| this.packageFiles = packageFiles; | ||
| this.gateway = gateway; | ||
| this.log = log; | ||
| this.fileSystem = fileSystem; | ||
| this.outputVariablesWriter = outputVariablesWriter; | ||
| this.repositoryAdapter = repositoryAdapter; | ||
| } | ||
|
|
||
| public ManifestUpdateResult ProcessSource(ApplicationSourceWithMetadata sourceWithMetadata) | ||
| { | ||
| var applicationSource = sourceWithMetadata.Source; | ||
| var annotatedScope = ScopingAnnotationReader.GetScopeForApplicationSource(applicationSource.Name.ToApplicationSourceName(), applicationFromYaml.Metadata.Annotations, applicationFromYaml.Spec.Sources.Count > 1); | ||
|
|
||
| log.LogApplicationSourceScopeStatus(annotatedScope, applicationSource.Name.ToApplicationSourceName(), deploymentScope); | ||
|
|
||
| if (!deploymentScope.Matches(annotatedScope)) | ||
| return new ManifestUpdateResult(false, string.Empty, []); | ||
|
|
||
| log.Info($"Writing files to repository '{applicationSource.OriginalRepoUrl}' for '{applicationFromYaml.Metadata.Name}'"); | ||
|
|
||
| var sourceUpdater = new CopyTemplatesSourceUpdater(packageFiles, log, fileSystem, deploymentConfig.PurgeOutputDirectory); | ||
|
|
||
| var sourceUpdateResult = repositoryAdapter.Process(sourceWithMetadata, sourceUpdater); | ||
|
|
||
| if (sourceUpdateResult.PushResult is not null) | ||
| { | ||
| outputVariablesWriter.WritePushResultOutput(gateway.Name, | ||
| applicationFromYaml.Metadata.Name, | ||
| sourceWithMetadata.Index, | ||
| sourceUpdateResult.PushResult); | ||
|
|
||
| return new ManifestUpdateResult(true, sourceUpdateResult.PushResult.CommitSha, sourceUpdateResult.PatchedFiles); | ||
| } | ||
|
|
||
| log.Info("No changes were commited"); | ||
| return new ManifestUpdateResult(false, string.Empty, []); | ||
| } | ||
| } |
123 changes: 123 additions & 0 deletions
123
source/Calamari/ArgoCD/Conventions/ManifestTemplating/ApplicationUpdater.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| using System.Collections.Generic; | ||
| using System.Linq; | ||
| using Calamari.ArgoCD.Domain; | ||
| using Calamari.ArgoCD.Dtos; | ||
| using Calamari.ArgoCD.Git; | ||
| using Calamari.ArgoCD.Models; | ||
| using Calamari.Common.Plumbing.FileSystem; | ||
| using Calamari.Common.Plumbing.Logging; | ||
|
|
||
| namespace Calamari.ArgoCD.Conventions.ManifestTemplating; | ||
|
|
||
| public class ApplicationUpdater | ||
| { | ||
| readonly AuthenticatingRepositoryFactory repositoryFactory; | ||
| readonly DeploymentScope deploymentScope; | ||
| readonly ArgoCommitToGitConfig deploymentConfig; | ||
| readonly ILog log; | ||
| readonly ICalamariFileSystem fileSystem; | ||
| readonly IArgoCDApplicationManifestParser argoCdApplicationManifestParser; | ||
| readonly ArgoCDOutputVariablesWriter outputVariablesWriter; | ||
| readonly IPackageRelativeFile[] packageFiles; | ||
|
|
||
|
|
||
| public ApplicationUpdater(AuthenticatingRepositoryFactory repositoryFactory, DeploymentScope deploymentScope, ArgoCommitToGitConfig deploymentConfig, ILog log, | ||
| ICalamariFileSystem fileSystem, | ||
| IArgoCDApplicationManifestParser argoCdApplicationManifestParser, | ||
| ArgoCDOutputVariablesWriter outputVariablesWriter, | ||
| IPackageRelativeFile[] packageFiles) | ||
| { | ||
| this.repositoryFactory = repositoryFactory; | ||
| this.deploymentScope = deploymentScope; | ||
| this.deploymentConfig = deploymentConfig; | ||
| this.log = log; | ||
| this.fileSystem = fileSystem; | ||
| this.argoCdApplicationManifestParser = argoCdApplicationManifestParser; | ||
| this.outputVariablesWriter = outputVariablesWriter; | ||
| this.packageFiles = packageFiles; | ||
| } | ||
|
|
||
| public ProcessApplicationResult ProcessApplication( | ||
| ArgoCDApplicationDto application, | ||
| ArgoCDGatewayDto gateway) | ||
| { | ||
| log.InfoFormat("Processing application {0}", application.Name); | ||
| var applicationFromYaml = argoCdApplicationManifestParser.ParseManifest(application.Manifest); | ||
| var containsMultipleSources = applicationFromYaml.Spec.Sources.Count > 1; | ||
| var applicationName = applicationFromYaml.Metadata.Name; | ||
|
|
||
| LogWarningIfUpdatingMultipleSources(applicationFromYaml.Spec.Sources, | ||
| applicationFromYaml.Metadata.Annotations, | ||
| containsMultipleSources); | ||
|
|
||
| ValidateApplication(applicationFromYaml); | ||
|
|
||
| var repositoryAdapter = new RepositoryAdapter(repositoryFactory, deploymentConfig.CommitParameters, log, new CommitMessageGenerator()); | ||
| var sourceUpdater = new ApplicationSourceUpdater(applicationFromYaml, | ||
| gateway, | ||
| deploymentScope, | ||
| deploymentConfig, | ||
| packageFiles, | ||
| log, | ||
| fileSystem, | ||
| outputVariablesWriter, | ||
| repositoryAdapter); | ||
|
|
||
| var updatedSourcesResults = applicationFromYaml | ||
| .GetSourcesWithMetadata() | ||
| .Select(applicationSource => new | ||
| { | ||
| UpdateResult = sourceUpdater.ProcessSource(applicationSource), | ||
| applicationSource | ||
| }) | ||
| .Where(u => u.UpdateResult.Updated) | ||
| .ToList(); | ||
|
|
||
| //if we have links, use that to generate a link, otherwise just put the name there | ||
| var instanceLinks = application.InstanceWebUiUrl != null ? new ArgoCDInstanceLinks(application.InstanceWebUiUrl) : null; | ||
| var linkifiedAppName = instanceLinks != null | ||
| ? log.FormatLink(instanceLinks.ApplicationDetails(applicationName, applicationFromYaml.Metadata.Namespace), applicationName) | ||
| : applicationName; | ||
|
|
||
| var message = updatedSourcesResults.Any() | ||
| ? "Updated Application {0}" | ||
| : "Nothing to update for Application {0}"; | ||
|
|
||
| log.InfoFormat(message, linkifiedAppName); | ||
|
|
||
| return new ProcessApplicationResult( | ||
| application.GatewayId, | ||
| applicationName.ToApplicationName(), | ||
| applicationFromYaml.Spec.Sources.Count, | ||
| applicationFromYaml.Spec.Sources.Count(s => deploymentScope.Matches(ScopingAnnotationReader.GetScopeForApplicationSource(s.Name.ToApplicationSourceName(), applicationFromYaml.Metadata.Annotations, containsMultipleSources))), | ||
| updatedSourcesResults.Select(r => new UpdatedSourceDetail(r.UpdateResult.CommitSha, r.applicationSource.Index, r.UpdateResult.ReplacedFiles, [])).ToList(), | ||
| [], | ||
| updatedSourcesResults.Select(r => r.applicationSource.Source.OriginalRepoUrl).ToHashSet()); | ||
| } | ||
|
|
||
| void LogWarningIfUpdatingMultipleSources( | ||
| List<ApplicationSource> sourcesToInspect, | ||
| Dictionary<string, string> applicationAnnotations, | ||
| bool containsMultipleSources) | ||
| { | ||
| if (sourcesToInspect.Count > 1) | ||
| { | ||
| var sourcesWithScopes = sourcesToInspect.Select(s => (s, ScopingAnnotationReader.GetScopeForApplicationSource(s.Name.ToApplicationSourceName(), applicationAnnotations, containsMultipleSources))).ToList(); | ||
| var sourcesWithMatchingScopes = sourcesWithScopes.Where(s => deploymentScope.Matches(s.Item2)).ToList(); | ||
|
|
||
| if (sourcesWithMatchingScopes.Count > 1) | ||
| { | ||
| log.Warn($"Multiple sources are associated with this deployment, they will all be updated with the same contents: {string.Join(", ", sourcesWithMatchingScopes.Select(s => s.s.Name))}"); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| void ValidateApplication(Application applicationFromYaml) | ||
| { | ||
| var validationResult = ValidationResult.Merge( | ||
| ApplicationValidator.ValidateSourceNames(applicationFromYaml), | ||
| ApplicationValidator.ValidateUnnamedAnnotationsInMultiSourceApplication(applicationFromYaml) | ||
| ); | ||
| validationResult.Action(log); | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you know if this filename also needs to be stripped of the leading "./"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think, the way we're using it in the tests - the answer is "no" - but that's luck.
The reason we have a problem in production code is when the argo app's path is "./" :(