ApplicationSet found, but no diff rendered + SSH Authentication issues

Hey!

Thanks for a wonderful initiative.

I'm having trouble rendering my ApplicationSets. It seems like they are found, but not used in diff generation, or loaded to ArgoCD. See here:

🤖 Fetching all files in dir: base-branch
🤖 Found 10 applications
🤖 Patching applications for branch: main
🤖 Patching 10 Argo CD Application[Sets] for branch: main
🤖 Fetching all files in dir: target-branch
🤖 Found 10 applications
🤖 Patching applications for branch: target-branch
🤖 Patching 10 Argo CD Application[Sets] for branch: target-branch
🚀 Creating cluster...
🚀 Cluster created successfully
🤷 No secrets found in ./secrets
🦑 Installing Argo CD Helm Chart version: 'latest'
🦑 Waiting for Argo CD to start...
🦑 Argo CD is now available
🦑 Logging in to Argo CD through CLI...
🦑 Argo CD installed successfully
🌚 Getting resources from base-branch
🌚 Got all resources from 0 applications for base
🧼 Removing applications
🧼 Removed applications successfully
🌚 Getting resources from target-branch
🌚 Got all resources from 0 applications for target
💥 Deleting cluster...
🔮 Generating diff between main and target-branch
🙏 Please check the ./output/diff.md file for differences
🎉 Done in 101 seconds

Here is an example of an ApplicationSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: metadata-name
  namespace: my-namespace
spec:
  goTemplate: true
  goTemplateOptions: [ "missingkey=error" ]
  generators:
    - git:
        repoURL: git@github.com:my-repo.git
        revision: HEAD
        directories:
          - path: envs/appsets/*
  template:
    metadata:
      name: '{{.path.basename}}-prod'
      namespace: my-name-space
    spec:
      project: my-project
      sources:
        - repoURL: git@github.com:juliankock/my-repo.git
          ref: local-files
        - repoURL: git@github.com:juliankock/my-repo.git
          targetRevision: HEAD
          path: 'apps/{{.path.basename}}'
          helm:
            valueFiles:
              - '$local-files/envs/values-k8s.yaml'
      destination:
        server: https://kubernetes.default.svc
        namespace: my-namespace
      syncPolicy:
        automated:
          prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
          selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).

I did look into the codebase, and I do not immediately see any overwrite of spec.template.spec.destination.namespace. May that be it?

update: tried to remove spec.template.spec.destination.namespace didn't fix it.

@juliankock

Hi @juliankock,

Thank you so much!

Is your repository private?
I noticed that the tool isn't being provided with any secrets for repository access (🤷 No secrets found in ./secrets), which could prevent it from creating applications based on the git generator.

I hope this helps! If not, let me know, and we can investigate further.

I wonder if this only affects deploying a new directory ("Application") via an ApplicationSet? I am seeing similar behavior:

Adding a new directory doesn't cause a diff in the PR Comment:

Testing adding new app seanturner026/argocd-applicationset#41
The github action has 💤 1 Applications timed out. in the logs?

Removing an existing directory actually caused an error:

Testing established app removal seanturner026/argocd-applicationset#42

Modifying an existing directory had the desired behavior (diff rendered):

Testing established app modification seanturner026/argocd-applicationset#43

ApplicationSet is here: https://github.com/seanturner026/argocd-applicationset/blob/8b2ad93e8c3cf920f073f4c79520d39251559d86/clusters/staging/infra-appset.yaml

And is deployed by this Application: https://github.com/seanturner026/argocd-applicationset/blob/8b2ad93e8c3cf920f073f4c79520d39251559d86/clusters/staging/argocd-app.yaml

Seems to be the case on 0.0.18 and 0.0.24? Please let me know if I should make separate Issues here. Also having a look on my end.

@seanturner026

Thank you for the examples and investigation, @seanturner026.
I now realize that the git-generators do not work properly! The tool is not updating the git-generator-revision-field to match the Pull request branch. I will have to fix that 🚀 Sorry for the inconvenience.

No inconvenience at all!

@juliankock

Hi @juliankock,

This issue should now be fixed in version 0.0.26! You can check it out here: [release] 🚀
Here’s a working example: #80.

Please let me know if you’re still encountering this issue 🙏🏻 It’s possible that the behavior you’re seeing is caused by something else

@juliankock

Hi @juliankock, Just wanted to follow up on this - have you had a chance to check out the new version? :) The issue you experienced might also be related to: issues 85

@dag-andersen

@dag-andersen

It seems like it now finds the ApplicationSets. However, since I use SSH AUTH reference to the repository, I now get the following failing GitHub action:

Error: CommandError { stderr: "time=\"2025-01-08T06:52:51Z\" level=fatal msg=\"rpc error: code = Internal desc = unable to resolve git revision : error creating SSH agent: \\\"SSH agent requested but SSH_AUTH_SOCK not-specified\\\"\"\n" }

I do mount the secrets as specified here: https://dag-andersen.github.io/argocd-diff-preview/github-actions-workflow/

So maybe I need to add an SSH key to the container?

This is only a local constraint for our running ArgoCD instance. So if possible, the library here could just change to an HTTP repoURL.

@juliankock

@juliankock

How does your current real Argo CD instance authenticate with GitHub? Can't you use the same credentials/authentication method from the argocd-diff-preview cluster? :)

@juliankock

@juliankock

How does your current real Argo CD instance authenticate with GitHub? Can't you use the same credentials/authentication method from the argocd-diff-preview cluster? :)

I'm having the same issue as Julian. We currently authenticate to our repos by manually adding a key via the UI in argocd. I suppose the correct fix would be to gitops those keys, but is there a way to avoid needing to do that?

@myanello-rescale

Hi @myanello-rescale,

I recommend avoiding checking your keys into Git.

Instead, you can create an ephemeral secret within your pipeline, as explained here: [link].

Since you're using SSH, your secret should look something like this ⬇️. You can verify the correct format by checking your real ArgoCD cluster with the following command:
kubectl get secret -n argocd -o yaml

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  type: git
  url: git@github.com:tiwarisanjay
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    blahblahblah
    -----END OPENSSH PRIVATE KEY-----

Let me know if that helps :)

@juliankock

Hi @juliankock and @myanello-rescale,

Did this solve your problem? If so, I’ll go ahead and close the issue. :)

Hi @juliankock and @myanello-rescale,

Did this solve your problem? If so, I’ll go ahead and close the issue. :)

I haven't had a chance to really dig into this again yet, so feel free to close as my free time comes and goes. Thank you for the tip forward though

@dag-andersen

@dag-andersen
I finally gave it a shot, I'm now hitting:

Failed to generate applications from ApplicationSet ... broken pipe\" logger=\"UnhandledError\"\ntime=\"2025-03-17T14:05:57Z\" level=fatal msg=\"rpc error: code = Internal desc = unable to resolve git revision : failed to list refs: error creating SSH agent: \\\"SSH agent requested but SSH_AUTH_SOCK not-specified\\\"\"\n: exit status 20"

I'm using a private GitHub App SSH key which is installed in the repo. Any ideas?

@juliankock

@juliankock

Would you mind sharing your pipeline with me?

Someone else had SSH issues last week [link], and they updated the [docs] with their solution :)

Let me know if that helps - otherwise, we’ll have to dig a bit deeper.

@dag-andersen

@dag-andersen

I updated it to follow the links, thanks!

The workflow looks like this:

name: Argo CD Diff Preview

on:
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4
        with:
          path: pull-request

      - uses: actions/checkout@v4
        with:
          ref: main
          path: main
          
      - name: Prepare secrets
        run: |
          mkdir secrets
          cat > secrets/secret.yaml << EOF
          apiVersion: v1
          kind: Secret
          metadata:
            name: private-repo
            namespace: argocd
            labels:
              argocd.argoproj.io/secret-type: repository
          stringData:
            type: git
            url: git@github.com:<repo-omitted>.git
            sshPrivateKey: |
          $(echo "${{ secrets.GH_APP_PRIVATE_KEY }}" | sed 's/^/    /')
          EOF

      - name: Generate Diff
        run: |
          docker run \
            --network=host \
            -v /var/run/docker.sock:/var/run/docker.sock \
            -v $(pwd)/main:/base-branch \
            -v $(pwd)/pull-request:/target-branch \
            -v $(pwd)/output:/output \
            -v $(pwd)/secrets:/secrets \
            -e TARGET_BRANCH=${{ github.head_ref }} \
            -e REPO=${{ github.repository }} \
            dagandersen/argocd-diff-preview:v0.1.0

      - name: Post diff as comment
        run: |
          gh pr comment ${{ github.event.number }} --repo ${{ github.repository }} --body-file output/diff.md --edit-last || \
          gh pr comment ${{ github.event.number }} --repo ${{ github.repository }} --body-file output/diff.md
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Now I get this error:

broken pipe\" logger=\"UnhandledError\"\ntime=\"2025-03-17T15:23:47Z\" level=fatal msg=\"rpc error: code = Internal desc = unable to resolve git revision : failed to list refs: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain\"\n: exit status 20"

It seems like my SSH key is not scoped correctly. I'm using GitHub Apps to do this, which is installed to the GitHub repository. It should have READ contents set.

EDIT:
I think I might be hitting this: argoproj/argo-cd#9122

I will try to generate a deploy key instead, since GitHub apps make a .pem key.

@dag-andersen

Using a deploy key worked! I would recommend adding this reference to the docs: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys if you want to use SSH access.

Thanks for the help! :)

This issue can be closed, but I will let you close it @dag-andersen so you can see this.

Navigation Menu

Repository navigation

ApplicationSet found, but no diff rendered + SSH Authentication issues #64

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ApplicationSet found, but no diff rendered + SSH Authentication issues #64

Description

Activity

dag-andersen commented on Nov 14, 2024

seanturner026 commented on Nov 20, 2024

dag-andersen commented on Nov 21, 2024

seanturner026 commented on Nov 21, 2024

dag-andersen commented on Dec 22, 2024

dag-andersen commented on Jan 7, 2025

juliankock commented on Jan 8, 2025

dag-andersen commented on Jan 12, 2025

myanello-rescale commented on Jan 14, 2025

dag-andersen commented on Jan 14, 2025

dag-andersen commented on Jan 26, 2025

myanello-rescale commented on Jan 26, 2025

juliankock commented on Mar 17, 2025

dag-andersen commented on Mar 17, 2025

juliankock commented on Mar 17, 2025

juliankock commented on Mar 17, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions