Skip to content

Argo CD sometimes applies very old Git commit during sync retry despite targetRevision=HEAD #26530

@skwair

Description

@skwair

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

When syncing an Application that tracks targetRevision: HEAD, Argo CD sometimes applies manifests from an older Git commit (not the current HEAD).

This results in unexpected rollback-like behavior. Old infrastructure changes (e.g. removed initContainers, or any manifest updates) are re-applied/rolled-back, which can cause downtime or data corruption.

To Reproduce

Unfortunately, I can't find a way to reliably reproduce the issue, but I've seen it happen a couple of times in the last few weeks.

Today it happened with Argo CD v3.3.1 deployed in HA mode, but I've also seen this issue with v3.2.5.

What happened this time:

  • An Application generated by an ApplicationSet (see below) detected a new commit from the remote Git repository
  • It transitioned to OutOfSync, as expected
  • I have a sync hook that runs database migrations for my application
  • Due to a bug in the migration script (not related to Argo CD at all), It could not complete so it retried until reaching the max backoff, as expected
  • Eventually a very old commit was applied, editing live resources of the Syncing Application (for example in this case, it re-introduced initContainers that were removed months ago among other changes)
  • The app is now stuck (which is expected in my case) in OutOfSync, but live resources are reverted to very old commits

I'm not sure where to look for info on what's going on, but I see this in the repo server logs:

{"level":"info","msg":"manifest cache miss: \u0026ApplicationSource{RepoURL:https://github.com/xxxxx/infra.git,Path:kube/apps/xxxxx/overlays/dev/demo/demo,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,Name:,}/cb6ebcceab445f9d32ffca733a2816dc4617c12d","time":"2026-02-19T16:54:34Z"}

cb6ebcceab445f9d32ffca733a2816dc4617c12d is an old commit from Dec 17, 2025:

Image

The HEAD commit for this repo is currently 78924535ebc1ee48dbd86dfd2f3aa9c0c5a82b4b:

Image

The Application correctly says it must sync to this commit:

Image

The Application diff tool correctly shows the diff between cb6ebcce and 78924535.

Here's the ApplicationSet used to deploy the Application:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: 'xxxxx'
  namespace: 'argocd'
spec:
  goTemplate: true
  goTemplateOptions: ['missingkey=error']
  generators:
    - git:
        repoURL: 'https://github.com/xxxxx/infra.git'
        revision: 'HEAD'
        directories:
          - path: 'kube/apps/xxxxx/overlays/dev/*/*'
  template:
    metadata:
      name: 'xxxxx-{{ index .path.segments 5 }}-{{ index .path.segments 6 }}'
      namespace: 'argocd'
      labels:
        cluster: 'dev'
        type: '{{ index .path.segments 5 }}'
    spec:
      project: 'apps'
      source:
        repoURL: 'https://github.com/xxxxx/infra.git'
        targetRevision: 'HEAD'
        path: '{{ .path.path }}'
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: 'xxxxx-{{ index .path.segments 5 }}-{{ index .path.segments 6 }}'
      syncPolicy:
        syncOptions:
          - 'CreateNamespace=true'
          - 'ServerSideApply=true'
        managedNamespaceMetadata:
          labels:
            cluster-secret-store.external-secrets/apps: 'allow'
  strategy:
    type: 'RollingSync'
    rollingSync:
      steps:
        - matchExpressions:
            - key: 'cluster'
              operator: 'In'
              values:
                - 'dev'
          maxUpdate: 1
  ignoreApplicationDifferences:
    - jsonPointers:
        # Allow disabling automated sync for Applications generated by this ApplicationSet if needed.
        # See https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Controlling-Resource-Modification/#allow-temporarily-toggling-auto-sync
        # for more information.
        - '/spec/syncPolicy/automated'

The only times I noticed this issue is when the actual sync fails for some reason (like the sync job failing in this case), but I suspect it happens more often, it just "reconciles" itself most of the time because the sync is successful.

Expected behavior

Older commits shoud never be re-applied if not explicitly targeted by revision or targetRevision.

Version

argocd: v3.3.0
  BuildDate: 2026-02-02T15:53:30Z
  GitCommit: fd6b7d5b3cba5e7aa7ad400b0fb905a81018a77b
  GitTreeState: clean
  GitTag: v3.3.0
  GoVersion: go1.25.6
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v3.3.1
  BuildDate: 2026-02-18T11:44:48Z
  GitCommit: 326a1dbd6b9f061207f814049f88e73fd8880c55
  GitTreeState: clean
  GitTag: v3.3.1
  GoVersion: go1.25.5
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.8.1 2026-02-09T16:15:27Z
  Helm Version: v3.19.4+g7cfb6e4
  Kubectl Version: v0.34.0
  Jsonnet Version: v0.21.0

Logs

I can paste more logs but they are very chatty so I'm unsure which one would actually help?

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriage/pendingThis issue needs further triage to be correctly classified

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions