Skip to content

ApplicationSet stuck on out of sync #26899

@ethanlsemperis

Description

@ethanlsemperis

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

We have a semi large applicationset with around 50 applications. The applicationset rolls out perfectly initially but tends to end up in a weird state where it does not sync the application when the paramter changes for the image.tag from the:

    parameters:
      - name: image.tag
        value: $ARGOCD_APP_REVISION

We are using progressiveSync with 4 stages. The 4th stage holds all of our microservices that use the ARGOCD_APP_REVISION to have their image tags updated as code is pushed to main.
To Reproduce

  • Deploy an application set with progressiveSyncs on 4 stages, here is the applicationset with company IP redacted:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    meta.helm.sh/release-namespace: argocd
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
  name: 
  namespace: argocd
spec:
  generators:
  - list:
      elements: [] # REDACTED FOR PRIVACY
      template:
        metadata: {}
        spec:
          destination: {}
          project: ""
  goTemplate: true
  goTemplateOptions:
  - missingkey=error
  ignoreApplicationDifferences:
  - jsonPointers:
    - /spec/source/helm/parameters
    - /spec/source/targetRevision
  strategy:
    deletionOrder: Reverse
    rollingSync:
      steps:
      - matchExpressions:
        - key: group
          operator: In
          values:
          - infrastructure
      - matchExpressions:
        - key: group
          operator: In
          values:
          - platform
      - matchExpressions:
        - key: group
          operator: In
          values:
          - seed
      - matchExpressions:
        - key: group
          operator: In
          values:
          - services
    type: RollingSync
  template:
    metadata:
      labels:
        app: '{{.releaseName}}'
        environment: '{{.environment}}'
        group: '{{.appGroup}}'
      name: '{{.name}}'
    spec:
      destination:
        namespace: '{{.namespace}}'
        server: https://kubernetes.default.svc
      project: default
      source:
        chart: '{{.chartName}}'
        helm:
          releaseName: '{{.releaseName}}'
          values: |
            environment: "{{.environment}}"
            {{- if and (hasKey . "useDefaultValues") .useDefaultValues }}
            imagePullSecrets:
              - name: {{.imagePullSecretName}}
            autoscaling:
              minReplicas: {{.minReplicas}}
              maxReplicas: {{.maxReplicas}}
            externalSecret:
              hooks:
                argocd:
                  enabled: true
              enabled: {{.externalSecret}}
              secret_name: {{.environment}}-{{.secretTier}}
              # Name is formatted as such to keep in alignment with external-secrets chart resource name
              secretStoreRefName: azure-store-{{.tier}}
              refreshInterval: {{.refreshInterval}}
            {{- end }}
            {{- if .customValues }}
            {{- $custom := fromJson .customValues }}
            {{- toYaml $custom | nindent 0 }}
            {{- end }}
        path: '{{.chartPath}}'
        repoURL: '{{.repoURL}}'
        targetRevision: '{{.targetRevision}}'
      syncPolicy:
        automated:
          allowEmpty: false
          prune: true
          selfHeal: true
        retry:
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 3m
          limit: 5
        syncOptions:
        - CreateNamespace=true
        - ServerSideApply=true
        - Validate=true
        - PrunePropagationPolicy=foreground
        - PruneLast=true
        - RespectIgnoreDifferences=true
  templatePatch: "{{- if hasKey . \"compareOptions\" }}{{- if .compareOptions }}\nmetadata:\n
    \ annotations:\n    argocd.argoproj.io/compare-options: {{ .compareOptions }}\n{{-
    end }}{{- end }}\nspec:\n  source:\n    helm:\n      ignoreMissingValueFiles:
    {{ .ignoreMissingValueFiles }}\n      {{- if or (and (hasKey . \"valueFiles\")
    .valueFiles) (and (hasKey . \"parameters\") .parameters) }}\n      {{- if and
    (hasKey . \"valueFiles\") .valueFiles }}\n      valueFiles:\n      {{- range splitList
    \",\" .valueFiles }}\n      - \"{{ . }}\"\n      {{- end }}\n      {{- end }}\n
    \     {{- if and (hasKey . \"parameters\") .parameters }}\n      parameters:\n
    \     {{- $params := fromJson .parameters }}\n      {{- range $key, $param :=
    $params }}\n      - name: \"{{ $param.name }}\"\n        value: \"{{ $param.value
    }}\"\n      {{- end }}\n      {{- end }}\n      {{- end }}\n  {{- if and (hasKey
    . \"ignoreDifferences\") .ignoreDifferences }}\n  ignoreDifferences:\n  {{- $ignores
    := fromJson .ignoreDifferences }}\n  {{- toYaml $ignores | nindent 6 }}\n  {{-
    end }}\n  \n  {{- if eq .ownsNamespace true }}\n  syncPolicy:\n    managedNamespaceMetadata:\n
    \     annotations:\n        # Workaround for namespace deletion issues - https://github.com/argoproj/argo-cd/issues/7875\n
    \       argocd.argoproj.io/tracking-id: {{ .name }}:/Namespace:{{ .namespace }}/{{
    .namespace }}\n  {{- end }}\n"

Let the applicationset sit around for about a day, new changes are pushed to the services main branch, application show as out of sync, but the proressiveSync states as waiting.

Expected behavior
The applicationset controller should trigger a reconcile for the argocd application.

Screenshots

Image - application showing as out of sync but waiting Image - deployment showing as out of sync Image - showing desired image tag change

Version

v3.4.0-rc1

Logs
Applicationset controller logs

time="2026-03-18T21:55:23Z" level=info msg="generated 47 applications" applicationset=argocd/REDACTED-20260311182604
time="2026-03-18T21:55:23Z" level=info msg="ApplicationSet REDACTED-20260311182604 step list:" applicationset=argocd/REDACTED-20260311182604
time="2026-03-18T21:55:23Z" level=info msg="step 1: [REDACTED]" applicationset=argocd/REDACTED-20260311182604
time="2026-03-18T21:55:23Z" level=info msg="step 2: [REDACTED]" applicationset=argocd/REDACTED-20260311182604
time="2026-03-18T21:55:23Z" level=info msg="step 3: [REDACTED]" applicationset=argocd/REDACTED-20260311182604
time="2026-03-18T21:55:23Z" level=info msg="step 4: [REDACTED]" applicationset=argocd/REDACTED-20260311182604
time="2026-03-18T21:55:23Z" level=info msg="Application allowed to sync before maxUpdate?: map[REDACTED, ALL OF THESE SHOW TRUE]" applicationset=argocd/REDACTED-20260311182604
time="2026-03-18T21:55:23Z" level=info msg="unknown field \"status.health\"" ApplicationSet="{REDACTED-20260311182604 argocd}" controller=applicationset controllerGroup=argoproj.io controllerKind=ApplicationSet name=REDACTED-20260311182604 namespace=argocd reconcileID=8d3e308f-ec09-489f-89cc-cccd62d9ce4d
time="2026-03-18T21:55:23Z" level=info msg="end reconcile in 499.028333ms" applicationset=argocd/REDACTED-20260311182604 requeueAfter=0s
time="2026-03-18T22:02:29Z" level=info msg="Alloc=47173 TotalAlloc=709691392 Sys=123994 NumGC=24618 Goroutines=186"

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingbug/priority:urgentBug should be fixed immediatelyfeature:progressive-syncIssues related to the ApplicationSet progressive sync featuretriage/pendingThis issue needs further triage to be correctly classified

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions