Skip to content

Excessive never-ending retry loop of /api/v1/stream/applications and /api/v1/stream/applications/<app>/resource-tree when OIDC token has expired #24807

@illrill

Description

@illrill

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

When browsing an Application in the UI, and then leaves the browser tab open / untouched for a while (so that the OIDC token expires), JS-triggered GRPC requests to the /v1/stream API's continues with a very aggressive retry cadence. We've observed the following requests coming in at a very high request rate:

  • GET /argocd/api/v1/stream/applications?name=<APPLICATION>&appNamespace=argocd (~1000 requests per minute)
  • GET /argocd/api/v1/stream/applications/<APPLICATION>/resource-tree?appNamespace=argocd (~200 requests per minute)

They are all getting 401 responses (rightfully since the token is expired), however this doesn't trigger the "main" browser page to refresh the token / log in again, so the GRPC requests just continue forever until the user clicks a button or refreshes the browser to get a fresh token. Only then does the excessive GRPC calls stop.

The response body is the following (for both requests)

{
    "error": {
        "grpc_code": 16,
        "http_code": 401,
        "message": "invalid session: failed to verify the token",
        "http_status": "Unauthorized"
    }
}

Browser console:

Image

Logs from argocd-server:

time="2025-10-01T11:20:26Z" level=info msg="finished call" grpc.code=Unauthenticated grpc.component=server grpc.error="rpc error: code = Unauthenticated desc = invalid session: failed to verify the token" grpc.method=WatchResourceTree grpc.method_type=server_stream grpc.service=application.ApplicationService grpc.start_time="2025-10-01T11:20:26Z" grpc.time_ms=0.507 peer.address="[::1]:55068" protocol=grpc
time="2025-10-01T11:20:26Z" level=warning msg="Failed to verify token: failed to verify token: oidc: token is expired (Token Expiry: 2025-10-01 09:11:14 +0000 UTC)"
time="2025-10-01T11:20:26Z" level=info msg="finished call" grpc.code=Unauthenticated grpc.component=server grpc.error="rpc error: code = Unauthenticated desc = invalid session: failed to verify the token" grpc.method=WatchResourceTree grpc.method_type=server_stream grpc.service=application.ApplicationService grpc.start_time="2025-10-01T11:20:26Z" grpc.time_ms=0.496 peer.address="[::1]:55068" protocol=grpc
time="2025-10-01T11:20:26Z" level=warning msg="Failed to verify token: failed to verify token: oidc: token is expired (Token Expiry: 2025-10-01 09:11:14 +0000 UTC)"
time="2025-10-01T11:20:26Z" level=info msg="finished call" grpc.code=Unauthenticated grpc.component=server grpc.error="rpc error: code = Unauthenticated desc = invalid session: failed to verify the token" grpc.method=Watch grpc.method_type=server_stream grpc.service=application.ApplicationService grpc.start_time="2025-10-01T11:20:26Z" grpc.time_ms=0.558 peer.address="[::1]:55068" protocol=grpc
time="2025-10-01T11:20:26Z" level=warning msg="Failed to verify token: failed to verify token: oidc: token is expired (Token Expiry: 2025-10-01 09:11:14 +0000 UTC)"
time="2025-10-01T11:20:26Z" level=info msg="finished call" grpc.code=Unauthenticated grpc.component=server grpc.error="rpc error: code = Unauthenticated desc = invalid session: failed to verify the token" grpc.method=Watch grpc.method_type=server_stream grpc.service=application.ApplicationService grpc.start_time="2025-10-01T11:20:26Z" grpc.time_ms=0.465 peer.address="[::1]:55068" protocol=grpc
time="2025-10-01T11:20:26Z" level=warning msg="Failed to verify token: failed to verify token: oidc: token is expired (Token Expiry: 2025-10-01 09:11:14 +0000 UTC)"

Request statistics:

  • GET /argocd/api/v1/stream/applications?name=<APPLICATION>&appNamespace=argocd
  • ~1000 requests per minute
Image
  • GET /argocd/api/v1/stream/applications/<APPLICATION>/resource-tree?appNamespace=argocd
  • ~200 requests per minute
Image

To Reproduce

  • Use my OIDC configuration below.
  • Open any Application page in the UI.
  • Leave it open / untouched until the token has expired.
configs:
  params:
    server.rootpath: /argocd
    server.basehref: /argocd
  cm:
    oidc.config: |
      name: Azure
      issuer: https://login.microsoftonline.com/**REDACTED**/v2.0
      clientID: **REDACTED**
      clientSecret: $oidc.azure.clientSecret
      requestedIDTokenClaims:
        groups:
          essential: true
      requestedScopes:
        - openid
        - profile
        - email

Expected behavior

The main page should be redirected to log in again if these GRPC requests are getting 401 responses. And/or the GRPC requests should stop retrying after X amount of failed requests.

Version

argocd: v3.1.1+fa342d1
  BuildDate: 2025-08-25T17:01:33Z
  GitCommit: fa342d153e0e7942938256aea491a68439a53c44
  GitTreeState: clean
  GoVersion: go1.25.0
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v3.0.12+ed1e239
  BuildDate: 2025-07-25T19:36:41Z
  GitCommit: ed1e2397ef9af6d23b284e39d504308cdda1957b
  GitTreeState: clean
  GoVersion: go1.24.4
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.6.0 2025-01-14T15:12:17Z
  Helm Version: v3.17.1+g980d8ac
  Kubectl Version: v0.32.2
  Jsonnet Version: v0.20.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcomponent:uiUser interfaces bugs and enhancements

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions