troubleshooting warning kubernetes · ·

Fix ArgoCD OutOfSync With No Diff (Ghost Status)

Diagnose and fix ArgoCD OutOfSync errors with empty diffs caused by finalizers, aggregated RBAC, mutating webhooks, and metadata drift using CLI commands and...

argocd OutOfSync
Fix ArgoCD OutOfSync With No Diff (Ghost Status)
Advertisement

Problem

You check your ArgoCD dashboard and see OutOfSync. You open the diff panel expecting to find a config drift. The panel is empty. No changed fields. No added or removed lines. Nothing. The application status flickers between Synced and OutOfSync with no apparent reason.

This happens because ArgoCD’s reconciliation loop detects a difference between your desired state (in Git) and the live cluster state, but that difference lives in metadata or infrastructure the normal diff view ignores. The application behaves correctly, but the sync status is unreliable. In clusters with over 100 applications, this ghost OutOfSync creates noise that hides actual drift. You lose trust in the dashboard and cannot tell real drift from noise.

Root Causes

Four common culprits cause this ghost OutOfSync. Check them in order.

Finalizers and ownerReferences

Kubernetes resources with metadata.finalizers (for example, namespace termination protection or custom controllers) block deletion. When ArgoCD wants your resource to match a Git state without finalizers, but the live resource has them, ArgoCD sees a difference. The diff view does not show finalizers by default, so the panel looks clean. The same applies to dynamically injected ownerReferences from parent resources. Namespaces stuck in Terminating with a finalizer are the most common trigger.

Aggregated RBAC roles

ClusterRoles using aggregationRule (for example, system:aggregated-to-view) dynamically inherit rules from other ClusterRoles. When a platform operator adds or removes a label selector on an aggregated ClusterRole outside ArgoCD control, every role that aggregates it changes. ArgoCD detects the rule change and reports OutOfSync, but the diff view shows nothing because the aggregation happens server-side.

Conversion webhooks and mutating admission controllers

Tools like Istio, Kyverno or Open Policy Agent inject sidecars, annotations or labels after ArgoCD compares the manifest to the live state. ArgoCD sees the injected values as drift. The diff normalization in the UI strips these known injection points, so the panel appears empty. You get a false positive every time a new pod rolls out.

Resource normalization differences

ArgoCD normalizes certain fields before generating a diff. Default values injected by the Kubernetes API server (for example, status fields, metadata.generation) create drift that ArgoCD’s normalization rules hide from the diff view but not from the sync status calculation.

Solution

Diagnose systematically using the ArgoCD CLI. Run a hard refresh and force the raw diff to appear:

$ argocd app diff <your-app-name> --hard-refresh

The --hard-refresh flag forces ArgoCD to re-fetch the live state from the cluster and recompare. This often reveals the hidden fields as raw YAML patches. Look for Finalizers:, OwnerReferences: or injected annotation keys near the top of the output.

If you see finalizers causing the diff, edit the source manifest in Git to match the finalizers or remove them entirely. For namespaces managed by ArgoCD, add a PreSync hook that strips finalizers before sync:

$ kubectl patch namespace <ns-name> -p '{"metadata":{"finalizers":[]}}' --type=merge

Then commit the removal to your Git repository.

If the diff shows aggregated ClusterRole rules, your options are limited. Option A: remove aggregated ClusterRoles from your ArgoCD Application specification and let the cluster manage them outside GitOps. Option B: apply a resourceCustomization patch in the Application spec to ignore the aggregated rules. ArgoCD v2.6+ supports spec.ignoreAggregatedRoles: true directly (check your version).

For mutating webhooks (Istio sidecars, Kyverno policies), use ignoreDifferences in your Application spec:

spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/template/spec/containers/0/image
  syncPolicy:
    syncOptions:
    - RespectIgnoreDifferences=true

This tells ArgoCD to skip those specific fields during comparison. The RespectIgnoreDifferences option ensures the UI respects your ignored fields in the status calculation. For a deeper walkthrough of configuring sync policies, see this tutorial on Argo CD Sync Policies.

Prevention

Set up a consistent strategy from day one. For every Application that uses mutating webhooks, always include ignoreDifferences blocks before the first sync. Pin all aggregated ClusterRoles outside your Git repos. Use ArgoCD ApplicationSets with a single ignoreDifferences template to avoid repeating yourself across hundreds of applications. For production clusters with non-uniform admission controllers, run a weekly CLI audit with argocd app list -o wide | grep OutOfSync. If the count exceeds a reasonable baseline (for example, more than 2% of applications), investigate before it becomes noise that real drift hides behind.

ArgoCD’s official documentation covers normalization details in the Resource Diff section. Bookmark it for when you hit edge cases your ignoreDifferences blocks do not cover.

Advertisement

Get the next article in your inbox

Practical DevOps tips, tutorials, and guides. No spam, unsubscribe anytime.