🎉 DevOps Interview Prep Bundle is live — 1000+ Q&A across 20 topicsGet it →
All Articles

FluxCD Reconciliation Failing or OOMKilled: How to Fix It

FluxCD source-controller or kustomize-controller getting OOMKilled or stuck in reconciliation loop? Here's how to diagnose and fix it.

DevOpsBoys3 min read
Share:Tweet

FluxCD is usually rock-solid, but there are specific failure modes that catch people off guard — especially as your repo size or number of resources grows.

Common Symptoms

# Kustomization stuck
kubectl get kustomization -n flux-system
NAME           READY   STATUS
my-app         False   kustomize build failed: ...

# Source OOMKilled
kubectl get pods -n flux-system
source-controller-xxx    0/1   OOMKilled   5   10m

# Reconciliation loop
Events: Warning Progressing Kustomization/my-app 
  dependency 'flux-system/infrastructure' is not ready

Fix 1: source-controller OOMKilled

The source-controller fetches and indexes your Git repo. If your repo is large (many files, large history), it can run out of memory.

bash
# Check current resource limits
kubectl describe deployment source-controller -n flux-system | grep -A 10 "Limits"

Increase the memory limit:

yaml
# Create a patch file
# patches/source-controller-memory.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: source-controller
  namespace: flux-system
spec:
  template:
    spec:
      containers:
        - name: manager
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi    # increase from default 512Mi
            requests:
              cpu: 50m
              memory: 64Mi

Apply via Kustomize patch in your flux-system kustomization:

yaml
# flux-system/kustomization.yaml
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - path: patches/source-controller-memory.yaml
    target:
      kind: Deployment
      name: source-controller

Fix 2: Reconciliation Dependency Deadlock

FluxCD Kustomizations can depend on each other. If the dependency chain has a problem, everything gets stuck.

bash
# Check Kustomization status
kubectl get kustomization -n flux-system -o wide
 
# Get detailed status
kubectl describe kustomization infrastructure -n flux-system

Look for:

Status:
  Conditions:
    Message: dependency 'flux-system/cert-manager' is not ready
    Reason: DependencyNotReady
    Status: "False"

This means cert-manager kustomization is failing, which blocks infrastructure, which blocks apps. Find the root cause:

bash
# Walk the dependency chain
kubectl get kustomization cert-manager -n flux-system -o yaml | grep -A 20 "conditions:"

Fix 3: kustomize build Failed

bash
# Check the exact error
kubectl describe kustomization my-app -n flux-system | grep -A 20 "Message:"

Common errors:

Error: "accumulating resources: read file... no such file"

yaml
# Wrong path in kustomization.yaml
resources:
  - ../../wrong-path/deployment.yaml  # file doesn't exist in repo

Fix: correct the path and commit.

Error: "multiple matches for kind Deployment"

You have two resources with the same name/namespace. Check for duplicates:

bash
kustomize build ./overlays/prod/ 2>&1

Run kustomize locally to see the exact error before committing.

Error: "json: cannot unmarshal string into Go struct field"

YAML formatting issue in one of your manifests. Validate with:

bash
kustomize build ./overlays/prod/ | kubectl apply --dry-run=client -f -

Fix 4: Git Repository Not Syncing

bash
# Check GitRepository status
kubectl get gitrepository -n flux-system
 
# If URL or auth issue:
kubectl describe gitrepository flux-system -n flux-system

If you rotated your deploy key:

bash
# Delete old secret and re-create
kubectl delete secret flux-system -n flux-system
 
# Re-bootstrap or create the secret manually
kubectl create secret generic flux-system \
  -n flux-system \
  --from-file=identity=./deploy_key \
  --from-file=identity.pub=./deploy_key.pub \
  --from-file=known_hosts=./known_hosts

Fix 5: Force Reconciliation

If Flux is stuck and you want to trigger an immediate reconciliation:

bash
# Install flux CLI
brew install fluxcd/tap/flux
 
# Force reconcile everything
flux reconcile source git flux-system
flux reconcile kustomization flux-system
 
# Reconcile a specific app
flux reconcile kustomization my-app -n flux-system

Fix 6: Suspend and Resume

Sometimes you need to temporarily stop Flux from reconciling while you fix something manually:

bash
# Suspend reconciliation
flux suspend kustomization my-app
 
# Make your changes manually...
 
# Resume and force sync
flux resume kustomization my-app
flux reconcile kustomization my-app

Check Health at a Glance

bash
# All-in-one health check
flux check
 
# Get all failing resources
kubectl get kustomization,helmrelease,gitrepository,helmrepository -A | grep -v "True"

Monitoring Flux with Prometheus

Flux exposes metrics. Alert on these:

yaml
# Alert when Kustomization has been failing > 5 min
- alert: FluxKustomizationFailing
  expr: |
    gotk_reconcile_condition{kind="Kustomization",type="Ready",status="False"} == 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "FluxCD Kustomization {{ $labels.name }} is failing"

Most Flux issues come down to three things: memory on the source-controller, dependency chain problems, or git authentication. Start there and you'll resolve 90% of issues.

Docs: FluxCD troubleshooting guide

🔧

Today I Fixed

Short real fixes from production — posted daily

Browse fixes
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.

Related Articles

Comments