Fix Flux SOPS illegal base64 data at input byte 0

NEEDS_INTERNAL_LINKS: only 1 valid internal link found (additional link required)

Problem

The error illegal base64 data at input byte 0 appears in Flux’s kustomize-controller logs when it tries to decrypt a SOPS-encrypted Kubernetes Secret but finds data that is not valid base64. The decryption step (DecryptSecrets) expects the ciphertext inside the Secret’s data field to be base64-encoded. If the encrypted value is raw text or includes whitespace, base64 decoding fails and Flux rejects the manifest.

This is a common frustration for teams adopting GitOps with encrypted secrets. The error message is terse , it does not tell you which field is broken, only that the first byte of the input is illegal. You end up hunting through Secrets, age keys, and Kustomization configurations.

Root Causes

Three distinct misconfigurations trigger this error in practice.

1. Raw SOPS ciphertext in a Secret’s data field

When you run sops --encrypt secret.yaml, the output is a YAML file with plaintext keys like encrypted_regex and sops metadata. If you copy the value of an encrypted field directly into a Kubernetes Secret’s data block, that ciphertext is valid YAML , but it is not base64-encoded. The kustomize-controller wraps the ciphertext in base64 during processing, so feeding it raw base64-unencoded text causes the error.

2. Malformed age key secret

The age private key used by SOPS (stored in a Secret like sops-age-key) must also be base64-encoded in the Secret’s data.age-key field. A common mistake is to use stringData to inject the key as plaintext, but a trailing newline or extra whitespace in the key file produces invalid base64 when the value is later encoded. Even a single space character breaks decoding.

3. Truncated or corrupted secret data

Copying encrypted values from a terminal or a web UI often introduces invisible characters, line breaks, or truncated strings. If the ciphertext is even one character short, base64 decoding fails at the first byte.

Solution

The fix is to ensure that both the age key secret and the encrypted Kubernetes Secret store their data as valid base64. Follow these steps.

Step 1: Verify your age key secret

Get the current secret and decode its base64 value to confirm it matches the private key file:

$ kubectl get secret sops-age-key -n flux-system -o jsonpath='{.data.age-key}' | base64 -d | head -c 80

If the output looks like AGE-SECRET-KEY-1QQP... without trailing garbage, the key is valid. If you see base64: invalid input or garbled text, recreate the secret.

Generate a clean age key secret:

The safest method is to use --from-file, which tells kubectl to base64-encode the file content automatically:

$ kubectl create secret generic sops-age-key \
    --namespace=flux-system \
    --from-file=age-key=~/.config/sops/age/keys.txt

If you prefer --from-literal, strip the trailing newline and pass the raw key text (do not pre-encode it):

$ kubectl create secret generic sops-age-key \
    --namespace=flux-system \
    --from-literal=age-key="$(cat ~/.config/sops/age/keys.txt | tr -d '\n')"

Pre-encoding with base64 -w0 before --from-literal causes double base64 encoding, which breaks decryption.

Step 2: Re-encrypt the Secret using SOPS with the correct format

Do not manually construct the Secret YAML. Let SOPS handle the encoding:

$ sops --encrypt --age age1...your-public-key... secret.yaml

This produces a YAML file where the ciphertext is already the base64-encoded representation that Flux expects. The sops metadata block is automatically added and recognised by Flux’s decryption.

Step 3: Check the Kustomization decryption config

Ensure your Flux Kustomization object points to the correct key secret. For example:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  decryption:
    provider: age
    secretRef:
      name: sops-age-key

Run flux logs -n flux-system --level=error to confirm the error is gone after re-applying.

Step 4: Validate the encrypted Secret manually

Before committing, test that the Secret’s data field is valid base64:

$ kubectl get secret my-encrypted-secret -n my-namespace -o json | jq -r '.data | to_entries[] | .key + ": " + (.value[:40]) + "..."' | head -5

Then decode one value to ensure base64 decoding succeeds:

$ kubectl get secret my-encrypted-secret -n my-namespace -o jsonpath='{.data.my-key}' | base64 -d | head -c 100

If that fails, the ciphertext is corrupted. If it succeeds, you will see a block of encrypted text (not human-readable). That is normal.

Prevention

• Use sops --encrypt --output-type yaml to always produce a valid Flux-compatible Secret.
• Never edit encrypted Secrets by hand. Always go through SOPS.
• Validate base64 encoding in CI with a simple base64 -d check on the Secret’s data values.
• Store the age key secret declaratively in Git using a SOPS-encrypted YAML file that Flux decodes, rather than manually creating the key secret via kubectl.

For more on troubleshooting Flux reconciliation, see the official Flux SOPS guide and this practical Kubernetes debug walkthrough for applying similar diagnostic steps.