Manage Kubeflow AI Workloads with Headlamp Plugin

When a Kubeflow notebook hangs on startup or a Katib trial dies without a clear reason, you usually end up in a terminal running kubectl describe, kubectl get events, and kubectl logs across three namespaces to reconstruct what happened. The Headlamp Kubeflow plugin removes most of that. It teaches the Headlamp dashboard to read Kubeflow’s custom resources directly, so a stuck Notebook, a failed Pipeline run, or a bad AutoML suggestion shows up as a first-class object with its status conditions, owned pods, and configuration in one view.

This post walks through what the plugin actually surfaces, how to install it on a desktop or in-cluster Headlamp, and how it changes the way you debug ML workloads on Kubernetes. It was introduced on the Kubernetes blog in July 2026 and is maintained under Kubernetes SIG UI with an Apache 2.0 license.

The problem: Kubeflow hides behind CRDs

Kubeflow is not one application. It is a set of controllers that each register their own Custom Resource Definitions and reconcile them into ordinary Kubernetes objects. A Notebook becomes a StatefulSet and a Pod. A Katib Experiment fans out into Trial and Suggestion resources, each of which spawns Jobs. A Pipeline run creates a graph of Argo Workflow steps. The Kubeflow Central Dashboard gives data scientists a clean surface over all of this, but it is built for the person running experiments, not the person keeping the cluster healthy.

When something breaks, the operator’s questions are Kubernetes questions. Which pod is pending, and why? What does the resource’s status block say? Is the controller even reconciling this object? The Central Dashboard was not designed to answer those, so operators fall back to kubectl. That is fine for one broken notebook. It stops scaling the moment you are supporting a shared cluster with dozens of researchers and several Kubeflow components installed.

Headlamp already solves the general version of this problem. It is a lightweight, extensible Kubernetes UI that runs as a desktop app or in-cluster, and its plugin system lets anyone add first-class views for custom resources. The Kubeflow plugin is that idea applied to Kubeflow’s CRDs.

What the plugin surfaces

The plugin adds operator-focused views for the resources that matter when you are troubleshooting AI/ML workloads. As of the current release it recognizes these component groups:

For each object you get the same drill-down Headlamp gives any native resource: the live status conditions, the owned pods with their phases, the events, and the full YAML. That is the difference that matters during an incident. Instead of guessing that a stuck notebook is a scheduling problem, you open the Notebook, see its Pod stuck Pending, click through to the Pod, and read the FailedScheduling event that says there is no node with a free GPU.

Auto-detection: you install only what you run

Very few teams run all of Kubeflow. You might deploy only Katib for hyperparameter tuning, or only the Notebooks controller for a shared research environment. The plugin handles this by checking the cluster’s API for which Kubeflow CRDs are actually registered, then showing sidebar sections only for the components it finds.

The practical effect is that the plugin is safe to install everywhere. If a cluster has no Katib CRDs, the Katib section simply does not appear. There is no configuration file listing which features to turn on, and no broken menu entries pointing at resources that do not exist. Install a new component later with Helm or Kustomize, refresh Headlamp, and its section shows up on its own.

Installing on a desktop Headlamp

The fastest way to try the plugin is the desktop app. Download and run Headlamp, point it at a cluster your kubeconfig can reach, and open the Plugin Catalog from the sidebar. Search for the Kubeflow plugin, install it, and reload. If the target cluster has any Kubeflow CRDs, the new sections appear in the left navigation.

This path is ideal for an operator who wants to inspect a cluster from a laptop without deploying anything into it. The plugin runs inside your local Headlamp process and talks to the cluster through the same API access your kubeconfig already grants.

Installing in-cluster

For a shared dashboard the whole team uses, run Headlamp in the cluster and load the plugin through the Helm chart’s plugin manager. The chart supports declaring plugins in values.yaml, and a sidecar keeps them in sync:

config:
  pluginsDir: /headlamp/plugins
pluginsManager:
  enabled: true
  watchPlugins: true

There are two supported in-cluster patterns. The plugin manager shown above pulls and updates plugins for you, which is the recommended approach. The alternative is an initContainer that copies plugin files into a shared volume before Headlamp starts. Both end with the plugin’s static assets sitting in the directory Headlamp reads at boot. Pick the plugin manager unless you have a reason to bake plugins into an image yourself.

Whichever you choose, remember that Headlamp respects the RBAC of whoever is logged in. The Kubeflow views do not grant new access. A user who cannot get notebooks in a namespace will not see them in the plugin either, which is exactly what you want on a multi-tenant research cluster.

Try it on a throwaway cluster

You do not need a real Kubeflow install to see how the plugin behaves. Because it keys off CRDs, applying the definitions alone is enough to light up the UI. Spin up a local cluster with kind:

$ kind create cluster --name headlamp-kubeflow
$ kubectl config use-context kind-headlamp-kubeflow

Apply a component’s CRDs. For the Notebooks controller the upstream manifests install the Notebook definition among others:

$ kubectl apply -k "github.com/kubeflow/notebooks/notebook-controller/config/crd?ref=main"

Confirm the CRD registered:

$ kubectl get crd notebooks.kubeflow.org
NAME                     CREATED AT
notebooks.kubeflow.org   2026-08-05T09:14:22Z

Open Headlamp against this cluster and the Notebooks section appears. Create a sample Notebook object and you can watch the plugin render its status even before a real controller reconciles it, which is a fast way to learn the views without provisioning GPUs. When you are done, delete the cluster with $ kind delete cluster --name headlamp-kubeflow.

A realistic debugging pass

Here is how a shared-cluster incident looks with the plugin in place. A researcher reports that their training run “just stopped.” The steps you would otherwise do by hand collapse into a short click path:

1. Open the Katib section and find the researcher’s Experiment. Its status shows Failed with a condition message pointing at the last Trial.
2. Click into that Trial. The plugin shows the Job it created and the Pod that ran it, along with the Pod’s phase.
3. Open the Pod. Its last state is Terminated with reason OOMKilled and exit code 137.
4. Read the container spec in the same view: the memory limit is 4Gi, well under what the model needs.

The fix is a bigger memory request in the trial template, but the point is the diagnosis. You went from a vague “it stopped” to OOMKilled without typing a single kubectl command or context-switching between namespaces. Memory pressure and eviction are the same failure modes you already know from general Kubernetes work, and if you want a deeper reference on reading pod-level failure states, our guide on fixing CrashLoopBackOff covers the status conditions the plugin puts in front of you.

Where it fits alongside the Central Dashboard

The Kubeflow Central Dashboard and the Headlamp plugin are not competitors. They serve different people. The Central Dashboard is where a data scientist launches notebooks, submits pipelines, and reviews experiment results. The Headlamp plugin is where an operator or SRE answers infrastructure questions about those same objects: scheduling, resource limits, controller health, and pod lifecycle.

On a small team one person wears both hats and might use both tools. On a larger platform team, the split is cleaner. Researchers live in the Central Dashboard. The people running the cluster live in Headlamp, and the Kubeflow plugin means they no longer have to translate every ML abstraction back into raw pods by hand. If your team is standing up this kind of shared ML platform, it pairs naturally with broader workload visibility work like LLM observability on Kubernetes and the capacity planning behind horizontal pod autoscaling for bursty training jobs.

What it does not do

Be clear about the plugin’s scope so you do not expect the wrong thing from it.

• It is read and inspect focused. It surfaces state and configuration for troubleshooting; it is not a control plane for editing experiment definitions or launching training runs. Do that through Kubeflow’s own tooling.
• It reflects only what the CRDs and controllers expose. If a controller writes a thin status block, the plugin can only show that thin block. It reads the cluster, it does not add telemetry the cluster is not already recording.
• It does not replace metrics and logs pipelines. For GPU utilization trends, cost, or historical training throughput you still want Prometheus, your logging stack, and a proper observability setup. The plugin answers “what is the state of this object right now,” not “how has this behaved over the last week.”

None of that is a knock on the tool. It is a focused operator UI, and knowing the boundary keeps you from reaching for it when you actually need a time-series dashboard.

The takeaway

If you operate a cluster that runs Kubeflow, the Headlamp Kubeflow plugin is a low-cost addition that pays off the first time a notebook or trial breaks. It is open source under Kubernetes SIG UI, it detects which components you run so it is safe to install broadly, and it turns the CRD archaeology of an ML incident into a few clicks through resources you already understand. Install it on a desktop Headlamp to try it in minutes, then move it into an in-cluster deployment through the plugin manager once your team wants a shared view. The next time someone says their training run “just stopped,” you will find out why without opening a terminal.