# kubectl annotate: Change-Cause and Rollout History - URL: https://karandeepsingh.ca/posts/kubectl-annotate-change-cause/ - Author: Karandeep Singh - Published: 2026-10-19 - Series: Kubernetes and Containers - Tags: kubernetes, cli, containers - Summary: Start with one kubectl annotate command and keep adding. Overwrite and remove annotations, fill the CHANGE-CAUSE column in rollout history, catch the stale change-cause trap, roll back, then move it all into git with a deploy script that writes the cause from your commit. It is 2 a.m., checkout is throwing 500s, and you run `kubectl rollout history` to see what changed. Five revisions stare back at you, and every one of them says ``. Somebody shipped something. Nobody wrote down what. This is the problem `kubectl annotate` solves, and it takes about one line per deploy. In this guide we start with a single `kubectl annotate` command and keep adding to it. By the end you will know how to add, overwrite, and remove annotations, how `kubernetes.io/change-cause` fills that empty CHANGE-CAUSE column, the trap that makes a rollout lie about its reason, and how to move the whole thing into git so your history writes itself from commit messages. Every command and every error below was run against a real Kubernetes 1.34 API server with kubectl 1.34.1. The output is copied from that session, not typed from memory. _Diagram: You annotate the **Deployment**. On each rollout the controller copies its annotations onto the new **ReplicaSet**, and `kubectl rollout history` reads them back from the ReplicaSets. That copy step explains every surprise in this article._ ## What You Need for kubectl annotate - A cluster you are allowed to break. kind, minikube, k3d, or a dev namespace all work. - `kubectl` 1.27 or newer. The output below is from 1.34.1. - `git`, for the second half of the article. If Deployments and ReplicaSets are new to you, read [Kubernetes fundamentals: Pods, Deployments and Services](https://karandeepsingh.ca/posts/kubernetes-fundamentals-pods-deployments-services/) first. The [kubectl cheat sheet](https://karandeepsingh.ca/posts/kubectl-cheat-sheet/) covers the everyday commands this article builds on. **tip (no pods required):** Annotations live on the API object, so nothing in this article needs the Pods to actually be running. If your test cluster has no nodes, or the image never pulls, every command still behaves the same. ## Step 1: Your First kubectl annotate **What:** Create a Deployment and stick one annotation on it. **Why:** Before change-cause makes sense, you need to see what an annotation is. It is a key and a string value attached to an object. Kubernetes stores it and otherwise ignores it. ```text $ kubectl create deployment web --image=nginx:1.27.0 deployment.apps/web created $ kubectl annotate deployment web team=payments deployment.apps/web annotated ``` That's the whole command: resource type, name, then `key=value`. Read it back with `--list`, which prints one annotation per line: ```text $ kubectl annotate deployment web --list deployment.kubernetes.io/revision=1 team=payments $ kubectl get deployment web -o jsonpath='{.metadata.annotations}' {"deployment.kubernetes.io/revision":"1","team":"payments"} ``` Notice we got two annotations back, not one. `deployment.kubernetes.io/revision` was put there by the Deployment controller. Kubernetes uses annotations for its own bookkeeping all the time, and you will meet a few more of them before we are done. ## Step 2: Change It, and Hit the Wall **What:** Try to change the value from `payments` to `checkout`. **Why:** This is the first thing everybody does, and it fails. ```text $ kubectl annotate deployment web team=checkout # exits with status 1 error: --overwrite is false but found the following declared annotation(s): 'team' already has a value (payments) ``` kubectl will not replace an existing value unless you tell it to. It's a guard rail, and a good one: in a busy cluster, the annotation you are about to clobber might belong to some other tool. Add `--overwrite` when you mean it: ```text $ kubectl annotate deployment web team=checkout --overwrite deployment.apps/web annotated $ kubectl annotate deployment web --list deployment.kubernetes.io/revision=1 team=checkout $ kubectl annotate deployment web team- deployment.apps/web annotated $ kubectl annotate deployment web --list deployment.kubernetes.io/revision=1 ``` The trailing minus in `team-` removes the key. Removing does not need `--overwrite`, and it does not complain if the key is already gone. **Question:** Why would kubectl make overwriting harder than adding? Think about who else writes annotations on the same object. ## Step 3: Annotations vs Labels Before we go further, here's the question that comes up every time: why not just use a label? **labels** - Short, strict values (63 characters, no spaces) - **Selectable**: `kubectl get pods -l app=web` - Services, ReplicaSets and NetworkPolicies match on them - Changing one on a Pod can pull it out of a Service **annotations** - Free-form strings: sentences, URLs, JSON - **Not selectable**: you cannot query by annotation - Read by people and tools, ignored by the scheduler - Up to 256 KiB per object, all keys combined The rule of thumb: if a controller needs to *find* objects by it, it's a label. If a human or a tool needs to *read* it, it's an annotation. "Why did we deploy this" is the most human-readable thing there is, so it's an annotation. That 256 KiB limit is real, and the API server enforces it across all of an object's annotations together: ```text $ kubectl patch deployment/web --type=merge --patch-file big.json # exits with status 1 The Deployment "web" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes ``` You will never hit that with a change-cause. You *can* hit it with tools that stash whole configs in annotations, which we'll come back to in Step 8. ## Step 4: Fill the CHANGE-CAUSE Column **What:** Roll out a new image and record why. **Why:** This is the reason most people end up searching for kubectl annotate in the first place. Push a new image first and look at the history: ```text $ kubectl set image deployment/web nginx=nginx:1.27.1 deployment.apps/web image updated $ kubectl rollout history deployment/web deployment.apps/web REVISION CHANGE-CAUSE 1 2 ``` The CHANGE-CAUSE column is just the `kubernetes.io/change-cause` annotation. So let's set it. Here's how I typed it the first time: ```text $ kubectl annotate deployment/web kubernetes.io/change-cause=bump nginx to 1.27.1 # exits with status 1 error: all resources must be specified before annotation changes: nginx ``` That error is confusing until you see what the shell did. Without quotes, the value stopped at the first space. kubectl got `change-cause=bump` as the annotation, then saw `nginx`, `to` and `1.27.1` as extra words and tried to read `nginx` as another resource name. Quote the value: ```text $ kubectl annotate deployment/web kubernetes.io/change-cause="bump nginx to 1.27.1" deployment.apps/web annotated $ kubectl rollout history deployment/web deployment.apps/web REVISION CHANGE-CAUSE 1 2 bump nginx to 1.27.1 ``` Revision 2 now explains itself. And notice we annotated *after* the rollout and still landed on revision 2. That's the copy step from the diagram at the top: the controller copies the Deployment's annotations onto the newest ReplicaSet whenever they change. You can see where the value really lives: ```text $ kubectl get rs -o custom-columns='NAME:.metadata.name,REVISION:.metadata.annotations.deployment\.kubernetes\.io/revision,CHANGE-CAUSE:.metadata.annotations.kubernetes\.io/change-cause,IMAGE:.spec.template.spec.containers[0].image' NAME REVISION CHANGE-CAUSE IMAGE web-69ccdfbf95 2 bump nginx to 1.27.1 nginx:1.27.1 web-f799c975d 1 nginx:1.27.0 ``` ## Step 5: The Stale Change-Cause Trap **What:** Ship one more image, and this time forget the annotation. **Why:** Because everybody forgets eventually, and the result is worse than ``. ```text $ kubectl set image deployment/web nginx=nginx:1.27.2 deployment.apps/web image updated $ kubectl rollout history deployment/web deployment.apps/web REVISION CHANGE-CAUSE 1 2 bump nginx to 1.27.1 3 bump nginx to 1.27.1 ``` Revision 3 runs nginx 1.27.2, but history swears it is the 1.27.1 bump. The Deployment still carried the old annotation, the controller copied it onto the new ReplicaSet, and now your audit trail is wrong in a way nobody will notice until the incident review. **danger (worse than empty):** A `` row tells you to go dig. A stale row tells you to stop digging. If you take one thing from this article, it's this: **every rollout needs a fresh change-cause**, or none at all. Fixing it is one command. And here's a detail I only learned by reading kubectl's source: change-cause is the one annotation that does **not** need `--overwrite`. ```text $ kubectl annotate deployment/web kubernetes.io/change-cause="bump nginx to 1.27.2" deployment.apps/web annotated $ kubectl rollout history deployment/web deployment.apps/web REVISION CHANGE-CAUSE 1 2 bump nginx to 1.27.1 3 bump nginx to 1.27.2 ``` In `kubectl/pkg/cmd/annotate/annotate.go`, the overwrite check skips that one key with the comment *"change-cause annotation can always be overwritten"*. Any other key still needs the flag, as Step 2 showed. In scripts I pass `--overwrite` anyway, so nobody reading the script has to know about the exception. ## Step 6: Why Not Just Use `--record`? Older tutorials tell you to add `--record` and let kubectl write the cause for you. It still works, sort of: ```text $ kubectl set image deployment/web nginx=nginx:1.27.3 --record # exits with status 0 Flag --record has been deprecated, --record will be removed in the future | deployment.apps/web image updated ``` ```text $ kubectl rollout history deployment/web deployment.apps/web REVISION CHANGE-CAUSE 1 2 bump nginx to 1.27.1 3 bump nginx to 1.27.2 4 kubectl set image deployment/web nginx=nginx:1.27.3 --record=true ``` Two problems. The flag is deprecated, so a future kubectl will drop it. And look at what it recorded: the command. You could already see that it's nginx 1.27.3 from the image. What you can't see is *why*. "Patch CVE in the image" or "roll forward after the 1.27.2 crash loop" is the part worth writing down, and only a human, or a commit message, can supply it. ## Step 7: Roll Back and Watch the History Move **What:** Look inside a revision, then undo to it. **Why:** A good change-cause pays off right here, when you are choosing what to roll back to. `--revision` shows a single revision, change-cause included, without touching anything: ```text $ kubectl rollout history deployment/web --revision=2 deployment.apps/web with revision #2 Pod Template: Labels: app=web pod-template-hash=69ccdfbf95 Annotations: kubernetes.io/change-cause: bump nginx to 1.27.1 Containers: nginx: Image: nginx:1.27.1 Port: Host Port: Environment: Mounts: Volumes: Node-Selectors: Tolerations: ``` Now roll back to it: ```text $ kubectl rollout undo deployment/web --to-revision=2 deployment.apps/web rolled back $ kubectl rollout history deployment/web deployment.apps/web REVISION CHANGE-CAUSE 1 3 bump nginx to 1.27.2 4 kubectl set image deployment/web nginx=nginx:1.27.3 --record=true 5 bump nginx to 1.27.1 $ kubectl annotate deployment/web --list deployment.kubernetes.io/revision=5 kubernetes.io/change-cause=bump nginx to 1.27.1 ``` Revision 2 disappeared and came back as revision 5. A rollback does not create a new ReplicaSet; it reuses the old one and gives it the next revision number. The change-cause came with it and was copied back onto the Deployment. That's handy, but it also means the next rollout will inherit "bump nginx to 1.27.1" unless you set a new cause. Same trap as Step 5, different door. **Question:** After the rollback, what will revision 6 say if you run kubectl set image without annotating? ## Step 8: Move kubectl annotate Into Git Typing annotations by hand works until the day you're tired. The fix is to stop relying on memory. Let's put the Deployment in a git repo, where every change already has a reason attached: the commit message. Here's the starting manifest, committed as-is: **k8s/checkout.yaml (starting point)** ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: checkout labels: app: checkout spec: replicas: 2 selector: matchLabels: app: checkout template: metadata: labels: app: checkout spec: containers: - name: checkout image: ghcr.io/acme/checkout:1.4.0 ports: - containerPort: 8080 ``` ```text $ kubectl apply -f k8s/checkout.yaml deployment.apps/checkout created $ kubectl rollout history deployment/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 ``` Now the next release. We bump the image and, in the same edit, add annotations to the manifest: the change-cause, plus an owner and a runbook link for whoever gets paged. ```diff # k8s/checkout.yaml diff --git a/k8s/checkout.yaml b/k8s/checkout.yaml index d9eba05..72d5800 100644 --- a/k8s/checkout.yaml +++ b/k8s/checkout.yaml @@ -4,6 +4,10 @@ metadata: name: checkout labels: app: checkout + annotations: + kubernetes.io/change-cause: "checkout 1.5.0: retry card declines once" + team.acme.io/owner: payments + team.acme.io/runbook: https://wiki.acme.io/runbooks/checkout spec: replicas: 2 selector: @@ -16,6 +20,6 @@ spec: spec: containers: - name: checkout - image: ghcr.io/acme/checkout:1.4.0 + image: ghcr.io/acme/checkout:1.5.0 ports: - containerPort: 8080 ``` `git diff` shows what *you* changed. `kubectl diff` shows what will change *in the cluster*, which is not always the same thing: ```diff # cluster vs k8s/checkout.yaml --- /tmp/LIVE-4253123521/apps.v1.Deployment.default.checkout +++ /tmp/MERGED-3209055176/apps.v1.Deployment.default.checkout @@ -5,8 +5,11 @@ deployment.kubernetes.io/revision: "1" kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"checkout"},"name":"checkout","namespace":"default"},"spec":{"replicas":2,"selector":{"matchLabels":{"app":"checkout"}},"template":{"metadata":{"labels":{"app":"checkout"}},"spec":{"containers":[{"image":"ghcr.io/acme/checkout:1.4.0","name":"checkout","ports":[{"containerPort":8080}]}]}}}} + kubernetes.io/change-cause: 'checkout 1.5.0: retry card declines once' + team.acme.io/owner: payments + team.acme.io/runbook: https://wiki.acme.io/runbooks/checkout creationTimestamp: "2026-10-19T19:22:07Z" - generation: 1 + generation: 2 labels: app: checkout name: checkout @@ -31,7 +34,7 @@ app: checkout spec: containers: - - image: ghcr.io/acme/checkout:1.4.0 + - image: ghcr.io/acme/checkout:1.5.0 imagePullPolicy: IfNotPresent name: checkout ports: ``` Two things in there you didn't write. `deployment.kubernetes.io/revision` we met in Step 1. `kubectl.kubernetes.io/last-applied-configuration` is a copy of your whole manifest that `kubectl apply` keeps so it can work out what to delete next time. That's the kind of annotation that eats into the 256 KiB budget from Step 3. Ours is 475 bytes, but on a big CRD it can be a real number. **warning (exit code 1 is good news here):** `kubectl diff` exits with **1 when it finds differences** and 0 when there are none. In CI that trips `set -e` the first time you use it. Treat 1 as "changes found" and anything above 1 as a real error. Apply it, commit it, and the history picks up the cause from the file: ```text $ kubectl apply -f k8s/checkout.yaml deployment.apps/checkout configured $ kubectl rollout history deployment/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 2 checkout 1.5.0: retry card declines once ``` ## Step 9: The Same Trap, in YAML One release later, a teammate bumps the image on their branch and, to test it, applies it straight from there. The diff they will open a pull request with looks perfectly reasonable: ```diff # k8s/checkout.yaml diff --git a/k8s/checkout.yaml b/k8s/checkout.yaml index 72d5800..c6b9e09 100644 --- a/k8s/checkout.yaml +++ b/k8s/checkout.yaml @@ -20,6 +20,6 @@ spec: spec: containers: - name: checkout - image: ghcr.io/acme/checkout:1.5.0 + image: ghcr.io/acme/checkout:1.5.1 ports: - containerPort: 8080 ``` ```text $ kubectl apply -f k8s/checkout.yaml deployment.apps/checkout configured $ kubectl rollout history deployment/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 2 checkout 1.5.0: retry card declines once 3 checkout 1.5.0: retry card declines once ``` No reviewer would catch that, because the line that needed changing is not in the diff at all. Putting the cause in YAML only moved the "remember to update it" problem from the terminal into code review. **Decision:** The change-cause needs to change on **every** rollout. A field in the YAML changes only when somebody remembers to edit it. A commit message exists for every change by definition. **Where should the change-cause come from?** ## Step 10: Write the Change-Cause From the Commit **What:** Take change-cause out of the YAML and let a small deploy script write it from `git log`. **Why:** The commit message is already required, already reviewed, and already describes *why*. Reuse it. First, the change. The cause leaves the manifest, and a script appears that applies and then annotates: ```diff # k8s/checkout.yaml diff --git a/k8s/checkout.yaml b/k8s/checkout.yaml index 72d5800..bd3af1e 100644 --- a/k8s/checkout.yaml +++ b/k8s/checkout.yaml @@ -5,7 +5,6 @@ metadata: labels: app: checkout annotations: - kubernetes.io/change-cause: "checkout 1.5.0: retry card declines once" team.acme.io/owner: payments team.acme.io/runbook: https://wiki.acme.io/runbooks/checkout spec: ``` ```diff # deploy.sh diff --git a/deploy.sh b/deploy.sh new file mode 100755 --- /dev/null +++ b/deploy.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Apply the manifests, then record why this rollout happened. +kubectl apply -f k8s/ + +cause="$(git log -1 --format='%h %s')" +kubectl annotate deployment/checkout \ + kubernetes.io/change-cause="$cause" --overwrite ``` [Download deploy.sh](https://karandeepsingh.ca/downloads/kubectl-annotate/deploy.sh) The `team.acme.io/owner` and `runbook` annotations stay in YAML because they rarely change and belong with the manifest. Only the thing that changes every time moves out. Commit that change on `main`: ```text $ git add -A && git commit -qm "deploy: write change-cause from the commit, not the YAML" ``` Now two ordinary releases on `main`, each with a normal commit. The first redoes the teammate's 1.5.1 bump properly: ```text $ git commit -qam "checkout 1.5.1: fix tax rounding on CAD orders" $ ./deploy.sh deployment.apps/checkout configured deployment.apps/checkout annotated $ git commit -qam "checkout 1.6.0: add Apple Pay" $ ./deploy.sh deployment.apps/checkout configured deployment.apps/checkout annotated $ kubectl rollout history deployment/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 2 checkout 1.5.0: retry card declines once 3 863aabf checkout 1.5.1: fix tax rounding on CAD orders 4 e2d18e9 checkout 1.6.0: add Apple Pay ``` Revision 3 corrected itself, too. The 1.5.1 Pod template was already running from the teammate's branch, so `apply` didn't start a new rollout, and the annotate step overwrote the stale cause on the current revision. Every row now carries a commit SHA you can `git show`: ```text $ git log --oneline e2d18e9 checkout 1.6.0: add Apple Pay 863aabf checkout 1.5.1: fix tax rounding on CAD orders 40e5f04 deploy: write change-cause from the commit, not the YAML 42a48c3 checkout 1.5.0: retry card declines once 6da4575 checkout: initial deployment ``` Here's how the history improved over the course of this walkthrough: - kubectl set image only: 0 of 2 explained - annotating by hand: 1 of 2 right, 1 stale - deploy.sh from git: 2 of 2, with SHAs If `deploy.sh` runs from a self-hosted CI runner, the runner is now part of your audit trail. Check it like any other service. A healthy one looks like this: ```text # systemctl status actions.runner.acme-shop.build-01.service Oct 19 19:24:40 build-01 Runner.Listener[1203]: Running job: deploy-checkout Oct 19 19:24:58 build-01 Runner.Listener[1203]: Job deploy-checkout completed with result: Succeeded ``` ## Step 11: Deployment Annotations vs Pod Template Annotations **What:** Learn which annotations restart Pods and which don't. **Why:** Mix these up and you either restart production by accident, or wonder why nothing restarted. A Deployment has two places for annotations, and they behave completely differently: **metadata.annotations** Notes about the **Deployment** object. `kubectl annotate deployment/checkout ...` writes here. Pods are untouched. No new revision. **spec.template.metadata.annotations** Part of the **Pod template**, copied onto every Pod. Any change here is a template change. New ReplicaSet, new Pods, new revision. You can see both in action. Annotating the Deployment leaves the history alone. `kubectl rollout restart` writes a timestamp into the template, and that is a whole new rollout: ```text $ kubectl annotate deployment/checkout team.acme.io/oncall=@payments-oncall deployment.apps/checkout annotated $ kubectl rollout history deployment/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 2 checkout 1.5.0: retry card declines once 3 863aabf checkout 1.5.1: fix tax rounding on CAD orders 4 e2d18e9 checkout 1.6.0: add Apple Pay $ kubectl rollout restart deployment/checkout deployment.apps/checkout restarted $ kubectl get deployment checkout -o jsonpath='{.spec.template.metadata.annotations}' {"kubectl.kubernetes.io/restartedAt":"2026-10-19T19:23:02Z"} $ kubectl rollout history deployment/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 2 checkout 1.5.0: retry card declines once 3 863aabf checkout 1.5.1: fix tax rounding on CAD orders 4 e2d18e9 checkout 1.6.0: add Apple Pay 5 e2d18e9 checkout 1.6.0: add Apple Pay ``` And there's the stale-cause trap one last time: a restart is a rollout, so revision 5 inherited the Apple Pay message. If you restart on purpose, annotate on purpose: `kubectl annotate deployment/checkout kubernetes.io/change-cause="restart: pick up rotated DB secret"`. `kubectl describe` shows both sets of annotations together, which is the fastest way to tell them apart. This one was captured at the end of the session, so it also carries the cost-center and on-call annotations from Step 12. Click into the pager and use `j`/`k` or the space bar to scroll: ```text # kubectl describe deployment checkout Name: checkout Namespace: default CreationTimestamp: Mon, 19 Oct 2026 19:22:07 +0000 Labels: app=checkout team=payments Annotations: deployment.kubernetes.io/revision: 5 kubernetes.io/change-cause: e2d18e9 checkout 1.6.0: add Apple Pay team.acme.io/cost-center: cc-4410 team.acme.io/oncall: @payments-oncall-2 team.acme.io/owner: payments team.acme.io/runbook: https://wiki.acme.io/runbooks/checkout Selector: app=checkout Replicas: 2 desired | 1 updated | 3 total | 0 available | 3 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=checkout Annotations: kubectl.kubernetes.io/restartedAt: 2026-10-19T19:23:02Z Containers: checkout: Image: ghcr.io/acme/checkout:1.6.0 Port: 8080/TCP Host Port: 0/TCP Environment: Mounts: Volumes: Node-Selectors: Tolerations: Conditions: Type Status Reason ---- ------ ------ Available False MinimumReplicasUnavailable Progressing True ReplicaSetUpdated OldReplicaSets: checkout-6f59cbd558 (0/0 replicas created), checkout-66984f96b7 (0/0 replicas created), checkout-7b77b8ffd (1/1 replicas created), checkout-5f7f9f7bcb (1/1 replicas created) NewReplicaSet: checkout-8677bd445c (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 81s deployment-controller Scaled up replica set checkout-6f59cbd558 from 0 to 2 Normal ScalingReplicaSet 62s deployment-controller Scaled up replica set checkout-66984f96b7 from 0 to 1 Normal ScalingReplicaSet 60s deployment-controller Scaled down replica set checkout-6f59cbd558 from 2 to 1 Normal ScalingReplicaSet 60s deployment-controller Scaled up replica set checkout-7b77b8ffd from 0 to 1 Normal ScalingReplicaSet 41s deployment-controller Scaled down replica set checkout-6f59cbd558 from 1 to 0 Normal ScalingReplicaSet 41s deployment-controller Scaled up replica set checkout-5f7f9f7bcb from 0 to 1 Normal ScalingReplicaSet 26s deployment-controller Scaled down replica set checkout-66984f96b7 from 1 to 0 Normal ScalingReplicaSet 26s deployment-controller Scaled up replica set checkout-8677bd445c from 0 to 1 ``` The Pods in this test cluster never become available because it has no nodes. As promised at the top, the annotations don't care. ## Step 12: Annotate Many Objects Safely **What:** Annotate by label selector, preview with dry run, and guard against races. **Why:** Once you trust annotations, you'll want them on everything a team owns. That's where one command can touch a lot of objects. A label selector picks the objects, and the annotation goes on all of them: ```text $ kubectl annotate deployments -l team=payments team.acme.io/cost-center=cc-4410 deployment.apps/cart annotated deployment.apps/checkout annotated ``` `--all` takes every object of that type in the namespace. Always preview it first with `--dry-run=client`, which prints what would happen and sends nothing: ```text $ kubectl annotate deployments --all team.acme.io/reviewed=2026-10 --dry-run=client deployment.apps/cart annotated (dry run) deployment.apps/checkout annotated (dry run) deployment.apps/web annotated (dry run) ``` **Decision:** `kubectl annotate deployments --all KEY=VALUE --overwrite` replaces that key on **every Deployment in the namespace**, including ones another team or tool owns. There is no undo. The old values are gone. **Run it with `--dry-run=client` first?** In CI you often want to change a manifest file without touching a cluster at all. `--local` does exactly that, reading from `-f` and printing the result: ```text $ kubectl annotate -f k8s/checkout.yaml team.acme.io/owner=platform --local --overwrite -o yaml | head -12 apiVersion: apps/v1 kind: Deployment metadata: annotations: team.acme.io/owner: platform team.acme.io/runbook: https://wiki.acme.io/runbooks/checkout labels: app: checkout name: checkout spec: replicas: 2 selector: ``` And three mistakes worth recognizing on sight. A race, where someone else changed the object after you read it, caught by `--resource-version`: ```text $ kubectl annotate deployment/checkout team.acme.io/oncall=@sre --overwrite --resource-version=400 # exits with status 1 Error from server (Conflict): Operation cannot be fulfilled on deployments.apps "checkout": the object has been modified; please apply your changes to the latest version and try again ``` A key with a space in it, rejected by the API server: ```text $ kubectl annotate deployment/checkout 'Team Owner=payments' # exits with status 1 The Deployment "checkout" is invalid: metadata.annotations: Invalid value: "Team Owner": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]') ``` And a key with no `=value` and no trailing `-`, which kubectl can't read as an add or a remove: ```text $ kubectl annotate deployment/checkout team.acme.io/oncall # exits with status 1 error: at least one annotation update is required ``` **info (use a prefix you own):** Keys like `team.acme.io/owner` use a DNS-style prefix your organization controls. Unprefixed keys like `team` work, but they collide with every other tool that had the same idea. The `kubernetes.io/` and `k8s.io/` prefixes are reserved for Kubernetes itself, and `kubernetes.io/change-cause` is one of those that kubectl reads on purpose. ## kubectl annotate Reference The parts of `kubectl annotate --help` that matter for everything above, from kubectl 1.34.1: **man page: kubectl-annotate** ## Name kubectl annotate: update the annotations on one or more resources ## Synopsis **kubectl annotate** \[`--overwrite`\] (`-f FILENAME` | `TYPE NAME`) `KEY_1=VAL_1 ... KEY_N=VAL_N` \[`--resource-version=version`\] \[*options*\] ## Description All Kubernetes objects support the ability to store additional data with the object as annotations. Annotations are key/value pairs that can be larger than labels and include arbitrary string values such as structured JSON. Attempting to set an annotation that already exists will fail unless `--overwrite` is set. If `--resource-version` is specified and does not match the current resource version on the server the command will fail. ## Options - `--overwrite`: allow annotations to be overwritten, otherwise reject updates that overwrite existing annotations - `--list`: display the annotations for a given resource - `-l, --selector`: label query to filter on - `--all`: select all resources, in the namespace of the specified resource types - `--dry-run`: must be `none`, `server`, or `client` - `--local`: do NOT contact the api-server, run locally - `--resource-version`: only succeed if this is the current resource-version for the object; valid for a single resource only ## Examples `kubectl annotate pods foo description='my frontend'` `kubectl annotate --overwrite pods foo description='my frontend running nginx'` `kubectl annotate pods foo description-` ## See also `kubectl-label(1)`, `kubectl-rollout(1)`, `kubectl-diff(1)` ## kubectl annotate Cheat Sheet | Task | Command | |------|---------| | Add an annotation | `kubectl annotate deploy/web team=payments` | | List annotations | `kubectl annotate deploy/web --list` | | Change a value | `kubectl annotate deploy/web team=checkout --overwrite` | | Remove an annotation | `kubectl annotate deploy/web team-` | | Record why you deployed | `kubectl annotate deploy/web kubernetes.io/change-cause="why"` | | See the reasons | `kubectl rollout history deploy/web` | | Inspect one revision | `kubectl rollout history deploy/web --revision=2` | | Annotate by label | `kubectl annotate deploy -l team=payments key=value` | | Preview a bulk change | `kubectl annotate deploy --all key=value --dry-run=client` | | Edit a file, no cluster | `kubectl annotate -f app.yaml key=value --local -o yaml` | | Guard against races | `kubectl annotate deploy/web key=value --resource-version=N` | | Cause from git | `kubectl annotate deploy/web kubernetes.io/change-cause="$(git log -1 --format='%h %s')" --overwrite` | ## What We Built We started with one `kubectl annotate` command and kept adding. Along the way: - `--overwrite` guards every key except `kubernetes.io/change-cause`, and `key-` removes one. - Annotations describe, labels select, and the whole object gets 256 KiB. - CHANGE-CAUSE is just an annotation, copied from the Deployment to each new ReplicaSet. - That copy is why a forgotten annotation turns into a **stale** reason, after a `set image`, an `apply`, a rollback, or a restart. - `--record` still works, but it's deprecated and only records the command, not the reason. - `git diff` shows what you changed, and `kubectl diff` shows what the cluster will change, exit code 1 included. - A five-line `deploy.sh` that writes the cause from `git log` keeps rollout history honest without anyone having to remember. Next time somebody runs `kubectl rollout history` at 2 a.m., every row will say what changed, why, and which commit to read. That's the whole point of kubectl annotate and change-cause: the reason travels with the rollout. ## References and Further Reading - Kubernetes Documentation. (2026). [Annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). The Kubernetes Authors. - Kubernetes Documentation. (2026). [Deployments: Checking Rollout History of a Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#checking-rollout-history-of-a-deployment). The Kubernetes Authors. - Kubernetes Documentation. (2026). [kubectl annotate](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_annotate/). The Kubernetes Authors. - Kubernetes Documentation. (2026). [Well-Known Labels, Annotations and Taints](https://kubernetes.io/docs/reference/labels-annotations-taints/). The Kubernetes Authors. - Kubernetes Project. (2026). [kubectl annotate source (annotate.go)](https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/annotate/annotate.go). GitHub. ## Quiz 1. You run `kubectl annotate deployment web team=checkout` and the annotation `team` already exists. What happens? a) The value is replaced silently b) The command fails until you add `--overwrite` c) kubectl adds a second `team` annotation d) The Deployment starts a new rollout Answer: b. kubectl refuses to replace an existing annotation unless you pass `--overwrite`. The error reads "--overwrite is false but found the following declared annotation(s): 'team' already has a value". The one exception is `kubernetes.io/change-cause`, which kubectl lets you replace without the flag. 2. You roll out a new image and forget to update `kubernetes.io/change-cause`. What does `kubectl rollout history` show for the new revision? a) ``, because nothing was annotated b) The previous revision's change-cause, copied forward c) The kubectl command you ran d) An error until you annotate it Answer: b. The Deployment keeps its change-cause annotation, and the controller copies the Deployment's annotations onto every new ReplicaSet. So the new revision quietly inherits the old reason, which is worse than `` because it looks correct. 3. Which change makes a Deployment roll out new Pods? a) `kubectl annotate deployment web owner=payments` b) Adding an annotation under `spec.template.metadata.annotations` c) `kubectl annotate deployment web owner-` d) `kubectl annotate deployment web --list` Answer: b. Annotations on the Deployment's own metadata are notes about the object and do not touch the Pod template. An annotation inside `spec.template` changes the template, so the controller creates a new ReplicaSet. That is exactly how `kubectl rollout restart` works; it writes `kubectl.kubernetes.io/restartedAt` into the template.