Diff stage

Overview ¶

kapp compares resources specified in files against resources that exist in Kubernetes API. Once change set is calculated, it provides an option to apply it (see Apply section for further details).

There are five different types of operations: create, update, delete, noop (shown as empty), exists (added in v0.43.0). Seen in Op column of diff summary table. Additionally there is Op strategy column (shorted as Op st.), added in v0.31.0+, that shows supplemental information how operation will be performed (for example fallback on replace for update operation).

There are three different types of waiting: reconcile (waits until resource has converged to its desired state; see apply waiting for waiting semantics), delete (waits until resource is gone), noop (shown as empty). Seen in Wait to column of diff summary table.

Diff strategies ¶

There are two diff strategies used by kapp:

1. kapp compares against last applied resource content (previously applied by kapp; stored in annotation kapp.k14s.io/original) if there were no outside changes done to the resource (i.e. done outside of kapp, for example, by another team member or controller); kapp tries to use this strategy as much as possible to produce more user-friendly diffs.

2. kapp compares against live resource content if it detects there were outside changes to the resource (hence, sometimes you may see a diff that shows several deleted fields even though these fields are not specified in the original file)

Strategy is selected for each resource individually. You can control which strategy is used for all resources via --diff-against-last-applied=bool flag.

Related: rebase rules.

Versioned Resources ¶

In some cases it’s useful to represent an update to a resource as an entirely new resource. Common example is a workflow to update ConfigMap referenced by a Deployment. Deployments do not restart their Pods when ConfigMap changes making it tricky for wide variety of applications for pick up ConfigMap changes. kapp provides a solution for such scenarios, by offering a way to create uniquely named resources based on an original resource.

Anytime there is a change to a resource marked as a versioned resource, entirely new resource will be created instead of updating an existing resource.

To make resource versioned, add kapp.k14s.io/versioned annotation with an empty value. Created resource follow {resource-name}-ver-{n} naming pattern by incrementing n any time there is a change.

Example:

---
apiVersion: v1
kind: Secret
metadata:
  name: secret-sa-sample
  annotations:
    kapp.k14s.io/versioned: ""

This will create versioned resource named secret-sa-sample-ver-1

Namespace  Name                    Kind    Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
default    secret-sa-sample-ver-1  Secret  -       -    create  -       reconcile  -   -  

Op:      1 create, 0 delete, 0 update, 0 noop, 0 exists
Wait to: 1 reconcile, 0 delete, 0 noop

Additionally kapp follows configuration rules (default ones, and ones that can be provided as part of application) to find and update object references (since new resource name is not something that configuration author knew about).

apiVersion: kapp.k14s.io/v1alpha1
kind: Config
templateRules:
  - resourceMatchers:
      - apiVersionKindMatcher: {apiVersion: v1, kind: ConfigMap}
    affectedResources:
      objectReferences:
        - resourceMatchers:
            - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment}
          path: [spec, template, spec, containers, {allIndexes: true}, env, {allIndexes: true}, valueFrom, configMapKeyRef]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  annotations:
    kapp.k14s.io/versioned: ""
data:
  special.how: very-good
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
          env:
            - name: SPECIAL_LEVEL_KEY
              valueFrom:
                configMapKeyRef:
                  name: special-config
                  key: special.how

As of v0.38.0+, kapp.k14s.io/versioned-keep-original annotation can be used in conjunction with kapp.k14s.io/versioned to have the original resource (resource without -ver-{n} suffix in name) along with versioned resource.

Example:

---
apiVersion: v1
kind: Secret
metadata:
  name: secret-sa-sample
  annotations:
    kapp.k14s.io/versioned: ""
    kapp.k14s.io/versioned-keep-original: ""

This will create two resources one with original name secret-sa-sample and one with -ver-{n} suffix in name secret-sa-sample-ver-1.

Namespace  Name                    Kind    Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
default    secret-sa-sample        Secret  -       -    create  -       reconcile  -   -  
^          secret-sa-sample-ver-1  Secret  -       -    create  -       reconcile  -   -  

Op:      2 create, 0 delete, 0 update, 0 noop, 0 exists
Wait to: 2 reconcile, 0 delete, 0 noop

You can control number of kept resource versions via kapp.k14s.io/num-versions=str(int) annotation, e.g. kapp.k14s.io/num-versions: "5".

As of v0.41.0+, the kapp.k14s.io/versioned-explicit-ref can be used to explicitly refer to a versioned resource. This annotation allows a resource to be updated whenever a new version of the referred resource is created.

Multiple annotations with the prefix kapp.k14s.io/versioned-explicit-ref.(Note the “.” at the end) can be used to define multiple explicit references.

Example:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-1
  annotations:
    kapp.k14s.io/versioned: ""
data:
  foo: bar
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-2
  annotations:
    kapp.k14s.io/versioned-explicit-ref: |
      apiVersion: v1
      kind: ConfigMap
      name: config-1      
data:
  foo: bar

Here, config-2 explicitly refers config-1 and is updated with the latest versioned name when config-1 is versioned.

@@ create configmap/config-1-ver-2 (v1) namespace: default @@
  ...
  1,  1   data:
  2     -   foo: bar
      2 +   foo: alpha
  3,  3   kind: ConfigMap
  4,  4   metadata:
@@ update configmap/config-2 (v1) namespace: default @@
  ...
  8,  8         kind: ConfigMap
  9     -       name: config-1-ver-1
      9 +       name: config-1-ver-2
 10, 10     creationTimestamp: "2021-09-29T17:27:34Z"
 11, 11     labels:

Changes

Namespace  Name            Kind       Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
default    config-1-ver-2  ConfigMap  -       -    create  -       reconcile  -   -  
^          config-2        ConfigMap  -       14s  update  -       reconcile  ok  -  

Try deploying redis-with-configmap example and changing ConfigMap in a next deploy.

Controlling diff via resource annotations ¶

kapp.k14s.io/disable-original ¶

kapp, by default, records the resource copy into its annotation kapp.k14s.io/original while applying the resource to the cluster.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-1
  namespace: default
data:
  foo: bar

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-1
  namespace: default
  annotations:
    kapp.k14s.io/original: '{ "apiVersion": "v1", "kind": "ConfigMap", ...snip... }'
data:
  foo: bar

kapp.k14s.io/disable-original annotation controls whether to record provided resource copy (rarely wanted)

Possible values: "" (empty). In some cases it’s not possible or wanted to record applied resource copy into its annotation kapp.k14s.io/original. One such case might be when resource is extremely lengthy (e.g. long ConfigMap or CustomResourceDefinition) and will exceed annotation value max length of 262144 bytes.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-1
  namespace: default
  annotations:
    kapp.k14s.io/disable-original: ""
data:
  foo: bar

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-1
  namespace: default
  annotations:
    kapp.k14s.io/disable-original: ""
data:
  foo: bar

Controlling diff via deploy flags ¶

Diff summary shows quick information about what’s being changed:

• --diff-summary=bool (default true) shows diff summary, listing how resources have changed
  Example

  Sample config

  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: sample
  stringData:
    foo: bar
  

  $ kapp deploy -a sample-secret -f config.yaml --diff-summary=true
  Target cluster 'https://127.0.0.1:56540' (nodes: kind-control-plane)
  
  Changes
  
  Namespace  Name    Kind    Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
  default    sample  Secret  -       -    create  -       reconcile  -   -  
  
  Op:      1 create, 0 delete, 0 update, 0 noop, 0 exists
  Wait to: 1 reconcile, 0 delete, 0 noop
  
  Continue? [yN]: 
  

Diff changes (line-by-line diffs) are useful for looking at actual changes, when app is re-deployed:

• --diff-changes=bool (-c) (default false) shows line-by-line diffs
• --diff-context=int (default 2) controls number of lines to show around changed lines
• --diff-mask=bool (default true) controls whether to mask sensitive fields
  Example

  Sample config

  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: sample
  stringData:
    foo: bar
  

  # deploy sample-secre app
  $ kapp deploy -a sample-secret -f config.yaml
  
  #update config
  ...
  stringData:
    foo: bars
  ...
  
  # re-deploy sample-secret app with required diff-changes flag to see line by line changes 
  $ kapp deploy -a sample-secret -f config.yaml --diff-changes=true --diff-context=4
  Target cluster 'https://127.0.0.1:56540' (nodes: kind-control-plane)
  
  @@ update secret/sample (v1) namespace: default @@
    ...
   30, 30     resourceVersion: "244751"
   31, 31     uid: b2453c2a-8dc8-4ed1-9b59-791547f78ea8
   32, 32   stringData:
   33     -   foo: <-- value not shown (#1)
       33 +   foo: <-- value not shown (#2)
   34, 34   
  
  Changes
  
  Namespace  Name    Kind    Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
  default    sample  Secret  -       7m   update  -       reconcile  ok  -  
  
  Op:      0 create, 0 delete, 1 update, 0 noop, 0 exists
  Wait to: 1 reconcile, 0 delete, 0 noop
  
  Continue? [yN]: 
  
  # --diff-mask=true by default, note the masked value for secret data
  
  # try out kapp deploy -a sample-secret -f config.yaml --diff-mask=false --diff-changes=true --diff-context=2
  

Controlling how diffing is done:

• --diff-against-last-applied=bool (default true) forces kapp to use particular diffing strategy (see above).
• --diff-run=bool (default false) set the flag to true, to stop after showing diff information.
• --diff-exit-status=bool (default false) controls exit status for diff runs (0: unused, 1: any error, 2: no changes, 3: pending changes)
  Example

  Sample config

  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: sample
  stringData:
    foo: bar
  

  # deploy secret-sample app
  $ kapp deploy -a secret-sample -f config.yaml  --diff-run=true --diff-exit-status=true
  Target cluster 'https://127.0.0.1:56540' (nodes: kind-control-plane)
  
  Changes
  
  Namespace  Name    Kind    Conds.  Age  Op      Op st.  Wait to    Rs  Ri  
  default    sample  Secret  -       -    create  -       reconcile  -   -
  
  Op:      1 create, 0 delete, 0 update, 0 noop, 0 exists
  Wait to: 1 reconcile, 0 delete, 0 noop
  
  kapp: Error: Exiting after diffing with pending changes (exit status 3)
  
  # note that kapp exits after diff and displays the exit status
  

Diff filter allows to filter changes based on operation (add/update/delete), newResource (configuration provided to kapp) and existingResource (resources in Kubernetes cluster)

• --diff-filter='{"and":[{"ops":["update"]},{"existingResource":{"kinds":["Deployment"]}]}' will keep the resources which are getting updated and were of kind Deployment.