Carvel Logo

Overlay module

Overview

ytt’s Overlay feature provides a way to combine YAML structures together with the help of annotations.

There are two (2) structures involved in an overlay operation:

  • the “left” — the YAML document(s) (and/or contained maps and arrays) being modified, and
  • the “right” — the YAML document (and/or contained maps and arrays) that is the overlay, describing the modification.

Each modification is composed of:

  • a matcher (via an @overlay/(match) annotation), identifying which node(s) on the “left” are the target(s) of the edit, and
  • an action (via an @overlay/(action) annotation), describing the edit.

Once written, an overlay can be applied in one of two ways:

Programmatic overlays are applied immediately (while the contained template is being evaluated). Declarative overlays are applied in the “Apply Overlays” step as described in How it works.

(For a step-by-step primer on writing and using overlays, watch Primer on ytt Overlays.)

Overlays as files

As ytt scans input files, it pulls aside any YAML Document that is annotated with @overlay/match, and considers it an overlay.

After YAML templates are rendered, the collection of identified overlays are applied. Each overlay executes, one-at-a-time over the entire set of the rendered YAML documents.

Order matters: modifications from earlier overlays are seen by later overlays. Overlays are applied in the order detailed in Overlay order, below.

In the example below, the last YAML document is an overlay (it has the @overlay/match annotation). That overlay matcher’s selects the first YAML document only: it’s the only one that has a metadata.name of example-ingress.

#@ load("@ytt:overlay", "overlay")

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: another-example-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /

#@overlay/match by=overlay.subset({"metadata":{"name":"example-ingress"}})
---
metadata:
  annotations:
    #@overlay/remove
    ingress.kubernetes.io/rewrite-target:

yields:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  annotations: {}
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: another-example-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /

See also: Overlay files example in online playground.

(For a high-level overview of ytt, see How it works.)

Overlay order

(as of v0.13.0)

Overlays are applied, in sequence, by:

  1. left-to-right for file flags
    • e.g. in -f overlay1.yml -f overlay2.yml, overlay1.yml will be applied first
  2. if file flag is set to a directory, files are alphanumerically sorted
    • e.g. in aaa/z.yml xxx/c.yml d.yml, will be applied in following order aaa/z.yml d.yml xxx/c.yml
  3. top-to-bottom order for overlay YAML documents within a single file

Next Steps

Familiarize yourself with the overlay annotations.

Programmatic access

Overlays need not apply to the entire set of rendered YAML documents (as is the case with the declarative approach).

Instead, the declared modifications can be captured in a function and applied to a specific set of documents via overlay.apply() in Starlark code.

In this example we have left() function that returns target structure and right() that returns the overlay, specifying the modifications.

overlay.apply(...) will execute the execute the overlay and return a new structure.

#@ load("@ytt:overlay", "overlay")

#@ def left():
key1: val1
key2:
  key3:
    key4: val4
  key5:
  - name: item1
    key6: val6
  - name: item2
    key7: val7
#@ end

#@ def right():
#@overlay/remove
key1: val1
key2:
  key3:
    key4: val4
  key5:
  #@overlay/match by="name"
  - name: item2
    #@overlay/match missing_ok=True
    key8: new-val8
#@ end

result: #@ overlay.apply(left(), right())

yields:

result:
  key2:
    key3:
      key4: val4
    key5:
    - name: item1
      key6: val6
    - name: item2
      key7: val7
      key8: new-val8

Next Steps

Familiarize yourself with the two kinds of overlay annotations:

@overlay Annotations

There are two groups of overlay annotations:

Matching Annotations

These annotations are used to select which structure(s) will be modified:

@overlay/match

Specifies which nodes on the “left” to modify.

Valid on: Document, Map Item, Array Item.

@overlay/match [by=Function|String, expects=Int|String|List|Function, missing_ok=Bool, when=Int|String|List]
  • by=Function|String — criteria for matching nodes on the “left”
  • expects=Int|String|List|Function — (optional) expected number of nodes to be found in the “left.” If not satisfied, raises an error.
    • Int — must match this number, exactly
    • String (e.g. "1+") — must match at least the number
    • Function(found):Bool — predicate of whether the expected number of matches were found
      • found (Int) — number of actual matches
    • List[Int|String|Function] — must match one of the given criteria
    • Default: 1 (Int) (i.e. expecting to match exactly one (1) node on the “left”).
  • missing_ok=Bool (optional) shorthand syntax for expects=[0,1]
  • when=Int|String|List (optional) criteria for when the overlay should apply. If the criteria is met, the overlay applies; otherwise, nothing happens.
    • Int — must equal this number, exactly
    • String (e.g. "1+") — must match at least the number
    • List[Int|String] — must match one of the given criteria

Notes:

  • expects, missing_ok, and when are mutually-exclusive parameters.
  • take care when expects includes zero (0); matching none is indistinguishable from a mistakenly written match (e.g. a misspelling of a key name)

Examples:

  • #@overlay/match by="id": expects to find one (1) node on the “left” that has the key id and value matching the same-named item on the “right.”
  • #@overlay/match by=overlay.map_key("name"): (same as above)
  • #@overlay/match by=overlay.all,expects="0+": has no effective matching expectations
  • #@overlay/match missing_ok=True: expects to find 0 or 1 matching nodes on the “left”
  • #@overlay/match expects=2: expects to find exactly two (2) matching nodes on the “left”
  • #@overlay/match expects="2+": expects to find two (2) or more matching nodes on the “left”
  • #@overlay/match expects=[0,1,4]: expects to find 0, 1 or 4 matching nodes on the “left”
  • #@overlay/match expects=lambda x: return x < 10: expects 9 or fewer matching nodes on the “left”
  • #@overlay/match when=2: applies changes only if two (2) nodes are found on the “left” and will not error otherwise
  • #@overlay/match when=2+: applies changes only if two (2) or more nodes are found on the “left” and will not error otherwise
  • #@overlay/match when=[0,1,4]: applies changes only if there were exactly 0, 1 or 4 nodes found on the “left” and will not error otherwise

History:

  • v0.28.0+ — added when keyword argument.

Custom Overlay Matcher Functions

The matcher functions from @ytt:overlay cover many use-cases. From time-to-time, more precise matching is required.

A matcher function has the following signature:

Function(indexOrKey,left,right):Boolean

  • indexOrKey
    • when left is an array item: (Int) — the potential match’s zero-based index in the list that is the left-side
    • when left is a map item: (any) — the potential match’s key in the map that is the left-side
  • left (yamlfragment or scalar) — the potential match/target node
  • right (yamlfragment or scalar) — the value of the annotated node in the overlay
  • returns True if left should be considered a match; False otherwise.

Most custom matchers can be written as a Lambda expression.

Lambda expressions start with the keyword lambda, followed by a parameter list, then a :, and a single expression that is the body of the function.

Examples:

Example 1: Key presence or partial string match

left contains right:

lambda indexOrKey, left, right: right in left

Returns True when right is “a member of” left

(see also: Starlark Spec: Membership tests for more details)

Example 2: Precise string matching

left contains a key of the same name as the value of right:

lambda indexOrKey, left, right: left["metadata"]["name"].endswith("server")

See also:

Example 3: Match on both key and value

Is the map item with key/value of: type: LoadBalancer.

lambda indexOrKey, left, right: indexOrKey == "type" and left == "LoadBalancer"

@overlay/match-child-defaults

Sets default values for expects, missing_ok, or when for the children of the annotated node. Does not set these values for the annotated node, itself.

Commonly used to avoid repeating @overlay/match missing_ok=True on each child node in maps.

Valid on: Document, Map Item, Array Item.

@overlay/match-child-defaults [expects=Int|String|List|Function, missing_ok=Bool, when=Int|String|List]

(see @overlay/match for parameter specifications.)

Examples:

Without the #@overlay/match-child-defaults, each of the four new annotation would have needed an @overlay/match missing_ok=True to apply successfully:

---
metadata:
  annotations:
    ingress.kubernetes.io/rewrite-target: /

#@overlay/match by=overlay.all
---
metadata:
  #@overlay/match-child-defaults missing_ok=True
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: 2000
    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/client-body-buffer-size: 1M

Action Annotations

The following annotations describe how to modify the matched “left” node.

They are:

@overlay/merge

Merge the value of “right” node with the corresponding “left” node.

Valid on: Map Item, Array Item.

@overlay/merge

(this annotation has no parameters.)

Note: This is the default action; for each node in an overlay, either the action is explicitly specified or it is merge.

@overlay/remove

Deletes the matched “left” node.

Valid on: Document, Map Item, Array Item.

@overlay/remove

(this annotation has no parameters.)

@overlay/replace

Substitutes matched “left” node with the value of the “right” node (or by that of a provided function).

Valid on: Document, Map Item, Array Item.

@overlay/replace [or_add=Bool, via=Function]
  • or_addBool (optional) determines what should be done in case matched node is not found
    • or_add=True indicates that node should be added. Can be used in combination with @overlay/match missing_ok=True.
  • via=Function(left, right): (any) (optional) determines the value to substitute in. If omitted, the value is right.
    • left (yamlfragment or scalar) — the matched node’s value
    • right (yamlfragment or scalar) — the value of the annotated node

History:

Examples:

Example 1: Use value from “right”

Replaces the corresponding “left” with the value "v1"

#@overlay/replace
apiVersion: v1

Example 2: Edit string value

#@overlay/replace via=lambda left, right: "prefix-"+left

See also:

@overlay/insert

Inserts “right” node before/after the matched “left” node. The inserted node is either the “right” node or that provided by a function.

Valid on: Document, Array Item.

@overlay/insert [before=Bool, after=Bool]
  • before=Bool whether to insert the “right” node immediately in front of the matched “left” node.
  • after=Bool whether to insert the “right” node immediately following the matched “left” node.
  • via=Function(left, right): (any) (optional) determines the value to substitute in. If omitted, the value is right.
    • left (yamlfragment or scalar) — the matched node’s value
    • right (yamlfragment or scalar) — the value of the annotated node

Examples:

Add a ConfigMap into each Namespace:

#@ def configMap(namespace):
apiVersion: v1
kind: ConfigMap
metadata:
  name: insert
  namespace: #@ namespace.metadata.name
#@ end

#@overlay/match overlay.subset({"kind": "Namespace"})
#@overlay/insert after=True, via=lambda namespace, _: configMap(namespace)
---

@overlay/append

Inserts the “right” node after the last “left” node.

Valid on: Document, Array Item.

@overlay/append

(this annotation has no parameters.)

Note: This action implies an @overlay/match selecting the last node. Any other @overlay/match annotation is ignored.

History:

  • v0.32.0 — omitting @overlay/append annotation implies a merge operation and 0 matches, defaulting to insert after the last item.

@overlay/assert

Checks assertion that value of “left” matched node equals that of the annotated “right” node (or a provided predicate).

Valid on: Document, Map Item, Array Item.

@overlay/assert [via=Function]
  • Default: checks that the value of the matched “left” node equals the value of the annotated “right” node.
  • via=Function(left, right):(Bool|Tuple(Bool|String)|None) (optional) predicate indicating whether “left” passes the check
    • left (yamlfragment or scalar) — the matched node’s value
    • right (yamlfragment or scalar) — the value of the annotated node
    • Return types:
      • Bool — if False, the assertion fails; otherwise, nothing happens.
      • Tuple(Bool|String) — if False, the assertion fails and specified string is appended to the resulting error message; otherwise nothing happens.
      • None — the assertion assumes to succeed. In these situations, the function makes use of the @ytt:assert module to effect the assertion.

History:

  • v0.26.0 — works with yamlfragment values.
  • v0.24.0 — introduced

Examples:

Example 1: Range check

Fails the execution if left not between 0 and 1000, exclusively.

#@overlay/assert via=lambda left, right: left > 0 and left < 1000

Example 2: Well-formedness check

Fails the execution if left contains anything other than lowercase letters or numbers.

#@overlay/assert via=lambda left, right: regexp.match("[a-z0-9]+", left)

See also:

Functions

The @ytt:overlay module provides several functions that support overlay use.

To use these functions, include the @ytt:overlay module:

#@ load("@ytt:overlay", "overlay")

The functions exported by this module are:

overlay.apply()

Executes the supplied overlays on top of the given structure.

overlay.apply(left, right1[, rightN...])

Notes:

Examples:

overlay.apply(left(), right())
overlay.apply(left(), one(), two())

See also: Overlay example in the ytt Playground.

overlay.map_key()

An Overlay matcher function that matches when the collection (i.e. Map or Array) in the “left” contains a map item with the key of name and value equal to the corresponding map item from the “right.”

overlay.map_key(name)
  • name (String) — the key of the contained map item on which to match

Note: this matcher requires that all items in the target collection have a map item with the key name; if this requirement cannot be guaranteed, consider using overlay.subset(), instead.

Examples:

Example 1: Over an Array

With “left” similar to:

clients:
- id: 1
- id: 2

the following matches on the second array item:

clients:
#@overlay/match by=overlay.map_key("id")
- id: 2

Example 2: Over a Map

(as of v0.26.0+)

With “left” similar to:

clients:
  clientA:
    id: 1
  clientB:
    id: 2

the following matches on the second map item:

---
clients:
  #@overlay/match by=overlay.map_key("id")
  _:
    id: 2

(note: the key name _ is arbitrary and ignored).

overlay.index()

An Overlay matcher function that matches the array item at the given index

overlay.index(i)
  • i (Int) — the ordinal of the item in the array on the “left” to match (zero-based index)

Example:

#@overlay/match by=overlay.index(0)
- item10

overlay.all()

An Overlay matcher function that matches all contained nodes from the “left”, unconditionally.

overlay.all()

(this function has no parameters.)

Examples:

Example 1: Documents

Matches each and every document:

#@overlay/match by=overlay.all
---
metadata:
 annotations: ...

Example 2: Array Items

Matches each and every item in the array contained in items on the “left”:

items:
#@overlay/match by=overlay.all
- item10

Example 3: Map Items

(as of v0.26.0+)

Matches each and every item in the map contained in items on the “left”:

items:
  #@overlay/match by=overlay.all
  _:
    name: item10

(note: the key name _ is arbitrary and ignored)

overlay.subset()

An Overlay matcher function that matches when the “left” node’s structure and value equals the given target.

overlay.subset(target)
  • target (any) — value that the “left” node must equal.

Examples

Example 1: Scalar

To match, scalar values must be equal.

#@overlay/match by=overlay.subset(1)
#@overlay/match by=overlay.subset("Entire string must match")
#@overlay/match by=overlay.subset(True)

(if a partial match is required, consider writing a Custom Overlay Matcher function)

Example 2: Dict (aka “map”)

To match, dictionary literals must match the structure and value of left.

#@overlay/match by=overlay.subset({"kind": "Deployment"})
#@overlay/match by=overlay.subset({"metadata": {"name": "istio-system"}})

Example 3: YAML Fragment

To match, yamlfragment’s must match structure and value of left.

#@ def resource(kind, name):
kind: #@ kind
metadata:
  name: #@ name
#@ end

#@overlay/match by=overlay.subset(resource("Deployment", "istio-system"))

overlay.and_op()

(as of v0.26.0+)

An Overlay matcher function that matches when all given matchers return True.

overlay.and_op(matcher1, matcher2, ...)

Examples:

#@ not_sa = overlay.not_op(overlay.subset({"kind": "ServiceAccount"}))
#@ inside_ns = overlay.subset({"metadata": {"namespace": "some-ns"}})

#@overlay/match by=overlay.and_op(not_sa, inside_ns),expects="1+"

overlay.or_op()

(as of v0.26.0+)

An Overlay matcher function that matches when at least one of the given matchers return True.

overlay.or_op(matcher1, matcher2, ...)

Examples:

#@ config_maps = overlay.subset({"kind": "ConfigMap"})
#@ secrets = overlay.subset({"kind": "Secret"})

#@overlay/match by=overlay.or_op(config_maps, secrets)

overlay.not_op()

(as of v0.26.0+)

An Overlay matcher function that matches when the given matcher does not.

not_op(matcher)

#@overlay/match by=overlay.not_op(overlay.subset({"metadata": {"namespace": "app"}}))
(Help improve our docs: edit this page on GitHub)