YAML and Annotations

Overview ¶

Templates in ytt — rather than being YAML-like text files with templating injected in — are well-formed YAML files with templating annotated on.

It’s reasonable to have expectations of how to write templates, especially if we have prior experience. But in the shift from a free-form text file to a structured YAML document set, some of those expectations are foiled. And when they are, it can be rather frustrating.

If we take a few minutes to cover:

• what YAML is — its format and constituent parts; and
• how to annotate YAML in ytt,

it can mitigate much of that frustration.

What is YAML? ¶

YAML is a tree of nodes. Each node is of one of these six types:

• a Document Set (which is list of Documents)
• a Map that has Map Items
• an Array that has Array Items

We’ll start from the top of the tree…

Documents and Document Sets ¶

A file containing text in YAML format is parsed into a Document Set: a collection of Documents.

Figure 1: A file, parsed into a document set (dotted blue) containing two documents (solid blue).

Note: --- marks the start of a new document.

The corresponding tree of nodes looks something like this:

Figure 2: Corresponding tree: this document set has two documents.

A given document has exactly one (1) value. It is usually either:

• a Map,
• an Array, or
• a Scalar

We’ll look at each of these, in turn. Maps are most common…

Maps and Map Items ¶

A “map” is collection of key-value pairs. Each pair is referred to as a “map item”.

Figure 3: Each document has a map (dotted green) each containing two map items (solid green).

The complete tree of nodes looks like this:

Figure 4: Corresponding tree: each document has a map for its value; each map has two items.

Let’s zoom in on the first document, and explicitly reveal the contents of a map item:

Figure 5: Each map item has a key and a value; here, both map items’ key is a string and their value is an integer.

Like documents, a map item has exactly one (1) value. And just like documents, it’s either:

• a Map,
• an Array, or
• a Scalar

Let’s see what it looks like for a map item to have a map, itself.

Figure 6: The document’s value is a map with two map items; the second map item has a key of “metadata” and a value that is another map.

Arrays and Array Items ¶

An “array” is a list of “array items” (zero-indexed).

Like documents (and map items), an array item has exactly one (1) value. Once more, it’s either:

• a Map
• an Array, or
• a Scalar

Figure 7: The map item “foo” has an array (dotted gold) for a value; that array has three array items (solid gold).

Let’s see what it looks like when an array item’s value is a map…

Figure 8: “foo” has an array with one array item whose value is a map containing two map items: name and factor.

We’ve seen scalars in use above. Let’s cover them explicitly.

Scalars ¶

A “scalar” is a fundamental value. YAML supports:

• integers (e.g. 13, 42, 137, 32767)
• floating point numbers (e.g. 1.0, 3.14, 137.03599913)
• strings (e.g. “”, “fine structure”, “$ecr3t”)
• boolean values: true or false
• null (when a value is omitted, that implies null; ~ == null)

Summary ¶

In short:

• a YAML file is parsed into a Document Set which is merely a list of Documents.
• a given Document has one value.
• Both Maps and Arrays are nothing more than collections of items.
  • a Map Item has a key and a value
  • an Array Item has a value
• a value (whether held by a document, map item, or array item) is either a Map, an Array, or a Scalar.

With this structure in mind, we can now look into how ytt annotates it.

Annotating YAML ¶

A ytt “annotation” is a YAML comment of a specific format; it attaches behavior to a node.

For example:

Figure 9: A YAML file with a ytt annotation.

Can be understood to mean that the value of foo is the result of evaluating the expression 13 + 23 + 6.

The diagram reveals how the annotation is attached to the YAML tree:

Figure 10: The annotation (dotted black) is attached to the map item (solid green).

This is a document, whose value is a map, that contains a single map item. To the right of the map item is an annotation that contains an expression that, when evaluated, becomes a value. The annotation is attached to the map item.

How an Annotation Finds its Node ¶

When attaching annotations, ytt follows these two rules (in order):

1. the annotation is attached to the value-holding node on its left, if there is one; otherwise,
2. the annotation is attached to the value-holding node directly below it.

In the example above, the value-holding node to the left of the annotation is the map item.

As noted in the summary above, “value-holding” nodes are:

• Documents,
• Map Items, and
• Array Items.

Example: Annotating Map Items ¶

Let’s see these rules at play:

Figure 11: A templated document with three annotations.

We’ll ignore what these annotations mean, for now, and focus on where they attach. There are three annotations in total.

Visualizing the YAML tree can help:

Figure 12: Each annotation attaches to the value-holding node to its left or bottom.

Taking each annotation in turn:

• @overlay/match by=overlay.all:
  • is in the document set, but that is not a value-holding node
  • has a document node just below it and so is attached to that node.
• @overlay/replace:
  • has no node to its left;
  • has a map just below it but maps are not value-holding;
  • the next node is a map item, and so attaches to that node.
• @ 13 + 23 + 6:
  • has a map item to its left, and so attaches to that node.
  • there is another map item below it (i.e. bar: true), but a home has already been found for this annotation.

Example: Annotating an Array ¶

When an array item contains a map, it can be tricky to know which item a particular annotation attaches to.

In this example, we reinforce the value of knowing the two rules that determine how an annotation finds its node.

Figure 13: An overlay ensuring each book has been marked as reviewed. The blank line after the array item is intentional.

There are three annotations in this example. The first one clearly annotates the document. But what about the next two?

Visualizing the layer, it can start to become more clear…

Figure 14: The newline after the array item ensures the last annotation attaches to the map item.

Taking each annotation in turn:

• the first @overlay/match by=overlay.all:
  • has a document node just below it and so is attached to that node.
• the second @overlay/match by=overlay.all:
  • has a map item (books:), but that is above this annotation;
  • books: contains an array, but arrays are not value-holding nodes;
  • that array’s array item (denoted by -) is just below the annotation and so is attached to that node.
• finally, @overlay/match missing_ok=True:
  • if this had been the same line as the array item, it would have also attached to the same node as the previous annotation (but it is not, and is it won’t);
  • the next node is a map, but again, maps are not value-holding;
  • the next node is a map item (reviewed:), so the annotation attaches to that node.

Further Exploration ¶

While we’ve thoroughly covered the fundamentals here, these concepts only become real with use:

• put your understanding to the test by picking an example from the Playground and tweaking it in various ways to see what actually happens.
• take a simple example and run it through ytt --debug and study the ### ast section of the output. Note:
  • in this output, annotations are referred to as meta.
  • “AST” is short for “Abstract Syntax Tree” which is the technical name for the trees we’ve been visualizing in this guide.
  • we recommend making small changes and study how it modifies the “AST”
• if you’d like to obsess over YAML itself, look no further than the YAML spec.