Blog Posts

Using Carvel Terraform Provider to manage Kubernetes workloads

by Soumik Majumder — Nov 29, 2021

The Carvel tools are designed to empower our users to manage their Kubernetes workloads effectively. We realise that engineers leveraging Terraform to declaratively define their platforms might want to use Carvel tools to set up applications and workloads on their Kubernetes clusters in a predictable manner. Carvel’s terraform provider allows engineers to do exactly this using Terraform configurations.

In this blog, we will be using the provider to deploy this sample guestbook application on a Kubernetes cluster.

Do make a copy of the folder in your working directory if you want to follow along!

Setting things up ¶

The Carvel provider is published on the Terraform registry.

The resources used, require their respective binaries to be available on $PATH. We will be using ytt to apply an overlay to our manifests and then deploy the resources using kapp. We can make the Carvel binaries available by using our install script

$ wget -O- https://carvel.dev/install.sh | bash
# or with curl...
$ curl -L https://carvel.dev/install.sh | bash

See our Install section for alternative installation methods.

If you do not have Terraform set up on your system, you can refer to the installation page in the official docs for the same.

We will be using the latest version of the Carvel provider available as of today. We declare this requirement in main.tf

terraform {
  required_providers {
    carvel = {
      source = "carvel-dev/carvel"
      version = "0.10.0"
    }
  }
}

We will then define our Terraform configuration which sets up our resources on the cluster in the file app.tf

Connecting to the cluster ¶

The kapp resource requires the kubeconfig attribute to connect to the cluster. We will be asking the kapp resource to use the config used by kubectl in the same environment.

Look here for more ways of authenticating while connecting to a cluster.

provider "carvel" {
  kapp {
    kubeconfig {
      from_env = true
    }
  }
}

Overlaying using ytt ¶

We can use ytt to template and apply overlays to our manifests, this allows our manifests to be dynamic. In this blog, we will be asking ytt to apply an overlay that ensures that all Deployment resources spawn one replica.

data "carvel_ytt" "guestbook" {
  files = ["ytt-config"]
  ignore_unknown_comments = true

  config_yaml = <<EOF
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind":"Deployment"}),expects="1+"
    ---
    spec:
      replicas: 1
  EOF
}

Deploying with kapp ¶

kapp helps us deploy resources to our clusters in a safe and predicatable manner, apply them in a certain order and then wait for the resources to reach their desired state among other things. We will be asking kapp to group the resources declared in the manifest as an application called “guestbook” and deploy it to the default namespace. The manifest consumed is the one produced by the ytt data source.

resource "carvel_kapp" "guestbook" {
  app = "guestbook"
  namespace = "default"
  config_yaml = data.carvel_ytt.guestbook.result
}

Get, Set, Deploy! ¶

At this point we can install the provider:

$ terraform init

and then validate our configuration:

$ terraform validate

For readers following along, the working directory looks something like this at this point:

├── app.tf
├── main.tf
├── terraform.tfstate
├── terraform.tfstate.backup
└── ytt-config
    ├── frontend.yaml
    ├── guestbook-all-in-one.yaml
    └── redis-slave.yaml

The final contents of app.tf being:

terraform {
  required_providers {
    carvel = {
      source = "carvel-dev/carvel"
      version = "0.10.0"
    }
  }
}

provider "carvel" {
  kapp {
    kubeconfig {
      from_env = true
    }
  }
}

data "carvel_ytt" "guestbook" {
  files = ["ytt-config"]

  # Configure all deployments to have 1 replica
  config_yaml = <<EOF
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind":"Deployment"}),expects="1+"
    ---
    spec:
      replicas: 1
  EOF
}

resource "carvel_kapp" "guestbook" {
  app = "guestbook"
  namespace = "default"
  config_yaml = data.carvel_ytt.guestbook.result
}

We can now create the declared resources.

$ terraform create

Terraform creates the carvel_kapp.guestbook resource and waits for it to reach the desired state. (Thanks kapp!). We can run:

$ kubectl get deployment frontend -o yaml

To verify that each Deployment resource spawns one replica even thought the original manifest specified three replicas. (Thanks to ytt’s overlay!)

That’s cool! How does this make my life easier? ¶

If you are using Terraform to declaratively provision GKE or EKS clusters and developers working on your platform need to be able deploy workloads which use cert-manager to allow secure connections. You can use the kapp resource to install cert-manager on your cluster as a part of your Terraform configuration.

This stands true for other resources you would want to deploy on your cluster after provisioning it while the ytt data source allows you to template and overlay your manifests on the go.

We would love to know how you are using the Carvel tools and our Terraform provider on this thread!