Infrastructure as Code with Terraform

The last few months/years my focus and attention has shifted more and more from purely development to automation, CI/CD, testing and making my life easier with it :)

A while ago I ran into something called Terraform. It is a tool created by HashiCorp and designed to make your work around creating infrastructure easier and maintainable.

For those of you familiar with setting up environments, servers, networking, firewalls etc.. You know that it can become a lot very fast. And keeping track of it all requires good documentation. And since development online is shifting more and more towards cloud providers like GCP and AWS products are added and changed more frequently and you probably want to keep a persistent state of your infrastructure.

This is where Terraform gets in the picture. What this does is enabling you to create infrastructure as code (IaC). So say for example you want to setup a new Kubernetes cluster. You can use the interface in Google or Amazon or DigitalOcean to click your way through and choose the settings you need. Most of these interfaces are easy to use and work perfectly fine. But what if you need to do this several times, or you have a fixed configuration you use per customer or something like that. You could write it down and use your cheat sheet every time. Works fine too. But Terraform can be your sheet cheat and your executor in doing this. Allowing you to drink your coffee and not worry if you've clicked the right settings last time around ;)

So how does this work? Basically you download and install the Terraform cli tool. You write down a few Terraform configuration files, plan, check and execute the everything. Sounds easy enough right? Well it is, of course you and I both know things get bigger and more complex then just a few servers you want to spin up. But Terraform has some solutions for that as well. So let's see how we go about it.

Creating a cluster for Development and Production

I've created an example here. The usecase here is a relatively simple one but it shows the power and possibilities of Terraform. What do we want to do here?

• Create a cluster configuration for DigitalOcean
• Differentiate configurations between development and production contexts
• Define a network location for the cluster
• Choose how many nodes our default node pool should have
• What type/size should the nodes be
• Ensure the configuration plans are persisted
• Check the plan and apply it
• Change configuration and see what happens
• Destroy everything afterwards

Install Terraform cli

The first thing you need is the cli to actually use Terraform. Go to the downloads page and install it: https://www.terraform.io/downloads.html

First configuration

You can create an empty directory and use whatever editor you prefer. Terraform collects the configuration from all the files ending with the .tf extension. The configuration language itself is created by HashiCorp for Terraform so it is a fully custom configuration language.

The first thing we do is create a file and we'll call it terraform.tf. Here we'll configure the fact that we want to use the DigitalOcean Terraform provider. Providers are created by the companies like DigitalOcean, Google etc.. So you can use Terraform with their products. This is the DigitalOcean provider for example: https://registry.terraform.io/providers/digitalocean/digitalocean/latest

In the terraform.tf file, place the following lines:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "2.4.0"
    }
  }
}

provider "digitalocean" {
  token = var.do_token
}

variable "do_token" {
  type = string
}

What this says is, we require the digitalocean/digitalocean provider and we need to give it our DigitalOcean api token as a variable.

Next let's define how many clusters and nodes we need.

variable "cluster_count" {
  type = map(number)
  default = {
    "development" = 1
    "production" = 1
  }
}

variable "node_count" {
  type = map(number)
  default = {
    "development" = 2
    "production" = 3
  }
}

We created variables with default values that differ based on workspace. So we say here the amount of clusters we need is 1 for development as well as for production. The cluster node count is 2 in development and 3 in production.

variable "auto_upgrade_nodepool" {
  type = bool
  default = true
}

variable "size" {
  type = map(string)
  default = {
    development = "s-1vcpu-2gb"
    production = "s-1vcpu-2gb"
  }
}

variable "region" {
  type = string
  default = "ams3"
}

Here we say that the we always want to upgrade our nodes. We want for development as production the same types of nodes and we want our network region to be ams3 (Amsterdam, The Netherlands).

And we want to have some information on the Kubernetes version that DigitalOcean offers.

data "digitalocean_kubernetes_versions" "k8s_versions" {}

Now we've setup most variables and information we need. But we've not yet defined that we actually need resources. Let's do that.

Create a file called resources.tf. Again, the name doesn't matter much as long as it ends with .tf Terraform will read it.

In this file we'll define that we want a cluster and we'll use our defined variables.

resource "digitalocean_kubernetes_cluster" "main-cluster" {
  name = "main-cluster-${terraform.workspace}"
  region = var.region
  version = data.digitalocean_kubernetes_versions.k8s_versions.latest_version
  auto_upgrade = var.auto_upgrade_nodepool

  tags = [ "main-cluster-${terraform.workspace}" ]

  node_pool {
    name = "worker-pool"
    node_count = var.node_count[terraform.workspace]
    size = var.size[terraform.workspace]
  }
}

Ok so let's break this down.

The first line says:

resource "digitalocean_kubernetes_cluster" "main-cluster" {

This tells Terraform that you want a DigitalOcean Kubernetes cluster and in our configuration we're gonna call it "main-cluster".

The name:
name = "main-cluster-${terraform.workspace}"

This is the name DigitalOcean is going to give to the cluster. We've defined the prefix "main-cluster" and appended it with our workspace name.

So when we create it in the development workspace the cluster is gonna be called "main-cluster-development" and in production "main-cluster-production"

Region:
region = var.region

This set's our region based on our defined region variable.

The Kubernetes version is a bit more interesting.

version = data.digitalocean_kubernetes_versions.k8s_versions.latest_version

Remember the line: data "digitalocean_kubernetes_versions" "k8s_versions" {} we set in the first file. This is that data we're using. DigitalOcean provides us with a list of Kubernetes versions they have available and they have a reference to latest_version. Which at the time of writing is 1.20.

So we set the version to the latest version DigitalOcean uses.

auto_upgrade = var.auto_upgrade_nodepool

Here we configure the auto_upgrade settings to be true as we set in our variable.

tags = [ "main-cluster-${terraform.workspace}" ]

Tags you can fill with whatever you can use/need for reference. I've added this one as an example.

node_pool {
    name = "worker-pool"
    node_count = var.node_count[terraform.workspace]
    size = var.size[terraform.workspace]
 }

The default node pool we setup is again based on the variables we've set in the first file. So the size of the nodes are the same and amount of nodes will 2 in development and 3 in production.

Next we'll tell Terraform what we want as output after the execution is done, so what information will be shown and can be used in possible concurrent automation setups.

Create a file called output.tf and add the line:

output "cluster_information" {
  value = digitalocean_kubernetes_cluster.main-cluster
}

This will return the cluster called "main-cluster" information as that is what we called it in our resources definition.

Create workspace

As we have created variables based on workspaces we need to actually create it. Otherwise the reference will fail.

First initalize terraform in your directory. Head to the directory in terminal and execute:

terraform init

Then create the workspace development

terraform workspace new development

The workspace will automatically be switched to development.

Plan your deployment

Before you actually execute your configuration be sure to plan and check it. Terraform  allows you to do this as following:

terraform plan -out development.tfplan

You might be asked for your api token. Enter it and continue.

This will show you what will be created, changed or destroyed. If everything goes as planned you should see the right configuration for your cluster outputted on your screen.

Apply your deployment

If you're happy with what you see apply the created plan:

terraform apply "development.tfplan"

This can take a few minutes as your cluster is now being provisioned. My experience is that it takes around 8-10 minutes.

Afterwards you'll see the output you've configured, that being your created cluster. And when you go to your DigitalOcean account you can now see your cluster.

I love it when a plan comes together
- John "Hannibal" Smith

Changes

You might decide later on that development doesn't need two nodes in the default worker pool. You might be fine with just one.

Change the variable default to 1 in the terraform.tf file.

Plan it again, see the changes that terraform will make and apply it if you're happy with what you see.

Destroy

You're done with your development cluster? Good let's remove all that was created.

Simply run

terraform destroy

And all is removed for you.

Production

You might also want to create your production setup at a certain point.

Simply create/switch to production workspace.

terraform workspace new production

Plan again

terraform plan -out "production.tfplan"

And apply again

terraform apply "production.tfplan"

Again it is that easy.

Persisted state

You might ask yourself "Why all this trouble for something I can do online as well?".

It is a fair question. You might not need all this automation. When you just have 1 setup this might be overkill. But when you're doing this multiple times, things need to change, demands change, customers change etc.. This will allow you document and persist the state of your infrastructure in readable and manageable manner. You can put all the configuration and outputted plans in git and keep track of it. Persisted state is worth a lot when you're doing this multiple times and other people might need to continue with it.

What's next

This was a very basic and quick example of the possibilities Terraform offers. What we've done in this example is only create the infrastructure. Now you can actually use the resources you've created. Or hand the credentials of it to your colleague or customer that needed this.

You can think of ways to combine this in automation pipelines, ansible, Jenkins etc.. The possibilities are a many. And there is much more to learn about Terraform and it's possibilities.

So I challenge you try it out yourself. You can use DigitalOcean as I did, the costs are kept to a minimum if you remove it after you've tried it.

Conclusion

Terraform is a great tool for keeping infrastructure setup maintainable, manageable and readable. And your setups are persisted. So it offers great advantages if you're managing multiple setups and need to deploy changes and such.

There is much more to learn but I'm certain that I'm going to use this a lot more in the future as HashiCorp has done some great work creating this tool.

I hope you enjoyed the read!