Deploy TKGs cluster with Terraform on TMC (SaaS)

TKG cluster can be deployed in a many ways. You can use simple YAML file and run it from the Supervisor. You can use some automation tool like Jenkins or create Kubernetes nodes from the TMC.
Apart from that, Terraform has a dedicated Tanzu Mission Control provider (plugin) to automate many processes.

In this article I will show you how to deploy TKGs cluster (v1beta1) with Terraform on TMC (SaaS).

Prerequisites:

  • vSphere with Tanzu – in my case it’s a vSphere 8.0 U2a with AVI LB
  • vSphere with Tanzu connected with TMC
  • Linux/Mac/Windows machine with Terraform installed

1. Generate API Token from the TMC

1. At first, we need to generate an API Token from the TMC. It will be used to connect Terraform to TMC. Log in to your Organization: https://console.cloud.vmware.com/

2. Go to My Account under User Settings.

3. Click Generate Token to move to the token settings.

4. Type a Token Name, Token TTL and Define Scopes. For testing purposes, I selected all available roles. Click Generate to create a token.

5. Token generated! Copy somewhere this data.

2. Terraform installation

Depending on your operating system, you will need to install right version of the Terraform. Check documentation – https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli and choose a appropriate binaries. In this tutorial, I have installed Terraform v1.6.6 on MacOS.

3. Create Terraform configuration files

Before we begin, remember to setup vSphere Namespace with your parameters.

I’ve created a dedicated catalog: tkgs-cluster-terraform-tmc where I will store all the files.

1. Create a first file: tmc-conf.tf

It’s a Terraform provider file. Configuration describes name, version and TMC SasS parametersendpoint and API token.
!! REMEMBER to use minimum v1.4.1 version of this provider !!

More info about Terraform Tanzu Mission Control provider:
https://registry.terraform.io/providers/vmware/tanzu-mission-control/latest/docs

terraform {
  required_providers {
    tanzu-mission-control = {
      source = "vmware/tanzu-mission-control"
      version = "1.4.1"
    }
  }
}


# Provider configuration for TMC SaaS
provider "tanzu-mission-control" {
  endpoint            = var.endpoint            
  vmw_cloud_api_token = var.vmw_cloud_api_token

}

2. Create a second file: terraform.tfvars

Here, you need to paste generated API Token from the 1st step and enter endpoint name. Endpoint is a FQDN of the TMC SaaS infrastructure.

vmw_cloud_api_token     = "Generated_API_Token_From_1st_Step"
endpoint      = "NAME.tmc.cloud.vmware.com"

3. Create a third file: cluster.tf

This is a configuration of the new TKGs cluster.

Bolded data can be different than yours. Fill this parameters according to your needs.

Default Cluster template can be found here:
https://registry.terraform.io/providers/vmware/tanzu-mission-control/latest/docs/resources/cluster#tanzu-kubernetes-grid-service-workload-cluster

resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tkgs_cluster" {
  name                    = "cl01-terraform-tmc" // name of the TKGs cluster
  management_cluster_name = "vmware-vcs-01" // name of the Management Cluster on the TMC
  provisioner_name        = "ns01" // vSphere namespace

  spec {
    cluster_group_name = "default"

    topology {
      version           = "v1.26.5+vmware.2-fips.1-tkg.1"
      cluster_class     = "tanzukubernetescluster"
      cluster_variables = jsonencode(local.tkgs_cluster_variables)

      control_plane {
        replicas = 1

        os_image {
          name    = "photon"
          version = "3"
          arch    = "amd64"
        }
      }

      nodepool {
        name        = "worker"
        description = "vmattroman-workers"

        spec {
          worker_class = "node-pool"
          replicas     = 3
          overrides    = jsonencode(local.tkgs_nodepool_a_overrides)

          os_image {
            name    = "photon"
            version = "3"
            arch    = "amd64"
          }
        }
      }

      network {
        pod_cidr_blocks = [
          "172.20.0.0/16",
        ]
        service_cidr_blocks = [
          "10.96.0.0/16",
        ]
        service_domain = "cluster.local"
      }
    }
  }

  timeout_policy {
    timeout             = 60
    wait_for_kubeconfig = true
    fail_on_timeout     = true
  }
}

4. Create a fourth file: variables.tf

At the beginning, there are variables regarding to the endpoint and API token.
After that, there are variables regarding to the configuration of the ClusterClass. If you want to deploy TKGs cluster with an API v1beta1 it’s a strong requirement.

Bolded data can be different than yours. Fill this parameters according to your needs. Fill this parameters according to your needs.

Default template can be found here:
https://registry.terraform.io/providers/vmware/tanzu-mission-control/latest/docs/resources/tanzu_kubernetes_cluster#example-usage-1

variable "endpoint" {
  description = "endpoint"
  type        = string
}

variable "vmw_cloud_api_token" {
  description = "vmw_cloud_api_token"
  type        = string
}

// ClusterClass configuration below

locals {
  tkgs_cluster_variables = {
    "controlPlaneCertificateRotation" : {
      "activate" : true,
      "daysBefore" : 30
    },
    "defaultStorageClass" : "tanzu-vmware-ssd",
    "defaultVolumeSnapshotClass" : "tanzu-vmware-ssd",
    "nodePoolLabels" : [

    ],
    "nodePoolVolumes" : [
      {
        "capacity" : {
          "storage" : "20G"
        },
        "mountPath" : "/var/lib/containerd",
        "name" : "containerd",
        "storageClass" : "tanzu-vmware-ssd"
      },
      {
        "capacity" : {
          "storage" : "20G"
        },
        "mountPath" : "/var/lib/kubelet",
        "name" : "kubelet",
        "storageClass" : "tanzu-vmware-ssd"
      }
    ],
    "ntp" : "0.pl.pool.ntp.org",
    "storageClass" : "tanzu-vmware-ssd",
    "storageClasses" : [
      "tanzu-vmware-ssd"
    ],
    "vmClass" : "best-effort-xsmall"
  }

  tkgs_nodepool_a_overrides = {
    "nodePoolLabels" : [
      {
        "key" : "vmattroman-worker-label",
        "value" : "terraform-tmc"
      }
    ],
    "storageClass" : "tanzu-vmware-ssd",
    "vmClass" : "best-effort-xsmall"
  }
}

4. TKGs cluster deployment

Now, when all things are ready, we can move on and create a new cluster!

1. Open shell in the place, where files are created. Type a command terraform init to initialize Terraform provider plugins.

terraform init

2. To check that everything is OK before deployment, run a command terraform plan.
The output generates execution plan and description about “installation” steps.

terraform plan

3. Run a command terraform apply -auto-approve to start deployment.
Now, Terraform starts cluster installation. You can check progress in the shell, vCenter and Tanzu Mission Control.

terraform apply -auto-approve

4. Done! After a few minutes, TKGs cluster was created and attached to the TMC! 🥳

4. Logging in to the TKGs Cluster

Newly created cluster is accessible through the shell and looks healthy! 😀

5. Destroy cluster

To delete and detach TKGs cluster from the vCenter and TMC, type a command: terraform destory -auto-approve

terraform destroy -auto-approve

Wrapping up

And that’s it. We deployed TKGs cluster using Terraform in Tanzu Mission Control.
This is the beginning and basic possibilities offered by Terraform TMC provider.
In the future, I will show more options that can be used when setting up a cluster.

Leave a Reply

Your email address will not be published. Required fields are marked *