Adventures of a SysAdmin

Boostraping Private Gke Cluster With Terraform

2022-08-14T00:00:00+00:00

Kubernetes is an orchestration tool which automates deployment, scaling, and management of containerized applications. GKE (Google Kubernetes Engine) is a manged Kubernetes platform offering by Google Cloud, it takes away the pain of managing Kubernetes control plane. In this post, I will be sharing the steps required to easily bootstrap a private GKE cluster with Terraform.

Creating a service account

First, you will need to create a service account within Google Cloud Platform (GCP). Open the GCP console and navigate to IAM & Admin -> Service Accounts -> Create service account (at the top) and fill out the details.

There are multiple ways for service account to authenticate with GCP, but for convenience we will be using service account keys. In the same service account window, click on 3 dots under actions and select Manage keys. In the keys window, click Add key -> Create new key -> Select JSON -> Create, the key will be automatically downloaded.

Configuring Terraform providers

First create a root folder where you will hold Terraform files, in this post the folder name we will be using is gke. In the gke folder, create the following files:

config.tf - will contain provider information
main.tf - will have our resource definitions

You should also move the service account key file to this folder, I name my service account key file account.json. Your directory structure should look like:

gke
├── account.json
├── config.tf
├── main.tf

Open the config.tf file and fill it with the following text:

terraform {
  required_providers {
    google = {
      version = "~> 4.31.0"
    }
  }
}

We specify that we will be using Google provider of version 4.31.x, ~> specifies that the rightmost component of the version can be incremented.

Bootstrapping the GKE cluster

Now that we have our provider configured, we are ready to define Terraform resources, that allows us to create a GKE cluster with one terraform apply command.

Enabling required GCP APIs

By default, the GKE API is not enabled and without it, Terraform won’t be able to provision a GKE cluster. This could be enabled manually, but we can also enable it via Terraform. Open main.tf and add the following lines (you should change “your-project-id” with your actual project id):

locals {
  api = [
    "cloudresourcemanager.googleapis.com",
    "container.googleapis.com",
  ]
}

resource "google_project_service" "apis" {
  count = length(local.api)

  service                    = local.api[count.index]
  project                    = "your-project-id"
  disable_dependent_services = true
}

Here we define a local variable api which is a list of APIs that we will be enabling, and we later access it via the local keyword (as can be seen in service value). cloudresourcemanager.googleapis.com enables Cloud Resource Manager API, which allows us to interact with GCP resources, while container.googleapis.com enables Kubernetes Engine API, which allows us to provision GKE clusters. We are using count to create multiple google_project_service resources with a single definition.

Creating a VPC network

For the GKE cluster, we will also create a private VPC network. We will be using the predefined Google Network module. A module is basically a collection of resources that are used together. Append your main.tf with:

module "gke_network" {
  source  = "terraform-google-modules/network/google"
  version = "~> 5.2.0"

  network_name = "gke-network"
  project_id   = "your-project-id"
  subnets = [
    {
      subnet_name           = "gke-subnetwork"
      subnet_ip             = "10.1.0.0/16"
      subnet_private_access = "true"
      subnet_region         = "europe-west3"
    },
  ]
  secondary_ranges = {
    "gke-subnetwork" = [
      {
        ip_cidr_range = "10.2.0.0/16"
        range_name    = "pod-ip-range"
      },
      {
        ip_cidr_range = "10.3.0.0/16"
        range_name    = "service-ip-range"
      },
    ]
  }

  depends_on = [google_project_service.apis]
}

With source we tell Terraform where to look for the module and everything else is pretty self-explanatory - we create a VPC and a subnetwork within that VPC, then we specify secondary IP ranges which will be used for Kubernetes Pods and Services (for real life clusters you should carefully consider subnet ranges as per docs). Finally, we use depends_on to ensure that Terraform won’t attempt to provision these resources until APIs are enabled.

Provisioning GKE cluster

Finally, we are ready to provision our GKE cluster. Again, we will be using a Terraform module to make our life a lot easier. Add these lines to main.tf:

module "gke" {
  source  = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
  version = "~> 22.1.0"

  enable_private_nodes = true
  ip_range_pods        = "pod-ip-range"
  ip_range_services    = "service-ip-range"
  master_authorized_networks = [
    { cidr_block = "1.2.3.4/32", display_name = "First IP that will have access to Control Plane" },
    { cidr_block = "5.6.7.8/32", display_name = "Second IP that will have access to Control Plane" },
  ]
  name           = "gke-cluster"
  network        = module.gke_network.network_name
  network_policy = true
  node_pools = [
    {
      name         = "my-node-pool"
      machine_type = "e2-small"
      min_count    = 1
      max_count    = 3
      disk_size_gb = 30
    },
  ]
  project_id               = "your-project-id"
  region                   = "europe-west3"
  regional                 = false
  remove_default_node_pool = true
  subnetwork               = "gke-subnetwork"
  zones                    = ["europe-west3-a"]

  depends_on = [google_project_service.apis, module.gke_network]
}

module "auth" {
  source  = "terraform-google-modules/kubernetes-engine/google//modules/auth"
  version = "~> 22.1.0"

  cluster_name = module.gke.name
  location     = module.gke.location
  project_id   = "your-project-id"
  
  depends_on = [google_project_service.apis, module.gke]
}

resource "local_file" "kubectlconfig" {
  content  = module.auth.kubeconfig_raw
  filename = "gke-cluster-config"
}

Things worth mentioning:

enable_private_nodes = true ensures that created nodes do not have external IP address. As we are creating a private cluster, we want to limit public endpoint accessibility, we could disable it, but instead we will filter IPs which can access this public endpoint. We achieve that with master_authorized_networks.
You can use node_pools to define your list of node pools that should be created. Here we create only one pool with e2-small instances. We allow a maximum of 3 nodes for autoscaler to scale if we lack resources.
As this is a tutorial, we are creating a zonal cluster by setting regional = false and zones = ["europe-west3-a"]. Zones specify in which zones the cluster can be hosted. Zonal cluster has only 1 Control plane, so if you want high availability you should change regional to true, and then you can omit zones variable as it is not required for regional cluster.
module "auth" and resource "local_file" "kubectlconfig" are optional, however they are convenient as they will create you a config file that allows you to authenticate with your GKE cluster.

terraform init && terraform apply

Our main.tf is ready and all that is left for us to do it initialize terraform providers/modules and apply the configuration. Before you do that, you need to set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to our account.json file. You can achieve this with: export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gke/account.json

Once the environment variable is set, run terraform init followed by terraform apply. After running terraform apply you will be provided with a list of resources that will be provisioned and will be asked for confirmation - type yes press Enter on your keyboard and go grab a drink as the provisioning will take some time, usually it is around 15-20 minutes.

Final Words

And here you have it - an easy way to provision a Private Cluster in GKE using Terraform! It is worth mentioning that the configuration here is simplified - usually you would want to define variables in variables.tf and use those variables instead of direct values as in this post, also you would probably want to separate resources in different files, in this case you could move API enabling to api.tf and VPC creating to network.tf.

Complete main.tf

locals {
  api = [
    "cloudresourcemanager.googleapis.com",
    "container.googleapis.com",
  ]
}

resource "google_project_service" "apis" {
  count = length(local.api)

  service                    = local.api[count.index]
  project                    = "your-project-id"
  disable_dependent_services = true
}

module "gke_network" {
  source  = "terraform-google-modules/network/google"
  version = "~> 5.2.0"

  network_name = "gke-network"
  project_id   = "your-project-id"
  subnets = [
    {
      subnet_name           = "gke-subnetwork"
      subnet_ip             = "10.1.0.0/16"
      subnet_private_access = "true"
      subnet_region         = "europe-west3"
    },
  ]
  secondary_ranges = {
    "gke-subnetwork" = [
      {
        ip_cidr_range = "10.2.0.0/16"
        range_name    = "pod-ip-range"
      },
      {
        ip_cidr_range = "10.3.0.0/16"
        range_name    = "service-ip-range"
      },
    ]
  }

  depends_on = [google_project_service.apis]
}

module "gke" {
  source  = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
  version = "~> 22.1.0"

  enable_private_nodes = true
  ip_range_pods        = "pod-ip-range"
  ip_range_services    = "service-ip-range"
  master_authorized_networks = [
    { cidr_block = "1.2.3.4/32", display_name = "First IP that will have access to Control Plane" },
    { cidr_block = "5.6.7.8/32", display_name = "Second IP that will have access to Control Plane" },
  ]
  name           = "gke-cluster"
  network        = module.gke_network.network_name
  network_policy = true
  node_pools = [
    {
      name         = "my-node-pool"
      machine_type = "e2-small"
      min_count    = 1
      max_count    = 3
      disk_size_gb = 30
    },
  ]
  project_id               = "your-project-id"
  region                   = "europe-west3"
  regional                 = false
  remove_default_node_pool = true
  subnetwork               = "gke-subnetwork"
  zones                    = ["europe-west3-a"]

  depends_on = [google_project_service.apis, module.gke_network]
}

module "auth" {
  source  = "terraform-google-modules/kubernetes-engine/google//modules/auth"
  version = "~> 22.1.0"

  cluster_name = module.gke.name
  location     = module.gke.location
  project_id   = "your-project-id"
  
  depends_on = [google_project_service.apis, module.gke]
}

resource "local_file" "kubectlconfig" {
  content  = module.auth.kubeconfig_raw
  filename = "gke-cluster-config"
}

Preparing Usb That Auto Installs Debian 11

2022-08-07T00:00:00+00:00

I have been setting up my home lab, which consists of 5 servers and instead of spending 30-40 minutes installing Debian interactively I decided to spend 4-5 hours automating the process, so all you would have to do is choose the boot option as USB for the OS to be installed. After trial and error of multiple re-installs, here are the steps I took to succeed:

Create bootable USB flash drive
Create preseed file
Update grub.cfg, isolinux.cfg and txt.cfg

Creating bootable USB flash drive

I am not going to go into detail how to do this, as this is a pretty straight-forward process and there are different ways to achieve it. I used amd64 image which can be downloaded from Debian page. And then used Rufus to create a bootable USB from that image.

Creating preseed file

Once your bootable USB flash drive is prepared, you will need to create a preseed file. This file is used during the installation to automatically provide the values to the questions that you are asked normally during interactive installation. There are many options which can be found on Debian Appendix B, but my goal was to set up a basic server with SSH installed, as I later use Ansible to finish configuring the server as desired, thus your preseed file might differ based on your requirements. Empty lines or lines starting with # are ignored during installation.

preseed.cfg

# Set language, country, locale, keyboard 
d-i debian-installer/language string en
d-i debian-installer/country string US
d-i debian-installer/locale string en_US.UTF-8
d-i keyboard-configuration/xkb-keymap select us

# If possible, automatically selects a network interface that has a link 
d-i netcfg/choose_interface select auto
# Sets hostname and domain
d-i netcfg/get_hostname string hostname
d-i netcfg/get_domain string domain.com
# Disables WEP key dialog
d-i netcfg/wireless_wep string

# Set up mirror settings that may be used to download additional components for the installed and to setup sources.list
d-i mirror/country string manual
d-i mirror/http/hostname string http.us.debian.org
d-i mirror/http/directory string /debian
d-i mirror/suite string testing
d-i mirror/http/proxy string

# Specifies a disk to partition. If there is only one disk, installer automatically defaults to it
d-i partman-auto/disk string /dev/sda
# Specifies partition method to be LVM
d-i partman-auto/method string lvm
# Specifies to use all available space for LVM partition
d-i partman-auto-lvm/guided_size string max
# Skip the warning if there is an old LVM configuration
d-i partman-lvm/device_remove_lvm boolean true
# Skip the warning if there is an old RAID array
d-i partman-md/device_remove_md boolean true
# Skip the confirmation to write the LVM partition
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
# Specifies to put all files in one partition
d-i partman-auto/choose_recipe select atomic
# Makes partman automatically partition without confirmation
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# Installs grub automatically to MBR if not other OS is detected on the machine
d-i grub-installer/only_debian boolean true
# Specifies a disk to which to install grub
# Required as installer might fail to properly identify between USB and the CD and without this option might still ask you to choose
d-i grub-installer/bootdev  string /dev/sda

# Set root password (might want to change this part if your USB will be shared)
passwd passwd/root-password password somepassword
passwd passwd/root-password-again password somepassword
# Skip creating additional users
passwd passwd/make-user boolean false

# Installs standard tools and SSH server
tasksel tasksel/first multiselect standard, ssh-server
# Accept to report back on what software is installed and what software is used
d-i popularity-contest/participate boolean true
# Skip the last message that installation is complete
d-i finish-install/reboot_in_progress note

Put this file in the root directory of USB, it should look like:

Updating grub.cfg, isolinux.cfg and txt.cfg

With the preseed.cfg file in place, you need to tell the installer to use it. For that, you have to update 2 files - grub.cfg and txt.cfg.

grub.cfg

Using above image as root directory, you will find grub.cfg in boot/grub directory - boot/grub/grub.cfg, open the file and find the following lines:

menuentry --hotkey=i 'Install' {
    set background_color=black
    linux    /install.amd/vmlinuz vga=788 --- quiet 
    initrd   /install.amd/initrd.gz
}

Update the linux line, so it would be:

menuentry --hotkey=i 'Install' {
    set background_color=black
    linux    /install.amd/vmlinuz preseed/file=/cdrom/preseed.cfg auto-install/enable=true vga=788 --- quiet  
    initrd   /install.amd/initrd.gz
}

preseed/file=/cdrom/preseed.cfg specifies the preseed file to use.

auto-install/enable=true delays the asking of localization questions, without this line you wouldn’t be able to preseed them in preseed.cfg

txt.cfg

txt.cfg file can be found by navigating from the root directory to isolinux - isolinux/txt.cfg, open the file and edit the append line from:

label install
	menu label ^Install
	kernel /install.amd/vmlinuz
	append vga=788 initrd=/install.amd/initrd.gz --- quiet

To:

label install
	menu label ^Install
	kernel /install.amd/vmlinuz
        append preseed/file=/cdrom/preseed.cfg auto-install/enable=true vga=788 initrd=/install.amd/initrd.gz --- quiet

isolinux.cfg

Finally, we want to update isolinux.cfg file, which can be found in the same directory as txt.cfg - isolinux/isolinux.cfg. We need to change default line from vesamenu.c32:

# D-I config version 2.0
# search path for the c32 support libraries (libcom32, libutil etc.)
path 
include menu.cfg
default vesamenu.c32
prompt 0
timeout 0

To install:

# D-I config version 2.0
# search path for the c32 support libraries (libcom32, libutil etc.)
path 
include menu.cfg
default install
prompt 0
timeout 0

This allows us to skip the menu which asks you what type of installation you want and directly goes to install option.

That is all

Now you are ready - plug the USB into the device you want to install Debian to, select that USB as a boot option and watch how the installation goes through without any input required from you.

Gke Access Management With Kubernetes Rbac

2022-07-30T00:00:00+00:00

Lately, I have been working on setting up proper accesses for our GKE (Google Kubernetes Engine) cluster. There are 2 options for controlling access within GKE cluster - GCP Identity and Access Management (IAM) and Kubernetes role-based access control (RBAC).

GCP IAM - allows managing permissions to GCP resources, however when it comes to GKE it becomes a bit limited in its granularity. For example, you could assign a Kubernetes Engine Viewer role to a user to give view permissions to Kubernetes resources, but you can’t limit it to a specific namespace.

Kubernetes RBAC - allows managing permissions to objects within Kubernetes cluster, it is a great option if you need to assign fine-grained permissions, i.e., limit view access only to pods within specific namespace.

We had the following requirements:

Users should be able to view and manage resources only within their namespace
Accesses should be easy to maintain - if you need to add/remove permissions for a user, it should be enough to add/remove that user from a list
Should be automated (no one likes manual work…)

Implementation

As we needed more granular control, we decided to create a custom GCP IAM role, which would grant minimal required GKE permissions and then leave the rest of access management to Kubernetes RBAC. We took the following steps:

Use Terraform for automation
Create a YAML file with list of users
Create and assign a custom GCP IAM role
Create Kubernetes Role and RoleBinding objects

Note: I won’t be describing how to set up Terraform providers in this post, and will assume that you already have them. For convenience, we will have all Terraform resources described in one *main.tf file.*

Creating a YAML file

As Google Groups were not an option, we decided to hold the list of users in a YAML file (users.yaml):

first.user@gmail.com: ["default", "backend", "frontend"]
second.user@gmail.com: ["frontend"]
third.user@gmail.com: ["backend"]

As you can see, the format is pretty simple: users_email: [list of namespaces to grant access to], it allows us to easily add or remove users, as well as effortlessly control which namespaces they can access. We will be using yamldecode to read this information in Terraform:

locals {
  users = yamldecode(file("users.yaml"))
}

Creating and assigning custom GCP IAM role

As per GKE docs at minimum users require container.clusters.get permission, so users could authenticate with a cluster. We also decided to add container.clusters.list to allow users to list those clusters.

resource "google_project_iam_custom_role" "gke_minimal_permissions" {
  role_id     = "minimalGKEPermissions"
  title       = "Minimal GKE permissions"
  permissions = ["container.clusters.get", "container.clusters.list"]
  project     = "project-name"
  description = "This role provides minimal GKE permissions"
}

resource "google_project_iam_member" "gke_minimal_role" {
  for_each = local.users

  member  = "user:${each.key}"
  role    = google_project_iam_custom_role.gke_minimal_permissions.name
  project = "project-name"

}

Create Kubernetes Role and RoleBinding objects

Finally, we needed to create Role and RoleBinding objects.

Role Object - is a set of permissions within a specific namespace.

RoleBinding Object - allows you to grant permissions of a Role object to list of Kubernetes subjects (users, groups, service accounts).

Knowing above, we had to ensure that a Role would be created in all namespaces and that a RoleBinding would assign a Role in a correct namespace. For that, we had to format the input from users.yaml file:

locals {
  users      = yamldecode(file("users.yaml"))
  namespaces = ["default", "frontend", "backend"]
  members = flatten(
    [for k, v in local.users :
      [for ns in v :
        { user = k, namespace = ns }
      ]
    ]
  )
}

We loop through every user and then loop again through every namespace assigned to a user to create a list of objects in format:

{
  "namespace" = "namespace_name"
  "user"  = "users_email"
}

flatten() function creates one list from multiple lists and is critical here, as without it, we would just have a list of list of objects, which would be of no use for us. With our users.yaml file, the output would be:

Having data in a format we can use, we can finally create Role and RoleBinding objects (we will be granting view permissions for all resources in namespaces that are assigned to a user):

resource "kubernetes_role" "resource_viewer" {
  for_each = toset(local.namespaces)

  metadata {
    name      = "resource-viewer"
    namespace = each.key
  }

  rule {
    api_groups = [""]
    resources  = ["*"]
    verbs      = ["get", "list", "watch"]
  }
}

resource "kubernetes_role_binding" "resource_viewer_users" {
  for_each = { for member in local.members : "${member.user}-${member.namespace}" => member }

  metadata {
    name      = "resource-viewer-user"
    namespace = each.value.namespace
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.resource_viewer[each.value.namespace].metadata[0].name
  }

  subjects {
    api_group = "rbac.authorization.k8s.io"
    kind      = "User"
    name      = each.value.user
  }
}

Few things to note

for_each = toset(local.namespaces) we are converting a list to a set, so we could use for_each instead of count. It is important for setting correct Role name when creating a RoleBinding.

for_each = { for member in local.members : "${member.user}-${member.namespace}" => member } from our formatted data we are creating a new object, so we could access required fields. "${member.user}-${member.namespace}" => member this part specifies how the new object key => value should look. With our data, we have:

Complete main.tf

locals {
  users      = yamldecode(file("users.yaml"))
  namespaces = ["default", "frontend", "backend"]
  members = flatten(
    [for k, v in local.users :
      [for ns in v :
        { user = k, namespace = ns }
      ]
    ]
  )
}

resource "google_project_iam_custom_role" "gke_minimal_permissions" {
  role_id     = "minimalGKEPermissions"
  title       = "Minimal GKE permissions"
  permissions = ["container.clusters.get", "container.clusters.list"]
  project     = "project-name"
  description = "This role provides minimal GKE permissions"
}

resource "google_project_iam_member" "gke_minimal_role" {
  for_each = local.users

  member  = "user:${each.key}"
  role    = google_project_iam_custom_role.gke_minimal_permissions.name
  project = "project-name"

  depends_on = [google_project_iam_custom_role.gke_minimal_permissions]
}

resource "kubernetes_role" "resource_viewer" {
  for_each = toset(local.namespaces)

  metadata {
    name      = "resource-viewer"
    namespace = each.key
  }

  rule {
    api_groups = [""]
    resources  = ["*"]
    verbs      = ["get", "list", "watch"]
  }
}

resource "kubernetes_role_binding" "resource_viewer_users" {
  for_each = { for member in local.members : "${member.user}-${member.namespace}" => member }

  metadata {
    name      = "resource-viewer-user"
    namespace = each.value.namespace
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.resource_viewer[each.value.namespace].metadata[0].name
  }

  subjects {
    api_group = "rbac.authorization.k8s.io"
    kind      = "User"
    name      = each.value.user
  }
}

Final words

GCP IAM is a great tool to manage GKE permissions, if you do not need fine-grained control, however if more precise permissions are required, Kubernetes RBAC is the way to go. As seen above, Kubernetes RBAC can be easily managed in an automated way with Terraform.

How We Chose Our Goteleport Setup

2022-07-22T00:00:00+00:00

Recently, I had the pleasure of setting up GoTeleport in highly available fashion. As described in their GoTeleport docs:

Teleport is a certificate authority and access plane for your infrastructure. With Teleport you can:

Use a single solution to access your SSH servers, Kubernetes clusters, databases, desktops, and web applications.

Define sophisticated access policies for every component of your infrastructure, with fine-grained audit logs and session recordings.

Automatically on– and off-board users via integrations with single sign-on providers like GitHub, Okta, and Google Workspace.

We required high availability as our infrastructure would be reachable only via Teleport, so downtime meant users wouldn’t be able to reach required servers, thus we wanted no downtime or at most few minutes of it if one of Teleport components would fail.

Initial setup

Teleport has the following data types:

Core cluster state - cluster configuration and identity
Audit events - events from the audit log
Session recordings - raw terminal recordings of interactive sessions
Teleport instance state - ID and credentials of a non-auth teleport instances

Each of the data type has its own supported storage backends - default is a local directory, but GCP/AWS are also supported (teleport instance state data type supports only local directory). Core cluster state can also be stored in etcd, which is great as GCP/AWS were not an option for us.

Our initial setup was 1 Load Balancer, 2 Teleport proxy servers, 2 Teleport auth servers and 3 etcd servers:

This setup had the following benefits:

Different Teleport components were in different data centers
Easy to scale

Different data centers

Different components were in different data centers, meaning if Teleport Proxy 1 datacenter would experience issues, Teleport Proxy 2 would still be up (same for Teleport Auth and etcd servers). This means that some serious problems would need to occur for Teleport to be completely down.

Scaling

This setup also made it effortless to scale the Teleport cluster, as you could just add required components.

However, this setup has few drawbacks.

Audit events and session recordings are on separate Teleport auth servers
Price
Speed

Audit events and session recordings

If you were to try to inspect logs/recordings, you wouldn’t see the whole picture. This wasn’t a dealbreaker as audit events could be aggregated easily as they are in JSON format. Each session recording is in a separate file, so gathering them in one place wouldn’t be an issue. This would be a bigger problem if one would only rely on Teleport Web UI to view this information.

Price

More servers you have, more money you pay. Though the costs weren’t concerning, it still felt a little of an overkill as our infrastructure doesn’t consist of many servers or users.

Speed

This was the part which caused us to reconsider our setup. Connection time increased by 3-4 seconds, and it was even worse via VPN. This doesn’t sound too bad, but it felt too much for our infrastructure.

Current setup

After some discussion, we decide to go with much simpler option - Active/Passive setup with 2 servers each of them has both Teleport Proxy and Teleport Auth and stores all data types locally:

Compared to original setup, we observed following advantages:

Speed increase
Cost reduction
Fewer things to maintain
Audit events and session recordings in one place

Speed increase

With this setup, connection time increase is up to 2 seconds via VPN, and it is barely noticeable without VPN.

Cost reduction

From 8 servers, it went down to 2, which naturally reflects in costs.

Fewer things to maintain

Apart from cost reduction, you also have fewer things to maintain, it is a lot easier to secure and keep software up to date of 2 servers than 8.

Audit events and session recordings

Now as you have only 1 active Teleport Auth server at a time all session recordings and audit event logs will be in one place, so aggregating is no longer necessary and Teleport Web UI become reliable for inspecting them.

Sadly, everything has its disadvantages:

Core cluster state backups are a must
Harder to scale
Downtime

Backups

In the original setup, core cluster state was stored in etcd cluster, meaning if one of the Teleport Auth servers would go down, the second server would be fine. Now, if an active Teleport Auth server goes down, and you don’t have a backup of a core cluster state, you will lose all user data. Luckily, it is an SQLite file, so backing it up is a minor inconvenience.

Scaling

Since Teleport Proxy and Auth are on the same server, it became a lot harder to scale. Scaling Teleport Auth in this setup is impossible and to scale Teleport Proxy you would need to separate components.

Downtime

If an active Teleport server goes under, you would instantly have downtime, which would last until you switch DNS records to point to the passive server. Furthermore, if backed up core cluster state is not present on a passive server, there would be additional downtime until it would be moved there. As our cluster state is backed up few times a day, and it is always present on a passive server, downtime wouldn’t last longer than few minutes, which was fine with us.

Possible user data inconsistency is another minor problem that rises due to this setup. If the active Teleport server goes down some hours after the latest backup, a passive server might not have the newest information. One of the scenarios where user could feel it would be password change, if a user changes the password and the backup didn’t go through yet, after the switch to passive server user’s password would be the old one. Luckily, it is something that we are alright with.

Wrapping up

As discussed, each setup has its advantages and disadvantages, so in the end we had to make a choice and sacrifice some advantages of one setup to get the benefits of another. Fortunately, our requirements allowed us some downtime, which in turn led to faster connection times, reduced maintenance work and costs.