You have a shiny new kubernetes cluster and you’re ready to host some applications. That’s great, but if you’re not intimately familiar with the applications you can prepare to have a bad time. Even in 2020, most applications don’t take well to being distributed across ephemeral containers. In my experience this means spending weeks (or months) helping development teams debug their application code*.

In some cases that means giving them control over the infrastructure underlying their kubernetes apps. One of the benefits of using kubernetes is the ability to run workloads from many different teams under one control plane. Following the principle of least privilege, you want to delegate the development team the least number of privileges possible but still enough to let them do their jobs. Kubernetes is among the many systems that have adopted Role-Based Access Control (RBAC) for authorization. How to best use RBAC is beyond the scope of this post but we found a neat way to do it with terraform, which is an awesome tool for managing your cloud resources!

* but not any of our customers.

Steps (what this Terraform modules does for you)

This Terraform module is admittedly not something you would want to use in a production system and is something of a hack for the unique use-case that is Azure. Kubernetes doesn’t manage users itself, it’s set up to defer authentication to an external source like IAM. AWS does a great job with their EKS service by integrating the two right out of the box (or more aptly out of the cloud). Microsoft Azure lacks the sophistication that AWS has cultivated while being the number one cloud provider since it invented the idea so only recently began allowing authentication with their identity provider, Azure Active Directory. Previously AKS came with one superuser that authenticated with X509 Client Certs.

Create a Kubernetes Namespace, Role and Rolebinding

Delineating code into modules and projects and controlling the “blast radius” of your Terraform project is a complex topic with no clear answers. At Rhythmic, we’ve had success separating code into what we’ve termed “projects”, which live inside the main Terraform repository for the account. It isn’t the sexiest Terraform setup, but its simplicity allows easy onboarding of developers and the ability to quickly look at Terraform code and know what’s deployed into an environment, so really, its simplicity is its strength. Those basic principles and what the repositories it leads to look like will have to be a blog post all their own

Creates an RSA Key Pair

Kubernetes has several objects that make managing permissions easy; Namespaces, Roles, and RoleBindings. The concept of namespaces are used across computing, most notably in the Linux kernel, and like in the Linux Kernel they isolate processes and other objects within a subsection, called a “Namespace”, in your Kubernetes cluster. Within any given namespace there will be many users and privileged containers that need to access the Kubernetes API. Instead of creating unique permissions for every different user Kubernetes allows you to create “Roles” that list the different actions it can apply, the resources it can apply them to, and the API groups it can talk to. You then delegate each of these roles to users with RoleBindings. The default Role in this module creates a namespace admin user that has full control over their namespace.

Creates a Certificate Signing Request (CSR)

This is used to identify our new users. If you’re unfamiliar with public-key cryptography you can think of a key pair as a lock and key. Granted they are only two text files but the analogy is useful because you use the public key to “lock” your message (by encrypting it) and then only the key can “unlock” that message (decrypt it). This analogy doesn’t capture another benefit of public-key cryptography and that’s authentication. The private key can also be used to encrypt messages that can only be decrypted by the public key. That means you can encrypt a message with your private key and any recipient of the message can verify that it was sent by you by decrypting it with the public key. This second benefit of public-key cryptography is what we’ll be taking advantage of today.

Creates a Kubeconfig File

The CSR is an encoded document that asks a Certificate Authority (CA) to recognize your private key as you. This is how you’re going to “sign-in”. If you’re unfamiliar with cryptography you can think of the private key as a fancy password and the CSR as you giving them your email so you can create an account. This is a vast oversimplification but it conveys how these pieces are going to come together.

There’s no Terraform resource for a CertificateSiginginRequest so we’re templating the YAML files and using kubectl to create it. This should change with Terraform’s recent announcement about their kubernetes alpha provider, which will let you create any Kubernetes in Terraform with a specification that directly mimics the YAML configs.

# certificate_signing_request.yaml.tpl
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${name}
spec:
  groups:
  - system:authenticated
  - ${namespace}
  request: ${base64_csr}
  usages:
  - digital signature
  - key encipherment
  - client auth

Creates a Kubeconfig File

With great power comes great responsibility, so be careful with these files! You will want to securely give these to your users because this file is how they will authenticate to the cluster using the kubectl tool. Hopefully, you’re familiar with kubectl and some of the awesome kubectl plugins. If you read through this file you can see that it contains all the information needed to connect with Kubernetes including the server’s address and the certificate data we just created.

# kubeconfig.yaml.tpl
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${CA_DATA}
    server: ${API_SERVER}
  name: ${CLUSTER_NAME}
contexts:
- context:
    cluster: ${CLUSTER_NAME}
    namespace: ${namespace}
    user: ${username}
  name: ${CLUSTER_NAME}
current-context: ${CLUSTER_NAME}
kind: Config
preferences: {}
users:
- name: ${username}
  user:
    client-certificate-data: ${CLIENT_CRT_DATA}
    client-key-data: ${CLIENT_KEY_DATA}

How to run it

Let’s do an example! This is directly pulled from some code I recently wrote which is why I haven’t included all the variables. It creates a Kubernetes cluster, a namespace, and an admin. If you already have a Kubernetes cluster to test on you can see a simpler example on the GitHub repo.

Create a Kubernetes Cluster

We’re using Terraform so this is a breeze. As it so happens we have Terraform modules for Azure Kubernetes Service (AKS) and Elastic Kubernetes Service (EKS). This code was originally created to run with Azure so here’s an example with AKS. I’m going to assume enough proficiency in Terraform that you’re able to declare and fill out these variables on your own. If you need some help you can leave an issue on our AKS terraform module’s GitHub page, where there’s another example. This post isn’t explicitly on creating AKS clusters but Azure has made that process fairly straightforward if all you need is a simple, publicly available cluster.

########################################
# Service Principal
# which is required to integrate AKS with ACR
########################################
provider "azuread" {
  version = ">=0.6.0"
}

data "azuread_service_principal" "aks" {
  display_name = var.sp_display_name
}

resource "azurerm_role_assignment" "acr_contributor" {
  principal_id         = data.azuread_service_principal.aks.id
  role_definition_name = "acrpull"
  scope                = var.acr_id
}

resource "azurerm_role_assignment" "network_contributor" {
  principal_id         = data.azuread_service_principal.aks.id
  role_definition_name = "Network Contributor"
  scope                = var.default_node_pool.vnet_subnet_id
}

########################################
# the AKS cluster
########################################
module "aks" {
  source  = "rhythmictech/aks/azurerm"
  version = "5.1.0"

  CLIENT_ID                            = data.azuread_service_principal.aks.application_id
  CLIENT_SECRET                        = var.sp_password
  default_node_pool                    = var.default_node_pool
  default_node_pool_availability_zones = var.default_node_pool_availability_zones
  kubernetes_version                   = local.aks_version
  location                             = var.location
  network_profile                      = var.network_profile
  prefix                               = var.name
  rbac_enabled                         = true
  tags = var.tags
}

Create a Namespace Admin for Pierre

This bit of Terraform invokes our namespace admin module, passing in the necessary outputs from the AKS cluster created above. It will go about creating the namespace, roles, rolebindings, key pairs, and CSR requests necessary. Then it outputs kubeconfigs that can be used to connect to kubernetes!

########################################
# Kubernetes Namespace
# and our admin, named "pierre"
########################################
module "admins" {
  source = "git::https://github.com/rhythmictech/terraform-kubernetes-namespace-admins.git?ref=master"

  client_certificate     = module.aks.outputs.client_certificate
  client_key             = module.aks.outputs.client_key
  cluster_ca_certificate = module.aks.outputs.cluster_ca_certificate
  cluster_name           = module.aks.outputs.cluster_name
  host                   = module.aks.outputs.host
  name                   = "pierres-namespace"
  namespace              = "eus1-ops-pierres-namespace"
  namespace_admins = [
    "pierre"
  ]
}

As a final test it’s always best to make sure the kubeconfig files work as intended. There shouldn’t be any pods in our new namespace but this command would get any there.

#/bin/bash
kubectl \
    --kubeconfig kubeconfigs/pierre.yaml \
    --namespace eus1-ops-pierres-namespace \
    get pods

Next Steps

Kubernetes and many other systems use certificate management as an authentication mechanism internally, but this isn’t a best practice for the use-case of delegating cluster access to end-users. Certificates can be hard to revoke and they lack the conveniences of a managed identity-management system like IAM. You can read more about why not to do this here.

Another consideration is that as your cluster scales, and you need to delegate more permissions, this approach won’t scale well. User pools and roles are normally stored somewhere else like Azure Active Directory (AAD) or IAM and you’re going to need to tie in this Authentication natively. Luckily this is how Kubernetes was designed to operate. The use case for this was unique, we had apps running in an older AKS cluster that predated AKS’s AAD integration. Other services like EKS already come integrated with an authentication system (in the case of AWS that’s IAM) so you can create users through that system rather than having the cluster CA sign certificates. The process of delegating permissions out with Roles and RoleBindings will be the same though.  Tools exist to make this easier, particularly for the use case described above. One that I’ve heard good things about is Fairwinds RBAC-Manager, which can be deployed as a helm chart and help you automate RBAC in kubernetes.