Tsuru in Production with Kubernetes

This post will teach how to install Tsuru in Kubernetes and how to add the cluster to be used by Tsuru.

This how-to was made using Digitalocean Kubernetes. We will update the article when tested with other kubernetes like (AKS, GKE, EKS…).

Here we assume that you have a kubernetes cluster running and the kubectl context properly configured.

Installing Helm/Tiller

First install helm: https://github.com/helm/helm/blob/master/docs/install.md

With helm installed, lets install tiller. (it’s just one way to install tiller, you can see other ways at: https://github.com/helm/helm/blob/240e539cec44e2b746b3541529d41f4ba01e77df/docs/rbac.md

$ cat rbac-tiller-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
$ kubectl create -f rbac-tiller-config.yaml

Now lets install tiller:

helm init --service-account tiller

Great! now you can install cert-manager and tsuru using helm charts :)

Prepare cluster to install Cert-manager

We will use cert-manager to manage tls certificates with letsencrypt.

You can get the steps from https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html#steps, but tsuru charts will install cert-manager as a dependency, so you only need to do the follow steps:

Create the custom-resource:

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml

And label the namespace that we will install cert-manager (without this label cert-manager-webhook will fail to validate certificates)

Create the namespace and the label:

kubectl create namespace tsuru-manager
kubectl label namespace tsuru-manager certmanager.k8s.io/disable-validation=true

Installing Tsuru

To install Tsuru and its dependencies we will use a helm chart (thanks for gfleury who created the chart).

First clone the chart:

git clone https://github.com/tsuru/tsuru-helm-chart && cd tsuru-helm-chart

Install dependencies:

helm dep up .

Install the chart with ingress disabled (we will enable it later):

helm install --name tsuru --namespace tsuru-manager . \
    --set ingress.enabled=false

Create the letsencrypt issuer (change the email field!!):

$ cat letsencrypt-prod.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod 
  namespace: tsuru-manager
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: YOUR_EMAIL@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-prod-pk
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx
$ kubectl apply -f letsencrypt-prod.yaml

Now you need to get the ingress-controller loadbalancer IP:

export LB_HOSTNAME=$(kubectl -n tsuru-manager get services tsuru-nginx-ingress-controller -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
echo $LB_HOSTNAME

If you have a domain, it’s time to point the dns to LB_HOSTNAME.

In order to use with kube-router, you need to add a wildcard domain pointing to the LB_HOSTNAME.

If you don’t have or don’t want to use a domain now, you can use nip.io.

Changing hostnames in tsuru installation:

$ HOSTNAME="cloud.$LB_HOSTNAME.nip.io" # or your own domain
helm upgrade tsuru --namespace tsuru-manager . \
  --set ingress.enabled=true \ 
  --set ingress.domain=$HOSTNAME \  
  --set ingress.tls[0].hosts[0]=tsuru.$HOSTNAME \
  --set docker-registry.ingress.hosts[0]=registry.$HOSTNAME \
  --set docker-registry.ingress.tls[0].hosts[0]=registry.$HOSTNAME \
  --set ingress.tls[0].secretName=tsuru-api \
  --set docker-registry.ingress.tls[0].secretName=registry-tls \
  --set registry.enabled=true \
  --set registry.url=registry.$HOSTNAME

Now you have tsuru installed with tls!!

Configuring Tsuru

Create the admin user on tsuru:

export POD_NAME=$(kubectl -n tsuru-manager get pods -l “app=tsuru,release=tsuru” -o jsonpath=”{.items[0].metadata.name}”)
kubectl -n tsuru-manager exec -it $POD_NAME tsurud root-user-create admin@example.com # CHANGE IT TO YOUR ADMIN USER #

Create the tsuru-target and log in:

tsuru target-add -s default https://$HOSTNAME
tsuru login

Create pool, and add kubernetes cluster to be managed by tsuru:

tsuru pool-add mypool -p -d --provisioner kubernetes
export SA_NAME=$(kubectl -n tsuru-manager get sa -l "app=tsuru-tsuru-api,release=tsuru"  -o jsonpath="{.items[0].secrets[0].name}")
  
export K8S_TOKEN=$(kubectl -n tsuru-manager get secret $SA_NAME -o jsonpath="{.data.token}" | base64 -d)
kubectl -n tsuru-manager get secret $SA_NAME -o jsonpath="{.data.ca\.crt}" | base64 -d > ca.crt
tsuru cluster-add kube-prod kubernetes \
  --addr=https://kubernetes.default.svc.cluster.local \  
  --cacert ca.crt \ 
  --custom token=$K8S_TOKEN \
  --default

We need to update nodes to add pool:

tsuru node list -q | xargs -I{} sh -c "tsuru node-update {} pool=mypool"

Create one team:

tsuru team-create admin

Create and Deploy tsuru-dashboard app:

tsuru app-create dashboard --router-opts tls-acme=true
tsuru app-deploy -a dashboard -i tsuru/dashboard

Build Platforms:

tsuru platform-add python
tsuru platform-add go

Create an app to test:

mkdir example-go && cd example-go
wget https://raw.githubusercontent.com/tsuru/platforms/master/examples/go/tsuru.yml
wget https://raw.githubusercontent.com/tsuru/platforms/master/examples/go/web.go
tsuru app-create example-go go 
tsuru app-deploy -a example-go .

Check the app info and get the url:

tsuru app-info -a example-go

Please leave us a comment telling if everything went well, if you had any problems or if you needed an extra step, etc …