Multi-Region Kubernetes Cluster Federation in Virtuozzo PaaS

If you have multiple Kubernetes (K8s) clusters in different regions and need to run the same application across all of them, it is reasonable to use the so-called Kubernetes Cluster Federation or KubeFed.

Kubernetes Federation is a multi-cloud or multi-region implementation for centralized deployment and management of applications and services across multiple Kubernetes clusters.

Federation allows to create a set of interconnected clusters, into which you can deploy applications according to specific rules. The most demanded use case for Federation is scaling applications across different clusters located in different regions. You can bundle components of your application to federated resources and distribute the specified number of replicas across all member clusters.

In this article, we’ll demonstrate how to set up Virtuozzo Kubernetes Services (JK8s) as a Federation located across different regions.

Federation Prerequisites

Supposedly, within one Virtuozzo PaaS, we have five clusters in different regions, and we'd like to deploy applications to any of these clusters. One of them is a Host Cluster that acts as a Federation Control Plane, it propagates and pushes configurations to the Member Clusters.

Thus, we need to decide which payload we want to distribute, and which member clusters should handle it.

So, let’s get down to business and create a Federation in Virtuozzo PaaS.

Sign in to your account and create two Kubernetes clusters in different regions. Actually, you may create as many as you need, though, but here we create a Host Cluster and Member Cluster 1 only. All of the actions below can be applied to any number of Member Clusters. So, deploy:

• Federation Host Cluster: fedhost.vip.jelastic.cloud
• Federation Member Cluster: member1.demo.jelastic.com

Remote Access to the Clusters

The next step is to establish remote access to the clusters.

Log in to the master node of the Host Cluster via SSH and start with the configuration. Some command's output will be shown to let you be sure you are doing right:

1. First, install the KubeFed chart with helm in kube-federation-system namespace :
Add repository:

fedhost~$ helm repo add kubefed-charts

https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts

Install the latest version available of kubefed. Here and below, we use 0.7.0 as the current version:

fedhost~$ helm install kubefed kubefed-charts/kubefed --version 0.7.0 --namespace kube-federation-system --create-namespace

2. Download the latest version of kubefedctl command line tool as well and copy it to /usr/local/bin directory:

fedhost~$ wget

https://github.com/kubernetes-sigs/kubefed/releases/download/v0.7.0/kubefedctl-0.7.0-linux-amd64.tgz

fedhost~$ tar xvf kubefedctl-0.7.0-linux-amd64.tgz

fedhost~$ mv kubefedctl /usr/local/bin

3. To let KubeFed federating deployments, it must be able to interact with all of the selected Member Clusters. To get this, you can use the following RBAC config file to create the necessary role to ensure connection from the Host Cluster. Log in to the master node of the Member Cluster via SSH and create a configuration file, for example, member1.yaml, and paste the content below into it.

apiVersion: v1
kind: ServiceAccount
metadata:
 labels:
   name: member1
 name: member1
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 labels:
   name: member1
 name: member1
rules:
 - apiGroups: ['*']
   resources: ['*']
   verbs: ['*']
 - nonResourceURLs: ['*']
   verbs: ['*']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 labels:
   name: member1
 name: member1
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: member1
subjects:
 - kind: ServiceAccount
   name: member1
   namespace: default

Apply the configuration file:

member1~$ kubectl apply -f member1.yaml

4. To gain access to all members, you have to create a context for each of them. Each context contains a K8s cluster name, a cluster endpoint, a username with credentials, and the namespace.

Where the credentials are:

• RBAC token
• certificate

Get the token name of the newly created service account member1

member1~$ kubectl get secret | grep member1

Retrieve its content and copy into the clipboard:

member1~$ kubectl describe secret member1-token-zkctp

5. Create user kubefed-member1 and provide token for it from the clipboard:

fedhost~$ kubectl config set-credentials kubefed-member1

--token='eyJhbGciOiJSUzI1Ni…….JYNCzwS4F57w'

6. Retrieve the Member Cluster endpoint that the Host Cluster will be connecting to:

member1~$ cat /root/.kube/config | grep server

Set up the cluster name kubefed-remote-member1 and add the Member Cluster endpoint:

fedhost~$ kubectl config set-cluster kubefed-remote-member1

--server='https://k8sm.member1.demo.jelastic.com:6443'

Note: In case the Member Cluster is located on a different Virtuozzo platform, the endpoint is different as well. Use an API Endpoint from a successful installation window. For example:

fedhost~$ kubectl config set-cluster kubefed-remote-member1

--server='https://member1.demo.jelastic.com/api/'

7. Retrieve the certificate and copy its content into the clipboard.

member1~$ cat /root/.kube/config | grep certificate-authority-data

Substitute the <certificate> with the value from the clipboard and add to the context:

fedhost~$ kubectl config set clusters.kubefed-remote-member1.certificate-authority-data '<certificate>'

8. Finally, describe the context itself for the Member Cluster using the proper cluster name, context name, and user name:

fedhost~$ kubectl config set-context member1 --cluster=kubefed-remote-member1 --user=kubefed-member1 --namespace=default

Repeat steps 3-8 for each Member Cluster in Federation.

Cluster Remote Access Testing

Now, the Member Clusters are described in the Host Cluster configuration file /root/.kube/config. In case everything was done properly, you may interact with any cluster configured, simply by selecting the context. Let’s see the list of nodes on both clusters kubefed-remote-member1 and fedhost:

fedhost~$ kubectl --context member1 get nodes
fedhost~$ kubectl --context fedhost get nodes

Joining the Federation

Now that the communication is available between two clusters located in different regions, we can use the kubefedctl tool to let them both join the Federation.

Add Host Cluster to Federation:

fedhost~$ kubefedctl join fedhost --v=2 --host-cluster-context fedhost

--kubefed-namespace=kube-federation-system

Add Member Cluster to Federation:

fedhost~$ kubefedctl join member1 --v=2 --host-cluster-context fedhost

--kubefed-namespace=kube-federation-system

If everything passed smoothly with no errors, you should have similar output:

To make sure that the join is working, you can check the status of the Federation with the command:

fedhost~$ kubectl -n kube-federation-system get kubefedclusters

Congratulations! Now you are aware of how to build Kubernetes Federation across multiple regions of Virtuozzo PaaS service providers. The next step is to start federating the resources, and we’ll cover this in more detail within our upcoming publication.

Want to see how it works for you?
Give it a try with a free trial from a local Virtuozzo Application Platform provider — and see how easy cloud can be.

Want to offer powerful cloud services under your brand?
Get in touch with Virtuozzo to learn how to install the Application Platform on your own infrastructure.

Related Articles

How to Federate Resources across Kubernetes Clusters for Unified Deployment

Jelastic Released Kubernetes Package with Integrated Auto-Clustering and Pay-per-Use Pricing Model

Kubernetes Cluster Setup with Automated Scaling and Pay-per-Use Pricing