On-premise Clusters

Scale up your ML infra with on-premise GPUs

Overview

In the background, VESSL Clusters leverages GPU-accelerated Docker containers and Kubernetes pods. It abstracts the complex compute backends and system details of Kubernetes-backed GPU infrastructure into an easy-to-use web interface and simple CLI commands. Data Scientists and Machine Learning Researchers without any software or DevOps backgrounds can use VESSL’s single-line CURL command to set up and configure on-premise GPU servers for ML.

VESSL’s cluster integration is composed of four primitives.

• VESSL API Server — Enables communication between the user and the GPU clusters, through which users can launch containerized ML workloads.
• VESSL Cluster Agent — Sends information about the clusters and workloads running on the cluster such as the node specifications and model metrics.
• Control plane node — Acts as the 🔗 cluster-wide control tower and orchestrates subsidiary worker nodes.
• Worker nodes — Run specified ML workloads based on the runtime spec and environment received from the control plane node.

Integrating more powerful, multi-node GPU clusters for your team is as easy as integrating your personal laptop. To make the process easier, we’ve prepared a single-line curl command that installs all the binaries and dependencies on your server.

OS, Network requirements

• OS: Ubuntu 18.04+ / Centos 7.9+
• Each machine should be connected to internet
• Nodes reachable to following port/protocols (refer to this k0s link)

Protocol	Port	Service	Direction	Notes
TCP	2380	etcd peers	controller <-> controller
TCP	6443	kube-apiserver	Worker, CLI => controller	Authenticated Kube API using Kube TLS client certs, ServiceAccount tokens with RBAC
TCP	179	kube-router	worker <-> worker	BGP routing sessions between peers
UDP	4789	Calico	worker <-> worker	Calico VXLAN overlay
TCP	10250	kubelet	Master, Worker => Host *	Authenticated kubelet API for the master node kube-apiserver
TCP	9443	k0s-api	controller <-> controller	k0s controller join API, TLS with token auth
TCP	8132	konnectivity	worker <-> controller	Konnectivity is used as “reverse” tunnel between kube-apiserver and worker kubelets

NVIDIA GPU, CUDA requirements

For NVIDIA GPUs, you must install CUDA and NVIDIA drivers. We strongly recommend to use CUDA >= 11.1 and NVIDIA Driver >= 450.80.02.

Here is the command you can use to check CUDA and NVIDIA driver versions.

nvidia-smi

Step-by-step Guide

1. Install dependencies

You can install all the dependencies required for cluster integration using a single-line curl command. The command

• Installs 🔗 Docker if it’s not already installed.
• Installs and configures 🔗 NVIDIA container runtime.
• Installs 🔗 k0s, a lightweight Kubernetes distribution, and designates and configures a control plane node.
• Generates a token and a command for connecting worker nodes to the control plane node configured above.

If you wish to use your control plane solely for the control plane node — meaning not running any ML workloads on the control plane node and only using it for admin and monitoring purposes — add a --taint-controller flag at the end of the command.

curl -sSLf https://install.dev.vssl.ai | sudo bash -s -- --role=controller

Upon installing all the dependencies, the command returns a follow-up command with a token. You can use this to add worker nodes to the control plane. If you don’t want to add an additional worker node you can skip to the next step.

curl -sSLf https://install.dev.vssl.ai | sudo bash -s -- --role worker --token '[TOKEN_HERE]'

You can confirm that your control plane and worker node have been successfully configured using a k0s command.

sudo k0s kubectl get nodes

Please try a manual installation if you encounter an error while installing with magic script.

Manual install

Install Docker

To install docker, you can refer to official docs link.

Install NVIDIA Docker

Using NVIDIA GPU in containers requires you to install nvidia-docker via this link.

• Ubuntu

# Fix apt repository
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

# NVIDIA container toolkit
DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.list | \
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo sed -i -e '/experimental/ s/^#//g' \
/etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update

# remove existed nvidia docker
apt-get remove -y libnvidia-container-tools libnvidia-container1 nvidia-container-runtime nvidia-container-toolkit nvidia-docker2
apt-get install -y nvidia-docker2

# Change default runtime to nvidia
cat << EOF > /etc/docker/daemon.json
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
EOF

systemctl restart docker

To test if a setup works, run the following container:

docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

Install k0s

k0s is an open source, all-inclusive Kubernetes distribution, which is configured with all of the features needed to build a Kubernetes cluster. VESSL recommends installing Kubernetes in on-premise machines using k0s.

• Control plane

• Worker node

curl -sSLf https://get.k0s.sh | sudo K0S_VERSION="v1.23.17+k0s.1" sh

sudo mkdir -p /etc/k0s
k0s config create > /etc/k0s/k0s.yaml

# change config values from k0s.yaml (e.g. advertise address, sans)

sudo k0s install controller -c /etc/k0s/k0s.yaml \
	--enable-worker \
    --cri-socket=docker:unix:///var/run/docker.sock \
	--kubelet-extra-args="--network-plugin=cni"
sudo k0s start

After installation, you can check the k0s status in the control plane.

k0s status
k0s kubectl get nodes

Install Helm

As a final step, install helm.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

2. Install vessl agent

First, make sure that you set the kubeconfig in your home directory.

chmod +r /var/lib/k0s/pki/admin.conf
mkdir -p ~/.kube/config
cp /var/lib/k0s/pki/admin.conf ~/.kube/config

You are now ready to integrate the Kubernetes cluster with VESSL. Make sure you have VESSL Client installed on the server and configured for your organization.

pip install vessl --upgrade

vessl configure

The following single-line command connects your Kubernetes-backed GPU cluster to VESSL. Note the —mode multi flag, specifying multi-node cluster integraiton.

vessl cluster create --name='[CLUSTER_NAME_HERE]' --mode=multi

By this point, you have successfully completed the integration.

You can use VESSL CLI command or visit 🗂️ Clusters to confirm your integration.

vessl cluster list

Destroy and delete the cluster

In order to destroy a cluster created by VESSL, follow these steps:

k0s stop
k0s reset

To complete the deletion, you may need to reboot your machine.

After destroying a cluster, you can delete it from the cluster page.

Troubleshooting

Cannot find the cluster after installation

If you can’t find the newly added cluster in the cluster page,

1. Check if kubernetes cluster is created.

k0s status
k0s kubectl get nodes

2. Check if vessl-agent pod is running in vessl namespace.

kubectl get pods -n vessl

3. Contact [e-mail] with logs from vessl-agent pods.

Node IP is assigned to other internet interface (e.g. Private IP instead of Public IP)

Add an additional —node-ip argument to —kubelet-extra-args in the k0s install command.

--kubelet-extra-args='--network-plugin=cni --node-ip=165.132.46.110'

Preventing unprivileged access to GPUs in Kubernetes

Sometimes your workloads without GPU requests can have the access to all machine GPUs. It is a native bug in NVIDIA Docker and Kubernetes, you can find the how to guide from NVIDIA in here. TLDR, you can use this command to fix.

cat << EOF > /etc/nvidia-container-runtime/config.toml
disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
accept-nvidia-visible-devices-envvar-when-unprivileged = false
accept-nvidia-visible-devices-as-volume-mounts = true

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
#no-cgroups = false
#user = "root:video"
ldconfig = "@/sbin/ldconfig.real"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
EOF

In case of relocating the Docker root repository

Please refer to this link.