Serve YAML

VESSL Serve is configured through a single YAML file.

Revision YAML Field Types

Name

Service name created by the user.

Name	Type	Required	Description
name	string	Requried	Service name

name: vessl-test-service

Message

Write a message for the Service Revision. We recommend writing an identical message for each revision to distinguish them.

Name	Type	Required	Description
message	string	Requried	Description of the revision.

message: vessl-yaml-revision

Image

The name of the docker image that will be used for inference. You can also use a custom docker image.

Name	Type	Required	Description
image	string	Requried	Docker image url.

image: quay.io/vessl-ai/torch:2.3.1-cuda12.1-r5

Resources

Write down the compute resources you want to use for Service. You can specify the resources you want to use in the Cluster settings.

Name	Type	Required	Description
cluster	string	Optional	The cluster to be used for the run. (default: VESSL-managed cluster)
preset	string	Optional	The preset to be used for the run.
spot	one of (empty, SpotOnly, SpotFirst, OnDemandOnly)	Optional	Whether to use spot instances for the run or not.

resources:
  cluster: vessl-gcp-oregon
  preset: gpu-l4-small-spot

Import

Write the datasets and volumes imported in the Revision container when the Revision is deployed.

Prefix	Type	Required	Description
git://	string	Optional	Mount a git repository into your container. The repository will be cloned into the specified mount path when container starts.
hf://	string	Optional	Mount a huggingface repository into your container. The repository will be cloned into the specified mount path when container starts.
vessl-dataset://	string	Optional	Mount a dataset stored in VESSL. Replace organizationName with the name of your organization and datasetName with the name of the dataset.
s3://	string	Optional	Mount an AWS S3 bucket into your container. Replace bucketName with the name of your S3 bucket and path with the path to te file or folder you want to mount.
gs://	string	Optional	Mount an GCP GCS bucket into your container. Replace bucketName with the name of your GCS bucket and path with the path to te file or folder you want to mount.
vessl-model://	string	Optional	Mount a VESSL Model into your container.
vessl-artifact://	string	Optional	Mount a VESSL Artifact into your container.

import:
  /root/git-examples: git://github.com/vessl-ai/examples
  /root/hf: hf:///opt/data1
  /input/config: gs://config.yaml
  /input/data2: s3://192.168.10.2:~/
  /input/data3: vessl-dataset://{organization_name}/{dataset_name}

Mount

Write down the volumes that will be mounted in the Revision Service container when the Revision is deployed.

Prefix	Type	Required	Description
hf://	string	Optional	Mount a huggingface repository into your container. The repository will be cloned into the specified mount path when container starts.
hostpath://	string	Optional	Mount a file or directory from the host node’s filesystem into your container. Replace path with the path to the file or folder you want to mount.
nfs://	string	Optional	Mount a Network File System(NFS) into your container. Replace ip with the IP address of your NFS server and path with the path to the file or folder you want to mount.
cifs://	string	Optional	Mount a Command Internet File System(CIFS) into your contianer. Replace ip with the IP address of your NFS server and path with the path to the file or folder you want to mount.
gcs-fuse://	string	Optional	Mount a GCS with FUSE csi driver.

Run

Write down what commands you want to run on the service container when the Revision is deployed.

Name	Type	Required	Description
workdir	string	Optional	The working directory for the command.
command	string	Required	The command to be run.

run:
  - workdir: /root/git-examples
    command: |
      vessl model launch service.py:Service -p 3000

Env

Write down the environment variables that will be set in the Revision Service container.

Name	Type	Required	Description
env	map	Optional	Key-value pairs for environment variables in the run container.

env:
  learning_rate: 0.001
  batch_size: 64
  optimizer: sgd

Ports

Write down the ports and protocols that the Revision Service container should open.

Name	Type	Required	Description
name	string	Required	The name for the opening port.
type	string	Required	The protocol the port will use.
port	int	Required	The number of the port.

ports:
  - name: web-service
    type: http
    port: 8000
  - name: web-service-2
    type: http
    port: 8001

Autoscaling

Sets the value for how the Revision Pod will autoscale.

Name	Type	Required	Description
min	string	Required	Minimum number of Pods to autoscale.
max	string	Required	Maximum number of Pods to autoscale.
metric	int	Required	Determine what conditions you want to autoscale under. You can select cpu, gpu, memory, and custom
target	int	Required	A metric threshold percentage. If the metric is above the target, then the Autoscaler automatically scale-out.

autoscaling:
  min: 1
  max: 3
  metric: cpu
  target: 50