Dataset for a Run

Multiple ways to configure dataset.

1. Download with Run commmand

The first method involves downloading your dataset directly using a run command. This method works well with tools such as wget or the kaggle API. Here is an example of how to use this approach:

1.1 Using YAML

Using wget and kaggle API

run:
  - workdir: /root
    command: |-
      wget [your-wget-command]
      kaggle datasets download -d [dataset]

In the provided example, replace [your-wget-command] and [dataset] with your actual wget command and the kaggle dataset identifier respectively.

1.2 Using Web Console

In the VESSL Web Console, locate the Start commands section and simply type in the command.

2. Import a VESSL Dataset

VESSL Dataset supports multiple data sources. Register your dataset in VESSL Web Console and reference it when you create a run.

2.1 Using YAML

You can download data from a VESSL Dataset into your run container with import field.

import: 
  /input: vessl-dataset://{organizationName}/{datasetName}

2.2 Using Web Console

Select your dataset from the list of VESSL Dataset under Import > Dataset option.

3. Import a Object Storage

If you haven’t register your dataset as a VESSL Dataset, you can still import AWS S3 bucket or Google Cloud Storage.

3.1 Using YAML

Download the dataset from the cloud provider bucket into the run container with import field.

• Import from AWS S3

• Import from Google Cloud Storage

  import:
    /input: s3://{bucketName}/{prefix}

3.2 Using Web Console

Choose the cloud provider option and specify the Bucket Name and Prefix.

4. Import the Uploaded Local Files

You can manually upload local files via the Web Console as follows, and then refer to them in your run.

5. Mount a VESSL Dataset

Among the registered VESSL Dataset, the On-Premise Storage type can be directrly mount to your run container without donwloading a copy of it.

5.1 Using YAML

Directly mount and access the dataset in the run container with mount field.

mount:
  /input: vessl-dataset://{organizationName}/{datasetName}

5.2 Using Web Console

Select your dataset from the list of VESSL Dataset under Mount > Dataset option.

6. Mount an On-Premise Storage

Similar to importing an Object Storage, you can mount the On-Premise Storage type of data without registering it as a VESSL Dataset.

6.1 Using YAML

With mount field, you can mount hostpath or nfs dataset and directly access it in the run container.

• Hostpath

• NFS

  mount:
    /input: hostpath://{path}

6.2 Using Web Console

Choose the dataset mount option(hostpath or nfs) and specify the Server and Path.

What’s Next

For more detailed YAML reference and definitions, please visit:

YAML Reference

A complete descriptions of YAML references.

YAML Cheat Sheet

A complete list of YAML definitions.