Dataset for a Run
1. Download with Run commmand
The first method involves downloading your dataset directly using a run command. This method works well with tools such as wget
or the kaggle API
. Here is an example of how to use this approach:
1.1 Using YAML
run:
- workdir: /root
command: |-
wget [your-wget-command]
kaggle datasets download -d [dataset]
In the provided example, replace [your-wget-command] and [dataset] with your actual wget command and the kaggle dataset identifier respectively.
1.2 Using Web Console
In the VESSL Web Console, locate the Start commands
section and simply type in the command.
2. Import a VESSL Dataset
VESSL Dataset supports multiple data sources. Register your dataset in VESSL Web Console and reference it when you create a run.
2.1 Using YAML
You can download data from a VESSL Dataset into your run container with import
field.
import:
/input: vessl-dataset://{organizationName}/{datasetName}
2.2 Using Web Console
Select your dataset from the list of VESSL Dataset under Import > Dataset
option.
3. Import a Object Storage
If you haven’t register your dataset as a VESSL Dataset, you can still import AWS S3 bucket or Google Cloud Storage.
3.1 Using YAML
Download the dataset from the cloud provider bucket into the run container with import
field.
Import from AWS S3
Import from Google Cloud Storage
import:
/input: s3://{bucketName}/{prefix}
3.2 Using Web Console
Choose the cloud provider option and specify the Bucket Name
and Prefix
.
4. Import the Uploaded Local Files
You can manually upload local files via the Web Console as follows, and then refer to them in your run.
5. Mount a VESSL Dataset
Among the registered VESSL Dataset, the On-Premise Storage type can be directrly mount to your run container without donwloading a copy of it.
5.1 Using YAML
Directly mount and access the dataset in the run container with mount
field.
mount:
/input: vessl-dataset://{organizationName}/{datasetName}
5.2 Using Web Console
Select your dataset from the list of VESSL Dataset under Mount > Dataset
option.
6. Mount an On-Premise Storage
Similar to importing an Object Storage, you can mount the On-Premise Storage type of data without registering it as a VESSL Dataset.
6.1 Using YAML
With mount
field, you can mount hostpath or nfs dataset and directly access it in the run container.
Hostpath
NFS
mount:
/input: hostpath://{path}
6.2 Using Web Console
Choose the dataset mount option(hostpath
or nfs
) and specify the Server
and Path
.
What’s Next
For more detailed YAML reference and definitions, please visit: