Llama 3.1 deployment
Serve & deploy vLLM-accelerated Llama 3.1-8B
This example deploys a text generation API using Llama-3.1-8B and vLLM. It illustrates how VESSL AI facilitates the common logics of model deployment from launching a GPU-accelerated service workload to establishing an API server.
Upon deployment, VESSL AI also offloads the challenges in managing production models while ensuring availability, scalability, and reliability.
VESSL guides you to smooth and seamless performance with the following items:
- Autoscaling the model to handle peak loads and scale to zero when it’s not being used.
- Routing traffic efficiently across different model versions.
- Providing a real-time monitoring of predictions and performance metrics through comprehensive dashboards and logs.
Read our announcement post for more details.
What you will do
- Define a text generation API and create a model endpoint
- Define service specifications
- Deploy model to VESSL AI managed GPU cloud
Set up your environment
We’ll start with the Llama 3.1 example, which demonstrates how to deploy an AI service using a single YAML file. Follow these steps to prepare:
Deploy a vLLM Llama 3.1 Server with VESSL Service
Configure resource and environment to run vLLM Llama 3.1 server through YAML file as follows.
For YAML manifest details, refer to the YAML schema reference.
Deploy your server easily using the YAML configuration and VESSL CLI with the following command:
Upon activation, access your model via the provided endpoint, as depicted below:
vessl serve create -f quickstart.yaml
) may temporarily result in unexpected errors. If this occurs, please use VESSL CLI with Python 3.11 for the time being. We are working on it.Explore the API Documentation
Access the API documentation by appending /docs
to your endpoint URL:
Test the API with an OpenAI Client
For compatibility with OpenAI clients, install the OpenAI Python package:
Test your deployed API using the api-test.py
script. Replace YOUR-SERVICE-ENDPOINT
with your actual endpoint and execute the command below:
Troubleshooting
-
NotFound (404): Requested entity not found error while creating Revisions or Gateways via CLI:
- Use the
vessl whoami
command to confirm if the default organization matches the one where Service exists. - You can use the
vessl configure --reset
command to change the default organization. - Ensure that Service is properly created within the selected default organization.
- Use the
-
What’s the difference between Gateway and Endpoint?
- There is no difference between the two terms; they refer to the same concept.
- To prevent confusion, these terms will be unified under “Endpoint” in the future.
-
HPA Scale-in/Scale-out Approach:
- Currently, VESSL Service operates based on Kubernetes’ Horizontal Pod Autoscaler (HPA) and uses its algorithms as is. For detailed information, refer to the Kubernetes documentation.
- As an example of how it works based on CPU metrics:
- Desired replicas =
ceil[current replicas * ( current CPU metric value / desired CPU metric value )]
- The HPA constantly monitors this metric and adjusts the current replicas within the
[min, max]
range.
- Desired replicas =
Let’s go ahead and deploy the model.
Once deployed, you can check the status of the model, including the endpoint, logs, and metrics under Services.
To use the Llama 3 text generation service, execute the following curl
command. Be sure to replace ENDPOINT_URL
and API_KEY
with your own credentials. You can find your API_KEY
on the FastAPI
page under the /v1/chat/completion
section.
What’s next?
Next, let’s see how you can serve your model with serverless mode with Text Generation Inference(TGI).