Skip to main content

VertexAI - Google [Gemini, Model Garden]

Open In Colab

Pre-requisites​

  • pip install google-cloud-aiplatform
  • Authentication:
    • run gcloud auth application-default login See Google Cloud Docs
    • Alternatively you can set application_default_credentials.json

Sample Usage​

import litellm
litellm.vertex_project = "hardy-device-38811" # Your Project ID
litellm.vertex_location = "us-central1" # proj location

response = litellm.completion(model="gemini-pro", messages=[{"role": "user", "content": "write code for saying hi from LiteLLM"}])

OpenAI Proxy Usage​

Here's how to use Vertex AI with the LiteLLM Proxy Server

  1. Modify the config.yaml

Use this when you need to set a different location for each vertex model

model_list:
- model_name: gemini-vision
litellm_params:
model: vertex_ai/gemini-1.0-pro-vision-001
vertex_project: "project-id"
vertex_location: "us-central1"
- model_name: gemini-vision
litellm_params:
model: vertex_ai/gemini-1.0-pro-vision-001
vertex_project: "project-id2"
vertex_location: "us-east"
  1. Start the proxy
$ litellm --config /path/to/config.yaml

Set Vertex Project & Vertex Location​

All calls using Vertex AI require the following parameters:

  • Your Project ID
import os, litellm 

# set via env var
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811" # Your Project ID`

### OR ###

# set directly on module
litellm.vertex_project = "hardy-device-38811" # Your Project ID`
  • Your Project Location
import os, litellm 

# set via env var
os.environ["VERTEXAI_LOCATION"] = "us-central1 # Your Location

### OR ###

# set directly on module
litellm.vertex_location = "us-central1 # Your Location

Model Garden​

Model NameFunction Call
llama2completion('vertex_ai/<endpoint_id>', messages)

Using Model Garden​

from litellm import completion
import os

## set ENV variables
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
os.environ["VERTEXAI_LOCATION"] = "us-central1"

response = completion(
model="vertex_ai/<your-endpoint-id>",
messages=[{ "content": "Hello, how are you?","role": "user"}]
)

Gemini Pro​

Model NameFunction Call
gemini-procompletion('gemini-pro', messages), completion('vertex_ai/gemini-pro', messages)
Model NameFunction Call
gemini-1.5-procompletion('gemini-1.5-pro', messages), completion('vertex_ai/gemini-pro', messages)

Gemini Pro Vision​

Model NameFunction Call
gemini-pro-visioncompletion('gemini-pro-vision', messages), completion('vertex_ai/gemini-pro-vision', messages)
Model NameFunction Call
gemini-1.5-pro-visioncompletion('gemini-pro-vision', messages), completion('vertex_ai/gemini-pro-vision', messages)

Using Gemini Pro Vision​

Call gemini-pro-vision in the same input/output format as OpenAI gpt-4-vision

LiteLLM Supports the following image types passed in url

Example Request

import litellm

response = litellm.completion(
model = "vertex_ai/gemini-pro-vision",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Whats in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
}
}
]
}
],
)
print(response)

Chat Models​

Model NameFunction Call
chat-bison-32kcompletion('chat-bison-32k', messages)
chat-bisoncompletion('chat-bison', messages)
chat-bison@001completion('chat-bison@001', messages)

Code Chat Models​

Model NameFunction Call
codechat-bisoncompletion('codechat-bison', messages)
codechat-bison-32kcompletion('codechat-bison-32k', messages)
codechat-bison@001completion('codechat-bison@001', messages)

Text Models​

Model NameFunction Call
text-bisoncompletion('text-bison', messages)
text-bison@001completion('text-bison@001', messages)

Code Text Models​

Model NameFunction Call
code-bisoncompletion('code-bison', messages)
code-bison@001completion('code-bison@001', messages)
code-gecko@001completion('code-gecko@001', messages)
code-gecko@latestcompletion('code-gecko@latest', messages)