LiteLLM

In this tutorial, we will guide you through the steps to create a Chainlit application integrated with LiteLLM Proxy

The benefits of using LiteLLM Proxy with Chainlit is:

• You can call 100+ LLMs in the OpenAI API format
• Use Virtual Keys to set budget limits and track usage
• see LLM API calls in a step in the UI, and you can explore them in the prompt playground.

​

Prerequisites

Before getting started, make sure you have the following:

• A working installation of Chainlit
• The OpenAI package installed
• LiteLLM Proxy Running
• A LiteLLM Proxy API Key
• Basic understanding of Python programming

​

Step 1: Create a Python file

Create a new Python file named app.py in your project directory. This file will contain the main logic for your LLM application.

​

Step 2: Write the Application Logic

In app.py, import the necessary packages and define one function to handle messages incoming from the UI.

from openai import AsyncOpenAI
import chainlit as cl
client = AsyncOpenAI(
    api_key="anything",            # litellm proxy virtual key
    base_url="http://0.0.0.0:4000" # litellm proxy base_url
)

# Instrument the OpenAI client
cl.instrument_openai()

settings = {
    "model": "gpt-3.5-turbo", # model you want to send litellm proxy
    "temperature": 0,
    # ... more settings
}

@cl.on_message
async def on_message(message: cl.Message):
    response = await client.chat.completions.create(
        messages=[
            {
                "content": "You are a helpful bot, you always reply in Spanish",
                "role": "system"
            },
            {
                "content": message.content,
                "role": "user"
            }
        ],
        **settings
    )
    await cl.Message(content=response.choices[0].message.content).send()

​

Step 3: Run the Application

To start your app, open a terminal and navigate to the directory containing app.py. Then run the following command:

chainlit run app.py -w

The -w flag tells Chainlit to enable auto-reloading, so you don’t need to restart the server every time you make changes to your application. Your chatbot UI should now be accessible at http://localhost:8000.