LLM Providers allow you to configure the prompt playground so that it can work with any LLM provider.

​
Provider library

We currently offer the implementation for OpenAI, AzureOpenAI and Anthropic. This enables you to get started quickly without having to re-implement the connection to these LLM providers.

The HuggingFace provider example is available to help you customize it to the HugginFace-hosted model of your choice.

​
OpenAI

Required environment variables:

OPENAI_API_KEY=

​
AzureOpenAI

Required environment variables:

AZURE_OPENAI_API_KEY=
AZURE_OPENAI_API_VERSION=
AZURE_OPENAI_ENDPOINT= # something like https://mydomain.openai.azure.com
AZURE_OPENAI_DEPLOYMENT_NAME= # something like gpt-35-turbo

Optional environment variables:

AZURE_OPENAI_API_KEY="" # AZURE_OPENAI_API_KEY is mandatory so you can't set it to None, but you can set it to an empty string
AZURE_AD_TOKEN=
AZURE_AD_TOKEN_PROVIDER=
AZURE_DEPLOYMENT=

​
Anthropic

Required environment variables:

ANTHROPIC_API_KEY=

​
Custom provider

You can define your own provider to connect with your LLM provider of choice. Below is an example of how it would look like.

​
Step 1: Define the LLM Provider

The create_completion method is required, it is where you should place the logic where you communicate with your LLM provider.

The format_message and message_to_string function are encouraged. Defining them allows the provider to work with both defining prompts as one string or as a list of messages.

The environment variables are required to auto-detect which provider is configured. Chainlit only shows configured LLM providers in the prompt playground.

The inputs is a list of controls that this LLM provider offers that Chainlit will display in the side panel of the prompt playground.

The is_chat property is a toggle to define whether you feed a list of messages to the LLM provider (like OpenAI’s gpt-4 model) or one text string (like Anthropic’s claude-2 model).

from fastapi.responses import StreamingResponse

from chainlit.input_widget import Select, Slider
from chainlit.playground.config import BaseProvider, add_llm_provider
from chainlit.prompt import PromptMessage


class ExampleProvider(BaseProvider):
    def format_message(self, message, prompt):
        """Optional method to transform a PromptMessage before sending it to the provider"""
        message = super().format_message(message, prompt)
        # Additional formatting...
        return message

    def message_to_string(self, message: PromptMessage):
        """Optional method that transforms a Prompt Message to a string that a non-chat LLM can understand."""
        return message.to_string()

    async def create_completion(self, request):
        await super().create_completion(request)

        # Check the OpenAI provider for a streaming example
        # https://github.com/Chainlit/chainlit/blob/main/src/chainlit/playground/providers/openai.py#L174

        stream = ["This ", "is ", "the ", "test ", "completion"]

        async def create_event_stream():
            for token in stream:
                await cl.sleep(0.1)
                yield token

        return StreamingResponse(create_event_stream())


example_env_vars = {"api_key": "EXAMPLE_API_KEY"}

Example = ExampleProvider(
    id="example",
    name="Example",
    env_vars=example_env_vars,
    inputs=[
        Select(
            id="model",
            label="Model",
            values=["text-001", "text-002"],
            initial_value="text-002",
        ),
        Slider(
            id="temperature",
            label="Temperature",
            min=0.0,
            max=1.0,
            step=0.01,
            initial=0.9,
        )
    ],
    is_chat=False,
)

​
Step 2: Register the LLM Provider:

Once you have defined the provider, you need to tell Chainlit that it exists.

add_llm_provider(ExampleProvider)

​
Langchain Provider

Adding an LLM Provider from a Langchain LLM class is straightforward.

Langchain Providers won’t have settings editable in the Prompt Playground. If you need the settings, you should extend the BaseProvider class and create your own provider.

Here is an example with HuggingFaceHub (works the same for other LLMs).

import os
from chainlit.playground.config import add_llm_provider
from chainlit.playground.providers.langchain import LangchainGenericProvider
from langchain.llms import HuggingFaceHub

# Instantiate the LLM
llm = HuggingFaceHub(
    model_kwargs={"max_length": 500},
    repo_id="declare-lab/flan-alpaca-large",
    huggingfacehub_api_token=os.environ["HUGGINGFACEHUB_API_TOKEN"],
)

# Add the LLM provider
add_llm_provider(
    LangchainGenericProvider(
        # It is important that the id of the provider matches the _llm_type
        id=llm._llm_type,
        # The name is not important. It will be displayed in the UI.
        name="HuggingFaceHub",
        # This should always be a Langchain llm instance (correctly configured)
        llm=llm,
        # If the LLM works with messages, set this to True
        is_chat=False
    )
)

Was this page helpful?

Formatted PromptsOverview