Prompt executors

Prompt executors provide a higher-level abstraction that lets you manage the lifecycle of one or multiple LLM clients. You can work with multiple LLM providers through a unified interface, abstracting from provider-specific details, with dynamic switching between them and fallbacks.

Executor types

Koog provides two main types of prompt executors that implement the PromptExecutor interface:

Type	Class	Description
Single-provider	SingleLLMPromptExecutor	Wraps a single LLM client for one provider. Use this executor if your agent only requires switching between models within a single LLM provider.
Multi-provider	MultiLLMPromptExecutor	Wraps multiple LLM clients and routes calls based on the LLM provider. It can optionally use a configured fallback provider and LLM when the requested client is unavailable. Use this executor if your agent needs to switch between LLMs from different providers.

Creating a single-provider executor

To create a prompt executor for a specific LLM provider, perform the following:

1. Configure an LLM client for a specific provider with the corresponding API key.
2. Create a prompt executor using SingleLLMPromptExecutor.

Here is an example:

val openAIClient = OpenAILLMClient(System.getenv("OPENAI_KEY"))
val promptExecutor = SingleLLMPromptExecutor(openAIClient)

Creating a multi-provider executor

To create a prompt executor that works with multiple LLM providers, do the following:

1. Configure clients for the required LLM providers with the corresponding API keys.
2. Pass the configured clients to the MultiLLMPromptExecutor class constructor to create a prompt executor with multiple LLM providers.

val openAIClient = OpenAILLMClient(System.getenv("OPENAI_KEY"))
val ollamaClient = OllamaClient()

val multiExecutor = MultiLLMPromptExecutor(
    LLMProvider.OpenAI to openAIClient,
    LLMProvider.Ollama to ollamaClient
)

Pre-defined prompt executors

For faster setup, Koog provides the ready-to-use executor implementations for common providers.

The following table includes the pre-defined single-provider executors that return SingleLLMPromptExecutor configured with a specific LLM client.

LLM provider	Prompt executor	Description
OpenAI	simpleOpenAIExecutor	Wraps OpenAILLMClient that runs prompts with OpenAI models.
OpenAI	simpleAzureOpenAIExecutor	Wraps OpenAILLMClient configured for using Azure OpenAI Service.
Anthropic	simpleAnthropicExecutor	Wraps AnthropicLLMClient that runs prompts with Anthropic models.
Google	simpleGoogleAIExecutor	Wraps GoogleLLMClient that runs prompts with Google models.
OpenRouter	simpleOpenRouterExecutor	Wraps OpenRouterLLMClient that runs prompts with OpenRouter.
Amazon Bedrock	simpleBedrockExecutor	Wraps BedrockLLMClient that runs prompts with AWS Bedrock.
Amazon Bedrock	simpleBedrockExecutorWithBearerToken	Wraps BedrockLLMClient and uses the provided Bedrock API key to send requests.
Mistral	simpleMistralAIExecutor	Wraps MistralAILLMClient that runs prompts with Mistral models.
Ollama	simpleOllamaAIExecutor	Wraps OllamaClient that runs prompts with Ollama.

Koog also provides the pre-defined multi-provider executor DefaultMultiLLMPromptExecutor. This is an implementation of MultiLLMPromptExecutor that wraps OpenAILLMClient, AnthropicLLMClient, and GoogleLLMClient models.

Here is an example of creating pre-defined single and multi-provider executors:

// Create an OpenAI executor
val promptExecutor = simpleOpenAIExecutor("OPENAI_KEY")

// Create a DefaultMultiLLMPromptExecutor with OpenAI, Anthropic, and Google LLM clients
val openAIClient = OpenAILLMClient("OPENAI_KEY")
val anthropicClient = AnthropicLLMClient("ANTHROPIC_KEY")
val googleClient = GoogleLLMClient("GOOGLE_KEY")
val multiExecutor = DefaultMultiLLMPromptExecutor(openAIClient, anthropicClient, googleClient)

Running a prompt

To run a prompt using a prompt executor, do the following:

1. Create a prompt executor.
2. Run the prompt with the specific LLM using the execute() method.

Here is an example:

// Create an OpenAI executor
val promptExecutor = simpleOpenAIExecutor("OPENAI_KEY")

// Execute a prompt
val response = promptExecutor.execute(
    prompt = prompt("demo") { user("Summarize this.") },
    model = OpenAIModels.Chat.GPT4o
)

This will run the prompt with the GPT4o model and return the response.

Note

The prompt executors provide methods to run prompts using various capabilities, such as streaming, multiple choice generation, and content moderation. Since prompt executors wrap LLM clients, each executor supports the capabilities of the corresponding client. For details, refer to LLM clients.

Switching between providers

When you work with multiple LLM providers using MultiLLMPromptExecutor, you can switch between them. The process is as follows:

1. Create an LLM client instance for each provider you want to use.
2. Create a MultiLLMPromptExecutor that maps LLM providers to LLM clients.
3. Run a prompt with a model from the corresponding client passed as an argument to the execute() method. The prompt executor will use the corresponding client based on the model provider to run the prompt.

Here is an example of switching between providers:

// Create LLM clients for OpenAI, Anthropic, and Google providers
val openAIClient = OpenAILLMClient("OPENAI_API_KEY")
val anthropicClient = AnthropicLLMClient("ANTHROPIC_API_KEY")
val googleClient = GoogleLLMClient("GOOGLE_API_KEY")

// Create a MultiLLMPromptExecutor that maps LLM providers to LLM clients
val executor = MultiLLMPromptExecutor(
    LLMProvider.OpenAI to openAIClient,
    LLMProvider.Anthropic to anthropicClient,
    LLMProvider.Google to googleClient
)

// Create a prompt
val p = prompt("demo") { user("Summarize this.") }

// Run the prompt with an OpenAI model; the prompt executor automatically switches to the OpenAI client
val openAIResult = executor.execute(p, OpenAIModels.Chat.GPT4o)

// Run the prompt with an Anthropic model; the prompt executor automatically switches to the Anthropic client
val anthropicResult = executor.execute(p, AnthropicModels.Sonnet_3_5)

You can optionally configure a fallback LLM provider and model to use when the requested client is unavailable. For details, refer to Configuring fallbacks.

Configuring fallbacks

Multi-provider prompt executors can be configured to use a fallback LLM provider and model when the requested LLM client is unavailable. To configure the fallback mechanism, provide the fallback parameter to the MultiLLMPromptExecutor constructor:

val openAIClient = OpenAILLMClient(System.getenv("OPENAI_KEY"))
val ollamaClient = OllamaClient()

val multiExecutor = MultiLLMPromptExecutor(
    LLMProvider.OpenAI to openAIClient,
    LLMProvider.Ollama to ollamaClient,
    fallback = MultiLLMPromptExecutor.FallbackPromptExecutorSettings(
        fallbackProvider = LLMProvider.Ollama,
        fallbackModel = OllamaModels.Meta.LLAMA_3_2
    )
)

If you pass a model from an LLM provider that is not included in the MultiLLMPromptExecutor, the prompt executor will use the fallback model:

// Create a prompt
val p = prompt("demo") { user("Summarize this") }
// If you pass a Google model, the prompt executor will use the fallback model, as the Google client is not included
val response = multiExecutor.execute(p, GoogleModels.Gemini2_5Pro)

Note

Fallbacks are available for the execute() and executeMultipleChoices() methods only.