🧠Connect an LLM

Note: If you want to run the backend in the cloud, you can either use a cloud-based LLM API, like OpenAI or Together.ai or you can proxy the traffic from the cloud to your local Ollama. See below for instructions.

Ollama (default)

By default, the app tries to use Ollama to run it entirely locally.

1. Download and install Ollama.

2. Open the app or run ollama serve in a terminal. ollama serve will warn you if the app is already running.

3. Run ollama pull llama3 to have it download llama3.

4. Test it out with ollama run llama3.

Ollama model options can be found here.

If you want to customize which model to use, adjust convex/util/llm.ts or set npx convex env set OLLAMA_MODEL # model. If you want to edit the embedding model:

1. Change the OLLAMA_EMBEDDING_DIMENSION in convex/util/llm.ts and ensure: export const EMBEDDING_DIMENSION = OLLAMA_EMBEDDING_DIMENSION;

2. Set npx convex env set OLLAMA_EMBEDDING_MODEL # model.

Note: You might want to set NUM_MEMORIES_TO_SEARCH to 1 in constants.ts, to reduce the size of conversation prompts, if you see slowness.

OpenAI

To use OpenAI, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = OPENAI_EMBEDDING_DIMENSION;

Set the OPENAI_API_KEY environment variable. Visit https://platform.openai.com/account/api-keys if you don't have one.

npx convex env set OPENAI_API_KEY 'your-key'

Optional: choose models with OPENAI_CHAT_MODEL and OPENAI_EMBEDDING_MODEL.

Together.ai

To use Together.ai, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = TOGETHER_EMBEDDING_DIMENSION;

Set the TOGETHER_API_KEY environment variable. Visit https://api.together.xyz/settings/api-keys if you don't have one.

npx convex env set TOGETHER_API_KEY 'your-key'

Optional: choose models via TOGETHER_CHAT_MODEL, TOGETHER_EMBEDDING_MODEL. The embedding model's dimension must match EMBEDDING_DIMENSION.

Other OpenAI-compatible API

You can use any OpenAI-compatible API, such as Anthropic, Groq, or Azure.

• Change the EMBEDDING_DIMENSION in convex/util/llm.ts to match the dimension of your embedding model.

• Edit getLLMConfig in llm.ts or set environment variables:

npx convex env set LLM_API_URL 'your-url'
npx convex env set LLM_API_KEY 'your-key'
npx convex env set LLM_MODEL 'your-chat-model'
npx convex env set LLM_EMBEDDING_MODEL 'your-embedding-model'

Note: if LLM_API_KEY is not required, don't set it.

Note on changing the LLM provider or embedding model:

If you change the LLM provider or embedding model, you should delete your data and start over. The embeddings used for memory are based on the embedding model you choose, and the dimension of the vector database must match the embedding model's dimension. See below for how to do that.