The llm example allows you to run advanced AI chatbots and services on your own computer with just one command. Once you’ve downloaded the model, you can use it without needing a constant internet connection.
services-flake provides two main uses:
- Running services in development projects with source code access.
- Creating end-user apps that run multiple services.
Our example is based on the second use. These apps can be launched with nix run or installed using nix profile install.
Running the app
To run the local AI chatbot and launch the Web UI,
# You can also use `nix profile install` on this URL, and run `services-flake-llm`
nix run "github:juspay/services-flake?dir=example/llm"
Before launching the Web UI, this will download the phi3 model, which is about 2.4GB. To reduce or avoid this delay, you can:
- Choose a different model, or
- Use no model at all
See further below for more options.
Demo
Want to run your own AI chatbot (like ChatGPT) locally? You can do that with Nix.
— NixOS Asia (@nixos_asia) June 18, 2024
Powered by services-flake (#NixOS), using @OpenWebUI and @ollama.
See example: https://t.co/dyItC93Pya pic.twitter.com/DeDow8bEPw
Default configuration & models
The example runs two processes Ollama and Open WebUI
Key points:
-
Data storage:
-
Ollama data is stored in
$HOME/.services-flake/llm/ollama -
To change this location, edit the
dataDiroption inflake.nix
-
Ollama data is stored in
-
Model management:
-
By default, the
phi3model is automatically downloaded -
To change or add more models: a. Edit the
modelsoption inflake.nix, or b. Use the open-webui interface to download additional models.
-
By default, the