Run the stack locally

deploy/platform is a Docker Compose stack: the AI gateway, the NuFi app, the console, observability and monitoring, wired together. Keep it running in the background. Every other guide in this section starts one app from source against it.

What comes up

The databases have no host ports. The apps you run from source talk to the gateway on 4000 and to the chat on 3080, which is all they need.

Before you start

Install the tools in the prerequisites table, then sign in to the registry once. The chat and console images are private:

# a GitHub personal access token with the read:packages scope
echo ghp_xxxxxxxxxxxxxxxxxxxx | docker login ghcr.io -u <your-github-username> --password-stdin

Ollama is optional. Install it only if you want a free local model; a cloud key or any OpenAI-compatible server on your network works instead.

Apple Silicon

The two images from ghcr.io/dudaji-vn are built for linux/amd64 only. On an arm64 Mac, docker compose up stops with no matching manifest for linux/arm64/v8, and because the pull is one transaction it also takes down anything that was already running. Create deploy/platform/docker-compose.override.yml (it is gitignored, and Compose reads it automatically):

services:
  librechat:
    platform: linux/amd64
  console:
    platform: linux/amd64

Both then run under emulation: correct, slower, and hungrier for memory.

Bring it up

cd deploy/platform
./scripts/bootstrap.sh --backend ollama --model qwen2.5:3b   # a local model through Ollama
./scripts/bootstrap.sh --backend cloud                       # an OpenAI, Anthropic, Together or Groq key
./scripts/bootstrap.sh --backend skip                        # the stack only; register a model later

Without flags it asks. In order, the script:

1. Checks for docker, docker compose, openssl, curl, git, yq, and that the Docker daemon is up.
2. Copies .env.example to .env if there is none, then fills every secret still set to replace-me. Re-running is safe: values already in .env are kept.
3. Pulls the chat image, then docker compose up -d.
4. Waits up to five minutes for the gateway to report healthy.
5. Registers your model with add-model.sh (see Add or change a model).
6. Runs the smoke test, then prints the URLs and the Langfuse login.

Other flags: --backend remote for a vLLM, TGI or any OpenAI-compatible server on your network; --backend mock-npu to clone a registered model tagged as NPU for testing routing; --domain <host> to rewrite the public URLs; --skip-pull to skip ollama pull; --skip-smoke-test.

One line it prints is wrong: the URL list says Grafana is on 3002 and "not yet deployed". Grafana is on 3030.

Check it works

Then run the smoke test:

./scripts/smoke-test.sh

It prints eight numbered checks and all checks passed: gateway liveness, the model list, a chat completion, a streamed completion, a 4xx for an unknown model, a Langfuse trace for the request it just made, the request counter in Prometheus, and a recorded decision from the prompt-injection control.

The three health endpoints, when you want them by hand:

curl -s http://localhost:4000/health/liveliness   # "I'm alive!"
curl -s http://localhost:3080/health              # OK   (it is /health; /api/health returns 404)
curl -s http://localhost:3001/_health             # {"ok":true}

After git pull

The gateway and the scanner are built on your machine from litellm/ and scanner/. docker compose up -d never rebuilds them. When a pull touches either directory, or litellm/config.yaml:

docker compose build litellm-proxy nufi-scanner
docker compose up -d

The symptom of forgetting is a gateway that restarts in a loop. Its log ends with unknown threshold key(s) naming a policy key the old image does not know:

docker compose logs --tail 50 litellm-proxy

Day to day

docker compose ps                     # every service, with (healthy) or not
docker compose logs -f litellm-proxy  # follow one service
docker compose restart librechat
docker compose down                   # stop; keep the data
docker compose down -v                # stop and drop every volume

What to edit, and what to do afterwards:

Secrets live in deploy/platform/.env and nowhere else; the other apps read them from there when you run them from source.

Variables the stack reads that .env.example does not list

Add these to .env yourself when you need them.

When it does not come up