LLM Notes¶

This guide is a work in progress.

The models and commands below reflect my personal setup.

A collection of notes and configuration snippets for running large language models locally, covering installation, model serving, and integration with coding agents and editors.

llama.cpp¶

https://llama.app

macOS Windows Linux

Install via Homebrew:

brew install llama.cpp

Pre-built binaries are also available from the GitHub releases page.

Install via WinGet:

winget install llama.cpp

Pre-built binaries are also available from the GitHub releases page.

See the official build instructions.

Pre-built binaries are also available from the GitHub releases page.

Router Mode¶

Router Mode allows you to serve multiple models with on-demand loading and unloading, with speculative decoding to improve inference speed. See the llama.cpp Router Mode documentation for more details.

Models must be available first.

Models must be downloaded before they can be served by the router. See Models for the commands which will download the models.

models.ini

[*]
ctx-size = 65536
flash-attn = on
load-mode = none
no-mmproj = true

[ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_0]
alias = gemma4-26b-a4b-it-qat
no-reasoning-preserve = true

[ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M]
alias = ornith-1.5-35b-a3b
spec-type = draft-mtp

[ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M]
alias = qwen3.6-35b-a3b
spec-type = draft-mtp

[unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M]
alias = qwen3-coder-30b-a3b-instruct

llama serve --cors-origins localhost \
  --sleep-idle-seconds 300 \
  --models-max 1 \
  --models-preset models.ini

Argument	Explanation
`--cors-origins`	comma-separated list of allowed origins for CORS (default: *)
`--sleep-idle-seconds`	number of seconds of idleness after which the server will sleep
`--models-max`	for router server, maximum number of models to load simultaneously
`--models-preset`	path to INI file containing model presets for the router server

For the full server options and flags reference, see the llama.cpp server documentation.

Ollama¶

https://ollama.com

macOS Windows Linux

To install Ollama, run the following command:

curl -fsSL https://ollama.com/install.sh | sh

Or download the installer from https://ollama.com/download/mac.

To install Ollama, run the following command in PowerShell:

irm https://ollama.com/install.ps1 | iex

Or download the installer from https://ollama.com/download/windows.

To install Ollama, run the following command:

curl -fsSL https://ollama.com/install.sh | sh

Models¶

Gemma-4-26B-A4B-IT¶

https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/

llama.cppOllama

Download model:

llama download -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_0 --mtp

Download and serve model:

llama serve -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_0 \
  -a gemma4-26b-a4b-it-qat \
  -c 65536 \
  --cors-origins localhost \
  --flash-attn on \
  --load-mode none \
  --no-mmproj \
  --no-reasoning-preserve

ollama pull gemma4:26b-a4b-it-qat

Ornith-1.5-35B-A3B¶

https://ornith.ai/ornith_1_5.html

llama.cppOllama

Download model:

llama download -hf ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M --mtp

Download and serve model:

llama serve -hf ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M \
  -a ornith-1.5-35b-a3b \
  -c 65536 \
  --cors-origins localhost \
  --flash-attn on \
  --load-mode none \
  --no-mmproj \
  --spec-type draft-mtp

ollama pull ornith-1.5:35b

Qwen3.6-35B-A3B¶

https://qwen.ai/blog?id=qwen3.6-35b-a3b

llama.cppOllama

Download model:

llama download -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M --mtp

Download and serve model:

llama serve -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M \
  -a qwen3.6-35b-a3b \
  -c 65536 \
  --cors-origins localhost \
  --flash-attn on \
  --load-mode none \
  --no-mmproj \
  --spec-type draft-mtp

ollama pull qwen3.6:35b-a3b-mtp-q4_K_M

Qwen3-Coder-30B-A3B-Instruct¶

https://qwen.ai/blog?id=qwen3-coder

llama.cppOllama

Download model:

llama download -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M --mtp

Download and serve model:

llama serve -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M \
  -a qwen3-coder-30b-a3b-instruct \
  -c 65536 \
  --cors-origins localhost \
  --flash-attn on \
  --load-mode none \
  --no-mmproj

ollama pull qwen3-coder:30b

Coding Agents¶

Tip

You can change the model name for both Copilot and Claude by updating the value passed to --model in the commands below.

Copilot CLI¶

https://github.com/features/copilot/cli

llama.cppOllama

COPILOT_PROVIDER_BASE_URL=http://localhost:8080/v1 COPILOT_PROVIDER_WIRE_API=responses copilot --model gemma4-26b-a4b-it-qat

COPILOT_PROVIDER_BASE_URL=http://localhost:11434/v1 COPILOT_PROVIDER_WIRE_API=responses copilot --model gemma4:26b-a4b-it-qat

Claude Code¶

https://code.claude.com/docs/en/overview

llama.cppOllama

ANTHROPIC_BASE_URL=http://localhost:8080 ANTHROPIC_AUTH_TOKEN=ollama claude --model gemma4-26b-a4b-it-qat

ANTHROPIC_BASE_URL=http://localhost:11434 ANTHROPIC_AUTH_TOKEN=ollama claude --model gemma4:26b-a4b-it-qat

Pi¶

https://pi.dev

llama.cppOllama

The llama.cpp extension auto-discovers models from a running llama.cpp server and registers them as the llama-cpp provider in pi.

pi install git:github.com/huggingface/pi-llama

Pi also supports the llama.cpp router server. The router discovers multiple GGUF models and loads or unloads them on demand. See the llama.cpp documentation for more details.

~/.pi/agent/models.json

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "models": [
        {
          "id": "gemma4:26b-a4b-it-qat"
        }
      ]
    }
  }
}

OpenCode¶

https://opencode.ai

~/.config/opencode/opencode.json

llama.cppOllama

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "gemma4-26b-a4b-it-qat": {
          "name": "gemma4-26b-a4b-it-qat",
          "limit": {
            "context": 65536,
            "output": 8192
          }
        }
      }
    }
  }
}

Adding more models.

Add model objects to the models section with the model ID as key and update the name and limit properties as needed.

OpenCode probes http://127.0.0.1:11434, discovers completion models, and adds them to /models without an account connection.

See the Ollama provider documentation for more details.

IDEs & Editors¶

VSCode¶

https://code.visualstudio.com

llama.cppOllama

The Custom Endpoint provider lets you connect any compatible API endpoint to chat in VS Code. See the language models documentation for more details.

Model configuration file location:

macOS Windows Linux

~/Library/Application Support/Code/User/chatLanguageModels.json

%appdata%\Code\User\chatLanguageModels.json

~/.config/Code/User/chatLanguageModels.json

[
  {
    "name": "llama.cpp",
    "vendor": "customendpoint",
    "models": [
      {
        "id": "gemma4-26b-a4b-it-qat",
        "name": "gemma4-26b-a4b-it-qat",
        "url": "http://localhost:8080/v1",
        "toolCalling": true,
        "vision": false,
        "thinking": true,
        "contextWindow": 65536,
        "maxOutputTokens": 8192
      }
    ]
  }
]

Adding more models.

Copy the model object and update the id and name.

The Ollama extension adds models from your running Ollama server to the VS Code model picker.

Ollama for VS Code