Skip to content

LLM Notes

This guide is a work in progress.

The models and commands below reflect my personal setup.

A collection of notes and configuration snippets for running large language models locally, covering installation, model serving, and integration with coding agents and editors.

llama.cpp

Install via Homebrew:

brew install llama.cpp

Pre-built binaries are also available from the GitHub releases page.

Install via WinGet:

winget install -e --id ggml.llamacpp

Pre-built binaries are also available from the GitHub releases page.

See the official build instructions.

Pre-built binaries are also available from the GitHub releases page.

Router Mode

Router Mode allows you to serve multiple models with on-demand loading and unloading, with speculative decoding to improve inference speed. See the llama.cpp Router Mode documentation for more details.

Models must be available first.

Models must be downloaded before they can be served by the router. See Models for the serve commands which will download the models.

models.ini

[*]
ctx-size = 65536
load-mode = none

[ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_0]
alias = gemma4-26b-a4b-it-qat
spec-type = draft-mtp

[deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4_K_M]
alias = ornith-1.0-35b
no-mmproj = true
reasoning-preserve = true

[ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M]
alias = qwen3.6-35b-a3b
spec-type = draft-mtp
no-mmproj = true
reasoning-preserve = true

llama serve --models-preset models.ini

Ollama

To install Ollama, run the following command:

curl -fsSL https://ollama.com/install.sh | sh

Or download the installer from https://ollama.com/download/mac.

To install Ollama, run the following command in PowerShell:

irm https://ollama.com/install.ps1 | iex

Or with WinGet using the following command:

winget install -e --id Ollama.Ollama

Or download the installer from https://ollama.com/download/windows.

To install Ollama, run the following command:

curl -fsSL https://ollama.com/install.sh | sh

Models

Gemma-4-26B-A4B-IT

llama serve -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_0 \
  -a gemma4-26b-a4b-it-qat \
  -c 65536 \
  --load-mode none \
  --spec-type draft-mtp

ollama pull gemma4:26b-a4b-it-qat

Ornith-1.0-35B

llama serve -hf deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4_K_M \
  -a ornith-1.0-35b \
  -c 65536 \
  --no-mmproj \
  --load-mode none \
  --reasoning-preserve

ollama pull ornith:35b-q4_K_M

Qwen3.6-35B-A3B

llama serve -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M \
  -a qwen3.6-35b-a3b \
  -c 65536 \
  --no-mmproj \
  --load-mode none \
  --reasoning-preserve \
  --spec-type draft-mtp

ollama pull qwen3.6:35b-a3b-mtp-q4_K_M

Coding Agents

Note

You can change the model name for both Copilot and Claude by updating the value passed to --model in the commands below.

Copilot CLI

mise use -g copilot

COPILOT_PROVIDER_BASE_URL=http://localhost:8080/v1 COPILOT_PROVIDER_MAX_PROMPT_TOKENS=57344 COPILOT_PROVIDER_MAX_OUTPUT_TOKENS=8192 copilot --model gemma4-26b-a4b-it-qat

COPILOT_PROVIDER_BASE_URL=http://localhost:11434/v1 COPILOT_PROVIDER_MAX_PROMPT_TOKENS=57344 COPILOT_PROVIDER_MAX_OUTPUT_TOKENS=8192 copilot --model gemma4:26b-a4b-it-qat

Token limits.

COPILOT_PROVIDER_MAX_PROMPT_TOKENS and COPILOT_PROVIDER_MAX_OUTPUT_TOKENS are separate budgets that must sum to within the model's context window (57344 + 8192 = 65536). This split does not apply to OpenCode, which derives the input budget from its limit.context and limit.output settings.

Claude Code

mise use -g claude

ANTHROPIC_BASE_URL=http://localhost:8080 ANTHROPIC_AUTH_TOKEN=ollama claude --model gemma4-26b-a4b-it-qat

ANTHROPIC_BASE_URL=http://localhost:11434 ANTHROPIC_AUTH_TOKEN=ollama claude --model gemma4:26b-a4b-it-qat

Pi

mise use -g pi

The llama.cpp extension auto-discovers models from a running llama.cpp server and registers them as the llama-cpp provider in pi.

pi install git:github.com/huggingface/pi-llama

Pi also supports the llama.cpp router server. The router discovers multiple GGUF models and loads or unloads them on demand. See the llama.cpp documentation for more details.

~/.pi/agent/models.json

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "models": [
        { "id": "gemma4:26b-a4b-it-qat" }
      ]
    }
  }
}

OpenCode

mise use -g opencode

~/.config/opencode/opencode.json

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://localhost:8080/v1"
      },
      "models": {
        "gemma4-26b-a4b-it-qat": {
          "name": "gemma4-26b-a4b-it-qat",
          "limit": {
            "context": 65536,
            "output": 8192
          }
        }
      }
    }
  }
}

Adding more models.

Add model objects to the models section with the model ID as key and update the name and limit properties as needed.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "gemma4:26b-a4b-it-qat": {
          "name": "gemma4-26b-a4b-it-qat"
        }
      }
    }
  }
}

Adding more models.

Add model objects to the models section with the model ID as key and update the name property as needed.

IDEs & Editors

VSCode

The Custom Endpoint provider lets you connect any compatible API endpoint to chat in VS Code. See the language models documentation for more details.

Model configuration file location:

~/Library/Application Support/Code/User/chatLanguageModels.json

%appdata%\Code\User\chatLanguageModels.json

~/.config/Code/User/chatLanguageModels.json

[
  {
    "name": "llama.cpp",
    "vendor": "customendpoint",
    "models": [
      {
        "id": "gemma4-26b-a4b-it-qat",
        "name": "gemma4-26b-a4b-it-qat",
        "url": "http://localhost:8080/v1",
        "toolCalling": true,
        "vision": true,
        "thinking": true,
        "maxInputTokens": 57344,
        "maxOutputTokens": 8192
      }
    ]
  }
]

Token limits.

maxInputTokens and maxOutputTokens are separate budgets that must sum to within the model's context window (57344 + 8192 = 65536). This split does not apply to OpenCode, which derives the input budget from its limit.context and limit.output settings.

Adding more models.

Copy the model object and update the id, name, and feature flags as needed.

Model Features

Model Tool Calling Vision Thinking
Gemma-4-26B-A4B-IT
Ornith-1.0-35B
Qwen3.6-35B-A3B

The Ollama extension adds models from your running Ollama server to the VS Code model picker.

Ollama for VS Code

Resources

Installation & Tools

Platforms & Services

Models

Coding Agents

IDEs & Editors