Deploying Local LLMs for Agent Use: A Comprehensive Guide

Introduction

#

Running large language models (LLMs) locally has become increasingly practical as quantization techniques and hardware capabilities have advanced. For agent frameworks like OpenClaw and Hermes, having a local LLM with a large context window (100k+ tokens) is essential for maintaining conversation history and executing complex multi-step tasks.

This guide covers the best models available in June 2026 for machines with 128GB VRAM (such as Strix Halo setups), how to deploy them using Ollama in Docker, and how to leverage speculative decoding with Multi-Token Prediction (MTP) for faster inference.

Why Run LLMs Locally?

#

Before diving into specific models, here are the key reasons to run LLMs locally for agent use:

• Privacy: Your data never leaves your machine
• Latency: No network round-trips means faster response times
• Cost: No API fees after the initial hardware investment
• Customization: Fine-tune and modify models without restrictions
• Offline capability: Agents work without internet connectivity

Best Models for 128GB VRAM (June 2026)

#

Here are the top models that can run on a 128GB VRAM machine with context windows of 100k+ tokens:

Cohere Command A+ (218B total, 25B active)

#

• Context: 128k input / 64k output
• VRAM: ~110GB at 4-bit quantization
• Strengths: Specifically trained for conversational tool use, making it ideal for agent frameworks that need to call external APIs and tools

DeepSeek Coder V2 (236B total, 20.9B active)

#

• Context: 128k
• VRAM: ~118GB at 4-bit quantization
• Strengths: Specialized for code generation with 338-language support, perfect for code-centric agents

DeepSeek V4 Flash (284B total, 13B active)

#

• Context: 128k native (up to 1M with speculative decoding)
• VRAM: ~80GB at 4-bit quantization
• Strengths: Flagship MoE model with excellent reasoning capabilities

Gemma 4 26B/31B

#

• Context: 128k (E4B) to 256k (26B/31B)
• VRAM: ~33GB at 4-bit quantization
• Strengths: Google’s latest open model with excellent long-context support

Qwen3-8B-128K

#

• Context: 128k
• VRAM: ~4GB at 4-bit quantization
• Strengths: Ultra-lightweight, perfect for running multiple agent instances

Deploying with Ollama in Docker

#

The easiest way to run these models locally is using Ollama in a Docker container. Here’s how to get started:

Step 1: Pull the GGUF Model

#

Most models are available as GGUF files on Hugging Face. For example, to get the DeepSeek V4 Flash Q2 model:

1
2
3

git clone https://github.com/antirez/ds4
cd ds4
./download_model.sh q2  # Downloads ~80GB Q2 quantized model

Step 2: Set Up Ollama in Docker

#

Create a directory for your models and start the Ollama container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Create models directory
mkdir -p ~/models
cp ds4/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf ~/models/

# Start Ollama container with model mount
docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ~/models:/models \
  -e OLLAMA_HOST=0.0.0.0 \
  ollama/ollama

Step 3: Import the Model into Ollama

#

1
2
3
4
5
6

# Import the GGUF file
docker exec -it ollama bash -c \
  "ollama import deepseek-v4-flash-q2 /models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf"

# Run the model
docker exec -it ollama bash -c "ollama run deepseek-v4-flash-q2"

Understanding Quantization: Q2 vs Q4

#

When deploying large models, quantization is essential for fitting them into VRAM. Here’s what the quantization levels mean:

For a 128GB VRAM machine:

• Q4 quantization is recommended for models up to ~236B parameters
• Q2 quantization allows you to run even larger models (284B+) with some quality trade-off

Imatrix vs Standard Quantization

#

Some models offer “imatrix” variants that use importance matrices during quantization:

• Standard Q2: Uses a synthetic fallback heuristic based on weight energy
• Q2-imatrix: Uses real activation statistics from a calibration dataset

The imatrix version provides better quality at the same file size because it allocates bits more intelligently to the most important parameters.

Speculative Decoding with MTP

#

One of the most exciting advances in local LLM inference is Multi-Token Prediction (MTP) combined with speculative decoding. Here’s how it works:

The Problem with Standard Inference

#

Traditional autoregressive generation produces one token at a time:

1
2
3
4

Step 1: Predict token 1 (full forward pass)
Step 2: Predict token 2 (full forward pass)
Step 3: Predict token 3 (full forward pass)
...

This is sequential and doesn’t utilize GPU parallelism effectively.

How MTP + Speculative Decoding Works

#

1. Draft Phase: MTP heads predict multiple candidate tokens in parallel using a single forward pass
2. Verify Phase: The base model verifies all candidates in one additional forward pass
3. Accept/Reject: Tokens that match are kept; mismatches trigger regeneration from that point

The Speed-up

#

With an 85-90% acceptance rate (as reported by DeepSeek), you can achieve:

• 1.8x throughput improvement (tokens per second)
• Reduced latency for multi-token generation

Using MTP with Ollama

#

To enable MTP in Ollama, create a Modelfile that references both the base model and the MTP helper:

1
2

FROM /models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf
MTP /models/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf

Then create and run the model:

1
2

ollama create deepseek-v4-flash-q2-mtp -f Modelfile
ollama run deepseek-v4-flash-q2-mtp

Choosing the Right Model for Your Use Case

#

Here’s a quick reference guide:

Performance Tips

#

1. Quantize to 4-bit for the best balance of quality and VRAM usage
2. Use GGUF format with llama.cpp-based runners for automatic CPU offloading
3. Enable MTP for DeepSeek V4 Flash to get 1.8x speed improvement
4. Run multiple smaller models instead of one large model if you need concurrent agents
5. Monitor VRAM usage with nvidia-smi to avoid out-of-memory errors

Conclusion

#

Running local LLMs for agent use has never been more accessible. With 128GB VRAM, you have the flexibility to choose from several excellent models with 128k+ context windows. Whether you prioritize tool-use capabilities (Command A+), code generation (DeepSeek Coder V2), or raw performance (DeepSeek V4 Flash), there’s a model that fits your needs.

The combination of Ollama for easy deployment and speculative decoding for faster inference makes local agent deployment both practical and efficient. As quantization techniques continue to improve, we can expect even larger models to become accessible on consumer hardware.

References

#

• Cohere Command A+ Model Card
• DeepSeek V4 Flash GGUF
• Gemma 4 Model Card
• Ollama Documentation
• Speculative Decoding Research