Local Large Language Model (2): Comparison of vLLM and Ollama and Deploy Open WebUI Frontend

Introduction

#

Previously, we discussed building private large language model services, where we used Ollama as the backend service and LobeChat as the frontend, communicating through API calls to the backend Ollama to implement conversations.

Recently, I have been trying to use OpenClaw, but its token consumption rate is too fast, so I thought about running a local large language model to provide tokens. After talking with ChatGPT, it told me that in this case, it’s better to use vLLM rather than Ollama, because vLLM provides API compatibility with OpenAI, supports Copilot, and has better performance. So I decided to switch to vLLM.

Introduction

#

vLLM

#

vLLM and Ollama are both tools for running large language models (LLM) on your own computer or on a server, but their use cases differ. Simply put, vLLM is focused on efficiently running large language models, while Ollama focuses on making it easy to run large language models.

Additionally, vLLM provides API compatibility with OpenAI. If the services we want to run depend on commercial model APIs like OpenAI’s, and support custom model providers, we can connect to our own deployed vLLM.

Ollama

#

For an introduction and deployment instructions for Ollama, please refer to the first article in this series.

Since Ollama uses some custom REST APIs that are incompatible with OpenAI’s API, it cannot directly connect to many services that use large language models. However, developers have created a tool LiteLLM that can call Ollama and is compatible with OpenAI’s API.

We only need to deploy this tool along with the Ollama container and make some settings, and it will be able to provide API compatibility with OpenAI’s API.

Comparison and Considerations for vLLM and Ollama

#

1. GPU Memory Usage

   When running, the biggest difference we can perceive between vLLM and Ollama is that deploying vLLM requires loading the entire model into the GPU’s memory, even when the model is idle (not being called), vLLM will still occupy this portion of memory.

   In other words, vLLM can hardly run alongside other programs that heavily use the graphics card. For example, after running vLLM, if you want to play games, you need to take vLLM offline first.

   Ollama, on the other hand, does not have this problem. Ollama only loads the model into memory when it is called. When you want to play games, you don’t need to shut down Ollama, as long as you don’t call Ollama during your gaming session, it can coexist peacefully with your game.

2. Number of Callable Models

   After deploying Ollama, we can download multiple models, and users can decide which one to call.

   However, after deploying vLLM, the model loaded into GPU memory is fixed — one vLLM container can only load one model. To choose between models, you would need to deploy multiple vLLM containers simultaneously. This is clearly impractical for personal users, as our GPUs generally cannot handle that. If you want to load multiple models, you can only use very small ones, which defeats the purpose.

Deploying the Large Language Models and Frontend

#

Overview

#

I have two computers each with a GPU with 16GB of VRAM. I chose to deploy vLLM on Computer A and deploy Ollama with a vLLM-compatible interface on Computer B (let’s call it vOllama for now). The interfaces of vLLM and vOllama are exposed to the public internet via domain names.

The Open WebUI frontend is deployed on any one of the computers (either A or B, or another Computer C), which makes it easy to connect to our self-deployed large language models.

Deploying vLLM

#

docker-compose

#

The docker-compose.yml file is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

services:
  vllm:
    image: vllm/vllm-openai:v0.17.0
    container_name: vllm
    restart: unless-stopped
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - HF_HUB_DISABLE_XET=1
      - HF_HUB_ENABLE_HF_TRANSFER=0
      - VLLM_API_KEY=${VLLM_API_KEY:-dummy}
      - TZ=${TZ:-UTC}
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ipc: host
    command: >
      ${MODEL}
      ${MODEL_PARAMETERS}
    networks:
      traefik-net:
        aliases:
          - vllm

networks:
  traefik-net:
    external: true

The MODEL, MODEL_PARAMETERS, and other variables are defined in the environment variable file .env:

1
2
3
4
5

HF_TOKEN=hf_your_token
MODEL=Qwen/Qwen2.5-Coder-14B-Instruct-AWQ
MODEL_PARAMETERS="--host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.81 --dtype auto --kv-cache-dtype fp8 --enforce-eager"
VLLM_API_KEY=your_api_key
TZ=Europe/Paris

Here, HF_TOKEN is the Hugging Face token, which we need to download large language models from Hugging Face.

vLLM can only load one model at a time. We chose Qwen/Qwen2.5-Coder-14B-Instruct-AWQ, which requires approximately 14GB of VRAM to load.

Testing

#

Enter a command like the following in the terminal to test whether vLLM has been deployed successfully:

1
2
3
4
5
6
7

curl https://vllm.example/v1/chat/completions \
  -H "Authorization: Bearer your_vllm_api_key" \   
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-Coder-14B-Instruct-AWQ",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

If you receive a response similar to the following, vLLM has been successfully deployed and is accessible via the domain name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

{
  "id": "chatcmpl-82d9387336a446bd",
  "object": "chat.completion",
  "created": 1774276907,
  "model": "Qwen/Qwen2.5-Coder-14B-Instruct-AWQ",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today?",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 30,
    "total_tokens": 40,
    "completion_tokens": 10,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

In the terminal, this JSON will be displayed compressed on a single line.

Deploying vOllama

#

Configuration Files

#

The docker-compose.yml file is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

services:
  ollama:
    container_name: ollama
    image: docker.1ms.run/ollama/ollama:0.17.7
    environment:
      - OLLAMA_ORIGINS=*
      - OLLAMA_HOST=0.0.0.0
      - TZ=America/New_York
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: ["gpu"]
            count: all
    volumes:
      - ./ollama-data:/root/.ollama
    restart: always
    networks:
      - traefik-net

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    command: ["--config", "/app/config.yaml"]
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    networks:
      - traefik-net

networks:
  traefik-net:
    external: true

Here we deploy two containers, ollama and litellm, both using the traefik-net network we created earlier. We still use Cloudflare Tunnel to reverse proxy the vOllama service.

litellm requires a configuration file litellm-config.yml, as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

model_list:
  - model_name: qwen3.5:0.8b
    litellm_params:
      model: ollama/qwen3.5:0.8b
      api_base: http://ollama:11434

  - model_name: mistral:7b
    litellm_params:
      model: ollama/mistral:7b
      api_base: http://ollama:11434

general_settings:
  master_key: your_ollama_api_key

The configuration file serves two main purposes:

• model_list tells litellm which large language models we have running in ollama. When we download and install additional models in Ollama, we need to update this file to include the new models.
• Since we are exposing the models on the public internet, we need to set an API key (i.e., master_key) to control access. Anyone who does not provide the API key will not be able to call our models.

Installing Large Language Models

#

1. Use docker compose up -d to deploy the Ollama container.
2. After a successful deployment, Ollama is empty with no models downloaded. Use docker exec -it ollama ollama run qwen3.5:0.8b to download and run the qwen3.5:0.8b model.
3. After installation, you can continue downloading other models.
4. Use docker exec -it ollama ollama list to list the installed models.

Testing

#

Use the following command to test whether the deployment was successful:

1
2
3
4
5
6
7

curl https://ollama.example/v1/chat/completions \
  -H "Authorization: Bearer your_ollama_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:0.8b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

If you receive a response similar to the following, vOllama has been successfully deployed and is accessible via the domain name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

{
  "id": "chatcmpl-5844b507-f967-4412-b8fa-815c4e3235b1",
  "created": 1774276549,
  "model": "qwen3.5:0.8b",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Greetings! How's going? 😊 Feel free to ask me anything—whether you need help, just wanted to say hello, or maybe something else. I'm here for fun and helpful!",
        "role": "assistant"
      }
    }
  ],
  "usage": {
    "completion_tokens": 111,
    "prompt_tokens": 15,
    "total_tokens": 126
  }
}

In the terminal, this JSON will be displayed compressed on a single line.

Deploying Open WebUI

#

Using only curl or APIs to call the deployed models is clearly inconvenient — we need to deploy a web-based client. Here we choose the open-source client Open WebUI.

Introduction to Open WebUI

#

Open WebUI is an open-source web client for large language models, with a style similar to OpenAI’s ChatGPT interface. It allows us to connect to our self-deployed vLLM or Ollama models, and also supports connecting to various commercial models (API key required).

Deploying Open WebUI

#

The docker-compose.yml file is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

version: "3.9"

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:0.8.10
    container_name: open-webui
    volumes:
      - ${DATA_DIR}:/app/backend/data
    environment:
      - OPENAI_API_BASE_URL=http://vllm:8000/v1
      - OPENAI_API_KEY=${VLLM_API_KEY:-dummy}
      - TZ=${TZ:-UTC}
    restart: unless-stopped
    networks:
      traefik-net:
        aliases:
          - open-webui

networks:
  traefik-net:
    external: true

The OPENAI_API_BASE_URL and OPENAI_API_KEY here point to our locally deployed vLLM. These can be left blank and configured later in the graphical interface.

Use the following command to start the open-webui container:

1

docker compose up -d

Open WebUI Configuration

#

After setting up the reverse proxy, we can access Open WebUI via the domain name. On first access, Open WebUI will ask us to set up an administrator account and password.

After logging in with the administrator account, click the avatar in the upper right corner and you will see the “Admin Panel” option. Click to enter and configure the backend model services that Open WebUI needs to connect to.

In the “Settings” page of the “Admin Panel”, there is a “Connections” option where you can add the Ollama service, vLLM service, or vOllama service we deployed earlier, as well as commercial large language model services (such as OpenAI, Azure, Anthropic, etc.).

After adding them, click the “Models” option in the “Settings” page, and you can see the available models in the Ollama or vLLM service. These models are set to “private” by default. You can click the edit button on the right side of a model and click the “Access” option in the model settings to configure which users can access that model.

Finally, after configuring the models, you can use Open WebUI just like ChatGPT. After entering a question, Open WebUI will call the vLLM or Ollama service we deployed earlier to get a response:

Troubleshooting

#

Installing Qwen3.6 with Ollama

#

When using Ollama to install the uncensored Qwen3.6 model Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive, you may encounter installation failures. The error message is as follows:

1

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35moe'

This issue has been discussed on GitHub:

https://github.com/ollama/ollama/issues/14503

The reason is that after downloading the Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive model, the Modelfile generated by Ollama is incorrect. It contains two From /root/.ollama/xxx lines, indicating that Ollama is trying to load two model files when in fact only one is needed. We need to delete the second From /root/.ollama/xxx line or comment it out with #, then save and re-run the model.

However, the Modelfile does not seem to be explicitly created. We need to enter the container first:

1

docker exec -it ollama bash

Then execute the following command inside the container to generate the file:

1

ollama show --modelfile hf.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:IQ2_M | cat > /root/Modelfile

After commenting out or deleting the second From /root/.ollama/xxx line, save and exit, then recreate the model:

1

ollama create Qwen3.6-35B-A3B-Uncensored-IQ2M -f /root/Modelfile