07 Module · Why Ollama

The prompt is
a data transfer.

In a European company the question that decides AI adoption is usually jurisdiction rather than capability, and the architecture follows from that.

First released: 2023
Written in: Go
Endpoint: OpenAI-compatible
Licence: MIT

A prompt containing an unreleased product plan, a patient record or a client's source code is a data transfer, whatever the product calls it.

We have this conversation often. An engineering team adopts a hosted assistant because it is useful. Six months later somebody asks what has been pasted into it, and nobody knows, because the interface made pasting easy and nothing logged it.

The usual response is a policy: a document telling people what they may not paste. Policies of that shape have a poor record against a tool that is helpful and one keystroke away.

The architectural response is to move the model inside the perimeter. The policy question then changes from "what may leave?" to "what, specifically, is worth sending out?", which has a much shorter answer.

Argumenten · The case

Seven reasons,
in order of how often they matter.

Ranked by what decides the procurement conversation rather than by technical interest.

01 Nothing crosses the perimeter

Prompts, source code, documents and customer data stay on hardware you own. The data-protection discussion then comes down to describing the network, instead of arguing about a sub-processor's sub-processor.

02 It speaks the protocol you already use

An OpenAI-compatible endpoint means existing editors, scripts and libraries point at your own machine with one changed base URL. Nothing above it needs rewriting, and nothing is locked to us.

03 The cost is electricity, not tokens

One GPU serves a team all day for the cost of running it. Nobody has to ration usage or watch a meter, and finance discusses it once at procurement instead of every month.

04 Model management is a package manager

ollama pull, and the weights are on disk. Pinning a specific model version is trivial, which matters when a hosted model changes underneath you and last month's prompts stop working.

05 Latency is a local network

There is no internet round trip, no rate limit and no provider incident to take your tooling down. This matters most for editor integrations, where the model runs every time you pause typing and people abandon it if it is slow.

06 Retrieval over your own documents

Embedding an internal corpus locally means the index never leaves either. Questions can be answered against your own handbook, contracts or codebase without uploading anything to build the index.

07 No training-data question

There is no clause to read about whether your inputs are retained or used for training, because there is no counterparty.

Jämförelse · Side by side

Against a hosted API

These are not equivalent. The table shows where the trade-offs fall.

Concern	Hosted frontier API	Ollama, on your GPU
Raw capability	Meaningfully higher	Good, and closing
Where the prompt goes	Their infrastructure	Your rack
Data-protection paperwork	DPA, SCCs, a transfer assessment	None, as nothing is transferred
Cost model	Per token, forever	Capital, then electricity
Cost at heavy team use	Grows with adoption	Flat
Latency	Internet round trip	Local network
Availability	Their status page	Your status page
Model stability	Changes underneath you	Pinned until you move it
Context window	Very large	Smaller, and VRAM-bound
Upfront work	An API key	Hardware, sizing, deployment

Hur vi driftar · Our install

What a Chenborg Ollama install contains.

Most of the value is in sizing and wiring. Installing Ollama takes an afternoon; the engagement is mostly choosing hardware that matches the models your team will use and connecting it to where the work happens.

GPU host sized against your actual workload, procured or rented at a European provider
Ollama declared in NixOS, so the machine is reproducible like every other one
An OpenAI-compatible endpoint on your WireGuard network, never published
Retrieval over your own repositories and documents, indexed locally
Integrations where the work is: editors, Emacs, Redmine's REST API, internal search
A written acceptable-use and data-handling note your DPO can sign

The Ollama engagement in detail

ollama.nix

# The GPU host, declared once.
services.ollama = {
  enable            = true;
  acceleration      = "cuda";
  host              = "10.44.0.9";  # wg0 only
  loadModels        = [
    "qwen3:32b"
    "nomic-embed-text"
  ];
  environmentVariables = {
    OLLAMA_KEEP_ALIVE = "30m";
  };
};

# not reachable from the internet,
# by construction rather than by policy
networking.firewall.interfaces.wg0
  .allowedTCPPorts = [ 11434 ];

Nackdelar · The downsides

Where the gap is.

Open-weight models running on one GPU are not frontier models, whatever a hardware vendor tells you. What matters is whether the gap falls on work you do.

Hard reasoning is where you feel it most. On long multi-step problems, subtle architectural questions and unusual mathematics, a frontier model is better by a wide margin.
Context windows are smaller and limited by VRAM. Feeding an entire large repository into one prompt is a hosted-model habit; locally you use retrieval instead, which is more work to build.
Somebody must own the machine. A GPU host is a server like any other and needs patching, monitoring and capacity planning.
Hardware is a capital cost. A GPU is a real purchase, and if adoption fails it is a real write-off. For a small team experimenting, a hosted API with a strict policy may be the better first step.
The frontier keeps moving. Local models are improving quickly, but so are the best hosted ones, so expect to revisit the sizing every eighteen months or so.

Where the gap does not matter is the bulk of daily work: summarising, classifying, drafting, rewriting, explaining unfamiliar code, and answering questions against internal documents. That is most of the work, and it is also the part that touches your most sensitive material.

So we usually recommend both: local by default for anything touching your own data, and a clear written line about the specific work that may leave the building. Drawing that line is part of what we deliver.