Infrastructure · Observability
LLM on Kubernetes
Getting an open-source model to run is the easy half. Knowing, continuously, whether it is still producing tokens fast enough is the half nobody instruments.
Built
2026
Stack
- Kubernetes (kind)
- Prometheus
- Grafana
- Ollama
- Llama 3.2 1B
- React
01 — The problem
“The pod is running” is not the same as “the model works”.
In most tutorials, reliability means the container has not crashed. In production it means an engineer asking the model something gets a useful answer, and that when they do not, something alerts before the user complains.
Ollama exposes no metrics endpoint at all, so there was nothing to scrape and nothing to alert on. The exporter is written from scratch: 192 lines of Python with exactly one dependency, publishing eight metric families and running a capped five-token probe so that “is it generating?” is answered by generation rather than by a health check.
- Metric families
- 8
- gauges, a histogram and two counters
- Grafana panels
- 11
- no nested rows, all fed by the exporter
- Alert rules
- 3
- down, p95 latency above 10s, probe errors rising
- Exporter
- 192 lines
- one dependency, python:3.12-slim
Three rules, and they fired on something real.
An alert that has never fired is a hypothesis. These fired during real degradation on the cluster, which is the only way to find out whether the thresholds were set anywhere near the right place.
My own stack loses, and the numbers are published anyway.
Ten prompts across factual recall, reasoning, code generation and summarisation, run against the self-hosted Llama 3.2 1B and against a commercial API model, each answer scored out of fifteen. The local stack scored 121 of 150. The commercial model scored 148.
Quality score per prompt, out of 15
Self-hosted Llama 3.2 1B on CPU against gpt-4o-mini. Tied on three of ten prompts; behind on the rest.
The judge is gpt-4o-mini — the same model family as one of the two contestants. Position is randomised per prompt but the assignment is not persisted, so the randomisation cannot be audited from the artifact. Every record in results.json carries a judge_bias_disclosed flag for exactly this reason.
Source scripts/benchmark/results.json
Table
| Prompt | Self-hosted | Commercial | Local time |
|---|---|---|---|
| factual 1 | 15/15 | 15/15 | 235.2s |
| factual 2 | 9/15 | 15/15 | 17.2s |
| factual 3 | 12/15 | 15/15 | 164.8s |
| reasoning 1 | 10/15 | 15/15 | 238.1s |
| reasoning 2 | 15/15 | 15/15 | 88.2s |
| reasoning 3 | 10/15 | 15/15 | 32.6s |
| code 1 | 11/15 | 15/15 | 14.2s |
| code 2 | 10/15 | 13/15 | 25.3s |
| summarise 1 | 14/15 | 15/15 | 15.9s |
| summarise 2 | 15/15 | 15/15 | 13.3s |
The gap that matters is not really quality. It is throughput: 2.83 tokens per second aggregate on CPU against 42.09 from the API, and two local responses that took 238 and 235 seconds against 7 and 12. The self-hosted stack costs nothing per call and answers roughly fifteen times slower — which is the actual trade, stated in numbers instead of adjectives.
- Self-hosted
- 80.7%
- 121 of 150, $0 in API cost
- Commercial
- 98.7%
- 148 of 150, $0.001498 for the run
- Local throughput
- 2.83 tok/s
- aggregate across all ten prompts
- API throughput
- 42.1 tok/s
- roughly fifteen times faster
What it deliberately does not do
-
It does not claim a neutral judge. Using gpt-4o-mini to score a contest that gpt-4o-mini is in is a real methodological weakness. It is disclosed on every record rather than quietly omitted, because the alternative — a human rubric over ten prompts — was not run.
-
It is one node, not a cluster. A single-node kind cluster on an 8 GB laptop hosting both the model and its own monitoring stack. The observability design would carry to a real cluster; the capacity numbers would not.
-
There is no CI. No workflow file exists in the repository. The gates were run by hand.
What it gets wrong
The exporter catches a bare Exception on its probe path, which will swallow a programming error as a probe error and inflate that counter. It is a known rough edge in a 192-line file, not a designed behaviour.