(llm translate, orig=german)
In recent posts I've been writing a lot about (subscription) models and OpenClaw. I've gathered a lot of experience with those and things are working well so far. Here I'd like to dump a small side project from the past few weeks: fluxer.app (a Discord alternative) + an OpenClaw plugin. More on that in a separate post.
So the software runs, but I was unhappy with the HTPC. Built from an i7-7700, 16 GB of slow DDR4 RAM, a 6700 XT with 12 GB of RAM, and a massively oversized 850 W power supply. The idea when I built that (Frankenstein) HTPC: occasionally play games on the TV in full HD. For that, it was perfect — I just rarely did. But I did use it for testing OpenClaw for months, and I was never really happy with it. The machines running 24/7 bothered me. So, a lot of research — hardware prices are expensive right now. The result was a GMKtec K12:
- Ryzen 255H, Radeon 780M iGPU (Vulkan), 32 GB DDR5, 1 TB NVMe
- RAM bandwidth: ~89 GB/s — the bottleneck
What changed: – The HTPC had a dedicated GPU (6700 XT) — the K12 only has an iGPU (780M) – The K12 is ~5× smaller and quieter, but also slower with large models
The first question people will ask: why a mini-PC for a local LLM? Surely you need an NVIDIA GPU with 32 GB of VRAM.
Sure, and something with 48 GB would be even better — then you can run models in good quality, or several at once. But it also depends on what you want to do. OpenClaw handles a lot of things in the background for me: research on topic X, creating SVGs. None of that is particularly urgent. And, well, price plays a role too.
My goal was: I want to be able to run OpenClaw entirely on a local model if I have to, even if slowly. That means a certain model size is required, with compromises on speed.
The backstory: https://text.tchncs.de/wiulinu/openclaw-alfred-uebernehmen-sie-teil-6
Results: – For chat, editing configs, and anything that needs a bit more care: a 30B model or larger – For cron jobs: Gemma4-12B works well and saves a lot of tokens on the subscription
Cron jobs, script-based, also run on a small 2B model. Not perfect, but very efficient: LFM2.5-2.6B Q8, enough context length and good tool calling. Any smaller didn't work — below that, quality dropped and there was a lot of pseudo-data.
Next challenge: chat and maintenance of OpenClaw.
Consideration 1: buy a GPU with lots of VRAM for the HTPC. So a lot of research into new hardware, used hardware, NVIDIA, Intel, AMD. Used P40s, Instinct MI50/60/100 … or maybe a second 6700 XT? That would at least be 24 GB of VRAM. But 24 GB of VRAM still isn't ideal. And what stays true for all of these solutions: high power consumption and old hardware.
One of my favorite candidates was an Intel Arc Pro B60 with 24 GB of VRAM. It requires PCIe 5.0, though, and the HTPC only has PCIe 3.0. That, along with the idea of two 6700 XTs, gave me a hint about what actually matters for tokens per second: bandwidth.
RAM bandwidth
| RAM type | Clock | Channels | Bandwidth |
|---|---|---|---|
| DDR4-2600 | 2600 MT/s | Dual | ~41.6 GB/s |
| DDR4-3200 | 3200 MT/s | Dual | ~51.2 GB/s |
| DDR5-4800 | 4800 MT/s | Dual | ~76.8 GB/s |
| DDR5-5600 | 5600 MT/s | Dual | ~89.6 GB/s (K12/780M) |
| DDR5-6400 | 6400 MT/s | Dual | ~102.4 GB/s |
| LPDDR5X-8000 | 8000 MT/s | Quad | ~215 GB/s (Strix Halo) |
PCIe bandwidth
| Standard | Lanes | Bandwidth |
|---|---|---|
| PCIe 3.0 ×16 | 16 | ~15.75 GB/s |
| PCIe 4.0 ×16 | 16 | ~31.5 GB/s |
| PCIe 4.0 ×4 | 4 | ~7.88 GB/s |
| PCIe 5.0 ×16 | 16 | ~63 GB/s |
| USB4 | — | ~5 GB/s |
GPU VRAM bandwidth
| GPU | VRAM | Memory bus | Bandwidth | Power draw |
|---|---|---|---|---|
| Radeon 6700 XT | 12 GB GDDR6 | 192-bit | 384 GB/s | 230 W |
| RX 9060 XT | 16 GB GDDR6 | 128-bit + 32 MB IC | 322 GB/s | 150 W |
| Intel Arc Pro B60 | 24 GB GDDR6 | 192-bit | 456 GB/s | ~150 W |
| Radeon 8060S (Strix Halo) | 96 GB unified | LPDDR5X | ~215 GB/s | 50–120 W |
| NVIDIA P40 | 24 GB GDDR5X | 384-bit | 346 GB/s | 250 W |
| AMD MI50 | 32 GB HBM2 | 4096-bit | 1 TB/s | 300 W |
| RTX 4090 (reference) | 24 GB GDDR6X | 384-bit | 1,008 GB/s | 450 W |
What this means for LLM inference
Rule of thumb: tok/s ≈ bandwidth ÷ active parameters (Q4)
- 780M (89 GB/s) + 3.8 B active = ~23 tok/s
- 6700 XT (384 GB/s) + 7 B active = ~55 tok/s
- B60 (456 GB/s) + 27 B active = ~17 tok/s
- MI50 (1 TB/s) + 27 B active = ~37 tok/s
My conclusion
With more than one GPU you could split the model across both, but every time the GPUs exchange data, PCIe slows things down — especially since I only have PCIe 3.0.
A big GPU is too expensive, and the rest would still be old hardware that isn't ideal for 24/7 operation. But there has to be some halfway viable solution.
The next idea was to upgrade the DDR4 RAM. But that's too slow as well. Really? Time for the next hacking test. I had stumbled across the option of splitting models several times before — but it was always roughly 50% per GPU. Deep enough down the rabbit hole, the option appeared: “just offload the experts of the mixture-of-experts model to the CPU.”
My first thought: that doesn't help me, I still don't have more room in VRAM for the model. You can offload the experts, but that's an additional requirement — it only distributes the load. An example:
Qwen3.6-35B-A3B needs 18 GB of VRAM/RAM. On my workstation (yes, there's more than just the HTPC) I have a 9060 XT with 16 GB of VRAM, so it doesn't fit. But there's also 32 GB of DDR4 (very slow). So more research: llama.cpp can offload in both directions.
ramalama serve --backend vulkan \
--image ghcr.io/ggml-org/llama.cpp:full-vulkan-b10775 \
--runtime-args="-cmoe --cache-ram 0 --spec-type draft-mtp --spec-draft-n-max 2 --jinja" \
hf://unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-IQ4_XS \
--port 8081 --threads 6 --thinking false \
--ctx-size 256000 --host 0.0.0.0
So now 18 GB of DDR4 RAM are occupied and the experts live in the 9060 XT. It still got tight on the 9060 XT: 256,000 context + experts take about 10 GB of VRAM, and the desktop also helped itself to a bit.
So, how fast?
| Qwen3.6-35B-A3B IQ4_XS | Mode | Speed |
|---|---|---|
| Qwen3.6-35B-A3B IQ4_XS | -cmoe (experts → RAM, attention → VRAM) | ~15–25 tok/s |
| Qwen3.6-35B-A3B IQ4_XS | -cmoe + MTP n=3 | ~25–40 tok/s |
25 tok/s is okay, 40 tok/s I consider good. Now I knew: MoE model + at least 50 GB/s of bandwidth is fine — anything below that is an absolute no-go.
From here it was clear: with DDR5 this is doable. But you still don't want to compute on the CPU. And then I stumbled across terms I had never heard of before, which work like magic!
GTT
GTT stands for Graphics Translation Table — a memory pool in the kernel that the AMD driver (amdgpu) reserves for GPU operations.
Check: cat /sys/class/drm/card0/device/mem_info_gtt_total
That's for Vulkan. There are other parameters for ROCm, deprecated parameters, and who knows what else.
On to the next insight: a Ryzen APU will very likely serve the purpose too. Still unclear: how VRAM allocation works in the BIOS. Resizable BAR was another keyword that came up. Then there were very different reports: one saying “you can set the GPU to 24 of 32 in BIOS, that works,” another saying “set 512 MB, the rest is handled by GTT” — ROCm, Vulkan, Intel, all mixed together. For me, only Vulkan was an option.
The magic of GTT + Vulkan is this: GTT's default value is half of the available RAM.
- Default config: 32 GB RAM, 6 GB “VRAM” → 12 GB GTT — meaning 18 GB available for the model.
- Variant 1: 1 GB VRAM, ~15 GB GTT, 16 GB available for the model.
- Variant 2: 24 GB VRAM, 3 GB GTT, 27 GB available for the model. Depends on the BIOS; not possible on mine.
- Variant 3: Take variant 1 and change the GTT value via a kernel parameter (see below).
My variant: 16 GB VRAM + 8 GB GTT. Enough for the model + 128K context and some headroom. No fiddling, just set it once in the BIOS.
LLM performance on the K12
Why MoE almost always wins here: the 780M has ~89 GB/s of bandwidth (DDR5-5600 dual channel). Per token, the model weights are streamed from RAM. A dense 27B reads ~16 GB per token, an MoE reads only ~3.8 GB per token. Same bandwidth budget, but 4× more tokens per second.
Theoretical maximum: 89 GB/s ÷ active parameters (Q4) — e.g. 89 ÷ 3.8 = ~23 tok/s. Measured: 23–25, so it checks out.
Check: cat /sys/class/drm/card0/device/mem_info_gtt_used
The proper solution: ttm.pages_limit and ttm.page_pool_size on the kernel cmdline, calculated as (GB × 1024 × 1024 × 1024) / 4096.
Qwen3.6-35B-A3B on the K12 — measured live
| Parameter | Value |
|---|---|
| Model | Qwen3.6-35B-A3B IQ4_XS (~18.2 GB) |
| Model weights + MTP | ~3.38 GiB |
| Context | 84K |
| MTP | n=2 |
| tok/s | 23 without MTP, 26–28 with MTP |
| GTT used (84K) | 5.20 GiB |
| GTT limit | 7.69 GiB |
| GTT free | ~2.49 GiB |
| KV cache (84K) | ~1.70 GiB |
| KV cache per token | ~20.6 KiB |
Extrapolation to larger contexts:
| Context | KV cache (approx.) | GTT used (approx.) | Free | Status |
|---|---|---|---|---|
| 12K | ~0.24 GiB | ~3.62 GiB | ~4.07 GiB | ✅ comfortable |
| 64K | ~1.28 GiB | ~4.66 GiB | ~3.03 GiB | ✅ plenty of room |
| 84K | ~1.70 GiB | 5.20 GiB | ~2.49 GiB | ✅ stable (measured) |
| 128K | ~2.58 GiB | ~6.08 GiB | ~1.61 GiB | ✅ fits |
| 160K | ~3.30 GiB | ~6.80 GiB | ~0.89 GiB | ⚠️ tight, no headroom |
Plus the raw GTT numbers at 84K context (measured live):
mem_info_gtt_used: 5,581,713,408 bytes = 5.20 GiBttm/pageslimit: 2,015,931 pages = 7.69 GiB limit- KV cache at 84K: ~1.70 GiB → ~20.6 KiB per context token
The actually interesting number: a 128K context fits on 32 GB of RAM — with ~1.61 GiB of headroom. This was previously unclear because the hybrid architecture of Qwen3.6 (Gated DeltaNet + Attention) only produces a KV cache in ~10 of its 40 blocks. That makes the KV cache ~75% smaller than a standard transformer's.
I tested the 128K that way too, and it's now in production. It runs at about 25 tokens per second.
HTPC sold, mini-PC bought, total cost 400 euros.
Bonus: the PC runs (mostly) quietly, draws less power, looks nicer, and has enough room for the model.