Most people assume an iGPU "can't run an LLM" because of the tiny shared VRAM and limited bandwidth. I had a LAN box with an Intel Arc 140T sitting around, so I threw a 7.9B MoE model — Ling-3.0-tiny — at it and measured everything.

Short version: the iGPU handles it, and faster than you'd expect. Short-context prefill hits 556 t/s — over 8x the same machine on pure CPU. Full data below.

Test environment

The CPU name is masked as "Genuine Intel 0000" by the system, but 14C/14T at 2.0 GHz is real. The iGPU runs through the Vulkan backend with -ngl 99 for full offload.

Step 1: Proving the iGPU is actually working

This matters. The llama-bench table has a backend = Vulkan column — but that's not evidence. I confirmed it also says Vulkan with -ngl 0 (pure CPU). You need hard proof.

The cleanest cross-check is to turn off GPU offload and run pure CPU:

The iGPU delivers roughly 4x the prefill speed. At just 4.49 GB, the MoE weights land right in the Arc 140T's shared-VRAM sweet spot.

Main result: 1K to 24K context speed curves

This is the table I most wanted to produce. Numbers come from the server-side print_timing log, which excludes HTTP round-trip overhead.

Two decay laws

• Prefill roughly halves: 556 → 332 → 178 → 88 → 58 t/s. At 24k, prefill alone takes 7 minutes — that's the dominant cost of long context.
• Decode decays far more gently: 40.8 → 15.5 t/s, only a 62% drop. Decode is bottlenecked on streaming weights from shared memory and kernel launch overhead, not attention.

End to end: 24K input + 128 output ≈ 7.1 minutes, 98% of it prefill. Cutting 24k to 8k drops prefill from 419s to 46s (9x) while any parameter tweak buys at most 6%. For long context, shorten the input — don't tune flags.

Why this model doesn't blow up VRAM at long context

This is the key design of the bailingmoe3 architecture. It's hybrid linear attention: 18 of 24 layers use a recurrent state (only 19.27 MiB), and just 6 layers carry full-attention KV.

n_head_kv = [0,0,0,1, 0,0,0,1, 0,0,0,1, 0,0,0,1, 0,0,0,1, 0,0,0,1]
                              ↑ 只有 6 层有 KV，其余 18 层是 recurrent state

So KV cache at 32k context is only 114.75 MiB — f16 fits easily and KV quantization is pointless here. Memory barely grows with context, and prefill decays far more gently than a pure Transformer.

Tuning: the -ub sweep

I swept -ub (physical batch size) with real 8192-token full prefills, 3 random prefixes per point, median taken:

• 256 → 512 is the biggest jump (+9.5%) — that's the cheapest win.
• Returns peak at 1024–1536 (~187 t/s); going larger (2048/4096) slightly regresses — memory bandwidth is saturated and bigger physical batches only add FA scratch memory.
• Use -ub 1536: 8k prefill at 187.5 t/s, 5.8% faster than 768, and no OOM at 24k full prefill.

Thread count and ngl

4/6/8/10 are all within noise (763–768), but 12/14 clearly regress. Vulkan needs CPU threads for submission and over-subscription hurts. With 14 cores, 8 threads is the sweet spot.

As for -ngl: with only 25 layers, -ngl 30 already offloads everything — identical to -ngl 99. Small models make this parameter irrelevant.

Against a pure-CPU machine

Pure CPU (-ngl 0) vs the iGPU on the same box — the gap is larger than you'd guess:

8 pitfalls I hit

1. pp512 does not represent real long-input prefill

pp512 reports 768 t/s, but real 8k input gives 178 t/s — over 4x apart. Prefill throughput collapses with length; always compare benchmarks at the same length.

2. Confirm you have exactly one client

With -np 1, concurrent requests just queue. I misread results once because a buffer-blocked "hung" client doubled the sample count for the same 16k target.

3. Don't run the Vulkan service as SYSTEM

As SYSTEM in session 0, -c 32768 -ctk f16 just hangs: process alive, port listening, but /health never responds. Running in an interactive logon session fixes it — GPU contexts need an interactive session.

4. PowerShell 5.1 dies on BOM-less UTF-8 scripts

Any Chinese in a BOM-less file gets read as GBK, producing invalid tokens and a ParserError where not one line executes. The symptom is deceptive: zero-byte log file, no error on stdout. Fix: put test logic in Python 3 and keep .ps1 as an ASCII-only launcher.

5. Start-Process children die when SSH disconnects

Even with -RedirectStandardOutput. Fix: register llama-server.exe as a scheduled task and launch it with Start-ScheduledTask. The task is the process, so SSH disconnects don't touch it; it's ready in 6–12s.

6. Paths with spaces break Start-Process

-ArgumentList doesn't quote paths with spaces. My model path ...\Default Project\... hit this exactly: silent startup failure, zero-byte log. Fix: hand-quote in the scheduled task's -Argument string.

7. scp chokes on spaces in the path

Windows OpenSSH scp reports ambiguous target. Fix: copy to a space-free C:\temp\ first, then Move-Item remotely.

8. Break the prefix cache before measuring

Three identical 8k prompts in a row all hit cache, producing a fake prompt eval time ≈ 1 token/~1s. My fix: prepend a 64-char random prefix to break the shared prefix, and explicitly pass "cache_prompt": false.

Recommended flags

llama-server.exe -m "C:\path\Ling-3.0-tiny-Q4_K_M.gguf" \
  -ngl 99 -t 8 -ctk q8_0 -ctv q8_0 -fa on \
  -ub 1536 -b 2048 -c 32768 \
  --host 0.0.0.0 --port 8088 -np 1 -lv 3 --no-warmup

• Use this daily: 8k prefill goes from 177 to 187.5 t/s (+5.8%), no OOM at 24k.
• Shorten the input for long context rather than tuning flags. 24k→8k saves 9x; any -ub change buys at most 6%.
• Short context (≤8k) feels best: 8k in + 128 out ≈ 51s, with decode at 27 t/s — comfortably interactive.
• KV quantization isn't the bottleneck: 114.75 MiB at 32k means -ctk f16 -ctv f16 is fine if you want lossless.