I built this on one machine and measured it there: a Lenovo ThinkPad L16 Gen 2 running Arch Linux and Hyprland, with a Ryzen 7 PRO 250, a Radeon 780M and a Phoenix XDNA 1 NPU. The project set out to answer two questions and ended up answering a third.
Speculative decoding without ROCm
ROCm support on this iGPU is a moving target, so I ran llama.cpp through Vulkan instead and paired Llama 3.1 8B with Llama 3.2 1B as the draft model, both at Q4_K_M. The small model proposes tokens and the large one verifies them in a batch, which turns many small matmuls into fewer large ones.
| Workload | Speedup over the non-speculative baseline |
|---|---|
| Code | 1.59× |
| Reasoning | 1.61× |
| Overall | 1.42× |
A daemon that stays out of the way
presenced watches the UVC webcam at 2 Hz on the CPU and locks Hyprland once I stop being in front of it. A vision loop running beside a saturated inference job costs you throughput without announcing it, so I measured presenced against the same suite: at worst it cost 0.82 % of inference performance. Kill either service and the other keeps running. The suite logged no OOM events and no thermal throttling.
The NPU, through AMD's stack
The device enumerates. amdxdna loads. XRT lists it. That is where the AMD-stack story ended for llama.cpp and OpenCV: VitisAI falls back to the CPU without saying so, and both OGA Hybrid and FastFlowLM target Windows or XDNA 2.
The repository publishes that gap rather than burying it, because a silent CPU fallback is worse than an error. A benchmark still produces numbers, and they look like the NPU worked.
The NPU, through the open stack
Phase 8 changed the ending. Dropping Vitis and the VitisAI execution provider for the open IRON, mlir-aie and Peano toolchain, I climbed a ladder of hand-written kernels up to whole models:
- M1 passthrough, then M2 and M4 matmul.
- M3 int8 conv2d.
- M5 real CNNs, including ResNet and Google's Magika.
- M6 Llama-3.2-1B-Instruct generating text with every weight matmul on the NPU, as bf16 GEMV.
The M6 output is token-identical to the fp32 reference, with logits at cosine 0.999992. A 1B-parameter model generating text on a Phoenix NPU under Linux is the result I did not expect to reach when the AMD stack dead-ended.