Unified-Memory Hardware Matrix
The First-Principles Physics of Unified-Memory AI Inference
In modern autoregressive Large Language Model generation, every single new token requires streaming the active parameter weights and reading the full KV cache through memory. Because batch size for personal local inference is typically 1, generation is strictly memory-bandwidth bound.
1. Why identical memory bandwidth yields identical decode ceilings:
Both the Sipeed MaixStation and NVIDIA DGX Spark feature 256-bit LPDDR5X memory buses delivering approximately 273 GB/s peak theoretical bandwidth. As a result, their theoretical physical decode ceilings for any given model and context are identical:
Decode Ceiling (tok/s) = Peak Memory Bandwidth (GB/s) / Total Bytes Read per Token (GB)
2. Decode vs. Prefill Distinction:
While decoding is bandwidth-limited, the prompt processing (prefill) phase computes attention across all input tokens simultaneously, making it compute-bound (measured in dense TFLOPS / TOPS). High NPU TOPS accelerates prompt ingestion (Time-to-First-Token), but will not exceed the memory bus speed during token generation.
3. Software Ecosystem Maturity Matters:
NVIDIA (CUDA) and Apple (Metal) boast battle-tested, out-of-the-box runtimes in llama.cpp, vLLM, and PyTorch. In contrast, proprietary NPU toolchains (such as Axera Pulsar2) currently require custom model compilation and operator mapping, representing experimental developer territory.