Unified-Memory AI Box Planner

Project Purpose & Problem Space

Personal local Large Language Model (LLM) inference has entered an architectural transition. Traditional hardware sizing calculators were engineered specifically for discrete PCIe GPU topologies (such as a 24 GB NVIDIA GeForce RTX 4090 or dual RTX 3090 configurations), where GPU VRAM is isolated from system DRAM by a PCIe bus.

The emergence of unified-memory platforms—including the Sipeed MaixStation (Axera AX8860P), NVIDIA DGX Spark, and AMD Ryzen AI Halo, alongside Apple Mac Studio M2 Ultra—introduces shared addressable memory pools from 128 GB to 192 GB with theoretical peak memory bandwidth from 256 GB/s to 800 GB/s. On these machines, CPU, GPU, and NPU cores share a single physical memory architecture.

Unified-Memory AI Box Planner was developed to explore this distinct technical space. It provides local developers, AI researchers, and workstation builders with a mathematical calculation engine to model whether large models (such as Llama 3.1 70B, Qwen 2.5 32B, DeepSeek V2 Lite, and GPT-OSS 120B) can fit in addressable memory and what theoretical decode speed upper bounds could be achieved under batch-1 workloads.

How the Calculation Engine Works

The sizing engine models memory footprint and theoretical bandwidth limits:

1. Theoretical Memory Bandwidth Upper Bound for Decoding

In batch-1 autoregressive token generation, when a workload is memory-bandwidth bound, computing each subsequent token requires streaming active model parameters and reading the accumulated Key-Value (KV) cache through the memory controllers. Under ideal bus saturation, the theoretical physical upper bound is expressed as:

Decode Ceiling (tok/s) = Peak Memory Bandwidth (GB/s) / Total Bytes Read per Token (GB)

This formula represents an ideal theoretical ceiling, not a guaranteed empirical benchmark. Real-world decode speeds are typically lower due to runtime framework overhead, kernel launch latency, memory controller efficiency, and compute boundaries during prompt prefill.

2. Attention KV Cache Memory Sizing

Memory requirements scale with sequence context length ($N$). The engine calculates KV cache footprint across three distinct attention architectures:

• Grouped-Query Attention (GQA / MHA): Utilized by Meta Llama 3.1 (8 KV heads, 70B/8B) and Qwen 2.5 (8 KV heads, 32B). Scales as $2 \times L \times H_{KV} \times D \times N \times \text{bytes}$.
• Multi-Head Latent Attention (MLA): Utilized by DeepSeek V2 Lite, utilizing low-rank latent KV compression vectors ($d_c = 512, d_R = 64$) to reduce memory pressure at extended contexts.
• Sliding-Window Attention (SWA): Hybrid sliding attention configurations with bounded buffer allocations.

3. Deterministic 95% Modeling Threshold

Unified RAM is shared with operating system kernel tasks, display servers, dynamic driver allocations, and runtime activations. A flat 4 GB baseline reserve is modeled for base OS processes. To help users assess memory allocations, the planner evaluates configurations against a 95% modeling threshold:

• Fits Memory (≤ 95% of Usable Memory): Total modeled footprint (quantized weights + context KV cache + 4 GB OS reserve + dynamic activations) remains within 95% of addressable unified memory. This provides a conservative planning heuristic, though it is not an absolute guarantee against OS paging or swapping under heavy multitasking.
• Tight Fit (95% to 100% of Usable Memory): Modeled consumption enters the final 5% margin of addressable memory, indicating high risk of memory pressure or swap thrashing.
• Doesn’t Fit as Configured (> 100% of Usable Memory): Total required memory exceeds addressable capacity. In accordance with honest disclosure rules, speed metrics are strictly suppressed, and the exact shortfalls in GB are calculated with adjustment recommendations.

Dual-Status Matrix & Provenance Standards

The planner audits 120 potential configurations (5 model architectures × 8 quantization formats × 3 KV precisions) into two explicit categories:

• Evidence-Backed Configurations: Supported by identifiable public model artifacts (e.g., published Base Llama 3.1 GGUFs or OpenAI MXFP4 checkpoints) and documented runtime engine support (such as llama.cpp server FP16/Q8_0 KV modes or vLLM native architectures). These configurations display an active provenance box linking directly to publisher repositories.
• Theoretical / Unverified Configurations: Configurations where public artifacts or runtime execution evidence has not been established (e.g., unverified quantization combinations or unsupported KV formats). These configurations persistently display cautionary warning banners clarifying that displayed numbers are mathematical estimates only.

Whenever empirical benchmark calibration is missing (such as prefill compute or device-level kernel efficiency factors), numerical estimates are suppressed with an explicit evidence gap notification rather than presenting ungrounded guesses.

Release History

Release 1.0.0 (October 31, 2026)

• Dual-Status Combination Matrix: Audited 120 configurations into evidence-backed versus theoretical regimes with visible UI provenance.
• Model Context Limits & Clamping: Aligned context windows with official publisher model cards (131,072 for Llama 3.1, Qwen 2.5, GPT-OSS; 32,768 for DeepSeek V2 Lite) with active notification on slider clamping.
• Qualified GPT-OSS 120B Native MXFP4 Sizing: Adopted OpenAI's published 60.8 GiB checkpoint reference, qualified capacity verdicts with runtime unverified disclaimers, and suppressed decode metrics due to unverified MoE active parameter and unembedding breakdown.
• Exceeds-Memory Rules: Suppressed tok/s and TTFT on out-of-memory configurations while displaying exact GB shortfall and adjustment guidance.
• Suppressed Uncalibrated Prefill: Replaced speculative prefill numbers with honest suppression notices citing unvalidated empirical benchmarks.
• Verification Automation: Enforced by automated test vectors (6/6 passing), Release 1 verification suite (11 sections passing), and real browser Playwright end-to-end tests.

Privacy & Data Handling Practices

The privacy practices of this website are based on its local code and standard web hosting architecture:

To inspect the references and documentation citations used across this planner, visit the Sources & Evidence Directory.

Contact & Feedback

For inquiries, technical corrections, or feedback regarding hardware specifications and sizing calculations, contact: dorykriesel869@gmail.com.