BREAKING
397B Model Runs on a MacBook Pro
0
B
total params
0
B
active params
0
GB
unified memory
How SSD Expert Streaming Works
1
mmap 5.5GB non-experts
↓
2
Stream experts from SSD
↓
3
OS page cache holds 42GB
↓
4
FMA Metal kernel runs
Speed by Quantization Scheme
2-bit
5.74
4-bit + FMA
4.36
4-bit base
3.9
Promise vs PoC Limits
Upside
On-device
●
Tool calling on a laptop
●
Privacy, low latency
●
iPhone 17 Pro fork appeared
Limits
PoC
●
2-bit breaks JSON
●
Heat and throttling risk
●
Heavy SSD dependence
Built in 24 Hours, 90+ Experiments
AI NEWS BLITZ
A new engine called Flash-MoE runs a 397 billion parameter model on a laptop.