Breaking

Google Demos Gemma 4 26B MoE Running 16-Way Parallel at 300 tok/s on a Single DGX Spark

June 28, 2026 at 12:34 EDT

Foundation Models
Infra & Chips
Open Source

Google has shown a demo running its MoE-based language model Gemma 4 26B A4B as 16 parallel instances on a single NVIDIA "DGX Spark" desktop AI workstation, reaching an aggregate throughput of roughly 300 tok/s. The company says the setup can scale to as many as 32 simultaneous instances.

June 23, 2026 · Gemma Team

16 AI models, one desktop box — running side by side

An official demo runs 16 simultaneous instances of the MoE model Gemma 4 26B-A4B on a single NVIDIA DGX Spark — about 18 tok/s each, ~300 tok/s combined, and scalable to 32 instances.

concurrent model instances on one machine (scalable to 32)

~300

tok/s aggregate throughput across 16 instances

~16 GB

loaded footprint per model (NVFP4 quantized)

Speed: solo vs. shared

Same unit — tokens/sec. One model running alone vs. each model when 16 share the box, and the combined total.

~52

1 instance
(solo)

~18

per instance
(×16)

~300

16 combined
(aggregate)

Why it fits: MoE activates only a slice

26B total parameters, but only ~4B fire per token (128 experts). Less to read each step suits bandwidth-limited unified memory.

26B

total params

→

~4B

active / token

The machine: NVIDIA DGX Spark

128 GB

unified memory (LPDDR5X)

~273 GB/s

memory bandwidth

GB10

Grace Blackwell Superchip

$3,999+

desktop-class price

What developers liked

High parallelism + snappy local responses
Practical for chat, reasoning, tool calling, agents
Feels "purer" and easier than a high-end gaming GPU
Private, low-latency, multi-instance serving

The caveats

Needs patches, Docker flags & a quantization plugin
MoE

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.