Running 800B-parameter LLMs on consumer hardware with InferSSD
What happened
InferSSD lets consumer GPUs run 800‑billion‑parameter Mixture‑of‑Experts models by keeping most expert weights on NVMe storage and loading only the needed subset into RAM or VRAM. On an RTX 5090 with 32 GB RAM and 32 GB VRAM, it processes DeepSeek prompts at 164.5 tokens/s and generates at 7.27 tokens/s. A 48 GB M5 Max MacBook Pro achieves 26.27 tokens/s prefill and 2.70 tokens/s decode. The same model on an RTX 5060 Ti with 8 GB RAM and 16 GB VRAM reaches 75.1 tokens/s prefill and 2.55 tokens/s decode. The system uses llama.cpp, supports GGUF files, and offers an OpenAI‑compatible API.
Why it matters
This approach could let hobbyists and small teams run very large language models on modest hardware, reducing the need for expensive cloud inference. It demonstrates that SSD‑based offloading can make high‑parameter models more accessible to non‑enterprise users.
Reality check
An automated, sceptical read of the sourceIncremental. The technique builds on existing SSD‑offloading ideas and uses llama.cpp; it improves performance but does not introduce a fundamentally new paradigm.
Is this a big deal?
Get stories like this every morning
The day's AI news in plain English, in your inbox at 8:30 IST. Free.
More AI news
Every Way to Get 192GB for Local AI (Without a $9,500 Mac) [video]Industry news · 14 Nov 2026DeepSeek V4 Pro 0813DeepSeek · 14 Nov 2026DeepSeek V4 Pro 0813DeepSeek · 14 Nov 2026Show HN: Portal2Bench – Can AI agents solve Portal 2?Industry news · 14 Nov 2026Anthropic Upgrades Claude Code Projects to Run Parallel AI Workers Hands-FreeAnthropic · 14 Nov 2026This page was collected from wici.ai and summarised automatically by AIPulsee's pipeline. See how it runs.