Preview: PrismML's Bonsai 27B, open-weight MoEs, and local agent control planes.
Hey — Running larger models locally has always been a game of hardware compromise. This week, we saw some clever software and quantization tricks that suggest we might be closer to bypassing the VRAM bottleneck entirely.
🔥 This Week's Big One
Bonsai 27B by PrismML
Bonsai 27B is a 27-billion parameter multimodal model (derived from Qwen3.6-27B) compressed down to 1-bit (3.9 GB) and ternary (5.9–7.2 GB) variants that can run directly on consumer devices. By running locally in-browser via custom WebGPU kernels, it demonstrates that near-FP16 precision is achievable on a 10GB memory footprint, making phone-based local inference of large models a practical reality. https://huggingface.co/prism-ml/Bonsai-27B-gguf
🧠 Model Releases
- Inkling by Thinking Machines — A 975-billion parameter open-weights Mixture-of-Experts model (41B active) featuring adjustable reasoning effort and a 1-million-token context window. https://huggingface.co/thinkingmachines/Inkling
- QLLM (qllm2) — An experimental attention-free architecture that represents tokens as complex numbers in Hilbert spaces, offering constant-time inference and O(1) memory scaling for long context without a growing KV cache. https://github.com/gowrav-vishwakarma/qllm2
- OvisOCR2 by ATH-MaaS — A highly compact 0.9B multimodal model capable of parsing full-page document images directly into structured markdown, bypassing complex layout pipelines. https://huggingface.co/ATH-MaaS/OvisOCR2
🛠 LM Studio / Ollama
- Ollama v0.32.0 — This stable release introduces a CLI-based Agent UI, turns Ollama into an interactive agent environment, and adds parser support for Qwen3.5/Next formats. https://github.com/ollama/ollama/releases/tag/v0.32.0
- LM Studio Bionic — A privacy-first desktop application for open models that adds local voice dictation, project checkpointing, and zero-data-retention APIs. https://lmstudio.ai/bionic
🤖 Agents & Frameworks
📦 Tools & Repos
- Colibri (JustVugg/colibri) — A dependency-free C inference engine that uses "expert streaming" from NVMe SSDs to run massive MoE models like GLM-5.2 (744B) on machines with as little as 25 GB of RAM. https://github.com/JustVugg/colibri
- Mesh LLM — A distributed peer-to-peer inference system built on the iroh protocol that allows developers to pool GPU and CPU resources across multiple local machines to run large models. https://github.com/Mesh-LLM/mesh-llm
⚡ Quick Hits
- Ollama vs. llama.cpp Overhead — Community benchmarks indicate that running models directly via bare llama.cpp yields a 10% to 30% speedup over the Ollama wrapper daemon due to less API overhead. [link]
- GitLost Vulnerability — Security researchers disclosed an indirect prompt injection vector where unauthenticated attackers can use public GitHub issues to hijack autonomous repository coding agents. [link]
Until next week, keep your inference local and your cache warm.
— the LocalAILab team
