YHK - yet another hacker news clone

Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

60 points JonSchneider 1 hour ago 14 comments

kamranjon 56 minutes ago | parent

Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!

kadoban 29 minutes ago | parent

You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.

This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.

spijdar 22 minutes ago | parent

I run Unsloth's UD-Q4_K_S on 20 GB of VRAM (RX 7900 XT) and I get ~90k tokens of context without quantizing KV cache. With 8-bit quantization, I get about a 134k token context window. That's with only one slot, but for me, it works pretty darn well, with 20-35 tok/s depending on how full that window is.

abraxas 54 minutes ago | parent

I'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?

kamranjon 53 minutes ago | parent

"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."

pizza234 29 minutes ago | parent

Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!

sisve 5 minutes ago | parent

They mention 5090 with regards to speed, Q6 will not have that speed?

And speed matters a lot for many use cases

Havoc 34 minutes ago | parent

Their first 27B bonsai was able to run on an iphone.

Aurornis 26 minutes ago | parent

These are small enough that you can run them entirely in the browser https://huggingface.co/spaces/webml-community/ternary-bonsai...

Remember to clear the downloaded weights afterward.

Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.

outofpaper 19 minutes ago | parent

So you have some fun examples?

z2 21 minutes ago | parent

I'd love to see a Bonsai model start with a 100B+ parameter model and get that down to <30 GB. But maybe at that point we call it Topiary?

JonSchneider 11 minutes ago | parent

I'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.

simonw 9 minutes ago | parent

If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...