YHK - yet another hacker news clone

Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard

33 points guregodevo 1 hour ago 12 comments

walrus01 51 minutes ago | parent

I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.

https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...

guregodevo 10 minutes ago | parent

will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx

egeres 42 minutes ago | parent

This also appeared on HN last week: https://news.ycombinator.com/item?id=50228167, it's the other method they mention, GVS5H

roenxi 41 minutes ago | parent

That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.

Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.

[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.

SwellJoe 31 minutes ago | parent

Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.

MrDrMcCoy 25 minutes ago | parent

Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.

SwellJoe 14 minutes ago | parent

I think that requires testing, rather than assuming. There's only so far a harness can go.

nicce 26 minutes ago | parent

Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.

mkl 39 minutes ago | parent

GVS5H, which this is based on, was discussed here a week ago: https://news.ycombinator.com/item?id=50228167

guregodevo 28 minutes ago | parent

Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.

xlayn 8 minutes ago | parent

Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...

Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...

fedpost 4 minutes ago | parent

We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.