Yet another Hacker News

Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)

28 points anemll 1 hour ago 10 comments

spacecoder2 1 hour ago | parent

Anyone know how hot an iPhone gets running the Neural Engine flat out for 10 minutes?

anemll 1 hour ago | parent

It's warm but not hot. On M6 it's about 50% less energy consumption in my tests vs GPU. I think iPhone should be similar

GeekyBear 54 minutes ago | parent

The neural engine is optimized for energy efficiency, and the GPU is optimized for performance.

You're going to get much better battery life and much less heat running on the neural engine.

analog_daddy 1 hour ago | parent

Hi. Could anyone share any recommendations (hopefully with actual links) for App Store apps to try running such local models? I’m not really interested in side-loading since it currently seems (to me) more of an hassle than it’s worth.

App Store search is broken in a sense that there might be so many apps claiming to be local but once you read reviews, it’s just another scam where it’s it’s usually sending your query to remote server.

simonw 57 minutes ago | parent

My current favorite is this one: https://apps.apple.com/us/app/locally-ai-by-lm-studio/id6741...

It's free, not annoying, and has a bunch of model options.

layer8 48 minutes ago | parent

Non-app-store link: https://locallyai.app/

However this uses the GPU, not the Neural Engine like the submission.

jdgoesmarching 46 minutes ago | parent

Note that the model discussed in this article isn’t available in this app running on the phone. Still, it has some good local options like Gemma 4 and can link to the desktop LM Studio app which is cool.

woadwarrior01 46 minutes ago | parent

I'm the dev behind the Private LLM app, which was the first on-device LLM app on the App Store. Don't download it yet. I'm currently finalizing a brand new v2 rewrite which can run models that the copycat app (locally) cannot run, like Bonsai 2 27B, Gemma 4 12B etc.

https://x.com/private_llm/status/2101769928252473488

Retr0id 53 minutes ago | parent

OP, asking for upvotes is against the HN guidelines: https://x.com/anemll/status/2111531629529456771

createful 35 minutes ago | parent

Not Qwen, Bonsai. Although it is based on Qwen, and achieves similar scores, it is not the "full" Qwen.

Last I remember, when trying to run a model above 1-2 GB on iOS, it would refuse to do so, at least from Locally AI. iOS only allows apps to use about 1-2 GB of RAM before being killed. Though using their Neural APIs might bypass this.