Yet another Hacker News

Desert Ant Labs: local, fast models that run on device

323 points willwhitedc 8 hours ago 86 comments

sipjca 7 hours ago | parent

at first i got very excited about a new fast transcription model (voz) but turns out its just parakeet v3 with some new inference code which is macOS/iOS specific

gorgmah 6 hours ago | parent

I was also thinking that this is almost too good to be true

pveugen 6 hours ago | parent

It's an ANE optimized version of Parakeet, with our own inference, which enabled us to push performance to about 300x realtime speed on an iPhone 16/17. Our next gen Voz model is trained from scratch and will be at least twice as fast. Android and other platforms will land soon.

Muromec 5 hours ago | parent

Nice, now I know what I actually need.

nullbio 7 hours ago | parent

This is a cool idea. The most useful one for me would be something that can process pdf files into a json schema. Title and tag generation from a post would also be useful. I'm interested in web app though.

pveugen 6 hours ago | parent

OCR on steroids. Our Schemer model will soon be available (free form text to structured JSON). Once that lands, we want to jump into image to JSON.

capevace 4 hours ago | parent

But Schemer will be text-only, correct?

Once image support drops this could be a cool addition to https://struktur.sh. Will it be on OpenRouter too or will all inference have to be self-managed? I’m thinking about server-side use cases, where budgets are low, so small models shine.

edit: ah I saw mainly iOS for now. But an integration would be possible on macOS then, right?

pveugen 1 hour ago | parent

Schemer is text only. But we're working on sort like models for images to data too.

We aim to make all our models available cross-platform. There's a subset currently only iOS/macOS, because they take some more effort to port over to Android and web.

Check out the CLI for easy experimentation: https://github.com/Desert-Ant-Labs/desert-ant-cli#desert-ant...

ironsmoke 1 hour ago | parent

You should check out IBM's Docling.

ashenke 7 hours ago | parent

A lot of the models would be useful in a web context, to improve on the CMS we're making for clients. But they look like most of them are iOS only, few have a node package or something other, and all the benchmark are running it on modern iPhones so I doubt it would be that fast on a 20$ VPS.

pveugen 6 hours ago | parent

Just a few are iOS first (pure practical timing/sequencing). We plan to make all models available cross-platform in the coming weeks.

library8848 7 hours ago | parent

Shiny layer of marketing and proprietary code on top of open models?

Voz is Parakeet 0.6B v3

Clear is DeepFilterNet 3

Ear is the language predictor from whisper-tiny

...

sipjca 6 hours ago | parent

seems to be…

Muromec 5 hours ago | parent

AI beige theme and obnoxious AI writing signaled as much.

sudb 4 hours ago | parent

It is now also reasonably straightforward if you have access to frontier LLMs, a recent-ish mac and a recent-ish iPhone to point them at the job of porting a given model to run on the ANE - it's a reasonably easy task to hill-climb at this point!

joshuat 3 hours ago | parent

but they're European!

illright 7 hours ago | parent

I wonder why they only support Apple platforms, citing CoreML. Doesn't Android have a similar framework, ML Kit?

pveugen 5 hours ago | parent

We plan to make most our models available for Android and web too. Some are a bit harder to port to the different platforms and will take a bit longer to properly land on Android or web. Mostly sequencing (Voz, Clips, Title). Soon!

1dom 7 hours ago | parent

This is a cool way of approaching AI models. I'm a big fan of local LLMs, local specific models like this should be even more powerful.

> Every model is free up to 100k monthly active devices. No tokens, no logins.

I dunno about the business model though. Cloud LLM billing makes sense: you're getting another computer to do work with each request, and using their compute via their gateway that they bill you.

These local models are like old school software. They're producing the weights, and then giving them to people. If I'm happy with the weights you've given me, and I'm not using your compute for inference and not wanting or needing any updates off you, why should you continue getting money off me and my customers?

The whole "but we need to keep it updated for your security" doesn't really work as well for software designed to run fully offline like these local models are.

I'm not saying they shouldn't get paid, but I guess I feel a personal sadness that it's less obvious how to successfully monetise such a sincerely useful and beneficial approach towards AI models.

donavanm 6 hours ago | parent

> why should you continue getting money off me and my customers?

Welcome to the concept of fair market value. Less snarkily you're conflating the concepts of price and cost; theyre not the same thing and theyre not the same for you or the seller.

Muromec 5 hours ago | parent

How does piracy of digital content and software factor into this concept?

handfuloflight 6 hours ago | parent

How is it not obvious and fair that they are asking you to pay them when you see success (defined as >100K MAU)? How more aligned can you and them be besides this?

Muromec 5 hours ago | parent

When I buy I chair I don't pay a share of my income to the furniture shop when I get rich. I would buy another chair at some point too, maybe fancier one (or the same). Because chairs are commodity. I do however pay taxes to the government based on my income because it keeps doing ongoing maintenance on everything.

Everyone wants to be paid forever for something they produced once is some kind of a mind virus. Make me a better chair and maybe I will buy it, but don't expect to become a trillion dollar company. It's deeply unfair to everyone who wants to be a trillion dollar company of course.

What is really funny to me -- the ones that do make it to collect the rent indefinitely also decrease their own taxes paid to the government AND also decrease the amount of contribution to society by making less and shittier stuff.

handfuloflight 5 hours ago | parent

Analogy doesn't apply to this case, as you are paying for the chair upfront but not paying for the model weights upfront.

Muromec 5 hours ago | parent

I'm also not downloading a chair, because why would it?

handfuloflight 4 hours ago | parent

Exactly, so your analogy is useless.

1dom 4 hours ago | parent

The only reason my chair manufacturer would be entitled to the proceeds of my success whilst I was in their chair is if I hadn't already paid them for the work they put into making the chair for me. Even then, why would my success with their product have anything to do with how much they can charge me for their labour? What other things work like that outside of the modern tech world?

Seriously, look around you, pick a thing, imagine a world where the price you paid for that thing hinged on your personal individual success with that thing.

edit: to clarify, my point is why can't we pay for the model weights up front for offline models? Why would we continue paying for things we don't want/need anymore, like updates for offline software that does it's job fine?

handfuloflight 4 hours ago | parent

If they charged for the weights up front, then your critique would simply reverse.

wredcoll 2 hours ago | parent

What, why?

ricardobeat 2 hours ago | parent

> look around you, pick a thing, imagine a world where the price you paid for that thing hinged on your personal individual success with that thing

YCombinator. Unreal Engine. CryEngine. FMOD. Steam, App stores in general. Qt.

+ 90% of enterprise software that charges per seat / user.

You could argue that most of these provide ongoing services, but that will also be the case if you license one of their models commercially, contracts come with strings attached both ways.

Muromec 1 hour ago | parent

>The only reason my chair manufacturer would be entitled to the proceeds of my success whilst I was in their chair is if I hadn't already paid them for the work they put into making the chair for me

Unless you could lease a chair and limit your cost to some finite number (which is higher then upfront cost), because chairs are easy to get by. You could also buy very cheap chairs from China for example and rent them by the hour (or that's what I heard).

The actual reason to why is -- because they can make you do it.

>What other things work like that outside of the modern tech world?

taxes

visarga 4 hours ago | parent

> When I buy I chair

You should, sitting on that chair got you in the zone where you made it rich. /s

Muromec 1 hour ago | parent

Right, right. Maybe correct posture gave me exact edge over the competition I would have missed otherwise.

adhi001 4 hours ago | parent

A model's weights can be pirated unlike a chair and that is the difference..?

HeatrayEnjoyer 3 hours ago | parent

What does that have to do with the business model?

chadash 6 hours ago | parent

> If I'm happy with the weights you've given me, and I'm not using your compute for inference and not wanting or needing any updates off you, why should you continue getting money off me and my customers?

I definitely understand the appeal of the desire to “buy it once”, but I think there are a few issues:

- almost no software is static. Look at a package like python Requests and even though it does the simplest thing and has barely changed from a user perspective, it gets updated all the time. This is true for most software. This is doubly true for something like local AI models where both the software and the hardware are changing constantly. Subscriptions motivate sellers to keep their software up to date.

- If I’m an app developer, the idea that I can try something out for $X/month is very appealing versus making an upfront investment of (let’s say) $X*20. This is doubly true for something like local models where I will almost certainly want something new when the models improve.

- To add to the first point, I work at a startup. No one asks questions when I want $20/month licenses. But let’s say I want something that’s gonna be in the 5 figure range annually. If I go to my CFO and ask for $50k upfront and then we implement something and the project fails, I look like an idiot. If I ask for $2000/mo budget for something and then we try it for two months and it fails, no one cares. Subscriptions are just safer in this sense.

TechSquidTV 6 hours ago | parent

LLMs are static though.

sejje 5 hours ago | parent

In the way that, say, Garmin Maps are static, I agree.

Caracas288 5 hours ago | parent

Very much like map CD-rom updates, I think a lot of people wouldn't mind paying per-update for these models, and would understand the implications. That is, if there weren't so many open models available.

TechSquidTV 5 hours ago | parent

... So we both agree.

1dom 4 hours ago | parent

I think the issues you're describing are all modern internet software issues. I'd guess you're possibly a bit younger than me, or maybe forgotten how software worked before about 2010.

- All compiled software is static. New versions are released. I'm happy to pay for new versions/features if I want/need them. But if the version I have is working offline, it doesn't need security updates, and if it does what I need, it doesn't need new features. Same applies if I'm building software for others, if the library does what I need for my software for my customers and I need nothing more off the dev, why should I keep paying the dev?

- I'm struggling not be sarcastic here, sorry, but this might shock you to hear: most humans find the idea of being able to try whatever they want and get paid for it appealing. That's not really how money or life works though.

- I don't know how to respond to this point, it's a bit of a strawman. Yes, if the only other option is to pay $50k up front for a local model, then your point is valid.

As a kid, I spent literally hundreds of £ on the first Sims, and Sims expansion packs at £20 /pop. EA got loads of money off me, I got loads of new features I wanted. I moaned at the time, but on reflection, everyone was a winner. Sims is still going now (possibly as a subscription!)

Again, apologies for a bit of sass here (that's a me problem), but the number of people who seem to sincerely believe the points you've put forward is why I'm personally sad at the state of modern software economics.

chadash 4 hours ago | parent

I'm old enough to remember buying software in cardboard boxes at Bestbuy.

> I don't know how to respond to this point, it's a bit of a strawman.

Why is it a strawman? I can sell you software for $100 once or I can sell it to you for $10/month. I don't think anyone is suggesting that I'm gonna sell it for $10 once though.

In particular, Desert Ant is saying their software is free for 100k active devices per SDK. That's likely a lot more users than most peoples' side projects. Given that, I'm guessing that $50k (even annually) is not a crazy number (yes, obviously $50k is not what a consumer would ever pay).

1dom 4 hours ago | parent

It's a strawman because it only makes sense if the one off up front cost is $50k, which wasn't the argument I was making. If the one-off up front cost is $20, then your point makes no sense - your boss would have no issue.

An alternative would be instead of 1 of charging over 100k users $50k, charge every user a one off $20.

I guess it's hard to discuss alternative billing methods without knowing what need to be recuperated.

Thanks for the response, appreciated. Again, sorry for the sassyness.

horsawlarway 4 hours ago | parent

50k annually for a static set of weights is actually crazy, though.

As in - I make software for a living and they can fuck off. Produce the item. Sell the item at a cost you find reasonable. Convince me to buy it again with an upgrade that's worth buying.

Period.

There's no "service" happening here. If anything, the proposed cost model just makes me incredibly unlikely to engage at all, because it means they're going to insist on trying to track my users (a thing I don't like doing...) with whatever bloatware they shove into their "sdk".

weitendorf 2 hours ago | parent

They're not selling it to you/it doesn't come out of your salary?

They're selling it to your employer. You're literally not even the customer for this product. There's no need to get angry that they're charging enterprise/vendor rates for a product.

And it's free for up to 100k devices. If you are running anything on 100k devices you should be able to afford $0.50/device and. If you aren't then why do you care? It's still a very customer-permissive and flexible business model so I cannot understand why this is upsetting or offensive

horsawlarway 1 hour ago | parent

they literally would be selling it to me.

I deploy browser extensions to millions of devices, and small local models that I can run in that space are incredibly appealing.

But the "per user" model is just especially difficult to justify here, and comes with a TON of fucking downsides like usage tracking, feature availability, and product structure (ex - is a trial user a user? how long? etc...)

---

And it's not even like they're justifying it with model download support or caching... they shove it entirely onto hugging face, or have you self serve.

As their literal target audience - I'd rather buy the weights for a fixed price (even a high one), and not let them influence my product design decisions with saas style pricing.

wredcoll 2 hours ago | parent

> - All compiled software is static. New versions are released. I'm happy to pay for new versions/features if I want/need them. But if the version I have is working offline, it doesn't need security updates, and if it does what I need, it doesn't need new features. Same applies if I'm building software for others, if the library does what I need for my software for my customers and I need nothing more off the dev, why should I keep paying the dev?

Who is allowed to make copies of the library?

pavlov 39 minutes ago | parent

Are you forgetting that the upfront price for software used to be a lot higher than it is today?

You’re talking about times before 2010 and an EA game, but that’s not the right comparison. These Desert Ant AI models are an SDK, a professional middleware library. Licensing that kind of software in the pre-2010 days used to cost tens of thousands of dollars.

In other words, the proper comparison here is the various libraries that EA licensed to make the game, not the game itself.

Aurornis 6 hours ago | parent

Because they own the IP and they get to decide the terms of how it’s licensed.

This is like a company taking open source software, saying they like the code the community has given them, and asking why they should continue having to respect the terms of the license after downloading the code. The availability of the software (or models) does not equal a free license to use as you please.

A license allowing 100K devices for free is very generous. The businesses selling more than 100K units of anything will be significant operations. It’s fair that they’re asked to contribute financially.

sejje 6 hours ago | parent

He's not arguing about their right to license it how they please. He's arguing that someone ought to choose to do it another way.

You made up a position to argue against.

Aurornis 6 hours ago | parent

No, you’re arguing against the first part of my comment without reading the whole thing where I addressed the value proposition.

A 100K free device license is a generous gift to small companies. Past 100K, it’s more than fair to ask that companies contribute. That’s a significant operation at that point and the model has obviously provided value.

If the model doesn’t provide any value to a >100K device company then they should train their own or not use it.

I could see how people unfamiliar with per-device or per-unit licensing fees would be confused, but this is a common business model. Scaling payments with the customer’s business success is one of the more fair ways to align vendor and customer while also providing a nice way to give freely to smaller businesses like this.

It’s a good thing. It’s silly that a company giving away a license for 100K devices, which covers small and medium hardware operations, is being criticized for this of all things.

serf 5 hours ago | parent

>Because they own the IP and they get to decide the terms of how it’s licensed.

how'd that work out for the rest of IP owners that had their stuff used as training data for use in model creation?

was the material they trained on produced in-house?

Aurornis 5 hours ago | parent

Use as training data has been tested in court multiple times and cleared.

I know some people want it to be a legal violation, but it’s not.

Also these are audio models, if you hadn’t noticed. The training sets for this type of work is very different than the corpus of scraped GitHub repos.

Chris2048 3 hours ago | parent

Presumably it's fine to train a model on their model then?

HeatrayEnjoyer 3 hours ago | parent

This is not even remotely settled law yet, what an absolute falsehood!

It will take years for the hundreds of jurisdictions in the world to write new laws and (in common law countries) establish new precedent.

Dlemlo 5 hours ago | parent

Was the material, I was trained, produced in-house?

Philosophical questions don't make sense if you use them to counter argue about something which is a legal issue.

avazhi 3 hours ago | parent

> how'd that work out for the rest of IP owners that had their stuff used as training data for use in model creation? was the material they trained on produced in-house?

Not only did you pull a whataboutism but the whataboutism isn't even relevant to the topic of conversation lol.

wredcoll 2 hours ago | parent

> Because they own the IP and they get to decide the terms of how it’s licensed.

Well, the laws that society decide what is legal in terms of buying and selling.

People, somewhat unfortunately, are starting to act as if "licenses" are some kind of natural, pre-existing part of the world.

The fact that they exist at all is because we invented them, and we can easily uninvent them.

embedding-shape 5 hours ago | parent

How would a model where you pay per each model/weight update feel like? More similar to traditional software, where you'd get the current version, maybe even free/rebated version upgrade to the next, but then have to do another purchase to get the latest version? Or something similar to that?

Feels like it'd be more aligned with how the business and users would produce and use it, it'd be something like: "We've done a bunch of research and output, here is the thing, you can buy it, and we'll work on the next, then you can pay again for the next if you'd like".

1dom 4 hours ago | parent

This seems like common sense. I guess it's not seen as common sense any more because the assumption is that all software is going to be using someone else's compute, or be connected to the internet and therefore require regular updates.

If neither of those are true, a subscription or recurring license or similar regular fee just feels like greed/robbery.

DrScientist 2 hours ago | parent

As far as I can see there is no clear pricing once you go beyond the free tier - it's contact sales.

This is actually a classic enterprise software approach - as there are no fixed per unit production costs to cover, the pricing is very elastic.

ie each license is probably a negotiation - if you see your app for a one off fee then they want a one-off cut - if you sell a subscription they may want a cut of that instead.

Obviously the risk is if you bake it in and your app becomes successful they may want a larger cut than you can afford, but if they don't negotiate in good faith they won't last very long.

sureMan6 33 minutes ago | parent

What's worrying for them is nothing they're doing is impossible to be recreated, if you bake it in and you don't want to pay rent you can fine tune your own model

ricardobeat 2 hours ago | parent

What do you mean by "continue" if you haven't paid anything yet at that point?

wredcoll 2 hours ago | parent

> These local models are like old school software. They're producing the weights, and then giving them to people. If I'm happy with the weights you've given me, and I'm not using your compute for inference and not wanting or needing any updates off you, why should you continue getting money off me and my customers?

You are, presumably, making copies of their software for your customers. So I guess the real question here is, should copyright exist? For software?

mtlynch 6 hours ago | parent

I love this idea and hope to see more on-device models. How do they make money, though?

I tried out their demo for Clear, the audio quality improvement model.[0] I'm not sure if it's just I don't have refined enough an ear or their demo is broken, but the "raw" and "enhanced" versions sounded exactly the same to me.

[0] https://desertant.com/models/clear/

sudb 6 hours ago | parent

The underlying model (DFNet3) is not particularly great but it is very small and fast - imo the best commercially usable denoising model is MossFormer2 (no affiliation - it's just excellent) with one drawback in that it can't remove reverb.

Nvidia's RE-USE model can do what MossFormer2 does _and_ can remove reverb, but it is non-commercial licensed.

bronlund 6 hours ago | parent

The website looks amazing.

pveugen 6 hours ago | parent

Thanks!

wuisce 5 hours ago | parent

How was it built?

pveugen 5 hours ago | parent

We design in Figma. Build a design system out that with components for all our public facing products, demos and marketing assets. The website is built with the components, and we have a set of skills and tools to keep both in sync. We also use skills to sync between HuggingFace, our SDK on GitHub and the website. We write and review most our core copy by hand and use that to help expand into different pages. We prob do a write-up about this on our blog later.

tuxguy 3 hours ago | parent

brilliant ! Would love to see a deep dive on your component design system & copy process.

Here is to more beauty in the world - both physical & digital :)

markdog12 6 hours ago | parent

> opinionated on-device intelligence

> Hate speech triage. On-device moderation that flags hateful, abusive and threatening text

What could go wrong here?

lemome 5 hours ago | parent

I don't think you understood. It means these are specialized models. Their toxic model could be ideal for video game lobbies without investing a ton of money if you're an indie dev

This also could be ideal if you want your child to play online to have auto-censorship

ch_sm 4 hours ago | parent

I don’t know if i want to expose my child to auto-censorship (or online gaming anyway).

Muromec 5 hours ago | parent

1. Not having human in the loop to review it because humans are expensive.

2. Having human in the loop to review it and subject said human to the worst other humans produce.

nater5000 5 hours ago | parent

I definitely think there's a lot to be done with small models dedicated to specific tasks. I've always thought the REAL value is in having large models be able to easily build small models for custom tasks (which I know is kind of a thing), but perhaps just providing the small models directly is the more accessible approach.

>accessible via one SDK for Swift, Kotlin, and JavaScript

Lol well let me know when there's a Python SDK and I'll give it a try then. Obviously this isn't a deal breaker if you have a real case, but as someone who is willing to spin something up and try it out if there's a quick "pip install" command, this is getting put back on the shelf for now.

embedding-shape 5 hours ago | parent

The founders and website seems to talk a lot about mobile devices and how many are being sold/shipped, that's the "unused compute power" they're targeting. So obviously mobile-first platforms and SDKs seem to be the focus first, hence those languages. Understandable, given their target, but just like you I wish there was an easier way to give it a try on a desktop computer.

ChickeNES 4 hours ago | parent

Same, just wish I could afford the computer to train my own ngl

lukevp 4 hours ago | parent

I would love to use Voz and Ear, but I’d need a version that is competitive with other audio transcription LLMs for platform availability - meaning macOS, Windows and Linux, and supporting GPUs if available.

faangguyindia 4 hours ago | parent

Cool! is there a local model for LLM command approval?

init0 4 hours ago | parent

Awaiting web version...

viccis 3 hours ago | parent

I wish we could pop a tiny one into my phone so that when I type "Will see you" and swipe the word "later" it chooses that instead of "lasso"

agcat 3 hours ago | parent

Honestly this is a really cool idea and i am glad that there are companies being built in this space. This is closest to the vision of what i want to do next.

ricardobeat 2 hours ago | parent

> Ranks a transcript's best non-overlapping moments: each clip gets scored and ranked. Build strong selections or unique editing features to pick the best sentences in video or audio recordings.

This is very impressive for a 248MB model. I wonder how good the results are, as an LLM 10x the size is still quite bad at that.

Dwedit 1 hour ago | parent

Testing out Tongue, it detected "馬鹿外人" as Chinese.

robotresearcher 3 minutes ago | parent

For the many people here complaining about the per seat pricing (once a good scale is reached), does it help you to consider that you’re making a copy of the model and providing it to each of your customers?

From that point if view, by wrapping it up in your value-added app, you’re an integrator/reseller of the tech, and you can charge for that appropriately.

Of course it’d be lovely and cheap for you and your users if you could pay a modest one time fee instead. But the original dev has to make a living, just like you do.