Yet another Hacker News

94 points pretext 1 hour ago 53 comments

swyx 1 hour ago | parent

their full talk is here https://www.youtube.com/watch?v=Ok6QdZHCcI8

mtrovo 1 hour ago | parent

30 cents per million tokens is wild.

The relevant question is how easy is to setup 56x B200s in a cluster, are we talking about a adhoc infra team job or clicking a button on AWS?

rrr_oh_man 25 minutes ago | parent

Sometimes the infra team job feels easier…

gosen 1 hour ago | parent

> Per David Abitbol, who leads AI ops there: 56 Nvidia B200s, a blended 30 cents per million tokens, roughly a tenth of what they paid Anthropic and OpenAI.

So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.

We'll see if that's a trend for mid-big companies or just an experiment.

All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.

InsidiousWombat 1 hour ago | parent

I doubt local will be all that common aside from regulatory requirements but buying them off your cloud provider will be the new norm for many enterprises.

The slice of the pie where frontier models are required is just gonna be smaller and smaller. And at point where AI wrote docs, tests, and all the easy and medium difficulty functions you are already saving a ton per employee

aurareturn 1 hour ago | parent

Personally, I think Anthropic and OpenAI will introduce enterprise plans such as $2,000/month/engineer plans that will give 100x more tokens than the current max plans and then charge API pricing.

Yes, margins will go down a bit but the frontier labs still have some leverage.

AidenVennis 1 hour ago | parent

The B200 was launched in 2024 if I'm not mistaken. The big gamble with this might be how fast these will deprecate in value/use as soon as Nvidia launches a new and better product.

At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?

mongrelion 54 minutes ago | parent

> So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling).

Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.

yolo3000 44 minutes ago | parent

It says in the video that they run them on Together.ai

gosen 28 minutes ago | parent

yeah, seems like they're renting (according to the info from the video), but doesn't it make the bill worse? I mean the provider makes money somehow, so it should be more expensive for revolut. I guess it's a lot anyway

f311a 35 minutes ago | parent

Unless you have very strict requirements, it does not make any sense to invest in hardware. Most companies just rent it. Also, it's pretty hard to get the hardware.

InsidiousWombat 1 hour ago | parent

If investors knew how powerful open models are the AI companies valuation would tank overnight

sunaurus 1 hour ago | parent

Well, with the recent news about OpenAI being unable to find investors, I feel like it's already happening

m00dy 34 minutes ago | parent

Don’t trust the news, especially now that AI is seen as part of US national security. We’re in a war, both financially and technologically.

yolo3000 49 minutes ago | parent

You still need to run them on GPUs

jabron 41 minutes ago | parent

AI companies aren't valued on the GPUs they own.

m00dy 30 minutes ago | parent

I think he meant NVIDIA is the winner in the end.

wg0 40 minutes ago | parent

And GPUs anyone can buy.

abc123abc123 24 minutes ago | parent

And if the market teaches us one thing, it is that GPU prices will drop over time. The current absurd prices is attracting competitors, and the more competitors, the lower the price.

I would not be surprised if, in 5 years, there's a neat, low-power GPU in high end laptops, alternatively, a cheap, say 500 USD or so, GPU in a box, that you can use for your AI-center at home or in the office.

Tade0 20 minutes ago | parent

MoE models can be largely offloaded to system RAM - recent versions of llama.cpp automatically distribute parts of the model, with all the dense parts going into VRAM and just the routed experts residing in system RAM.

I'm currently running Qwen3 Coder Next, which is a 80B A3B model, at 15tps on a 8GB 7700S and 64GB RAM.

It's an idiot (no Thinking), so it has to go over its work several times to get it right, but it's the largest idiot I can feasibly run at the moment.

Also by accident I'm undercutting the only provider of this model on OpenRouter by 40%, doing $0.48/mTok.

Of course at my token rate it would take an entire day to even reach a million tokens, but it's the principle that matters.

rrr_oh_man 35 minutes ago | parent

No, it would be hyped up even more for a pump & dump :)

aurareturn 1 hour ago | parent

It's probably true that open weight models are "good enough" for most software engineering tasks.

But the goal posts will move and I think frontier labs will find new areas that have a need for absolute frontier.

cassianoleal 1 hour ago | parent

"Frontier labs" makes it sound like there's some special sauce to these companies. There isn't, other than very smart people working on problems.

Guess who else has that? The Chinese companies developing open weight models. In fact, the vast majority of AI science comes from China. They might still be playing catch up in some areas, but they're also leading in others (famously DeepSeek's inference architecture as an example).

There is IMO a non-trivial chance those labs will eventually surpass the US ones and become themselves "frontier", and that may happen sooner than a lot of people expect.

Of course, it still remains to be seen if they will keep publishing the weights at that point.

aurareturn 41 minutes ago | parent

Chinese labs are gated by lack of compute, at least until they can build their own EUV machines.

At some point, they'll stop releasing models for free and will need a license.

noir_lord 38 minutes ago | parent

> Of course, it still remains to be seen if they will keep publishing the weights at that point.

I mean if they don't then the US companies could just release their weights and chop the Chinese off at the knee's at that point (reversing what is currently happening).

Opening the weights isn't just a technical thing, it's a geopolitical decision which likely in part comes from the top of the CCCP (by either request or just implicitly allowing it).

Havoc 36 minutes ago | parent

> But the goal posts will move

On both frontiers and open. And lately the gap has been closing so if anything yeeting Anthropic etc is getting more viable over time

f311a 33 minutes ago | parent

> will find new areas that have a need for absolute frontier

And they will keep it to themselves to extract the value for as long as possible.

irthomasthomas 1 hour ago | parent

https://xxcancel.com/guilleflorvs/status/2117379073785696395

0192373u5 32 minutes ago | parent

Thank you! I only opened the comments to find this link.

jacquesm 1 hour ago | parent

We did the same thing, on much less powerful hardware but it has served us very well and we don't need to worry at all about the question whether or not the suppliers of inference services can be trusted long term with our data. The answer to that question in my opinion is 'absolutely not'.

mkesper 58 minutes ago | parent

Can you share some details about your setup? Would probably be interesting for a lot of people.

jacquesm 51 minutes ago | parent

Sure:

https://jacquesmattheij.com/local-inference-with-large-langu...

If you have more questions feel free to mail or ask here in the thread and I'll do my best to answer.

The hardware related parts are near the end of the article.

krona 13 minutes ago | parent

How do security conscious self-hosters sandbox their runtimes?

dannyw 5 minutes ago | parent

Your inference server is just an inference server that spits out tokens. You can use the same harness, etc. you’re basically just changing the API base URL.

torginus 57 minutes ago | parent

This is the right call - there's little proof that the quality gap between open models and OAI/Anthropic stuff translates to business results, but you surrender all control of your data to a third party.

The last cause might be a straight up blocker - certain financial regulations straight up prevent you from giving third parties to your data (esp outside national/EU jurisdiction), no matter what documents you sign.

injidup 55 minutes ago | parent

This is probably only relevant if you go over the 150 engineers putting you in the enterprise bucket and forcing you to pay by the token rather than the subsidized Claude code plans. If you need 150 engineers all with frontier model accounts maybe you are doing something wrong ;)

kasey_junk 42 minutes ago | parent

If you want zero data retention you have to go api only.

rrr_oh_man 32 minutes ago | parent

I feel ZDR is the same as the EU data privacy shield

traceroute66 30 minutes ago | parent

> If you want zero data retention you have to go api only.

Time for the daily reminder that ZDR is a pinky-promise.

ZDR is a contract clause. That means:

    - It is subject to the rest of the contract, e.g. the good old "subject to lawful request or court order" carve-out clauses
    - It is subject to random provider exceptions, e.g. see Anthropic and Fable.
    - You have no way of verifying it beyond "trust me bro".

Basically you the only options are:

     - Run your own hardware which is obviously the ideal situation
     - If you can't run your own hardware, use confidential computing which runs the AI in secure enclave, e.g. Privatemode[1] (Germany) or Tinfoil[2] (US)

[1] https://www.privatemode.ai/ [2] https://tinfoil.sh/

redmalang 51 minutes ago | parent

For the past several months about 90% of the work we do goes to open source models. If you ignore the noise around 'cheap' etc, you might find that with a little extra effort to more precisely define what you need not only do these models generally work as well, and they will generally more closely follow instructions - on longer timeframes this is actually pretty helpful.

They are less good on ambiguous prompts like 'fix all the problems' (which to my chagrin is a very common prompt it turns out..)

jacquesm 33 minutes ago | parent

Yes, that's a good point: you really need to work on the prompts, even if the harness is the same. Ambiguity is a real problem as is what we call 'the negative space', what isn't in the prompt but what is implied by the prompt to be undesirable. You need to spell that out in more detail it is not enough just to say 'do x' but you also need to make sure to say 'and not y or z', and knowing what y or z are requires a bit more insight into what hobby horses are encoded in a model.

To give one example: I'm doing a lot of stuff that involves hardware. When settings things up for fixing a bug I have to make sure to include that the hardware has been tested and works well. If I don't do that the model will almost certainly spend a ridiculous amount of time speculating about all the different ways in which the hardware could be broken (including theorizing about chip mask errors and all kinds of other obscure and rarely seen in practice defects). It's going to end up telling itself that for sure the 500 lines it just produced are absolutely perfect and the device that is on the i2c bus is broken in a way that it just really can't progress with solving the issue because there is nothing wrong with the code.

These kinds of blind-spots are funny when they happen once in a month but when they happen regularly it begins to be irritating. Adding a line or two to the prompt will take care of the issue. Frontier models tend to be better at this sort of thing.

redmalang 8 minutes ago | parent

Us too! Hardware is very interesting - I find in fact that 'smart' (aka expensive) models can overthink complex hardware state machines and end up doing atrocious code because it will try to handle every possible edge case (unbidden, mind you). What has worked very well is harvesting and aggregating lessons and feeding this automatically into our pi sessions, this allows the models which listen to instructions to avoid a lot of time burning tokens on internal dialogue to ascertain the nature of reality.. but the expensive ones: a) they dont listen b) they ironically know too much!

littlecranky67 34 minutes ago | parent

"the gains showed up when tokens stopped being rationed" - I can totally see that. I am leading (as an external consultant) a larger dev team for a huge insurance company. The devs are quite junior or in general more the "I became a sw-eng here because that company is the only big employer in a 1h radius" type of people (meaning: No rockstar developers). Most of them do not play with AI in their spare time, and they have been told to spend claude tokens wisely. Everybody is too afraid now to play with it and explore during work hours for fear of being token shamed.

At the same time, most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription and use it for toy projects. I have younger devs (early 20s) in the team that told me they won't spend private money, or simply have no usecase in their private life for AI. We talk about software engineers here, and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.

user43928 26 minutes ago | parent

If you were to take away my work's $500 budget for Opus 5.5 and replace it with unlimited use of GLM-5.3, it would be a huge downgrade in my opinion.

If your employer is so cheap as to spend a $100 or less per month per engineer, while being large enough not to have access to the subscription plans, a move to self hosted or cheap providers does make sense in my view.

littlecranky67 23 minutes ago | parent

The thing is the have been told to use the tokens wisely, but no limit is communicated. Still everybody is hesitant. To be fair, the company runs claude on AWS bedrock for data privacy requirements at a different pricelevel. But I guess that is the drawback for german job-security: each and all of them know they basically can't get laid off.

I also attribute a lot of it to be just plain outright denial. I have a feeling a lot of people don't WANT to understand that their coding skills they acquired over years have dropped a lot in worth. It is the "as long as I don't step on a scale, I can't be overweight" kind of denial. You are afraid to see what is behind the curtain, so you simply don't do it.

latexr 10 minutes ago | parent

> But I guess that is the drawback for german job-security: each and all of them know they basically can't get laid off.

Help me understand what exactly is the drawback you’re mentioning. In your previous comment you were suggesting these people spend their own personal money and personal time on projects to benefit their work time. They (quite reasonably, it seems from the limited context) decided that no, they won’t dedicate life and resources from outside work to more work. Fortunately, it seems they have some job security (German, but probably some from the EU as well) which allows them to make whatever choice they believe is healthier to them, without fearing repercussions. What’s the drawback?

littlecranky67 5 minutes ago | parent

No you misunderstood my comment. I suggested they spent their private money - 20€ a month for the lowest Max flatrate tier - because to me it is obvious that you absolutely need to develop the skill to work with AI. They only have the "pay-per-token" model on their work accounts and laptops, because that is what large corporations get - no flatrate. That leads them to not work with AI. Most of them just refuse to get that 20€/month subscription to experiment with it on their private laptops. I didn't say private time - company policy is no US cloud services for work related topics so they would have to install it on their private computers. They totally could work on toy projects during their time, as the team is in homeoffice a couple a days per week.

andy99 28 minutes ago | parent

This is a sane version of what “sovereign” AI can look like for many. In Canada we’ve just had this national plan to overpay for an inferior AI product in the name of sovereignty. Even if GLM is slightly behind, this is likely close to the Pareto frontier.

They also could consider putting this on openrouter when they have spare capacity to make the most effective use of the hardware.

chr15m 26 minutes ago | parent

Anthropic's missing last quarter results in a nutshell.

world2vec 16 minutes ago | parent

At dayjob we're also using self-hosted GLM-5.3 via our flavour of pi.dev but teams still have discretionary power over using Claude Code or not (there's a team's budget to spend on whatever the team decides). Some teams like mine still like to use Claude Code for a lot of the work but most stuff doesn't require the latest and greatest LLM and can be done with "lesser" LLMS by simply throwing a lot of tokens and compute to it.