Yet another Hacker News

Qualcomm Cloud AI 100 Ultra – 150W AI Accelerator Card with 128GB at 548GB/s

44 points peter_d_sherman 1 hour ago 18 comments

pearl_panel5 1 hour ago | parent

Curious how the memory bandwidth actually holds up under sustained inference loads compared to the spec sheet numbers.

varispeed 1 hour ago | parent

Aren't these specs already outdated? Seems meh. I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.

Give me 1TB RAM and 5TB/s or something that exciting.

tempest_ 56 minutes ago | parent

5 year old GPUs are still at crazy prices.

Plenty of businesses want in local inference and are priced out.

entrope 51 minutes ago | parent

For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.

mciancia 39 minutes ago | parent

8 of those cards give you 1TB of ram and 4.4TB/s

PhunkyPhil 34 minutes ago | parent

It may be a semantic thing but I don't think bandwidth pooling works like that? Unless your blocks are distributed perfectly between cards there's going to be data passed between each card and from VRAM to compute. Correct me if I'm wrong though

aetherspawn 57 minutes ago | parent

Ok so it’s significantly worse than Apple M6 and we don’t know if it’s cheaper or not

Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.

dist-epoch 47 minutes ago | parent

Apple M6 has 80-100 TOPS.

Qualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.

ipsum2 22 minutes ago | parent

Considering that memory bandwidth is the most important for inference for most neural networks, that matters more.

zitterbewegung 52 minutes ago | parent

Sort of interesting design It has half a terabyte of SRAM on the accelerator with a 128 GB of RAM replacing HBM with SRAM but keeping regular RAM.

2001zhaozhao 51 minutes ago | parent

half a gigabyte not terabyte lol

zrm 31 minutes ago | parent

Yeah, this thing is unusual. It resembles a consumer GPU but using normal DRAM with more channels, so then you get more RAM but less memory bandwidth than GDDR/HBM, which is the same thing Apple does. And the TDP is pretty attractive for what it is.

But what is that amount of SRAM for? It simultaneously seems like a lot, because SRAM is so expensive, but still not very much, because what can you fit in 0.5GB? Is there a specific workload that needs something that looks like this?

tokepi 49 minutes ago | parent

If the specs seem outdated, it’s because this chip launched in 2024. It’s available at cloud providers including Cirrascale

thway15269037 38 minutes ago | parent

What's the price?

simlevesque 27 minutes ago | parent

According to their own website, they cost 1 cent.

thway15269037 14 minutes ago | parent

Yeah, I clicked that too. But knowing qualcomm they obviously cost an arm and a leg. Still interesting how much.

dingdingdang 10 minutes ago | parent

Honestly, let me buy one, it's many times faster than the minipc I run local ai on so if the price is right I'm in.

swiftcoder 22 minutes ago | parent

So it's an M4 Max Mac Studio, RAM-wise, with a GPU's worth of compute?