Yet another Hacker News

35 points joshmgross 2 hours ago 19 comments

jeffbee 1 hour ago | parent

The fact that you can just pay to scale out point reads is not news to anyone.

AdamProut 55 minutes ago | parent

yeah, this is a definitely a "best case" workload for a sharded database. Single row reads on the key used to shard with no hotspots (no shard to shard network traffic at all).

samlambert 1 hour ago | parent

It cost $250,000 to do this run but it feels worth it.

handfuloflight 1 hour ago | parent

I did not know men could build such things.

whalesalad 1 hour ago | parent

I estimated the cluster to achieve this was ~$3-4k per-hour. I am thinking there is a typo on the r8g.16xlarge and they are actually r8gd.16xlarge (notice the d) which comes with directly attached nvme disks.

svuiv 54 minutes ago | parent

We used r8g.16xlarge instances with EBS disks, no nvmes

whalesalad 22 minutes ago | parent

would love to hear more about the ebs volumes, iops/size/raid configuration

svuiv 19 minutes ago | parent

each shard had a 4TiB volume with 65k IOPS and 1,500 mbps of throughput

rcrowley 28 minutes ago | parent

It felt rude to take so many r8gd instances away from our customers who really love those (and i8g and i8ge).

jeffbee 47 minutes ago | parent

That's roughly 250x more than it would cost to perform this stunt using on-demand Cloud Bigtable, if my math checks out (~1150 nodes @ 85¢/hour for 1h).

znpy 1 hour ago | parent

If this is closed source then i have zero interest in it.

noir_lord 36 minutes ago | parent

Someone posted a twitch conversation yesterday about this, I poked around on the page realised there was no open source version and noped out immediately.

I'm sure it's a great product (it seems like planetscale do good engineering and the folks I know who use them seem fine with it) but I don't do vendor lock-in as a service personally, I'll use whatever employer uses because that's the deal but for personal stuff, well this isn't designed for that really, wrong order of magnitude on scaling.

jjice 32 minutes ago | parent

I believe multigress is the similarly aged open equivalent from Supabase. Haven't used it myself and don't know what the differences are in usability, but I'm a bit more interested in that since it's open.

spongeboi 24 minutes ago | parent

unfortunately they haven't been able to move forward w/ the project, it can't even shard yet

samlambert 20 minutes ago | parent

Neki right now:

Multiple live shards: yes

Query routing across shards: yes

Online shard splitting: yes

Zero-downtime resharding: yes

Multiple independent shard groups: yes

Data topology management: yes

HA / automated failover: yes

Multi-AZ: yes

Connection pooling: yes

Online schema changes: yes

Workflow-driven migrations/cutovers: yes

Zero-downtime imports: yes

CDC / logical replication: yes

Online Postgres version upgrade workflows: yes

Cross-shard transactions: coming

Multigres today:

Multiple live shards: no

Query routing across shards: no

Online shard splitting: no

Resharding: no

Multiple shard groups: no

HA / failover: yes

Multi-AZ: yes

Connection pooling: yes

Logical replication/import work: in progress

Distributed migration/resharding workflows: no

How it is an it's an alternative? Do you just say things without validating?

AdamProut 54 minutes ago | parent

I'm curious why the test needed so many router hosts:

512 shards, each with one Postgres primary each on an r8g.16xlarge

480 Neki routers, each on its own 8xlarge instance

That's ~250K queries/sec per router which seems lowish for this type of workload? The routers won't be doing very much (parse query, route it to proper shard?).

svuiv 20 minutes ago | parent

that's over 13k queries/sec per router core, about 50% of it is spent doing syscalls, the other 50%: parsing, doing grpc, tls, go gc, resolving the shards, waiting for the responses

neki is still in platform preview, this experimentation allowed us to collect profiles at such scale and ship some nice optimizations, more are coming

danbruc 30 minutes ago | parent

87.3 % served from cache. Does that mean it returned a result existing in the cache because the very same query was executed before? Probably still a relevant result, if you have to process millions of queries every second, it seems not unlikely that you will see a lot of repeated queries. But at that point you are measuring cache performance more than query performance. But unless you run some standardized query benchmark, a single queries per second number is not that informative anyway because query complexity and therefore execution time can span many others of magnitude. Looking up a name by ID and aggregating across a billion rows from seventeen tables joined together are both a single query.

farazbabar 15 minutes ago | parent

In 2015, I was able to get to 1 million read/write queries per second on only a couple nodes and tested this with multiple databases, it required (at the time) decent network tuning and node placement inside AWS but it cost me about 10 to 15 dollars per run if I recall correctly, obviously there is the matter of scaling such performance and so I want to recognize the engineering effort gone into this but this is too much money. This reminds of when one of my teams used Hadoop to process only a a few terabytes of offline data and were able to process the WHOLE THING in only a few hours. I did not have the heart or courage to tell them during the demo that this was overkill, but I did write a very simple (and small) piece of code that could extract all the signals from the offline files in mere seconds with careful network planning and storage optimization and invited them for a demo/lunch and learn next week.