Yet another Hacker News

59 points ksec 2 hours ago 17 comments

Tiberium 1 hour ago | parent

If anyone's curious - https://planetscale.com/docs/postgres/search/get-started#loc...:

They're not providing a local extension with the same performance at the time - it's only offered on their cloud services.

The local version https://github.com/planetscale/lead is mainly just for testing the syntax, it doesn't have the same perf characteristics.

noir_lord 1 hour ago | parent

Becoming more the norm for them, Neki is the same.

Immediately rules out ever using them (though I don't currently have any problems that would benefit from that level of scale currently, have in the past though).

Postgres's license allows this but for me (personally) it leaves a bad taste.

samlambert 39 minutes ago | parent

why?

dbbk 35 minutes ago | parent

Why would you need super fast search for local testing?

Onavo 22 minutes ago | parent

To avoid vendor lock-in.

dbbk 10 minutes ago | parent

It's Postgres search any coding agent can switch you to something else in 5 minutes

zombodb 16 minutes ago | parent

Why don’t you like Postgres’ license? It’s as permissive as a license gets.

pqdbr 23 minutes ago | parent

The problem is that they don’t support bare metal. I’d love to use PlanetScale in our bare metal servers.

bob1029 1 hour ago | parent

I struggle with FTS inside SQL (SQLite and MSSQL). There is often a fairly significant impedance mismatch between the relational concerns and how the documents need to be stored.

I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?

zombodb 34 minutes ago | parent

I’m one of TIN’s developers and if you google my username you’ll see I’ve been in this space for a long time.

The answer to your first question is simply: yes

As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.

gfody 31 minutes ago | parent

in my experience it’s pretty common to find big inverted indexes for text directly in the database - not necessarily large docs but certainly free text records in volume. using bm25 and unicode’s breakiterator is a very good way to build it. like putting lucene in the database basically - makes a lot of sense when the database is already large. places that bend over backwards to move search out of the db are usually trying to avoid having a very large db (and often end up with one anyway, getting the worst of both worlds)

zombodb 23 minutes ago | parent

They also end up with all the infrastructure and processes necessary to keep the external search system in sync, resync/reindex, pkey shipping back to their source of truth in queries, application-side joins and enrichment between both sources. It’s brutal.

Having everything in one place eliminates entire classes of development and especially operational problems.

andrenotgiant 34 minutes ago | parent

I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world.

It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search

Timescale has pg_textsearch https://github.com/timescale/pg_textsearch

Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se...

Now PlanetScale.

AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification

CodesInChaos 25 minutes ago | parent

ParadeDB's implementation builds on the Tantivy crate, which predates AI coding.

samwillis 16 minutes ago | parent

There is a lot of truth to this, but it's also very much down to domain experts being able to do this to move faster.

Planetscale (assuming they used a agentic development practice) will have pulled this off, to the level of performance that they have, because they have a team of very highly experienced Postgres developers. Their knowlage of Postgres internals will have given them the insights needed to steer the models to a plan that used the architecture as described in the post. That's not something a model can do on its own*

World experts + LLMs = moving mountains.

(* we're obviously seeing something a little different from inside the research teams in the labs. They are showing that the models, when you burn the level of tokens only they can, are able to do novel things from the models own insights.)

alexnewman 14 minutes ago | parent

What’s funny is I worked with a company with planet in the name Who could really use a full text search that was great in the Postgres