YHK - yet another hacker news clone

Cool thought! We have an internal version of map reduce implemented in waymark that worked pretty well (though uses Redis for its intermediate values so it's technically not an end-to-end integrated solution).

In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.

Where did you hit the most snags with caching?

MIT licensed, just clarified in the repo.

As for async - background jobs are better modeled as async because of their natural parallelism across nodes imo. Logic in actions themselves are free to be as synchronous as you want. Why the preference?

I don’t understand these kinds of strong emotions async python triggers in some people. It reminds me of the irrational hate towards systemd in the Linux world.

This is a platform for running workflows distributed on multiple machines, sometimes long-running and with lots of IO. Every single contender in the space advertises the same code, from temporal to hatchet to dagster. Because it makes sense to write it this way.

Yep, that’s exactly it. imo the main value add for agents:

- Each LLM call and tool call becomes a checkpoint. Without durable execution you're pretty much stuck doing db saves/restores if you detect that this agent might have a previous context window. Agent frameworks are fundamentally just harnesses around the underlying model: you `while loop` until some server gives you some payload back, you add that to your context window list, and you keep going until it terminates. Durable execution moves that loop into the workflow body itself (a quick example here https://www.waymark.sh/python/tutorials/building-a-from-scra...).

- Long waits. Sometimes you do want your agent to wait for an arbitrary amount of time (ie. wait for this compilation job to finish - look again in 5mins, or maybe 5 hours). Durable execution makes sure that sleep timer will wake up regardless of what happens in the system state.

- Easier monitorability. Instead of having to sift through interleaved logs to figure out what was going on, you can see every large function call with a clean input/output contract. Useful especially in tracing down raised Exceptions and the data flow that it took to get there. Good for agents and regular code alike.

Of course, if your system is largely idempotent or your agents are mostly classifiers that are working in short internals - a simple db save/restore can work perfectly well too.

Durable execution and orchestration are essential in some specialised uses, such as regulatory reporting for financial institutions, or other snapshot based data pipelines. The transition is happening, and temporal and conductor are the usual tools there, but good tools are not always behind good solutions to put it lightly.

I like this one, as in the context of workloads like these, going with anything more complex than a Postgres is overkill.

We'll look into it! The db layer is already modeled basically like a plugin; should be pretty easy to add in whatever database support you want. Sign up to the newsletter on the website if you want the updates when it's ready :)

> all the intro and docs assume you understand the problem and couch the discussion in somewhat domain specific language. It might be interesting to me. It might even be relevant to me and I don't know it yet. The problem is I need to work really hard to get to the point of understanding that.

> This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.

I agree with this. For Show HN posts, this would really help broaden the audience as well as lift up the audience’s knowledge and enthusiasm.