50 points bootlegbilly 2 hours ago 11 comments

77rushi77 2 hours ago | parent

The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?

reilly3000 43 minutes ago | parent

The whole wide events thing is Honeycomb’s marketing pitch. It’s essentially offering developer velocity, ie you don’t have to think about what should be a metric in advance. They have an open source project called Refinery which is a smart sampling proxy that samples successful requests but always forwards errors, slow queries, etc.

You can do anything you like with any approach but if they are mostly about ad hoc queries, dumping everything into BigQuery as a wide event can be the most economical. However if you want to run alerting, SLOs, etc with frequent queries over aggregated metrics, the best bet is using OTel Collector and metrics store of your choice like Prometheus or Mimir.

Most companies could save a bundle on observability if they were to move to a hybrid that includes both of those approaches instead of cloud provider logging, log-based metrics, and alerting.

x3n0ph3n3 2 hours ago | parent

Why is 77rushi77's comment flagged + dead? Seemed like a good question.

Edit: Looks like it's been restored.

gnabgib 2 hours ago | parent

Looks like it's because of:

> Don't post generated text or AI-edited text. HN is for conversation between humans.

But who can tell, you could ask the mods (email found in the guidelines)

kqr 1 hour ago | parent

Doesn't seem particularly robot-generated to me, nor to the classifier I made for a related purpose: https://xkqr.org/aicomment/#lang=none&prior=50&c=Pc_BbcMwEAT...

fakwandi_priv 40 minutes ago | parent

Then that check accounts history and tell me if you see a pattern or not, this one is really obvious.

delamon 2 hours ago | parent

4ndrewl 42 minutes ago | parent

Potential PII nightmare. You're now potentially circumventing access controls from sensitive data.

jakewins 19 minutes ago | parent

To me this seems easier than the alternative model (logs, metrics, traces), in that you only have one abstraction to contemplate handling PII in?

In what way is this worse than the alternative?

drdrey 7 minutes ago | parent

wide events feel like structured logs, great for high cardinality data but not so much for time series