YHK - yet another hacker news clone

The downtime also starts happening in 2020, which is a while after the acquisition. 2020 was the year many people started learning to code while being stuck at home.

Might be worth considering a /GB/MO fee. Let's face it, VCS is a (glorified) storage service.

Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it.

Instead you have to realize that you are entering into a high growth phase and design around that.

It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly.

At some point you have to admit that Github ready for the moment.

OK so then use the additional revenue to add more resources and employees? What the f*k are they doing over there?

There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc.

When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience.

There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills.

Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today.

But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.

The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is.

My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.

If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.

Something more serious is at play. And I want to hear from the actual underlings.