104 points gateauxd 1 hour ago 73 comments

defenestration 1 hour ago | parent

GitHub actions staying 'Queued', started 16 minutes ago for me

nix0n 45 minutes ago | parent

I'm seeing at least one job showing up that way on the Github website that has already completed (on a self hosted runner, which is how I can tell).

EduardoBautista 1 hour ago | parent

I am going to try out Cursor Origin, GitHub is just too much of a bottleneck these days. The only thing that is not readily available is integration with claude code, but with the new claude code mods feature I think it's something I could vibe code quickly.

rosslh 46 minutes ago | parent

Isn't it still just essentially a UI wrapper around GitHub?

outworlder 1 hour ago | parent

At my company, we migrated basically all CI/CD to Github Actions. Also our Gitops processes. I have warned that putting all the eggs in one basket and they key to the fridge in the same basket was a bad idea. People didn't care. So I don't care anymore.

Github seems to have degraded measurably after they got acquired. As is tradition.

FLeXMurphy 59 minutes ago | parent

>Github seems to have degraded measurably after they got acquired. As is tradition.

This old chestnut.

jbs789 59 minutes ago | parent

Sounds like you now have your audience for the previously proposed solution.

outworlder 52 minutes ago | parent

After we spent a couple of years ripping and replacing the previous solution? You may have a point.

I guess I can embrace the corporate way, repackage the old solution with a few improvements as something brand new, and pitch to execs.

dewey 39 minutes ago | parent

The attention for ideas like that usually fades as quickly as it came after the downtime is over.

coldpie 55 minutes ago | parent

> People didn't care. So I don't care anymore.

Yeah. Enjoy the rest of your day off. Winter's coming, maybe go hit a park for a few hours or play a new video game or something. Or take some time to learn a new skill or something. Not your fault if the company can't provide the tools to do the job, go find something better to do.

hack1312 45 minutes ago | parent

Great ideas if working remote, can’t imagine busting out my Steam library in the office though.

coldpie 41 minutes ago | parent

Time to bring back the Boss Button https://en.wikipedia.org/wiki/Boss_button :)

esseph 40 minutes ago | parent

Coward! /s

worble 30 minutes ago | parent

Then its time to break out the good old MUDs

coldpie 16 minutes ago | parent

Dungeon Crawl Stone Soup's text UI could be mistaken for Perl at a glance https://crawl.develz.org/

JsonDemWitOster 32 minutes ago | parent

Isn't this basically https://xkcd.com/303/ but "Github is down"?

(Man, unbelievable the mileage we're getting out of this comic.)

I mean this really seriously. Maybe not a game on your computer, that's unprofessional (unless your TL/manager does it first). But

> Not your fault if the company can't provide the tools to do the job, go find something better to do.

is just Truth. What should I do in this case to best deliver value to the company, fly to SF and hand Github a ringbound volume of my changeset? Patch our corporate network so we can interface to Github via RFC 1149?

More "professional" ideas:

- read a book; a novel or TAOCP for nerdmaxxing

- play a game on your phone

- learn to draw. Draw hands!

999900000999 8 minutes ago | parent

That’s a good way to get blamed and let go.

But sometimes it’s a catch 22. I’ve had numerous times where systems simply didn’t work.

I brought it to my managers attention and they didn’t care.

Then later I leave in disgust or get fired. All my fault. I should have used my magical powers to convince my manager he’s wrong while not making him feel stupid.

Ultimately if you’re on the titanic it’s easier to grab a life boat vs trying to avoid the ice berg.

gateauxd 55 minutes ago | parent

Curious what even is the alternative. The stickiness is quite high and risk is scary to swap providers.

embedding-shape 52 minutes ago | parent

Self-hosted $whatever, managed your way. CI/CD doesn't have to be dog-slow or ever go down when you need it, only when you do your upgrades yourself, as god intended it.

I personally use Forgejo + their built-in Actions stuff (mostly GitHub compatible), have one old Linux desktop as a server which runs Linux + Windows runners, one Mac Mini for macOS/iOS builds, using NixOS so I don't have to remember how things work, and it just keeps running 24/7, always there when I need it.

tracker1 43 minutes ago | parent

There's something to be said for the social availability that github brings. I'm far more likely to submit a pr for a bugfix to a project on gh than a random self-host site.

You can do your own ci/cd runner system, but you still rely on webhooks or triggers, when you aren't hosting the code.

embedding-shape 34 minutes ago | parent

> There's something to be said for the social availability that github brings. I'm far more likely to submit a pr for a bugfix to a project on gh than a random self-host site.

Yeah, that used to be the single favorable argument for GitHub, nowadays I think it might be the opposite, at least personally. If the contributor can't be arsed to join the community proper (and maybe get a sign-in code to their email, or whatever login you use), I think I prefer life without their drive-by PRs.

x86a 43 minutes ago | parent

Claude/codex has made the stickiness much less of a problem. We just say "no new repos in GitHub" and then put agents on refactoring the existing repos for GitLab compatibility.

msdz 43 minutes ago | parent

The alternative is, I’m guessing, probably any other hosted git + CI/CD service that doesn’t offer a free tier to zillions of users (human and LLM alike) so that their computing capacity isn’t in a constant arms race with growth.

outworlder 41 minutes ago | parent

See, my main issue is with Actions. If that breaks, there's little alternative for the more complex flows. Minor infrastructure changes could be done manually through AWS, but complex deployments would have to be replaced. That's a big task even with AI.

The pull request system and related machinery is useful, but Gitlab and the ilk can do the same. Even self-hosted solutions like Gitea. But I don't particularly have a problem with that living in Github. If there's an outage, break glass, force push to main.

The actual Git repository hosting part is trivial. If that breaks, Git is distributed anyways, push and pull to any random remote storage. All you need is SSH. The main headache is that automations would break as they are usually hard-coded to a given origin. That is solvable in a pinch, and we have setup mirrors in the past.

It really depends on your use-case and how much you are tied to them (Github authentication?)

EngineeringStuf 35 minutes ago | parent

swarmfile was built to deal with this type of problem

sssilver 29 minutes ago | parent

What makes it sticky? I just don't see it. The declarative Github Actions manifests?

I imagine very few people use the wiki, because it's so subpar.

The issues are a problem, but this is why I always thought issues should be a core part of the Git repo, checked into it, sort of how Fossil does it. And so should the wiki, for that matter.

ionetan 24 minutes ago | parent

Agreed. Epiq also does that (a repo native issue tracker). It is persisted as an event graph in your repo in a dedicated branch. I'm it's author for disclosure. Vendor agnostic and works offline

Bnjoroge 19 minutes ago | parent

Forgejo, and Gitea are decent alternatives if you want to fully move off Github! If you are hoping to move away from only Github Actions, I've been working on a ~drop-in replacement for their control-plane + runner for the past few months. Each each job runs in a cross-platform vm so you can run your ci locally and update your checks. It's still beta, so I certainly wouldnt use it in prod just yet, but it's great if you want to run some of your CI locally. it allows me to extend it where possible to introduce some new features like JIT pipelines, improve performance or security. https://github.com/preloopdev/preloop if anyone's curious!

nrmitchi 50 minutes ago | parent

> Github seems to have degraded measurably after they got acquired. As is tradition.

I'm not disagreeing with you that it has degraded, however it was acquired almost 8 years ago at this point, and the software world has been turned on it's head in the last... 3?

Software generally degrades post-acquisition, I'm not denying that. Github has definitely degraded. I'm not sure I would agree that the core reason in this case is "the acquisition" and not "AI making software and free-CI-load demand 3-5 orders-of-magnitude higher than it was 3 years ago"

shimman 39 minutes ago | parent

These are incredibly complex and massive systems, they do not fail instantly. The failures happen incrementally until they become insurmountable and the system collapse. Incompetence doesn't show its head immediately, it can take quite a while.

Either way, it's clear the acquisition has not helped github in any meaningful way. GitHub is owned by a multi trillion dollar transnational corporation. They have enough money, resources, and human labor to achieve whatever they want, but this is what they want and this is what they've achieved.

JohnMakin 36 minutes ago | parent

The latest problems correlate pretty closely when they tried to migrate the entire platform onto azure in an insane timeline, something like ~1 year (for a project of that size, that's borderline insane).

boxed 50 minutes ago | parent

> Github seems to have degraded measurably after they got acquired. As is tradition.

Their explosive growth caused by LLMs seems like it must be a much better indicator than the acquisition. I bet Microsoft are losing a shit ton of money on GitHub right now and seriously questioning the decision to buy them.

maccard 7 minutes ago | parent

No, their decision to focus on a move to azure is the indicator, as the reliability issues started shortly after that, but before their boom in agentic commits

sandeepkd 46 minutes ago | parent

I feel sorry for the teams at this point. These kind of incidents at this scare and frequency are bound to create an acrimonious environment where everyone would be looking for scapegoats on the pretext of finding root causes.

rfgplk 44 minutes ago | parent

They haven't degraded, their service is being hit by an enormous amount of traffic. PRs/commits are what? Doubling every quarter?

neya 4 minutes ago | parent

It's Microsoft's touch. Anything they touch gets fucked up. It's because of their shitty infrastructure aka Azure. There was an employee here who actually gave insight into how bad the situation was a while back. Just reading that should make you re-evaluate anything owned by Microsoft.

serial_dev 1 hour ago | parent

I find it baffling that I knew about the incident 20 minutes before the official status update…

jacquesm 1 hour ago | parent

20 minute rule: if you fix it before 20 minutes pass it never happened.

amw-zero 1 hour ago | parent

Why is that baffling? You usually have to diagnose what's going on before summarizing it into a status update.

Or do you mean you want to see: "Something is wrong, we're looking into it."

serial_dev 55 minutes ago | parent

Yes. An acknowledgement of the issue should be fine, even if they don’t know why it happened. If a random dev with one PR can see that no jobs are picked up, GitHub should see it as well and add an entry to their status “hey, actions aren’t getting picked up, we are looking into it”. I wasted 20 minutes looking into out actions config looking got an issue, thought maybe we messed something up, but in turned out it was GitHub.

pasc1878 21 minutes ago | parent

Yes then you know the problem is not in yopur set up and so do not have to waste time looking into that possibility.

esafak 59 minutes ago | parent

To all those in the middle of conjuring a Github-killer into existence, more power to you, and please ship it already; I can't take this any longer.

ainar-g 54 minutes ago | parent

Why not just use GitLab?

esafak 39 minutes ago | parent

Community sentiment on GitLab has been poor. I want something newer and better like Cursor's.

molodec 39 minutes ago | parent

GitLab is often recommended when GitHub issues occur, but GitLab Cloud is not any better in terms of reliability and has plenty of outages https://status.gitlab.com/pages/history/5b36dc6502d06804c083...

GitHub also has more traffic, compare to GitLab

x86a 54 minutes ago | parent

Self-hosted gitlab works great for us

beardsciences 54 minutes ago | parent

self hosted forgejo is easy/great

gvfsa 52 minutes ago | parent

We are not short of GitHub alternatives.

willio58 59 minutes ago | parent

I wonder if this will ever end (meaning they've solved scaling for AI), and if not, is that due to the problem being unsolvable at this scale or bad decisions on GitHub's part? For example, they always bring up AI as a cause, but it's also apparent the move to Azure didn't help.

icameron 58 minutes ago | parent

I’m going to hope they can figure things out and improve their uptime. It’s so engrained in my teams workflow at this point, tearing it out and switching to anything else would be extremely disruptive.

I did work for a group using Atlassian bitbucket once, and that had its own problems at the time. I’m still pretty sure even a degraded GitHub beats it. That is to say I don’t know what alternatives for GitHub even are

gateauxd 54 minutes ago | parent

That's the thing what is the alternative feels like we are just at mercy to their outages.

hack1312 43 minutes ago | parent

You could self-host GHE. Or Gitlab. Or Forejo.

macrofriendly 33 minutes ago | parent

swarmfile supports git, self-hosted runners, is self-hosted swarmfile.com/public/swarmfile/swarmfile-server

cwyers 13 minutes ago | parent

Did a swarmfile write this message

macrofriendly 7 minutes ago | parent

A real human did :)

OlivOnTech 40 minutes ago | parent

We're using gitlab.com with self hosted runners (that can access lan).

Outages only come from our infra.

Migrating means migrating your ci workflows, but a decent llm can do that easily.

somehowadev 58 minutes ago | parent

It's almost a running joke at my company that around 15:00 UTC we're going to get the weekly Github actions outage.

I'm assuming it's something to do with the USA users getting their first work in?

FLeXMurphy 56 minutes ago | parent

It would not surprise me if they are going to shift to Plan B sometime soon, because this isn't working. They can't keep provisioning hardware resources in perpetuity to keep up with volume that the AI is pummeling them at. I think most other repo providers have either experienced downtime and/or have locked themselves up to prevent the onslaught.

It is impressive they've been able to keep going, but this is a regular denial-of-service attack at this point, nonstop and relentless.

dannyw 50 minutes ago | parent

They have plenty of levers to pull.

For example, GitHub can reduce/eliminate free GitHub Action minutes.

Or just cap concurrent jobs for free users from 20 to 2 or 1… lots of levers, their inaction is sad to see.

tonyhart7 50 minutes ago | parent

maybe just charge money if you use more compute ????

rvz 51 minutes ago | parent

Last time this happened was a week ago. [0]

There’s no CEO of GitHub to contact and Microsoft does not care as Copilot, and Tay continue to wreck the platform as the incidents continue.

No better time to self-host, since as this point your self hosted setup would be far most stable than GitHub as predicted 6 years ago. [1]

[0] https://news.ycombinator.com/item?id=50057012

[1] https://news.ycombinator.com/item?id=22868406

kingcauchy 46 minutes ago | parent

I know there's a lot of GitHub annoyed people out there but after investigating a lot of the instances of outages it does seems like A) They're abusing ElasticSearch in a way is was never meant to be used B) They've seen an unprecedented increase in scale and now have to provide agent slop a lot of free action minutes until they figure out how to separate commercial support from free support C) Who knows how many other systems were never meant to see this scale there

Still grateful for a lot of the free services and the semblance of community that GitHub still offers. It does seem like it's still the yellowpages of tech.

haute_cuisine 46 minutes ago | parent

Their infra is under constant DDoS + they provide a lot of free stuff for open source projects. There's also a constant churn of dependencies, patches, abandoned libraries, etc. We should give them some appreciation, dealing with this stuff is hard.

privacyishuman 45 minutes ago | parent

Agreed. I don't think people realize how much more people are using their free services. It's staggering.

Chyzwar 42 minutes ago | parent

Then just reduce free tier if they cannot handle. I paying for a service but I get best effort freeware quality.

boredatoms 43 minutes ago | parent

They need to cut back on the free tiers if it’s causing outages for paying customers

ianm218 7 minutes ago | parent

I think you are giving them way too much leeway. For the service they provide there’s no technical reason it can’t be infinitely scalable. For example their “pull request” page constantly goes down. Serving that kind of workload at high burst load is a fully solved problem.

If they are genuinely compute constrained on CI they should obviously reduce the free offerings rather than take down all their paying customers constantly.

JohnMakin 45 minutes ago | parent

They should just call it Monday now

cwyers 41 minutes ago | parent

There is a screamingly obvious answer here and I don't understand why they aren't taking it -- raise prices, make the free tier less generous. Either slow down the growth or up the revenue pipe so you can throw more compute at it.

Normal_gaussian 20 minutes ago | parent

As it stands, customers aren't leaving because of the degraded performance. Free is generating paying customers.

Raising prices risks churn, reducing free tier risks lowering customer acquisition rate.

torben-friis 6 minutes ago | parent

What if the load isn't fully the cause, and they end up with a significantly smaller userbase still getting downtime?

renehsz 27 minutes ago | parent

https://GiveUpGithub.org - why and how to ditch GitHub.