r/github • u/Suspicious_Orchid770 • 18d ago
Discussion What’s gone wrong at GitHub?
https://leaddev.com/software-quality/whats-gone-wrong-at-githubReliability issues are driving users away in their droves.
92
u/Dense_Gate_5193 17d ago edited 15d ago
GitHub used to be a gold standard of work excellence. Their uptime was incredible and their team worked entirely async. No office hours, flat hierarchy, people picked what they wanted to work on, and had performance curmudgeons squeezing out every millisecond of performance out of the platform.
Then, Microsoft purchased them. it’s been particularly bad since AI and the enshittification of everything began
edit:
This is not just pre/post AI.
i looked it up and mentioned in another comment
Metric (24 Months Pre- vs. Post-Acquisition)
Reported Incidents
Pre-Acquisition (2016–2018)
89
Post-Acquisition (2018–2020)
126
+41%
Incident Minutes
Pre-Acquisition (2016–2018)
6,110 minutes
Post-Acquisition (2018–2020)
12,074 minutes
+97%
microsoft nearly doubled incidents reported and downtime across the board.
18
u/Afflictionista 17d ago
When the GitHub CEO left I moved to GitLab because of what they were saying who is going to "lead" Github, the AI department. GitLab maybe doesn't have that big visibility but it is working.
14
u/spigolt 17d ago
It would be nice if GitLab was a good alternative, but I had a completely like abysmally awful experience with them when I tried to actually pay for it. Having to deal with all kinds of separate companies that work somehow separately to Gitlab managing the subscription. Total complete mess.
8
u/EphemmyEuphie 17d ago
Selfhost Forgejo
-4
u/Round_Ad_5832 17d ago
no, self host gitea
1
u/philipmather 16d ago
Between personal project to side-hustle, maybe even to startup level potentially but latter... nah.
1
1
u/Itz_Santhosh 16d ago
Wow I hate only one thing in gitlab that they ask credit card to verify for even small actions(ci/cd) but as a student how I even suppose to have credit card??
1
u/silentkode26 17d ago
Why not selfhosted? We use it and it’s perfect.
1
u/RubbelDieKatz94 17d ago
Yup, same here. Today I have an appointment with the DevOps guy who'll explain how to set up a new project in GitLab and hook it up to Argo. Quite interesting.
2
u/Leapin_Langosta_ 13d ago
Just a question, where is the gitops repo for argo stored?
1
u/RubbelDieKatz94 13d ago
Each project gets its own "infrastructure" repo with its gitops settings
1
u/Leapin_Langosta_ 13d ago
In another comment you said gitlab was deployed using argo so the only thing that is unclear to me is where that gitops repo to deploy gitlab is located; in said deployed gitlab instance (which doesnt make much sense?) or do you have mirrors or idk what else?
2
u/RubbelDieKatz94 13d ago
Ah, seems like I was mistaken then. GitLab itself runs outside the main cluster, but its runners run inside the cluster. Argo's own config seems to live outside the cluster too.
0
10
u/Business-Row-478 17d ago
Gitlab doesn’t have the issues because it doesn’t have the users
1
u/ferocity_mule366 15d ago
gitlab tries so hard to be firefox, but it just sucks and cost more instead of a better alternative it promises
3
u/robotsmakinglove 16d ago
I’ve needed to use GitLab for some consulting and am constantly underwhelmed by the product. Every feature feels a bit more poorly designed than GitHub. This is shocking to me because it is basically a ripoff.
1
1
1
-1
u/Tomik080 16d ago
There was more PRs opened and commits pushed in the last month than in 2024 and 2025. Somehow Microsoft is to blame their infra is taking a toll. People are so ignorant.
4
u/Dense_Gate_5193 16d ago
i can’t be the only one who noticed a quality difference between pre microsoft and post microsoft acquisition.
1
u/TechIncarnate4 15d ago
Are you sure this isn't pre-AI vs post-AI?
2
u/Dense_Gate_5193 15d ago
This is not just pre/post AI.
Guarantee you that when microsoft purchased GitHub that there was a mandate to use windows servers for hosting.
Metric (24 Months Pre- vs. Post-Acquisition)
Reported IncidentsPre-Acquisition (2016–2018)
89
Post-Acquisition (2018–2020)
126
+41%Incident Minutes
Pre-Acquisition (2016–2018)
6,110 minutes
Post-Acquisition (2018–2020)
12,074 minutes
+97%microsoft nearly doubled incidents reported and downtime across the board.
1
u/TechIncarnate4 15d ago
Sure... "reported" is the key word there. There is a good post further below showing that the "before MS" stats may not be accurate, including a time where there was a 24 hour outage, but their stats showed 100% uptime for the month.
1
u/ferocity_mule366 15d ago
okay? but github also increase in scale exponetially and you havent taken account for that?
1
u/inebriatus 11d ago
Can’t believe you’re getting downvoted for this. I have no love for M$ but acting like this isn’t mostly an AI traffic increase issue is committing critical thinking malpractice.
21
u/Key-Temporary-8759 17d ago
Microsoft acquiring GitHub is possibly one of the cause for poor reliability, because of the loss of independence. And when GitHub copilot was launched, their focus shifted to AI features, instead of performance and infrastructure.
And also, increase in number of PRs can cause performance issues, as most of the PRs today are written by agents
5
u/mavenHawk 17d ago
Microsoft owned Github since 2018...
6
u/EarthGoddessDude 17d ago
3
u/tigerhuxley 17d ago
Hahaha- i haven’t seen this visual graphic before.
2
u/Business-Row-478 17d ago
Graphic is misleading for multiple reasons.
The biggest dips on that graphic are still >99.5% uptime which isn’t terrible.
Before Microsoft purchase, it is listed at 100% uptime, which is not realistic or really possible. It’s very likely that Microsoft just has better observability and publication of incidents.
Total traffic volume has also exploded since Microsoft purchased it
5
u/NiceVu 16d ago
FYI 99.5% uptime is absolutely terrible, that is a two days a year, or 4 hours a month or 7 minutes a day downtime. For any kind of production standards that’s really bad.
And if you look at breakdown the uptime went as low as 98%
Yes the traffic skyrocketed but that uptime is really bad for a tool like Github. The goal for that kind of service should be 99.99% uptime
0
u/Business-Row-478 16d ago
99.5% isn’t ideal but that’s the lowest it dropped in a month, it’s not like that is their overall uptime.
Obviously the goal is higher uptime, but for products that operate at the scale of GitHub, that’s pretty much impossible.
Even AWS, which certainly isn’t a gold standard of uptime, has had multiple single 9 months just this year.
1
1
1
u/coderstephen 16d ago
99.5% is terrible uptime where I come from, but I work in enterprise software. Our target is 99.999% uptime ("five nines").
Then again, we are paying GitHub quite a lot, and GitHub portrays their higher tiers of business services as an enterprise service...
1
u/Business-Row-478 16d ago
99.5 is the lowest it dipped in a month, not their total uptime. Obviously it isn’t great, but any service operating at that level of scale isn’t going to be able to maintain 5 nines.
2
u/coderstephen 16d ago
Obviously it isn’t great, but any service operating at that level of scale isn’t going to be able to maintain 5 nines.
I would disagree with that.
Amazon S3 operates at a higher scale than GitHub. Probably by orders of magnitude. Yet it has a track record of even better than 5 nines.
1
u/Business-Row-478 16d ago
S3 had one of the worst outages in the history of AWS. And besides that, bucket storage is a lot simpler than the entirety of the gh platform, so availability numbers aren’t really comparable.
The vast majority of GitHub outages have affected runners
1
u/coderstephen 16d ago
Yep, S3 did have that really big outage that impacted a lot of things. And yet, its uptime is still better than 99.5%.
I agree that S3 and GitHub are different in complexity but I don't think the comparison is unfair.
1
u/foramperandi 16d ago
Amazon S3 is famous for being one of the most available SaaS services of all time. They've done tons of formal methods verification and the entire internet depends on it. Amazon S3 and GitHub are not even vaguely comparable. You would not want to pay for what GitHub would cost, if it had 5 nines of uptime.
1
u/coderstephen 16d ago
I mean, the graphs show that GitHub used to have much better uptime (not 5 nines, but better than 1-2 nines) so its definitely possible. I am not expecting GitHub to have similar reliability to S3, I'm just contesting the opinion that, "Oh it's just too hard to have good uptime when you're at scale."
→ More replies (0)3
u/foramperandi 16d ago edited 16d ago
That graph is lying with numbers. What that graph is actually showing is that GH had shitty operational maturity before the MS acquisition and rarely declared incidents. If you believe that uptime from 2016 to 2018 was nearly 100%, I've got some land to sell you.
GitHub's worst single outage of all time was in Oct 2018 when they were down for over 24 hours: https://github.blog/news-insights/company-news/oct21-post-incident-analysis/. That graph says they had 99.99369% availablity that month.
Edit: my mistake, I misread. The graph actually says that in a month that GitHub had 24 hours of downtime, it's availability was 100%. Amazing!
4
u/Special_Rice9539 15d ago
I competed in a hackathon and had a senior Microsoft engineer on my team and he was blown away when I added automated tests to the project…
15
9
6
u/autisticpig 17d ago
Anyone who took on GitHub was not prepared for what agentic coding was going to do. The sheer volume happening 24/7 would send any service to its knees.
We brought all our runners in house and built cache layers for our images and libs/pkgs built with runs that sync every could of days to the cache. We still rely on GitHub to orchestrate and hold the repos but if that's down it's not the end of the world... Can still deploy and test just fine internally.
These tools are ruining so much, it's going to take time to adjust resources to absorb the abuse.
People need to calm down. Esp people expecting 99% uptime for something they're not paying for.
3
2
1
0
u/coderstephen 16d ago
They used to have 99+% uptime. The issue people noticed is that that changed.
8
u/ZoolanderBOT 17d ago
Without really knowing anything of the situation, I blame Microsoft. Seems like a safe bet?
3
u/SentientSeamoth 16d ago
It's okay to blame Microsoft for just about everything. In this case they also own GitHub, so yes you can blame them twice as hard lol
2
u/ajax81 14d ago
I think it’s funny that they bought GitHub to gain access to the worlds codebase to train their AI and co-pilot STILL sucks balls for coding.
2
u/lambdalab 14d ago
I actually quite like it, though it’s expensive for what it is. So if I want something done in bulk, I’ll use Claude, and I do that for the majority of work where I can trust Claude to do.
But when I need or want to write something myself, at that point copilot can be helpful. I’ve limited it to only producing one line at a time, and this saves me a ton of time. I’d start a line, and a lot of the time it will be correct about the rest of it. Some of the time when it isn’t, I just type it in myself, and the next one is more likely to be correct. So I write code much more quickly, but also retain total understanding and control of what I wrote.
To be fair if I wasn’t lazy I could probably find a local model that can do that for me, but alas.
1
2
1
4
11
u/spigolt 18d ago
I'm trying to just create a repo and it's failing the last 10 minutes, which led me here to see whether it's just me... my understanding is that it's fallen off a cliff since the Microsoft purchase... like it was fine for a while following the purchase, but when the long-time CEO stepped down and github got reorganized under Microsoft's "Core AI" product division, and thus it's goal shifted from serving developers to acting as a data and distribution funnel for Microsoft’s AI products.... after that it has fallen off a cliff and just keeps getting worse.
8
u/Prior-Data6910 17d ago
"Since the Microsoft purchase" is, in my view, a lazy reason (no offence intended).
Do you know what the uptime was of Github Actions before the acquisition? 0% because it didn't launch until after. So basically, you're comparing the uptime of a storage system and basic issue tracker against a system that can run your entire software development life cycle from planning through to production. That's before we look at the difference in scale (more users, more repositories, more daily work).
All ignoring the fact that Github did used to have outages, and when it did the whole system went offline, not just parts of it.
1
u/spigolt 17d ago
Like I wrote - it wasn't immediately after the Microsoft purchase, it was when it got reorganized under Microsoft's "Core AI" product division and the original founder left that things started really going downhill.
2
u/Prior-Data6910 17d ago
I'm not sure who you're talking about, but none of the founders remained at Github post acquisition. Microsoft installed a new CEO (Nat Friedman) from day one.
The timing about the reorg also coincides with a massive increase in traffic, which is more likely to be the cause.
Don't get me wrong, Microsoft have destroyed many acquisitions over the years (RIP Skype, gone but not forgotten) but I don't think Github is one of them.
2
u/foramperandi 16d ago
The original founders had not been involved for years when MS bought them. They had been looking for a new CEO for years at that point, in 2018. Tom had been pushed out over misconduct allegations and Chris had lost interest and had been AWOL for years. The company was being run by the Chief Legal Officer to the extent anyone was in charge. The company was directionless and the product was stagnating.
You have no idea what you're talking about.
4
u/gkreitz 18d ago
Impeccable timing to post an article about Github being unreliable a few minutes before they completely killed off push:ing anything. https://www.githubstatus.com/incidents/smtdswd1h2w4
SSH keys get rejected, so it seems impossible to push anything. Their description is "Pull Requests is experiencing degraded availability." :)
8
2
u/bartread 17d ago
It's not even the downtime, it's reliability on reporting and front-end as well. You'll be waiting for a PR to build or whatever and your actions will look like they're running but, actually, they finished 5 minutes ago. Sometimes it updates the UI automatically, presumably via WebSockets... and sometimes it just doesn't until you refresh the page.
I get bitten by that regularly and it's incredibly frustrating. I'm seriously thinking of implementing a ViolentMonkey script that simply refreshes any GitHub page in the focussed tab every 10 - 15 seconds because otherwise, if you've got a bunch of PRs going through, it's easy to miss that the front-end is stale on one of them.
2
2
u/_verel_ 15d ago
They desperately wanted slop. They got slop. GitHub broken.
Does anyone care? No because it's a Microsoft product which means every company on earth needs to have it for some fucking reason.
You could literally be like "Bro setup forgejo with CI and migrate my GitHub repo" but people are so fucking lazy that paying for GitHub is what they'll do
1
u/Great-Use-3149 17d ago
Microsoft has gone wrong. It always has .
But hey, I'm sure their AI-native direction made them 100x more productive /s.
1
1
1
1
u/GeorgeRNG 17d ago
The bots pushing these posts are so annoying. What does droves even mean as a metric.
1
u/imagebiot 17d ago
You mixed your business with your engineering and then shifted capital from Eng to business departments
1
1
u/ConspicuouslyBland 16d ago
Why do you ask? Don't you know it's acquired by Microsoft? Or did you really think that was not the answer to your question?
1
1
u/Patient-Pollution46 16d ago
You can give it a try with tools like Bitrise. It started as CI/CD, and now we cover the whole path from code to release. So, agents can work in our Remote Dev Environments (macOS and Linux machines with simulators), and developers review what they built. CI and Build Cache build and test it, and Release Management and CodePush ship it to users. And if you want to stay with GitHub/GitHun Actions, you can just swap runners for the faster runners on the market with Bitrise Build Hub
You can use one piece in your existing stack or run all of them together.
1
u/SpudgunDaveHedgehog 16d ago
Wait until you learn the difference between GitHub cloud and GitHub enterprise. 😆😉
1
u/Eastern_Picture_3879 16d ago
AI commit volume is overloading the service. Simple as. I don't envy their network team.
1
u/SafatK 13d ago
What is wrong is that everyone dislikes AI and says it doesn’t work but somehow is pushing millions of lines of code a month to their repos!
People who could never code beyond that calculator tutorial they copied from some YouTube video are now pushing endless code to their random endless projects.
Cut GitHub some slack!
1
1
1
113
u/sreekanth850 17d ago
nobody saying the amount of slops that people pushing every hour. you cannot infinitely scale systems.