The writing is in the wall. First it was UX annoyances. Then it was GitHub Actions woes. Now it is paying money for running their software on your own hardware. It's only going to go downhill. Is it a good time now to learn from our mistakes and convince our teams and management to use community-maintained, libre alternatives? They may be inferior. They may lack features. But they're not going to pull user hostile tricks like this on you and me. And hey, if they are lacking features, maybe we should convince our management to let us contribute time to the community to add those features? It's a much better investment than sinking money into a software that will only grow more and more user hostile, isn't it?
Every company I've been at that tried to self-host something like GitLab, later moved to GitHub. Nobody in business cares if it's open source/free software. They care about managed hosting, centralized services, invoicing, etc. DIY is great for hobbyists and the cash-strapped.
But also, many hours spent building jobs and the like that are off-the-shelf on Github Actions.
This is the main issue though, it feels like doing anything but what the largest companies offer just costs more time and money. It reminds me of the decades long push to move away from Microsoft, only for e.g. the office 365 offering to show up and make everyone's (software, account management) work easier and cheaper, forever.
OSS can build truly incredible libraries and frameworks. User facing products? ehhhhh not so much.
GitHub has gotten worse over the years but it's not like there's some gold standard open source alternative. And remember, early GH was filled with a pretty amazing group of developers and open source advocates.
If the counter is that, instead of buying github, we could have invested in building some other tool, well, that's not how this works. People need to build what the company is actually building for its users.
This is far from truth imo. It is very possible to only use (F)OSS. Github, AWS, Azure, Vercel are not at all more pleasant or easier to work with than on-prem Gitlab/gitea/codeberg/jenkins/k8s/kibana/prometheus/grafana.
I could spend an hour and have a full setup done on physical or VPS to have 1) remote git hosting 2) pipelines running on changes 3) pipelines publishing images or some artifacts 4) automated deployment for these images/artifacts
I'm struggling to see what am I missing. What is worthwhile that Github offers? It is popular and easy to set up org+repos, but that seems it. A few years ago I was working on Azure Devops with the azure pipeline and that was the worst developer experience I've ever had. AFAIK Github actions uses the same syntax and works the same way, at least it was at that time.
This sounds like a week or two of work to me (I'm a novice though). You should write a guide.
Bespoke CI is easy to build but no one wants to be in charge of rolling out a critical security patch to that on-prem box no one's touched since that consultant from 2 years ago.
Huh, I should make an Apache plugin that launches docker exported containers uploaded into a directory.
And the companies with specific needs (funnily enough, at the other end of the cash spectrum) or have a lower internal cost than the products out there.
There's more of them out there than we give credit for I think. In particular, running one's own stack on the corner seldom makes the news.
Interesting, my experience has been the opposite. 99% of companies I've seen self host their VCS, it has been Gitlab (with some rare sel-hosted GitHub Enterprise everyone seems to hate, and the very rare Bitbucket).
To be fair most of them started with it when Gitlab was really really ahead, features wise. The gap has somewhat closed, but Gitlab is still a superior product IMO. Just the fact that you can have an actual organisational structure, and move it around, and share variables/configs between groupings, beats anything GitHub have to offer which is slightly nicer than GitLab.
And had to live with their constant outages :)
everything else is trash.
Github Actions changed the landscape.
They're composable.
The only two other things that come close is Concourse.CI and CircleCi.... and circle-ci is 100% trash
If it works for you, great. But it is far from being good.
I've worked with Github Actions, Gitlab-CI and CircleCI in the last 10 years, and they've all been such an improvement over Jenkins, or god forbid, CVS with manual deployments, that I'm generally just counting my blessings.
For me the pain only came when not adhering to KISS. All the mentioned VCS are pretty much feature complete and only really differ on meta-topics (cost, license, lock-in) or niche topics (Actions marketplace, matrix builds, SSH on Runners). I've not yet run into an issue that would have actually blocked me, because there's always sh to fall back to in case of a bug or missing feature.
You can run all your CI locally if you don't embed your logic into the workflows, just use CI for orchestation. Use an env manager(Mise, Nix etc) to install tooling(you'll get consistency across your team & with CI) and call out to a task runner(scripts, Make, Task etc).
- everything else is trash.
- Github Actions changed the landscape.
- They're composable.
And I still hate github actions! Aside from anything else, they have one major flaw, which is there is no good development/test loop for writing them.
If you write most of your CICD in some kind of script, then you can run it locally, and do some basic checks around environment etc before deploying.
If you write most of your CICD in github actions or any alternative, you will be doomed to push 100 commits with messages like "maybe be?", "hmmm. . ." before eventually squashing them all down when it turns out several hours later that you mispelt an environment variable.
Something like linked and dependent PRs in a chain would go someway to replicating Gerrit but again this basic functionality is not available out of the box for whatever reason.
What you can say for it, is that it was free and near infinitely hackable.
If someone is forcing you towards high stakes tight-coupling with no thought whatsoever towards the lock-in, you should get it in writing that "we at ${org} are fully committed to ${vendor} with ${platform}, on ${cloud} using ${tech} come what may, now and forever" and lots of sign off so that everyone knows who to blame when this is inevitably wrong.
I grant you pipelines are the best bit about ado, but the fact that you can’t test them is a pain.
And the webhooks and templating are pretty messy and unpleasant quickly.
We’re changing from ADO to GitHub (had to be an MS product for corporate) and the infra people are looking forward to GHA as they prefer their maintenance to ADO pipelines.
Actually there were alternatives that were far superior (seriously, no way to group projects?) but also more than 2x as expensive. If GH "fixes the glitch" then it will be plan B time.
For example, Azure has had a script runner service for ages that you can hook up to your ”own” vm, by installing an agent. But then you pay double, the fee (per second) for the service as long as the script is running, and the fee (per second) for the vm in azure as long as it exists. So, as with GitHub actions, it’s cheaper to run it on their provided crap instances.
To get rid of the double costs I guess you could install your own CI server and agents, that polls the GitHub repo, but then you don’t get the integration in their web gui. That was what you did before gh actions came around, a local Jenkins for example.
It also doesn't randomly fail and if it would, you can probably fix it yourself.
I don't think actions on a git repository host is a good way to fix poor deployment strategies if it goes beyond pushing a package to npm and co. Just to poke at the wound again.
But Gitea has interfaces here as well, didn't try them though.
I'm all for everyone going full Libra - we do it at my co-op - but it makes sense to me that venture funded companies would "play the game" and light investor money on fire because, first, who gives a shit, and second, the investors want you to do that anyway so they can find out as fast as possible if you're a unicorn.
At my co-op, I spend hours writing future proof code and integrating FOSS solutions that I hope will serve us forever. When I'm at a startup, I'm looking for the fastest, maybe cheapest solution. YC gave us 200k in AWS credit? Guess we're on AWS. Another company in the cohort is some LLM IDE ala cursor and gave us a year free? Sure, burn tokens their investors are paying for, more agents for me. Vercel offers us a year of free hosting? Great, I hate nextjs but Claude loves it so fuck it, we deploy a nextjs app on vercel and lock ourselves deep into that ecosystem. Our product may not look like this at all in a year so I may be rewriting it in Vue or whatever when the vercel bills start coming in. Doesn't matter.
Edit: this got flagged, but if you end up in an unprofitable position because you chose Oracle as a vendor and they squeezed your company so hard they need to choose between paying you (either via a raise or via actually employing you) or staying with a vendor thats squeezing them; they will choose the latter, as its short term cheaper.
(facepalm)
1) company uses exclusively free software, spends more time dealing with the shortcomings of said software than developing product, product is half baked and doesn't sell well, company dies.
2) company uses proprietary but cheap/free (as in beer) software that does the job really well, focuses on developing product, product is good and sells well, company how has a ton of money they could use to replicate the proprietary product from scratch if they wanted to.
A purist approach like in scenario 1 leaves everyone poor. A pragmatic approach like scenario 2 ends up earning enough money that can be used to recreate the proprietary software from scratch (and open-source it if you wanted to).
In this case the problem isn't even the proprietariness of the software, it's the fact that companies are reliant on someone else hosting the software (GH being FOSS wouldn't actually change anything here - whoever is hosting it can still enforce whatever terms they want).
FOSS alternatives already exist, it's just that our industry is so consumed by grifters that nobody knows how to do things anymore (because it's more profitable for every individual not to); running software on a server (what used to be table stakes for any shop and junior sysadmin) is nowadays lost knowledge. Microsoft and SaaS software providers are capitalizing on this.
In reality you have to also make concessions with proprietary software, so the moat is not as large as your comment makes it seem imo.
That depends, not always. Sometimes the employees of said company manages to contribute back upstream, on the dime of the company. If the "free software" they used and contributed to have a lot of users, it's certainly not "leaves everyone poor" but rather "helps everyone, beyond monetary gain".
Sure, you can make the argument that it isn't that great for the company, and you may be right. But the world is bigger than companies making money, killing a few companies along the way to make small iterative steps on making free software for absolutely everyone is probably a worthwhile sacrifice, if you zoom out a bit.
In the end the purists approach results in better more productive software across even slightly longer timescales. That ultimately produces more value and thus a richer society than the kind of short term pump and dump schemes which SV is so fond of. Who captures that value is a different story than was that value created.
And your last paragraph hits the nail on the head, people are afraid to run their own software.
Any self hosted service in an enterprise means that you're dealing with all the headaches that come with that including: backups, user/role creation and mapping maintenance, infrastructure scaling needs, OTEL or other monitoring, etc.
It's an easier decision for VPs to pay GitHub anything less than the man hours required to execute the above tasks because it's a "not our problem" fee.
Running builds on a designated server would also protect against malware on a developer’s machine silently embedding itself into the resulting artifact and then deployed to production.
The first checks I setup are build and test. The rest is “extra”.
- You host open-source software on your own hardware.
- You pay a company for setup and maintenance by the hour.
it changes and you move on.
> When we shipped Actions in 2018, we had no idea how popular it would become.
We were on codeberg for a couple years and it was fine.
4x 4c/16gb instances will perform much better than one 16 core 64GB instance.
Isn't it written in this super scaling language that everybody says scales super well?
What is the problem with it?
How is there not a collective decision to dissolve them?
Bitbucket also implemented 2FA, but it's 100% optional, so I'm sticking with that for the moment.
We currently self-host on kubernets/aws. The thing that really got to me isn't the new charge per se. It's the fact that GHA has a ton of problems. I can hold my nose and deal with them when it's free. But now that you're squeezing me, at least you could have created something like GHA 2.0 and added a charge for that. Instead, there are vague roadmap promises which don't even include things that I care about. Specifically:
- Jenkins had better kubernetes integration years ago. It's crazy that GHA can't beat that.
- "Reintroducing multi-label functionality" - yeah, so they first broke it. They did supply "reasons", which looked like they never talked to a customer. [1]
- Still no SDK of any kind.
- "Actions Data Stream" - or you can just fix your logging.
There are dozens more complains, which are easy enough to find. This kind of an approach just makes me want to make sure that I don't use GHA again. Even if I end up paying another vendor, at least I'll be treated as a customer.
[1] - https://github.com/orgs/community/discussions/160682#discuss...
"Thank you for your interest in this GitHub action, however, right now we are not taking contributions.
We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time. The GitHub public roadmap is the best place to follow along for any updates on features we’re working on and what stage they’re in."
We had it done with a week to spare.
Apples to oranges, naturally, but like this, our infra-jenkins master would pay for itself in hosting in a week of ansible integration testing compared to what GHA would cost. Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.
And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
Depending on your workplace, there's a whole extra layer of bureaucracy and compliance involved if you self-host things. I aggressively avoid managing any VMs for that reason alone.
Most modern development workflows should just pickup a host with some container engine and do their work in stateless containers with some external state mapped in, like package caches. It's much easier for both sides in a majority of cases.
This is kinda where I am. No one really feels like they are selling a premium "just works" product. Its all jank. So why it the jank I chose at the price I chose?
At the moment I'm self hosting gitlab runners. Its jank. But it's free.
Hilariously, the official Terraform provider for GitHub is full of N+1 API call patterns — aka exponential scaling hotspots — so even generating a plan requires making a separate (remote, rate-limited) API call to check things like the branch protection status of every “main” branch, every action and PR policy, etc. As of today it takes roughly 30 minutes to do a full plan, which has to run as part of CI to make sure the pushed TF code is valid.
With this change, we’ll be paying once to host our projects and again for the privilege of running our own code on our own machines when we push changes…and the bill will continue to grow exponentially b/c the speed of their API serves to set an artificial lower bound on the runtime of our basic tests.
(To be fair, “slow” and “Terraform” often show up and leave parties at suspiciously similar times, and GitHub is far from the only SaaS vendor whose revenue goes up when their systems get slower.)
Self-hosting Jenkins on an EC2 instance is probably going to result in a _better_ experience at this point. Github Cache is barely better than just downloading assets directly, and with Jenkins you can trivially use a local disk for caching.
Or if you're feeling fancy and want more isolation, host a local RustFS installation and use S3 caching in your favorite tools.
Hardware is getting cheaper and cheaper, but the fear-mongering around running a Linux machine has successfully prevented most businesses from reaping those cost reductions.
Unfortunately not anymore and not in the foreseen future if we don't see some AI investment corrections.
But having an _option_ to not download everything every time is great. You can add a periodic cache flushing, after all.
They’ve lowered their runner costs to compete, and introduce minimum charge to discourage abd make sure they still get paid.
They could have made their service better or more competitive like with price but instead they chose this route. SMH.
The runner configuration and registration process is unnecessarily byzantine. [1]
They can't cancel jobs cleanly. [2]
There are consistency problems everywhere. [3]
Their own documentations describes horrible things unless you use runners in JIT mode. Though JIT runners are not always removed after exit.
If there is a worse self-hosted CI runner, I haven't yet met it.
[1] https://docs.github.com/en/actions/how-tos/manage-runners/se...
The rest is correct. (Though you can hardlink the installation.) And you can disable self-update, though it does it by default.
Create/start/stop/terminate EC2.
Of course, if you can just fence in your competition and charge admission, it'd be silly to invest time in building a superior product.
> Actions is down again, call Brent so he can fix it again...
Scheduling jobs, actually getting them running, is virtually instant with GitLab but it's slow AF for GitHub for no discernable reason.
Not sure if a Phoenix Project reference, but if it is, it's certainly in keeping with Github being as fragile as the company in the book!
The only self-hosted runners I've used have been for internalized deployments separate from the build or (pre)test processes.
Aside: I've come to rely on Deno heavily for a lot of my scripting needs since it lets me reference repository modules directly and not require a build/install step head of time... just write TypeScript and run.
When you've got many 100s of services managing these in actions yaml itself is no bueno. As you mentioned having the option to actually be able to run the CI/CD yourself is a must. Having to wait 5 minutes plus many commits just to test an action drains you very fast.
Granted we did end up making the CI so fast (~ 1 minute with dependency cache, ~4 minutes without), that we saw devs running their setup less and less on their personal workstations for development. Except when github actions went down... ;) We used Jenkins self-hosted before and it was far more stable, but a pain to maintain and understand.
I’m definitely sure it’s saving me more than $140 a month to have CI/CD running and I’m also sure I’d never break even on the opportunity cost of having someone write or set one up internally if someone else’s works - and this is the key - just as well.
But investment in CI/CD is investing in future velocity. The hours invested are paid for by hours saved. So if the outcome is brittle and requires oversight that savings drops or disappears.
When CI and CD stop being flat and straightforward, they lose their power to make devs clean up their own messes. And that's one of the most important qualities of CI.
Most of your build should be under version control and I don't mean checked in yaml files to drive a CI tool.
I’m also someone who paid for JetBrains when everyone still thought it wasn’t worth money to pay for a code editor. Though I guess that’s again now. And everyone is using an MS product instead.
This is like if Dropbox started charging you money for the files you have stored on your backup hard drives.
Using GitHub actions to coordinate the Valhalla builds was a nice-to-have, but this is a deal-breaker for my pull request workflows.
A lot of it is wasted in build time though, due to a lack of appropriate caching facilities with GitHub actions.
[0] https://github.com/Barre/ZeroFS/tree/main/.github/workflows
tl;dr uses a local slot-based cache that is pre-warmed after every merge to main, taking Sidecar builds from ~10-15 minutes to <60 seconds.
SlateDB, the lower layer, already does DST as well as fault injection though.
``` on: push ```
is the event trigger to act on every push.
You'll waste a lot of CI on building everything in my opinion, I only really care about the pull request.
In my experience gitlab always felt clunky and overly complicated on the back end, but for my needs local forgejo is better than the cloud options.
- github copilot PR reviews are subpar compared to what I've seen from other services: at least for our PRs they tend to be mostly an (expensive) grammar/spell-check
- given that it's github native you'd wish for a good integration with the platform but then when your org is behind a (github) IP whitelist things seem to break often
- network firewall for the agent doesn't seem to work properly
raised tickets for all these but given how well it works when it does, I might as well just migrate to another service
(People seem to object to this comment. I genuinely do not understand why.)
Actions let you test things in multiple environments, to test them with credentials against resources devs don't have access to, to do additional things like deploys, managing version numbers, on and on
With CI, especially pull requests, you can leave longer running tests for github to take care of verifying. You can run periodic tests once a day like an hour long smoke test.
CI is guard rails against common failure modes which turn requiring everyone to follow an evolving script into something automatic nobody needs to think about much
... Is nobody in charge on the team?
Or is it not enough that devs adhere to a coding standard, work to APIs etc. but you expect them to follow a common process to get there (as opposed to what makes them individually most productive)?
> You can run periodic tests once a day like an hour long smoke test.
Which is great if you have multiple people expected to contribute on any given day. Quite a bit of development on GitHub, and in general, is not so... corporate.
I don't deny there are use cases for this sort of thing. But people on HN talking about "hosting things locally" seem to describe a culture utterly foreign to me. I don't, for example, use multiple computers throughout the house that I want to "sync". (I don't even use a smartphone.) I really feel strongly that most people in tech would be better served by questioning the existing complexity of their lives (and setups), than by questioning what they're missing out on.
>... Is nobody in charge on the team?
This isn't how things work. You save your "you MUST do these things" for special rare instructions. A complex series of checks for code format/lint/various tests... well that can all be automated away.
And you just don't get large groups of people all following the same series of steps several times a day, particularly when the steps change over time. It doesn't matter how "in charge" anybody is, neither the workplace nor an open source project are army boot camp. You won't get compliance and trying to enforce it will make everybody hate you and turn you into an asshole.
Automation makes our lives simpler and higher quality, particularly CI checks. They're such an easy win.
Perhaps that isn't most use of it; the big projects are really big.
Fundamentally, yes, what you run in a CI pipeline can run locally.
That's doesn't mean it should.
Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally.
There are a rather lot of people who do argue that? Like, I actually agree that non-local CI is useful, but this is a poor argument for it.
I'm not aware of people arguing for self-hosting team or enterprise services.
The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub actions runners for 4 years. 100s of thousands of runners have taken jobs, done the work, destroyed themselves, and been replaced with clean runners, all without a single issue with the runners themselves.
Workflows on the other hand, they have problems. The whole design is a bit silly
been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered
we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates
That’s not hard, the status page is updated manually, and they wait for support tickets to confirm an issue before they update the status page. (Users are a far better monitoring service than any automated product.)
Webhook deliveries do suffer sometimes, which sucks, but that’s not the fault of the Actions orchestration.
The one for azure devops is even worse though, pathetic.
Charging for self-hosted runners is an interesting choice. That's the same cost as their smallest hosted runners [1]
[1] - https://docs.github.com/en/billing/reference/actions-runner-...
(ofc, that'd only mean they stop updating the status page, so eh)
After like day 2 my workflows would take 10-15 minutes past their trigger time to show up and be queued. And switching to the self hosted runners didn't change that. Happens every time with every workflow, whether the workflow takes 10 seconds or 10 minutes.
Edit: Confused GitLab and Bitbucket
ZIRP ended, its remaining monopoly money has been burnt through, and the projected economy is looking bleak. We're now in the phase where everything that can be monetized is being monetized in every way that can be managed.
Free tiers evaporate. Fees appear everywhere. Ads appear everywhere, even where it was implied they wouldn't. The lemons must be squeezed.
And because everybody of relevance is in that mode, there's little competitive pressure to provide a specific rationale for a specific scheme. For the next few years, that's all the justification that there needs to be.
I thought that "Bitbucket" was in your original post and you added only your edit message to say that it was, in fact, Gitlab and not Bitbucket that added cost for self-hosted runners.
I don't know if it's worth the amount they are targeting, but it's definitely not zero either.
This is $2.88/day, $86.4/month, $1051.2/year. For them to do essentially nothing.
Most notably, this is the same price as their hosted "Linux 1-core" on a per-minute basis. Meaning they're charging you the same for running it yourself, as you'd pay for them to host it for you...
Orchestration, logging, caching, result storage.
It's not nothing. Whether it's worth it to you is a value judgement, and having run a bunch of different CI systems I'd say this is still at least competitive.
Additionally, I thought that caching came out of a separate limit, and was not billed like artifact storage?
Maybe this is designed to scare people away from self-hosting altogether?
It is a way of increasing lock-in for smaller-medium clients, without driving away their medium-large ones.
If its the price of runs, then its not always running.
If its price of the agent to exist, then thats not paying per runs- then you’re right that people tend to leave their runners online 24/7- but I’ve never worked anywhere that had workers building 24/7.
This is not uncommon in some orgs - less number of concurrent runners, slow builds, loads of jobs because of automation or how hooks for the runners are setup.
In the context of discussion that doesn't matter, OP's point distills to that they use minimum of 720 hours / month of orchestration time or some multiple of that on self hosted runners running 24x7.
Github will now charge $84 extra per month for single self-hosted runner running 24x7 - i.e. that is the cost for 43,200 build minutes for only their orchestration alone.
In a more typical setup that is equivalent to say 5 self-hosted running running ~4.5 hours a day(i.e 144/hours/runner/month)
Either way, we will likely pay 8-hours4-pipelines5-days=160 hours per week, just shy of 168-hours for true 24/7. This currently costs $0 just for context.
Of course entirely expected after MS buyout, if anything I'm surprised it took that long
Don't they generally only kick in when you push or merge?
But you are right, this is ridiculous!
The real mistake was GH not charging anything for self-hosted runners in the first place, setting an expectation.
Now it's none of those three. Once again, choosing not to pirate is just an objectively wrong choice. It's a worse experience, with worse quality, worse availability, and at a higher price tag.
Choosing not to pirate and not to consume simultaneously is not necessarily a wrong choice. A difficult one? Yes. But I propose that it could be beneficial for your mental (and maybe physical) health.
With DVD, Netflix if something I wanted to watch wasn't on any of my streaming services, it was almost guaranteed to be on DVD Netflix. That fallback doesn't exist anymore.
But when streaming started to really go down the toilet I already had a homelab so I spun up radarr and Jellyfin behind seven proxies for family-scale piracy. It's wonderful. This is a new golden age for piracy.
What an incredibly silly accusation to make of a company/service that streams movies and television. Like you understand it is possible to dilute the concept of civic responsibility right?
Companies don't care about society, unless it affects profit. Companies are not people, they are cold machines that through different means try to reach the same purpose, make more money.
No one should anthropomorphize companies. They might look like they have human qualities, same way like the T800 in the Terminator looked human.
The GitHub encrapification finally affects me. I am militantly unwilling to pay per minute to use my own computer. Time to leave. I can trigger and monitor builds myself thank you very much.
The only variable is how long after acquisition before they gut it. It's almost never right away. GitHub was acquired 7 years ago, but it started showing symptoms perhaps 2 years ago.
With this I think it's clear the wound was fatal. GitHub will stumble on for a few more years with ever-decreasing quality, before going the way of Skype.
So, I guess we're all migrating to gitlab? Or is it time to launch gittube? Githamster?
They seem to care much less about free users than in the past but businesses still flock to it. GitLab is the only other platform I’ve seen in the workplace of anywhere I worked, with the exception of a big tech company I worked at. They had both GitHub enterprise and an internally maintained platform which was being phased out. if I recall correctly it based on Phabricator
The considerations for commercial users leaving github are probably pretty different, so perhaps they'll stay.
There’s also a large bunch of projects that will never leave, some of which ms have influence over. I mean, some high profile open source projects hasn’t even left source forge yet, even though it has been malware infested shithole for decades.
But other alternatives exist as well, like Sourcehut: https://sr.ht/
Considering that the lifetime of our sun system is finite that statement is undeniably true.
Also we don't know how a non-Microsoft GitHub could have developed.
Anyone using GitLab or any other VCM that you'd recommend? I'm absolutely done with Github. Or is everything else just as bad?
Alternatively, Forgejo, Gitea, or (based on praise I've seen from other people) maybe sourcehut.org.
I find GitLab's interface intolerable. Heavy reliance on javascript even for read-only access, nonintuitive organization, common operations hidden behind menus, mystifying icons... Every time I seek out a project's home and discover a GitLab instance, I find myself pausing to reconsider whether contributing to the project will really be rewarding enough to outweigh the unpleasant experience I'm about to have.
What does VCM stand for?
Particularly making a contribution should anyhow be trivial - you push the branch and it shows a banner in the repo asking if you want to open a MR for the recently pushed branch.
I don't know why anyone would use GitHub actions. They seem like a weird, less powerful version of the GitLab CI. Now they want to charge for runtime on your own runner.
For a company, I'd recommend self-hosting forgejo (which also has actions), which powers codeberg.
(forgejo started as a fork of gitea)
And the best (maybe?) part in your case is that the CI is based on GH Actions, so you can probably reuse your workflows without the need to adapt them.
And the entire solution just sucks. It's bad, bad software. It barely works a lot of the time.
The only reason they're doing this is because they can. Which is usually a really stupid reason. And here, it is. So, that's outrageous.
Sure, but there's a separate mechanism that you need to make it all work: the orchestration. Without that, you have only the capacity to run jobs -- it's potential energy, if you will, not doing real work.
That orchestrator thus provides real value. And it's not like it's free for them to build, operate, and maintain.
It does? I feel like it implies that they want third-party runners like Blacksmith out of the ecosystem, which is why they're now financially penalizing customers who use them.
1. Services like blacksmith and WarpBuild (I'm the founder) are still cheaper than GitHub hosted runners, even after including the $0.002/min self-hosting tax.
2. The biggest lever for controlling costs now is reducing the number of minutes used in CI. Given how slow Github's runners are, or even the ones on AWS compared to our baremetal processor single core performance + nvme disks, it makes even more sense to use WarpBuild. This actually makes a better case for moving from slow AWS instances running with actions-runner-controller etc. to WarpBuild!
3. Messaging this to most users is harder since the first reaction is that Github options make more sense. After some rational thought, it is the opposite.
Overall - it is worse for Github users, but options like blacksmith and WarpBuild are still the better option.
what makes you think they won't hike the control plane price again? They can turn this knob arbitrarily to put you out of business.
Reg. hiking it again, they'd have to either be extremely anti-competitive and selectively apply the pricing OR apply the hike uniformly by about double the current value to match our pricing while making it completely unviable for any large co to use self-hosted github actions in the first place.
Right now at my company our biggest complaint are macOS Intel runners from GitHub which somehow take 15+ minutes to provision and are the slowest of the bunch.
Nowadays GH has more sizes by WB continues to beat them in price and performance.
It’s highway robbery what GH charges for the crap they provide. I can highly recommend WarpBuild for Mac (and Linux) runners.
macOS Runners Apple Silicon and Intel support
- Introducing a cheap 1-core runner
- Lowering the price of GitHub-hosted runners
- Making it slightly more expensive to use self-hosted runners
- There is actually a fourth one: the vnet integration, which also allows you to run public runners in your own infra
As a bonus, for some people it means something that was free is now not free. Those who are willing to pay rather than go, might prefer to use GitHub-hosted if they are going to pay anyway.
This is clearly an incentive to use github-hosted, and their sales reps are also going this way.
https://docs.github.com/en/enterprise-cloud@latest/admin/con...
1. https://medium.com/@the_atomic_architect/github-vs-gitlab-20...
[0]: https://docs.gitlab.com/user/project/repository/branches/pro...
* its 7 VPS because we separated the tests by modules and we have 7 major modules. * edit: formatting
The split between tag and branch pipelines seems like intentional obfuscation with no upsides (you can't build non-latest commit from a branch, and when you use a tag to select the commit, GitLab intentionally hides all branch-related info, and skips jobs that depend on branch names).
"CI components" are not really components, but copy-paste of YAML into global state. Merging of jobs merges objects but not arrays, making composition unreliable or impossible.
The `steps` are still unstable/experimental. Composing multiple steps either is a mess of appending lines of bash, or you have go all the way in the other direction and build layered Docker images.
I could go on all day. Programming in YAML is annoying, and GitLab is full of issues that make it even clunkier than it needs to be.
There's been years of discussion about ways to fix it with nothing moving forward.
https://gitlab.com/gitlab-org/gitlab/-/issues/263401
And the most recent tracking issue:
No matter what I did, every time I touched our CI pipeline code I could be sure to run into yet another Gitlab bug.
It’s great if you have relatively simple CI. If you have anything slightly more complicated (like multiple child pipelines for a monorepo) you’re going to have a rough time.
Every time I thought I understood GitLab CI, it would fail/behave in non-obvious ways.
Oh and turns out GitHub also has that now: https://github.blog/changelog/2025-09-18-actions-yaml-anchor...
UPDATE: okay they botched it https://frenck.dev/github-actions-yaml-anchors-aliases-merge...
I'm happy for competition, but this seems a bit foul since we users aren't getting anything tangible beyond the promise of improvements and investments that I don't need.
Given that GitHub runners are still slow as ever, it actually is a point in our favor even compared to self-hosting on aws etc. However, it makes the value harder to communicate <shrug>.
https://tangled.org/tangled.org/core/blob/master/docs/spindl...
(no affiliation)
---
Blog post about Tangled's CI: https://blog.tangled.org/ci
https://bsky.app/profile/tangled.org
There looks to be a blog post here: https://blog.tangled.org/ci
> Also Nix supports MacOS.
Doesn't matter. Tangled CI requires Docker.
I'm not a fan of nix and would have picked containers regardless for a git forge CI offering
It was govt thing and they are required to put a new bid every few years and their bid was EVIDENTLY "just list what our current hosting provider has, we can't be arsed to spend months migrating infrastructure every few years", but the clever weasels in the sales managed to get them.
fun video on it: https://www.youtube.com/watch?v=E3_95BZYIVs
Can't we switch to something more advanced in terms of protocols (like one that always maintain 3 copies, and where people can give ressources (cpu/bandwidth/memory) in the forms of tokens)?
Is it that egregious?. I read it as they are redistributing the costs. It is in combination dropping the managed runner costs by a good margin and charging for the orchestration infrastructure. The log storage and real time streaming infra isn't free for them (not $84/month/runner expensive perhaps but certainly not cheap )
We don't need to use the orchestration layer at all, even if we want to use rest of the platform, either for orchestration or runners. Github APIs have robust hooks(not charged extra) and third-party services(and self-hostable projects) already provide runners, they will all add the orchestration layer now after this news.
--
Competition is good, free[2] kills competition. Microsoft is the master of doing that with Internet Explorer or Teams today.
Nobody was looking at doing the orchestration layer because Github Actions was good enough at free[1], now the likes of BuildJet, Namespace Labs etc are going to be.
[1] Scheduler issues in Github Actions not withstanding, it was hard to compete against a free product that costs money to build and run.
[2] i.e. bundled into package pricing,
It used to suprise me that people saw cool tech from Microsoft (like VSCode) and complain about it.
I now see the first innings of a very silly game Microsoft are going to start playing over the next few years. Sure, they are going to make lots of money, but a whole generation of developers are learning to avoid them.
Thanks for trying to warn us old heads!
Microsoft of Nadella is different. It looks more like a boring Silicon Valley monopoly. They had good products years ago and it got people hooked and now its a game of endless rugpulls. Microsoft of now doesn't care about the featureset. They just jump from one hype train to another. People keep paying them for the stuff they did in early 2000s. Nobody cares about newer stuff including Microsoft themselves.
* Codeberg https://codeberg.org/
* Sourcehut https://sr.ht/ [1]
Don't get me wrong — I am glad that they are doing what they're doing, but it's a long way until it becomes a real alternative.
As mentioned above — I am glad that they exist and do what they do. However, that doesn't mean I should ignore issues like this.
- charge the same you would pay for the GitHub runners
- you have to factor YOUR server cost also, so self hosted will cost more than the platform option
- you jump to the platform runners and save on servers, sysadmin, DevOps, etc.
And then they grab you by the balls and raise the prices.
Now the playing field is more level, yay. Fun for those who choose to migrate away.
Are Control Plane costs already separately charged?
So the question becomes: is $0.002/minute a good price for this. I have never run GitHub Actions, so I am going to assume that experience on other, similar, systems applies.
So if your job takes an hour to build and run though all tests (a bit on the long side, but I have some tests that run for days), then you are going to pay GitHub $.12 for that run. You are probably going to pay significantly more for the compute for running that (especially if you are running on multiple testers simultaneously). So this does not seem to be too bad.
This is probably going to push a lot of people to invest more in parallelizing their workloads, and/or putting them on faster machines in order to reduce the number of minutes they are billed for.
I should note that if you are doing something similar in AWS using SMS (Systems Management Service), that I found that if you are running small jobs on lots of system that the AWS charges can add up very quickly. I had to abandon a monitoring system idea I had for our fleet (~800 systems) because the per-hit cost of just a monitoring ping was $1.84 (I needed a small mount of data from an on-worker process). Running that every 10 minutes was going to be more than $250/day. Writing/running my own monitoring system was much cheaper.
I am not open to GitHub extracting usage-based rent for me using my own hardware.
This is the first time in my 15+ years of using GitHub that I'm seriously evaluating alternative products to move my company to.
So, like I said, the question for you is whether that $140/month of service is worth that money to you, or can you find a better priced alternative, or build something that costs less yourself.
My guess is that once you think about this some more you will decide it is worth it, and probably spend some time trying to drive down your minutes/month a bit. But at $140 a month, how much time is that worth investing?
I'd happily pay a fixed monthly fee for this service, as I already do for GitHub.
The problem here is that this is like a grocery store charging me money for every bag I bring to bag my own groceries.
> But at $140 a month, how much time is that worth investing?
It's not $140/month. It's $140/month today, when my company is still relatively small and it's just me. This cost will scale as my company scales, in a way that is completely bonkers.
Maybe they can market it as the Github Actions corkage fee
If it is so easy why don’t you write your own orchestrator to run jobs on the hardware you own?
This is an odd take because you're completely discounting the value of the orchestration. In your grocery store analogy, who's the orchestrator? It isn't you.
For scheduled work, cron + a log sink is fine, and for pull request CI there's plenty of alternatives that don't charge by the minute to use your own hardware. The irony here, unfortunately, is that the latter requires I move entirely off of GitHub now.
> My guess is that once you think about this some more you will decide it is worth it, and probably spend some time trying to drive down your minutes/month a bit. But at $140 a month, how much time is that worth investing?
It's $140 right now. And if they want to squeeze you for cents worth of CPU time (because for artifact storage you're already paying separately), they *will* squeeze harder.
And more importantly *RIGHT NOW* it costs more per minute than running decent sized runner!
That value to you is apparently less than $140/mo. Find the number you’re comfortable with and then move away from GH Actions if it’s less than $140.
More than 10 years of running my own CI infra with Jenkins on top. In 2023 I gave up Jenkins and paid for BuildKite. It’s still my hardware. BuildKite just provides the “services” I described earlier. Yet I paid them a lot of money to provide their services for me on my own hardware. GH actions, even while free, was never an option for me. I don’t like how it feels.
This is probably bad for GitHub but framing it as “charging me for my hardware” misses the point entirely.
It used to suprise me that people saw cool tech from Microsoft (like VSCode) and complain about it.
I now see the first innings of a very silly game Microsoft are going to start playing over the next few years. Sure, they are going to make lots of money, but a whole generation of developers are learning to avoid them.
Thanks for trying to warn us old heads!
It makes sense to do usage-based pricing with a generously-sized free tier, which seems to be what they're doing? Offering the entire service for free at any scale would imply that you're "paying" for/subsidizing this orchestration elsewhere in your transactions with GitHub. This is more-transparent pricing.
Although, this puts downward pressure on orgs' willingness to pay such a large price for GH enterprise licenses, as this service was hitherto "implicitly" baked into that fee. I don't think the license fees are going to go down any time soon, though :P
Now the GitHub pricing change definitely? costs more than both servers combined a month ... (They cost about 60$ together )
3 step GitHub action builds around 1200 nix packages and derivations , but produces only around 50 lines of logs total if successful and maybe 200 lines of log once when a failure occurs And I'm supposed to pay 4$ a day for that ? Wonder what kind of actual costs are involved on their side of waiting for a runner to complete and storing 50 lines of log
Despite what people say about "maintaining" Jenkins (whatever that means to them personally) - you can set it up in an IaaC way including the jobs. You can migrate/create jobs en masse via its API (I did this about 10 years ago for a large US company converting from what was then called TFS)
Nice profit margin…
Unless you're on the free org plan, they're hardly doing it "for free" today…
Right, instead, they now charge the full cost of orchestration plus runner for just the orchestration part, making the basic runner free.
(Considering that compute for "self-hosted" runners is often also rented from some party that isn't Microsoft, this is arguably leveraging the market power in CI orchestration that is itself derived from their market power in code hosting to create/extend market power in compute for runners, which sounds like a potential violation of both the Sherman Act and the Clayton Act.)
I get being charged per-run, to recoup the infra cost, but what about my total runtime on my machine impacts what GH needs to spend to trigger my build?
This argument is disingenuous. Companies pay GitHub per seat for access to PR functionality etc. What's next, charging per repository? Because of a decision to no longer provide the repositories "for free"? It's not for free, you're paying already, it's included in the per-seat pricing. If you charge per seat then sometimes there are users who hardly use it and sometimes there are users who use it a lot. The per-seat pricing model is supposed to make the service profitable overall regardless of the usage levels of individual users.
Absolutely not, since it's the same price as their cheapest hosted option. If all they're doing is orchestration, why the hell are they charging per-minute instead of per-action or some other measure that recognizes the difference in their cost between self-hosted and github-hosted?
It was free, so anything other than free isn't really a good price. It's hard to estimate the cost on github's side when the hardware is mine and therefore accept this easily.
(Github is already polling my agent to know it's status so whether is "idle" or "running action" shouldn't really change a lot on their side.)
...And we already pay montly subscription for team members and copilot.
I have a self-hosted runner because I must have many tools installed for my builds and find it kinda counter productive to always reinstall those tools for each build as this takes a long time. (Yeah, I know "reproducible builds" aso, but I only have 24h in most of my days)
Even for a few hundreds minutes a month, we're still under a few $ so not worth spending two days to improve anything... yet.
It is not only not good. It is outrageous. The amount of compute required for orchestration is small (async operations) and also they already charge your for artifacts storage. You need to understand that the orchestration just receives details (inbound) from the runner. It needs very little resources.
Or shortly summarized: lock in through pricing.
Pretty sure this will explode straight in their faces though. And pretty damn hard.
So they make CI a bit cheaper but a future migration to Forgejo harder.
In fact they could easily pull off some typical sleazy Microsoft bullshit and eventually make it a shit ton harder to migrate out of GitHub once you migrated back in.
I don’t know if that’s actually why they’re doing this, but it sounds plausible.
Now, if you are already looking at migrating, its also potentially a kick in the butt to do it now. But if you aren’t, the path of least resistance—or at least, the path of least present recurring cost—is a path to a greater degree of lock-in.
Microsoft’s sales reps know this.
And trust me, they are running a lot of public and private repositories.
And there are many more orgs and govs throughout Europe doing similar things because there's a (growing) zeitgeist here that the Trump administration nor any American SaaS company can be trusted. This started, by the way, after Microsoft suspended the ICJ from using Microsoft 365 on orders from the White House.
I have seen this sentiment more and more, which is welcome to me as it’s a drum I have been banging for 15 years.
I have never had so many empathetic conversations than I have recently.
Everybody now is like "Hey, we can take something like Kubernetes which is open source and is backed by a worldwide community, and you know like OpenStack which is open source and is backed by a worldwide community and we can build our own computing platform and deploy services and online communities and stuff on top of that"
And I was like "Wait, you guys are realizing that NOW?!? I've been an activist and part of a movement urging you all to try and be less dependent on US Big Tech and focus more on decentralization for YEARS"
Like you I am really happy things seem to get rolling now, though :)
> Or shortly summarized: lock in through pricing.
how would increasing price make you locked in more ?
> If you don't really care about the metadata all it pretty much takes is moving git repositories with their history.
moving PR/CI/CD/Ticket flow is very significant effort, as in most companies that stuff is referenced everywhere. Having your commits refer ticket ID from system that no longer exists is royal PITA
just rewrite the short links in your front-end to point to the migrated issues/PRs. write a redirect rule for each migrated issue/PR, easy
hard-coded links in commit messages are annoying, you can redirect in the front-end too but locally you'd have to smudge/clean them on local checkout/commit
Where do you live that that seems like a bad idea?
What color are you?
I'm sure I can find a company that supports ethnostates if you need that for your next project.
Overall costs go up for everyone but we remain the better option.
If you don't want to pay, you'd have to not use GitHub Actions at all, maybe by using their API to test new commits and PRs and mark them as failed or passed.
Nothing kills morale faster than wrenching on the unreliable piece of infrastructure everyone hates. Every time I see an alert in slack github is having issues with actions (again) all I think is, "I'm glad that isn't me" and go about my day
Contrast with a declarative system like github actions: "I would like an immutable environment like this, and then perform X actions and send the logs/report back to the centralized single pane of glass in github". Google's "cloud run" product is pretty good in this regard as well. Sure, developers can add foot guns to your GHA/Cloud Run workflow, but since it is inherently git-tracked, you can simply revert those atomically.
I used Jenkins for 5-7 years across several jobs and I don't miss it at all.
People would be better served by not expecting anything different from Microsoft. As you say yourself, this is how they roll.
> The only learning to take away is never ever use anything from the big tech companies
Do you even believe in this yourself? Not being dependent on them would be a good start.
I mean maybe https://github.com/rust-lang/bors is enough to fully replace Github Actions? (not sure)
Listen to webhooks for new commits + PRs, and then use the commit status API to push statuses: https://docs.github.com/en/rest/commits/statuses?apiVersion=...
Rather than having to write some ad hoc code to do this
Post statuses, and add rulesets to require those statuses before a PR can be merged. The step after that is to lock out pushing to the branch entirely and perform the integration externally but that has its own challenges.
They don't care about people actually self-hosting. They care about people "self hosting" with these guys:
Anyway, GitHub actions is a dumpster fire even without this change.
I don’t make policy at GitHub and I don’t work at GitHub so go ask GitHub why they charge for infrastructure costs like any other cloud service. It has to do with the queueing and assignment of jobs which is not free. Why do they charge per minute? I have no idea, maybe it was easiest to do that given the billing infrastructure they already have. Maybe they tried a million different ways and this was the most reasonable. Maybe it’s Microsoft and they’re giving us all the middle finger, who knows.
I added some context that contradicts your assumption that the increased fees were to cover hosting/storage/scheduling costs.
But you (yes, you personally) have to collect the results and publish them to a webpage for me. For free.
Would you make this deal?
Except the alternative is I do this for free but also I'm doing all the testing and providing the hardware.
I'm only going to charge you if you do most of the work yourself
Maybe it's bad business dealing with lots of non-standardized external hosts, and it drags you down.
Maybe people are abusing the free orchestration to do non-CI stuff and they're compromising legitimate users.
Look, I understand it's frustrating to some consumers. However, it's not irrational from GitHub's point of view.
Would you keep charging the same rate per head?
it's 2025, for log files and a spicy cron daemon (you pay for the artifact storage), it's practically free to do so. this isn't like the days of Western Union where paying $0.35 to send some data across the world is a good deal
All the people complaining can just tap into this almost-free and acessible cheap resource you are referring to instead.
but realistically, publishing a web page is practically free. you could be sending 100x as much data and I would still be laughing all the way to the bank
If you think that's easy, do it for me. I have some projects to migrate, give me the link of your service.
I think it's cheap to maintain. let me know how many devs you have, how many runs you do, and how many tests (by suite) you have, and I can do you up a quote for hosting some Allure reports. can spread the up-front costs over the 3-year monthly commitment if it helps
My point was that they profit from accessing your code, which is why they made it free in the first place. Now they make you pay because they believe they will make more profit. But they certainly weren't losing money before.
> If you respect the license and make the AI comply to valid license reuse
I think that the de facto situation is that AI does not have to know about licences or copyright at all. If they hack your computer to train their AI, the illegal part is that they hacked your computer, not that they trained their AI with the stolen data.
That is simply not true.
Companies can get into legal trouble if they don't.
Copilot does that bookeeping:
https://docs.github.com/en/copilot/how-tos/get-code-suggesti...
Heard of Meta torrenting copyrighted material? What kind of trouble did they get into?
i think it should be illegal or otherwise extremely damaging to do this kind of thing
Why would a public repo use a self-hosted runner ? because the self-hosted runner storage available is only 14GB !!
8h job is definitely more expensive to them than a 1 minute one, but I'd guess that the actual reason is that this way they earn more money, and dissuade users from using a third party service instead of their own runners.
the only rational outside rationale is this has the best financial projections, equitability with the customer be damned
gotta make up for those slumping ai sales somehow, amiright?
This isn't aimed at people actually self-hosting; it's aimed at alternative hosted runners providers. See this list
The costs for GitHub doing action workflows (excluding running) is less related to job duration.
The most charitable interpretation is that per-minute pricing is easier to understand, especially if you already pay runners per minute.
The less charitable interpretation is that they charge that because they can, as they have the mindshare and network effect to keep you from changing.
Charging a per-workflow-minute platform fee makes a lot of sense and the price is negligible. They're ingesting logs from all the runners, making them available to us, etc. Helps incentivize faster workflows, too, so pretty customer-aligned. We use self-hosted runners (actually WarpBuild) so we don't benefit from the reduced default price of the Github-hosted runners, but that's a nice improvement as well for most customers. And Actions are still free for public repos.
Now if only they'd let us say "this action is required to pass _if it runs_, otherwise it's not required" as part of branch protection rules. Then we'd really be in heaven!
It's there a particular reason you're extending the benefit of the doubt here? This seems like the classic playbook of making something free, waiting for people to depend on it, then charging for it, all in order to maximize revenue. Where does the idea that they're really doing this in order to deliver a more valuable service come from?
Charging "more than nothing" is certainly not what I would call maximizing revenue, and even it they were maximizing revenue I would still make the same decision to purchase or abandon based on its value to me. Have you interacted with the economy before?
and you expect it to stay this way?
> I would still make the same decision to purchase or abandon based on its value to me.
(plz correct me if i'm wrong)
Now Microsoft will charge "data plane usage" (CRUDing a row that contains (id, ts, state_enum, acc_id ...) in essence) 2.5 more than what Ubicloud offers for WHOLE compute. Also to have "fair pricing" they'll make you pay 2.5 more the compute's price for being able to use their data plane.
cool.
To spell it out: jobs can hang forever because of some ridiculously bad code on their end, they have a 6 hour cap, so that's 6 hours of billable $$$ per-instance of the bug (assuming it wasn't manually canceled). I know I've seen jobs hang forever regularly over the course of my years using GitHub for work.
Note: pretty sure this has been resolved.
As far as I can tell from that article, these changes will not affect me; it says "Standard GitHub-hosted or self-hosted runner usage on public repositories will remain free" and another section says "This will not impact Actions usage in public repositories". Hopefully, this information would behelpful for other people who use GitHub Actions. However, I don't know if I missed something else that is important, from the article.
For private repositories, each GitHub account gets 2000 free minutes of runtime per month. Both self-hosted runners and GitHub-hosted runners count against that quota.
I'm sure we'll feel it too at https://sprinters.sh, but probably a bit less than others as our flat $0.01 per job fee for runners on your own AWS account will still work out to about 80% average savings in practice, compared to ~90% now when using spot instances.
1. Self-hosting runners is still cheaper than not Despite the $0.002/minute self-hosted runner tax, self-hosting runners on your cloud (aws/gcp/azure/...) remains the cheaper option.
2. Prefer larger runners If your workflow scales with the number of vCPUs, prefer larger runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, using actions-runner-controller with heavy jobs running on 1 vcpu runners is not a good idea. Instead, prefer a 2vcpu runner (say) if it runs the job ~2x faster.
3. Prefer faster runners All else being equal, prefer faster runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, if you're self-hosting on aws and using a t3g.medium runner, it's better to use a t4g.medium runner since the newer generation is faster, but not much more expensive.
4. Prefer fewer shards If you have a lot of shards for your jobs (example: tests on ~50 shards), consider reducing the number of shards and parallelizing the tests on fewer but larger runners.
5. Improve job performance This is not new advice, but it's now more important than ever because of the additional GitHub self-hosted runner tax.
6. Use GitHub hosted runners for very short jobs For linters and other very short jobs, it's better to use GitHub hosted runners.
Note: I make WarpBuild, where we provide github actions runner compute. Our compute is still cheaper than using github hosted runners (even with the $0.002/min tax) and our runners are optimized for high performance to minimize the number of mins consumed. I'm generally biased, but I think the points 1-6 apply irrespective of WarpBuild.
This feels like one of the big issues that OSS projects might face when migrating to an alternative.
What might a less GitHub centric CI ecosystem look like for OSS community?
Without GitHub's free CI for public repos, the small projects and indies will get hit the hardest imo.
However, I do not know hard numbers to quantify the impact.
This is going to be the downfall of GA
From this perspective this is a huge price jump, but self-hosting to save money can still work out.
Honestly, GitHub Actions have been too flaky for me and I'm begrudgingly reaching for Jenkins again for new projects.
[1] https://instances.vantage.sh/aws/ec2/m7i.large?currency=USD&...
I do not know what route are these companies taking. Microsoft has been crazy for past 2-3 years, but it is sad to see BitBucket and other alternatives also taking similar route :/
(I work exclusively on public repo open source at the moment, and get Github actions for free).
I was just using act (https://github.com/nektos/act) on my local server to build the X64 packages for my project, since I want to streamline it with ARM64 support, I migrated to the github self hosted runners.
This is really ridiculous, is M$ really lack that money just to schedule the Jobs running not in there infra?
Today it's possible to spin up a company that sells GitHub Actions runners with a lower price and higher performance than GitHub's own hosted runners. These new fees will make that a lot less economically viable.
1. Services like WarpBuild (I'm the founder) are still cheaper than GitHub hosted runners, even after including the $0.002/min self-hosting tax.
2. The biggest lever for controlling costs now is reducing the number of minutes used in CI. Given how slow Github's runners are, or even the ones on AWS compared to our baremetal processor single core performance + nvme disks, it makes even more sense to use WarpBuild. This actually makes a better case for moving from slow AWS instances running with actions-runner-controller etc. to WarpBuild!
3. Messaging this to most users is harder since the first reaction is that Github options make more sense. After some rational thought, it is the opposite.
We use feature branch deployments, so we trigger a lot of builds.
It’s been awhile since I looked. What’s a good alternative?
Jenkins has been rock solid, we are trying to migrate to Argo Workflows/Events, but there are a complaints (like deploying argo workflows with helm, such fun!)
- runs locally
- has a language server: python, typescript, go, java, OR elixer
- has static typing
- the new caching mechanisms introduced in 0.19.4 are chef's kiss
I do not work for dagger and pay for it using the company credit card. A breath of fresh air after the unceasing misery and pain that is Gitlab and GHA.
-- Winston Churchill (probably)
I get that self-hosted runners generate huge egress traffic, but this is still wild. Hope it pushes more companies to look into self-hosted Gitea / Forgejo / etc.
Running workers ourselves was the last resort, we tried everything else but it was impossible to get fast (and consistent) build times otherwise.
In a way we are now going to get charged for Github's poor execution on Actions.
[0] https://docs.gitea.com/usage/actions/overview
[1] https://gitea.com/gitea/act / https://gitea.com/gitea/act_runner
I have cron jobs on several github projects that runs once a day and I have never been charged anything for it (other than my github membership). Should I expect to be charged for this?
Focus on the enterprise. Something like a 3000$ minimum yearly price. Direct customer support with real engineers no questions asked.
Need someone to setup your CICD, that's another fee, but on staff engineers will get it done.
Edit: I'd even imagine a company like this can bootstrap, I'd need help though. Would probably take 4 skilled SWEs about 6 months for an MVP.
What I'd really like to see is some new CI/CD systems though. Actions is garbage in multiple dimensions. Can't somebody do something clever and save us from this flaky insecure YAML hell?
To your second statement, I generally agree. Sounds strange to say given we're in the business of GHA runners. But it's just not a performant or reliable system at scale. This change from GitHub doesn't smell of a company that wants to do right by it's users.
If you are interested in what is up next for us at Depot, feel free to ping me via the email in my bio. I think you'll be quite interested in what we are doing.
Part of this is fair since there is a cost to operating the control plane.
One way around this is to go back to using check runs. I imagine a third party could handle webhooks, parse the .github/workflows/example.yml, then execute the action via https://github.com/nektos/act (or similar), then post the result.
To solve the problem, I created a simple vm in Google Cloud with a lot of CPU and memory that runs Ubuntu. I installed enough stuff on it to be able to check out code and run our build script (a jvm and gradle basically). And then I modified the Github action to 1) start the vm, 2) trigger the build script via ssh 3) pause the vm so we don't get billed for it. That vm runs for maybe an hour per month or so. It would probably cost us hundreds of euros per month if we ran it 24/7. But 1/3600th of that barely registers on our bills. And it's nice and fast.
This has been working flawlessly for a few years now. The Github action takes about 3 minutes. That includes starting the vm, running the script, and shutting the vm down again.
Wonky in a way. But also simple and robust enough. People over engineer/over think this stuff for the wrong reasons. For example, I could of course automate the provisioning of that vm. But I haven't. Because I only ever touch it once a year or so to run a quick apt-get update. I rebuilt it a few weeks ago in a different region. That was like a 20 minute job. Terraform or Ansible for vms you only create once every few years is redundant and might take more time than you would save. I can always do that when that stops being true.
I've been running this startup on the freemium layer in Github for five years now. It's great as a free service. I would actually pay for it if I needed to. I did actually pay for it before MS acquired Github in a previous startup when business usage wasn't free. But so far, there's no need for me to do that. I also run some monitoring scripts as Github actions. Simple curl jobs against our servers that trigger alerts when they fail. That has to run somewhere. It might as well be Github actions. But if/when that becomes inconvenient, I can improvise other solutions.
This is not new, not unexpected. This is ongoing. Nothing stops this because who wins elections? How do they pay for all that publicity. Certainly "contributing" to campaigns is much cheaper than paying your taxes.
Supposedly this is a place for hackers. Hackers can build a better alternative.
It'll be interesting to see how this affects third party companies providing GitHub runners.
I think most do. Or at least the infrastructure/compute costs are not coming from their own dept budget anymore ;)
This is an insult to anyone who bought into GitHub. It's an insult to all of us who have been doing OSS there for years. This is how you kill your business and any loyalty or trust in your brand.
What a disaster.
GitHub's log streaming also sucks. It's very laggy and chunked, whereas GitLab's is pretty much real-time.
1. Self-hosting runners or using WarpBuild/blacksmith runners is still cheaper Despite the $0.002/minute self-hosted runner tax, self-hosting runners on your cloud (aws/gcp/azure/...) or using WarpBuild/... runners remains the cheaper option.
2. Prefer larger runners If your workflow scales with the number of vCPUs, prefer larger runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, using actions-runner-controller with heavy jobs running on 1 vcpu runners is not a good idea. Instead, prefer a 2vcpu runner (say) if it runs the job ~2x faster.
3. Prefer faster runners All else being equal, prefer faster runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, if you're self-hosting on aws and using a t3g.medium runner, it's better to use a t4g.medium runner since the newer generation is faster, but not much more expensive.
4. Prefer fewer shards If you have a lot of shards for your jobs (example: tests on ~50 shards), consider reducing the number of shards and parallelizing the tests on fewer but larger runners.
5. Improve job performance This is not new advice, but it's now more important than ever because of the additional GitHub self-hosted runner tax.
6. Use GitHub hosted runners for very short jobs For linters and other very short jobs, it's better to use GitHub hosted runners.
Hope this helps. Note: I'm the founder of WarpBuild. I'm biased, but the points above hold.
If you've been running your runners on your own infra for cost reasons, you're not really that interesting to the Github business.
There are multiple competitors in this space. If you are (or were) paying for Github runners for any reason, you really shouldn't be.
Performance is the primary lever to pay less $0.002/min self hosting tax and we strive to provide the best performance runners.
Self-hosted runners help bridge the gap with on-prem servers, since you can pop a runner VM inside your infra and give it the connectivity/permissions to do deployments.
This announcement pisses me off, because it's not something related to abuse/recouping cost, since they could impose limits on free plans or whatever.
This will definitely influence me to ensure all builds/deployments are fully bash/powershell scripted without GH Action-specific steps. Actions are a bit of a dumpster fire anyway, so maybe I'll just go back to TeamCity like I used before Actions.
You can throw tons of cores and ram locally at problems without any licensing costs.
Your data may be local, makes sense to work with it locally.
the control plane clearly has value to people beyond the compute used for running the actions, and it seems reasonable that they should charge for that if you're using it.
For example, in our pipeline we have 5 different linter tasks (for different subprojects), running each only a few seconds. Nonetheless, we’ll get billed for 5 minutes on every commit.
It's not outrageous money today, but it's a clear signal about where they want CI to live.
- Git repo - Ticketing, Kaban - Full helpdesk - Complete and full CI/CD - everything links via custom workflow - self hosted
I still dont know why everyone hasn't switch yet to that banger.
My org sadly has a lot of github actions workflows, even after this it's not expensive enough to justify migrating away, but with all their downtime and bugs they are really pushing us closer and closer.
to top it all off, they round up to the nearest whole minute instead of billing for actual usage which i assume they'll use for this new charge.
Earthly did not work out, and dagger had the problem of we support everything but but nothing is great
People, now, are going to be annoyed and/or pissed off about this and look for/move to alternatives. It's not even that difficult to move, and if you're already self-hosting runners you're also in the position to self-host your Forge or move elsewhere.
Actions isn't even good enough to demand this. They're slow, buggy, and full of shit.
Feels super much like the classic Microsoft short-sighted bullshit. Take something that's been running well, and people were happy, then abruptly change it in disruptive ways and slowly kill your products that were doing just fine.
Github can just drop Actions pricing and leave the self-hosted stuff alone, and people would even have extended more goodwill. Is MS this short-sighted and greedy as to push further toward killing a golden goose?
But nothing they've done in the last few years has demonstrated improvement in this area. As the person with both purchasing and final authority on these things in my org, it's hard to stomach.
"Any Network Egress to CircleCI will be charged. At this current time, this includes CircleCI Caches, Workspaces, and Artifacts and will be charged at the normal rate according to your Usage Controls.
The only network traffic that will result in billing is accrued through restoring caches and workspaces, and downloading artifacts to self-hosted runners. Retention of artifacts, workspace, and cache objects will result in billing for storage usage.
Since your builds will not be running on CircleCI's Infrastructure, you will not be charged compute credits"
https://support.circleci.com/hc/en-us/articles/2064321965685...
I think that's fair. In my personal opinion most people started using GitHub Actions because it “came for free with the VCS and/or our MS contract” and it was “good enough for the job”. Now might be a good time to look around at the alternatives again. There is a reason that f.e. CircleCI is doing fully focused CI/CD for 10+ years and is still going strong. Plenty of businesses don’t want to put all their eggs in one (MS) basket, for all kinds of reasons. I guess today one of these reasons became obvious.
Disclaimer: I work at CircleCI.
There simply is no free lunch, somewhere someone needs to spend effort and time on managing the orchestration layer for the runners, and there is also network traffic and storage in play that costs money. If you need a future-proof CI/CD platform, it takes some investment. I agree that the Github "pay per minute" approach doesn't feel right, most people would probably find a "pay per orchestration job" or something more acceptable.
Anyway, there are alternatives out there :)
GitLab CI and others seem to be perfectly serviceable.
The biggest issue is the compatibility, forgejo doesn’t have all the actions available that GitHub does nor some of the same functionality
I suspect we'll be doing that sometime in January or February.
I guess forgejo is the easiest migration path? https://forgejo.org/
Holy s***
That's more expensive than an m8i.large.
What on earth.
I realise 100% utilisation isn't realistic, but that still sounds very expensive when you're already BYOB.
It's worse than unrealistic. It's ludicrous. Any company running more than an hour of actions workflows per week on GitHub can afford a few dollars a month for infrastructure. The per-minute charge is less than the cost of a millisecond of engineering labor time.
Charging per minute for self-hosted runners seems absolutely bananas!
I hate GH Action runners with a passion. They are slow, overpriced, and clearly held together with duct tape and chewing gum. WarpBuild, on the other hand, was a breeze to setup and provided faster runners and lower prices.
This is a really shitty move.
Hey GitHub, your Microsoft is showing...
However, my experience with GitHub Actions was really poor. Some build that would run perfectly on my local machine and any other servers we have hosted would always time out on GitHub runners. I went back and forth from small runners to large runners and the result was always the same. Then I found that there are third-party companies just offering replacement runners for GitHub Actions at less than half the price for an amazing reliability and cost. It was a night and day difference.
Now... this move by GitHub is almost unbelievable. Charging folks to use their own machines
Given github ran 11.5 billion mins of actions in 2025, and most of them would've been on self-hosted runners, this move makes some sense from their POV.
However, this is still an... interesting... move, especially after bitbucket got all that hate a few weeks ago for doing something similar.
a nice feature would be if they limit the number of branches, too: - <=2 branches: free - <=5 branches: 3.00$ per user per month - >5 branches: contact enterprise sales
Not sure about the "up to" implications, but I guess it's just Microsoft trying to make github a bit more freemium tm
> And we’re reducing the net cost of GitHub-hosted runners by up to 39%, depending on which machine type is used.
> The price reduction you will see in your account depends on the types of machines that you use most frequently – smaller runners will have a smaller relative price reduction, larger runners will see a larger relative reduction.
30 users + 500 builds.
However I don't know what counts as a build, since a typical commit to an open PR uses 10 GH runner machines simultaneously doing odd jobs like integration tests, releases, deploys, etc...
Pricing should mostly just be users + build minutes (for cloud runners) + storage. There is a few other optional, feature specific costs. Self hosted runners are free, but you need to self host caches/workspaces - our native ones have an egress bill to self hosted runners.
If self-hosted runners are free that would change our equation a bit. I'll talk to some folks here, I liked using this product at another company I worked at - but this would most likely shake out AFTER Github charges us the first time.
When building on self-hosted windows machines, you actually pay three times.
Oh I wish I could make my customers pay three times for everything I deliver, I might be as rich as Bill by now.
> Hosted Agents > > 2,000 minutes/month
:-o
Now the only alternative is to move builds, CI, etc. off of GitHub's platform entirely, and maybe your source control as well. In other words, a big pain. Github seems to have entered peak encrapification: the point where they openly acknowledge rent-seeking as their product approach, fully deprecating "building the best, most reliable, trustworthy product." Now it's just "Pay us high margins because the effort to migrate off is big and will take too long to break even."
Basically the modern day Heroku business model.
GitHub still supports e.g. PR checks that originate from other systems. We had PR checks before GHA and it's easy enough to go back to that. Jenkins has some stuff built in or you can make some simple API calls.
It's not as convenient, but it works just fine.
These people will forever unto the end of time into their afterlife have a Harem of old ladies following them around laughing at their never ending hilarious hot takes.
- DroneCI
- ConcourseCI
- forgejo can use github actions
https://docs.github.com/en/actions/reference/workflows-and-a...
My mistake was a combination of triggering workflows on every push, using macOS runners (which I didn't realize had a 10x multiplier compared to Linux), and forgetting to set aggressive timeouts.
I'm sharing this because the support experience was actually the highlight. I opened a ticket expecting to just eat the cost, but they sent a detailed breakdown of my usage/mistakes the next day. Even though it was 100% my error (tho I used to think macOS runners were only 3x? True, did that change? anyway), they gave me a $50 coupon to offset the overage. Amidst the pricing discussions, I think it's worth noting that their support team is still very human and responsive.
We will continue to do our best to provide the fastest GHA runners and keep them cheaper than GitHub-hosted runners.
Hard for me to feel like our industry is innovating and not just gouging with the rest in the battle for enshittification.
I will intentionally start exploring other options even if the cost isn't high, because I don't want to support this type of thing.
It seems GitLab has a much better experience in this department, but their pricing is hard to justify for us...
Genuinely curious if folks here had better experiences or recommendations for a smooth CI/CD experience.
Basically I'll gladly pay for a service, but I don't like getting locked into that service. If the payed service is using FOSS, I will always have the option to migrate if the provider starts to misbehave
I'm in the era of writing my own tools, not to share just for me or whatever group I'm working in. If you're going to charge me for something rife with annoying struggles, I might as well be annoyed by a tool I control.
That being said even with no free platform minutes my Blacksmith usage will only $1.20 a month in platform fees, so inconsequential.
Thanks, enshittification.
In 2010, people were saying it was very reasonable to start prioritizing publishers' ability to reach you over your organic contacts. After all, Facebook is providing this utility for free; shouldn't they be able to extract some additional revenue from their platform? And here we are in 2025...
"Here is how platforms die: First, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die."
We are on step 2: then they abuse their users to make things better for their business customers.
It's an unnecessary fee to use self-hosted (i.e., not GitHub-hosted) components in CI pipelines.
That's not true for _all GitHub Actions usage_.
https://resources.github.com/actions/2026-pricing-changes-fo...
> Standard GitHub-hosted or self-hosted runner usage on public repositories will remain free.
[1] https://nesbitt.io/2025/12/06/github-actions-package-manager... [2] https://github.com/github/roadmap/issues/592
(Which, yes, has implications for energy use/climate change too for sure).
It doesn't look like i currently have access to the usage data on any of the lots-of-runners-lots-of-PRs projects I currently work on (which are still probably way less than some large companies).
Any "large companies" don't give a shit about things at this cost level. They spend more on the time it takes you to open the door. The number of CI minutes could be astronomical and it still wouldn't rate above the threshold of caring. The time people in this thread have spent wringing their hands is way more expensive.
On my larger organization, we have on average 20 to 30 *active* runners during business hours. Assuming 5 on the off-hours, my napkin math says it comes down to about 10 fully-utilized-runners per month, so about 864$/mo. For the size of my organization that is honestly totally acceptable.
This is assuming 0.002$ per minute of job being actively executed. If it turns out to be 0.002$ per minute of *runner being registered* on the control plane, it would increase quite a bit. We are still using the old HorizontalRunnerAutoscaler with actions-runner-controller, with quite a pool of prewarmed runners idling to pick up a job. It would be a strong reason to use the new RunnerScaleSet (to take advantage of the reactive webhook-based scaling) and keep a very lean pool of prewarmed runners.