After years of dealing with this (first Jenkins, then GitLab, then GitHub), my takeaway is:
* Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code.
* Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing/debugging pipelines becomes a nightmare.
* Avoid YAML as much as possible, period.
* Don't bind yourself to some fancy new VC-financed thing that will solve CI once and for all but needs to get monetized eventually (see: earthly, dagger, etc.)
* Always use your own runners, on-premise if possible
It was a large enterprise CMS project. The client had previously told everyone they couldn't automate deployments due to the hosted platform security, so deployments of code and configs were all done manually by a specific support engineer following a complex multistep run sheet. That was going about as well as you'd expect.
I first solved my own headaches by creating a bunch of bash scripts to package and deploy to my local server. Then I shared that with the squads to solve their headaches. Once the bugs were ironed out, the scripts were updated to deploy from local to the dev instance. Jenkins was then brought in an quickly setup to use the same bash scripts, so now we had full CI/CD working to dev and test. Then the platform support guy got bored manually following the run sheet approach and started using our (now mature) scripts to automate deployments to stage and prod.
By the time the client found out I'd completely ignored their direction they were over the moon because we had repeatable and error free automated deployments from local all the way up to prod. I was quite proud of that piece of gorilla consulting :-)
There's probably a lesson in there.
I swear by TeamCity. It doesn't seem to have any of these problems other people are facing with GitHub Actions. You can configure it with a GUI, or in XML, or using a type safe Kotlin DSL. These all actually interact so you can 'patch' a config via the GUI even if the system is configured via code, and TeamCity knows how to store config in a git repository and make commits when changes are made, which is great for quick things where it's not worth looking up the DSL docs or for experimentation.
The UI is clean and intuitive. It has all the features you'd need. It scales. It isn't riddled with insecure patterns like GH Actions is.
CI is just the thing no one wants to deal with, yet everyone wants to just work. And like any code or process, you need engineering to make it good. And like any project, you can't just blame bad tools for crappy results.
Whereas, I could articulate why I didn't like Jenkins just fine :)
The thing I want to change are things that I do in the build system so that it is checked in and previous versions when we need to build them (we are embedded where field failure is expensive so there are typically branches for the current release, next release, and head). This also means anything that can fail on CI can fail on my local system (unless it depends on something like the number of cores on the machine running the build).
While the details can be slightly different, how we have CI is how it should be. most developers should have better things to do than worry about how to configure CI.
(I don't recall _loving_ it, though I don't have as many bad memories of it as I do for VSTS/TFS, GitLab, GH Actions, Jenkins Groovyfiles, ...)
We needed two more or less completely different configurations for old a new versions of the same software (think hotfix for past releases), but TeamCity can't handle this scenario at all. So now we have duplicated the configuration and some hacky version checks that cancel incompatible builds.
Maybe their new Pipeline stuff fixes some of these short comings.
This is making me realize I want a CI with as few features as possible. If I'm going to spend months of my life debugging this thing I want as few corners to check as I can manage.
I tend to stick with the GUI because if you're doing JVM style work the complexity and tasks is all in the build you can run locally, the CI system is more about task scheduling so it's not that hard to configure. But being able to migrate from GUI to code when the setup becomes complex enough to justify it is a very nice thing.
Any CI product play has to differentiate in a way that makes you dependent on them. Sure it can be superficially nicer when staying inside the guard rails, but in the age of docker why has the number of ways I configure running boring shell scripts gone UP? Because they need me unable to use a lunch break to say "fuck you I don't need the integrations you reserve exclusively for your CI" and port all the jobs back to cron.
And that's why jenkins is king.
If you make anything more than that, your CI will fail. And you can do that with Jenkins, so the people that did it saw it work. (But Jenkins can do so much more, what is the entire reason so many people have nightmares just by hearing that name.)
We build Docker images mostly so ymmv.
I have a "port to github actions" ticket in the backlog but I think we're not going to go down that road now.
You'll have to explain the weird CPS transformations, you'll probably end up reading the Jenkins plugins' code, and there's nothing fun down this path.
Probably 'guerilla', but I like your version more.
Wikipedia: Gorilla Suit: National Gorilla Suit Day:
https://en.wikipedia.org/wiki/Gorilla_suit#National_Gorilla_...
Put the Gorilla back in National Gorilla Suit Day:
https://www.instagram.com/mad.magazine/p/C2xgmVqOjL_/
Gorilla Suit Day – January 31, 2026:
https://nationaltoday.com/gorilla-suit-day/
National Gorilla Suit Day:
You can use Nix with GitHub actions since there is a Nix GitHub action: https://github.com/marketplace/actions/install-nix. Every time the action is triggered, Nix rebuilds everything, but thanks to its caching (need to be configured), it only rebuilds targets that has changed.
> How do you automate running tests and deploying to dev on every push
Nix is a build tool and it's main purpose is not to deploy artifacts. There are however a lot of tools to deploy artifacts built by Nix: https://github.com/nix-community/awesome-nix?tab=readme-ov-f...
Note there are also several Nix CI that can do a better job than a raw GitHub actions, because they are designed for Nix (Hydra, Garnix, Hercules, ...).
devShells.default = pkgs.mkShell {
packages = with pkgs; [ opentofu terragrunt ];
};
jobs:
terragrunt-plan:
runs-on: [self-hosted, Linux, X64]
defaults:
run:
shell: nix develop --command bash -e {0}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Plan
run: terragrunt --terragrunt-non-interactive run-all plan
> How do you automate running tests
You just build the Nix derivation that runs your tests, e.g. `nix build #tests` or `nix flake check` in your workflow file.
> deploying to dev on every push
You can set up a Nix `devShell` as a staging area for any operations you'd need to perform for a deployment. You can use the same devShell both locally and in CI. You'd have to inject any required secrets into the Action environment in your repository settings, still. It doesn't matter what your staging environment is comprised of, Nix can handle it.
Every CI "platform" is trying to seduce you into breaking things out into steps so that you can see their little visualizations of what's running in parallel or write special logic in groovy or JS to talk to an API and generate notifications or badges or whatever on the build page. All of that is cute, but it's ultimately the tail wagging the dog— the underlying build tool should be what is managing and ordering the build, not the GUI.
What I'd really like for next gen CI is a system that can get deep hooks into local-first tools. Don't make me define a bunch of "steps" for you to run, instead talk to my build tool and just display for me what the build tool is doing. Show me the order of things it built, show me the individual logs of everything it did.
Same thing with test runners. How are we still stuck in a world where the test runner has its own totally opaque parallelism regime and our only insight is whatever it chooses to dump into XML at the end, which will be probably be nothing if the test executable crashes? Why can't the test runner tell the CI system what all the processes are that it forked off and where each one's respective log file and exit status is expected to be?
Nix really helps with this. Its not just that you do everything via a single script invocation, local or ci, you do it in an identical environment, local or ci. You are not trying to debug the difference between Ubuntu as setup in GHA or Arch as it is on your laptop.
Setting up a nix build cache also means that any artefact built by your CI is instantly available locally which can speed up some workflows a lot.
- Everything sandboxed in containers (works the same locally and in CI)
- Integrate your build tools by executing them in containers
- Send traces, metrics and logs for everything at full resolution, in the OTEL format. Visualize in our proprietary web UI, or in your favorite observability tool
I work on garnix.io, which is exactly a Nix-based CI alternative for GitHub, and we had to build a lot of these small things to make the experience better.
All of that is a lot more than what a local dev would want, deploying to their own private test instance, probably with a bunch of API keys that are read-only or able to write only to other areas meant for validation.
Maybe add some semi-structured log/trace statements for the CI to scrap.
No hooks necessary.
How much better would it be if the CI web client could just say, here's everything the build tool built, with their individual logs, and here's a direct link to the one that failed, which canceled everything else?
But how do you get that sweet, sweet vendor-lock that way? /s
When I joined my first web SaaS startup I had a bit of a culture shock. Everything was running on 3rd party services with their own proprietary config/language/etc. The base knowledge of POSIX/Linux/whatever was almost completely useless.
I'm kinda used to it now, but I'm not convinced it's any better. There are so many layers of abstraction now that I'm not sure anybody truly understands it all.
It blows my mind what is involved in creating a simple web app nowadays compared to when I was a kid in the mid-2000s. Do kids even do that nowadays? I’m not sure I’d even want to get started with all the complexity involved.
If you want to use a framework The React tutorials from Traversy media are pretty good. You can even do cross platform into mobile app with frameworks like React Native or Flutter if you want iOS/Android native apps.
Vite has been a godsend for React/Vue. It’s no longer the circus it was in the mid 2010s. Google’s monopoly has made things easier for web devs. No more babel or polyfill or createReactApp.
People do still avoid frameworks and use raw HTML/CSS/Javascript. HTMX has made sever fetches a lot easier.
You probably want a decent CSS framework for reponsive design. Everyone used to use minimalist ones like Tailwimd have become more popular.
If you need a backend and want to do something simple you can use BaaS (Backend as a service) platforms like Firebase. Otherwise setting up a NodeJS server with some SQL or KV store like SQLLite or MongoDB isn’t too difficult
CI/CD systems exist to streamline testing and deployment for large complex apps. But for individual hobbyist projects it’s not worth it.
It’s demonstrably worse.
> The base knowledge of POSIX/Linux/whatever was almost completely useless.
Guarantee you, 99% of the engineering team there doesn’t have that base knowledge to start with, because of:
> There are so many layers of abstraction now that I'm not sure anybody truly understands it all.
Everything is constantly on fire, because everything is a house of cards made up of a collection of XaaS, all of which are themselves houses of cards written by people similarly clueless about how computers actually operate.
I hate all of it.
Your Jenkins experience is more valuable and worth replicating when you get the opportunity.
Once you get on Dagger, you can turn your CI into minimal Dagger invocations and write the logic in the language of your choice. Runs the same locally and in automation
I personally hold Dagger a bit different from most, by writing a custom CLI and using the Dagger Go SDK directly. This allows you to do more host level commands, as everything in a Dagger session runs in a container (builds and arbitrary commands).
I've adopted the mono/megarepo organization and have a pattern that also includes CUE in the solution. Starting to write that up here: https://verdverm.com/topics/dev/dx
Between that and upgrading for security patches. Developing user impacting code is becoming a smaller and smaller part of software development.
I heavily invested in a local runner based CI/CD workflow. First I was using gogs and drone, now the forgejo and woodpecker CI forks.
It runs with multiple redundancies because it's a pretty easy setup to replicate on decentralized hardware. The only thing that's a little painful is authentication and cross-system pull requests, so we still need our single point of failure to merge feature branches and do code reviews.
Due to us building everything in go, we also decided to have always a /toolchain/build.go so that we have everything in a single language, and don't need even bash in our CI/CD podman/docker images. We just use FROM scratch, with go, and that's it. The only exception being when we need to compile/rebuild our ebpf kernel modules.
To me, personally, the Github Actions CVE from August 2024 was the final nail in the coffin. I blogged about it in more technical detail [1] and guess what was the reason that the TJ actions have been compromised last week? Yep, you guessed right, the same attack surface that Github refuses to fix, a year later.
The only tool, as far as I know, that somehow validates against these kind of vulnerabilities, is zizmor [2]. All other tools validate schemas, not vulnerabilities and weaknesses.
[1] https://cookie.engineer/weblog/articles/malware-insights-git...
If you take a look at the pull requests in e.g. the changed-files repo, it's pretty obvious what happened. You can still see some of the malformed git branch names and other things that the bots tried out. There were lots of "fixes" that just changed environment variable names from PAT_TOKEN to GITHUB_TOKEN and similar things afterwards, which kind of just delays the problem until malware is executed with a different code again.
As a snarky sidenote: The Wiz article about it is pretty useless as a forensics report, I expected much more from them. [1]
The conceptual issue is that this is not fixable unless github decides to rewrite their whole CI/CD pipeline, because of the arbitrary data sources that are exposed as variables in the yaml files.
The proper way to fix this (as Github) would be to implement a mandatory linter step or similar, and let a tool like zizmor check the file for the workflow. If it fails, refuse to do the workflow run.
[1] https://www.wiz.io/blog/github-action-tj-actions-changed-fil...
Mise can install all your deps, and run tasks
From dagger.io...
"The open platform for agentic software.
Build powerful, controllable agents on an open ecosystem. Deploy agentic applications with complete visibility and cross-language capabilities in a modular, extensible platform.
Use Dagger to modernize your CI, customize AI workflows, build MCP servers, or create incredible agents."
So now we are trying to capitalize on it, hence the ongoing changes to our website. We are trying to avoid the "something something agents" effect, but clearly, we still have work to do there :) It's hard to explain in marketing terms why a ephemeral execution engine, cross-language component system, deep observability and interactive CLI can be great at running both types of workloads... But we're going to keep trying!
Internally we never thought of ourselves as a CI company, but as an operating system company operating in the CI market. Now we are expanding opportunistically to a new market: AI agents. We will continue to support both, because our platform can run both.
If you are interested, I shared more details here: https://x.com/solomonstre/status/1895671390176747682
The risk of muddling is limited to the marketing, though. It's the exact same product powering both use cases. We would not even consider this expansion if it wasn't the case.
For example, Dagger Cloud implements a complete tracing suite (based on OTEL). Customers use it for observability of their builds and tests. Well it turns out, you can use the exact same tracing product for observability of AI agents too. And it turns out that observability is huge unresolved problem of AI agents! The reason is because, fundamentally, AI agents work exactly like complicated builds: the LLM is building its state, one transformation at a time, and sometimes it has side effects along the way via tool calling. That is exactly what Dagger was built for.
So, although we are still struggling to explain this reality to the market: it is actually true that the Dagger platform can run both CI and AI workflows, because they are built on the same fundamentals.
Or I'll ask v0.dev to reimplement it, but I think it'd be more complete if you did it
I can understand what you're trying to say, but because I don't have clear "examples" at hand which show me why in practice handling such cases are problematic and why your platform makes that smooth, I don't "immediately" see the value-added
For me right now, the biggest "value-added" that I perceive from your platform is just the "CI/CD as code", a bit the same as say Pulumi vs Terraform
But I don't see clearly the other differences that you mention (eg observability is nice, but it's more "sugar" on top, not a big thing)
I have the feeling that indeed the clean handling of "state" vs "side-effects" (and what it implies for caching / retries / etc) is probably the real value here, but I fail to perceive it clearly (mostly because I probably don't (or not yet) have those issues in my build pipelines)
If you were to give a few examples / ELI5 of this, it would probably help convert more people (eg: I would definitely adopt a "clean by default" way of doing things if I knew it would help me down the road when some new complex-to-handle use-cases will inevitably pop up)
They do seem to have a nice "quickstart for CI" they haven't abandoned, yet: https://docs.dagger.io/ci/quickstart
(As much as I personally like working with CI and build systems, it's true there's not a ton of money in it!)
Literally from comment at the root of this thread.
https://www.boringbusinessnerd.com/startups/dagger
Mise indeed isn't, but its scope is quite a bit smaller than Dagger.
A lot of us could learn... do one thing and do it well
target: $(DOCKER_PREFIX) build
When run in gitlab, the DOCKER_PREFIX is a no-op (it's literally empty due to the CI=true var), and the 'build' command (whatever it is) runs in the CI/CD docker image. When run locally, it effectively is a `docker run -v $(pwd):$(pwd) build`.
It's really convenient for ensuring that if it builds locally, it can build in CI/CD.
I let GitHub actions do things like the initial environment configuration and the post-run formatting/annotation, but all of the actual work is done by my scripts:
https://github.com/Hammerspoon/hammerspoon/blob/master/.gith...
https://github.com/williamcotton/webdsl/blob/main/.github/wo...
The other thing I would add is consider passing in all environment variables as args. This makes it easy to see what dependencies the script actually needs, and has the bonus of being even more portable.
Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines. This is insane. You’re actually introducing much needed sanity into the process by admitting that a real programming language is the tool to use here.
I can’t imagine the cognitive dissonance Lisp folks have when dealing with this madness, not being one myself.
After a decade trying to fight it, this one Lisper here just gave up. It was the only way to stay sane.
I remain hopeful that some day, maybe within our lifetimes, the rapid inflation phase of software industry will end, and we'll have time to rethink and redo the fundamentals properly. Until then, one can at least enjoy some shiny stuff, and stay away from the bleeding edge, aka. where sewage flows out of pipe and meets the sea.
(It's gotten a little easier now, as you can have LLMs deal with YAML-programming and other modern worse-is-better "wisdom" for you.)
It would really benefit from a language that intrinsically understood its being used to control a state machine. As it is, that is what nearly all folks want in practice is a way to run different things based on different states of CI.
A lisp DSL would be perfect for this. Macros would make things alot easier in many respects.
Unfortunately, there's no industry consensus and none of the big CI platforms have adopted support for anything like that, they all use variants of YAML (I always wondered who started it with YAML and why everyone copied that, if anyone knows I'd love to read about it).
Honestly, I can say the same complaints hold up against the cloud providers too. Those 'infrastructure as code' SDKs really don't lean into the 'as code' part very well
"Application and configuration should be separate, ideally in separate repos. It is the admin's job to configure, not the developer's"
"I do not need to learn your garbage language to understand how to deploy or test your application"
"...as a matter of fact, I don't need to learn the code and flows of the application itself either - give me a binary that runs. But it should work with stale configs in my repo."
"...I know language X works for the application but we need something more ubiquitous for infra"
Then there was a crossover of three streams, as I would call it:
YAML was emerging "hard" on the shoulders of Rails
Everyone started hating on XML (and for a good reason)
Folks working on CI services (CruiseControl and other early solutions) and ops tooling (chef, ansible) saw JSON's shortcomings (now an entire ecosystem has configuration files with no capability to put in a comment)
Since everybody hated each other's languages, the lowest common denominator for "configuration code" came out to be YAML, and people begrudgingly agreed to use it
The situation then escalated severely with k8s, which adopted YAML as "the" configuration language, and a whole ecosystem of tooling sprung up on top using textual templating (!) of YAML as a layer of abstraction. For k8s having a configuration language was an acute need, because with a compiled language you need something for configuration that you don't have to compile with the same toolchain just to use - and I perfectly "get it" why they settled for YAML. I do also get why tools like Helm were built on top of YAML trickery - because, be it that Helm were written in some other language, and have its charts use that, they would alienate all the developers that either hate that language personally, or do not have it on the list of "golden permitted" at their org.
Net result is that YAML was chosen not because it is good, but because it is universally terrible in the same way for everyone, and people begrudgingly settled on it.
With CI there is an extra twist that a good CI setup functions as a DAG - some tasks can - and should - run in parallel for optimization. These tasks produce artifacts which can be cached and reused, and a well-set CI pipeline should be able to make use of that.
Consequently, I think a possible escape path - albeit an expensive one - would be for a "next gen" CI system to expose those _task primitives_ via an API that is easy to write SDKs for. Read: not a grpc API. From there, YAML could be ditched as "actual code" would manipulate the CI primitives during build.
I know this isn't a definite answer to your question, but it was still super interesting to me and hopefully it will inspire someone else to dig into finding the actual answer
The best guess I have as far as CI/CD specifically appears to be <https://en.wikipedia.org/wiki/Travis_CI#:~:text=travis%20ci%...> which launched in 2011 offering free CI and I found a reference to their .travis.yml in GitLab's repo in 2011, too
- CruiseControl (2004) was "ant as a service," so it was XML https://web.archive.org/web/20040812214609/http://confluence...
- Hudson (2007) https://web.archive.org/web/20140701020639/https://www.java.... was also XML, and was by that point driving Maven 2 builds (also XML)
- I was shocked that GitHub existed in 2008 https://web.archive.org/web/20081230235955/http://github.com... with an especial nod to no longer a pain in the ass and Not only is Git the new hotness, it's a fast, efficient, distributed version control system ideal for the collaborative development of software but this was just "for funsies" link since they were very, very late to the CI/CD game
- I was surprised but k8s 1.0.0 still had references to .json PodSpec files in 2010 https://github.com/kubernetes/kubernetes/blob/v1.0.0/example...
- cloud-init had yaml in 2010 https://github.com/openstack-archive/cloud-init/blob/0.7.0/d... so that's a plausible "it started here" since they were yaml declarations of steps to perform upon machine boot (and still, unquestionably, my favorite user-init thing)
- just for giggles, GitLab 1.0.2 (2011) didn't even have CI/CD https://gitlab.com/gitlab-org/gitlab/-/tree/v1.0.2 -- however, while digging into that I found .travis.yml in v2.0.0 (also 2011) so that's a very plausible citation <https://gitlab.com/gitlab-org/gitlab/-/blob/v2.0.0/.travis.y...>
- Ansible 1.0 in 2012 was also "execution in yaml" https://github.com/ansible/ansible/blob/v1.0/examples/playbo...
I've been using YAML for ages and I never had any issue with it. What do you think is wrong with YAML?
Many of us would rather use a less terrible programming language instead.
Some people talk about YAML being a turing complete language, if people try to do that in your CI/CD system just fire them
I'll allow helm style templating but that's about it.
It's miles better than Jenkins and the horrors people created there. GitLab CI can at least be easily migrated to any other GitLab instance and stuff should Just Work because it is in the end not much more than self contained bash scripts, but Jenkins... is a clown show, especially for Ops people of larger instances. On one side, you got 50 plugins with CVEs but you can't update them because you need to find a slot that works for all development teams to have a week or two to fix their pipelines again, and on the other side you got a Jenkins instance for each project which lessens the coordination effort but you gotta worry about dozens of Jenkins instances. Oh and that doesn't include the fact many old pipelines aren't written in Groovy or, in fact, in any code at all but only in Jenkins's UI...
Github Actions however, I'd say for someone coming from GitLab, is even worse to work with than Jenkins.
Instead of wrapping the aws cli command I wrote small Go applications using the boto3 library.
Removed the headaches when passing in complex params, parsing output and and also made the logic portable as we need to do the builds on different platforms (Windows, Linux and macOS).
I have seen people doing absolutely insane setups because they thought they have to do it in yaml and pipeline and there is absolutely no other option or it is somehow wrong to drop some stuff to code.
I'm not sure I understood what you're saying because it sounds too absurd to be real. The whole point of a CICD pipeline is that it automates all aspects of your CICD needs. All mainstream CICD systems support this as their happy path. You specify build stages and build jobs, you manage your build artifacts, you setup how things are tested, deployed and/or delivered.
That's their happy path.
And you're calling the most basic usecases of a standard class if tools as "insanity"?
Please help me explain what point you are trying to make.
All aspects of your CICD pipeline - rebasing PRs is not 'basic CICD' need.
CICD pipeline should take a commit state and produce artifacts from that state, not lint and not autofix trivial issues.
Everything that is not "take code state - run tests - build - deploy (eventualy fail)" is insanity.
Autofixing/linting for example should be separate process waay before CICD starts. And people do stuff like that because they think it is part of integration and testing. Trying to shove it inside is insanity.
This is the dumbest thing I see installers do a lot lately.
It tends to be that folks want to shoehorn some technology into the pipeline that doesn't really fit, or they make these giant one shot configurations instead of running multiple small parallel jobs by setting up different configurations for different concerns etc.
It's really easy to extend and compose jobs, so it's simple to unit test your pipeline: https://gitlab.com/nunet/test-suite/-/tree/main/cicd/tests?r...
This way I can code my pipeline and use the same infrastructure to isolate groups of jobs that compose a relevant functionality and test it in isolation to the rest of the pipeline.
I just wish components didn't have such a rigid opinion on folder structure, because they are really powerful, but you have to adopt gitlab prescription
I maintained a Javascript project that used Make and it just turned into a mess. We simply changed all of our `make some-job` jobs into `./scripts/some-job.sh` and not only was the code much nicer, less experienced developers were suddenly more comfortable making changes to scripts. We didn't really need Make to figure out when to rebuild anything, all of our tools already had caching.
The main argument I wanted to make is that it works very well to just use GitHub actions to execute your tool of choice.
It allows you to define a central interface into your project (largely what I find people justify using Make for), but smoothes out so many of the weird little bumps you run into from "using Make wrong."
Plus, you can an any point just drop into running a script in a different language as your command, so it basically "supports bash scripts" too.
So if even remotely possible we write all CI as a single 'one-click' script which can do it all by itself. Makes developing/testing the whole CI easy. Makes changing between CI implementations easy. Can solve really nasty issues (think: CI is down, need to send update to customer) easily because if you want a release you just build it locally.
The only thing it won't automaticaly do out of the box is being fast, because obviously this script also needs to setup most of the build environment. So depending on the exact implementation there's variation in the split between what constitutes setting up a build environment and running the CI script. As in: for some tools our CI scripts will do 'everything' so starting from a minimal OS install. Whereas others expect an OS with build tools and possibly some dependencies already available.
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
Yes, a thousand time.
Deploy scripts are tougher to deal with, as they'll naturally rely on a flurry of environment variables, protected credentials etc.
But for everything else writing the script for local execution first, and generalizating them for CI one they run well enough is the absolute best approach. It doesn't even need to run in the local shell, having all the CI stuff in a dedicated docker image is fine if it requires specific libraries or env.
- Treat pipelines as code. - Make pipelines parts composable, as code. - Be mindful of vendor lock-in and/or lack of portability (it is a trade-off).
For on-promise: if you're already deeply invested in running your own infrastructure, that seems like a good fit.
When thinking about how we build Namespace -- there are parts that are so important that we just build and run internally; and there are others where we find that the products in the market just bring a tremendous amount of value beyond self-hosting (Honeycomb is a prime example).
Use the tools that work best for you.
I fully agree with the recommendation to use maintainable code. But that effectively rules out shell scripts in my oppinion. CI shell scripts tend to become big ball of mud rather quickly as you run into the limitations of bash. I think most devs only have superficial knowledge of shell scripts, so do yourself a favor and skip them and go straight to whatever language your team is comfortable with.
One can learn to use it to the point where it's usable to do advanced automation... but why, when there are so many better options available?
The second one has been, from someone else: if you can use anything else than bash, do that.
Jokes aside... it's so trendy to bash bash that it's not funny anymore. Bash is still quite reliable for work that usually gets done in CI, and nearly maintenance free if used well.
THIS 10000% percent.
My personal favourite solution is Bazel specifically because it can be so isolated from those layers.
No need for Docker (or Docker in Docker as many of these solutions end up requiring) or other exotic stuff, can produce OCI image artifacts with `rules_oci` directly.
By requiring so little of the runner you really don't care for runner features, you can then restrict your CI/CD runner selection to just reliability, cost, performance and ease of integration.
That's also a very valid takeaway for life in general
Doesn’t matter Jenkins or actions - it is just complicated. Making it simpler is on devs/ops not the tool.
Each systemd service could represent a step built by running a script, and each service can say what it depends on, thus helping parallelize any step that can be.
I have not found anyone trying that so far. Is anybody aware of something similar and more POSIX/cross platform that allows writing a DAG of scripts to execute?
Facts.
However I’ll go a step further and say “only implement your logic in a tool that has a debugger”.
YAML is the worse. But shell scripts are second worst. Use a real language.
That said, if you absolutely need to use shell script for reasons, keep it all in single script, define logging functions including debug logs, rigorously check every constraint and variable, use shellcheck, factor the code well into functions - I should sometimes write a blog post about it.
Unfortunately, this isn't a good plan going forward... :( Going forward I'd wish for a tool that's as ubiquitous as Git, has good integration with editors like language servers, can be sold as a service or run completely in-house. And it would allow defining the actions of the automated builds and tests, have a way of dealing with releases, expose interface for collecting statistics, integrate with bug tracing software for the purpose of excluding / including tests in test runs, allowed organizing tests in groups (eg. sanity / nightly / rc).
The problem is that tools today don't come anywhere close to being what I want for CI, neither free nor commercial tools aren't even going in the desired direction. So, the best option is simply to minimize their use.
Why does YAML have any traction when JSON is right there? I'm an idiot amateur and even I learned this lesson; my 1 MB YAML file full of data took 15 seconds to parse each time. I quickly learned to use JSON instead, takes half a second.
Because it has comments, which are utterly essential for anything used as a human readable/writable configuration file format (your use case, with 1 MB of data, needs a data interchange format, for which yes JSON is at least much better than YAML).
YAML has comments. YAML is easily & trivially written by humans. JSON is easily & trivially written by code.
My lesson learned here? When generating YAML, instead generate JSON. If it's meant to be read and updated by humans, use something that can communicate to the humans (comments). And don't use YAML as a data interchange format.
For short configs, YAML is acceptable-ish. For anything longer I'd take TOML or something else.
- mise for lang config
- direnv for environment loading
- op for secret injection
- justfile for lint, build, etc
Here's a template repo that I've been working on that has all of this implemented:
https://github.com/iloveitaly/python-starter-template
It's more complex than I would like it to be, but it's consistent and avoids having to deal with GHA too much.
I've also found having a GHA playground is helpful:
However, CI is not "configured", it is coded. It is simply the wrong tool. YAML was continuously extended to deal with that, so it developed into much more than just "markup", but it grew into this terrible chimera. Once you start using advanced features in GitLab's YAML like anchors and references to avoid writing the same stuff again and again, you'll notice that the whole tooling around YAML is simply not there. How does the resulting YAML look like? How do you run this stuff locally? How do you debug this? Just don't go there.
You will not be able to avoid YAML completely, obviously, but use it the way it was originally intended to.
Finally! I was always struggling to explain to others why YAML is OK-ish as a language, but then never seems to work well for the things people tried doing with it. Especially stuff that needs to run commands, such as CI.
> How does the resulting YAML look like? How do you run this stuff locally? How do you debug this? Just don't go there.
Agreed. GitHub actions, or any remote CI runner for that matter, makes the problem even worse. The whole cycle of having to push CI code, wait 10 minutes while praying for it to work, still getting an error, trying to figure out the mistake, fixing one subtle syntax error, then pushing the code again in the hope that that works is just a terrible workflow. Massive waste of time.
> You will not be able to avoid YAML completely, obviously, but use it the way it was originally intended to.
Even for configurations YAML remains a pain, unfortunately. It could have been great for configs, but in my experience the whole strict whitespace (tabs-vs-spaces) part ruined it. It isn't a problem when you work from an IDE that protects you from accidentally using tabs (also, auto-formatting for the win!) but when you have to write YAML configuration (for example: Netplan) on a remote server using just an editor it quickly becomes a game of whack-a-mole.
I don't understand what problem you could possibly be experiencing. What exactly do you find hard about running commands in, say, GitLab CICD?
iterating a GitHub Actions workflow is a gigantic pain in the ass. Capturing all of the important logic in a script/makefile/whatever means I can iterate it locally way faster and then all I need github to do is provision an environment and call my scripts in the order I require.
What's wrong with this?
https://docs.github.com/en/actions/writing-workflows/choosin...
What exactly do you find hard in writing your own scripts with a scripting language? Surely you are not a software developer who feels conditionals and variable substitutions are hard.
> it ends up being 20 steps in a language that isn't shell but is calling shell over and over again, and can't be run outside of CI.
Why are you writing your CICD scripts in a way that you cannot run them outside of a CICD pipeline? I mean, you're writing them yourself, aren't you? Why are you failing to meet your own requirements?
If you have a requirement to run your own scripts outside of a pipeline, how come you're not writing them like that? It's CICD 101 that those scripts should be runnable outside of the pipeline. From your description, you're failing to even follow the most basic recommendations and best practices. Why?
That doesn't sound like a YAML problem, does it?
In order to use this domain-specific language properly, you first must learn it, and learning YAML is but a small part of that. Moreover, it is not immediately obvious that, once you know it, you actually want to avoid it. But you can't avoid it entirely, because it is the core language of the CI/CD platform. And you can't know how to avoid it effectively until you have spent some time just using it directly. Simplicity comes from tearing away what is unnecessary, but to discern necessary from unnecessary requires judgment gained by experience. There is no world in which this knowledge transfers immediately, frictionlessly, and losslessly.
Furthermore, there is a lot that GitHub (replace with platform of choice) could have done to make this better. They largely have no incentive to do so, because platform lock-in isn't a bad thing to the platform owner, and it's a nontrivial amount of work on their part, just as it is a nontrivial amount of work on your part to learn and use their platform in a way that doesn't lock you into it.
A: Easy! You just spin up a Kubernetes pod with Alpine image, map a couple of files inside, run a bash script of "date" with some parameters, redirect output to a mapped file, and then read the resulting file. That's all. Here's a YAML for you. Configuration, baby!
(based on actual events)
(Iterating even on this stuff by waiting for the runner is still annoying though. You need to commit to the repo, push, and wait. Hence the suggestion of having scripts that you can also run locally, so you can test changes locally when you're iterating on them. This isn't any kind of guarantee, but it's far less annoying to do (say) 15 iterations locally followed by the inevitable extra 3 remotely than it is having to do all 18 remotely, waiting for the runner each time then debugging it by staring at the output logs. Even assuming you'd be able to get away with as few as 15 given that you don't have proper access to the machine.)
Where?
> don't have anything particularly interesting in the .yml file, just the bare minimum plus some small number of uncomplicated script invocations to install dependencies and actually do the build
It is a very basic "how to" with no recommendations.
Moreover, they directly illustrate a bad practice:
- name: Run the scripts
run: |
./my-script.sh
./my-other-script.sh
This is invoking a shell and that's how shells typically work, one command at a time. Would it make you feel better if they added && or used a step like they also recommend to split these out? You can put the error handling in your script if need be, that's on you or the reader, most CI agents only understand true/false or in this case $?.
Nobody said they want that behavior, they're showing you the behavior. They actually show you the best practice behavior first, not sure if you didn't read that or are purposely omitting it. In fact, the portion you highlight, is talking about permissions, not making suggestions.
- name: Run a script
run: ./my-script.sh
- name: Run another script
run: ./my-other-script.shThat page is just one small part of a much larger reference document, and it doesn't seem opinionated at all to me. Plus there are dozens of other examples elsewhere in the same reference that are not simple invocations of one shell script and nowhere are you admonished not to do things that way.
No, it really isn't. I'll clarify why.
Pretty much all pipeline services share the same architecture pattern:
* A pipeline run is comprised of one or more build jobs,
* Pipeline runs are triggered by external events
* Build jobs have contexts and can output artifacts,
* Build jobs are grouped into stages,
* Stages are organized as a directed graph,
* Transitions between stages in the directed graph is ruled by a set of rules, some supported by default (i.e., if a job fails then the stage fails) complemented by custom rules (manual or automatic approvals, API tests, baking periods, etc).
This is the textbook scenario ideal for DSLs. You already are bound to an architecture pattern, this there is no point of reinventing the wheel each time. Just specify your stages and which jobs run as part of each stage, manage artifacts and promotion logic, and you're done.
You do not need to take my word for it. Take a look at GitLab CICD for a pipeline with build, test, and delivery stage. See what a mess you will put together if you support the same feature set with whatever scripting language you choose. There is no discussion or debate.
The problem starts when that graph cannot be determined in advance and needs to be computed in runtime. It's a bit better when it's possible to compute that graph as a first step, and it's a lot worse when one needs to do a couple of stages before being able to compute the next elements of the graph. The graph computation is terrible enough in e.g. Groovy, but having to do it in YAML is absolutely horrendous.
> Take a look at GitLab CICD for a pipeline with build, test, and delivery stage
Yeah, if your workflow fits in a kindergarten example of "build, test, and delivery", then yeah, it's YAML all the way baby. Not everyone is so fortunate.
Wrapping it in a DSL encoded as YAML has zero benefit other than it being easier for a team with weak design skills to implement and harder for users to migrate off of.
Pardon my pedantry, but the meaning of YAML's name was changed from the original “Yet Another Markup Language” to “YAML Ain't Markup Language” in a 2002 draft spec because YAML is, in fact, not a markup language :)
Compare:
Brings to mind the classic "Kingdom of Nouns" [0] parable, which I read to my kid just last week. The multi-line "run" nodes in GitHub actions give me the heebie-jeebies, like how MUMPS data validation was maintained in metadata of VA-Fileman [1].
0. https://steve-yegge.blogspot.com/2006/03/execution-in-kingdo...
I will take the Actions path 100% of the time. Building your own action is so insanely simple it makes me wonder if the people complaining about YAML understand the tooling because it's entirely avoidable. It also coincides with top comments about coding your own CI, if you're just "using" YAML you're barely touching the surface.
* Very easy to write the code you didn't mean to, especially in the context of CI where potentially a lot of languages are going to be mixed, a lot of quoting and escaping. YAML's string literals are a nightmare.
* YAML has no way to express inheritance. Nor does it have a good way to express variables. Both are usually desperately needed in CI scripts, and are usually bolted on top with some extra-language syntax (all those dollars in GitHub actions, Helm charts, Ansible playbooks etc.)
* Complexity skyrockets compared to the size of the file. I.e. in a language like C you can write a manageable program with millions of lines of code. In YAML you will give up after a few tens of thousands of lines (similar to SQL or any other language that doesn't have modules).
* Whitespace errors are very hard to spot and fix. Often whitespace errors in YAML result in valid YAML which, however, doesn't do what you want...
2. Trying to encode logic and control flow in a YAML document is much more difficult than writing that flow in a "real" programming language. Debugging is especially much easier in "real" languages.
YAML is great for the happy-flow where everything works. It's absolutely terrible for any other flow.
MSBuild, for example, is all XML, but it has debugging support in Visual Studio complete with breakpoints and expression evaluation.
It's a DSL. There is no execution, only configuration. The only thing that's executed are the custom scripts you create yourself, and any intro tutorial on the subject will eventually teach you that if you want to run anything beyond a single straight-forward command then you should move those instructions to a shell script to make them testable and reproducible.
Things are so simple and straight forward that you need to go way out of your way to create your own problems.
I wonder how many people in this discussion are blaming the tools when they even bothered to learn the very basics.
> It's a DSL. There is no execution, only configuration.
Jenkins pipelines are also DSL. I still can print out debugging information from them. "It's a DSL" is not an excuse for being a special case of shitty DSL.
> any intro tutorial on the subject will eventually teach you
Do these tutorials have a chapter on what to do when you join a company with 500 engineers and a ton of YAMLs that are not written in that way?
> you should move those instructions to a shell script to make them testable
Yeah, no. How am I supposed to test my script that is supposed to run on Github-supplied runner with a ton of injected secrets and Github-supplied JSON of 10,000 lines, when I don’t have the runner, the secrets, or the JSON?
The YAML is fed into an agent which reads it to decide what to execute. Any time you change the control flow of a system by changing data, you are doing a form of programming.
For example: Stripe uses constants for types of tax registration numbers (VAT/GST/TIN, etc.). So there is EU_VAT for European VAT numbers, US_TIN for US tax identification numbers, etc. But what value to use for tax-exempt organisations that don't have a tax number? Well... guess how I found out about NO_VAT...
On the bright side, I did learn that way that although Norway is in the Schengen zone, apparently they are not part of the EU (hence the separation of EU_VAT and NO_VAT). I guess the 'no' name collision has taught many developers something about Norway :-)
It would be better to delete your comment so nobody else has to has to ever have this crisis.
Why? I understand it in cases where security is critical or intellectual property is at stake. Are you talking about "snowflake runners" or just dumb executors of container images?
With self hosted Gitlab runners it was almost as fast as doing incremental builds. When your build process can take like 15-20 minutes (medium sized C++ code base), this brought down the total time to 30 seconds or so.
Imagine building Android - even "cloning the sources" is 200GB of data transfer, build times are in hours. Not having to delete the previous sources and doing an incremental build saves a lot of everything.
tldr; "A cache is one or more files a job downloads and saves. Subsequent jobs that use the same cache don’t have to download the files again, so they execute more quickly."
It will probably still be slower than a dedicated runner, but possibly require less maintenance ("pet" runner vs "cattle" runner).
Security is another thing where this can come in handy, but properly firewalling CI runners and having mirrors of all your dependencies is a lot of work and might very well be overkill for most people.
Buy a cheap Ryzen, and put it on your desk, that's a cheap runner.
So many times I was biting my fingers not being able to figure out the problems GitHub runners were having with my actions and was unable to investigate.
- I would go even further: Do not use bash/python or any duck-typed lang. (only for simple projects, but better just dont get started). - Leverage Nix (!! no its not a joke ecosystem) : devshells or/and build devcontainers out of it. - Treat tooling code, ci code, the exact same as your other code. - Maybe generate the pipeline for your YAML based CI system in code. - If you use a CI system, gitlab, circle etc, use one which does not do stupid things with your containers (like Github: 4 years! old f** up: https://github.com/actions/runner/issues/863#issuecomment-25...)
Thats why we built our own build tool which does that, or at least helps us doing the above things:
- I would go even further: Do not use bash/python or any duck-typed lang. (only for simple projects, but better just dont get started).
- Leverage Nix (!! no its not a joke ecosystem) : devshells or/and build devcontainers out of it.
- Treat tooling code, ci code, the exact same as your other code.
- Maybe generate the pipeline for your YAML based CI system in code.
- If you use a CI system, gitlab, circle etc, use one which does not do stupid things with your containers (like Github: 4 years! old f** up: https://github.com/actions/runner/issues/863#issuecomment-25...). Also one which lets you run dynamically generated pipelines.
Thats why we built our own build tool which does that, or at least helps us doing the above things:
This so much. This ties into the previous point about using as much shell as possible. Additionally I'd say environment control via Docker/Nix, as well as modularizing the pipeline so you can restart it just before the point of failure instead of rerunning the whole business just to replay one little failure.
To put the first 3 points into different words: you should treat the CI only as a tool that manages the interface and provides interaction with the outside world (including injecting secrets/configuration, setting triggers, storing caches etc.) and helps to visualize things.
Unfortunately, to do that, it puts constraints on how you can use it. Apart from that, no logic should live in the CI.
To an extent, yes. There should be one command to build, one to run tests, etc.
But in many cases, you do actually want the pipeline functionality that something like Gitlab CI offers - having multiple jobs instead of a single one has many benefits (better/shorter retry behaviour, parallelisation, manual triggers, caching, reacting to specific repository hooks, running subsets of tests depending on the changed files, secrets in env vars, artifact publishing, etc.). It's at this point that it becomes almost unavoidable to use many of the configuration features including branching statements, job dependencies etc. and that's where it gets messy.
The problem is really that you're forced to do all of that in YAML instead of an actual programming language.
Although we’re using temporal to schedule the workflows, we have a full-code typescript CI/CD setup.
We’ve been through them all starting with Jenkins ending with drone, until we realized that full-code makes it so much easier to maintain and share the work over the whole dev org.
No more yaml, code generating yaml, product quirk, groovy or DSLs!
This has been my entire strategy since I've been able to do this:
https://learn.microsoft.com/en-us/dotnet/core/deploying/#pub...
Pulling the latest from git, running "dotnet build" and sending the artifacts to zip/S3 is now much easier than setting up and managing Jenkins, et. al. You also get the benefit of having 100% of your CI/CD pipeline under source control alongside the product.
In my last professional application of this (B2B/SaaS; customer hosts on-prem), we didn't even have to write the deployment piece. All we needed to do was email the S3 zip link to the customer and they learned a quick procedure to extract it on the server each time.
My concern with this kind of deployment solution, where the customer is instructed to install software from links received in e-mails, is that someone else could very easily send them a link to a malicious installer and they would be hosed. E-mail is not authenticated (usually) and the sender can be forged.
I suppose you could use a shared OneDrive folder or something, which would be safer, as long as the customer doesn't rely on receiving the link to OneDrive by e-mail.
Docker and to some extent, however unwieldy, Kubernetes at least allow you to run anywhere, anytime without locking you into a third party.
A "pipeline language" that can run anywhere, even locally, sets up dependency services and initial data, runs tests and avoids YAML overload would be a much needed addition to software engineering.
Do not recommend this approach (of using docker for building).
It's very satisfying just compile an application with a super esoteric tool chain in docker vs the nightmares of setting it up locally (and keeping it working over time).
We used a single huge docker image with all the dependencies we needed to cross compile to all architectures. The image was around 1GB, it did its job but it was super slow on CI to pull it.
Let me at least recommend depot.dev for having absurdly fast runners.
I shared more context in this thread: https://x.com/solomonstre/status/1895671390176747682
* Consider whether it's not easier to do away with CI in the cloud and just build locally on the dev's laptop
With fast laptops and Docker you can get perfectly reproducible builds and tests locally that are infinitely easier to debug. It works for us.
I think builds must be possible locally, but i’d never rely on devs for the source of truth artifacts running in production, past a super early startup.
as always, enough decoupling is useful
> as long as it is proper, maintainable code
...in an imperative language you know well and which has a sufficient amount of type/null checking you can tolerate.
Also lol @deng
Are the Actions a little cumbersome to set up and test? Sure. Is it a little annoying to have to make somewhat-useless commits just to re-trigger an Action to see if it works? Absolutely. But once it works, I just set it and forget it. I've barely touched my workflows in ~4 years, outside of the Node version updates.
Otherwise, I'm very pleased with both. My needs must just be simple enough to not run into these more complicated issues, I guess?
GitHub CI is designed in a way which tends to work well for
- languages with no or very very cheap "compilation" steps (i.e. basically only scripting languages)
- relatively well contained project (e.g. one JS library, no mono repo stuff)
- no complex needs for integration tests
- no need for compliance enforcement stuff, especially not if it has to actually be securely enforced instead of just making it easier to comply then not to comply
- all developers having roughly the same permissions (ignore that some admin has more)
- fast CI
but the moment you step away from this it just falls more and more and more apart and I every company which doesn't fit the constraints above I have seen so far has non stop issues with GitHub Actions.
But the worst part, which maybe is where a lot of hatred comes from, is that it's there for cheap maybe even free (if you anyway pay for GitHub) and it doesn't need an additional contract, billing, etc. Not an additional vetting of 3rd party companies. Doesn't need managing your own CI service etc. So while it does cause issues non stop it also seems initially still "cheaper" solution for the company. And then when your company realizes it's not and has to setup their own GitHub runner etc. it probably isn't. But that is if you properly account dev time spend on "fixing CI issues" and even then there is the sunk cost fallacy because you already spend so much time to make github actions work and you would have to port everything over etc. Also, realistically speaking, a lot of other CI solutions are only marginally better.
I find github actions works very well for compliance. The ability to create attestations makes it easy to enforce policies about artifact provenance and integrity and was much easier to get working properly compared to my experience attempting to get jenkins to produce attestations.
https://docs.github.com/en/actions/security-for-github-actio...
https://docs.github.com/en/actions/security-for-github-actio...
What was your issue with it?
This is not true at all. It's fine with Haskell, just cache the dependencies to speed up the build...
- GitHub Action cache and build artifact handling is a complete shit show (slow upload, slow download and a lot of practical subtle annoyances, finished off with sub-par integration in existing build systems)
- GitHub runners are comparatively small, so e.g. larger linker steps can already lead to pretty bad performance penalties
and sure like I said, if you project is small it doesn't matter
Or even if you pay $$$ for big runners you can roll it onto your Azure bill rather than having to justify another SAAS service.
This is the key point. Every CI system falls apart when you get too far from the happy path that you lay out above. I don't know if there's an answer besides giving up on CI all together.
The problem isn't github actions but people overloading their build and CI system with all sorts of custom crap. You'd have a hard time doing the same thing twenty years ago with Ant and Hudson (Jenkin's before the fork after Oracle inherited that from Sun). And for the same reason. These systems simply aren't very good as a bash replacement.
If you don't know what Ant is. That was a popular build system for Java before people moved the problem to Maven and then to Gradle (without solving it). I've dealt with Maven files that were trying to do all sorts of complicated things via plugins that would have amounted to two or three lines of bash. Gradle isn't any better. Ant at least used to have simple primitives for "doing" things. But you had to spell it out in XML form.
The point of all this, is that build & CI systems should mainly do simple things like building software. They shouldn't have a lot of conditional logic, custom side effects, and wonky things that may or may not happen depending on the alignment of the moon and stars. Debugging that stuff when it fails to work really sucks.
What helps with Yaml is using Yaml generators. I've used a Kotlin one for a while. Basically, you get auto complete, syntactical sanity, type checking and if it compiles it runs. Also makes it a lot easier to discover new parameters, plugin version updates, etc.
That's supposedly CICD 101. I don't understand why people in this thread seem to be missing this basic fact and instead they vent about irrelevant things like YAML.
You set your pipeline. You provide your own scripts. If a GitHub Action saves you time, you adopt it instead of reinventing the wheel. That's it.
This whole discussion reads like the bike fall meme.
For you to make that comment, I'm not sure you ever went through any basic intro to GitHub Actions tutorial.
GitHub Actions has 'run'.
https://docs.github.com/en/actions/writing-workflows/workflo...
Now that we established that, GitHub Actions also supports custom actions, which is a way to create, share, and reuse high-level actions. Instead of copy-pasting stuff around, you do the equivalent of importing a third party module.
https://docs.github.com/en/actions/sharing-automations/creat...
Onboarding a custom GitHub Action does not prevent you from using steps.run.
I don't even know where to start regarding your comment on expression evaluation and conditions. Have you used a CICD system before?
The problem with half the comments in this thread railing against CICD in general, YAML, etc is that they clearly do not have a faintest idea about what they are doing, and are instead complaining about ther own inability.
I'm an experienced SaltStack user. If I found something I need is too complex to be described in YAML, I'll just write a custom module and/or state. Use YAML just to inform Salt what should happen, and shove the logic in the Python files.
People really should become generalists if they handle the plumbing.
I am with the author - we can do better than the status quo!
I commit code, push it, wait 45 seconds, it syncs to AWS, then all my sites periodically ping the S3 bucket for any changes, and download any new items. It's one of the most reliable pieces of my entire stack. It's comically consistent, compared to anything I try building for a mobile app or pushing to a mobile app store.
I look forward to opening my IDE to push code to the Actions for my web app, and I dread the build pipeline for a mobile app.
Well yeah because nobody is saying it isn't reliable. It's the setup stage that is painful. Once you've done it you can just leave it mostly.
I guess if your CI is very simple and always the same you are exposed to these issues less.
I would recommend looking at Fastlane[0] if you haven't already.
You notice a deprecation warning in the logs, or an email from GitHub and you make a 1 line commit to bump the node version. Easy.
Sure you can make typos that you don’t spot until you’ve pushed and the action doesn’t run, but I quickly learned to stop being lazy and actually think about what I’m writing, and get someone else to do an actual review (not just scroll down and up and give it a LGTM).
My experience is same as the commenter above, it’s relatively set and forget. A few minutes setup work for hours and hours of benefit over years of builds.
It's how it works now. It doesn't have to forever. We can imagine a future in which it works in a better way. One that isn't so annoying.
> I don't think there's some universally perfect solution that magically just works all the time and never needs intervention or updating.
Again you seem to be confused as to what the issue is. Maintenance is not painful. Initial development is.
When it takes all of a day to self host your own task runner on a laptop in your office and have better uptime, lower cost, better performance, and more correct implementations, you have to ask why anyone chooses GHA. I guess the hello-world is convincing enough for some people.
The world is full of kafkaesque nightmares of Dev-ops pipeline "designed" and maintained by committees of people.
It's horrible.
That said, for some personal stuff I have Google Cloud Build that has a very VERY simple flow. Fire, forget and It's been good.
But honestly, doesn't github now have a button you can press to retrigger actions without a commit?
GitHub Actions are least hassle, when you don't care about how much compute time you are burning through. Either because you are using the free-for-open-source repositories version, or because your company doesn't care about the cost.
If you care about the compute time you are burning, then you can configure them enough to help with that, but it quickly becomes a major hassle.
I wouldn't want to maintain GitHub actions for a large project involving 50 people and 5 languages...
I’ve noticed this phenomenon few times already, and I think there’s nothing worse than having a 30-60s feedback loop. The one that keeps you glued to the screen but otherwise is completely nonproductive.
I tried for many moons to replicate GHA environment on local and it’s impossible in my context. So every change is like „push, wait for GH to pickup, act on some stupid typo or inconsistency, rinse, repeat”.
It’s like a slot machine „just one more time and it will run”, eating away focus and time.
It took me 25 minutes to get 5s build process. Naive build with GHA? 3 minutes, because dependencies et al. Ok, let’s add caching. 10 hours fly by.
The cost of failure and focus drop is enormous.
There has to be a better way. How has nobody figured this out?
I wish GitLab/GitHub would provide a way to do this by default, though.
If the process is longer than a few minutes, I can switch tasks while I wait for it. It's waiting for those things in the 3-10 minute range that is intolerable for me: long enough I will lose focus, not long enough for me to context switch.
Now I can bullshit with the LLM about something related to the task while I wait, which helps me to stay focused on it.
Recently switched to a company using Github, and assumed I'd be blown away by their offering because of their size.
Well, I was, but not in the way I'd hoped. They're absolutely awful in comparison, and I'm beyond confused how it got to that state.
If I were running a company and had to choose between the two, I'd pick Gitlab every time just because of Github actions.
I have some GitHub actions for some side projects and it just seems so much more confusing to setup for some reason.
If you want to make a CI performant, you'll need to use some of its features like caches, parallel workers, etc. And GHA usability really fall short there.
The only reason I put up with it is that it's free for open source projects and integrated in GitHub, so it took over Travis-ci a few years ago.
> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
When the cli didnt have support for what I needed
one of them beeing echoing text to a file. to me, your comparison makes no sense.
Every time someone introduced a new way to use someone else's shared magic I feel nervous about using it. Like GitHub Actions. Perhaps it's time for me to dig into them a bit more and try to understand if/how they're safe to use. But I seem to remember just a few days ago someone mentioning a GitHub action getting hijacked?
I will be stunned if this doesn't become a more popular attack vector over the next few years. Lots of valuable stuff sits in github, and they're a nearly-wide-open hole to access it.
For instance, AWS has a lot of actions they maintain to assist with common CI/CD needs with AWS services.
// Luckily still a gitlab user, but recently forced to Microsoft Teams and office.
my condolences to you and your team for that switch; it's my 2nd used-and-disliked thing (right next to atlassian) - oh well
but one cool feature i found with ms teams that zoom did not have (some years ago - no clue now) is turning off incoming video so you dont have to be constantly distracted in meetings
edit: oh yeah, re github actions and the user that said: > Glad I’m not the only one
me too, me too; gh actions seem frustrating (from a user hardly using gh actions, and more gitlab things - even though gitlab seems pretty wonky at times, too)
Because the docs are crap perhaps? I prefer it, having used both professionally (and Jenkins, Circle, Travis), but I do think the docs are really bad. Even just the nesting of pages once you have them open, where is the bit with the actual bloody syntax reference, functions, context, etc.
A few years back I wanted to throw in the towel and write a more minimal GHA-compatible agent. I couldn't even find where in the code they were calling out to GitHub APIs (one goal was to have that first party progress UI experience). I don't know where I heard this, so big hearsay warning, but apparently nobody at GitHub can figure it out either.
It's really upsetting how little attention Actions is getting these days (<https://github.com/orgs/community/discussions/categories/act...> tells the story -- the most popular issues have gone completely unanswered).
Sad to see Earthly halting development and Dagger jumping on the AI train :(. Hopefully we'll get a proper alternative.
On a related note, if you're considering https://www.blacksmith.sh/, you really should consider https://depot.dev/. We evaluated both but went with Depot because the team is insanely smart and they've solved some pretty neat challenges. One of the cooler features is that their caching works with the default actions/cache action. There's absolutely no need to switch out popular third party actions in favor of patched ones.
Hi, Dagger CEO here. We're advertising a new use case for Dagger (running AI agents) while continuing to support the original use case (running complex builds and tests). Dagger has always been a general purpose engine, and our community has always used it for more than just CI. It's still the exact same engine, CLI, SDKs and observability stack. It's not like we're discontinuing a product, to the contrary: we're getting more workloads on the platform, which benefits all our users.
I think what we're doing is different: we built a product that was always meant to be general purpose; encouraged our community to experiment with alternative use cases; and are now doubling down on a new use case, for the same product. We are still worried about the perception of a FOMO-driven AI pivot (and the reactions on this thread confirm that we still have work to do there); but we're confident that the product really is capable of supporting both.
Thank you for the thoughtful comments, I appreciate it.
Example:
https://github.com/actions/runner/pull/2477#issuecomment-244...
What happened there?
What makes Depot so fast is that they use NVMe drives for local caching and they guarantee that the cache will always be available for the same builders. So you don't suffer from the cold-start problem or having to load your cache from slow object storage.
We also do native multi-platform builds behind one build command. So you can call depot build —platform Linux/amd64,linux/arm64 and we will build on native Intel and ARM CPUs and skip all the emulation stuff. All of that adds up to really fast image builds.
Hopefully that’s helpful!
You wouldn't really have to change anything on your dockerfile to leverage this and see significant speed up.
The docs are here: https://docs.warpbuild.com/docker-builders#usage
Do people really consider this best practice? I disagree. I absolutely don't want CI touching my code. I don't want to have to remember to rebase on top of whatever CI may or may not have done to my code. Not all linters are auto-fixable so anyway some of the time I would need to fix it from my laptop. If it's a trivial check it should run as a pre-commit hook anyway. What's next, CI should run an LLM to auto-fix failing test cases?
Do people actually prefer CI auto-fixing anything?
Doing it in CI sounds like making things more complicated by resetting to remote branches after pushing commits. And, in the worst case, something that actually brakes code that works locally.
Why do they have a say in this? This is up to tech leadership to set standards that need to be followed.
I’m also with the other commenter about settings these things at the Editor level, but also at the pre-push level.
We benchmark how long it takes to format/lint only changed files, usually no more than a second, maybe two, but I admit for some languages this may take more. An editor with a language server properly setup would have helped you find issues earlier.
We also have reports for our CI pipeline linters, so if we see more than 1 report there, we sent a message to the team: It means someone didn’t setup their editors nor their git hooks.
If the checks take more than a second, yeah, probably pre-commit is not the place/moment. Reliability is important, but so is user experience. I had companies where they ran the unit test suite at the pre-commit level, alright? And that is NOT fine. While it sounds like it’ll find issues earlier, it’ll screw your developer time if they have to wait seconds/minutes each time they fix a comma.
Because at the institutional level, there isn’t the appropriate will to mandate that devs fix their local environments, and I don’t feel like burning my own political capital on that particular fight.
Agreed on the performance comments.
I'd rather have linting pushed into the editing process, within my IDE/VS Code/vim plugins, whathaveyou, where it can feedback-loop with my actual writing process and not just be some ancillary command I run with lots of output I never read.
We have a lot of IDE checks, but they’re just warnings when debugging (because devs complained, IMO reasonably, that having them as errors during dev was too inconvenient during development/debugging). CI fails with any warnings, and we have devs who don’t bother to check their IDE warnings before committing and pushing to a PR.
Trivial mistakes in PRs are almost always signs of larger errors.
(Most of the time, the auto-fix is just running "cargo fmt".)
Things like 10 GB cache limits in GitHub, concurrency limits based on runner type, the expensive price tag for larger GitHub runners, and that's before you even get to the security ones.
Having been building Depot[0] for the past 2.5 years, I can say there are so many foot guns in GitHub Actions that you don't realize until you start seeing how folks are bending YAML workflows to their will.
We've been quite surprised by the `container` job. Namely, folks want to try to use it to create a reproducible CI sandbox for their build to happen in. But it's surprisingly difficult to work with. Permissions are wonky, Docker layer caching is slow and limited, and paths don't quite work as you thought they did.
With Depot, we've been focusing on making GitHub Actions exponentially faster and removing as many of these rough edges as possible.
We started by making Docker image builds exponentially faster, but we have now brought that architecture and performance to our own GHA runners [1]. Building up and optimizing the compute and processes around the runner to make jobs extremely fast, like making caching 2-10x faster without having to replace or use any special cache actions of ours. Our Docker image builders are right next door on dedicated compute with fast caching, making the `container` job a lot better because we can build the image quickly, and then you can use that image right from our registry in your build job.
All in all, GHA is wildly popular. But, the sentiment around even it's biggest fans is that it could be a lot better.
I guess that would be reasonable if we really needed the speedup, but if you're also offering a better QoL GHA experience then perhaps another tier for people like us who don't necessarily need the blazing speed?
We are fully usage based, no minimums etc., and our container builders are faster than others on the market.
We also have a BYOC option that gives 10x cost reduction and used by many customers at scale.
10,000,000,000 bytes should be enough for anyone! It really is a lot of bytes...
I used GitHub actions when building a fin services app, so I absolutely used the hash to specify Action dependencies.
I agree that this should be the default, or even the required, way to pull in Action dependencies, but saying "almost no one does" is a pretty lame excuse when talking about your own risk. What other people do has no bearing on your options here.
Pin to hashes when pulling in Actions - it's much, much safer
"Defaults matter" is a common phrase, but equally true is: "the pattern everyone recommends including example documentation matters".
It is fair to criticise the usage of GH Actions, just like it's fair to criticise common usage patterns of MySQL that eat your data - even if smarter individuals (who learn from deep understanding, or from being burned) can effectively make correct decisions, since the population of users are so affected and have to learn the hard way or be educated.
Yes, your builds will work as expected for a stretch of time, but that period will come to an end, eventually.
Then one day you will be forced to update those pinned dependencies and you might find yourself having to upgrade through several major versions, with breaking changes and knock-on effects to the rest of your pipelines.
Allowing rolling updates to dependencies helps keep these maintenance tasks small and manageable across the lifetime of the software.
Just make sure you don’t leak secrets to your PRs. Also I usually review changes in updated actions before merging them. It doesn’t take that much time, so far I’ve been perfectly fine with doing that.
[1]: https://docs.renovatebot.com/modules/manager/github-actions/...
Though, yes, I prefer pinning dependencies for my personal projects. I don't see why things should break when I explicitly keep them the same.
The real problem is security vulnerabilities in these pinned dependencies. You end up making a choice between:
1. Pin and risk a malicious update.
2. Don't pin and have your dependencies get out of date and grow known security vulnerabilities.
This is for composite actions. For JS actions what if they don't lock dependencies but pull whatever newest package at action setup time? Same issue.
Would have to transitively fork everything and pin it myself, and then keep it updated.
As for reducing boilerplate in the CI configs, GitHub Actions is a programming language with support for functions! It's just that function calls can only appear in very limited places in the program (only inside `steps`), and to define a function, you have to create a Git repository. The function call syntax is also a bit unusual, it's written with the `uses` keyword. So there is a lot of boilerplate that you can't remove this way, though there are several other yaml eDSLs hidden in GitHub Actions that address some points of it. E.g. you can create loops with `matrix`, but again, not general-purpose loops, they can only appear in a very specific syntactic location.
To really duplicate stuff, rather than copy-pasting blocks of yaml, without using a mix of these special yaml eDSLs, in the past I've used Nix and Python to generate json. Now I'm using RCL for this (https://rcl-lang.org). All of them are general-purpose yaml deduplicators, where you can put loops or function calls anywhere you want.
FYI there is also `on: workflow_call` which you can use to define reusable jobs. You don't have to create a new repository for these
https://docs.github.com/en/actions/writing-workflows/workflo...
In my experience, the best strategy is to minimize your use of it — call out to binaries or shell scripts and minimize your dependence on any of the GHA world. Makes it easier to test locally too.
I have done something similar with Jenkins and groovy CI library used by Jenkins pipeline. But it wasn't super simple since a lot of it assumed Jenkins. I wonder if there is a more cleaner open source option that doesn't assume any underlying platform.
Like teams.
GitHub Actions is the worst possible CI platform - except for all the others. Every single CI platform has weird limitations, missing features, gotchas, footguns, pain points. Every single one requires workarounds, leaves you tearing your hair out, banging the table trying to figure out how to do something that should be simple.
Of all of them I've tried, Drone is the platonic ideal of the best, simplest, most generally useful system. It is limited. But that limitation is usually easy to work around and doesn't impose artificial constrictions. However, you won't find nearly as many canned solutions or plugins as GitHub Marketplace, and the enterprise features are few.
GHA is great because of things like Dependabot, and the million canned Marketplace actions, and it's all tightly integrated with GH's features, so you don't have to work hard to get anything advanced or specific to work. Tight integration can save you weeks to months of development time on a CI solution. I've literally seen teams throw out versioning of dependencies entirely because they weren't updating their dependencies, because there's no Dependabot orb for CircleCI. If they had just been on GHA using Dependabot it would have saved them literal years of headaches.
Jenkins is, ironically, both the most full-featured, and the absolute worst to configure/maintain. Worst design, worst security, worst everything... except it does have a plugin for everything, and a UI for everything. I hate it with the fire of a million suns. But people won't stop using it, partially because it's so goddamn configurable, and they learned it years ago and won't stop using it. If anyone wants to write a replacement, I'm happy to help (I even wrote a design doc!).
Anyone who claims that GHA is garbage and any of the others are amazing is either doing something very basic or is crazy, or lying.
At the end of the day, you run shell scripts and commands using a YAML based config language (except for Jenkins). Amazingly, it's hard to build something that does that with the right abstractions and compromises between flexibility and good hygiene.
That may have been true before GitHub decided that PRs can't access repository secrets anymore. Apparently now you can at least add these secrets to Dependabot too (which is still duplicate effort for setup and any time you rotate secrets), but at the time when the change was introduced there were only weird workarounds.
I'm surprised nobody has mentioned dependabot yet. It automates this, keeping action dependencies pinned by hash automatically whilst also bringing in stable upgrades.
The only automation that I know of is cargo vet. Although it doesn’t work for GitHub Actions, the idea sounds useful. Basically, vet allows people who trust each other to vet updates. So one person verifies the diff and then approves the changes. Next, everyone who trusts this person can update the dependency automatically since it has been “vetted”.
We also, to your point, need more labels than @latest. Most of the time I want to wait a few days before taking latest, and if there have been more updates since that version, I probably don't want to touch anything for a little bit.
Common reason for 2 releases in 2 days: version 1 has a terrible bug in it that version 2 tries to fix. But we won't be certain about that one either until it's been a few more days with no patch for the patch for the patch.
For autoscaling we use terraform-aws-github-runner which will bring up ephemeral AWS machines if there are CI jobs queued on GitHub. Machines are then destroyed after 15 minutes of inactivity so they are always fresh and clean.
For defining build pipelines we use Nix. It is used both for building various components (C++, Go, JS, etc) as well as for running tests. This helps to make sure that any developer on the team can do exactly the same thing that the CI is doing. It also utilizes caching on an S3 bucket so components that don't change between PRs don't get rebuilt and re-tested.
It was a bit of a pain to set up (and occasionally a pain to maintain), but overall it's worth it.
Next to json I also used travisCI and appveyor for projects. And they all had the same (commit and pray) setup that ai hate. I wish if „act“ was a tool directly maintained by the GitHub folks though.
My experience is this works for simple scripts but immediately falls apart when you start to do things like “don’t run the entire battery of integration tests against a readme change”, or “run two builds in parallel”, or “separate the test step from the build and parallelise it even if the build is serial”.
It’s easy to wrap make build and go about your life, but that’s no easier than just using the GitHub action to call go build or mvn build. T
he complexity comes in “pull that dependency from this place that is in a private repository on GitHub/AWS because it’s 100x faster than doing it from its source”, and managing the credentials etc for all of that stuff. This is also where the “it differs from running locally” comes into it too, funnily enough.
I'm so much happier on projects where I can use the non-declarative Jenkins pipelines instead of GH Actions or BB pipelines.
These YAML pipelines are bad enough on their own, but throw in a department that is gatekeeping them and use runners as powerful as my Raspberry Pi and you have a situation where a lot of developers just give up and run things locally instead of the CI.
I think there's a place for making a builder that looks imperative, but can work out a tree of actions and run them. Gulp is a little bit this way, but again I haven't tried to breakpoint through it either.
If the next evolution in DevEx is not caring about what your code looks like in a stepping debugger, then the one after it will be. Making libraries that present a tight demo app for the Readme.md file and then are impossible to do anything tricky with or god forbid debug just needs to fucking stop. Yesterday. And declarative systems are almost always the worst.
It looks like they have a very specific and unique build process which they really should handle with something more customizable like Jenkins. Instead they're using something that's really intended for quick and light deployments for intense dev ops setup.
I really like GitHub actions, but I'm only doing very simple things. Don't call a fork bad because it's not great when you're eating soup
They either just write a long blog post about how they can't screw in nails with a hammer.
Or they leave their security rules wide open and about half the comments are like, we need tools which stop us from doing stupid things.
No other industry works like this.
What it really does is fire off a WebHook. Repository custom properties and the name of the action are properties that are included in the workflow_job webook. With this you can do anything you want and you're not at all constrained by YAML or runners.
If they had, we'd be reading a different article about how terribly complex and unintuitive Jenkins is.
CI is just a very very hard problem and no provider makes it easy.
[1] https://github.com/actions/runner/blob/6654f6b3ded8463331fb0...
I had a coworker who called it Visual Sorta-Safe which is just about the best parody name I've ever heard in my entire career.
If one action pushes a tag to the repo, `on:tag` does not trigger. The workaround apparently is to make the first action push the tag using a custom SSH key, which magically has the ability to trigger `on:tag`.
The workaround is to use a token tied to you instead of GitHub Actions, so you get charged (or run out of quota).
You get charged no matter what, a personal access token doesn’t change anything.
If they are concerned about infinite loops then put a limit on how many workflows can be triggered but another workflow. Each time a workflow chains off another pass along some meta data of “runsDeep” and stop when that hits X, which can be configured.
No, requiring a PAT to kick off a workflow from a workflow is gross and makes zero sense. I don’t want every tag associated with my user, I want it to be generic, the repo itself should be attributed. The only way to solve this is to create (and pay for) another GH user that you create PAT tokens under. A bunch of overhead, cost, and complexity for no good reason.
> When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN, with the exception of `workflow_dispatch` and `repository_dispatch`, will not create a new workflow run.
It has bitten me in the rear before too. I use this pattern a lot when I publish a new version, which tags a piece of code and then marks assets as part of that version (for provenance reasons I cannot rebuild code).
I'm using GitHub Actions to easily reuse some predefined job setup (like installing a certain Python version on Linux, macOS, Windows runners). For these tyoe of tasks, I find GitHub actions very useful and convenient. If you want to reuse predefined jobs, written by someone else, with GitLab CI/CD, what can I use?
Since the integration is done statically, it means gitlab can provide you a view of the pipeline script _after_ all components were included, but without actually running it.
We are using this and it is so nice to set up. I have a lot of gripes with other gitlab features (e.g. environments, esp. protected ones and their package registry) but this is one they nailed so far.
1: although TIL that the "script:" field itself is actually required in GLCI https://docs.gitlab.com/ci/yaml/#script
There is a gitlab CI feature `include`, but you pretty much have to write shell scripts inside YAML, losing on whole developer experience (shellcheck etc..). I would recommend this way only if you can't factor your code into a CLI in proper language.
* Barely reproducible because things like the settings of the server (environment variables are just one example) are not version controlled.
* Security is a joke.
* Programming in YAML or any other config format is almost always a mistake.
* Separate jobs often run in their own container, losing state like build caches and downloaded dependencies. Need to be brought back by adding remote caches again.
* Massive waste of resources because too many jobs install dependencies again and again or run even if not necessary. Getting the running conditions for each step right is a pain.
* The above points make everything slow as hell. Spawning jobs takes forever sometimes.
* Bonus points if everything is locked down and requires creating tickets.
* Costs for infra often keep expanding towards infinity.
We already have perfectly fine runners: the machines of the devs. Make your project testable and buildable by everyone locally. Keep it simple and avoid (brittle) dependencies. A build.sh/test.sh/release.sh (or in another programming language once it gets more complicated, see Bun.build, build.zig) and a simple docker-compose.yml that runs your DB, Pub-Sub or whatever. Works pretty well in languages like Go, Rust or TS (Bun). Having results in seconds even if you are offline or the company network/servers have issues is a blessing for development.
There are still things like the mentioned heavy integration tests, merges to main and the release cycle where it makes sense to run it in such environments. I'm just not happy how this CI/CD environments work and are used currently.
- env vars can be scripted, either in YAML or through dotenv files. Dotenv files would also be portable to dev machines
- how is security a joke? Do you mean secrets management? Otherwise, i don't see a big issue when using private runners with containers
- jobs can pass artifacts to each other. When multiple jobs are closely interwined, one could merge them?
- what dependency installation do you mean? You can use prebuilt images with dependencies for one. And ideally, you build once in a pipeline and use the binary as an artifact in other jobs?
- in my experience, starting containers is not that slow with a moderately sized runner (4-8 cpus). If anything, network latency plays a role
- not being able to modify pipelines and check runners must be annoying, I agree
- everything from on-prem license to SaaS license keeps costing more. Somewhere, expenses are made, but that can be optimized if you are in a position to have a say?
By comparing dev machines to runners, you miss one important aspect: portability, automation and testing in different environments. Except when you have a full container engine on your dev machine with flexible network configs, there can be missed issues. Also, you need to prime every dev to run the CI manually or work with hooks, and then you can have funny, machine-specific problems. So this already points to a central CI-system by making builds repeatable and in the same from-scratch envirnment. As for deployment, those shouldn't be made from dev machines, so automated pipelines are the go-to here. Also autmated test reporting goes out the window for dev machines.
Env vars can be scripted, many companies use a tree of instance/group/project scoped vars though, leading to easily breaking some projects when things higher up change. Solvable for sure, guidelines in companies make it a pain. There are other settings like allowed branch names etc. that can break things.
With security, yes I mean mostly secrets management. Essentially everyone who can push to any branch has access to every token. Or just having a typo or mixing up some variables lead to stuff being pushed to production. Running things in the public cloud is another issue.
Passing artifacts between jobs is a possibility. Still leads to data pushed between machines. Merging jobs is also possible, just defeats the purpose of having multiple jobs and stages. The examples often show a separation between things like linting, testing, building, uploading, etc. so people split it up.
With dependencies I mean everything you need to execute jobs. OS, libraries, tools like curl, npm, poetry, jfrog-cli, whatever. Prebuilt images work, but it is another thing you have to do yourself. Building more containers, storing them, downloading them. Also containers are not composable, so for each project or job has its own. The curse of being stateless and the way Docker works.
Starting containers is not slow on a good runner. But I noticed significant delays on many Kubernetes clusters, even if the nodes are <1% CPU. Startup times of >30s are common. Still, even if it would be faster it is still a delay that quickly adds up if you have many jobs in a pipeline.
I agree that dev machines and runners have different behavior and properties. What I mean is local-first development. For most tasks it is totally fine to run a different version of Postgres, Redis and Go for example. Docker containers bring it even closer to a realistic setup. What I want is quick feedback and being able to see the state of something when there a bugs. Not needing to do print debugging via git push and waiting for pipelines. Pipelines that setup a fresh environment and tear it down after are nice for reproducibility, but prevent me to inspect the system aside from logs and other artifacts. Certainly this doesn't mean you shouldn't have a CI/CD environment at all, especially for releases/production deployments.
Simple scripts like these are enough for most projects and it is a blessing if you can execute them locally. Having a CI platform doing it automatically on push/merge/schedule is still possible and makes migrations to other platforms easier.
For real tho, not every project can be build by everyone locally, but at least parts of it should be locally runnable for devs to be able work (at all IMO). What I am noticing is more and more coding is being done on some server somewhere Github Codespaces anyone? Google Colab? etc.
What I am also noticing is that this tools like GH-A there is not really a way to test the CI code other than.. commit, push, wait, commit, push, wait... That's just absurd to me. Obviously all CIs have some quirks that sometimes you have _just run it_ and see if it works but this... it's like that for everything! Abusrd I say!
Laptops are a lot cheaper then the cloud bills I have seen so far. Penny pinching every tiny thing for <100$/€, but cloud seems to run on an infinite magic budget...
Your opinions are so regressive you really should consider going into management.
Nowhere do I say you shouldn't use CI/CD at all. I just don't like the current CI/CD implementations and the environments/workflows companies I worked for so far provide on top of them.
The regressive thing is putting everything ONLY on a remote machine with limited access and control, taped together by a quirky YAML-based DSL as a programming language and still requiring me to program most stuff myself.
$ git l * cbe9658 8 weeks ago rejschaap (HEAD -> add-ci-cd) Update deploy.yml * 0d78a6e 8 weeks ago rejschaap Update deploy.yml * e223056 8 weeks ago rejschaap Update deploy.yml * 8e1e5ea 8 weeks ago rejschaap Update deploy.yml * 459b8ea 8 weeks ago rejschaap Update deploy.yml * a104e80 8 weeks ago rejschaap Update deploy.yml * 0e11d40 8 weeks ago rejschaap Update deploy.yml * 727c1d3 8 weeks ago rejschaap Create deploy.yml
$ git l
* cbe9658 8 weeks ago rejschaap (HEAD -> add-ci-cd) Update deploy.yml
* 0d78a6e 8 weeks ago rejschaap Update deploy.yml
* e223056 8 weeks ago rejschaap Update deploy.yml
* 8e1e5ea 8 weeks ago rejschaap Update deploy.yml
* 459b8ea 8 weeks ago rejschaap Update deploy.yml
* a104e80 8 weeks ago rejschaap Update deploy.yml
* 0e11d40 8 weeks ago rejschaap Update deploy.yml
* 727c1d3 8 weeks ago rejschaap Create deploy.ymlThis also reduces the lock in by orders of magnitude.
I wonder why they chose to move back to Github Actions rather than evaluate something like Buildkite? At least they didn't choose Cloud Build.
I think incremental progress in the CI front is chugging along nicely, and I really haven't seen any breathtaking improvements from other solutions I've tried, like CircleCI.
Totally agree with other comments saying to keep as much logic out of CI config as possible. Many CI features are too convoluted for their own good. Keep it really simple. You can use any CI platform and have a shit time if you don't take the right approach.
Instead, have tooling to do that before committing (vscode format-on-save, or manually run a task), then have a pre-commit hook just do a sanity-check on that. It only needs to check modified files, so usually very fast.
Then, have an additional check on CI to verify formatting on all files. That should rarely be triggered, but helps to catch cases where the hooks were not run, for example from external contributes. That also makes it completely fine for this CI step to take a couple of minutes - you don't need that feedback immediately.
Though it would be sort of interesting or maybe just amusing if you made something like ssh-agent but for 'git commit' and your test runner. Only allow commits when all files are older than your last green test run.
You have no idea how much I'd love that feature. Inasmuch as "save" is still a thing anyway. I don't miss explicit saves in IDEA, I see commit as the "real" save operation now, and I don't mind being able to hook that in an IDE-independent way.
I think the UX of git hooks has been sub-par for sure, but tools like the confusingly named pre-commit are helping there.
I don’t want to think about formatting, I just want everything to be consistent. A pre commit hook can run those tools for me, and if any changes occurred, it can add them to the commit.
If people want to die on a hill that is demonstrably causing problems for all of their coworkers then let em.
Example config: https://github.com/anttiharju/vmatch/blob/9e64b0636601c236a5...
The article is what you end up finding after that stage has been gone through.
The conclusion of course is that whoever invented this stuff really wasn't thinking clearly and certainly didn't have the time to write decent documentation to explain what they were thinking. And now the whole world has to try to deal with their mess.
My theory as to how this ends up happening is that the people creating the thing began with some precursor thing as their model. They made the new thing as "old thing, with a few issues fixed". Except they didn't fully understand the concepts in that thing, and we never got to see that thing. You'll see many projects that have this form: bun is "yarn fixed". Yarn is "npm fixed". And so on. None of these projects ever has to fully articulate their concepts.
I don't use sourcehut, but interpreting what you wrote I'd argue this is an antifeature and would be a dealbreaker for me. CI typically evolves with the underlying code and decoupling that from the code makes it difficult to go backwards. It loses cohesion.
If you put the build files in a .builds/ folder at the root of your repository, they will be run upon each commit. Just like in github or gitlab. You are just not forced into this way of life.
If you prefer, you can store the build files separately, and run them independently of your commits. Moreover, the build files don't need to be associated to any repository, inside or outside sourcehut.
https://news.ycombinator.com/item?id=18983586
https://rewiring.bearblog.dev/blog/?q=azure
PS: I am not the author of any of these posts.
The worst kind of downtime is when you go down but nobody else has.
on:
issues:
types:
- opened
pull_request:
types:
- opened
permissions:
contents: read
issues: write
pull-requests: write
jobs:
default:
runs-on: ubuntu-latest
steps:
- run: gh issue edit ${{ github.event.issue.number }} --add-assignee ${{ github.repository_owner }}
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
Documentation for GitHub Actions says, "If you specify the access for any of these permissions, all of those that are not specified are set to none." The article says "I do think a better "default" would be to start with no privileges and require the user to add whatever is needed", and it would seem that this is already the case if you explicitly add a "permissions" command into your GitHub Actions file. So, it would seem that the "default permissions" are only used if you do not add the "permissions" command, although maybe that is not what it means and the documentation is confusing; if so, then it should be corrected. Anyways, you can also change the default permission setting to restrictive or permissive (and probably ought to be restrictive by default).
Allowing to set finer permissions probably would also help.
Stop rebasing.
This should only happen if absolutely necessary to fix major merge mistakes.
Rebasing changes history and I've seen more problems prevented from removing it as a CI strategy.
Every CI strategy I've seen relying on rebasing had a better alternative in SDLC. You just need to level up your project management, period.
It also has the problem of not having a local dev runner for actions. The "inner loop" is atrociously slow and involves spamming your colleagues with "build failed" about a thousand times, whether you like it or not.
IMHO, a future DevOps runner system must be an open-source, local-first. Anything else is madness.
Right now we're in the "mainframe era" of DevOps, where we edit text files in baroque formats with virtually no tooling assistance, "submit" that to a proprietary batch system on a remote server that puts it into a queue... then come back after our coffee to read through the log printout.
I should buy a dot matrix printer to really immerse myself into the paradigm.
The entire code is a wonderful mess. We found that when we early-adopted ephemeral runners, that the control flow is full of races and the status code you get at the end is indicative of exactly nothing. So even if the backend is just having a hickup picking up a job with an obscure Azure error code, you better just throw that entire VM away, because you can't know if that runner will ever recover or has already done things to break the next run.
Although, I never saw a public announcement of this discontinuation, ADO is kind of abandoned AFAICT and even their landing page hints to use GitHub Enterprise instead [1].
It turns out, the last person to change cron __schedule__ (not the workflow file in general) is an 'actor' associated with this workflow. Very, very confusing implementation. Error messages are even more confusing - workflow runs are renamed as "{Unknown event}" and the message is "Email is unverified".
Link to docs: https://docs.github.com/en/actions/writing-workflows/choosin...
IIRC, GitHub recommends this practice in their docs, with a username of "YOUR_USERNAME-machine".
The machine user is just an ordinary GitHub user, added as a member of the organization, with all the necessary repo permissions, and a generated access token added to the GH repo Secrets. The organization owner then manages this GH machine account as well as the org, and their own personal (or work) login account.
We have not hit any rate limiting so far, but we're a relatively small team -- a dozen devs, a few hundred commits per day that trigger CI (we don't do CD), across half a dozen active repos.
Maybe it was just the pain of switching but that was my initial impression.
I pair it with bash scripts where I find important to run outside GitHub facilitating test and maintenance.
Although, I still find need to run multiple times or iterate over the actual GitHub action run time, which is a bit slow to iterate but find it best to catch any issues. If the dev feedback loop fixed it save me a lot of precious time. I know there’s a third party to run locally but it’s not the same…
Thus, bash scripting is great due to portability.
I take this a step further and approach CI with the mentality that I should be able to run all of my CI jobs locally with a decent interface (i.e. not by running 10 steps in a row), and then I use CI to automate my workflow (or scale it, as the case may be). But it always starts with being able to run a given task locally and then building CI on top of it, not building it in CI in the first place.
Nothing beats having a single script to bootstrap and run the whole pipeline e.g. `make ci`.
This means that for anything that needs to gracefully cancel, like for example terraform, it's screwed.
Want to cancel a run? Maybe you've got a plan being generated for every commit on a branch, but you push an update. Should be ok for GitHub to stop the previous run and run the action for the updated code, right? WRONG! That's a quick way to a broken state.
This is so frustrating. Having to inject a PAT into the workflow just so it will kick off another workflow is not only annoying but it just feels wrong. Also not lots of operations are tied to my user which I don't like.
> It doesn't help that you can't really try any of this locally (I know of [act](https://github.com/nektos/act) but it only supports a small subset of the things you're trying to do in CI).
This is the biggest issue with GH Actions (and most CIs), testing your flows locally is hard if not impossible
All that said I think I prefer GH Actions over everything else I've used (Jenkins and GitLab), it just still has major shortcomings.
I highly recommend you use custom runners. The speed increase and cost savings are significant. I use WarpBuild [0] and have been very happy with them. I always look at alternatives when they are mentioned but I don't think I've found another service that provides macOS runners.
We have a very complicated build process in my current project, but our CI pipelines are actually just a couple of hundred of lines of GHA yaml. Most of which are boilerplate or doing stuff like posting PR comments. The actual logic is in NX configuration.
Outside of locking down edit access to the .github workflow yml files I'm not sure how vulnerabilities like this can be prevented.
Presumably anything configured via a .github workflow wouldn't assure safety, as those files can be edited to trigger unexpected actions like deploys on working branches. Our Github Action workflow yml file had a check to only deploy for changes to the main branch. The deploy got triggered because that check got removed from the workflow file in a commit on a working branch.
You create an environment, restrict it to the main branch, add your secret to it and then tie your deploy workflow to it.
If someone runs that workflow against another branch it will run but it won’t be able to access those secrets.
[0] https://docs.github.com/en/actions/managing-workflow-runs-an...
But for actually good security CI and CD should be different tools.
The problem is it's still possible to work around those controls unless you create some YAML monstrosity that stops people from making the mistake in the first place.
- Self-hosting on your aws/gcp/azure account can get a little tricky. `actions-runner-controller` is nice but runs your workflows within a docker container in k8s, which leads to complex handling for isolation, cost controls because of NAT etc.
- Multi-arch container builds require emulation and can be extremely slow by default.
- The cache limits are absurd.
- The macos runners are slow and overpriced (arguably, most of their runners are).
Over the last year, we spent a good amount of time solving many of these issues with WarpBuild[1]. Having unlimited cache sizes, remote multi-arch docker builders with automatic caching, and ability to self-host runners in your aws/gcp/azure account are valuable to minimize cost and optimize performance.
[1] https://github.com/earthly/earthly/commit/6d7f6786ad9fa4392f... [2] https://github.com/earthly/earthly/commit/89d31fc014a8980a50...
I am really really hoping that someone (not me, I've already tried and failed) could slim it down into a single-purpose, self-contained, community maintainable tool ...
- name: Install Earthly
if: steps.check_changes.outputs.relevant_changes == 'true'
uses: earthly/actions-setup@v1
with:
version: v${{ env.EARTHLY }}
- name: Run tests and generate coverage summary
if: steps.check_changes.outputs.relevant_changes == 'true'
run: cd src/webapp && earthly --build-arg GO_VERSION=${{ env.GOLANG }} +coverage-summaryWhat a cool looking website. What kind of tool do you need to create those animations?
Go look at your workflows and see how much of the runtime is spent running installers upon installers for various languages, package managers and so on. Containers were not supposed to be like this.
Why not align these tools? Then there might be less pain. What a good idea.
Eventually we tried dropping that requirement and instead relied on testing main before deploying to production. It sped us up again, and main never broke because of bad merges while I was there.
Like othes have suggested, keep the actions simple by having lots of scripts which you can iterate on locally and making the actions dump to just run the scripts
None of it usefully explains how GHA works from the ground up, in a way that would help me solve problems I encounter.
https://github.com/StefMa/pkl-gha
It could save you already some time.
We moved to dagger to get replicable local pipeline runs, escape the gitlab DSL, and get the enormous benefits of caching.
We have explicitly chosen to avoid using the "daggerverse", and with that the cross-language stuff. Reason being that it makes modifying our pipeline slower and harder -- the opposite of the reason we moved to dagger.
So we use the Dagger python API to define and run our CI builds. It's great!
Like the other comments on this page about dagger, the move to "integrate AI" is highly concerning. I am hopeful that they won't continue down this path, but clearly the AI hype bubble is strong and at least some of the dagger team are inside it.
I'm speculating that if the dagger team doesn't drop the AI stuff, then the dagger project will end. A fork will pop-up and we'll move to using that. Not an expert (yet!) in the buildkit API, but it seems like the stuff we're benefiting from with dagger is really just a thin wrapper around buildkit. So potentially not too challenging to create a drop-in replacement if necessary later.
But, there are so many red flags in this post. Clearly this corp does not know how to build, test and release professional software.
I hope Actions stays bad tho. We need more folks to get off proprietary code forges for their open source projects—& a better CI + a better review model (PRs are awful) are 2 very low-hanging fruit that would entice folks off of the platform not for the philosophical reasons such as not supporting US corporations or endangering contributor privacy by making them agree to Microsoft’s ToS, but for technical superiority on the platform itself.
you just activate some probes and write SQL queries to sift through the information.
But two major pains I did not see : the atrocious UI and the pricing.
Their pricing model goes against any good practice, as it counts a minute for any job even if it runs for 2 seconds. Let's say you run 100jobs in parallel and they all take 30sec. You will pay 100 minutes instead of 50. Now translate this to an enterprise operating at a big scale and I assure you have seen crazy differences between actual time and billable time.
Most of the time, I'm just running dotnet build to a zip file and s3 bucket. Then, some code or script picks it up on the other side. Things get much trickier when you're using multiple services, languages, runtimes, database technologies, etc.
There are better solutions out there.
I was doing things more than 20 years ago in Hudson that GHA can't do now.
A 1000% yes, because it means the default experice most devs have of CI is using ephemeral runners which is a massive win for security and build rot.
Every company I've worked at with stateful runners was a security incident begging to happen, not to mention builds that would do different things depending on what runner host you got placed on (devs manually installing different versions of things on hosts, etc)
And what are those?
There's also the Woodpecker CI fork, which has a very similar user experience: https://woodpecker-ci.org/
When combined with Docker images, it's quite pleasant to use - you define what environment you want for the CI/CD steps, what configuration/secrets you need and define the steps (which can also just be a collection of scripts that you can run locally if need be), that's it.
Standalone, so you can integrate it with Gogs, Gitea or similar solutions for source control and perhaps a bit simpler than GitLab CI (which I also think is lovely, though maintaining on-prem GitLab isn't quite a nice experience all the time, not that you have to do that).
Could we not usually say most software could be documented better. I do not think GitHub Actions ranks near the bottom in terms of documentation and user experience overall. I do understand your point though.