r/cicd • u/Apprehensive_Air5910 • Dec 04 '25
What’s one CI/CD mistake you keep seeing teams repeat?
As someone who is just building his team's pipelines, share your experience with me and help me avoid some common pain
r/cicd • u/Apprehensive_Air5910 • Dec 04 '25
As someone who is just building his team's pipelines, share your experience with me and help me avoid some common pain
r/cicd • u/PuddingGeneral4250 • Sep 01 '26
I was recently brought into a company to help them implement CI/CD.
I noticed that they had a couple of projects where they would branch for each release (which are deployed every 3 months), and have projects shared by teams who are on a branch each, who have to come together at the end of each sprint to mob together to fix merge conflicts.
I didn’t mind helping them out fixing this, going to trunk based and implementing feature flagging, but it turned out that’s not what they meant by implementing CI/CD, they meant they wanted help setting up automated testing on gitlab.
To me, CI/CD as a methodology means trunk based branching and feature flagging, but it’s a little difficult to find good sources on this since it seems everywhere it means devops pipeline.
Are there good sources anyone can share to give the definition, or has this changed now? E.g, CI at least was its own thing since the 90s, but it’s hard to find a definition of CI/CD describing it as a workflow.
I also don’t really mind changing definitions of things if it avoids confusion etc
r/cicd • u/LevelRelationship732 • Sep 29 '25

Hello everyone,
I've been working on a brief series of articles about orchestration techniques for releases. I figured I'd post it here in case it helps anyone.
The goal of the series is to provide a useful summary of various methods and strategies for planning releases in contemporary development settings.
If you have any thoughts or experiences with release orchestration, please share them with us!
r/cicd • u/nrlnathan1 • Aug 08 '23
I've been using StrongDM for managing access in our system, and for the most part, it's been pretty smooth sailing. Every now and then, though, we hit a snag when their proxy servers go down. Earlier this week, the dev team was unable to work for a good half hour because of this, which isn’t that much in the grand scheme of things, but they were on my case. Thankfully, downtime is pretty rare, and to be fair, it could have been due to a configuration issue on our side, but when this happens, it’s super stressful.
StrongDM has been good to me for the most part, but this was a real eye-opener. Anyone else had this issue? How did you deal with it?
r/cicd • u/xescugc • Jun 02 '26
Been building a self-hosted CI/CD called PikoCI. Started because I needed custom environments for my own projects that GitHub Actions couldn't provide, and everything self-hosted I found was either too complex to deploy or too opinionated about infrastructure.
The core idea: start with a binary and a pipeline file, nothing else. Add SQLite when you want persistence. Add Postgres and distributed workers when you scale. The tool never changes.
Key things:
- Single binary, in-memory by default, no external dependencies to start
- HCL pipelines: Terraform-style syntax, not YAML
- Run jobs locally — `pikoci run -p pipeline.hcl -j test`, no server needed
- Services: ephemeral processes (Postgres, Redis, anything) that start before tasks and stop after, guaranteed. No Docker-in-Docker.
- Five sourceable abstractions: resource types, runners, service types, secret backends, and notification types. All defined in HCL, all pullable from a URL.
- Grows with you: start in memory, add SQLite, add Postgres and distributed workers at scale. The pipeline config never changes.
- Public pipelines: share build status without an account
- Prometheus metrics out of the box
PikoCI deploys itself. Live at ci.pikoci.com/teams/main/pipelines/pikoci, no login needed.
GitHub: https://github.com/pikoci/pikoci
Docs: https://docs.pikoci.com
r/cicd • u/Wash-Fair • Nov 25 '25
Building a pipeline from scratch, and could really use some real-world input. Which tools have given you the least pain and the most reliability? Open to any stack you’ve had a genuinely good experience with.
r/cicd • u/Consistent_Serve9 • Dec 16 '25
My deployment pipelines do the basic stuff. Unit tests, build a docker image, deploy on kubernetes. Sometimes we have additionnal checks before integration in the main branch.
I'm wondering; What is something you are really proud to have added to your pipeline? One extra step that you show people or other teams and say; yeah, we do that! Isn't it great? Let's get inspiration and flex a little!
r/cicd • u/the-tech-tadpole • Dec 12 '25
I’ve been trying to make our CI/CD pipelines better across a few projects. Most discussions focus on build time, deploy frequency, or failure rate, but we’ve found a few less obvious metrics that turned out to be really useful.
How often tests fail randomly versus failing for real reasons
How often we reuse existing build artifacts instead of rebuilding everything
I’m curious - what’s a CI/CD metric you track that doesn’t get talked about much, but has actually helped your team? How do you measure it, and what did it change for you?
r/cicd • u/kamilchm • Jul 10 '25
Hey r/cicd,
I've been building and managing CI/CD pipelines for a long time, and I've seen countless teams struggle with the same architectural issues: a patchwork of CI/CD tools that don't integrate well, inconsistent workflows, and a general lack of a unified strategy that leads to reinventing the wheel.
To bring some order to the chaos, I've started formalizing my own methodology, which I call the "CI/CD Pipeline Architecture Framework." I wanted to share the core concepts here to get your thoughts and feedback.
It's built on two main ideas:
1. The Golden Path: This is the non-negotiable, 6-step foundation that every solid pipeline needs. It's the core workflow: commit → build → test → staging → production → monitoring
2. The 7 Pipeline Pillars: These are the strategic capabilities you can build on top of the Golden Path. They aren't sequential; you implement them based on your team's biggest pain points.
Here are the pillars:
Multiple Environments & Promotion: Beyond just staging and prod. How do you handle dev, qa, uat?
Progressive Delivery Strategies: Decoupling deployment from release to reduce risk, using techniques like canary releases, blue-green deployments, and feature flags.
Metrics & Observability: The foundation for safe progressive delivery. This pillar moves beyond simple pass/fail to answer critical questions: Are our builds getting slower? How much developer time is wasted on flaky tests vs. real bugs? Can we see the performance impact of a new release by grouping metrics by version?
Advanced Testing Strategies: Going beyond basic unit/integration tests (e.g., contract testing, mutation testing).
Pipeline Control & Orchestration: Giving developers safe, self-service control over their pipelines.
Multi-Platform & Multi-Cloud Support: Building pipelines that aren't locked into a single vendor.
Access Control & Security Architecture: Integrating security into every step of the pipeline (DevSecOps).
I'm particularly interested in which of these pillars you've found most challenging or rewarding to implement. In my experience as a Platform Engineer, getting Metrics & Observability right is a total game-changer. It's crucial for having the confidence that changes to the pipeline won't break anything.
What are your experiences? Does this framework resonate with the challenges you face?
r/cicd • u/todaywasawesome • Jan 09 '23
Here's to a great 2023 🥂
r/cicd • u/QuoteForward5477 • Sep 23 '26
Jenkins is not upto the mark these dayss. Firstly you need to add some 250 plugins for it to effectively work and when you actually do it... there are plugins that breaks the update...
It's technical debt with a web interface.
I've seen teams spend more time maintaining their Jenkins instance than their actual product. Pipeline scripts that require a PhD to understand.
Meanwhile, their competitors are shipping features daily with GitHub Actions or GitLab CI.
Your CI/CD tool should be invisible. If you're spending more than 5% of your time maintaining it, you chose wrong.
Simple tools, simple problems. Complex tools, complex nightmares.
r/cicd • u/Ill_Faithlessness245 • Dec 16 '25
Hey Techies,
We’re running:
Terraform for IaC
Kubernetes for workloads
GitHub Actions for CI
GitOps for delivery (cluster state reconciled from git)
My biggest question is about testing—specifically for platform add-ons like:
cert-manager
external-dns
ingress controller / gateway
external-secrets / sealed-secrets
storage drivers / CSI bits
monitoring stack (Prometheus, etc.)
Static checks are easy-ish (render manifests, schema validation, policy checks), but those don’t prove the add-on actually behaves correctly.
What I’m trying to learn from people doing this at scale:
Do you test every add-on on every PR, or do you tier it (core vs non-core) and only run deep tests on core?
Do you spin up an ephemeral cluster in CI (kind/k3d) and run smoke tests? If yes, what are your “minimum viable” assertions?
For cert-manager, do you test real issuance (self-signed issuer + test cert), webhook readiness, etc.?
For external-dns, do you:
run --dry-run and assert expected planned DNS changes, or
hit a real sandbox DNS zone/account in staging?
Where do you draw the line between:
fast PR checks (render/schema/policy)
ephemeral cluster smoke tests
staging integration tests (real cloud LB/DNS/IAM)
War stories welcome—especially “we tried X and it was a trap.”
r/cicd • u/Apprehensive_Air5910 • Dec 07 '25
Are you generating tests, reviewing configs, predicting failures, enforcing standards… or avoiding AI completely?
What’s worked and what flopped?
r/cicd • u/Snoo52413 • Feb 23 '26
this tool is something I built for my own pipelines and automation scripts, mainly because nothing similar existed.
I’ve been using it for about six months, only last week I wrote proper documentation and moved it from my private Git server to GitHub and made it public.
https://github.com/azolfagharj/tmq
Complete standalone TOML CLI processor . tmq is a lightweight, portable, cross-platform, and fully featured command-line TOML processor. Like jq for JSON and yq for YAML, tmq is for TOML. supporting query, modification, and format conversion
r/cicd • u/QualityHot6485 • Dec 14 '25
I am trying to deploy a CI/CD pipeline using GitHub Actions for CI and Argo CD for CD.
My goal is to whenever there is a commit in the dev branch I want to create a docker image and store it in the GitHub image registry.
Now I have a specific repo in which argo cd tracks for changes in that repo. I want the docker image to updated based on the latest docker image tag.
I am using kubernetes so it has to update to the helm chart.
Then argo cd will build/recreate the pods based on lts docker image.
How can I achieve this??
I initially planned to try with argocd image updater but in my openshift container platform it is not available.
In the GitHub actions itself can I mention it to modify the package by updating it to the latest image (by creating a task and cloning it and modifying it).
Or is there any better alternative for this ??
r/cicd • u/Mother-Pear7629 • Dec 04 '25
Hi y'all,
I’m currently working on an open-source CI/CD engine and API (not a full CI/CD product), intended to be used as a building block for creating custom CI/CD platforms.
The idea is to provide a small, extensible core that other developers and platform teams can use to build their own CI/CD platforms on top of it.
It’s designed to be:
lightweight and self-hosted
API-first and event-driven
easy to extend with custom pluggable runners/drivers
usable in air-gapped, edge, or internal platforms
If this sounds like something you’d find useful or interesting, I’d really appreciate:
early technical feedback (Do you think such an API-first CI engine actually makes sense in practice?), and
a star ⭐ on GitHub to help with visibility.
You can find it on Github here:- https://github.com/open-ug/conveyor
r/cicd • u/Basic_Let7303 • Aug 13 '26
We are redesigning our CI/CD pipelines and trying to figure out the cleanest way to structure deployments.
Which route do you guys prefer?
One unified job: A single parameterized job where you just pass in the environment variable (develop,staging, prod, etc.).
Split jobs (same file): Explicitly separate jobs likE edeploy-develop ,deploy-staging and deploy-prod sitting in the same workflow.
Hard split: Completely separate files for lower envs(develop and stage) vs. production.
I want to avoid copy-pasting YAML, but I also don't want a massive, over-engineered "smart" job that's hard to debug. What's the sweet spot?
What strategy are you following in your org?
r/cicd • u/Puzzleheaded-Flan37 • Dec 19 '25
We were tired of manually maintaining Jenkins just to run Jenkinsfiles for CI/CD, Is it just me?.. TT
so I built a lightweight Git server that supports CI automation similar to GitLab Runner — while still using Jenkinsfiles.
The project consists of two applications:
jgitkins-server (Spring Framework + Eclipse JGit Server)
jgitkins-runner (Spring Framework + Jenkinsfile Runner)
P.S. This is still an MVP and under active development.
You can try it out on the develop branch.
Feedback is very welcome if you’re dealing with the same CI pain.
Thanks :)
r/cicd • u/zer0_snot • Jan 28 '24
How do I get better at improving an existing CI/CD process? The work in my current company is kinda chaotic and I'm trying to improve it. How can I learn more about :
best practices,
process problems and how they were solved
examples of good processes (practically explained)
I don't mean getting better at Github Actions or Jenkins pipeline on a code-level. I'm not talking about learning specific code.
I mean purely at the process-level, where you are seeing a birds-eye of what's going on, what devs are doing, what QAs are doing and then improving the whole process to make it less chaotic.
Can anyone point me towards books, courses, or any other material that can help me get better at it?
r/cicd • u/Karan-Sohi • Mar 14 '23
Hello,
Are you tired of dealing with microservice failures? Check out DoorDash Engineering's latest blog post to learn about common failures and the drawbacks of local countermeasures. The post also explores load shedding, circuit breakers, auto-scaling, and introduces Aperture - an open-source reliability management system that enhances fault tolerance in microservice architectures.
If you're interested in learning more about Aperture, it enables flow control through Aperture Agents and an Aperture Controller. Aperture Agents provide flow control components, such as a weighted fair queuing scheduler for prioritized load-shedding and a distributed rate-limiter for abuse prevention. The Aperture Controller continuously tracks deviations from SLOs and calculates recovery or escalation actions.
Deploy Aperture into your service instances through Service Mesh (using Envoy) or Aperture SDKs. Check out the full post and start building more reliable applications with effective flow control.
DoorDash Engineering Blog Post: https://doordash.engineering/2023/03/14/failure-mitigation-for-microservices-an-intro-to-aperture/
r/cicd • u/Drousch • Feb 02 '22
Hello!
With my team, we are building a library of open source jobs for CI/CD pipelines: https://r2devops.io
Some jobs are plug and play, meaning you can implement them directly in your yaml file and your pipeline will work. Of course, all the jobs are customizable, in order to fit the most of your project needs.
Today the jobs only work in GitLab, but the goal is to become platform-agnostic. We want to create a standard and produce resources usable no matter where your code is hosted! We are actually working on GitHub actions, trying to find an easy way to convert jobs from GitLab into actions for GitHub (and vice-versa).
We love feedback, so let me know what you think about R2Devops! And don't hesitate if you have any questions :)
r/cicd • u/BusyPair0609 • Apr 16 '26
My organization recently gave us access to codex, claude and gemini pro to try and evaluate all for the daily workflows on both engineering and DevOps side. With a couple of weeks into it, here is my take as a DevOps Engineer-
Codex - Amazing at long running tasks. Handle huge context decently (When working with multiple repos). Skills come handy when you want to offload mundane stuff.
Gemini - Great at doing research, grasping errors from screenshots and working with google ecosystem. Have mostly used this with the Google's Anti-gravity IDE and sadly it not the best out there. The agent often fails with error on high load and needs to be nudged again and again. The auto-completion though works amazingly even across the files.
Claude - Great capabilities, unmatched results. Writes a very clean and modular code. Claude in chrome is amazing to troubleshoot pipeline running in Github and Gitlab (Waiting for the official plugins to move to GA). Only limitation - the amount of tokens it burns is insane.
As a DevOps engineer one of my primary duties is to build CI/CD pipelines which earlier used to take me a couple of hours and can now be completed in minutes(developed, tested, shipped) using AI tools.
My questions is -
1. How deep is AI adoption in you org. in a high trust domain such as CI/CD?
2. As an DevOps engineer how to keep yourself ahead of the curve so that you are not replaced by AI someday?
r/cicd • u/Fit_Champion1940 • 10d ago
I’ve been collecting the usual suspects for CI-only failures. Most of the time it turns out CI isn't the problem, it's something my local machine was quietly giving the test for free:
- Speed: the CI runner is slower, so any check that doesn't wait or retry (isVisible(), reading text straight after a click) passes locally and fails there.
- Leftover state: locally I'm already logged in or the test data already exists. CI starts clean.
- Environment: a .env that only exists on my machine, or the runner being on UTC so date assertions break.
- Parallel workers: tests that pass alone but fight over the same data when run together.
My rule now is to open the trace from the failed CI run before touching retries. It usually points at one of these within a few minutes.
What am I missing? Curious about the weird ones: fonts, screen size, locale, DNS, anything.
r/cicd • u/rwflex • Aug 06 '26
Genuine question because I feel like I'm doing this wrong.
My current workflow for fixing a broken github actions pipeline is to change one line of YAML, commit, push again, try waiting for another minutes, watch it fail, add an echo statement, commit, push, wait again. Yesterday it took me like 7 commits to fix something.
Is there some setup everyone else knows about that I don't? Some way to actually pause a job and poke around? Or is commit-push-pray and wait just... the industry standard and we've all quietly accepted it?
r/cicd • u/QuoteForward5477 • Sep 09 '26
i made a super small change this morning (literally just updated a single config value in a backend service) and pushed it expecting a 3 minute pipeline run. ended up sitting there for nearly 25 minutes waiting for the CI to finish running.
took a look at the logs to see where the bottleneck was, and it was mostly fresh dependency installs and un-cached docker build layers. for some reason our runner pulled down every single package from scratch instead of using the cache, plus running the full end-to-end integration test suite that really didn't need to run for a tiny patch.
we use github actions on a shared runner setup, and it feels like as our repo grew, nobody ever went back to optimize the workflow files. now every small PR feels like a coffee break.
curious how you guys keep your builds lean? do you aggressively break pipelines into smaller conditional jobs based on changed files, or just pay for faster/bigger runners and call it a day?