Questions to Find the Right Continuous Integration System for You

Continuous integration is fancy shell scripting. It even uses shell scripting terminology like pipelines, but is obscured with marketing and many seemingly complex and different methods of configuration.

The purpose of this post is to discuss Continuous Integration (CI) Systems at a high level if you are considering switching to a new service. I want to introduce you to a few talking points that aren't discussed enough in order to help you make a decision.

What are the goals of Continuous Integration (CI)?πŸ”—

The functions of continuous integration are very flexible and therefore, and the goals are super simple. Automatically run scripts upon the occurrence of an event.

Common events can be a git commit or a timer event set to run every 5 minutes.

How does CI differ from shell scripting?πŸ”—

Continuous integration has a narrow context in which it is intended to run. It has extra structure in order to tell a tighter story. (The CI story leads into another story called Continuous Delivery/Deployment (CD), but we will only be focusing on CI)

A best practice of CI is to automatically build and/or run tests on every commit in order to validate whether the codebase builds/tests are complete without errors.

Basic structure defines conditions as well as commands or actions that should be executed when an event occurs. Conditions such as a new commit in a specific branch, or file paths. Or keywords in commit messages.

On the more advanced side, platform specific features like plugins may be introduced for sending or receiving events from other systems, initiating external triggers based on other pipelines, caching, cleaning up and notifications.

What type of CI systems that exist today?πŸ”—

There is no shortage of software that is designed for Continuous Integration. What I believe sets these platforms apart from each other is:

If your organization regularly exercises the creation of new environments, like in consulting orgs, it is reasonable to gain the experience with an open source tool that won’t cost you any money with licensing.

Otherwise, I might suggest picking a system the reduces the friction for pipeline creation and maintenance.

Styles of setting up new pipelines falls into two categories.

Stateful (WebUI) pipeline configπŸ”—

Jenkins is the primary example of configuration via WebUI, as it is the functionality out of the box. Adding pipelines in this system requires a user to log into the WebUI, configure connections to builder systems and methodically define pipelines.



Stateless (File-based) pipeline configπŸ”—

This is the modern style of CI system. Your TravisCI, Github Action, Azure DevOps

Typically we find this system is hosted through your git provider, or as a separate hosted service that we allow access to the repos. YAML is the most popular config format.



My opinionπŸ”—

I strongly prefer the stateless type of configuration because it provides a more code-like arena where documentation can be provided.

Mixing the workflow of CI pipeline configuration with your organization’s version control practices can be disruptive if you need to push code in order to see the results of changes to your pipeline. It is a price to pay for the ability to easily correlate changes in codebase with changes in pipeline.

The balance to strike rests on your code hosting provider needs (i.e. costs per user) and the types of technologies and platforms you need to support for building (web, native, mobile, embedded, etc). This flexibility mostly skews support towards building for web-based technologies.

Also, Jenkins technically has file-based pipeline support via Jenkinsfile, but unless my org was already deeply invested in staying with Jenkins, I would steer clear. It is not the future. It is harder than necessary to use.

What are the CI Hosting Options?πŸ”—


Self-hosting means running and maintaining a process on a physical or virtual machine that you have access to.



System as a Service (Hosted)πŸ”—

External systems as a service (Hosted services) allow users to just log in and connect their version control.



Issues to Consider when Switching between CI platformsπŸ”—

These are some categorized rhetorical questions that you or your organization should ask if you are migrating CI platforms

Threat modelπŸ”—

Contributor modelπŸ”—


Cost of Build minutesπŸ”—

Responsiveness needsπŸ”—

Technologies usedπŸ”—

Delivery/Deployment modelπŸ”—


Not all continuous integration is considered equal. But they all provide basically the same functions. In most cases, it is possible to self-host and do everything yourself. But sometimes it is worth paying money to a product / service that takes care of your needs. It is really all about making sure you know what the trade-offs are for your specific case.