A Perfect* CI Process

When you commit and push code to a Git repository, it can kick off a series of events. A pipeline for your software, as it were. This whole process is often called “continuous integration,” or CI for short. It’s pretty crucial to modern software development and falls under the umbrella of “DevOps” in current developer parlance.

Services like CircleCI and Travis CI helped establish it, but now there are loads of competing services. GitHub Actions (which is Azure Pipelines under the hood, at least originally, if I’m not mistaken) is popular because it’s built right into the Git tool most of us already use. It did follow in the footsteps of GitLab CI in this instance. Semaphore seems popular these days, and I’m partial to Buddy, to name just a few others. Hosts like Netlify and Vercel will run a build command for you, making them defacto CI.

What could a CI process do? Well, let me answer that by telling you what I think a CI process should do. Buckle up, it’s kind of a lot.

First, a few points:

• Not all CI should run all the time. You can get really specific with CI on what parts of it run under what conditions. Broadly, I think it makes sense to think of it as 1️⃣ run on any push to any branch, 2️⃣ run against pushes to PR branches only, or 3️⃣ only run on a push to the main branch. I’ve tried to indicate that under each title below.
• Things that run in CI very likely don’t only run in CI. You can (and probably do) run all these things locally because you have to view the site, because you’re testing it, or for any other reason it may be advantageous to do so.
• There is a concept of commit hooks. I can imagine some level of disagreement with me about what should run in CI, because there is workflow overlap on what hook and CI can do for a team. For example, you could force a pre-commit hook to run tests and prevent the Git push at all if they fail. I’d argue that you should do that, but then also run them in CI. The chances of CI working well are way higher than the chances of any given developer’s machine working well.
• Things in CI can fail. That’s part of the point and a strength of CI. If tests fail (for any reason), then the Pull Request (PR) that this CI ran against should be prevented from being merged. CI isn’t just for lookin’ at, it’s for providing real guards.

Format all your code

Branch | Pull Request Branch | Main

Code Formatting (like Prettier) should run inside all your developer’s code editor. That’s the most satisfying place for it to run because it helps keep code clean as coding is happening. But you could and should run it in CI as well, so you can guarantee it is happening. It could be broken on one particular user’s machine, one developer could decide they don’t like it while they code, one developer could have diverging local settings, etc.

What you are trying to avoid is another developer pulling down new code that is unformatted, having their machine format it, and then having to push it back up like it is their change.

It could be argued that this problem-preventative formatting actually belongs in a pre-commit hook. Those are part of the development environment and not subject to the whims of one developer. As a pre-commit hook, the code is formatted before it goes up in commits, which solves the issues. This is less awkward than running formatting in CI, where if changes are made, those changes need to be auto-committed to the new branch and thus pulled before any new commits. More plumbing, but still, CI is more bullet-proof than pre-commit hooks.

Lint all your code

Branch | Pull Request Branch | Main

By linting, I mean like ESLint for JavaScript/TypeScript, Stylelint for CSS, golangci-lint for Go, RuboCop for Ruby, etc.

Surely less controversial than formatting in CI is linting in CI. Linting errors are, at best, broken agreements about code style between you and your team and, at worst, actual bugs. You need to know if code is trying to get into your code base has a listing error — so run linting in CI.

Linting is still a multi-faceted thing:

• Have linters installed on developer’s code editors, so most linting problems are caught while coding.
• Run linting as a pre-commit hook, so linting problems don’t make it into commits.
• Run linting in CI on all pull request (PR) branches, just in case a linting error slips through. Do not allow merging a PR with a linting error.
• Your choice to run linting in CI against the main branch. You would not if you trust the PR system. You would if you don’t. Consider if a code red if a linting error on the main branch is found.

Optimize images

Branch | Pull Request Branch | Main

If you’re ultimately deploying untouched original images, well, fine, don’t optimize those (they should probably use Git LFS). But if you’re deploying any sort of image that is intended to go across the network to a user and there is no other processing middleman, that image should be sized, formatted, and optimized for web performance. Images are hard, so let CI help you as much as possible.

Calibre has a GitHub Action for this they’ve put out — read about it in Automatically Compress Images on Pull Requests. Don’t worry about remembering to optimize images before you commit them, make CI do it for you. This is already how we think with other assets — CI minifies our CSS and JavaScript for sure — so extend that thinking to anything else that needs optimization.

SVG is another example, run those automatically through SVGO before a PR goes in.

Run all build processes

Branch | Pull Request Branch | Main

The spirit of Git is that you keep “build artifacts” out of the repo. As in, anything in the conventional dist/ directory. Say you’re running a Next.js website. The command next build is what produces a production-ready version of the site in a .next/ directory. That .next/ the directory is in your .gitignore file, which solves the problem of keeping those build artifacts out of the repo, but then what files can actually be deployed? The CI process needs to run next build to get the files to deploy (in another CI action).

But a build process might do all sorts of things aside from running the main build command of a site building tool. Perhaps it needs to run a CSS processor like Sass, plus PostCSS, then Tailwind, then Purge CSS (phew). Perhaps it needs to do JavaScript processing on its own, like Babel and Rollup, then do minification of those assets. Perhaps it needs to post-process templates to update URLs. You might do all your tasks one-by-one as your build process, or use a tool like Gulp to orchestrate them, or a newer tool like Vite.

Everybody’s build process is different. You’ll be running this in development, too, but it will also be running in CI and likely with different settings (different ENV variables, more aggressive bundling, etc).

Run Unit Tests

Branch | Pull Request Branch | Main

These are your simple tests that text that the correct output happens from calling code in one part of your code base. Your npx jest command or ruby test or go run test — or however you have it all set up.

Other than running the build process, running tests is probably the most important thing a CI can do. You’ll probably recognize “badges” like this, graphics that show on GitHub repository main pages that indicate the status of tests.

You can and should run your unit tests locally as well, but running them in CI is vital because it prevents problematic code from going to production. If tests fail, that failure status can prevent a PR from merging to main. That is the power of tests.

Run Integration Tests

Branch | Pull Request Branch | Main

Really anything with “tests” as a concept should be run in CI, but let’s do them the justice of naming them. Integration tests test multiple systems combined. Perhaps fetching data from your API and processing it as expected.

These might be using similar frameworks to run the test and make assertations (e.g. Jest), but likely require more setup work before they run. For example, if you really are testing an API, you’ll need to spin up a server to host that API, seed it with data, and have it be entirely ready to accept requests from another system.

This may be the place to do slightly more exotic and custom testing, like running Madge to check for circular dependencies or other things unique to your needs.

Run End to End Tests

Branch | Pull Request Branch | Main

These are distinct from integration tests in that they involve using a simulated browser to automate tasks that users will do on your site. Go to the homepage, click signup, fill out the form, does it take you to the next expected step? Does it error correctly with invalid info? Can I make it through paid signup?

These kind of tests are very valuable in that they test that complete systems are working as intended. If they prevent you from deploying a broken signup flow once, that can be worth the all the effort.

These are tools like Cypress, Reflect, or even something like jest-puppeteer if you want to keep it low-level and in Jest. Cypress is the one I’ve used the most, and I like how they help you run it on GitHub.

Since you’ll need to be hitting URLs in a browser for these tests, that marries up with the idea that your CI should be building you deploy previews or deploying to a staging environment that you can test against.

Run a Performance Report / Check against a Performance Budget

Branch | Pull Request Branch | Main

Chrome’s tool:

Lighthouse CI is a suite of tools that make continuously running, saving, retrieving, and asserting against Lighthouse results as easy as possible.

You should know the performance implications of what you are about to ship. This can help you adhere to a performance budget. Even nicer would be to enforce the performance budget, treating it as a failing test if certain performance metrics are not met.

At the very least, it keeps you honest. It’s possible some developer decided to add some fancy way to have relative datestamps in 8 languages and it added 2MB to the JavaScript bundle. A code review might miss that, but a performance report will not.

Run Visual Regression Tests

Branch | Pull Request Branch | Main

One of the ways you can be super sure you didn’t make an unintended change to the website while updating code is to test literally exactly how it looks. If you change red to purple intentionally, that’s fine, but if you make some CSS update that changes how a modal is positioned on some page you didn’t intend, that’s not fine.

A tool like Percy can screenshot URLs of your PR branch and compare them to what is on your main branch. If there is any change, that’s considered a failed test and will require you either to approve the change, or go back and fix what was making the unintended change.

Run Accessibility Tests

Branch | Pull Request Branch | Main

There are tools, axe most notably, that can look at your site and find likely/potential accessibility problems. The auto-detectable problems make up the bulk of accessibility problems on the web, which is ironic considering how easy they are to detect, and usually, to fix.

Rather than remembering to run these kinds of tests yourself, run what you can in CI.

Build a Sitemap

Branch | Pull Request Branch | Main

You could consider a sitemap.xml file a build artifact. You only need to produce it for your production website, and you might as well just do it once right as the site is deployed to production.

Deploy to Staging / Make a Deploy Preview

Branch | Pull Request Branch | Main

It’s a powerful thing to have a live URL of your site that anyone can check out before it goes to production. Put it behind auth if you need to. This can allow for real humans to sign off on it, ideally doing real QA on it.

But this URL also opens the door to do testing in CI against. Huge.

Deploy to Production

Branch | Pull Request Branch | Main

You made it!

All the other CI stuff is passing, and your PR is headed to main. Merges to main should fire off a unique bit of CI that deploys the site to production. Hosts like Netlify and Vercel will do this for you. Maybe you rsync stuff to a remote server. Maybe you wire up files to be moved over SFTP. Maybe you install git on the remote server and it pulls. Whatever! It’s time to go!

Not all code is a website, so this finish line actually might be something like “push the tagged release to npm” or something like that.

Tell Third Parties Stuff

Now is the time to, say, tell Cloudflare to empty certain caches. Time to tell a Slack channel a deployment went out. Tag the release in Sentry. You could be pinging 3rd parties about stuff at any time during CI, but it seems particularly prudent here at the end.

---

* “Perfect” is a stretch, I know. Everybody’s CI will be different and adjusted to their needs, and that’s a good thing. My main point? I think CI is underused. If all this were easier, more projects would benefit from a more comprehensive set of CI tasks.