Ship It! (How Wilco Does CI/CD)

Ship It! (How Wilco Does CI/CD)
Written by
Tom Peri
July 28, 2022
No items found.

When setting up a dev org for a new product, we tend to prioritize code-related tasks and workflows. After all, what can be more important than clean and efficient code, testing-driven development practices, and scalable architecture? Not to underestimate all of these, but another thing you should be thinking about is the delivery pipeline. 

At Wilco, we’re big fans of continuous integration and deployment. And while we’re a relatively new company, our process has changed quite a bit over the past year — as we learned more and more about what works for us and what doesn’t. 

How It Began

It all started with our take on Git-Flow: a developer creates a feature branch out of develop, works on their code, then opens a pull request back to develop, which is continuously deployed to our staging environment. After several features were merged into develop, we would open another pull request from develop to main, which was then deployed to production once merged.

GitFlow feature branch illustration, by Mobify

This workflow sounds fairly simple but raises a couple of issues; first and foremost, keeping a clean commit history. It might sound minor but is very useful to dissect issues and understand the scope of changes each feature introduces. 

Our merge strategy was to squash the feature branches into develop and merge commit develop into main. This way we had a clear view of what feature was added and when, but without all the mess of the work-in-progress commits. This, however, had a couple of major downsides:

  • Developers needed to open a dedicated “deployment” pull request from develop to main, just for the sake of updating the production environment.
  • Reverting changes on either develop or main could cause sync issues we later needed to address between develop and main. Pushing a hotfix directly to main desynced the two as well.
  • Developers often mistakenly use the wrong merge strategy, such as clicking rebase or merge-commit instead of squash, which can cause conflicts between develop and main. These can be hard to resolve, which results in hard resetting main to develop. Unfortunately, GitHub doesn’t offer to force a different merge strategy per branch.

We then began looking for a new development and deployment cycle, with the following requirements in mind:

  • Minimal change to our day-to-day development workflows.
  • Auto-deployment to staging.
  • The easiest way possible to deploy to production: through GitHub, without external interactions.
  • An easy way to revert, roll back, or manually deploy a different branch to production.
  • Increase the visibility of what’s currently deployed in each environment.

What we did

With our requirements defined, we decided to use trunk-based development.

Having a single branch (main) means we could no longer use Heroku’s automatic deployment like we did when we had two branches. This meant we couldn’t test features before shipping them. The solution was to implement our deployment pipeline using the releases mechanism: git tags with fancy readme and GitHub actions with a few nifty actions that increase our deployment visibility.

In a nutshell, this is our flow: 

  • Create a feature branch out of main.
  • Open a Pull Request.
  • Merge to main → auto-deploy to staging.
  • Publish a release → auto-deploy to production.

Release Drafter

We use the Release Drafter action in two workflows. When a pull request is opened, we trigger the Auto Labeler functionality, which adds a label based on a predefined rule: feature, fix, chore, or any other required label.

name: Auto Labeler
   types: [opened]

   runs-on: ubuntu-latest
     - uses: release-drafter/release-drafter@v5
         config-name: release-drafter-config.yml
         disable-releaser: true # only run auto-labeler for PRs
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Once a pull request is merged into main, we trigger the Release Drafter functionality, which automatically creates or updates a release draft in GitHub with the changelog from a previous release.

name: Release Drafter

     - main

   runs-on: ubuntu-latest
     - uses: release-drafter/release-drafter@v5
         config-name: release-drafter-config.yml
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

This is what a release draft looks like:

Publishing a new release is two clicks: edit a release draft, and click Publish.

Deployment to Heroku

Once a release is published, we want to deploy it to our production environment in Heroku. We run several integration and lint tests, and use the Deploy to Heroku action to deploy the new release.

name: Continuous Deployment

# This action works when creating a tag or release
     - "released"
     - "release-*"
     - "rollback-*"

# some ci-checks here
# some lint checks there

     - ci-checks
     - lint
   runs-on: ubuntu-latest
   timeout-minutes: 15
     - uses: actions/checkout@v2
     - uses: akhileshns/heroku-deploy@v3.12.12
         heroku_api_key: ${{secrets.HEROKU_API_KEY}}
         heroku_app_name: "<your app name here>"
         heroku_email: "<your email address>"
         team: "<team name>"
         dontautocreate: true # do not create the app if it doesn't exist

Note: you need to generate a Heroku API key and store it as a secret in your GitHub repository or organization.

Revert and Rollback

Once the deployment flow is up and running, it’s time to address the unhappy path: rollbacks and reverts.

The easiest way is to create a new release, pointing to a different commit, using the GitHub UI. You can select a commit (it has to be on the base branch), release it, and the deployment workflow will automatically deploy that release to production.

Want to deploy a different branch? You’ve probably already noticed that the deployment workflow isn’t triggered only by publishing a new release, but also when a tag is pushed. This allows us to add a rollback tag on any commit (no matter the branch), push it to the repository, and deploy it as a temporary version to production—while our engineers fix any problem that might have happened.

git checkout <commit/tag>
git tag rollback-<number>
git push origin rollback-<number>

What comes next

We are now several months after implementing this new flow, and couldn’t have been happier about the results. Most of our pain points are resolved, without the need for additional maintenance. Onboarding new employees is quick, easy, and requires no complicated training on our CI processes.

Like all things in a startup, we evaluate our flow and code constantly, introduce changes and iterate fast. Our flow can be further extended - add Slack notifications, automatic rollbacks and more, but for now, this serves us well 🙂