Hotfix Releases

Hotfix Releases #

Classification: Confidential (describes release-pipeline source code and infrastructure).

A Slack-driven path to ship a fix to production without waiting for the release train, with the Slack thread acting as the incident record.

Read Production Release Pipeline first — this page only describes how a hotfix differs from it. Both flows are driven from Slack by the same admin user group: /release for the normal train, /hotfix for this one.

Shipping a hotfix #

Run /hotfix create in the channel. n8n verifies you are in the admin Slack user group — the same group that gates /release for regular releases — then resolves the latest release tag (= what is in production), branches hotfix/<latest-tag> from it, and posts a message with three buttons. That message is the thread root — do all your work in that thread.

/hotfix create
  └─ creates hotfix/<latest-tag>, posts [Release] [Status] [Abort]

       [Release] → re-checks the base is still the latest tag
                 → dispatches Create Release Tags against the branch
                 → opens a PR to main
       [Status]  → branch exists? base still latest?
       [Abort]   → deletes the branch

Then: push your fix to hotfix/<tag>, press Release, and merge the PR to main once the dust settles.

Buttons rather than slash subcommands because Slack slash commands cannot be invoked in threads at all — only built-ins like /topic work there.

Rules that will bite you #

  • One hotfix at a time. /hotfix create refuses if any hotfix/* branch exists.
  • Release re-checks that your base is still the latest tag, and refuses if it is not. If a regular release landed while your hotfix was open, production has moved and your branch no longer contains it — releasing would silently revert that release. Policy on failure: discard and start over.
  • Merging the PR to main is not optional, and is not automated. Main may contain work your branch does not, so a human has to confirm the merge is safe. Until it merges, the regular release train is blocked — see below.
  • Hotfix tags take the next number in the normal date sequence (2026-09-09-1), not a separate namespace. A -hotfix-N suffix would sort after -1 lexicographically and make “latest” ambiguous.

What a hotfix skips, and why #

A hotfix ships a fix on top of an already-released tag. It carries none of the work sitting in the pre-production Linear statuses — that work belongs to the next regular release. Running the Linear automation would therefore mark unreleased tickets as Done and stamp the hotfix tag onto them.

So on a hotfix, prod-deploy.yaml skips:

Skipped Would otherwise have
update-linear-deploying Moved the next release’s tickets to Deploying to Prod
add-release-label Labelled those tickets with the hotfix tag
update-linear-done Marked them Done

The deploy, the builds, and the Slack notifications all still run.

The hotfix’s own ticket is not moved by this release. It gets its status transition normally when the hotfix PR merges to main and ships in the next regular release.

Release notes also differ: there is no Hasami Version Update PR behind a hotfix, so get-pr-description reports the commits between the base tag and the hotfix tag (compareCommits) instead, with the Slack button pointing at the hotfix PR.

Why regular releases get blocked #

Because production now holds a commit that main does not, cutting a regular release before the hotfix branch merges back would deploy main over the top of it and silently revert the fix.

release-tags.yml therefore refuses to cut a regular release tag while either of these exists:

  • an open hotfix/* → main PR, or
  • a hotfix/* branch whose PR is not merged (closed unmerged, never opened, or targeting another base)

Both checks are needed: a branch created but never PR’d is invisible to the first, and a branch deleted with its PR left open is invisible to the second.

A branch whose PR has merged does not block — main already has the fix, so releasing main cannot revert it. Such a branch is only litter from a failed auto-delete, and the guard logs a warning naming it rather than stopping the release. Blocking there would wedge the release train behind advice (“merge the hotfix PR”) that has already been followed.

The block message names each blocker’s exact state — PR #11 closed without merging, no PR opened, open PR #13 → next — because each is cleared differently.

The block posts to the release channel naming what is in the way. To clear it, do one of:

  1. Merge the hotfix PR to main (the normal resolution), then re-run Create Release Tags.
  2. Abort the hotfix from its Slack thread, if it was created but never released.
  3. Override, by dispatching Create Release Tags with allow_open_hotfix=true. Only do this if you have confirmed the hotfix is genuinely not in production.

Note that by the time the guard fires, /release has already merged the Hasami PR — re-running /release will not help, because there is no longer an open PR for it to merge. See When a release is blocked by an open hotfix.

It deliberately does not check commit ancestry. This repo squash merges, so after the PR lands the original hotfix commit is reachable from nowhere — an ancestry check would block every future release permanently. PR and branch state clear themselves: merging closes the PR and auto-deletes the branch.

Implementation notes #

The Slack side is two n8n workflows on n8n Cloud (one per webhook — slash command and interactivity), doing HMAC verification and talking to the GitHub REST/GraphQL APIs. The real work is a workflow_dispatch of Create Release Tags, so if n8n is unavailable the same release can be triggered from the Actions UI by hand.

Known rough edges:

  • Buttons never expire. chat.update strips them after Release/Abort, and the handlers re-check live state, but an old thread can still show live-looking buttons.
  • The release result does not reach the hotfix thread. prod-deploy.yaml opens a fresh thread in the release channel; the hotfix thread only ever sees “:rocket: dispatched”.
  • The GitHub credential is a fine-grained PAT needing Contents: RW, Actions: RW, Pull requests: RW. Note that insufficient scope returns 404, not 403 — check x-accepted-github-permissions in the response headers.