Production Release Pipeline #

Classification: Confidential (describes release-pipeline source code and infrastructure).

What actually happens between “the Hasami PR was merged” and “it’s live”. Hasami covers release notes and version bumping; this page covers the deploy, the Linear status automation, and how the pipeline knows what kind of release it is.

The chain #

Releases are cut from Slack, not from the GitHub UI.

/release                (Slack, admins only)
  └─ n8n ─────────────── merges the "Hasami Version Update" PR to main
       └─ release-tags.yml ─── creates an annotated tag YYYY-MM-DD[-N]
            └─ (tag push) ──── prod-deploy.yaml builds, deploys, updates Linear

Nobody merges the Hasami PR by hand. /release sends a webhook to n8n, which verifies the requesting Slack user is in the admin Slack user group and then merges the PR through the GitHub API. That merge is what fires release-tags.yml — it triggers on pull_request: closed against main/next and checks the title is exactly Hasami Version Update.

/release merges immediately. There is no confirmation step and no button to click — unlike /hotfix create, which posts buttons and waits. Typing /release ships to production.

The same admin group gates /hotfix. See Hotfix Releases.

Two conventions hold this together, and most surprises come from forgetting one of them:

Tags are dates, not versions — 2026-09-09, then 2026-09-09-1, -2 for further releases the same day. Sort by parsed (date, seq), never lexicographically: 2026-09-09-10 sorts before 2026-09-09-2 as a string.
The latest tag is what is in production. Everything else, including the hotfix flow, depends on this.

`release-tags.yml` — cutting the tag #

Triggered by merging a PR titled exactly Hasami Version Update into main/next, or by manual dispatch. It picks today’s date, scans existing tags for a free suffix, and creates the tag.

Tags are annotated, not lightweight. The difference matters: a lightweight tag is just a ref pointing at a commit, with no room for metadata. An annotated tag is a real git object with a message, and that message is where the release type is recorded:

Release	Tag message subject
Regular	`release: main`
Hotfix	`hotfix: cut from hotfix/<base-tag>`

prod-deploy.yaml reads that line back to decide whether to run the Linear automation. The two strings are a contract between the two workflow files — reword one and the release fails loudly by design (see Release-type detection).

Tags created before this mechanism existed are lightweight and read as regular releases, which is correct: every hotfix tag is annotated.

`prod-deploy.yaml` — deploying it #

A tag push is the only trigger. There is no workflow_dispatch, so there is no button to deploy an arbitrary tag on demand — to re-deploy a tag, re-run that tag’s original workflow run, which replays the same push event.

Note the trigger is tags: "**", so any tag push deploys to production, not just date-shaped ones.

get-git-tag-name ──┬─ detect-hotfix ──┬─ get-pr-description ─ slack-notify-start ─ slack-post-release-notes
                   │                  │
                   │                  └─ update-linear-deploying   [skipped on hotfix]
                   │                       │
                   ├───────────────────────┴─ prod-deploy (deploy.yaml: ~25 builds + ~25 deploys)
                   │                       └─ build-tofu-internal-mcp ─ deploy-tofu-internal-mcp
                   │                            │
                   └────────────────────────────┴─ add-release-label   [skipped on hotfix]
                                                     └─ update-linear-done  [skipped on hotfix]
                                                          └─ slack-notify-result

notify-failures ─ PagerDuty, on any job failure

prod-deploy calls the shared deploy.yaml, which builds and deploys every service; each of its jobs checks out the pushed tag itself.

Linear status automation #

This is the part that surprises people. A production release bulk-moves Linear tickets by status, not by what is actually in the diff:

Job	Action
`update-linear-deploying`	Every issue in `In Dev`, `Deploying to Dev`, `Ready for Release`, `Needs Revert` → `Deploying to Prod`
`add-release-label`	Labels every `Deploying to Prod` issue with the release tag
`update-linear-done`	Every `Deploying to Prod` issue → `Done`

There is no per-ticket check that the work is in this release. The sweep assumes a regular release ships everything merged to main, which is true for the normal train and false for a hotfix — hence the skip logic below.

A consequence worth knowing: re-running an old tag’s deploy re-runs the sweep against whatever is in those statuses now, not what was there originally.

Release-type detection #

detect-hotfix resolves the tag over the GitHub API — getRef to learn whether the ref points at a tag object, then getTag to read its message. It needs no checkout, because it reads no repo files.

It accepts exactly two message subjects and fails on anything else:

release: <branch> → regular release
hotfix: cut from hotfix/<base> → hotfix, and <base> is exported as hotfix-base
a lightweight tag → regular release (pre-dates the mechanism)
anything else → job fails

The strict third case is deliberate. The marker is a free-text string written in one file and parsed in another; if they drift, a silent fall-through to “regular” would sweep the next release’s tickets on a hotfix — the exact bug the detection exists to prevent. Production tags are never created by hand, so there is no benign case to tolerate.

Two traps this design avoids:

Never read a commit message. git for-each-ref --format='%(contents:subject)' on a lightweight tag falls through to the commit subject, so a commit titled hotfix: ... would be mistaken for the marker. Reading the tag object’s message removes commit messages from the picture entirely.
Never derive the release type from branch containment (git branch --contains). This was the first implementation and it is wrong three ways: after the hotfix merges to main the commit is reachable from main; a squash merge leaves it reachable from nowhere; and because a hotfix branch is cut from the latest tag, that regular tag is immediately contained by hotfix/*. Any re-run flips the answer. A tag object is immutable.

When detection fails #

Deliberately asymmetric, and the blast radius is wider than just Linear. detect-hotfix is needed by every Linear job and by get-pr-description, so a failure there gives you:

	On a `detect-hotfix` failure
`update-linear-deploying`, `add-release-label`, `update-linear-done`	skipped — Linear is left untouched
`get-pr-description`	skipped — no release notes are resolved
`slack-notify-start`	runs, announcing the tag
`slack-post-release-notes`	runs, posting “No release notes available”
`prod-deploy`, `build-tofu-internal-mcp`	run — the release still ships
`slack-notify-result`	reports `:x: Release Failed` — the deploy shipped, but Linear was not swept and needs manual cleanup
`notify-failures`	pages via PagerDuty

The reasoning for letting it fail: wrongly sweeping Linear is silent and awkward to unwind, whereas not sweeping is visible — tickets sit in Deploying to Prod until someone moves them. Shipping the deploy and waking a human is the right trade at 3am.

What makes that trade work is that on-call still gets context. slack-notify-start gates on needs.get-pr-description.result != 'cancelled' rather than == 'success', precisely so a skipped get-pr-description does not mute it — and because slack-post-release-notes and slack-notify-result both key off slack-notify-start succeeding, gating it on success would have taken the entire Slack chain down with it. A deploy that ships with no Slack trace at all, paged but uncorrelatable, is worse than one with missing release notes.

The same reasoning drives needs.update-linear-deploying.result != 'failure' on the deploy jobs: a skipped job reports skipped, and the stricter == 'success' would skip the deploy itself.

slack-notify-result is the one place that must not be applied loosely. Its success message names the hotfix case explicitly rather than accepting any non-failed update-linear-done, because that job is also skipped when deploy-tofu-internal-mcp fails — so the lenient form would post a green release while PagerDuty was paging.

When a release is blocked by an open hotfix #

release-tags.yml refuses to tag a regular release while a hotfix is unmerged (see Hotfix Releases). Worth understanding the ordering:

The guard runs after the merge, because the merge is what triggered the workflow. By the time it fires, /release has already merged the Hasami PR to main. So a blocked release leaves main version-bumped and untagged, and nothing ships.

Recovery is not to run /release again — there is no longer an open Hasami PR for it to merge. Instead:

Clear the hotfix (merge its PR to main, or abort it).
Re-run / dispatch Create Release Tags from the Actions UI. It will tag the current main.

Because the workflow run is the only other signal, the guard also posts to the release Slack channel naming what is in the way — that notification is load-bearing, not decoration.

Follow-up worth doing: this check belongs in the n8n /release handler before it merges, so a blocked release never half-happens. That is also the only place a check could stop it — /release merges with no confirmation step, so there is no human checkpoint between the command and the merge. The workflow-level guard should stay either way, since it also covers a manual dispatch.

Hasami Release Tool — release notes and version bumping
Hotfix Releases — shipping without the release train
Deployment Monitoring — watching a deploy
Linear — what the statuses mean to humans