TUTORIAL 05

Emergency Hotfix

It's Friday night. Users can't checkout. Payment processing crashes with a null pointer. S0 Critical.

⏱ ~15 min · Hands-on

/agile-ship-hotfix /agile-code-branch /agile-code-tdd /agile-code-ci /agile-code-pr-review /agile-code-merge /agile-ship-deploy /agile-ship-rollback

🚨 Incident Alert — S0 Critical

PagerDuty fires at 21:47 on a Friday. The checkout page throws a 500 error. Revenue is dropping. Users cannot complete purchases. The error log reads:

ERROR 2026-04-04T21:47:12Z PaymentService.calculateTotal()
  NullPointerException: Cannot read property 'amount' of null
    at OrderProcessor.applyDiscount(OrderProcessor.java:142)
    at PaymentService.calculateTotal(PaymentService.java:87)
    at CheckoutController.processOrder(CheckoutController.java:53)

Step 1: Assess Severity

Before writing any code, classify the incident. This determines which workflow to follow.

Severity	Impact	Response
S0 Critical	Revenue-impacting, users blocked	Hotfix flow (this tutorial)
S1 High	Major feature broken, workaround exists	Expedited bugfix sprint
S2 Medium	Minor feature degraded	Normal sprint backlog
S3 Low	Cosmetic / edge case	Backlog grooming

Checkout is completely down. Users cannot pay. This is S0 Critical — we use the hotfix flow, not a regular bugfix.

Step 2: Branch from Main with /agile-code-branch

Key Rule

Hotfix branches always branch from main, never from develop. Main represents what's in production. When an S0 incident is active, /agile-code-branch detects the hotfix context and automatically branches from main.

/agile-code-branch

🌿 Branch created: hotfix/FIX-null-payment

Source: main
Convention: hotfix/FIX-[short-description]
Incident: S0 — Payment processing null pointer

Switched to branch 'hotfix/FIX-null-payment'

This creates an isolated branch directly from the production codebase. No unfinished feature work leaks in.

Why use /agile-code-branch?

Instead of manually running git checkout -b, /agile-code-branch automatically applies your project's branch naming convention, ensures hotfix branches source from main (not develop), and links the branch to the incident. No typos, no risk of branching from the wrong base.

Step 3: Reproduce with TDD

Run /agile-code-tdd to start a test-driven fix. First, write a failing test that reproduces the crash.

/agile-code-tdd

RED: Write the Failing Test

@Test
void should_not_crash_when_discount_is_null() {
    Order order = new Order();
    order.setSubtotal(99.99);
    order.setDiscount(null); // No discount applied

    double total = paymentService.calculateTotal(order);

    assertEquals(99.99, total, 0.01);
}

Run it:

$ ./gradlew test --tests "PaymentServiceTest.should_not_crash_when_discount_is_null"

PaymentServiceTest > should_not_crash_when_discount_is_null FAILED
  java.lang.NullPointerException: Cannot read property 'amount' of null
    at OrderProcessor.applyDiscount(OrderProcessor.java:142)

1 test failed

Confirmed — the test reproduces the exact production crash. Now find the root cause:

// OrderProcessor.java:142 — THE BUG
public double applyDiscount(Order order) {
    return order.getSubtotal() - order.getDiscount().getAmount(); // 💥 No null check!
}

Step 4: Fix and Go GREEN

GREEN: Apply the Fix

// OrderProcessor.java:142 — FIXED
public double applyDiscount(Order order) {
    if (order.getDiscount() == null) {
        return order.getSubtotal();
    }
    return order.getSubtotal() - order.getDiscount().getAmount();
}

Run the test again:

$ ./gradlew test --tests "PaymentServiceTest.should_not_crash_when_discount_is_null"

PaymentServiceTest > should_not_crash_when_discount_is_null PASSED

1 test passed

GREEN. The fix works. The null discount no longer crashes the checkout.

Step 5: Run Full CI

A hotfix must pass the full pipeline — no shortcuts.

/agile-code-ci

CI Pipeline — hotfix/FIX-null-payment
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Build ................ ✅ PASS (12s)
Unit Tests ........... ✅ 247/247 passing (34s)
Integration Tests .... ✅ 18/18 passing (22s)
Lint ................. ✅ 0 issues
Security Scan ........ ✅ 0 vulnerabilities
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Result: ✅ ALL CHECKS PASSED

Step 6: Expedited Code Review

Even under pressure, every hotfix gets reviewed. Use /agile-code-pr-review with expedited flag.

/agile-code-pr-review

PR Review — hotfix/FIX-null-payment
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Reviewer: @lead (expedited — 30 min max)
Changed: 1 file, +4 lines, -1 line

Findings:
  (none — minimal, focused fix)

Verdict: ✅ APPROVED
Comment: "Clean fix. Null guard is correct.
  Consider Optional wrapper as follow-up."
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Time to approval: 14 minutes

⚠️ Expedited != Skipped

An expedited review has a 30-minute SLA, but the same quality standards apply. The reviewer checks correctness, test coverage, and no regressions. The only thing that's faster is the turnaround time.

Step 7: Merge to Main AND Develop with /agile-code-merge

A hotfix must land in both branches. Main for production. Develop so the fix isn't lost in the next release. /agile-code-merge detects the hotfix branch and handles both merges plus version tagging in one command.

/agile-code-merge

🔀 Merge — hotfix/FIX-null-payment
═══════════════════════════════════

Pre-merge checks:
  ✅ PR approved by @lead (expedited)
  ✅ CI passing on branch
  ✅ No merge conflicts (main)
  ✅ No merge conflicts (develop)

Merging: hotfix/FIX-null-payment → main
Strategy: merge --no-ff (hotfix convention)
  ✅ Merged to main

Merging: hotfix/FIX-null-payment → develop
Strategy: merge --no-ff (hotfix convention)
  ✅ Merged to develop

Tagging: v1.2.1 (patch increment from v1.2.0)
  ✅ Tag v1.2.1 created and pushed

Branch hotfix/FIX-null-payment deleted (was 9c4e1b2).

✅ Hotfix merged to main + develop, tagged v1.2.1.

Why use /agile-code-merge for hotfixes?

/agile-code-merge detects the hotfix/ branch prefix and automatically performs both merges (to main and develop), creates the patch version tag, and cleans up the branch. This eliminates the risk of forgetting the develop merge or mistagging the release.

Why both merges matter

If the fix only lands in main, the next release from develop will reintroduce the bug. /agile-code-merge enforces both merges as mandatory for hotfix branches — it will not complete until both succeed.

Step 8: Deploy to Production

/agile-ship-deploy

Deploy — v1.2.1 → production
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Rolling deploy ........ started
  pod-1/3 ............. ✅ healthy
  pod-2/3 ............. ✅ healthy
  pod-3/3 ............. ✅ healthy
Health checks ......... ✅ all endpoints responding
Smoke test: checkout .. ✅ order placed successfully
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Deploy complete: 22:31 UTC
Incident duration: 44 minutes

What if the deploy fails?

If the hotfix deploy causes errors — health checks fail, error rates spike, or smoke tests don't pass — roll back immediately:

/agile-ship-rollback

⏪ Rollback — v1.2.1 → v1.2.0
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Reason: deploy health check failure
Rolling back to: v1.2.0 (last known good)

  pod-1/3 ............. ✅ rolled back
  pod-2/3 ............. ✅ rolled back
  pod-3/3 ............. ✅ rolled back

Health checks ......... ✅ all endpoints responding
Error rate ............ ✅ returned to baseline

Rollback complete. Production is on v1.2.0.
⚠️ Investigate the failed hotfix before re-deploying.

/agile-ship-rollback reverts production to the last known good version. It's always safer to roll back and investigate than to push a second fix under pressure.

Step 9: Post-Mortem

Post-Mortem: Null Pointer in Discount Calculation

📌 Root Cause: OrderProcessor.applyDiscount() assumed discount is always present. Orders without discount codes triggered a NullPointerException.
🔍 Detection: PagerDuty alert from error rate spike. Time to detection: 3 minutes.
🛠 Resolution: Added null guard in applyDiscount(). Total fix time: 44 minutes.

Action Items

Add null safety linting rule to CI pipeline (prevent similar bugs) — @lead, next sprint
Create story: "Audit all payment path methods for null safety" — backlog, 5 points
Add integration test for checkout with no discount — @dev, next sprint

Hotfix Flow

🚨

S0 Incident assess severity

🌿

/agile-code-branch from main

🧪

/agile-code-tdd RED → GREEN

⚙️

/agile-code-ci full pipeline

👀

/agile-code-pr-review expedited 30min

🔀

/agile-code-merge main + develop + tag

🚀

/agile-ship-deploy production

⚠️ Hotfix is NOT a Shortcut

A hotfix follows the same quality standards as any other change: TDD, code review, full CI. The difference is speed of turnaround and priority of scheduling — not a reduction in rigor. Cutting corners under pressure is how you create the next S0 incident.

What You Practiced

🚨 Assessing severity to choose the right workflow (hotfix vs. regular bugfix)
🌿 /agile-code-branch to create a hotfix branch from main (not develop), ensuring the fix is based on production code
🧪 /agile-code-tdd to reproduce the crash, then fix with confidence
⚙️ /agile-code-ci to verify no regressions in the full suite
👀 /agile-code-pr-review with expedited review (30 min SLA)
🔀 /agile-code-merge to merge to both main and develop, tag the patch release — all enforced automatically for hotfix branches
🚀 /agile-ship-deploy to push to production and run smoke tests
⏪ /agile-ship-rollback as the safety net if the deploy fails or causes new issues
📝 Writing a post-mortem with root cause and action items

Knowledge Check

Where does a hotfix branch from?

← Fix a Bug Code Review Workflow →