Anatomy of a breaking change: one pull request, three teams, two alerts

The cast

The behaviour is Graphwake’s; the company, people and numbers are made up. Four teams are involved, though only one of them writes the change:

The visual at the top of this page shows that map for GET /orders/{id}. It comes from subscriptions: each consumer team marked the endpoints it calls. Graphwake uses that list to tell checkout and mobile-app and to leave search alone, so nobody has to remember who calls what during a review. It also shows storefront, a Discovery suggestion nobody has confirmed, and an endpoint with no subscriber.

The timeline

1. The spec changes in a pull request

John renames the response field total to totalAmount and adds a region query parameter so orders can be served from the nearest region. He makes region required, because the new code needs it. The spec change in his pull request looks like this:

# orders-api/openapi.yaml, pull request #42 (branch orders-region-param)
paths:
  /orders/{id}:
    get:
      parameters:
        - { name: id, in: path, required: true, schema: { type: string } }
+       - { name: region, in: query, required: true, schema: { type: string } }
      responses:
        "200":
          content:
            application/json:
              schema:
                properties:
                  id:          { type: string }
-                 total:       { type: number }
+                 totalAmount: { type: number }
                  currency:    { type: string }

As a code diff, this is a small pull request. As a contract it holds three changes, and two of them break callers that work today.

2. The CI step sends the spec as a draft

orders-api runs the Graphwake CI agent in its pipeline. On the pull request build, the agent sends the spec with the build’s context: pull request #42, its link, the author and the commit. Graphwake stores it as a draft for that pull request. The live spec stays exactly as it was, because nothing has been merged.

Because of this step, everything that follows happens before the merge. Without it, Graphwake can still poll a published spec URL every 30 minutes, but then it only sees the change after it is deployed.

3. Graphwake finds the breaking changes

Graphwake compares the draft with the live spec, endpoint by endpoint. We call a change breaking when it can make an existing client fail, which in practice means one of these:

• an endpoint or a response field is removed;
• a new required parameter or required request field is added;
• a field’s type changes;
• the accepted values are narrowed, for example an enum loses a value.

The rule of thumb behind the list is whether a client that follows today’s contract keeps working. Adding something a client can ignore is safe. Taking away something it reads, or demanding something it does not send, breaks it. A team that wants to hear about the safe changes too can opt in to every change.

4. The impact score is computed

Each change set gets a score from 0 to 100, made of four parts with public rules. Each part counts something, gives a fixed number of points per item and stops at a cap, so no single part can dominate. The impact score page has the full rules. For #42:

• Breaking changes: 2. The removed field and the new required parameter. The added field does not count.
• Affected teams: 2. checkout and mobile-app subscribe to GET /orders/{id}. search does not, so it does not count.
• People to notify: 10. The distinct members of the affected teams: 4 in checkout and 6 in mobile-app, nobody in both. Someone who follows the endpoint personally gets the alert by email but is not counted here.
• Blind spots: 0. Every endpoint with a breaking change has at least one subscriber, and no endpoint was removed.

54/100 is Medium. Low runs from 0 to 29, Medium from 30 to 59, and High is 60 and above. We publish the rules so that a reviewer can check every point.

Blind spots work differently. Suppose #42 had also removed GET /orders/{id}/invoice, which no team subscribes to. That is a third breaking change (30 instead of 20) and a blind spot (+10): 74/100, High. The score goes up because Graphwake has nobody to tell. When that happens, Discovery suggests teams that may be affected, each with a reason, and a suggested team becomes a subscriber only when someone confirms it.

The visual below is not pull request #42. It shows a larger Example, pull request #1284 in the same service, with three breaking changes and three affected teams, to show what a High score looks like: 30 + 36 + 13 people = 79/100.

5. Two teams are alerted, the third is not

With “Alerts on open pull requests” turned on in settings, Graphwake sends the alert while #42 is still open. It goes to the teams subscribed to the changed endpoint, each in the channel that team chose: checkout in #team-checkout, mobile-app in #mobile-alerts. A team that chose Microsoft Teams or email gets the same information there.

Breaking change in PR #42
Service: orders-api
Endpoint: GET /orders/{id}
Owned by payments-team
Change: <the removed field>; <the new required parameter>
Author: john
PR: https://github.com/example/orders-api/pull/42

The message carries what a consumer needs to act on it: the endpoint, what changed, who changed it and where to comment. The impact score stays in Graphwake, on the change set, next to the list of teams it reached.

search gets nothing. It calls orders-api, but not the endpoint that changed. In Graphwake’s impact view it is listed among the teams that consume the service but none of the changed endpoints, so the producer can see that it was considered and that skipping it was deliberate.

While the pull request stays open, Graphwake keeps one message per team and endpoint. If John pushes again, the existing message is edited in place instead of a new one being posted. An edit notifies nobody, so if a push makes a change breaking that was not before, Graphwake adds a thread reply: “This change is now breaking.” If a push removes the breaking change, the message is updated to say it is no longer breaking.

6. The teams prepare, then the pull request merges

checkout starts sending region, mobile-app moves to totalAmount, and each team marks the change as handled for itself in Graphwake. That conversation happened during the review. Usually it happens after an incident.

John merges #42. The default-branch build sends the spec as live, and the agent reports that the commit came from pull request #42. Graphwake settles the draft as merged: each team’s message is updated to “Published: Breaking change in PR #42”, with the thread reply “Published to live.” The handled marks carry over, because the breaking changes were the same in the draft and at the merge. Had John closed the pull request instead, the reply would have been “PR closed without merging.”

7. The change history keeps the record

The merge is recorded in the change history of orders-api and of GET /orders/{id}: the change set, its breaking changes, the team that published it, and which teams were told in which channel. The history is append-only. Months later, when someone asks why checkout knew about the region parameter, checkout’s subscription to the endpoint links to the change it was told about.

If an old pipeline on the main branch is re-run later, its spec does not quietly undo #42: Graphwake ignores a live spec from an older commit than the one it already holds.

Why routing beats broadcasting

The usual alternative is a post in a shared channel, and the hope that everyone who calls the endpoint reads it. In this example that means asking the whole company to work out whether they call GET /orders/{id}.

• With subscriptions, each consumer team records what it calls once, when it starts calling it. Nobody has to repeat that matching every time another team ships.
• search heard nothing in this example, and that tells search something: it is not affected. In a broadcast channel, hearing nothing only means you missed the post.
• An endpoint nobody subscribes to shows up as a blind spot and raises the score, where a broadcast would leave it to surface in production.

More on this in stop announcing API changes in Slack and API breaking change alerts.

How Graphwake does it

• Before the merge

  The CI step sends the pull request’s spec as a draft, so consumer teams can hear about a breaking change while it is still under review.

• Only the subscribed teams

  Consumer teams subscribe to the endpoints they call. A breaking change goes to those teams in Slack, Microsoft Teams or email, not to a company-wide channel.

• A score you can check

  Each change set gets a score from 0 to 100 from four public rules, shown line by line in Graphwake.

• A record that stays

  Merges, the teams told and their channels are kept in an append-only change history for each service and endpoint.

What to take from this

1. Review spec changes as contract changes. A rename is a removal, and a new required input breaks every caller that does not send it.
2. Count who is affected before the merge, not after. Two breaking changes on an endpoint with two consumer teams is a different conversation from the same changes on an endpoint nobody calls.
3. Tell the teams that call the endpoint, in their own channel, and let everyone else stay quiet.

For the review side, see the API breaking change checklist. For more examples of what counts as breaking, see OpenAPI breaking changes.

Questions

Related

• GuideAPI breaking change alerts: how to tell the right teams
• How it worksHow the 0-100 impact score is built
• How it worksOne Go binary, five CI systems: inside our open-source CI agent
• GuideOpenAPI breaking changes: a reference with examples
• OpinionStop announcing API changes in a shared Slack channel
• GuideAPI change notifications in Slack and Microsoft Teams