is-promise post mortem

Last Saturday, I made the decision to try and catch up on some of the many contributions to my open source projects. One of the first pull requests I decided to merge was one that adds a TypeScript declaration file to is-promise .

After merging it, I decided it would also be a good time to update the module to support ES Module style imports . Specifically I wanted to be able to import isPromise from 'is-promise'; without needing to have synthetic default imports enabled. After this, I ran the tests, which passed, and published a new version.

I had been intending to set up more of my projects to be automatically published via CI, instead of manually published from my local machine, but because is-promise is such a tiny library, I figured it probably wasn’t worth the effort. This was definitely a mistake. However, even if I had setup publishing via CI is-promise may not have had sufficiently thorough tests.

Mistakes

It turns out, I had made lots of mistakes in publishing this new update:

• Because the repository had a .npmignore file, I had assumed it wouldn’t have a "files" array in package.json, so I failed to update that array to include the new index.mjs file.
• When adding the "exports" field to package.json as a way to tell newer versions of node.js about the ES Modules version, I had assumed the paths worked the same as "main" but instead they require the ./ prefix.
• I had not understood that the "exports" field to package.json restricts not just what you can import, but how you can import it. So even though my change allowed you to import the index.js file, it would break your code if you were doing require('is-promise/index') or require('is-promise/index.js') .
• I also hadn’t considered the fact that the "exports" field in package.json prevented you from importing require('is-promise/package.json') .

The restrictions imposed by "exports" are probably good in the long run, but they really necessitate testing, and they made this a breaking change.

Timeline

• 2020–04–25T15:03:25Z — I published the broken version of is-promise . The primary issue is the "exports" field in package.json
• 2020–04–25T17:16:00Z — Ryan Zimmerman submitted a pull request that fixed things for most people.
• 2020–04–25T17:48:00Z — I receive a Facebook message asking me to check GitHub. This is the first point at which I realise something is wrong.
• 2020–04–25T17:54:00Z — I merge and release Ryan’s pull request . For the majority of people, this fixes the library, but many people still have cached versions and some people were importing via odd paths that "exports" in package.json now blocks.
• 2020–04–25T17:57:00Z — Having read through the various issue comments, I close them all and create a fresh ticket so we can have more productive discussion.
• 2020–04–25T18:06:00Z — Jordan Harband explains to me why the "exports" is still a breaking change.
• 2020–04–25T18:08:08Z — I remove "exports" from package.json altogether and release the fixed version
• 2020–04–25T19:20:00Z — I un-published the broken versions, in an attempt to force anyone who had lock files to update from them.

In total, the library was broken for approximately 3 hours.

Contributing Factors

Various factors enabled me to make the mistakes I made:

• Releasing from my local machine always makes it tempting to skip the important steps of creating a pull request and letting CI test my changes.
• Our tests only covered the actual code, they did not check what was published to npm.
• We were not testing on the latest version of node.js
• I was not easy to reach during this incident. Although there were multiple contributors on the GitHub repository, they did not have permission to deploy new versions.

Steps taken to prevent future problems

• Added node 12 and 14 to the CI test suite ( https://github.com/then/is-promise/pull/21 )
• Added tests that use npm pack to test the actual published API ( https://github.com/then/is-promise/pull/25 )
• Removed .npmignore to reduce chance of confusion ( https://github.com/then/is-promise/pull/28 )
• Made publishing automatic from CI using Rolling Versions ( https://github.com/then/is-promise/pull/25 )

Rolling Versions & is-promise 3.0.0

After this incident, I decided to set everything up to be as robust as I possibly could. I also decided that the safest thing to do was to publish all these changes as a new major version, to avoid breaking things for people even if they were using un-documented approaches to import.

I’ve spent the last few months building Rolling Versions, which is a tool to help you safely publish packages via continuous integration, along with great change logs. I have now added this to is-promise, which gives me much greater confidence in future releases.

Releasing from continuous integration is one of the biggest things you can do to help prevent incidents like this. Writing change logs is also a really effective way to help you review your own changes and think about their impact. This is really only effective if you write the change logs as part of the pull request, rather than after the fact. You can see the increase in detail in the change log as I move from writing them as an afterthought (< 3.0.0) to writing them as part of the pull request (≥ 3.0.0): https://github.com/then/is-promise/releases

If you want to know more about Rolling Versions, or would like to be kept up to date on my thoughts on software development and open source, you can subscribe to my mailing list here: