Researchers Explore the Cost of API Breaking Change

“Move fast and break things.” The one-time mantra of Facebook developers has become engrained in the consciousness of software developers from Boston to the Bay Area. But change comes with a cost; and how software development communities manage the cost of breaking changes - i.e., changes that cause software using the code to fail - is the topic of a recent study by Carnegie Mellon University researchers.

Today, software development takes place in the context of large ecosystems that provide reusable code fragments, libraries, and frameworks that accelerate development and enhance functionality. Yet it also makes projects vulnerable to change by developers who control these resources.

Looking across three software development ecosystems - Eclipse, R/CRAN, and Node.js - researchers from Carnegie Mellon’s Institute for Software Research have uncovered that differing community values lead to differences in practices, tools, and policies that dramatically shift the costs of change between the developer initiating change and the users of the code.

For example, in the case of Eclipse, the community values stability and works to prevent breaking existing client code. “As a result, Eclipse features yearly scheduled releases with an eye towards backwards compatibility that makes it easy in principle for users to always use the latest version without worrying about incompatibilities” notes Chris Bogart, one of the lead researchers on the project. “We can contrast this with Node.js, in which the community values developer autonomy and therefore places a great deal more emphasis on innovation and iterative design. While convenient for package developers, this can result in in great deal more rework on the part of users, as updates are quite frequent and sometimes disruptive.”

On the surface, it is easy to say that the faster-moving Node.js community would be the obviously superior choice. It would be more responsive to demand and would more rapidly develop the functionality that users seek. However, Bogart notes, there are trade-offs in each of the environments they studied. “Many would say that Eclipse with its focus on stability is stagnating and not attractive to new contributors. And while this may be true, Node.js’s rapid release cycle means that users struggle to keep up. A lot of time and money is spent just keeping their software up to date.”

The tradeoffs and practices each community employ to manage the cost of change is also, Bogart points out, not readily apparent outside the boundaries of each ecosystem. “Many of the participants we spoke to in our study were surprised and interested to learn more about what their peers in other environments were doing to manage this burden.” Bogart explains that this suggests there simply isn’t enough transparency. “This opaqueness may lead to misconceptions and may cause development teams to discount a particular ecosystem which may, in fact, be well suited to their needs.”

Extending the scope of their study to other development ecosystems, Bogart hopes they can expand upon their understanding of how these breaking changes are handled. “We have only looked at three ecosystems; but there are many more out there. We need to hear from members of these communities.” To this end, Bogart and his collaborators have launched a website which covers their findings in greater depth as well as allows developers to contribute their experiences with breaking changes via an online survey.

For more information or to participate in the online study, please visit http://breakingapis.org.