Continuous delivery of evolving API

Here’s what I believe high quality API means:

• evolves over time as growing adoption triggers appearance of new use cases, new creative and unanticipated usages of the API
• useful for the end users, solves real world use cases in an elegant way
• backwards compatible, safe-to-upgrade, follows best of semantic versioning

Now the challenge is to agree above criteria with frequent releases, or even better: with continuously deployed releases.

Longer release cadence is somewhat tempting for API development: there’s plenty of time to get the model and the interfaces right. Getting the API right is the key to generate happy users and to avoid the need for breaking changes. However, slow release cadence introduces many problems including those that I dread the most:

• release contains many changes and therefore it is riskier and have higher chance of introducing bugs. As long as there are changes in the source code and there are people using the software, there are bugs
• new features reach the users late sometimes causing frustration and damaging adoption rate

I don’t want to release rarely. I’ve been there with Mockito and it was dark and gloomy.

Say you start developing a brand new library or a tool. In order to release features often and still maintain the freedom to change the API easily you might be tempted to start versioning from 0.0.1. Semantic versioning lets you get away with breaking changes before the official 1.0. The users might not be that forgiving so often pre-1.0 software authors care for backwards compatibility. This typically indicates that the software should be 1.0 already.

0.x.y versioning scheme works great to discourage potential users and can tamper adoption. There is something magical about 1.0. Regardless of the software quality and usefulness, if it’s not 1.0, it will be regarded by some groups as unstable and "not ready". I’m not against 0.x versions - I’m merely describing my observations of the software industry. The starting version of Mockito back in 2007/2008 was 0.9 and it reached 1.0 in few weeks once it was used in production. If you want to start versioning with 0.x just keep in mind the semver docs (as of late 2014):

How do I know when to release 1.0.0? 

If your software is being used in production, it should probably already be 1.0.0. If you have a stable API on which users have come to depend, you should be 1.0.0. If you're worrying a lot about backwards compatibility, you should probably already be 1.0.0.

Let’s get back to the original criteria of beautiful and useful API combined with continuous delivery. It’s hard. Even if the authors spend good amount of time and resources on designing the API they can miss out some use cases. The users will experiment, hack, break and push the API to the limits. Sooner or later there will be legitimate scenarios that call for changes in the API. Considering consistency and clarity of the API it may not always be possible to evolve the API in backwards compatible way. It’s time to introduce best friends of great API and frequent releases: @Deprecated, @Incubating and @Beta.

Gradle (and Mockito) has @Incubating. Guava (and Spock) uses @Beta. At the moment, the documentation pretty neatly describes what those annotations are used for but I want to emphasize the “why”. It’s all about the capability to release frequently that ultimately leads to better product and happy users. Early access to new features is fundamental for a high quality product. Frequent, small releases are critical to risk mitigation. API authors cannot afford to design the API for too long to ensure its stability and usefulness. API authors need real life feedback and concrete use cases. If the tool that you use has @Incubating or @Beta features it is a very good sign! It means that the authors care great deal for backwards compatibility and the API design (and they want to provide you with new features on frequent, regular basis).

Every incubating feature of Gradle or Mockito, or a beta feature from Guava and spock received exactly same amount of love from the authors as the public features. It passed exactly the same design process, brainstorming and validation. It has tons of automated tests. It was test-driven. It is a high quality feature that calls for your feedback.

Evolution of the API based on incubation and deprecation of the features, high regard for backwards compatibility, continuous feedback loop between authors and the users is what we need for successful continuous delivery of the APIs. For a tool author like myself, it's great fun to work this way :)

java packages and great architecture

I write code quite often and I keep pondering on how to do it better. I’ve always wanted to find a reusable pattern for high quality java design. I wanted the pattern to be simple, so that it can be safely installed in brains of young adepts of engineering and then grow with the experience. I wanted the pattern to be strong so that I can use it in my code. I think I’m finally getting close :)

I want highly comprehensible architecture, open to change and evolution. There are a lot of principles and practices that drive this goal, all well documented in the literature. I’m looking for a clear and simple implementation pattern that helps achieving the goal. Something that can be described in a single blog entry and that can be combined with the existing patterns.

Before revealing the actual implementation let’s zoom into somewhat underrated element of java: package-protected types. They are not used very often from my experience. Some engineers (including myself) use package-protected methods and constructors occasionally to simplify testing. Traditionally, all classes are public. Usually new classes are created public without too much thinking. IDEs generate new classes with ‘public’ identifier by default. In Groovy, classes are public by default (I still like Groovy a lot :). In general, I find all that quite disturbing. Package-protected types are lovable and they can be used to drive great object oriented design. It starts from package design.

Java package is a beautiful software design tool. Thinking hard about the contents of the package, the couplings between packages, knowing and eliminating package cycles, even choosing the right names and hierarchy for the package structure is one hell of a design tool that can significantly contribute to the quality of the architecture. When the package design is close to heart, the architecture can be comprehended literally by zooming in and out to the package structure.

I like to model java packages as software components. A component has the public API and the implementation details. The former is the official way of using the component. The latter is the stuff that we don’t want to leak to the consumers of the component. Standard practice in software component design is that the public API is a set of interfaces that neatly hide the implementation details. This approach allows evolution of the library in a way it is safe for consumers (compatibility) and convenient for maintainers (open to improvements). This is also fundamental for extensibility of the component’s API. What I suggest is to apply practices of component design, on a smaller scale, to the java package design.

The implementation may go like this: public concrete classes are discouraged. The only public types a java package can export are interfaces. If none of the classes are public how can the package be used "outside"? One approach is to relax the rule and allow some classes of the package to be public. There could be a single public class in the package that acts like a entry point to the features provided by the component. The entry-point class can use public interfaces to avoid leaking the internal implementation. Since majority of classes are package-protected, we have also the compiler and the IDE support informing what can be used from the component.

There’s also a puristic approach - it is actually possible to meet the ‘no public concrete classes’ requirement. Interfaces can have constants and therefore it is possible to expose component’s features without compromising visibility of any of the concrete classes:

//public interface exported by the package:
public interface HealthServices {

  //constant that exposes features but hides the impl
  public final static HealthServices API
    = new DefaultHealthServices();
    //impl class is package-protected
}

No doubt this model of designing of java code has limitations. Perhaps some other time I will write more about lessons learned from implementing this model and consequences on the architecture. I am super excited about this (though people tell me I get excited easily).

If after reading this you’re thinking more kindly of package-protected classes I’m already happy. If you’re more willing to start using classycle with your project I’m totally happy (in Gradle we apply use classycle.gradle script plugin). If you start considering java package design as an important part of architecture design I’m excited. If you try out some of the ideas or you have already used a similar approach let me know!

PS. I need ‘no public concrete classes’ t-shirt :)