Sunday, February 1, 2015

enabling continuous deployment is not enough

One of the goals of my new team at LinkedIn is building a brand new platform for front end engineers so that they can deploy new features to production 3 times a day. However, we don't want to just "enable" other teams to deploy to prod daily. We want to automatically deploy. Always.

This is what I find critical in continuous delivery mind shift. Enabling (through automation) to go to production frequently is not enough. A team might be enabled to deploy to prod easily and automatically. However, only when the deployment is regular and automatic (for example: triggered on every push master - like in Mockito) then the change of team's habits, thinking and behavior truly begins. Team that is enabled to continuously deliver differs so much from the team that continuously delivers. At least, this those are my subjective observations :)

I chatted about it with Peter Niederweiser, the legendary creator of spock test framework. Spock is not continuously deployed at time of writing this post (this will change, right Peter!!!?? :). Publication of new Spock versions is more interesting than Mockito because the former needs to deal with groovy variants (compile, build and run against different groovy runtime, publish artifact variants, etc.). Given the complexity of the build process there is a temptation to automate everything to the point when all artifacts, documentation, release notes can be published via a one click. I'd say it's not enough. Enabling for continuous delivery is simply not as gratifying as doing it all the way through.

It's hard to explain what changes when the team does true continuous delivery. The attitude to a code commit is different. Commit messages get better. Tests are cleaner and deeper, more thoughtful. There is more vigor in battling flaky tests (fact of life: continous delivery == lots of tests == flaky tests). There is more energy to improve the automation even further. To make the automated release notes look better. So on, so on.

I'm on a mission to do continuous and automatic delivery and not merely enable it. I hope I managed to stir you up a bit, too :) Right now I'm lonely at Munich airport waiting for a connecting flight to Krakow. In a week, I'm moving my family over to Santa Clara CA. I got pretty agitated writing this post, hopefully the content makes sense. Keep in mind that it's my subjective experience.

Tuesday, January 6, 2015

Continuously delivering beta versions

I used to think that beta versions are lame. I associated “beta” with poor quality and with pushing validation to the users instead of ensuring high quality internally, through test driven development, clean design and excellent engineering practices. Now we are in progress of making Mockito 2.0 and we are using betas...

Beta version makes it possible to continuously deliver incompatible changes, incrementally. It is high quality: all tests pass, new coverage is meticulously added. Since there are several potentially incompatible changes, it would be hard to implement all of them in one go. We need more time and we still want to deliver continuously. Betas to the rescue!

It makes me think why making a new major version typically involves relatively significant effort. Why isn’t a major version release as simple as it should be? In theory one can develop a software library with high focus on compatibility: deprecate features and APIs with replacements, avoid incompatible changes at all cost, etc. In this mode, releasing a major version could be as simple as removing deprecated code, turning on some new behavior, changing the default settings and that’s it. A few commits and a single push. In practice, it is never that simple. For example, it might be costly to keep old and new behavior in parallel so that it can be toggled in the next major version. In this case, the authors may choose to remove the previous behavior and implement a new one as a part of major version implementation effort. In some cases, it might be actually the best for the end user because he could get the new features faster.

It makes me wonder about developing a software library that is always backwards compatible. I’m afraid it wouldn’t work. It feels that breaking changes are needed from time to time. Otherwise, the development can be slowed down or even paralyzed. Old features, APIs may stand in the way of innovation. It may become hard to tackle new important and challenging use cases. Industry does change, competition is restless, going after new ideas and use cases is unavoidable.

Being always compatible is often impractical. Major version releases typically involve extra effort. The latter bothers me. I’m going to work hard to avoid that in future. Continuous delivery makes releases hassle-free part of every-day routine. Semantic versioning and focus on compatibility make major version releases somewhat challenging and totally interesting. Hassle-free major version releases, on a regular basis, indicate very mature continuous delivery model. I want to be there.

Saturday, December 13, 2014

New day, new Mockito release

Mockito 1.10.15 that I have just released contains an interesting plugin for android testing. “I have released” is probably not the right phrase - I have coded for some time, pushed the changes to the master, went to sleep, this morning I took a look and I saw a new version published. How can I not love continuous deployment?

The full build and the release is relatively fast. Technically, I could have waited a couple of minutes before going to sleep. However, I was very confident the release process would proceed cleanly. I didn’t feel I needed to inspect automatically generated release notes. I didn’t feel I needed to inspect if the binary sits happily in Bintray’s jCenter. I didn’t feel I needed to do anything else but pushing code to the master. Quality code, of course, test-driven, designed and documented properly. I wish everybody took advantage of modern approach of continuous delivery. Being able to focus on quality code and avoid thinking about releases as they happen automatically, all the time, in the background.

I’m making progress on extracting the bits and pieces of the release automation out of Mockito so that the machinery can be reused in other projects. If you want to take a look how it is done you can fork Mockito on GitHub. Down the road I’m thinking about building a release toolkit that would have an opinionated framework sitting on top of it. The framework would most likely contain Gradle plugins. In the simplest case, it would be enough to apply the gradle plugin and augment it with a few lines of configuration in the build.gradle file. Then, the project fully embraces continuous delivery, engineers enter the new level of joy from developing code whereas the consumers smile more often as they get features frequently, in small increments that contain less bugs. That’s my dream :)

Tuesday, November 25, 2014

Continuous delivery of evolving API

Here’s what I believe high quality API means:

  • evolves over time as growing adoption triggers appearance of new use cases, new creative and unanticipated usages of the API
  • useful for the end users, solves real world use cases in an elegant way
  • backwards compatible, safe-to-upgrade, follows best of semantic versioning

Now the challenge is to agree above criteria with frequent releases, or even better: with continuously deployed releases.

Longer release cadence is somewhat tempting for API development: there’s plenty of time to get the model and the interfaces right. Getting the API right is the key to generate happy users and to avoid the need for breaking changes. However, slow release cadence introduces many problems including those that I dread the most:

  • release contains many changes and therefore it is riskier and have higher chance of introducing bugs. As long as there are changes in the source code and there are people using the software, there are bugs
  • new features reach the users late sometimes causing frustration and damaging adoption rate

I don’t want to release rarely. I’ve been there with Mockito and it was dark and gloomy.

Say you start developing a brand new library or a tool. In order to release features often and still maintain the freedom to change the API easily you might be tempted to start versioning from 0.0.1. Semantic versioning lets you get away with breaking changes before the official 1.0. The users might not be that forgiving so often pre-1.0 software authors care for backwards compatibility. This typically indicates that the software should be 1.0 already.

0.x.y versioning scheme works great to discourage potential users and can tamper adoption. There is something magical about 1.0. Regardless of the software quality and usefulness, if it’s not 1.0, it will be regarded by some groups as unstable and "not ready". I’m not against 0.x versions - I’m merely describing my observations of the software industry. The starting version of Mockito back in 2007/2008 was 0.9 and it reached 1.0 in few weeks once it was used in production. If you want to start versioning with 0.x just keep in mind the semver docs (as of late 2014):
How do I know when to release 1.0.0? 
If your software is being used in production, it should probably already be 1.0.0. If you have a stable API on which users have come to depend, you should be 1.0.0. If you're worrying a lot about backwards compatibility, you should probably already be 1.0.0.
Let’s get back to the original criteria of beautiful and useful API combined with continuous delivery. It’s hard. Even if the authors spend good amount of time and resources on designing the API they can miss out some use cases. The users will experiment, hack, break and push the API to the limits. Sooner or later there will be legitimate scenarios that call for changes in the API. Considering consistency and clarity of the API it may not always be possible to evolve the API in backwards compatible way. It’s time to introduce best friends of great API and frequent releases: @Deprecated, @Incubating and @Beta.

Gradle (and Mockito) has @Incubating. Guava (and Spock) uses @Beta. At the moment, the documentation pretty neatly describes what those annotations are used for but I want to emphasize the “why”. It’s all about the capability to release frequently that ultimately leads to better product and happy users. Early access to new features is fundamental for a high quality product. Frequent, small releases are critical to risk mitigation. API authors cannot afford to design the API for too long to ensure its stability and usefulness. API authors need real life feedback and concrete use cases. If the tool that you use has @Incubating or @Beta features it is a very good sign! It means that the authors care great deal for backwards compatibility and the API design (and they want to provide you with new features on frequent, regular basis).

Every incubating feature of Gradle or Mockito, or a beta feature from Guava and spock received exactly same amount of love from the authors as the public features. It passed exactly the same design process, brainstorming and validation. It has tons of automated tests. It was test-driven. It is a high quality feature that calls for your feedback.

Evolution of the API based on incubation and deprecation of the features, high regard for backwards compatibility, continuous feedback loop between authors and the users is what we need for successful continuous delivery of the APIs. For a tool author like myself, it's great fun to work this way :)

Monday, November 3, 2014

java packages and great architecture

I write code quite often and I keep pondering on how to do it better. I’ve always wanted to find a reusable pattern for high quality java design. I wanted the pattern to be simple, so that it can be safely installed in brains of young adepts of engineering and then grow with the experience. I wanted the pattern to be strong so that I can use it in my code. I think I’m finally getting close :)

I want highly comprehensible architecture, open to change and evolution. There are a lot of principles and practices that drive this goal, all well documented in the literature. I’m looking for a clear and simple implementation pattern that helps achieving the goal. Something that can be described in a single blog entry and that can be combined with the existing patterns.

Before revealing the actual implementation let’s zoom into somewhat underrated element of java: package-protected types. They are not used very often from my experience. Some engineers (including myself) use package-protected methods and constructors occasionally to simplify testing. Traditionally, all classes are public. Usually new classes are created public without too much thinking. IDEs generate new classes with ‘public’ identifier by default. In Groovy, classes are public by default (I still like Groovy a lot :). In general, I find all that quite disturbing. Package-protected types are lovable and they can be used to drive great object oriented design. It starts from package design.

Java package is a beautiful software design tool. Thinking hard about the contents of the package, the couplings between packages, knowing and eliminating package cycles, even choosing the right names and hierarchy for the package structure is one hell of a design tool that can significantly contribute to the quality of the architecture. When the package design is close to heart, the architecture can be comprehended literally by zooming in and out to the package structure.

I like to model java packages as software components. A component has the public API and the implementation details. The former is the official way of using the component. The latter is the stuff that we don’t want to leak to the consumers of the component. Standard practice in software component design is that the public API is a set of interfaces that neatly hide the implementation details. This approach allows evolution of the library in a way it is safe for consumers (compatibility) and convenient for maintainers (open to improvements). This is also fundamental for extensibility of the component’s API. What I suggest is to apply practices of component design, on a smaller scale, to the java package design.

The implementation may go like this: public concrete classes are discouraged. The only public types a java package can export are interfaces. If none of the classes are public how can the package be used "outside"? One approach is to relax the rule and allow some classes of the package to be public. There could be a single public class in the package that acts like a entry point to the features provided by the component. The entry-point class can use public interfaces to avoid leaking the internal implementation. Since majority of classes are package-protected, we have also the compiler and the IDE support informing what can be used from the component.

There’s also a puristic approach - it is actually possible to meet the ‘no public concrete classes’ requirement. Interfaces can have constants and therefore it is possible to expose component’s features without compromising visibility of any of the concrete classes:

//public interface exported by the package:
public interface HealthServices {

  //constant that exposes features but hides the impl
  public final static HealthServices API
    = new DefaultHealthServices();
    //impl class is package-protected

No doubt this model of designing of java code has limitations. Perhaps some other time I will write more about lessons learned from implementing this model and consequences on the architecture. I am super excited about this (though people tell me I get excited easily).

If after reading this you’re thinking more kindly of package-protected classes I’m already happy. If you’re more willing to start using classycle with your project I’m totally happy (in Gradle we apply use classycle.gradle script plugin). If you start considering java package design as an important part of architecture design I’m excited. If you try out some of the ideas or you have already used a similar approach let me know!

PS. I need ‘no public concrete classes’ t-shirt :)

Monday, October 20, 2014

Scaling down continuous deployment

Before I started working on Mockito 1.10 release I set a goal for myself. I decided that the next release, and every subsequent release will be completely automatic and will happen on ‘git push’ without even any ‘release’ button pressing. Getting the release out of the door, without investing into automation, would be tons easier. However, I din’t want to take this shortcut _again_. I wanted to consume the lessons learned from the past: shameful 2-year release cadence, totally _not_ reflecting my attitude and experience in continuous delivery and software automation. I wanted the 1.10 version to start the new era in Mockito ecosystem. Now I need to scale this down a bit :)

Driven by the ‘completely automated release’ use case I’ve set up Travis CI, Bintray, GitHub and the master tool: Gradle to let the magic happen. At the time, I was already thinking that releasing on every push might be too eager. However, I couldn’t come up with convincing use cases and it was getting hard to decide on the ‘right’ release trigger. So I’ve applied my favorite approach: get stuff DONE and then watch REAL use cases and feedback showing up.

We learned pretty quickly that certain category of project events don’t make sense as release triggers. Consider all kinds of non-code changes that occur surprisingly often: updates to, adding TODOs, minor changes to the automation machinery. We worked around this problem by including a “[ci skip]” incantation in commit messages. Travis CI does not trigger the build for changes annotated with “[ci skip]”. This approach helps a bit but removes the validation of the changes completely because the build does not run at all. Not to mention that it's easy to forget to include the magic string in the commit message which would lead to an awkward release. So, I’ve improved the automation by crafting a Gradle task that compares binaries between the current build and the previous release. The release is skipped if binaries are the same (the comparison method is simplistic).

The second category of events that I started observing is pretty interesting. Consider internal code refactorings, changes that do not affect any public API nor the software behavior. Those changes still update the code, so the byte code is different and the jars are different. Should such changes trigger a new publication of a library? What would be the content of the release notes?

I’m undecided on this matter yet but I’ll make the call soon. Interesting use cases are already showing up. For one of the recent PRs I’d love to get some refactorings done first, before the eager contributor picks up the main implementation. It feels that in this case, it’s more effective if I prepare a landing for a new feature, instead of continuing the pull request code-review cycle. This will push our current continuous delivery model to address the internal refactorings and find the right place for them in the process.

Thursday, October 9, 2014

maven central vs bintray

When I started developing continuous deployment pipeline for Mockito I was (naively) thinking that I could happily publish to Bintray and forget about Maven Central.

Not quite :)

The reason I started looking at Bintray in the first place was that I was hoping for simpler publication process. Specifically, I wanted to be able to publish a new version from a CI box I don't own (for example: Travis CI box). It's not quite that easy with Nexus OSS publication model. Maven Central fancies PGP signatures for the publications. To generate signatures, the gpg tooling is needed and the key ring with a private key. To perform the release from a box I don't own I need to provision the key and possibly gpg tooling, too. It's doable but kind of... hard. At least it seems hard enough to look at alternate solutions for publishing to Central (hey Bintray!).

There was one more gotcha with publishing directly to Nexus OSS. I would probably need to use Gradle's signing plugin for the pgp stuff. This plugin needs love - it does not yet work with Gradle's new beautiful publication DSL. The option of using old publication plugin was not appealing to me at all.

Setting up automated publication was a breeze with gradle-bintray-plugin. I became complacent. Everything was neatly set up: every git push triggered a completely automated release, an update to the release notes, a new real version published etc. A couple of days passed and the keywords "when", "Mockito", "Central" started dominating the community air. I was curious why users needed the jars in Central when they were available in Bintray's jCenter. There were a couple of good arguments and here's my favorite.

Happiness :) A lot of people still use Maven. I'm perfectly OK with this because they all will use Gradle eventually. It won't be long, Gradle is unstoppable. Maven projects use Central Repository by default so naturally, it's nicer if Mockito is available without any extra configuration. I'm happy that I don't have to maintain the documentation how to set up maven builds to consume Mockito dependency (yes, users were asking us about this). It's just nice when stuff works out of the box.

I really want to forget about Central but it turns out the world around me does not forget and pushes back. I continuously publish to Bintray and I enjoy very much the automation level it offers. I can sync to Central from Bintray (a step that I haven't yet automated). I don't have to mess around with PGP any more because I can hook up the key to "Mockito" organisation in Bintray UI and that's it. The whole Maven Central PGP requirement was questioned already by Bintray.

Thank you Bintray for making automation challenges simpler! Continuous deployment of Mockito wouldn't be there without you. I'd be still figuring out how to PGP from Travis CI and biting my nails with Gradle's old publication model (the new model is such a beauty!). In meantime Mockito users would wait another 2 years for the next release. Thank you Central for serving the jars to all Maven users (btw. jCenter can do this, too ;).

Next week I'm at JDD Conference in Krakow talking about continuous delivery. After that, I'm keen on getting out a nice blog post on how to cut down 2-year release cadence to 2 hours.