Beware of powerful test tools

This article was originally posted on LinkedIn.

Would you feel comfortable allocating your personal time to solving engineering use cases such us mocking private or static methods? I don’t feel too comfortable, but I’m pushing forward. Neither private or static methods should be ever mocked. Instead, engineers should write clean code and wrap legacy code with an abstraction layer.

Life is a sequence of experiments, leading to learning, leading to more experiments.

Helping Powermock to integrate cleanly with Mockito is an experiment. We will see how it impacts our point of view on testing legacy code, our skills in engineering changes across 2 separate Open Source projects, and our community, which will be given more powerful test tools.

The more powerful test tools are, the more awkward code can be written without punishment.

If something is hard to test, it’s a punishment for poorly designed, untestable code. That punishment is a strong signal to refactor the code. If the punishment is gone, as well can the refactoring.

We are getting really close to finalize Mockito API improvements that will enable clean static mocking with Powermock (hmmm, is there such thing as ‘clean static mocking’? :). Mockito is opinionated and for the time being we don’t plan to provide static mocking out of the box. However, Mockito will have flexible API so that frameworks like Powermock can cleanly integrate and offer features needed for testing legacy code.

After a long effort with Arthur Zagretdinov, the remaining problem was mocking constructors. At first it appeared as a really hard problem to tackle because the entire Mockito API was geared towards mocking methods and not constructors. Changing the API to work on higher abstraction than a Java method would mean a lot of rework and public API changes. Solution: create a sweet little interface that can adapt constructors to methods!

The work continues in GitHub ticket 1110. I predict that sometime in Q3 2017, the community will get clean static mocking with Mockito.

Mockito and Powermockito integrated properly, soon!

Authors of libraries that are integrated should work together on the APIs. This way, the products are of higher quality and the integration is safe for their users. I published this article originally on LinkedIn.

Enable static mocking in Mockito? This dilemma might be moot very soon.

Powermockito is a part of Powermock, a library that helps mocking static methods and other unmockable things. After many years of developing Powermockito in isolation, we now work together on exposing new public APIs from Mockito, so that using Powermockito is safe. Why is it unsafe now? Powermockito uses Mockito internal APIs and copies some Mockito code. The resulting solution is brittle, very susceptible to version changes and falls over on various edge cases. Arthur Zagretdinov, current maintainer of Powermockito engaged with Mockito team and we are making nice progress on cleaning this up. For Mockito team, it is a challenging engineering exercise. We need to design the integration APIs so that 3rd party tools such as Powermock can implement clean solutions on top of Mockito.

Opinionated but not dogmatic

Disclaimer: Mockito team thinks that the road to hell is paved with static methods. However, Mockito's job is not to protect your code from static methods. If you don’t like your team doing static mocking, stop using Powermockito in your organization. Mockito needs to evolve as a toolkit with an opinionated vision on how Java tests should be written (e.g. don't mock statics!!!). However, Mockito is not dogmatic. We don't want to block unrecommended use cases like static mocking. It's just not our job.

Clean integration needs teamwork

Powermockito currently has code copied from Mockito. When we finish, there will be zero code copied from Mockito. That's how it should have been done long time ago. When a library needs to deeply integrate with other library, the teams should cooperate rather than copy the code over and use internals. I absolutely respect Powermockito developers - sometimes you have to use internal APIs to make progress. The time we can allocate to Open Source is sometimes a fraction of what is actually needed to develop a clean solution. Even if Powermockito pushed Mockito team harder on providing the necessary APIs, there is no guarantee Mockito core developers have had the time to implement them. Fortunately, in mid 2016 Mockito team got some fresh blood. We shipped Mockito v2 in October last year and we continue shipping nice incremental improvements ever since. For example, check out strict stubbing feature for cleaner tests and improved productivity. Now it's time to sort out the integration with Powermockito.

Arthur, keep the steam up and let's complete this project! I promise that Mockito team will find time to implement the APIs. Exciting!

Simple and practical continuous delivery for your library

Below article was already shared with you on LinkedIn in early May. We are in middle May so let's refresh it and fill out any gaps! This article is about Mockito release tools, not so much about mocking Java code. Let's get started!

Imagine...

Imagine the world where you call pull in a new version of some Open Source library and not worry if it breaks compatibility. Imagine that you can submit a pull request to some project, have it reviewed timely, and have the new version with your fix available to you in minutes after your PR is merged. Imagine that for any dependency you consider upgrading, you can view its neatly and consistently maintained release notes. Imagine that you can set up practical Continuous Delivery automation in your project in minutes, by using a well behaving and documented Gradle plugin. Imagine that you can focus on code and features while the release management, versioning, publishing, release notes generation is taken care for you automagically.

This is the goal of the new Open Source project that me and a couple of friends started cooking together. We used to call it "mockito release tools" because it originates from the release automation goodness we developed for http://mockito.org project. Mockito is the most popular mocking framework for Java and it is used by 2M users.

We found a name that captures the our mission very well. "Shipkit" - a toolkit for shipping it! Check it out at http://shipkit.org

True North Star

The goal is to provide easy-to-setup Continuous Delivery tooling. We would like other teams to take advantage of rapid development, frictionless releases and semantic versioning just like we do it in Mockito. We plan to make our release tools generic and neatly documented. It will be a set of libraries and Gradle plugins that greatly simplify enabling Continuous Delivery for software libraries.

Beyond the North Star

The project is getting some steam and there are already 4 contributors. I wanted to shout out what we are doing because... we are getting more ambitious and the goals of the project are getting broader in scope. After all, if we don't want to conquer the world, then why bother? :)

We need help

If the project goals speak to your engineering heart join our efforts! You can help with

• Implementing features - see issues marked with "please contribute" label.
• Spreading the word of what we're doing, letting us know about other project with similar goals. You can our issue tracker for reaching out.

Mockito Continuous Delivery Pipeline 2.0

What do you think about Mockito release model? This article shows the diagrams of proposed changes to Mockito continuous delivery pipeline. Those are high level diagrams. They lack comprehensive commentary yet, but I will add more context and explanations soon.

Current Mockito Continuous Delivery Pipeline (described in details on our wiki page):

Mockito Continuous Delivery Pipeline 1.0

Given community feedback (documented in issue 618) we proposed a plan to change Mockito's release cadence (documented in issue 911).

Proposed Mockito Continuous Delivery Pipeline:

Mockito Continuous Delivery Pipeline 2.0

Do you have feedback to share? Join the discussion in issue 911 or drop me a comment.

Clean tests produce clean code - strict stubbing in Mockito

Clean tests produce clean code. It’s close to impossible to write clean tests for dirty code.

Do you agree with above? If you do (which I sincerely hope so) read on about a brand new concept in Mockito, the “strict stubbing” feature. In a nutshell it provides:

• better productivity by detecting incorrect stubbing early (covered in great detail in this article).
• cleaner tests by eliminating unnecessary stubbings (somewhat covered, assuming it is obvious)
• DRY (Don’t Repeat Yourself) test code by verifying stubbing automatically (sadly, not covered in this article, if you want to know about it more leave me a comment :).

Stubbing in Mockito is considered delightfully simple and intuitive:

//given

List mock = mock(List.class);
given(mock.get(3)).willReturn("ok");

//expect

assert mock.get(1) == null;
assert mock.get(3) == "ok";

"Things should be as simple as possible, but not simpler" said Albert Einstein. Mockito stubbing is very close to Enstein’s simplicity boundary. There are times when returning null or empty values when stubbing argument does not match causes a lot of pain to our users. The user can stare at the code not understanding why mocks don’t behave as they are configured to. It happens when mocks return nulls instead of stubbed values configured in the test. The user resorts to debugging and stepping program execution to figure out what’s going on.

I will stop here because I feel uncomfortable complaining about the API I designed myself… I hope that "stubbing argument mismatch” scenario does not impact your productivity too often and you still enjoy cutting some great tests with Mockito.

When you hit stubbing argument mismatch scenario with Mockito 1.x here are your resolution options:

• reviewing code: double checking the arguments passed to mocked method in the test and in the code
• debugging: stepping program execution, inspecting the value of arguments. Using debug statements
• reducing complexity: temporarily removing logic to oversimplify the code, temporarily replacing stubbing arguments with permissive argument matchers like "any()"
• leveraging Mockito API: Mockito.RETURNS_SMART_NULLS
• migrating to latest Mockito! (finally I get my punchline)

Mockito 2.1 comes with JUnit Rule that prints warnings to the console when stubbing argument mismatch is detected. When you stub a method with some argument(s) but then the same method is invoked with different argument(s), a warning is printed. Mockito’s painstakingly comprehensive Javadoc documents this behavior in MockitoHint class. (BTW. MockitoHint is a marker interface, existing only for documentation and Javadoc linking. Isn’t it a neat idea?)

The reason Mockito produces a warning instead of an immediate test failure is because... it’s really a “warning" not an “error". Sometimes, it is actually desired to call the stubbed method many times, with different arguments, some of those args match and some don’t. Arguably, those scenarios are quite rare. Originally I even had an example here in this blog post describing this use case in more detail. Eventually, I got rid of the example, it was not compelling enough. Take my word for it: there are legit cases where stubbing argument mismatch is _not_ an error. I failed to produce an example for this use case, perhaps you can share an something from your tests?

Mockito 2.1 also contains improved behavior of JUnit Runner which detects unused stubbings and reports them as failures. This helps keeping the tests clean. This feature is not directly connected to stubbing argument mismatch scenario. However, it is a part of the same design thought - making Mockito stubbing stricter for cleaner tests and improved productivity.

Just like the JUnit Rule, the Runner also prints warnings on stubbing argument mismatch. Have you noticed those warnings? Do you find them useful? Here are the traits of the warning system:

• Warnings are useful only when developers actually pay attention to the console output. 
• Program output can interleave with the warnings. Given enough noise in the output the warnings will simply be lost.
• Warnings can interleave with actual test failure which can lead to confusing test result. The user sees both: the warning and the error and tries to figure out what is the problem to fix.

To address the caveats with warnings, Mockito 2.3 comes with "strict stubbing" option for JUnit Rules. This feature makes the stubbing argument mismatch fail fast. The user immediately gets feedback that there something fishy about the stubbing in test _or_ the use of stubbed method in the production code. This new fail-fast behavior might be undesired for certain use cases therefore strict stubbing can be turned off per test. Strict stubbing is tentatively planned as default behavior for Mockito version 3 because we believe it improves productivity. We would really appreciate if you tried out strict stubbing and let us know how it works for you.

Strict stubbing with JUnit Rules:

@Rule public MockitoRule mockito = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);

Mockito 2.5.2 adds strict stubbing support for JUnit Runner:

@RunWith(MockitoJUnitRunner.StrictStubs.class)
public class SomeTest {

  //...

}

Currently strict stubbing in Mockito is only available with JUnit Rule or JUnit Runner. This is not a great news for TestNG fans or users that for some unknown reason decided to not use Mockito’s JUnit support (Why not? It’s really good for you. Let us know why!). We currently work on adding strict stubbing support without the need to use JUnit Runner or JUnit Rule. It will unlock the feature to everybody. This feature will be merged within few days so it’s your last chance to provide feedback to the design note or the pull request.

If you got that far I can only thank you for reading. Let us know what you think about strictness in Mockito.

May your tests be blessed with clarity!