Repeated tests: Don’t Run Your Ruby Minitest Classes Twice!

Minitest is a very popular testing framework for Ruby applications. It provides a complete suite of testing facilities supporting TDD, BDD, mocking, and benchmarking. Minitest is the Ruby test standard library included in the Ruby 1.9 version.

Minitest uses Ruby classes which support inheritance. If a Minitest class inherits from another class, it will also inherit its methods causing Minitest to run the parent’s tests twice.

I have seen different applications using subclasses in Minitest where the tests run multiple times, so I decided to write a post explaining why and in which cases the tests run. Also, I will show you a library that detects duplicated tests in your application.

Using subclasses in Minitest

Imagine that we have a class Product that has a Minitest class ProductTest. This class tests things like the title, the description and other functionalities and properties are correct.

Now, imagine that we want to add the feature of making the Product “great.” It will add extra functionality to the Product and potentially modify existing ones. To test it, a developer might want to try to create a Class GreatProductTest that subclass ProductTest like the following example:

ProductTest and GrandProductTest classes.

Now, if we run the tests, we see that it ran FOUR tests! And we only have three methods 🤔 How is that possible?.

Test execution for ProducITest and GrandProductTest classes.

The method test_parent ran twice!. The reason? It’s because Minitest uses Ruby classes, and the class will inherit all its parent methods. You can find a few issues in the Minitest repository mentioning this behaviour. This decision is by design, it can be questionable, but it’s not the purpose of this post.

Analyze your tests application with the Minitest Analyzer library

I have created a library that will help you detect duplicated tests in your code!. I ran the script in a few projects and cleaned up repeated tests I introduced 😂.

I applied reverse engineering to the Minitest source code to build this library. It was a great experience. It’s easy to follow the Minitest source code, and I’d suggest you look at it to see how it works.

The Analyzer library reads all the Minitest classes in your repository and builds an Acyclic Graph where each node contains information about the class and its methods. The edges are directional and determine the parent-child relationship between the classes.

Then, the graph is transversed to calculate the number of redundant tests per class doing count_of_class_test_methods * count_of_descendants per node. Then, it computes this information and prints the result in the console. The graph looks like this:

A graph containing duplicated Test information.

If we run the Ruby Minitest Analyzer library with the classes shown in the previous graph picture, we get the following results:

Minitest Analyzer against the last example graph.

The script tells us that there are 3 classes (GrandParentTest, ParentTest1 andParentTest2) that run duplicated tests, and there were 12 test runs that can be removed.

If a class doesn’t have any subclass then it doesn’t have any repeated tests. That’s the case for Child1Test, Child2Test and Child2Test.

Parent1TestClass has 2 descendants, which means that its method will be run 3 times: one for the class and two for its descendants. Similarly, Parent2Test has only one child; hence it will run its methods 2 times. Finally, GrandParentTest has 5 descendants; hence it will run its tests 6 times.

Summarizing each test will run:

Class GrandParentTest
test_a: 6 times -> 5 extra runs that can be avoided.
Class Parent1Test
test_b: 3 times -> 2 extra runs that can be avoided.
test_c: 3 times -> 2 extra runs that can be avoided.
Class Child1Test
test_y: 1 time
Class Child2Test
test_z: 1 time
Class Parent2Test
test_d: 2 times -> 1 extra runs that can be avoided.
test_e: 2 times -> 1 extra runs that can be avoided.
test_f: 2 times -> 1 extra runs that can be avoided.
Class Child2Test
test_o: 1 time
test_p: 1 time

A total of 12 extra runs can be avoided.

Conclusion

Writing Minitest subclasses is totally fine, but you must be aware that Minitest will run all the test’s parent(s) methods for each class since it uses Ruby classes.

If you are unsure whether you have duplicated test runs in your application, use the Ruby Minitest Analyzer gem.

Update: I added a Rubocop Cop that catches repeated runs of two classes in a single file. The cop isn’t as helpful as the gem since it doesn’t detect mixins or if classes are in different files, but it will definitely catch some cases and hopefully make devs aware of the behaviour.