Repeated tests: Don’t Run Your Ruby Minitest Classes Twice!
Minitest is a very popular testing framework for Ruby applications. It provides a complete suite of testing facilities supporting TDD, BDD, mocking, and benchmarking. Minitest is the Ruby test standard library included in the Ruby 1.9 version.
Minitest uses Ruby classes which support inheritance. If a Minitest class inherits from another class, it will also inherit its methods causing Minitest to run the parent’s tests twice.
I have seen different applications using subclasses in Minitest where the tests run multiple times, so I decided to write a post explaining why and in which cases the tests run. Also, I will show you a library that detects duplicated tests in your application.
Using subclasses in Minitest
Imagine that we have a class
Product that has a Minitest class
ProductTest. This class tests things like the title, the description and other functionalities and properties are correct.
Now, imagine that we want to add the feature of making the Product “great.” It will add extra functionality to the Product and potentially modify existing ones. To test it, a developer might want to try to create a Class
GreatProductTest that subclass
ProductTest like the following example:
Now, if we run the tests, we see that it ran FOUR tests! And we only have three methods 🤔 How is that possible?.
test_parent ran twice!. The reason? It’s because Minitest uses Ruby classes, and the class will inherit all its parent methods. You can find a few issues in the Minitest repository mentioning this behaviour. This decision is by design, it can be questionable, but it’s not the purpose of this post.
Analyze your tests application with the Minitest Analyzer library
I have created a library that will help you detect duplicated tests in your code!. I ran the script in a few projects and cleaned up repeated tests I introduced 😂.
I applied reverse engineering to the Minitest source code to build this library. It was a great experience. It’s easy to follow the Minitest source code, and I’d suggest you look at it to see how it works.
The Analyzer library reads all the Minitest classes in your repository and builds an Acyclic Graph where each node contains information about the class and its methods. The edges are directional and determine the parent-child relationship between the classes.
Then, the graph is transversed to calculate the number of redundant tests per class doing
count_of_class_test_methods * count_of_descendants per node. Then, it computes this information and prints the result in the console. The graph looks like this:
If we run the Ruby Minitest Analyzer library with the classes shown in the previous graph picture, we get the following results:
The script tells us that there are 3 classes (
ParentTest2) that run duplicated tests, and there were 12 test runs that can be removed.
If a class doesn’t have any subclass then it doesn’t have any repeated tests. That’s the case for
Parent1TestClass has 2 descendants, which means that its method will be run 3 times: one for the class and two for its descendants. Similarly,
Parent2Test has only one child; hence it will run its methods 2 times. Finally,
GrandParentTest has 5 descendants; hence it will run its tests 6 times.
Summarizing each test will run:
test_a: 6 times -> 5 extra runs that can be avoided.Class Parent1Test
test_b: 3 times -> 2 extra runs that can be avoided.
test_c: 3 times -> 2 extra runs that can be avoided.Class Child1Test
test_y: 1 timeClass Child2Test
test_z: 1 timeClass Parent2Test
test_d: 2 times -> 1 extra runs that can be avoided.
test_e: 2 times -> 1 extra runs that can be avoided.
test_f: 2 times -> 1 extra runs that can be avoided.Class Child2Test
test_o: 1 time
test_p: 1 time
A total of 12 extra runs can be avoided.
Writing Minitest subclasses is totally fine, but you must be aware that Minitest will run all the test’s parent(s) methods for each class since it uses Ruby classes.
If you are unsure whether you have duplicated test runs in your application, use the Ruby Minitest Analyzer gem.
Update: I added a Rubocop Cop that catches repeated runs of two classes in a single file. The cop isn’t as helpful as the gem since it doesn’t detect mixins or if classes are in different files, but it will definitely catch some cases and hopefully make devs aware of the behaviour.