Using RSpec bisecting to find the minimal reproducer
Sometimes, code breaks in a way we don't even know where to start fixing. The time when tests are unpredictable and flaky. This is where RSpec bisecting comes to save the day by finding the minimal reproducer we can use to fix the root cause. Let's see how RSpec bisect works, its practical applications, and how it can improve your testing workflow.
What's bisecting
Test bisecting is a feature that helps developers identify the smallest subset of tests that cause a particular test to fail. This tool is invaluable when dealing with large test suites, where pinpointing the source of a flaky test can be like finding a needle in a haystack.
RSpec does that by running a binary search across your whole test suite, repeatedly halving the number of tests until it finds the minimal set of tests needed to reproduce the failure. It might not be your everyday feature, but a lifesaver on those days where things break apart.
How to run bisect in RSpec
RSpec will start bissecting if run with the --bisect
option. You can also combine it with your fixed --seed
:
$ rspec --bisect --seed 12345
Bisect started using options: "--seed 12345"
Running suite to find failures... (0.16755 seconds)
Starting bisect with 1 failing example and 9 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent
Round 1: bisecting over non-failing examples 1-8 .. ignoring examples 6-9 (0.30166 seconds)
Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5 (0.30306 seconds)
Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.33292 seconds)
Round 4: bisecting over non-failing examples 1-2 . ignoring example 1 (0.16476 seconds)
Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.26 seconds.
The minimal reproduction command is:
rspec ./spec/my_model_spec.rb[1:1] ./spec/my_model_spec.rb[1:1] --seed 12345
As you can see, RSpec will provide us with the most minimal reproduction command at the end.
If we cancel the test run by hitting CTRL-C, RSpec will still provide us a reproduction command it has discovered until that point.
A real-world example
Consider a test suite of 500 tests where one of the tests fails sporadically due to a race condition. Trying to manually pinpoint the right order to get a fixed reproducer would be time-consuming.
Using RSpec bisecting to automatically reduce the test suite to the minimal set of examples that triggers the failure will allow us to focus on the root cause.
Conclusion
RSpec bisecting is a powerful part of RSpec and one of those things the sister test framework Minitest lack. By automating the tedious part to find a minimal reproducer, it saves time, especially on a large codebase.
Hello, I am Josef and I am on Rails since its 2.0 version. I always liked strong conventions and the Rails Omakase docrine. I am author of Kamal Handbook and Test Driving Rails. I think testing should be fun, not a chore.
