The lack of test automation, in most products, can be a crippling debt in trying to do quicker product learning through continuous delivery. This test gap makes delivery in small increments uncertain – when things fail unpredictably is it the code, the environment, or something else? Additionally, test debt can make fast product learning cost prohibitive. What are some things we can try?
- Start talking about tests earlier
- Use visual indicators for challenge areas
- Listen to what the tests are telling you
- Invest in test automation in product context
- Know your testing will need upkeep
Start Talking about Tests Earlier
The best time to start thinking about the tests is the beginning. When teams are chartering, we coach them to think of examples of what success looks like for the product we want to build or enhance. Using examples early makes it less of a stretch to have better tests when it is time to work on stories. As we narrow our product learning window – using personas, mapping, and journeys – we continue to use more focused and fine grained examples. ‘When our business traveler chooses a seat on their flight, if their preferred seat is not available, we should provide a message.’ That is a nice example, is testable, and able to be automated.
Use Visual Indicators for Challenge Areas
I like to use visual indicators with teams – not because it is the cool lean thing to do but rather it is a nice trigger / reminder for conversations. For test debt, I use a variety of visual indicators. Something as simple as test duration run over time or a visual tracking of challenging areas to test. I also am a fan of SonarQube and using its ‘hotspot’ feature to quickly show me high risk areas. Keeping these things visible doesn’t solve the problem but creates an environment where the important areas get attention.
Listen to what the Tests are Telling You
Frequently teams are challenged to start testing not for lack of want but because the code wasn’t created with testing in mind. When you try and write a test, the test code tells you ‘Yo, this is jacked.’ That is a trigger for refactoring. When you see your tests getting big and complex, that is a trigger for refactoring. When your tests are running a long time, need to run at night, or we don’t trust the test automation – these are all signs that the tests are telling you to look at your design.
Invest in Test Automation in Product Context
So you have a big gnarly code base and are looking to invest in test automation. One pitfall we see teams fall into is arbitrary test automation, usually because of an arbitrary ‘code coverage’ measure being established. In my experience, the easiest tests in nasty code bases tend to have less value. Instead of just adding some tests with the goal of getting some test coverage, we coach teams to look at the product they are building and ask – what would be the most valuable test to start with. Remember, this valuable test may tell us something – that we need to refactor. But having that first valuable test then starts making the refactoring safer.
Know Your Testing will Need Upkeep
Testing and test automation is not a one and done thing. Your tests live, just like your product and code live, and your testing will need to be continuously manicured as the product needs and learnings grow. Testing is not just about collecting the most tests but rather about having tests at the right levels and removing tests that have lost their value.
Addressing test debt is by no means an easy task. It is easy to set up measures around ‘code coverage’. It is much more difficult to embrace testing as a first class citizen, a product and design tool. In organizations where code coverage is a focus, testing is more viewed as a cost center than a product and design tool. Organizations that are invested in TDD don’t focus on code coverage but rather focus on the tests.
What have you done to start addressing test debt?