Software Testing

Software testing is an important topic. Testing is a lot like eating healthy and exercising. They are important, but often neglected when their is an upcoming deadline or other issue arise. In my recent reading of DevOps Handbook it made me think about this topic some more. This testing pyramid is well known in software development:

It is a pyramid, because the largest base should be the automated unit test, with less integration test, and fewer GUI test and manual test. In my real world experience I have seen this pyramid inverted with many manual test, many GUI tests, some module tests, and little unit test. This is the exact opposite of an ideal situation.

What are benefits of the testing? A few things:

• Automated testing process (as much as possible to reduce the effort of manual testing)
• Early feedback for failures (sooner is better!)
• Confidence when modifying source code (did it break? I hope not)
• Better software architecture (through practices like Test Driven Development).

Automation is key

There is also a tradeoff for these different parts of the pyramid. To execute a unit test is typically very fast (less than a second) as compared to an automated GUI test which can take minutes or hours in some cases with manual testing being the most costly. The benefit of the pyramid is to optimize the value/effort of the testing process and to cover more ground while getting the most value from each part. If the tests are automated it can save a lot of time because manual tests are not required for every module or component after a change.

Early Feedback for failures

Changes are constant in the software field. Every day, the code is being modified and committed by team members. If there is adequate test coverage, changes can be made with confidence. After a class is modified, the unit test can be run. Do these still pass? If the answer is yes, then you know the behavior before the class is the same as the behavior before the change. There is not as much worry about “Was some part broken while making changes?”. It is easier to answer this question.

It is good to know if something was broken but the timing of this feedback is also important. Let’s say, for example, a change is made. The testing is being done, and at night, a GUI test failed. The feedback took almost a day to get back, and it is hard to know — did my change cause this issue? Maybe? It is much better if a problem can be found quickly, and resolved. This is a win-win.

Changing with confidence

After a change is made, the question is often,

Does the rest of the code still behave the same?

When I first started I was more naive and thought, it will be easy to tell if the behavior is the same after making a change. In reality, this is not true. The problem is, after checking a few things, it is easy to think, “Okay, the behavior is the same.” But there are many edge cases you don’t see now, and could never even dream about. The behavior might look the same at first, but be warned, this might not be true.

It has happened multiple times that someone has changed software that I wrote. After a quick glance, I realized a few problems. It was this exact situation. The software appears to be the same, but it’s not quite the same. There are some edge cases that will cause issues soon.

Making changes with confidence. This is another benefit of testing and using this pyramid. It is not possible to completely remove this risk, but with testing it is possible to improve the situation for everyone on the team.

Software Architecture

Writing test can help to improve the architecture. When writing a test, the developer has to spend more time thinking about:

• What is the purpose for this class/module?
• How is the class used?
• What dependencies should it have?

Through writing unit tests for new classes, I have had to change the software architecture because the testing was challenging. This forced me to re-think the design of the classes.

I can think of one of the first times I wrote a unit test for a new class I created. In this particular situation, I started with one class. This class did many things, had logic, had some parameter subscriptions, it inherited from some other classes. Through the testing process, I realized it had many responsibilities, and found it beneficial (and necessary) to split this. At the end, there were 3 separate classes that could be tested together. This was a great learning opportunity to realize that

Writing unit test will force you to make better software architecture.

Other testing methods

While we are on the topic of testing, there are a few more types of test I would mention. I’ve learned about these recently and thought they are interesting:

— Doc Testing & Performance test.

Doc Testing

I learned about doc testing with elixir. Doc testing is really a unit test, but it is a certain kind of unit test. In elixir, you can create documentation for the functions inside of your module. This is in the form of some comments above the function names. This can be generated into an html document that can be used by other developers. As part of the documentation you can provide examples of using the module. In addition to generating documentation for your function, this example will also be executed as part of the unit tests. As a result, you get an additional unit test that will verify your documentation is accurate and current. How many times have you looked at documentation that is inaccurate or outdated? This can help with this situation.

Performance Testing

Performance is an important quality in software. The software should work, and as part of a good user experience, it should be performant. Performance testing can be used to identify when performance issues are introduced. The test is something like this —

Make a test with some predefined steps. Record the time for some key parts of this process and save it for the future. The next time this is executed, compare the execution time with the previous time. If this time is less than some threshold (say 1% longer) then the test passes. If the test takes significantly longer, than it is possible to quickly identify there is a performance issue that was recently introduced. If the issues are not identified now, 5 years down the road, the application will be slow, and then it is much harder to know the root cause. If they are quickly identified, they can also be fixed.