Ask This: How Is Your Software Tested?
If three test cases pass, it’s pretty much proof by induction that it works.
— Professor Matthew Might (@mattmight) November 8, 2013
At SC13, highly technical parallel computing software will be discussed, demoed, denounced, and debuted. Whenever I hear about a new software project, one question I like to ask is, “How is the software tested?”
The unfortunate answer is that much of the software that will be presented at SC13 is not tested very well. If it is tested, it’s tested under the sarcastic circumstances described by Professor Might in the tweet above.
Great software testing includes a broad suite of unit tests evaluating correctness and performance across a broad range of hardware and operating systems. Every source code commit should automatically be tested by the independent testing system.
For ArrayFire, we use Jenkins to test millions of lines of code each day. We highly recommend it.
I’ve worked on open source projects before with automated testing. ITK is a good example and here is the ITK testing dashboard.
Most academic projects unfortunately do not get tested very well. There is often no reward for academics to do the testing dirty work.
The next time you hear about a new software project, ask the maintainers how the software is tested.
What are your thoughts on software testing?