Monday, July 25, 2011

Testing 1-2-3

Testing is critically important to producing high-quality software. I'd like to take you behind the scenes at Roaring Penguin and talk a little about how we test CanIt.

Test Types

Our tests are broken into three basic sets:

  1. Unit Tests test each module of the software separately. Unit tests are designed to catch logic errors within each module.
  2. Regression Tests test the system as a whole. These do a full installation of the software, run mail through it, and verify that the results are as expected. Regression tests catch integration errors. They also ensure that once a bug is found and fixed, it stays fixed.
  3. Virtual Machine Tests test our RPMs and ISO images. We do clean installs into brand new virtual machines to make sure the CanIt system is installable. We do basic sanity checks to ensure it works as expected. These tests catch packaging and dependency errors.

Running Tests

The unit and regression tests are run each time we commit a change to our git repository. We have a web page with the status of each test. This lets us know as soon as possible if and when something is broken.

The virtual machine tests are run with a single command when we prepare for a release. They are quite time-consuming, and are therefore impractical to run with each check-in.

How many tests?

To give you an idea of the scope of our testing, here's how many tests we have:

  • Unit Tests: 20,775 individual tests.
  • Regression Tests: 5,389 individual tests, run both with and without Storage Manager.
  • Virtual Machine Tests: 260 individual tests

As you see, that's a lot of tests. While it's impossible to ensure bug-free releases, we do our best to make each release as robust as possible. We also do our best to make sure that fixed bugs stay fixed.