Layer-based automated testing, part 1: choosing the right testing approach

This is a series of articles about building an effective testing system of larger-scale code-driven tests for your .Net application. It shows how to use such concepts like IoC, MVVM, BDD, mocking and asynchronous programming together to minimize efforts on developing new tests and supporting old ones. Also it gives a working sample of such test-friendly architecture. Of course this approach is not applicable for all cases, but it has proven its effectiveness for many existing scenarios.

Generally speaking, when you develop automated tests for your application, you may use three different approaches:

  1. Small-scale code-driven tests (i.e. unit tests, in its classical meaning)
  2. Larger-scale code-driven tests
  3. Graphical UI tests (using such tools as Selenium or Coded UI)

Each of these approaches has its strengths and weaknesses. So, in fact, to make your testing system most effective, it should use all 3 approaches, choosing the best applicable approach for each particular task. However, how we can minimize our efforts so that we won't need to build three automated test systems, each covering all functionality of your project? To answer these questions, we need to look closer on each of this approaches.

Unit tests

In its classical meaning, unit tests are checking behavior of one class, mocking its surroundings. These tests are perfectly atomic, so if unit test breaks, it is clear which method behaves in incorrect way. Also it describes the requirements to the class in a language of tests. However for many cases this approach is too narrow-minded and causes many problems:

  1. Interfaces can change. Public interfaces of your classes is not something written in stone. Your class design just reflects the requirements, and maybe not in the best way. In practice you will be refactoring your code a lot, because sometimes you discover architectural flaws, and sometimes after getting new requirements you understand that you need to generalize code in different way. And changes interface requires a lot of changes in unit tests which are testing this interface. Sometimes even the whole understanding of what this unit test was actually doing can be lost during such refactorings.
  2. Too much mocking. You should always invent a way of emulating other classes which are outside of the scope of this test. And often this is a non-trivial task. And this behavior can change a lot, too.
  3. Tests are too simple. Often over-reliance on atomic tests can cause the situation when actual problem is not found by tests, because it is with interaction of several objects, its registration, event subscription etc. Trying to find all these "atoms" of interaction on unit tests level is a very non-trivial task and may require lot of time and effort.

This list is just a summary of concerns expressed in many sources, for example:
http://stackoverflow.com/questions/153234/how-deep-are-your-unit-tests
http://stackoverflow.com/questions/1094413/is-there-such-a-thing-as-too-...

So, small-scale tests is not a panacea, and over-reliance on small-scale tests can effectively consume time and resources of your project. But they are very good when you really have something non-trivial and atomic to test. For example, regular expressions, non-trivial numeric calculations, complex conditions. There's a lot of places where unit tests should be used.

Graphical UI tests

Tests which are actually emulating user interaction by running your program and emulating user clicks and other input. The obvious benefit of these tests is that they test what user actually sees, not some abstraction buried deep inside your code. So, they won't break because of internal refactorings when your specification has not changed. And they test interaction of your object in (almost) the same way as it is in the working program. But they have a number of their own problems:

  1. Slow performance. UI testing frameworks are not very fast, and creating testing environment also takes some time. So yes, graphical UI tests are heavy.
  2. Reliance on UI testing framework which has lots of its own peculiarities, limitations, and can become outdated.

It means that this approach shouldn't be overused too. It's good to have at least some tests of such kind, checking most crucial of scenarios. However if you have too much of these tests, your test run becomes too slow, and your team will spend too much time working with testing frameworks. But if your application has very rich UI and does not have very complicated business logic, probably graphical UI tests should be your main choice.

Layer-based tests

And now, introducing the approach these articles are about. If your application is properly structured, it allows you to separate one layer from another in automated way, substitute other layers by mocks, and test the whole layer, or set of layers, as a whole.

Layer-based testing allows to combine advantages of unit tests and graphical UI tests. They are fast, because all slower layers are mocked. They are testing the set of classes, so they are effective in finding bugs not only with "atomic" behavior, but also with interaction of classes. When you are doing internal refactoring inside a layer, layer tests won't change. Even when you are introducing new classes and interfaces internal to your layer, layer tests won't change.

This approach is especially effective when you have complicated business logic with a rather fixed set of inputs and outputs. For this case, such kind of tests are most simple and easy to write. It really preserves a lot of developer's time. It works even better when combined with BDD and SpecFlow.

BDD

Behavior-driven development is a methodology in which tests are linked to actual user scenarios written in humanly readable language (Gherkin). This approach ensures that your tests remain sane: that they test what is actually needed from your application. Also it provides an easy and natural way to reuse your test-related code by introducing a concept of reusable "steps" which can be repeated in different scenarios.

It is not required to base your layer tests on BDD. However it helps a lot:

  1. It really makes your test structure more clear and reusable.
  2. Even non-developers can easily understand which test has actually failed.
  3. Gherkin scenario works as a perfect comment to your tests, but even more - this comment is functional and calls the code, not the other way around.

So our sample code will use Gherkin and SpecFlow a lot. To read more about BDD, please refer to the following resources:

http://blogs.lessthandot.com/index.php/EnterpriseDev/application-lifecyc...
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
(and many others)

The bottom line

The choice of using code-driven vs graphical UI tests actually depends on the application you intend to test. The more rich and complicated is UI, the more there are reasons to use graphical UI tests. The more "pure" business logic you have, the more reasons there are to use code-driven approach. So you need to be smart and use both approaches, covering more business logic-heavy parts of your system with code-driven tests, and UI-heavy parts with graphical UI tests. And don't forget to use traditional atomic unit tests for parts of your system where it is appropriate.

Also, you shouldn't forget about other kinds of tests. For example, code-driven layer tests are not the only integration tests which can be useful. Integration tests on real database and real hardware devices are also very important! Only combining tests written in different methodologies you can build a really effective test system.

However, in future articles I'll concentrate purely on code-driven layer tests and explain in more details how we can write them effectively.