Steve Sanderson’s blog

This blog post is aimed at developers with at least a small amount of unit testing experience. If you’ve never written a unit test, please read an introduction and have a go at it first.

What’s the difference between a good unit test and a bad one? How do you learn how to write good unit tests? It’s far from obvious. Even if you’re a brilliant coder with decades of experience, your existing knowledge and habits won’t automatically lead you to write good unit tests, because it’s a different kind of coding and most people start with unhelpful false assumptions about what unit tests are supposed to achieve.

Most of the unit tests I see are pretty unhelpful. I’m not blaming the developer: Usually, he or she just got told to start unit testing, so they installed NUnit and started churning out [Test] methods. Once they saw red and green lights, they assumed they were doing it correctly. Bad assumption! It’s overwhelmingly easy to write bad unit tests that add very little value to a project while inflating the cost of code changes astronomically. Does that sound agile to you?

Unit testing is not about finding bugs

Now, I’m strongly in favour of unit testing, but only when you understand what role unit tests play within the Test Driven Development (TDD) process, and squash any misconception that unit tests have anything to do with testing for bugs.

In my experience, unit tests are not an effective way to find bugs or detect regressions. Unit tests, by definition, examine each unit of your code separately. But when your application is run for real, all those units have to work together, and the whole is more complex and subtle than the sum of its independently-tested parts. Proving that components X and Y both work independently doesn’t prove that they’re compatible with one another or configured correctly. Also, defects in an individual component may bear no relationship to the symptoms an end user would experience and report. And since you’re designing the preconditions for your unit tests, they won’t ever detect problems triggered by preconditions that you didn’t anticipate (for example, if some unexpected IHttpModule interferes with incoming requests).

So, if you’re trying to find bugs, it’s far more effective to actually run the whole application together as it will run in production, just like you naturally do when testing manually. If you automate this sort of testing in order to detect breakages when they happen in the future, it’s called integration testing and typically uses different techniques and technologies than unit testing. Don’t you want to use the most appropriate tool for each job?

(Note: there’s one exception where unit tests do effectively detect bugs. It’s when you’re refactoring, i.e., restructuring a unit’s code but without meaning to change its behaviour. In this case, unit tests can often tell you if the unit’s behaviour has changed.)

Well then, if unit testing isn’t about finding bugs, what is it about?

I bet you’ve heard the answer a hundred times already, but since the testing misconception stubbornly hangs on in developers’ minds, I’ll repeat the principle. As TDD gurus keep saying, “TDD is a design process, not a testing process”. Let me elaborate: TDD is a robust way of designing software components (“units”) interactively so that their behaviour is specified through unit tests. That’s all!

Good unit tests vs bad ones

TDD helps you to deliver software components that individually behave according to your design. A suite of good unit tests is immensely valuable: it documents your design, and makes it easier to refactor and expand your code while retaining a clear overview of each component’s behaviour. However, a suite of bad unit tests is immensely painful: it doesn’t prove anything clearly, and can severely inhibit your ability to refactor or alter your code in any way.

Where do your tests sit on the following scale?

Unit tests created through the TDD process naturally sit at the extreme left of this scale. They contain a lot of knowledge about the behaviour of a single unit of code. If that unit’s behaviour changes, so must its unit tests, and vice-versa. But they don’t contain any knowledge or assumptions about other parts of your codebase, so changes to other parts of your codebase don’t make them start failing (and if yours do, that shows they aren’t true unit tests). Therefore they’re cheap to maintain, and as a development technique, TDD scales up to any size of project.

At the other end of the scale, integration tests contain no knowledge about how your codebase is broken down into units, but instead make statements about how the whole system behaves towards an external user. They’re reasonably cheap to maintain (because no matter how you restructure the internal workings of your system, it needn’t affect an external observer) and they prove a great deal about what features are actually working today.

Anywhere in between, it’s unclear what assumptions you’re making and what you’re trying to prove. Refactoring might break these tests, or it might not, regardless of whether the end-user experience still works. Changing the external services you use (such as upgrading your database) might break these tests, or it might not, regardless of whether the end-user experience still works. Any small change to the internal workings of a single unit might force you to fix hundreds of seemingly unrelated hybrid tests, so they tend to consume a huge amount of maintenance time – sometimes in the region of 10 times longer than you spend maintaining the actual application code. And it’s frustrating because you know that adding more preconditions to make these hybrid tests go green doesn’t truly prove anything.

Tips for writing great unit tests

Enough vague discussion – time for some practical advice. Here’s some guidance for writing unit tests that sit snugly at Sweet Spot A on the preceding scale, and are virtuous in other ways too.

• Make each test orthogonal (i.e., independent) to all the others
  Any given behaviour should be specified in one and only one test. Otherwise if you later change that behaviour, you’ll have to change multiple tests. The corollaries of this rule include:
  
  • Don’t make unnecessary assertions
    Which specific behaviour are you testing? It’s counterproductive to Assert() anything that’s also asserted by another test: it just increases the frequency of pointless failures without improving unit test coverage one bit. This also applies to unnecessary Verify() calls – if it isn’t the core behaviour under test, then stop making observations about it! Sometimes, TDD folks express this by saying “have only one logical assertion per test”.
    Remember, unit tests are a design specification of how a certain behaviour should work, not a list of observations of everything the code happens to do.
  • Test only one code unit at a time
    Your architecture must support testing units (i.e., classes or very small groups of classes) independently, not all chained together. Otherwise, you have lots of overlap between tests, so changes to one unit can cascade outwards and cause failures everywhere.
    If you can’t do this, then your architecture is limiting your work’s quality – consider using Inversion of Control.
  • Mock out all external services and state
    Otherwise, behaviour in those external services overlaps multiple tests, and state data means that different unit tests can influence each other’s outcome.
    You’ve definitely taken a wrong turn if you have to run your tests in a specific order, or if they only work when your database or network connection is active.
    (By the way, sometimes your architecture might mean your code touches static variables during unit tests. Avoid this if you can, but if you can’t, at least make sure each test resets the relevant statics to a known state before it runs.)
  • Avoid unnecessary preconditions
    Avoid having common setup code that runs at the beginning of lots of unrelated tests. Otherwise, it’s unclear what assumptions each test relies on, and indicates that you’re not testing just a single unit.
    An exception: Sometimes I find it useful to have a common setup method shared by a very small number of unit tests (a handful at the most) but only if all those tests require all of those preconditions. This is related to the context-specification unit testing pattern, but still risks getting unmaintainable if you try to reuse the same setup code for a wide range of tests.

  (By the way, I wouldn’t count pushing multiple data points through the same test (e.g., using NUnit’s [TestCase] API) as violating this orthogonality rule. The test runner may display multiple failures if something changes, but it’s still only one test method to maintain, so that’s fine.)

• Don’t unit-test configuration settings
  By definition, your configuration settings aren’t part of any unit of code (that’s why you extracted the setting out of your unit’s code). Even if you could write a unit test that inspects your configuration, it merely forces you to specify the same configuration in an additional redundant location. Congratulations: it proves that you can copy and paste!

  Personally I regard the use of things like filters in ASP.NET MVC as being configuration. Filters like [Authorize] or [RequiresSsl] are configuration options baked into the code. By all means write an integration test for the externally-observable behaviour, but it’s meaningless to try unit testing for the filter attribute’s presence in your source code – it just proves that you can copy and paste again. That doesn’t help you to design anything, and it won’t ever detect any defects.

• Name your unit tests clearly and consistently
  If you’re testing how ProductController’s Purchase action behaves when stock is zero, then maybe have a test fixture class called PurchasingTests with a unit test called ProductPurchaseAction_IfStockIsZero_RendersOutOfStockView(). This name describes the subject (ProductController’s Purchase action), the scenario (stock is zero), and the result (renders “out of stock” view). I don’t know whether there’s an existing name for this naming pattern, though I know others follow it. How about S/S/R? 

  Avoid non-descriptive unit tests names such as Purchase() or OutOfStock(). Maintenance is hard if you don’t know what you’re trying to maintain.

Conclusion

Without doubt, unit testing can significantly increase the quality of your project. Many in our industry claim that any unit tests are better than none, but I disagree: a test suite can be a great asset, or it can be a great burden that contributes little. It depends on the quality of those tests, which seems to be determined by how well its developers have understood the goals and principles of unit testing.

By the way, if you want to read up on integration testing (to complement your unit testing skills), check out projects such as Watin, Selenium, and even the ASP.NET MVC integration testing helper library I published recently.

Let’s start with a quick recap:

• Unit tests test individual software components. They supply mocks or fake versions of dependencies (such as a database) so that the unit test doesn’t rely on any external code and any failures can be pinpointed exactly. Unit testing is central to Test Driven Development – the entire TDD process is driven by unit testing.
• Integration tests test your entire software stack working together. These tests don’t mock or fake anything (they use the real database, and real network connections) and are good at spotting if your unit-tested components aren’t working together as you expected. In general, it’s best to put most of your effort into building a solid suite of unit tests, and then adding a few integration tests for each major feature so you can detect any catastrophic incompatibilities or configuration errors before your customers do.

ASP.NET MVC famously has great support for unit testing. That’s well covered elsewhere on the web, so I won’t write more about it here. But what about integration testing?

Typically, ASP.NET MVC developers have written integration tests using a browser automation tool, such as Selenium or Microsoft’s Lightweight Test Automation Framework. These tools host a real web browser and automate a series of navigation steps, clicking on buttons and letting you write assertions about what should appear on the screen. Browser automation is fully capable of testing your JavaScript code just as easily as your business logic. However, there are also some drawbacks to browser automation:

• Browser automation test scripts are very fragile. As soon as you alter the HTML in your views, the tests may start failing and sometimes all need to be re-recorded. Many developers are happy with browser automation at first, but after a few months are sick of re-recording the tests or maintaining thousands of lines of scripts, and give up on it.
• Browser automation test scripts rely on parsing the generated HTML, which loses all the strongly-typed goodness of your MVC model objects.

So, how can we do integration testing without browser automation?

Introducing MvcIntegrationTestFramework

What if you could write NUnit tests (or xUnit or whatever you prefer) that directly submit a URL, query string, cookies, request headers, etc., into your application – without needing the app to be hosted in any web server but still running in the real (non-mocked) ASP.NET runtime – and get back the response text to make assertions about? In fact, what if you also got back the ActionResult (e.g., a ViewResult) that was executed, a reference to any unhandled exception it threw, and could also write assertions about cookie values or the contents of Session? What if you could string together a series of test requests to simulate a user’s browsing session, checking that a whole feature area works from end to end?

Well, my friend, all this can be yours with MvcIntegrationTestFramework

Download the demo project. To run the integration tests, you’ll need to install NUnit or some other test runner.

A Simple Integration Test Example

Consider the following action method:

public ActionResult Index()
{
    ViewData["Message"] = "Welcome to ASP.NET MVC!";
    return View();
}

You can write an integration for this using MvcIntegrationTestFramework as follows:

[Test]
public void Root_Url_Renders_Index_View()
{
    appHost.SimulateBrowsingSession(browsingSession => {
        // Request the root URL
        RequestResult result = browsingSession.ProcessRequest("/");
 
        // You can make assertions about the ActionResult...
        var viewResult = (ViewResult) result.ActionExecutedContext.Result;
        Assert.AreEqual("Index", viewResult.ViewName);
        Assert.AreEqual("Welcome to ASP.NET MVC!", viewResult.ViewData["Message"]);
 
        // ... or you can make assertions about the rendered HTML
        Assert.IsTrue(result.ResponseText.Contains("<!DOCTYPE html"));
    });
}

This is pretty similar to a typical unit test for that action method, except that you also get access to the finished rendered HTML, in case that interests you. Unlike a unit test, this integration test goes through the entire request-processing pipeline, so it tests your routing configuration, your controller factory, any dependencies your controller has, and even your view template.

Note: That doesn’t make it better than a unit test: it just solves a different problem. Unit tests are better for driving the TDD process and for proving the correctness of individual components in isolation, and they tend to be easier to maintain large volumes of over time. Integration tests, on the other hand, are good in small doses: they cover things that unit tests cannot (such as views, database connectivity, etc.) but aren’t so fine-grained as unit tests.

Integration Tests Cover End-to-end Scenarios

Consider the following sequence of behaviours:

1. When a logged-out user visits /Home/SecretAction, they should be redirected to the login screen

2. If the user then enters a valid username and password, they should be redirected back to /Home/SecretAction

3. Since the user is now logged in, they should be shown some secret information

That’s a lot of interaction between action methods, Forms Authentication, HTTP redirections, cookies, and filters. You wouldn’t typically cover that interaction in any unit test, because each of those components is a separate unit. However, if you have an end-to-end unit test that runs through the whole process, you can feel very confident that you haven’t unintentionally broken anything (e.g., by editing web.config).

Here’s a example end-to-end integration test for this whole sequence of behaviours:

[Test]
public void LogInProcess()
{
    string securedActionUrl = "/home/SecretAction";
 
    appHost.SimulateBrowsingSession(browsingSession => {
        // First try to request a secured page without being logged in                
        RequestResult initialRequestResult = browsingSession.ProcessRequest(securedActionUrl);
        string loginRedirectUrl = initialRequestResult.Response.RedirectLocation;
        Assert.IsTrue(loginRedirectUrl.StartsWith("/Account/LogOn"), "Didn't redirect to logon page");
 
        // Now follow redirection to logon page
        string loginFormResponseText = browsingSession.ProcessRequest(loginRedirectUrl).ResponseText;
        string suppliedAntiForgeryToken = MvcUtils.ExtractAntiForgeryToken(loginFormResponseText);
 
        // Now post the login form, including the verification token
        RequestResult loginResult = browsingSession.ProcessRequest(loginRedirectUrl, HttpVerbs.Post, new NameValueCollection
        {
            { "username", "steve" },
            { "password", "secret" },
            { "__RequestVerificationToken", suppliedAntiForgeryToken }
        });
        string afterLoginRedirectUrl = loginResult.Response.RedirectLocation;
        Assert.AreEqual(securedActionUrl, afterLoginRedirectUrl, "Didn't redirect back to SecretAction");
 
        // Check that we can now follow the redirection back to the protected action, and are let in
        RequestResult afterLoginResult = browsingSession.ProcessRequest(securedActionUrl);
        Assert.AreEqual("Hello, you're logged in as steve", afterLoginResult.ResponseText);
    });
}

In this test, multiple requests are made as part of the same browsing session. The integration testing framework will preserve cookies (and therefore session state) from one request to the next, so you can test interaction with things like Forms Authentication and ASP.NET MVC’s AntiForgeryToken helpers. You can see this whole thing working by downloading the demo project.

This has advantages over browser automation in that (in my opinion) it’s easier to write this test – it’s concise, you don’t need to learn a browser scripting language, and you don’t need to rerecord the script if you restructure the HTML. You can make assertions at the HTML level, or you can make assertions at the ActionResult level, with fully strongly-typed access to any ViewData or Model values that were being supplied.

On the flip side, browser automation is still the only way to test your JavaScript.

Integration Testing Your Own ASP.NET MVC Application

To add these kinds of tests to your own ASP.NET MVC application, create a new class library project called MyApp.IntegrationTests or similar. Add a reference to MvcIntegrationTestFramework.dll, which you can get by downloading the demo project and compiling it, and also add a reference to your chosen unit testing framework (e.g., NUnit or xUnit).

Next, if you’re using NUnit, create a basic test fixture class as follows:

using MvcIntegrationTestFramework.Hosting;
using MvcIntegrationTestFramework.Browsing;
 
[TestFixture]
public class MyIntegrationTests
{
    // Amend this path to point to whatever folder contains your ASP.NET MVC application
    // (i.e., the folder that contains your app's main web.config file)
    private static readonly string mvcAppPath = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + "\\..\\..\\..\\MyMvcApplication");
    private readonly AppHost appHost = new AppHost(mvcAppPath);
}

(If you’re using something other than NUnit, amend the syntax accordingly.)

Also – and seriously you can’t skip this step – you must add a post-build step so that each time you compile, Visual Studio will copy the assemblies from your integration test project’s \bin folder into your main ASP.NET MVC application’s \bin folder. I’ll explain why this is necessary later.

The test fixture you just created tells MvcIntegrationTestFramework to find your ASP.NET MVC application at the specified disk location, and to host it (without using IIS or any other web server). The resulting AppHost object then provides an API for issuing requests to your application, so you could write a test as follows:

[TestFixture]
public class MyIntegrationTests
{
    // Leave rest as before
 
    [Test]
    public void HomeIndex_DemoTests()
    {
        appHost.SimulateBrowsingSession(browsingSession => {
            RequestResult result = browsingSession.ProcessRequest("home/index");
 
            // Routing config should match "home" controller
            Assert.AreEqual("home", result.ActionExecutedContext.RouteData.Values["controller"]);
 
            // Should have rendered the "index" view
            ActionResult actionResult = result.ActionExecutedContext.Result;
            Assert.IsInstanceOf(typeof(ViewResult), actionResult);
            Assert.AreEqual("index", ((ViewResult)actionResult).ViewName);
 
            // App should not have had an unhandled exception
            Assert.IsNull(result.ResultExecutedContext.Exception);
        });
    }
}

The browsingSession object lets you read and write cookies and ASP.NET session values (note: ASP.NET initializes the Session collection only after the first request in any given browsing session). The RequestResult object that comes back from ProcessRequest() lets you access the rendered response text, the actual HttpResponse object, plus the MVC ActionExecutedContext and RequestExecutedContext objects that specify what the application did in ASP.NET MVC terms.

How Does it Work Internally?

Note: if you just want to use MvcIntegrationTestFramework, you don’t need to know any of the following. I provide this info just for your interest, and to give you a head start if you want to expand the framework or change it in any way.

MvcIntegrationTestFramework uses .NET’s built-in API for creating an ASP.NET-enabled .NET appdomain and using it to host an ASP.NET application from an arbitrary disk location. The test framework then does a bunch of funky reflection-assisted stuff to attach some interceptors and event handlers to the new HTTP runtime.

When you call ProcessRequest(), internally it calls ASP.NET’s HttpRuntime.ProcessRequest() method, which invokes the request-processing pipeline. The testing framework simulates an incoming HttpWorkerRequest according to the URL, HTTP method, querystring, form values, cookies, and headers that you specify.

While the request is running, MvcIntegrationTestFramework uses its knowledge of ASP.NET MVC to add interceptors to key points in the MVC request-processing pipeline. This is how it’s able to grab and expose the ActionResult and other MVC context objects so you can make assertions about them.

Of course, all this request processing has to happen within the ASP.NET-enabled appdomain, *not* the appdomain your test runner invokes the unit test on. So, we use the slightly bizarre appHost.SimulateBrowsingSession(browsingSession => { /* your code goes here */ } syntax to shuttle your test code across the appdomain boundary and run it there. Consequently, any variables you declare outside your anonymous delegate and then use inside it *must* be serializable, or you’ll get a serialization error when they get marshalled across the appdomain boundary. This is unlikely to be a problem in real world use.

Since your test code runs in the ASP.NET-enabled appdomain, the CLR’s assembly loader will need to load your test code assembly from a location accessible to your ASP.NET MVC application. That’s why it’s necessary to have a post-build event to copy your test code assembly (and any other assemblies it references) into your ASP.NET MVC app’s \bin folder. Or you could register your test assembly in the GAC I suppose, but that’s much more hassle.

Conclusion

Personally, I find this approach to integration testing more productive than browser automation in most cases, because the tests are fast to write and are less prone to break because of trivial HTML changes. I wouldn’t want to have thousands of integration tests, though – for one thing, they’d take ages to run – but end-to-end testing for key scenarios like logging in, registering, going through shopping carts, etc., proves pretty convincingly that the major functional areas of your app are working overall, leaving unit tests to prove the fine detail of each component.

Some of the code in MvcIntegrationTestFramework was adapted from Chot’s CodeProject article, some of the ideas came from a discussion at DDD South West with Richard Fennell who’s already doing something similar except with a dependency on TypeMock, and some bits of my HttpWorkerRequest simulation code were demonstrated years ago by Phil Haack in his HttpSimulator code, which is similar to MvcIntegrationTestFramework except it predates ASP.NET MVC.

Download the demo project. To run the integration tests, you’ll need to install NUnit or some other test runner.