Site Meter
 
 

Writing Great Unit Tests: Best and Worst Practices

This blog post is aimed at developers with at least a small amount of unit testing experience. If you’ve never written a unit test, please read an introduction and have a go at it first.

What’s the difference between a good unit test and a bad one? How do you learn how to write good unit tests? It’s far from obvious. Even if you’re a brilliant coder with decades of experience, your existing knowledge and habits won’t automatically lead you to write good unit tests, because it’s a different kind of coding and most people start with unhelpful false assumptions about what unit tests are supposed to achieve.

Most of the unit tests I see are pretty unhelpful. I’m not blaming the developer: Usually, he or she just got told to start unit testing, so they installed NUnit and started churning out [Test] methods. Once they saw red and green lights, they assumed they were doing it correctly. Bad assumption! It’s overwhelmingly easy to write bad unit tests that add very little value to a project while inflating the cost of code changes astronomically. Does that sound agile to you?

Unit testing is not about finding bugs

Now, I’m strongly in favour of unit testing, but only when you understand what role unit tests play within the Test Driven Development (TDD) process, and squash any misconception that unit tests have anything to do with testing for bugs.

In my experience, unit tests are not an effective way to find bugs or detect regressions. Unit tests, by definition, examine each unit of your code separately. But when your application is run for real, all those units have to work together, and the whole is more complex and subtle than the sum of its independently-tested parts. Proving that components X and Y both work independently doesn’t prove that they’re compatible with one another or configured correctly. Also, defects in an individual component may bear no relationship to the symptoms an end user would experience and report. And since you’re designing the preconditions for your unit tests, they won’t ever detect problems triggered by preconditions that you didn’t anticipate (for example, if some unexpected IHttpModule interferes with incoming requests).

So, if you’re trying to find bugs, it’s far more effective to actually run the whole application together as it will run in production, just like you naturally do when testing manually. If you automate this sort of testing in order to detect breakages when they happen in the future, it’s called integration testing and typically uses different techniques and technologies than unit testing. Don’t you want to use the most appropriate tool for each job?

Goal Strongest technique
Finding bugs (things that don’t work as you want them to) Manual testing (sometimes also automated integration tests)
Detecting regressions (things that used to work but have unexpectedly stopped working) Automated integration tests (sometimes also manual testing, though time-consuming)
Designing software components robustly Unit testing (within the TDD process)

(Note: there’s one exception where unit tests do effectively detect bugs. It’s when you’re refactoring, i.e., restructuring a unit’s code but without meaning to change its behaviour. In this case, unit tests can often tell you if the unit’s behaviour has changed.)

Well then, if unit testing isn’t about finding bugs, what is it about?

I bet you’ve heard the answer a hundred times already, but since the testing misconception stubbornly hangs on in developers’ minds, I’ll repeat the principle. As TDD gurus keep saying, “TDD is a design process, not a testing process”. Let me elaborate: TDD is a robust way of designing software components (“units”) interactively so that their behaviour is specified through unit tests. That’s all!

Good unit tests vs bad ones

TDD helps you to deliver software components that individually behave according to your design. A suite of good unit tests is immensely valuable: it documents your design, and makes it easier to refactor and expand your code while retaining a clear overview of each component’s behaviour. However, a suite of bad unit tests is immensely painful: it doesn’t prove anything clearly, and can severely inhibit your ability to refactor or alter your code in any way.

Where do your tests sit on the following scale?

image

Unit tests created through the TDD process naturally sit at the extreme left of this scale. They contain a lot of knowledge about the behaviour of a single unit of code. If that unit’s behaviour changes, so must its unit tests, and vice-versa. But they don’t contain any knowledge or assumptions about other parts of your codebase, so changes to other parts of your codebase don’t make them start failing (and if yours do, that shows they aren’t true unit tests). Therefore they’re cheap to maintain, and as a development technique, TDD scales up to any size of project.

At the other end of the scale, integration tests contain no knowledge about how your codebase is broken down into units, but instead make statements about how the whole system behaves towards an external user. They’re reasonably cheap to maintain (because no matter how you restructure the internal workings of your system, it needn’t affect an external observer) and they prove a great deal about what features are actually working today.

Anywhere in between, it’s unclear what assumptions you’re making and what you’re trying to prove. Refactoring might break these tests, or it might not, regardless of whether the end-user experience still works. Changing the external services you use (such as upgrading your database) might break these tests, or it might not, regardless of whether the end-user experience still works. Any small change to the internal workings of a single unit might force you to fix hundreds of seemingly unrelated hybrid tests, so they tend to consume a huge amount of maintenance time – sometimes in the region of 10 times longer than you spend maintaining the actual application code. And it’s frustrating because you know that adding more preconditions to make these hybrid tests go green doesn’t truly prove anything.

Tips for writing great unit tests

Enough vague discussion – time for some practical advice. Here’s some guidance for writing unit tests that sit snugly at Sweet Spot A on the preceding scale, and are virtuous in other ways too.

  • Make each test orthogonal (i.e., independent) to all the others
    Any given behaviour should be specified in one and only one test. Otherwise if you later change that behaviour, you’ll have to change multiple tests. The corollaries of this rule include:
      i>Don’t make unnecessary assertions
      Which specific behaviour are you testing? It’s counterproductive to Assert() anything that’s also asserted by another test: it just increases the frequency of pointless failures without improving unit test coverage one bit. This also applies to unnecessary Verify() calls – if it isn’t the core behaviour under test, then stop making observations about it! Sometimes, TDD folks express this by saying “have only one logical assertion per test”.
      Remember, unit tests are a design specification of how a certain behaviour should work, not a list of observations of everything the code happens to do.
    • Test only one code unit at a time
      Your architecture must support testing units (i.e., classes or very small groups of classes) independently, not all chained together. Otherwise, you have lots of overlap between tests, so changes to one unit can cascade outwards and cause failures everywhere.
      If you can’t do this, then your architecture is limiting your work’s quality – consider using Inversion of Control.
    • Mock out all external services and state
      Otherwise, behaviour in those external services overlaps multiple tests, and state data means that different unit tests can influence each other’s outcome.
      You’ve definitely taken a wrong turn if you have to run your tests in a specific order, or if they only work when your database or network connection is active.
      (By the way, sometimes your architecture might mean your code touches static variables during unit tests. Avoid this if you can, but if you can’t, at least make sure each test resets the relevant statics to a known state before it runs.)
    • Avoid unnecessary preconditions
      Avoid having common setup code that runs at the beginning of lots of unrelated tests. Otherwise, it’s unclear what assumptions each test relies on, and indicates that you’re not testing just a single unit.
      An exception: Sometimes I find it useful to have a common setup method shared by a very small number of unit tests (a handful at the most) but only if all those tests require all of those preconditions. This is related to the context-specification unit testing pattern, but still risks getting unmaintainable if you try to reuse the same setup code for a wide range of tests.

    (By the way, I wouldn’t count pushing multiple data points through the same test (e.g., using NUnit’s [TestCase] API) as violating this orthogonality rule. The test runner may display multiple failures if something changes, but it’s still only one test method to maintain, so that’s fine.)

  • Don’t unit-test configuration settings
    By definition, your configuration settings aren’t part of any unit of code (that’s why you extracted the setting out of your unit’s code). Even if you could write a unit test that inspects your configuration, it merely forces you to specify the same configuration in an additional redundant location. Congratulations: it proves that you can copy and paste!

    Personally I regard the use of things like filters in ASP.NET MVC as being configuration. Filters like [Authorize] or [RequiresSsl] are configuration options baked into the code. By all means write an integration test for the externally-observable behaviour, but it’s meaningless to try unit testing for the filter attribute’s presence in your source code – it just proves that you can copy and paste again. That doesn’t help you to design anything, and it won’t ever detect any defects.

  • Name your unit tests clearly and consistently
    If you’re testing how ProductController’s Purchase action behaves when stock is zero, then maybe have a test fixture class called PurchasingTests with a unit test called ProductPurchaseAction_IfStockIsZero_RendersOutOfStockView(). This name describes the subject (ProductController’s Purchase action), the scenario (stock is zero), and the result (renders “out of stock” view). I don’t know whether there’s an existing name for this naming pattern, though I know others follow it. How about S/S/R

    Avoid non-descriptive unit tests names such as Purchase() or OutOfStock(). Maintenance is hard if you don’t know what you’re trying to maintain.

Conclusion

Without doubt, unit testing can significantly increase the quality of your project. Many in our industry claim that any unit tests are better than none, but I disagree: a test suite can be a great asset, or it can be a great burden that contributes little. It depends on the quality of those tests, which seems to be determined by how well its developers have understood the goals and principles of unit testing.

By the way, if you want to read up on integration testing (to complement your unit testing skills), check out projects such as Watin, Selenium, and even the ASP.NET MVC integration testing helper library I published recently.

kick it on DotNetKicks.com

48 Responses to Writing Great Unit Tests: Best and Worst Practices

  1. Great post Steven! I recently worked on a project where I attempted to use TDD without really knowing what I was doing. At first, pretty much all of my tests were in that “Dirty Hybrid” zone and it was very frustrating. Eventually with experience (and help from others like you) it started to click and unit testing really made a huge difference in the quality of the project (I wrote about my learning experience on my blog to hopefully help others too). Thanks for great posts like this one!

  2. John

    First, great book.

    Second, how will .Net 4.0 code contracts work with TDD? I’m just getting my feet wet in TDD and I keep hearing over and over again that the main benefit of TDD is more about specifications than “tests”. I’ve even heard some people argue for a name change and remove the word “test”. But then I heard a Hanselminutes podcasts on spec# some time ago. Later, I listened to an SE-Radio podcast on Eiffel, which uses “code by contract”. After listening to those, it seemed to me like design by code and TDD were both about design specifications, but design by contract was generally better, because it baked the specification right into the code itself.

    Now, I find out that .Net 4.0 will have a code contracts library. So I’m wondering, should 80% of your NUnit code be replaced with code contracts and leave NUnit for rest that can’t be easily specified in a code contract.

  3. steve

    That is an awesome post. Easy to understand and some great tips.

  4. Geovanny Tejeda

    Fantastic post!
    If possible I’d love to see you write more specifically about how to conceptually think about the actual tests, even after all this time I often question myself at the very beginning of the project with things like, a test for “should create a product” which is just a test of the var product = new Product(); including its properties… hmm… that doesn’t sound very useful, of course it should create a product, aaaargh, constantly struggling what to and not to test.

    And even bigger confusion when trying to do it the-TDD-way (which I’m intellectually in love with but have still not quite grasp it 100%), how do I justify the creation of the product class with something smaller than “a Product should be added to a cart”; that’s supposed to be to big a step to start with; you get the picture, I always get very confused at the beginning, so anything you can write about this would be more than welcome =)

    thanks man,
    keep it up

  5. arnold de groot

    On the naming of unittests: it helps to have the verb ‘should’ somewhere in the name. E.g. ProductPurchaseAction_Should_Render_OutOfStockView_If_StockIsZero(). In general, carefully thinking about the name before coding helps me focusing.

    Another way unittests can start failing is when you execute them in another order. It is a hint that the setup-methods are omitting needed preconditions!.

  6. Fantastic Post and i liked the tips very much. TDD is mostly understood for finding bugs still today. Thanks very much. I am sharing this.

  7. si

    I like to name unit tests along the lines of:

    MethodOrProperty_ShouldDoThis_UnderTheseCircumstances

    Also, when in maintenance mode I find unit tests can be great for reproducing bugs. i.e. Bug gets reported -> Write (or fix:) unit test to reproduce bug (fail) -> Fix bug (pass)

  8. KevDog

    Steven,

    Would you consider it worthwhile to unit test data validation attributes on fields like phone numbers, e-mail addresses, etc.? It’s close to things like the authorize attribute you mentioned.

    Thanks,

    Kevin

  9. Jack

    Hi,

    Care to elaborate on this highly spurious statement?

    “unit tests are not an effective way to find bugs or detect regressions.”

  10. Steve

    @John – a fascinating question. In one sense, unit tests *are* simply code contracts – just rather complex and specialised ones. I’m looking forwards to seeing where things go in this area, but I can’t predict it!

    @Geovanny – not sure how to address your question directly, but I’ll certainly agree that TDD isn’t always obvious. I try to demonstrate the need for each new piece of code by adding a test for it first, but I wouldn’t normally add a test merely to assert that “new Foo()” isn’t null – that’s a statement about the CLR, not about my code, so it isn’t very meaningful to make specifications about it.

    @Arnold – Good point about the naming. “It is a hint that the setup-methods are omitting needed preconditions!” – If you end up with complex setup methods simulating a lot of preconditions, it suggests that you aren’t testing a single unit at once, which means the test suite runs the risk of getting unmaintainable over time.

    @si – indeed. If I understand correctly, you’re saying that your unit tests didn’t find the bug (someone else did), but they were a way to specify the additional required behaviour that fixes the bug.

    @KevDog – that’s a tricky one. Not sure I can pretend to have a one-size-fits-all answer for this. Assuming you have some sort of DoValidation() method that returns a list of validation errors, I think it’s likely that you’ll want to define the behaviour of the method using unit tests (i.e., examples of invalid data resulting in specific validation errors), rather than defining that certain attributes have to be present in your source code. But if your system gets more complex and you decide to have an external validation provider object, you’d merely want to test that certain model operations (e.g., saving a record) invoke it and respect its output, and then separately specify the behaviour of the validation provider.

    @Jack – was it really a spurious statement, given that I went on to spend the next two paragraphs explaining and elaborating on it?

  11. Pingback: Tune Up Your PC » Post Topic » Connected Show #15 – C# 4, It Ain’t THAT Complex!

  12. KevDog

    Steve,

    ” I think it’s likely that you’ll want to define the behaviour of the method using unit tests (i.e., examples of invalid data resulting in specific validation errors)”

    Cool, that’s the approach I have been taking. I’ve been doing it by checking that the list of validation errors contains one with the error message I am looking for, which is a little fragile, but I haven’t come up with a better way of doing it yet.

  13. Pingback: Weekly Web Nuggets #74 : Code Monkey Labs

  14. Pingback: Connected Show #15 – C# 4, It Ain’t THAT Complex! Related Resources: | Codedstyle

  15. This is great stuff. Subscribed!

    My only question is how do you feel about some of the BDD and its affect on writing unit tests? It’s more about multiple classes that set up different contexts (scenario) for each unit (class) of code. I’ve particularly started seeing this more and have tried it. It results in when_asking_the_container_for_an_object_and_the_object_is_registered.should_return_the_object
    .should_be_an_instance_of_the_correct_type
    etc

    when_asking_the_container_for_an_object_and_the_object_is_not_registered.should_throw_an_exception

    http://bombali.googlecode.com/svn/trunk/product/bombali.tests/infrastructure/containers/ContainerSpecs.cs

    http://blog.jpboodhoo.com/MoreNewConventionsForHowIOrganizeMyTests.aspx

  16. Michal

    Is it a good practice to test stored procedures in a unit test? One sp adds some data to a test db, and second one removes it. Third sp selects some test data and asserts if rows are equal to specific value.
    I’m new to test and TDD, that’s why I’m asking about it.
    I’m not sure if such test can be called unit test.
    Regards

  17. I think I still disagree that bad unit tests are worse than none. We’ve got quite a few bad tests in our test suite, but I would rather have the developers working to improve them without breaking them, than to simply throw them all away and and try to write better ones. That inevitably leads to refactoring without tests, and that is a recipe for bugs.

  18. I don’t entirely agree with the statement “Unit testing is not about finding bugs”.

    I’m working on a project where the BLL and DAL layers have been writen by several developers (and very hastily I might add) and I’ve spent the last week getting to 26% coverage and in doing so have found 12 bugs that no one had noticed.

    Writting unit test forces the developer to think “how do I test this one line” which is how I found the 12 mentioned above.

  19. Pingback: Servefault.com

  20. Thank you for the information, and found your practice of naming test very useful.

  21. Cool article, Really.

    I just wonder what do you think about datalayer testing?
    I’m really not sure what will be a good practice there.

    Keep up the good work ;-)

  22. Pingback: Selective Unit Testing – Costs and Benefits « Steve Sanderson’s blog

  23. Pingback: Writing unit testable JavaScript code « Rue marin

  24. Could you post some examples of good unit tests?

  25. Saket

    Great Post!

    Would love to hear from you on the topics of

    1. regression tests in some more detail as “Integration Tests” covers a wide gamut of tests.

    2. What would you recommend for Regression tests – white box tests, black box tests or grey box tests. Your post above seems to point towards black box. However, one school of thought says UI based tests are complex and unstable …
    Look forward to more posts from you :)

  26. Pingback: Write and implement good unit tests | Programming and Development | TechRepublic.com

  27. @Andrew (and Steve) – you might be interested in some unit testing on controllers that I am in the middle of contributing into MvcContrib. With a colleague, we have gone for a FluentController to ease the pain of TDD on the controller redirects. I am currently porting the code which uses moq into rhino mocks to fit with the library. There is public version/demo on here: http://github.com/toddb/MvcContrib/blob/master/src/MVCContrib.UnitTests/TestHelper/FluentController/FluentControllerTest.cs

    Here’s a sample of the tests:

    GivenController.As()
    .ShouldRedirectTo(RestfulAction.Index)
    .IfCallSucceeds()
    .WhenCalling(x => x.Create(null));

    GivenController.As()
    .ShouldRenderView(”New”)
    .IfCallFails()
    .WhenCalling(x => x.Create(null));

    This specific implementation isn’t for everyone but if it is it hides away a level of accidental complexity in controllers (and it keeps controller redirect design/tests squarely in the realm of unit tests)

  28. wytek

    Very good article. Suggesting that unit tests are about design is interesting but difficult to accept. I lean towards viewing it as a matter of testing behavior. A unit test is valuable where a method behaves more like a mathematical function, creating an expected output for an expected input. I suggest that integration testing is more suited to help in the design of the product.

  29. Pall

    Hey great
    article really helped me in my ppt :)

  30. Steve,

    On average, how much time should be allocated for unit testing? For example, if a particular programming project for a maintenance application (CRUD) takes 500 hours to program, how many hours should be budgeted for unit testing?

    Any guidenace on this topic would be very helpful for project planning purposes. Thanks!

  31. Pingback: Framing the Debate Over Software Testing and Coverage Reporting « Enterprise Continuous Integration

  32. Pingback: Unit testing | Too weak to give in

  33. Rick Pingry

    Thanks for a great article. I have been trying TDD and automated testing in various ways for the last 6 months or so, and I am still frustrated with it. I guess I am like Geovanny Tejeda, in love with the idea, but still not 100% up on it. I wish I could get that solid Aha experience that others seem to get where they get converted and never go back. Right now I get the feeling that using TDD causes me to take about 5x as long to write the same thing. Yeah, maybe there are times where it has helped me find something faster, maybe even 10x faster than otherwise, but I am not sure that it justifies the 5 fold increase on all of the things that are not broken to begin with. I am following all of the rules that I have read here and in other articles, but it still seems to take SOOOO much longer to do even the simplest things, that I feel like I am wasting the time of my clients. Please Help!

    I am not talking about simple “Bank Account” kinds of example classes that simply do some kind of calculation. I find those easy and even writing TDD makes sense to me. The problem I have is when I am trying to add functionality that interacts with another collaborating class. To do it in a BDD kind of way, I have to create a new interface, a stub class, a Mock representation (I am using C++, so I am sure I am light on the tools you Java and .NET guys have), and then I need to pass in references through constructors ala “Inversion of Control”, which means somewhere up the line I have some kind of composer to do all of that, and then FINALLY the tests just to say that another function got called. See what I mean? Easily 5x the code and 5x the time to write it all, not to mention needing to maintain all of the extra code. Refactoring seems HARDER to me because any refactoring that requires a class extraction or change in parameters means not only changes in tests but moving parts of tests.

    So that brings me to my first real question… (sorry for the long rant, I really do appreciate you and I am trying to “get it”. I think lots of others may have these same confusions). When you do an extract class refactoring, do you need to move the tests to the new code and then tests against mocks on the new class being called?

    Next question… What is a “Unit” do you test every class against mocks of its immediate collaborators? I have been playing with building mocks of the external interfaces of my system and then test from the outside, which seems to smack more of what you are calling Integration Tests, right? (Thanks for the great clarification on that BTW).

    For me, the siren of watching for Regressions is still what is calling me, but I am trying to get my head around writing tests to make better code. I can see that doing this makes for more testable code, but does that necessarily mean better (more maintainable, easier to debug) code? Perhaps I have a hard time making the judgement, but I did not think my code was THAT bad to start with ;)

    With that in mind, I can see many questions here about Unit Testing, and they seem to be talking about writing Unit Tests (with an emphasis on code coverage) AFTER the Unit is written. Does that fly in the face of TDD’s “write the test first”? Does writing test-after still help with code design quality? Perhaps it does if Testable Code == Good Code.

    Sorry for so many questions in the same post. I am truly excited and I hope you can help.

  34. Pingback: Testing with Stubs or testing with Mocks? | gonz0′s Blog

  35. Alex

    Hey Rick,

    To your question regarding that you have the feeling that it takes 5x longer to write the code:
    TDD basically is about design just how the author of this article says. It helps you when you begin writing your code because your mind is then free of implementation details. Thus, you can soley think about your interfaces. You could in some way compare that to you writing interfaces first and then the implementation class. But TDD in addition lets you specify exactly the contract of your methods(what are the results, preconditions etc.). As you write the tests, you are in the role of the client of that code. This lets you easier and faster find the correct design. As such it makes the development process faster. That brings me to another of your questions: If you write the implementation first and then the test, the above explanation doesn’t hold any longer as you do not gain those benefits that I just stated. At first it is difficult to really write the tests first as it’s a complete different way and we are not used to it. But I found that if you just try it you get better and better. It’s a process of learning.

    As to your question about what a “unit of code” is. Well I think that’s yours to define but I think in an OO system a single class is 99% of the time a reasonable choice. But you must not necessarily have one “Test Class” for each class altough that’s a common way to do it. If you extract a class then it would be indeed a wise decision to also move the test methods. But honestly, I use that refactoring maybe 2 times in a year ;-)

  36. Very good article, thanks.

  37. David W

    I’ve only recently adopted unit testing as a method for design and am amazed at how it’s helping my overall design. This article does a great job in explaining how to best apply this technique so I can avoid working into bad habits. Well done, and thank you!

  38. An efficient account of the most important things to pay attention to!

  39. Pingback: Coach sale

  40. Ayman Saleem

    It’s true that unit testing is not about regression or functional testing, but unit testing is not only about TDD either! Mixing the two with each other’s and introducing TDD as a synonym for unit testing would bring the other side of confusion.

    If unit testing is TDD, then it is not about detecting bugs; however, unit testing is about testing the unit itself -as you clarified- and for sure that means discovering unit-bugs rather than application/system bugs which would require more complicated cycle. Here, it does not matter how you have written your unit tests (using TDD or another approach).

    In other words, while unit tests can be one of TDD products, TDD has something else to do which might be more important for iterative development; that different product should not hide the fact that unit tests lead to discovering unit-bugs as TDD “development” methodology and Unit-Tests are not the same.

  41. This article does a great job in explaining how to best apply this technique so I can avoid working into bad habits.

  42. Unit test is very important for every projects, if you are looking for developers and testers please visit this site
    Arabic Montreal

  43. Is it my internet connection or is this website kinda slow. It took ages to load the article… Thank God I found what I was looking for.

  44. Great blog. Wish I could run a blog just like this one.

  45. yo mama

    suck my dick

  46. We absolutely love your blog and find nearly all of your post’s to be just what I’m looking for. Do you offer guest writers to write content for you personally? I wouldn’t mind publishing a post or elaborating on some of the subjects you write in relation to here. Again, awesome blog!

  47. Independent summary unwraps Unique brand new stuff regarding Buy HDMI adapter that absolutely no one is bringing up.

  48. Fantastic post! I frequently start writing a unit test only to realize that I really wanted an integration test. This will help me make a decision earlier and save a bunch of work. I’m linking to this from my blog, Hacker’s Valhalla (http://hackersv.blogspot.com/2012/04/now-i-see-realized-benefits-of-unit.html), in a related post (also about unit testing). Thanks again!