Site Meter
 
 

First steps with Lightweight Test Automation Framework

One of the ASP.NET projects that Microsoft quietly added to Codeplex in the last few months is Lightweight Test Automation Framework. It piqued my interest because I’ve been using Selenium RC to test the client-side portions of xVal, but for a while now have wanted to replace Selenium with something more streamlined (I’ll explain why later).

In this post I’ll explain what Lightweight Test Automation Framework does and how you can use it with ASP.NET MVC.

What is Lightweight Test Automation Framework?

Lightweight Test Automation Framework is an integration testing tool, which works by scripting the web browser to perform a sequence of actions against your application’s UI – entering text and clicking links and buttons exactly as a real user would – and then checking that the expected results can be seen in the browser window.

Just in case you don’t know the difference between unit testing and integration testing:

  • Unit tests work against the API of a particular component in your code. Ideally, they test only that single isolated unit (hence the name), which allows them to pinpoint any problems exactly where they occur. Unit testing tools for .NET include NUnit, mbUnit, and xUnit.
  • Integrations tests combine multiple components. Typically, they work at the UI level to test that your JavaScript, your HTML, your controller code, your model code, and your database all work together. Integration testing tools for .NET include Selenium, Watin, and Lightweight Test Automation Framework.

How does it work?

Note: If you’re not interested in the mechanisms that run behind the scenes, just skip ahead to the tutorial “How to use Lightweight Test Automation Framework with ASP.NET MVC” below.

With Lightweight Test Automation Framework, you write tests in C# (or another .NET language) – each test looks very similar to a traditional unit test. You write a series of commands (e.g., navigate to this URL, click that button, get the text from that element, assert that the text is XYZ).

Instead of running these through a traditional test runner, such as NUnit GUI or Visual Studio’s built-in unit test runner, Lightweight Test Automation Framework’s test runner actually is the web browser – any web browser. Each command (e.g., navigate to this URL, click that button) is delivered from the server to the client using Ajax.

All you have to do is add to your web application a folder called Tests containing the supplied Default.aspx and DriverPage.aspx, and then browse to /Tests. Here’s how it works:

image

As the tests run, TestcaseExecutor.js highlights passing tests in green and failing tests in red. You can click on any red test to see the server-generated stack trace corresponding the failed assertion or unhandled exception.

How does this compare to Selenium RC?

Selenium is a well-established open source web integration testing tool. Just like Lightweight Test Automation Framework, it automates your application’s UI so you can verify the results of performing a known sequence of actions. There are two main ways of using it:

  • Selenium IDE is a Firefox plugin that records a sequence of actions and generates a test script based on what you did.
  • Selenium Remote Control (RC) is a system for writing test scripts in C#, VB, JavaScript, Python, etc, which are then used to remotely control the browser UI and make assertions about what your application should display. This is what I was using to test xVal’s client-side behaviour.

Clearly, Lightweight Test Automation Framework is very similar to Selenium RC. There is a significant architectural difference, though. Selenium RC integrates with any traditional unit test runner – it does not use the browser as a test runner. Selenium RC gives you a browser automation API which works by making remote procedure calls (RPCs) to a Java-based “test server” running on your development workstation. The Java-based test server then in turn hosts a browser instance (Firefox, IE, or Safari) and uses the browser’s API to run your commands in the browser. There’s no Ajax involved. Here’s how it works:

image

So, how do we weigh up the pros and cons of each approach?

  • Selenium RC has the advantage that your application isn’t hosted in a frameset (so the test environment can’t interfere with the code being tested)
  • Selenium RC has the advantage that it works with any traditional unit testing framework (e.g., NUnit or xUnit) and any traditional unit test runner (e.g., NUnit GUI or ReSharper’s test runner). Your test code calls the Selenium RC API and makes plain old assertions about the results
  • Lightweight Test Automation Framework has the disadvantage that it relies on an asynchronous client-side polling loop (unlike Selenium RC’s RPCs, which run synchronously). Therefore, Lightweight Test Automation Framework is relatively slow – each test takes 1-3 seconds to run, versus Selenium RC which takes about 1 second per test. If you have a lot of tests then you’ll really care about this. Tip: Run your tests in Google Chrome, where tests run about 50% faster than in IE, and at least twice as fast as in Firefox.
  • Lightweight Test Automation Framework has the advantage that it doesn’t need the Java-based “test server”, and it works in any browser – not just ones it knows how to host.
Why I’m moving from Selenium RC to Lightweight Test Automation Framework

My scenario was automating tests for xVal’s client-side code. For every possible type of validation rule, we have a test to check certain inputs that should trigger a validation error message and other inputs that should be accepted. This implicitly tests all parts of xVal:

  • The system of defining and detecting each type of rule
  • The system of describing each type of rule to the client
  • The system of detecting and configuring the active client-side validation engine (e.g., jQuery Validator)
  • The system of customising validation error messages
  • The client-side validation engine itself

If any of these stop working, an integration test should detect it.

Selenium RC was doing a perfectly fine job of this, but the critical fault is that you have to run a “test server” on your workstation, and that test server is written in Java. Now I’m perfectly happy with Java, but this setup added a lot of unwanted friction in an open source .NET project. Most .NET developers don’t even have the Java runtime on their PCs, and even those that do wouldn’t realise that they have to install the Selenium RC server and run it before the unit tests could pass.

Lightweight Test Automation Framework eliminates the Java test server and makes it far easier for people to contrib
ute to xVal. That’s why I’m switching to it. The main disadvantage is that client-side tests now take longer to run, but as I pointed out before, Lightweight Test Automation Framework has the extra benefit of supporting Google Chrome, which itself is so fast that the speed loss is hardly noticeable.

How to use Lightweight Test Automation Framework with ASP.NET MVC

The ASP.NET QA team say that the Lightweight Test Automation Framework is designed to work with “ASP.NET”. They don’t say what form of ASP.NET they’re talking about, but from experience, I can tell you it’s designed with ASP.NET WebForms and ASP.NET AJAX in mind.

Nonetheless, it also works fine with ASP.NET MVC if you make a certain change to your routing config and avoid one or two features that rely on ASP.NET AJAX.

Let’s imagine you have a small ASP.NET MVC application that responds to Ajax requests. It renders an Ajax-powered form as follows:

<h2>The Magnificent Age Calculator</h2>
The following form is submitted via Ajax:
 
<% using(Ajax.BeginForm(new AjaxOptions { UpdateTargetId = "results" })) { %>
    <p>Current year: <%= Html.TextBox("currentYear", DateTime.Now.Year) %></p>
    <p>Your year of birth: <%= Html.TextBox("yearOfBirth") %></p>
 
    <p><input type="submit" id="submitButton" value="Calculate My Age" /></p>
<% } %>
 
<div id="results"></div>

Since we’re using Ajax.BeginForm(), the form will be submitted asynchronously, and the results will be injected into the DIV with ID “results”. The following action methods handles the Ajax requests:

[AcceptVerbs(HttpVerbs.Post)]
public string Index(int? currentYear, int? yearOfBirth)
{
    if(!currentYear.HasValue)
        return "Please enter the current year";
    if (!yearOfBirth.HasValue)
        return "Please enter your year of birth";
 
    var computedAge = currentYear.Value - yearOfBirth.Value;
    return "Your age is approximately: " + computedAge;
}

This behaves as shown:

image

This Ajax-powered UI behaviour cannot be tested using traditional unit testing tools, which focus on testing server-side code only. However, you can use Lightweight Test Automation Framework to automate the browser and test the application’s UI behaviour.

To get started, download Lightweight Test Automation Framework and then add a reference from your project to Microsoft.Web.Testing.Light.dll. Then create a test method as follows:

[WebTestClass]
public class AgeCalculatorTests
{
    private const int ajaxRequestTimeoutSeconds = 1;
 
    [WebTestMethod]
    public void Can_Calculate_Age()
    {
        // Arrange
        HtmlPage page = new HtmlPage("/");
 
        // Act
        page.Elements.Find("currentYear").SetText("2010");
        page.Elements.Find("yearOfBirth").SetText("1932");
        page.Elements.Find("submitButton").Click();
 
        // Assert
        var resultsElement = page.Elements.Find("results");
        resultsElement.WaitForInnerText("Your age is approximately: 78", ajaxRequestTimeoutSeconds);
    }
}

As you can see, it uses the test framework’s API to navigate to a certain URL, enter some text, click the submit button, and check the result.

To run the test, copy the /Test/ folder and its contents from the Lightweight Test Automation Framework project into your ASP.NET MVC project. To work around an incompatibility with ASP.NET MVC routing, add the following line to your routing config:

routes.IgnoreRoute("Test/{resource}.axd/{*pathInfo}");

Now you can open /Test/ in a browser and you’ll get the test runner. You can run the test you just created:

image

You can then proceed in a TDD fashion, adding a test for each new UI behaviour (before you’ve coded the behaviour!), seeing the test fail, adding the behaviour, and then seeing the test pass.

For example, perhaps there should be a special error message if you claim to be born in the future:

[WebTestMethod]
public void Gives_Error_If_Birth_Year_In_Future()
{
    // Arrange
    HtmlPage page = new HtmlPage("/");
 
    // Act
    page.Elements.Find("currentYear").SetText("2009");
    page.Elements.Find("yearOfBirth").SetText("2010");
    page.Elements.Find("submitButton").Click();
 
    // Assert
    var resultsElement = page.Elements.Find("results");
    resultsElement.WaitForInnerText("Year of birth must be in the past", ajaxRequestTimeoutSeconds);
}

I haven’t yet implemented this behaviour, so the test runner will show a failure:

image

If you want to play with this yourself, then download this demo project.

Bugs and Gotchas

Besides the minor issue with ASP.NET MVC routing, I also hit a few more confusing problems as I was porting xVal’s test code to Lightweight Test Automation Framework:

  • Test classes (i.e., ones marked with [WebTestClass]) must not be in a namespace, otherwise the test runner will break. I’m pretty sure this must be a bug, so I’ve reported it.
  • You can’t use the “WaitFor” API. This API would be very neat if it worked: it’s supposed to help you pause the test execution to leave time for asynchronous requests to complete. However, it assumes you’re doing all your Ajax requests through ASP.NET AJAX with a script manager, so it just throws an exception if you’re using jQuery or even ASP.NET MVC’s Ajax.* helpers. That’s why in my tests above I had to use resultsElement.WaitForInnerText() to wait for the Ajax requests to complete.

Other than these issues – which have obvious workarounds – it seems to work very nicely. I’m looking forward to see where this project goes in future.

kick it on DotNetKicks.com

27 Responses to First steps with Lightweight Test Automation Framework

  1. You’ve been tweeted (a good thing) – Tweetback from @elijahmanor

    http://twitter.com/elijahmanor/status/1403325985

  2. Do you have a sense of how this framework would integrate the results of the test into a CI build? Selenium RC could easily do it since a test runner is driving the tests.

  3. Neil

    Interesting stuff Steve.
    For me though, the Java bit of Selenium is a minor “inconvieneince”, compared to the relative robustness of the Selenium suite as a whole.

  4. Andrew Webb

    Hi Steve,

    Just discovered your blog – looks v interesting, and talks about the stuff we’re interested in: jQuery, ASP.NET MVC, etc. Bookmarked! A colleague found you because we’re interested in your Flash- and jQuery- based async uploader.

    May I report one thing? I get a JS error every time I hit one of your pages. Line 415:-
    st_go({blog:’3482857′,v:’ext’,post:’182′});

  5. Requirement to have a Java agent is minor, why was this the main reason for you to move to another framework?
    Can you share some of the issue you faced with Selenium ?

  6. Steve

    @Eric – good question! I haven’t tried that, because the xVal build process doesn’t involve CI. Lightweight Test Automation Framework can be invoked with query string parameters (to specify which tests to run) and can emit a log of the results to a text file, so it would be possible to integrate it. However, this is certainly not as easy as using Selenium RC which as you say works through a traditional test runner.

    @Andrew Webb – thanks for letting me know. However, I don’t get any such error, and there isn’t a JavaScript function called st_go() on my blog. If you have any more information about this then please email me directly.

    @Neil, @Ran – the requirement for running a Java app may be a minor inconvenience in some circumstances, but it can be a severe problem in others. The scenario I’m specifically addressing in this blog post is automating UI tests for a .NET open source project. In this scenario, the requirement for Java was bizarre and troubled developers who might otherwise have contributed to the project. I can’t expect them to install the entire Java runtime to their PC (plus Selenium RC itself) just to run xVal’s unit tests. That’s why it was the main reason for moving to another framework. However, I appreciate that when developing most proprietary commercial applications you have the freedom to make specifications about the development and build environment (such as “you need Java”) and in that case Selenium RC still has the edge for robustness and power.

  7. Hey Steve, Excellent work on the post. Love the graphics, thanks for trying out our Framework!

  8. David Kemp

    Excellent post – shame some of us are stuck with supporting ASP.NET 2.0 :’( (Yes, I know about MonoRail)
    Have you thought about using IKVM.NET to run the Selenium server? I’ve not used it, but it seems like the ideal solution to having to have JRE on your PC.

  9. Steve

    @David – I hadn’t heard of IKVM before! Interesting. I should point out that I’m perfectly happy to have the Java runtime environment on my PC (and in fact I do). I have no problem with Java! The scenario I’m trying to address is other developers who frequently don’t have JRE installed and in fact don’t want to install anything new at all (which counts out IKVM too).

  10. Diego Guidi

    looks cool :)

  11. If the Java requirement is your main beef, you might be interested to know that Selenium 2.0 is the product of a merger between WebDriver and Selenium. The result will be no intermediate “server” component unless you truly need true remote control capabilities. Perhaps you’d like to join us as we build it out?

  12. Pingback: Infonicles: Testing Automation

  13. Pingback: Integration Testing Your ASP.NET MVC Application « Steve Sanderson’s blog

  14. “In this scenario, the requirement for Java was bizarre and troubled developers who might otherwise have contributed to the project. I can’t expect them to install the entire Java runtime to their PC (plus Selenium RC itself) just to run xVal’s unit tests.”

    The ENTIRE runtime? C’mon – we expect our clients to install the .Net runtime, but you can’t expect your developers to install a Java runtime? Plus Selenium RC doesn’t need to be installed – you can simply include it in source control.

  15. Steve

    @Jaco – a point I’ve made repeatedly in this blog post and its comments (e.g., #6) is that this is about a specific .NET *open-source* project. I can’t control what software is on potential developers’ PCs. There’s no reason to expect a .NET developer to have Java Runtime Environment installed, and they’re more likely to abandon any potential contribution than to install JRE. I know I would.

    Lightweight Test Automation Framework is for me just a way to avoid nonstandard dependencies such as Java.

  16. That might make sense to me if the 2 frameworks were very similar – which they’re not. Selenium has an IDE which scripts your tests for you and can be integrated into an automated build process. This framework doesn’t have an IDE and is more difficult to integrate. As far as I can see you’re making your life more difficult. I just don’t like to see this ‘must-be-dotnet’ fixation. The pro’s simply don’t outweight the con’s here.

  17. Pahe

    Nice post.. Have been using webaii lately (http://www.artoftest.com/Resources/WebAii/), and Iam amazed that not more people talks about it.

  18. Bernhard Hofmann

    Thanks for writing about this, I had not heard of “LTAF” (as I’ll call it for now). The tests look very much like those in WatiN. You seemed to focus your comparison on Selenium vs. LTAF and WatiN was left out of the mix. I like how we can run our WatiN tests automatically as part of a build and CI. It seems to me from your article that LTAF does not support automated or scheduled execution of the tests. For us this is a major concern, so I think we’ll be focusing our comparisons on Selenium and WatiN.

  19. Fabio Milheiro

    Hi, I am trying to test the ajax form you have, but the response is not appended to the results div (I have JavaScript enabled and tested both in IE and Firefox). Instead, the response (the string return by the Index action) is shown in a new page. What’s wrong?

    I am learning and I still did not get to the Ajax part, so I count on you to help me with this so I can test the LTAF.

    Thank you Steve! Very good information!

  20. Fabio Milheiro

    I got already! I was missing the fact that I need the JS files in the head of the Site.Master file.

    Thank you anyway Steve!

  21. Fabio Milheiro

    Now, I got a question that makes more sense. Can we use LTAF to test non-Ajax forms?

    I’m trying to do that, but I am not succeeding because I cannot get the url of the send URL. If we can get the URL of the second form, that would be just great.

    Do you or anyone else have any thoughts about this?

    Thanks!

  22. Steve

    @Bernhard – I focused on Selenium vs LTAF purely because I have experience with Selenium but not so much with WatiN. I’ve heard a lot of positive things about WatiN, so I might well try that next time I need to automate a browser.

    @Fabio – You certainly can use it to test non-Ajax forms. If you have specific tech support questions then please direct them to the LTAF folks on Codeplex.

  23. Pingback: Servefault.com

  24. Pingback: Herding Code 77: Eric Hexter on MvcConf, C4MVC, and MvcContrib | Herding Code

  25. Pingback: New TesterTools – LTAF | Software Testing Tools Blog - Testertools

  26. dispenses employ a good web site decent Gives appreciate it for the work to support me

  27. Whats up very cool blog!! Man .. Excellent .. Amazing .. I will bookmark your blog and take the feeds also¡KI am glad to seek out so many helpful information here within the post, we want work out extra techniques in this regard, thanks for sharing. . . . . .