Site Meter
 
 

Experiments with Koa and JavaScript Generators

Koa is a new application framework for Node.js. Its whole point is to eliminate the callback madness that plagues many Node.js apps. It achieves this previously-impossible feat by using the powerful generators feature which is new in JavaScript ES6. Koa is built by the same people who created Express, the best-known Node.js application framework, so it’s at least worth a look.

Having written plenty of callback-heavy JavaScript, I was excited to give this thing a go and see how much simpler it could make my code.

What have generators got to do with async?

On the surface, nothing. Generators are a way of returning an arbitrary sequence of results from a function, with the function’s execution suspended in between results. Here’s one:

// This is a generator function. The star preceding the function name means
// it will return a 'generator' object, even though none of the code inside
// explicitly says so.
function *getAllSquareNumbers() {
    for (var i = 1; ; i++) {
        // Every time we 'yield', this function's execution pauses until
        // the generator is restarted by a call to 'next' (see below).
        yield i * i;
    }
}
 
// Now let's fetch some values
var generator = getAllSquareNumbers();
console.log(generator.next().value); // Outputs '1'
console.log(generator.next().value); // Outputs '4'
console.log(generator.next().value); // Outputs '9'
console.log(generator.next().value); // Outputs '16'
// We could go on. That's enough.

Each time you call the generator’s next function, it returns an object of the form { value: ..., done: ... }, where value is the next thing to be yielded by the generator function, and done is a boolean that becomes true if the generator function has exited (e.g., by calling return).

For C# developers, this is exactly equivalent to how the yield keyword has worked since C# 2.0 in 2005.

So again, how does this help with async programming?

When I first heard about Koa eliminating callbacks, I thought “Oh I see, I guess the generator emits values asynchronously“. This turns out to be completely incorrect, and in fact doesn’t even make sense when you have a clear mental picture of generators. But because I was fixated on this (wrong) view of how Koa would work, I was totally confused for some time, and couldn’t make it do what I thought it should do. Allow me to spare you any such confusion.

Generators yield values synchronously.

They cannot yield values asynchronously. It wouldn’t make sense for them to do so, because the code receiving these values doesn’t necessarily know anything about ES6 Generators, and it’s always been a basic axiom of JavaScript that your code’s execution can’t be blocked (except now for functions that opt in by having a star in their name, i.e., generator functions). So when you call generator.next() from regular code, you will receive the next value synchronously.

Well then, for the third time, how are generators going to help us with async code, considering that they have to be synchronous?

What if the sequence of values returned was in fact a sequence of promises, or other objects that describe asynchronous tasks? In that case, the code consuming the sequence could choose to wait for each one to complete before calling .next to get the next one. And that’s how Koa works – your application code is the generator, it emits a series promises (or other things I’ll show below), and Koa waits for each one to complete before resuming your code (passing back to you the result of the previous task).

And how does Koa pass back results into your generator? That’s another feature of ES6 generators. If you pass a value to next, e.g., generator.next(123), then that value is given to the generator code as the value of the currently-blocked yield expression. Hence code like var nextData = yield something; (it emits something, then some time later receives nextData).

Koa Example

Here, then, is a trivial Koa example:

var koa = require('koa'),
    app = koa();
 
app.use(function *() {
    // Here is the important bit of application logic for this example.
    // We make use of a series of async operations without callbacks.
 
    var city = yield geolocation.getCityAsync(this.req.ip);
    var forecast = yield weather.getForecastAsync(city);
 
    this.body = 'Today, ' + city + ' will be ' + forecast.temperature + ' degrees.';
});
 
app.listen(8080);

The application code provided to Koa is a generator function (in this case, an anonymous generator function, hence function *() { ... } ). Each yield expression supplies a promise (or other task description) to Koa and suspends execution. Koa waits for each such promise to complete, then resumes your generator, passing you the most recent async result value.

The net result is that you get to consume asynchronous operations using a syntax that barely differs from if they were synchronous.

Error handling

Also great news is that error handling works really cleanly too. If the async task reports an error (for example, promises have ‘failure’ callbacks), then Koa uses the generator’s throw feature to dispatch an exception to whatever yielded that task. You can therefore catch exceptions just like in synchronous code, or let them cause the request to fail (instead of being silently ignored, as async exceptions often are by mistake).

Doing async Koa, or “What can I yield?”

I looked for this info in the Koa docs and couldn’t find it, so here is a list of the possible things you can yield to Koa and have it work properly.

In fact this turns out to be documented for the co Node module, which is what Koa uses behind the scenes.

1. Promises

Typically the most convenient. If you have any library that describes async tasks as Promise-like objects, you can just yield them to Koa. Example:

// Example of a manually-constructed promise using the 'Q' promises library
var Q = require('q');
 
function delay(milliseconds) {
    var deferred = Q.defer();
    setTimeout(deferred.resolve, milliseconds);
    return deferred.promise;
}

Now inside an app.use generator you can do this to insert a pointless delay:

app.use(function *() {
    yield delay(100); // Note: this yield has no useful return value
});

Or for a more realistic example, you can use Q’s denodify feature to create a version of the request module that returns promises:

var request = Q.denodeify(require('request'));
 
// Example of calling library code that returns a promise
function doHttpRequest(url) {
    return request(url).then(function(resultParams) {
        // Extract just the response object
        return resultParams[0];
    });
}

And now inside your app.use generator you can:

app.use(function *() {
    // Example with a return value
    var response = yield doHttpRequest('http://example.com/');
    this.body = "Response length is " + response.body.length;
});

Yay, HTTP requests with no callbacks!

Note: This works with any “thenable”, which is the looser notion of a promise defined in the Promises/A+ spec. Every sane promises library conforms to this.

2. Thunks (ignore these if you’re not comfortable passing around function objects)

Thunks? Huh? Koa understands thunks to be functions you call with a single parameter, callback, to receive a result value (or error description) when it’s done.

To begin, let’s imagine you have the following existing code that uses Node.js’s traditional callback style to return a result. Ignore the fact that this totally does not need to be asynchronous. Imagine it’s a database query or something.

// Traditional Node-style callback async pattern (neither a promise nor a thunk)
function getSquareValueAsync(num, callback) {
    setTimeout(function() {
        var result = num * num;
        callback(/* error: */ null, result);
    }, 500);
}

Now, Koa can’t understand this function directly, because it has no idea how many (or what) parameters it takes. But you can turn it into a thunk by wrapping it as follows:

// Wrap the async operation in a 'thunk' so that Koa understands it
function getSquareValueThunk(num) {
    // A 'thunk' returns a function that accepts a callback
    return function(callback) {
        getSquareValueAsync(num, callback);
    };
}

The whole point of this wrapping is to eliminate all parameters other than callback, so that Koa knows how to consume it. Now you can yield this thunk to Koa inside any of your app.use generators, e.g.,

app.use(function *() {
    var square = yield getSquareValueThunk(16);
    this.body = "Square of 16 is " + square;
});

To clarify, calling getSquareValueThunk(16) does not actually do anything except create and return a new function instance, i.e., a function that takes a single callback parameter. Whenever you yield a Function instance to Koa, then Koa assumes it may invoke it with a single callback parameter, and that callback will eventually be invoked as callback(error, result).

If you don’t like or understand this, just use promises. Promises are almost certainly a better choice for application code. Thunks are only good as a low-level primitive used internally by Koa because they don’t involve any particular promises library.

3. Generators

OK, here it gets interesting. Generators can emit generators which emit generators, etc. This makes it possible to construct async tasks out of other async tasks, with arbitrary nesting.

Imagine we want to invoke getSquareValueThunk multiple times in series, and wrap that series up as a single operation. Easy: just make another generator.

function *getSquareOfSquareOfSquare(num) {
    // Each of these is an async operation
    var square = yield getSquareValueThunk(num);
    var squareOfSquare = yield getSquareValueThunk(square);
    return yield getSquareValueThunk(squareOfSquare);
}

Notice the star preceding the function name. This means it’s legal to use yield inside it, and that it returns a generator. This generator returns a series of thunks (though promises would work equally). Now you can insert this series into your async pipeline by yielding it from any app.use code:

app.use(function *() {
    var bigNumber = yield getSquareOfSquareOfSquare(16);
    this.body = "16 to the 8th power is " + bigNumber;
});

This is how a nontrivial Koa app will be structured. The top-level app.use generators will yield generators returned by mid-tier application code (such as business logic) which in turn will yield generators returned by lower-level application logic (such as data access). All the way down, your source code looks synchronous (no callbacks), but is actually fully async. Of course you aren’t necessarily doing three-tier architecture, but you get the idea :)

Just as an aside, this technique could be used to simplify the doHttpRequest function you saw earlier in this blog post. The original version is:

function doHttpRequest(url) {
    return request(url).then(function(resultParams) {
        return resultParams[0];
    });
}

… and it can be replaced with the following generator, with no need for a then callback:

function *doHttpRequest(url) {
    var resultParams = yield request(url);
    return resultParams[0];
}

4. Arrays

If you have multiple simultaneous tasks and want to wait for all of them, just yield an array of them, e.g.,

app.use(function *() {
    var urls = [
        'http://example.com/',
        'http://twitter.com/',
        'http://bbc.co.uk/news/'
    ];
 
    // On this line, all three requests begin in parallel
    // Note: doHttpRequest is defined earlier in this blog post
    var arrayOfPromises = urls.map(doHttpRequest);
 
    // When we yield the array of promises here, Koa will
    // wait for all of them to complete
    var arrayOfResponses = yield arrayOfPromises;
 
    this.body = "Results";
    for (var i = 0; i < urls.length; i++) {
        this.body += '\n' + urls[i] + ' response length is '
              + arrayOfResponses[i].body.length;
    }
});

As you can see, the result of yield arrayOfPromises; is another array, with the corresponding entries being the promise results.

The array you yield doesn’t have to be purely promises – it can also contain thunks, generators, other arrays, or key-value objects (see below). The structure can be nested arbitrarily.

5. Key-value objects

If you have multiple simultaneous tasks you want to wait for, and you want to give useful names to each of them, use an object. Example:

app.use(function *() {
    var tasks = {
        imposeMinimumResponseTime: delay(500),
        fetchWebPage: doHttpRequest('http://example.com/'),
        squareTenSlowly: getSquareValueThunk(10)
    };
 
    // All of the operations have already started. Yielding
    // the key-value object to Koa just makes it wait for them.
    var results = yield tasks;
 
    this.body = 'OK, all done.';
    this.body += '\nResult of waiting is: ' + results.imposeMinimumResponseTime; // Displays: undefined
    this.body += '\nWeb page status code is: ' + results.fetchWebPage.statusCode; // Displays: Typically 200 in this case
    this.body += '\nSquare of ten is: ' + results.squareTenSlowly; // Displays: 100
});

It’s just easier to read than referring to your tasks as tasks[0], tasks[1], etc.

Isn’t this a lot like C# 5′s async and await?

Yes, very much like that indeed! The * in function *() { } plays a similar role to C#’s async keyword, and JavaScript’s yield (at least when used inside Koa or a similar framework) plays a similar role to C#’s await.

The usage patterns of */async and yield/await are very similar. The main differences are to do with the underlying implementation. In C#, await handles asynchrony as a language feature and has a built-in standard notion of Task, whereas in JavaScript, yield can be used to handle asynchrony (when you supply some library code to describe async tasks and some other library code to wait for them), but it can also be used for other things, and the JavaScript language doesn’t know or care what you’re using yield for.

For C# developers, it would be fun to use the same technique to recreate C# 5′s async/await patterns purely using C# 2.0′s yield keyword.

Learning more about Koa

This blog post is purely about generators and async in Koa. If you want to actually build an app with Koa, go and read the framework’s website. If you’ve never used Express (the earlier incarnation of Koa’s ideas) then be sure to learn about middleware, the central architectural pattern in Koa, so you will see how the framework intends your application to be organised.

Knockout Projections – a plugin for efficient observable array transformations

knockout-projections is a new Knockout.js plugin that adds efficient “map” and “filter” features to observable arrays. This means you can:

  • Write myObservableArray.map(someMappingFunction) to get a new, read-only array-valued observable containing the mapped version of each input item.
  • Write myObservableArray.filter(someFilterFunction) to get a new, read-only array-valued observable containing a subset of the input items.

When the underlying myObservableArray changes, or whenever any observable accessed during the mapping/filtering changes, the output array will be recomputed efficiently, meaning that the map/filter callbacks will only be invoked for the affected items.

The point of all this: it can scale up to maintain live transforms of large arrays, with only a fixed cost (not O(N)) to propagate typical changes through the graph of dependencies.

Trivial mapping example

To illustrate the mechanics in an obvious way, consider this underlying array:

var numbers = ko.observableArray([1, 2, 3]);

Now if you’ve referenced knockout-projections, you can write:

var squares = numbers.map(function(x) { return x*x; });

Initially, squares will contain [1, 4, 9]. It’s observable, so you can use squares.subscribe to get notifications when it mutates, or you can bind it to a DOM element (e.g., foreach: squares) in KO’s usual way.

Now if you transform the underlying array:

numbers.push(8);

… then squares updates (to [1, 4, 9, 64]) and only calls your mapping function for the new item, 8.

Any transformation is permitted, e.g.:

numbers.reverse();

This has the effect of reversing squares (to [64, 9, 4, 1]), again without remapping anything.

If you remove item(s):

numbers.splice(1, 1);

… then squares is updated (here, to [64, 1]) without remapping anything.

In summary, anyObservableArray.map is an efficient, observable equivalent to Array.prototype.map. Also, it can be arbitrarily chained (and combined in chains with filter) to produce graphs of transformations.

Trivial filtering example

The filter feature works exactly as you’d expect, given the above. For example,

var evenSquares = squares.filter(function(x) { return x % 2 === 0; });

Initially, evenSquares will contain just [64]. When you mutate the underlying array,

numbers.push(4); // evenSquares now contains [64, 16]
numbers.push(5); // evenSquares doesn't change
numbers.push(6); // evenSquares now contains [64, 16, 36]

Again, it responds to arbitrary transformations:

numbers.sort();  // evenSquares now contains [16, 36, 64]

A more realistic use case

Typically you won’t just be playing around will small collections of numbers. Most Knockout apps work with collections of model objects – sometimes very large collections.

Here’s a simple model object:

function Product(data) {
    this.id = data.id;
    this.name = ko.observable(data.name);
    this.price = ko.observable(data.price);
    this.isSelected = ko.observable(false);
}

Many KO applications involve fetching a large collection of model objects and exposing that from a viewmodel:

function PageViewModel() {
    // Some data, perhaps loaded via an Ajax call
    this.products = ko.observableArray([
        new Product({ id: 1, name: 'Klein Burger', price: 3.99 }),
        new Product({ id: 2, name: 'Mobius Fries', price: 1.75 }),
        new Product({ id: 3, name: 'Uncountable Chicken Chunks', price: 3.59 }),
        new Product({ id: 4, name: 'Mandelbrot Salad', price: 2.40 }),
        ... etc ...
    ]);
}
 
ko.applyBindings(new PageViewModel());

Now this might be bound to the UI:

<ul data-bind="foreach: products">
    <li>
        <input type="checkbox" data-bind="checked: isSelected" />
        <strong data-bind="text: name"></strong>
        (Price: £<span data-bind="text: price().toFixed(2)"></span>)
    </li>
</ul>

This is all fine, and Knockout is already good at efficiently (i.e., incrementally) updating the UI when the products array changes. But what if you want to track which subset of products is “selected”?

The traditional approach would be something like:

this.selectedProducts = ko.computed(function() {
    return this.products().filter(function(product) {
        return product.isSelected();
    });
}, this);

This works (using Array.prototype.filter, not the Knockout-projections filter function). However, it’s inefficient. Every time the products array changes, and every time any of their isSelected properties changes, it re-evaluates the isSelected property of every product. It has to do so, because it has no built-in understanding of what you’re doing, so it can’t be clever and incremetally update the earlier selectedProducts array.

However, if you use Knockout-projections filter function, e.g.:

this.selectedProducts = this.products.filter(function(product) {
    return product.isSelected();
});

… it’s both syntactically cleaner, and way faster for large arrays: it now updates the selectedProducts array incrementally whenever either products changes or any of the isSelected values changes.

Similarly you might want to output the names of the selected items. So, you could chain on a new selectedNames property, and perhaps furthermore chain on selectionSummary:

this.selectedNames = this.selectedProducts.map(function(product) {
    return product.name();
});
 
this.selectionSummary = ko.computed(function() {
    return this.selectedNames().join(', ')  || 'Nothing';
}, this);

Now when products changes, or when an isSelected or name property changes, the effect will propagate out incrementally through the whole dependency graph with the minimum of callback-invoking, meaning a snappy user experience even with large data sets and low-end mobile devices.

Licence and origin

knockout-projections is open source under the Apache 2.0 license. It was built as a tiny component in a very large project I’m working on at Microsoft, and thanks to my boss (and his boss, and probably multiple people up the chain) we’ve decided to open-source it. Hopefully my team will be producing plentiful nuggets of OSS goodness for you as we go about our work :)

Just a clarification for the avoidance of any confusion, Knockout.js itself remains MIT licensed and is run by the KO community and the KO core team members (which includes me personally).

A standing desk with (mostly) Ikea parts

Most developers by now are familiar with the standing desk trend. This post isn’t about the pros/cons, suggestions about how long you should stand for, or anything like that. Let’s just say that like me you want to give it a go, but you’re not inclined to spend many hundreds of pounds on a brand new high-end desk until you’re sure you really will use it for the next five years or so.

Many others have built their own standing desks before, even with Ikea parts. So this is hardly revolutionary. But it’s a good looking, inexpensive design, that others may find interesting.

Finished result

This standing desk sits on top of your existing sitting desk, and is designed to take up roughly half of a typical desk width. So you’ll have a half-sitting, half-standing desk, and can easily move between the two postures:

Desk from front

Pros:

  • Transportable. Can easily move it around between desks/workplaces (unscrew the legs by hand, carry it over)
  • Good ergonomics. Gives a lot of room around your keyboard and mouse, and keeps the monitor a pleasing distance from your face, compared with alternatives.
  • Cheap(ish), versus commercial pre-built standing desks, yet looks professional.

Cons:

  • Not height-adjustable, unless you count taking a hacksaw to the legs.
  • Only wide enough for a single large monitor
    • To be fair, there’s nothing to stop you putting on a wider shelf. It’s easily strong enough to take a second large monitor. It would just look a bit more imposing.
    • However, since I switched back to having a single large monitor a few years ago, this is exactly how I want it. Less neck-swivelling = more comfort for me personally.
  • Cheaper DIY alternatives exist
  • Requires a trip to Ikea. But hey, meatballs, right?

Ingredients

Many of the following components can be substituted with alternatives, depending on what you have available. For example, there’s probably no real reason to use Sugru if you already have some strong glue such as Araldite. And any similar-sized piece of wood will do in place of the Ekby Laiva shelf.

From Ikea:

Order online:

Total cost: from £41.24 to £70.99 depending on optional parts. But I do recommend some kind of mat. And I assume you have tools (a drill, a hacksaw) and a range of small screws, since the ones that Ikea include are too big.

Method

When you see the list of parts, it’s pretty obvious how to construct it. So I’m not going to list every screw, cut, or measurement – I’ll just give an outline with some tips. Here’s another viewpoint on the desk to clarify its arrangement:

Desk at angle

Getting the right height

Start here because it’s the most important bit. I’ve read that your standing desk surface should be at roughly elbow-height, but I like mine about 8-10cm below that. You will need to account for the height of the underlying desk and thickness of the table top (obviously), and perhaps the height of your keyboard or anti-fatigue mat if either are big. You might want to first construct a mock-up by balancing the table top on some boxes or whatever.

When you’re sure of the height, measure the Adils legs from the circular bracket at the top downwards, and subtract from your measurement about 1cm to account for the feet you’ll attach later. Now hacksaw the legs to length, and retain the adjustable feet for later.

Screw the circular Adils leg brackets into the underside of the table top now, but don’t screw the legs into them just yet.

Fitting the monitor stand

Be careful when planning how far back to mount the monitor stand to ensure the monitor’s centre of gravity will end up within the desk’s four legs. For me, the desk’s size and the angle of the Capita brackets result in a pretty ideal natural distance from my face to the monitor, but individual preferences vary.

The Capita brackets are a bit baffling because there are so many possible ways of attaching them. I’d recommend first drilling the two 9mm(ish) holes in your table top, loosely bolting on the Capita brackets, and then marking out where the screw holes naturally land on the underside of the Ekby Laiva shelf.

Then drill pilot holes into the Ekby Laiva shelf, bolt the Capita brackets tightly to the table top, and finally screw the Ekby Laiva shelf to the top of the Capita brackets with some small screws.

Finishing the legs

You can now screw in the Adils legs to their brackets, and ta-da – your standing desk is usable. However the bottom end of the legs is just roughly sawn metal, which will make a mess of whatever desk it stands on.

Fortunately you can take the feet off the sawn-off parts of the Adils legs and attach them to the bottom of your new desk’s legs. You can stick them on with strong glue such as Araldite. Instead, I used rolled-out circles of Sugru (see photo below), partly just because I was in a “hack your stuff” mood, and partly because it let me compensate for some irregularities in my unskilled hacksawing:

Desk feet with Sugru

Review

I’ve only been using this for a few weeks now, but the desk feels great to use – it’s sturdy, spacious, and looks good in my office. My elbow pain (the reason I did this in the first place) was gone after the first two days. I tend to stand through the morning and sit in the afternoon, though maybe I should reverse the order to minimise post-lunch lag.

After the first couple of multi-hour standing-working sessions, it was clear I couldn’t just stand on carpet in socks (or shoes) for hours. Your feet will probably tell you the same. I stood on a foam pillow for the next few days, and it was hugely better. For a better, more long-term solution, follow the advice of standing-mat devotees everywhere and get a nice gel mat to stand on, such as the Dandy one I mentioned above. These mats are a bit pricey, but feel so much better.