Site Meter
 
 

Category: Knockout

Knockout Projections – a plugin for efficient observable array transformations

knockout-projections is a new Knockout.js plugin that adds efficient “map” and “filter” features to observable arrays. This means you can:

  • Write myObservableArray.map(someMappingFunction) to get a new, read-only array-valued observable containing the mapped version of each input item.
  • Write myObservableArray.filter(someFilterFunction) to get a new, read-only array-valued observable containing a subset of the input items.

When the underlying myObservableArray changes, or whenever any observable accessed during the mapping/filtering changes, the output array will be recomputed efficiently, meaning that the map/filter callbacks will only be invoked for the affected items.

The point of all this: it can scale up to maintain live transforms of large arrays, with only a fixed cost (not O(N)) to propagate typical changes through the graph of dependencies.

Trivial mapping example

To illustrate the mechanics in an obvious way, consider this underlying array:

var numbers = ko.observableArray([1, 2, 3]);

Now if you’ve referenced knockout-projections, you can write:

var squares = numbers.map(function(x) { return x*x; });

Initially, squares will contain [1, 4, 9]. It’s observable, so you can use squares.subscribe to get notifications when it mutates, or you can bind it to a DOM element (e.g., foreach: squares) in KO’s usual way.

Now if you transform the underlying array:

numbers.push(8);

… then squares updates (to [1, 4, 9, 64]) and only calls your mapping function for the new item, 8.

Any transformation is permitted, e.g.:

numbers.reverse();

This has the effect of reversing squares (to [64, 9, 4, 1]), again without remapping anything.

If you remove item(s):

numbers.splice(1, 1);

… then squares is updated (here, to [64, 1]) without remapping anything.

In summary, anyObservableArray.map is an efficient, observable equivalent to Array.prototype.map. Also, it can be arbitrarily chained (and combined in chains with filter) to produce graphs of transformations.

Trivial filtering example

The filter feature works exactly as you’d expect, given the above. For example,

var evenSquares = squares.filter(function(x) { return x % 2 === 0; });

Initially, evenSquares will contain just [64]. When you mutate the underlying array,

numbers.push(4); // evenSquares now contains [64, 16]
numbers.push(5); // evenSquares doesn't change
numbers.push(6); // evenSquares now contains [64, 16, 36]

Again, it responds to arbitrary transformations:

numbers.sort();  // evenSquares now contains [16, 36, 64]

A more realistic use case

Typically you won’t just be playing around will small collections of numbers. Most Knockout apps work with collections of model objects – sometimes very large collections.

Here’s a simple model object:

function Product(data) {
    this.id = data.id;
    this.name = ko.observable(data.name);
    this.price = ko.observable(data.price);
    this.isSelected = ko.observable(false);
}

Many KO applications involve fetching a large collection of model objects and exposing that from a viewmodel:

function PageViewModel() {
    // Some data, perhaps loaded via an Ajax call
    this.products = ko.observableArray([
        new Product({ id: 1, name: 'Klein Burger', price: 3.99 }),
        new Product({ id: 2, name: 'Mobius Fries', price: 1.75 }),
        new Product({ id: 3, name: 'Uncountable Chicken Chunks', price: 3.59 }),
        new Product({ id: 4, name: 'Mandelbrot Salad', price: 2.40 }),
        ... etc ...
    ]);
}
 
ko.applyBindings(new PageViewModel());

Now this might be bound to the UI:

<ul data-bind="foreach: products">
    <li>
        <input type="checkbox" data-bind="checked: isSelected" />
        <strong data-bind="text: name"></strong>
        (Price: £<span data-bind="text: price().toFixed(2)"></span>)
    </li>
</ul>

This is all fine, and Knockout is already good at efficiently (i.e., incrementally) updating the UI when the products array changes. But what if you want to track which subset of products is “selected”?

The traditional approach would be something like:

this.selectedProducts = ko.computed(function() {
    return this.products().filter(function(product) {
        return product.isSelected();
    });
}, this);

This works (using Array.prototype.filter, not the Knockout-projections filter function). However, it’s inefficient. Every time the products array changes, and every time any of their isSelected properties changes, it re-evaluates the isSelected property of every product. It has to do so, because it has no built-in understanding of what you’re doing, so it can’t be clever and incremetally update the earlier selectedProducts array.

However, if you use Knockout-projections filter function, e.g.:

this.selectedProducts = this.products.filter(function(product) {
    return product.isSelected();
});

… it’s both syntactically cleaner, and way faster for large arrays: it now updates the selectedProducts array incrementally whenever either products changes or any of the isSelected values changes.

Similarly you might want to output the names of the selected items. So, you could chain on a new selectedNames property, and perhaps furthermore chain on selectionSummary:

this.selectedNames = this.selectedProducts.map(function(product) {
    return product.name();
});
 
this.selectionSummary = ko.computed(function() {
    return this.selectedNames().join(', ')  || 'Nothing';
}, this);

Now when products changes, or when an isSelected or name property changes, the effect will propagate out incrementally through the whole dependency graph with the minimum of callback-invoking, meaning a snappy user experience even with large data sets and low-end mobile devices.

Licence and origin

knockout-projections is open source under the Apache 2.0 license. It was built as a tiny component in a very large project I’m working on at Microsoft, and thanks to my boss (and his boss, and probably multiple people up the chain) we’ve decided to open-source it. Hopefully my team will be producing plentiful nuggets of OSS goodness for you as we go about our work :)

Just a clarification for the avoidance of any confusion, Knockout.js itself remains MIT licensed and is run by the KO community and the KO core team members (which includes me personally).

Knockout 3.0 Release Candidate available

Last week we published the Knockout 3.0 Release Candidate. Hopefully this is the last prerelease version of KO 3.0! The final release should be out in the coming weeks depending on developer feedback.

What’s new?

The biggest chunk of new v3 functionality was already included in the 3.0 Beta release. The beta included a huge overhaul of the binding system that makes it more powerfully extensible. In particular, this included:

  • Node binding and rewriting which lets you define custom binding syntaxes, and

  • Independent and ordered bindings which fixes the most longstanding difficulties with complex combinations of bindings.

See my beta blog post for details and examples.

Since the beta, we’ve focused mainly on performance, stability, and backward compatibility. Of course, we couldn’t resist including a few new features too… So, new in the release candidate is:

  • Array change subscriptions – a super-fast way to find out how an observable array has changed (i.e., which items were just added/deleted/moved) without having to run any differencing algorithm. See the example later.
  • Binding to detached nodes so frameworks built on top of Knockout have an easier time organizing all the DOM fragments they work with in the background.
  • Clearer error reporting if binding hits a problem. After all, nobody enjoys debugging…
  • More helpful handling of arrays in which each entry is an observable (as distinct from observable arrays, which have of course always worked nicely).
  • Performance improvements
    • Computed properties no longer issue change notifications by default if their computed value is definitely unchanged since last time.
    • Reduced by almost half the number of hidden, internal observables that Knockout constructs to manage the state of your bindings.
    • Reduced the stack depth when processing chained observable notifications by four call frames per observable, permitting much longer chains.
  • Bug fixes including HTML-encoding dropdown-list captions, and reinstating the .toJSON function on the output from ko.toJS (which was inadvertently omitted in KO v3 beta).
  • New build system based on Grunt.js to make contributing to Knockout easier. At last all the custom Bash scripts are gone :)
    • Fun fact: as recently as 9 months ago, Knockout still included an MS-DOS batch-file build system. Ah, the magic of %~dp0

Array change subscriptions

Full docs will be available with the final release, but for now, here’s an example of how to receive array change descriptions with KO 3. This will be useful for anybody processing complex chains of observable arrays, or using them for anything besides binding to the UI (which is already automatic).

Whenever you have an observableArray, you can now subscribe to its new arrayChange notifications:

var myArray = ko.observableArray(["Alpha", "Beta", "Gamma"]);
 
myArray.subscribe(function(changes) {
 
    // For this example, we'll just print out the change info
    console.log(changes);
 
}, null, "arrayChange");

Now if you add/remove/reorder its contents, you’ll be given a concise description of what happened. For example, if you add an item (e.g., changing ['Alpha', 'Beta', 'Gamma'] to ['Alpha', 'Beta', 'Gamma', 'Delta']):

myArray.push("Delta");
// Console output: [{ index: 3, status: 'added', value: 'Delta' }]

Or if you delete an item (e.g., changing ['Alpha', 'Beta', 'Gamma', 'Delta'] to ['Alpha', 'Beta', 'Gamma']):

myArray.pop();
// Console output: [{ index: 3, status: 'deleted', value: 'Delta' }]

Or if you perform a combination of deletions and additions at once (e.g., changing ['Alpha', 'Beta', 'Gamma'] to ['Alpha', 'New value']):

myArray.splice(1, 2, "New value");
// Console output:
// [{ index: 1, status: 'deleted', value: 'Beta' },
//  { index: 1, status: 'added', value: 'New value' },
//  { index: 2, status: 'deleted', value: 'Gamma' }]

Or if you change the order of items (e.g., from ['Alpha', 'New value'] to ['New value', 'Alpha']):

myArray.reverse();
// Console output:
// [{ index: 0, moved: 1, status: 'deleted', value: 'Alpha' },
//  { index: 1, moved: 0, status: 'added', value: 'Alpha' }]

In this last case, notice that as a consumer of this information, you can optionally ignore the moved information and just interpret it as the original Alpha being deleted, and a different Alpha being added to the array’s end. Alternatively you can recognize that the moved information tells you that you can think of the added and deleted items being the same thing just changing position (by matching up the indexes). Either is a valid interpretation – it’s up to you.

So what? I could do this before using ko.utils.compareArrays

Yes, you could, but the benefits of arrayChange are:

  • Performance is O(1) in most cases, i.e., there’s basically no performance implication at all, because for straightforward operations, (push, splice, etc.) KO supplies the change log without running any diff algorithm. KO now only falls back on the diff algorithm if you’ve made an arbitrary change without using a typical array mutation function.
  • The change log is filtered down just to give you the items that actually changed.

Coming soon: A powerful new plugin, built using arrayChange, for efficiently processing chains of maps and filters on observable arrays. Just as soon as I get permission to publish it :)

Please help us by trying it out

We can only release the final Knockout v3 when we have a good level of confidence that it’s as solid and backward-compatible as we expect. To make that happen soon, please try the v3 RC with your existing or new apps and tell us how it goes.

We’ll provide more detailed upgrade information with the final Knockout v3 release, but for now, here a few tips on the (deliberate) breaking changes since v2.3.0:

  • Computed properties now only notify when their output changes. We polled Knockout developers, and the feedback was overwhelmingly in favour of this change. For most developers, this won’t affect you except by making your app perform better. In the rare case where you still want a ko.computed to issue “change” notifications after re-evaluation even if the result is identical to before, you must now configure it using notify like this:

    myComputed.extend({ notify: 'always' });
  • Bindings on a given element now update independently. This is another improvement that KO developers have wanted for a long time. Again, in most cases, it will only affect you by making your app perform better. But if you built custom binding handlers that relied on the undocumented implementation detail that v2.x updates all bindings on the same element at the same time, you may have to stop relying on that detail.

  • optionsCaption now HTML-encodes its output. This is desirable for safety and consistency with every other text-outputting binding (e.g., text). But if you were previously HTML-encoding the string used with optionsCaption, you’ll now have to stop doing so or it will get double-encoded.

Credits

As usual, huge thanks to the team and community members who put the work in to get this implemented and released. Special acknowledgement to Michael Best who designed and built the overhauled binding system (among much more), and to Larry Gordon who instigated the move to Grunt.js. And of course to everyone else who submitted pull requests and issue reports!

Knockout v2.3.0 released; v3.0.0 beta available

Lots of new goodies for Knockout.js developers today! You can download version 2.3.0 final or 3.0.0 beta from the downloads page right now.

Why two new versions on the same day? The KO team and community has been hard at work preparing for the next phase of the library’s evolution, and we decided to roll it out in the form of a choice:

  • Get v2.3.0 if you just want all the bugfixes, perf improvements, and small features we’ve added in recent months. This is ideal for existing production apps.
  • Get v3.0.0-beta if you want the kickass new binding system (plus all the v2.3.0 enhancements) which has tonnes of powerful new extensibility potential. It’s almost-but-not-100% back-compatible with the older system – details below.

What’s new in 2.3.0?

Although KO 2.3.0 is primarily a package of compatibility/performance improvements and bugfixes, we sneaked in a few new features too:

  • ko.observable, ko.computed and other non-DOM-related parts of KO can now run on the server in Node.js. NPM package available.
  • The options binding has been rebuilt and made much smarter:
    • It now inserts and removes <options> nodes incrementally (rather than recreating them all) as your underlying array changes, leading to much better performance on older browsers.
    • A new optionsAfterRender callback lets you add custom code to modify <options> nodes as they are added to the document, in case you need this to interoperate with third-party DOM libraries
  • The template binding now allows use of an observable template name. This gives you an extra way of switching templates dynamically at runtime.
  • The css binding now allows use of any legal CSS class name. Previously it didn’t allow dots or slashes in class names.
  • ko.unwrap is the new, shorter name for ko.utils.unwrapObservable. Don’t worry – the old name still works too!

Bugfixes and compatibilty/perf improvements include:

  • KO will now warn you with an error if you try to apply bindings to the same element more than once (this was always an error; we just didn’t report it before)
  • The hasfocus and options bindings now work around additional edge-case browser quirks, e.g., during page load when running in an iframe in IE9 in compatibility mode (seriously!). This gives you a more consistent developer experience.
  • Observable arrays have better handling of unexpected parameters (for example, when initialized with non-array values).
  • The template binding now has more predictable handling of malformed HTML markup
  • ko.toJS now usefully handles String/Number/Boolean object-type properties (as opposed to string/number/bool primitives, which have always worked)
  • ko.computed now recovers from exceptions thrown during your evaluation callback. Previously, an evaluator exception would stop all future updates for that ko.computed.
  • Better whitespace handling in string-based templates (e.g., you can now put spaces around the equals character in data-bind = '…', if you care about that kind of thing).
  • General perf improvements (e.g., we avoid evaluating scrollTop when not absolutely necessary, as it triggers a rendering reflow).
  • More than 20 bugfixes for minor issues

Along the way, we improved some of the KO engineering processes to help us keep the momentum and stability high:

  • Ported all specs to the excellent Jasmine testing framework (thanks Chris!), so they can run in a wider range of environments
  • Simplified build system by removing all batch files
  • Automatic test runners everywhere!
    • The build script now runs all specs in PhantomJS if installed
    • The build script also runs specs for the Node-compatible subset of KO directly in Node (not via PhantomJS).
    • All specs run in all supported browsers (even IE6 and Safari for iOS 5 :) ) on every push to GitHub thanks to the excellent Testling service.

For a more detailed list, see the 2.3.0 release notes page.

What’s coming in 3.0.0?

v3 is a forward-looking release that focuses on how KO binds your models to your DOM:

  • It aims to open up powerful new extensibility possibilities at the heart of KO’s binding system. This means, for example, you can define your own new custom markup syntaxes.
  • Along the way, it fixes some longstanding complications in the binding mechanism. For example, it guarantees that bindings update independently and in the desired relative order, so there’s much less scope for surprising edge-case behaviors.

So, how does that actually look?

Node and binding rewriting

With v3, advanced KO developers can define custom syntaxes by providing callbacks that rewrite DOM nodes and binding strings during the binding process.

As an example of a node rewriter, here’s a custom binding syntax I just put together. It lets you write {{ expr }} to insert the value of arbitrary viewmodel expressions in your DOM, so that you rarely need to bother using the text binding. Because of how node rewriting works, it automatically has full support for observable values, being embedded in foreach loops etc.:

Such syntaxes can be distributed as KO plugins. Popular ones might end up in KO core.

Many more advanced scenarios are possible. For example, developers could create a “widgets” plugin that rewrites custom elements such as <orderLine> with the output from some other markup defined in <template id='orderLine'>…</template>.

Independent and ordered bindings

Until v3, if you had a binding like this:

<input type="checkbox" data-bind="visible: needsTerms, checked: acceptsTerms" />

… then any changes to needsTerms would cause both visible and checked to be re-evaluated, so the “checked” state of the checkbox would be reapplied. That’s rarely a problem, but it’s not great for performance and there were some rare edge cases where custom and even built-in bindings could interact in unexpected ways.

Also, the order in which bindings were applied was given by their left-to-right order in your binding attribute (so in this case, visible was applied before checked). Again, that’s almost always OK, and usually what you actually want. But again there could be edge-case scenarios with custom or even built-in bindings interacted in unexpected ways depending on order.

v3 fixes this by ensuring that bindings only refresh when their bound model state has changed at a more fine-grained level. So in the above example, changing needsTerms would not cause checked to refresh. Also v3 lets bindings declare order dependencies, so if for some reason it was important for checked to run before visible, we could ensure that it does, regardless of the order you write them in your binding attributes.

Backward compatibility

As always, we’ve taken back-compat very seriously. Knockout v3.0.0 is intended to be fully backward-compatibile with applications built for KO 2.x with one necessary exception:

  • v2.x’s behavior about dependencies between bindings (described in the section about “Independent and ordered bindings” above), is an undocumented internal implementation detail so hopefully you aren’t relying on it. But if you are relying on that then obviously you will see a change of behavior because bindings are independent in v3 by design. You’ll need to stop relying on cross-binding dependencies, which by the way will make your code much cleaner and easier to understand.

Please try v3.0.0 beta with your apps, and let us know if you discover any unexpected changes of behavior. That’s what the beta is for :) .

Credits

As always, big thanks to the KO community and core team for the work involved in producing this. In particular, most of the v3 binding enhancements were designed and implemented by Michael Best in Hawaii. The migration of specs to Jasmine was led by Chris Price in Newcastle. You can thank Ryan Niemeyer who wrote a large portion of the new docs in Wisconsin. Numerous fixes came from Matteo Pagliazzi in Italy. Lots of other people helped too – see GitHub for the full log!