Kit Cambridge

Programming and other observations.

Say "Hello" to Lo-Dash

Lo-Dash is a low-level utility library that offers consistency, customization, and performance. Created as a fork of the Underscore project, Lo-Dash has grown in features and popularity, while remaining faithful to its original tenets. Unlike most libraries, Lo-Dash eschews almost all native iteration methods in favor of simplified loops, resulting in tight, lean code.

But Lo-Dash isn’t just a consistent, fast utility belt. It offers a fully configurable build process, with a plethora of options and goodies. You can target legacy or modern browsers, or mix and match individual methods to taste. We’ve also introduced new methods for deep cloning, deep merging, and object iteration. And now we’ve added source maps, intuitive chaining, right-associative partial application, and a shorter iterator syntax.

Lo-Dash is your utility belt.

Background

Most JavaScript utility libraries, such as Underscore, Valentine, and wu, rely on the “native-first dual approach.” This approach prefers native implementations, falling back to vanilla JavaScript only if the native equivalent is not supported. But jsPerf revealed an interesting trend: the most efficient way to iterate over an array or array-like collection is to avoid the native implementations entirely, opting for simple loops instead.

Library Race

Unfortunately, legacy engines — and even some modern ones — are plagued by a profusion of iteration bugs and inconsistencies. Our solution is to use function compilation to construct low-level methods that resolve these bugs. For the common case of array iteration, we use plain while loops. For all other collections and objects, including strings, arguments objects, and object instances, we delegate to the compiled methods.

Compilation allows us to work around implementation differences, while reducing the performance overhead caused by code forking. By starting with a solid foundation, we’re able to maintain our performance lead and add utility.

Consistency

By avoiding slower native methods, we’re able to add functionality, and sidestep common performance and consistency traps. Libraries that depend on native methods, however, cannot break this parity without introducing inconsistencies. These are particularly difficult to track down in older browsers, which have limited debugging tools.

Lo-Dash also detects and avoids shimmed methods. A low-level utility library, built for general-purpose use, can’t make erroneous assumptions about its environment. This is particularly important for widgets, which must be embeddable in environments over which their developers have no control.

First Two Hundred Lines of Lo-Dash

More importantly, Lo-Dash emphasizes consistency because it’s important for your code to “just work,” in newer and older environments. If you’re like most devs, you’re developing in a modern browser with modern dev tools, testing your code in legacy browsers as time permits.

But even modern engines have quirks, and can throw a wrench into low-level libraries that assume the contrary. And, because inconsistencies in low-level abstractions propagate to their dependencies, you’ll subsequently need to delve into your library’s implementation details. Shouldn’t your utility library free you from this tedium?

Dr. Jekyll and Mr. Hyde

By smartly opting into native methods — only using a native implementation if it’s known to be fast in a given environment — Lo-Dash avoids the performance cost and consistency issues associated with natives. For example, the native Object.keys and Function#bind methods are optimized in different engines. We automatically select the best option during initialization, and implement a fallback to preserve consistent behavior.

For _.bind, this means support for calling bound functions as constructors — environments as recent as PhantomJS 1.8.1 (the latest release, at the time of writing) and Mobile Safari on iOS 5 (mirrored by Safari 5.0 on the desktop) lack Function#bind. Another example is iterating over arrays with non-contiguous indices. Because engines disagree on how these indices should be treated — JScript, for instance, expands this definition to undefined array elements — even fully compliant shims will produce different results across engines. Finally, modern and legacy engines are inconsistent in iterating over arguments objects and the prototype property of constructors. This can cause issues, especially if your code isn’t anticipating these bugs.

But how significant are the differences? Let’s compare Lo-Dash to its progenitor, Underscore. We’ll test in Firefox 18 (a modern browser), Safari 5.0 (an older browser, but consistent with Mobile Safari on iOS 5), Internet Explorer 7 (a legacy browser, with Firebug Lite), and a headless JavaScript engine (PhantomJS 1.8.1).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
function Animal(name) {
  this.name = name;
}

_.extend(Animal.prototype, {
  'speak': function(message) {
    return this.name + ': ' + message;
  },

  'toString': function() {
    return this.name;
  }
});

// Should be `true`, even in IE <= 8 (which doesn't support `Object.keys`
// and fails to enumerate certain shadowed properties).
_.contains(_.keys(Animal.prototype), 'toString');

var animal = new Animal('Orion');
// Should log `"Animal: Orion"`. If the `toString` property wasn't copied
// over in IE <= 8, this will log `"Animal: [object Object]"` instead.
console.log('Animal: ' + animal);

// Create a bound constructor. When instantiated, the bound constructor
// should return a new instance and ignore the context. In Underscore 1.4.4,
// a new instance is not created, and a global `name` property is set
// instead. `_.partial` can be used instead, though it suffers from the
// same issue.
var Cat = _.bind(Animal, null, 'Cat');
console.log('Cat: ' + new Cat());

// Reduce an array with three non-contiguous indices. Because Underscore
// delegates to the native `Array#reduce` implementation, this will log
// `0` in environments with native support, and `3` without. Lo-Dash
// treats all non-contiguous indices as `undefined`, so the result will
// always be `3`.
_.reduce(Array(3), function(sum) { return ++sum; }, 0);

// Properties of the `arguments` object are not enumerable in IE <= 8,
// Safari < 5.1 (Mobile Safari 5), and PhantomJS. In these browsers,
// Underscore will report that the `arguments` object is empty.
console.log(function() { return _.isEmpty(arguments); }(1, 2, 3));

// This should print three lines, each containing the character, index, and
// original string. IE <= 7 does not support accessing string characters
// using square bracket notation, causing Underscore to print `undefined`
// instead of the character. Lo-Dash's `each` implementation also returns
// the string for chaining.
_.each('ABC', console.log, console);

// Safari < 5.1 and Mobile Safari will iterate over the `prototype`
// property of constructors. This can cause issues if you're copying
// static methods from one constructor to another, and aren't expecting
// this property to be enumerated.
_.contains(_.keys(Animal), 'prototype');

Here are the results. Underscore 1.4.4 is on the left; Lo-Dash 1.0.0 is on the right.

Firefox 18

Underscore delegates to the native Array#reduce implementation, which ignores non-contiguous indices.

Lo-Dash and Underscore in Firefox 18

Safari 5.0.5 and Mobile Safari on iOS 5

These versions of Safari don’t implement Function#bind, and suffer from the arguments and prototype iteration bugs discussed above. As in Firefox 18, Underscore uses Array#reduce.

Lo-Dash and Underscore in Safari 5.0.5

Internet Explorer 7

Internet Explorer 8 and older are susceptible to object and string iteration bugs, and don’t implement Function#bind or Array#reduce.

Lo-Dash and Underscore in IE 7

PhantomJS 1.8.1

PhantomJS doesn’t support Function#bind, and suffers from the same arguments iteration bug as Safari 5.

Lo-Dash and Underscore in PhantomJS 1.8.1

Readability

Lo-Dash Audio Book

When we first started the project, we included over 30 compiled methods, peaking at 32 in v0.8.2. Since then, in an effort to increase readability and comprehension of the source, we’ve reduced our compiled methods to five: _.each, _.forIn, _.forOwn, _.assign, and _.defaults. All other iterator functions use uncompiled, simple loops for the common array case, and fall back to one of the five compiled core methods for collections and objects.

Custom Builds

Lo-Dash includes a build tool that makes it easy to create custom, integrable distributions. You can mix and match individual functions to build a utility library that’s just right for you. Every aspect of the build process is configurable, from the module format and browser support, to the immediately-invoked function expression wrapper. There’s even an option for pre-compiling Lo-Dash templates.

With custom builds, there’s no need to include legacy engine support if you’re targeting mobile browsers, or deep merging and function composition if you’re only interested in Backbone compatibility. We’ve tested the build system with an extensive battery of over 9,000 unit tests, and regularly test the Underscore and Backbone builds against the official test suites of both projects.

Need AMD support, collection methods, and strict mode? lodash strict category=collections exports=amd. What about Browserify, with source maps for easier debugging? lodash -p exports=node iife="%output%". Once your build is ready, we’ll use a hybrid compression strategy (UglifyJS and Closure Compiler) to determine the optimal file size. No other project provides this extensive level of customization.

It's Over 9,000!

You can run the builder directly from the command line, or access it programmatically through Node. If you use Grunt, you can specify build options directly in your Gruntfile with the excellent grunt-lodashbuilder plug-in.

Lo-Dash isn’t prescriptive about how you build and consume it. Choose your functions, exports, and output type. You don’t need to saddle your project with unnecessary cruft, and you don’t need to maintain your own fork. Lo-Dash is your utility belt.

What’s New in 1.0?

In past releases, we added _.bindKey for “lazy” function definition, _.isPlainObject for determining if an object was created by the Object constructor, _.forIn for enumerating all object properties, and _.forOwn for own properties only — with all the requisite consistency fixes baked in. Now, we’ve added deep comparison support to _.where, _.at for extracting collection values, and _.partialRight for right-associative partial application.

In addition, we’d like to showcase three new features in this release: customization callbacks, shorthand syntax, and intuitive chaining.

Customization Callbacks

Deep merging (_.merge) and cloning (_.clone(..., true); aliased as _.cloneDeep) were introduced in v0.5.0, and optimized in v0.9.0. Along with _.isEqual, _.merge and _.cloneDeep support cyclic structures, and define predictable behavior for objects, collections, booleans, numbers, regular expressions, and dates. In 1.0, we’ve added the ability to specify customization callbacks and contexts to all three recursive methods, and the non-recursive _.assign and _.clone.

Prior to v0.8.0, if you wanted to extend the semantics of _.isEqual or _.clone, you had to define custom isEqual and clone methods on your objects. To use the default in-method behavior, you’d then need to invoke _.isEqual or _.clone yourself, keeping track of cyclic structures. This was incredibly cumbersome.

Customization callbacks allow you to maintain granular control over cloning, merging, and performing deep comparisons, without requiring special methods. You can specify the callback and context just as you would for an iterator function. If the callback returns undefined, Lo-Dash falls back to the in-method behavior. This avoids cluttering the callback with recursive invocations, and takes the burden of tracking circular references away from you.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
var element = document.createElement('span');
element.appendChild(document.createTextNode('Hiya!'));

// Clone DOM nodes, with deep cloning enabled. `_.clone(..., true)` is
// equivalent to `_.cloneDeep`.
_.clone(element, true, function (value) {
  if (_.isElement(value)) {
    return value.cloneNode(true);
  }
});
// => <span>Hiya!</span>

var food = {
  'fruits': ['apple'],
  'vegetables': ['beet']
};
var otherFood = {
  'fruits': ['banana'],
  'vegetables': ['carrot']
};

// Concatenate arrays, instead of overwriting the indices.
_.merge(food, otherFood, function (left, right) {
  return _.isArray(left) ? left.concat(right) : undefined;
});
// => { 'fruits': ['apple', 'banana'], 'vegetables': ['beet', 'carrot' ]}

And, with the new _.partialRight method, you can compose callbacks to create higher-order methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// Implement a deep version of `_.defaults`.
var defaultsDeep = _.partialRight(_.merge, _.defaults);

// Default options.
var defaults = {
  'method': 'GET',
  'headers': {
    'X-Requested-With': 'XMLHttpRequest'
  }
};

// Custom options.
var options = {
  'method': 'POST',
  'headers': {
    'Content-Type': 'application/json'
  },
  'data': [1, 2, 3]
};

defaultsDeep(options, defaults);
// => { 'method': 'POST', 'headers': { 'X-Requested-With': 'XMLHttpRequest',
//      'Content-Type': 'application/json' }, 'data': [1, 2, 3]}

// Implement custom semantics for comparing DOM nodes.
var isEqualDOM = _.partialRight(_.isEqual, function (left, right) {
  if (_.isElement(left) && _.isElement(right)) {
    return left.nodeName == right.nodeName;
  }
});

_.isEqual(document.createElement('div'), document.createElement('div'));
// => `false`; default semantics.

isEqualDOM(document.createElement('div'), document.createElement('div'));
// => `true`; extended semantics.

Shorthand Syntax

Lo-Dash provides a new shorthand syntax that allows property names and objects to be used in place of callback functions. If you specify a property name, Lo-Dash will create a callback that retrieves the property value for each element in the collection, similar to _.pluck. Alternatively, if you provide an object, the created callback will return true for items that match the properties of the given object, and false otherwise. The object shorthand syntax performs deep comparisons, similar to _.where.

You can take advantage of this feature to write elegantly tight code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// Retrieve all disabled form elements.
_.filter(document.querySelectorAll('input'), 'disabled');

// Sort by string length.
_.sortBy(['hello', 'world', 'this', 'is', 'nice'], 'length');

var projects = [{
  'name': 'John-David',
  'projects': ['Benchmark', 'Lo-Dash', 'FuseJS']
}, {
  'name': 'Mathias',
  'projects': ['Lo-Dash', 'Benchmark']
}, {
  'name': 'Kit',
  'projects': ['Lo-Dash', 'JSON 3']
}];

// A more complex example with deep equality.
_.first(projects, {
  'projects': ['Benchmark', 'Lo-Dash']
});
// => [{ 'name': 'John-David', ...}]

Intuitive Chaining

Lo-Dash replaces Underscore’s .chain() syntax with jQuery-style intuitive chaining. Methods that operate on and return new arrays, functions, and collections can be chained together, just as you’d expect. You can break the chain and access the wrapped value by explicitly calling .value() on the wrapper. Alternatively, if you call a method that returns a Boolean (isArray, isString, isEqual) or single value (reduce, reduceRight, clone), Lo-Dash will automatically end the chain for you. Like jQuery, intuitive chaining allows you to “write less, do more.”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
_([1, 2, 3, 4, 5]).reject(function (value) {
  return !(value % 2);
}).invoke('toString', 2).sortBy('length').join('');
// => Returns "111101".

var users = [{
  'name': 'John-David',
  'company': 'Microsoft',
  'teams': ['BestieJS', 'Lo-Dash', 'Benchmark']
}, {
  'name': 'Mathias',
  'company': 'Qiwi',
  'teams': ['BestieJS', 'Lo-Dash']
}, {
  'name': 'Blaine',
  'company': 'IcedDev',
  'teams': ['IcedDev', 'hackPHX', 'EnyoJS']
}, {
  'name': 'Sindre',
  'teams': ['TodoMVC', 'TasteJS', 'Yeoman', 'Grunt', 'Bower', 'Basket']
}];

var collection =  _(users);

// Clones the collection of users.
var clone = collection.cloneDeep();
collection.isEqual(clone); // => true

// Uses property and object shorthand to filter the list of users and
// group by company name.
var companies = collection.where({
  'teams': ['BestieJS', 'Lo-Dash']
}).groupBy('company');

companies.isPlainObject(); // => true
companies.keys().contains('Qiwi'); // => true
companies.value(); // => { 'Microsoft': [...], 'Qiwi': [...] }

collection.find({
  'name': 'Mathias',
  'teams': ['BestieJS']
});
// => Returns Mathias' user info.

// Bind a function, and delay its invocation for three seconds.
_(function (message) {
  console.log(this.name + ': Greetings! ' + message);
}).bind({
  'name': 'Kit'
}).delay(3000, 'I bring fresh fruit.');
// => Prints "Kit: Greetings! I bring fresh fruit." after three seconds.

What’s Next?

In less than a year, Lo-Dash has been adopted by several major projects. By emphasizing performance, we’ve cleared the way for features, consistency, and customization. We’re now looking to the future, with modern environment support, ES 6 compatibility, and modular builds.

Following jQuery’s lead, we’ve created a modern build for devs who want performance gains and extra features, but without the overhead of legacy support. This build excludes object iteration fixes and other workarounds for older environments. We are not dropping our consistent support for these environments; rather, we’re making it easier for devs to target the browsers and engines that they work with the most. Although we’ll continue to provide consistent support for legacy environments well into the future, we’re looking forward to even more efficient, slimmed-down methods.

We’ve modified existing methods for future compatibility with ES 6 as well — _.template supports the new template delimiter syntax, and _.extend only iterates over own properties for parity with the Object.assign proposal. We’ll also continue refining source map support as UglifyJS and Closure Compiler expose additional customizations.

Finally, to make it easier for other libraries to adopt Lo-Dash, we’re working on a modularize build option. This option will separate Lo-Dash into self-contained modules — by method or category — for more granular consumption. To that end, we recently joined the Dojo Foundation, and can’t wait to see where this takes us.

Make Lo-Dash your own this Valentine’s Day!

— The Lo-Dash Core Team