Memoizing Angular filters to stop digest errors

tl;dr Use underscore's memoize method with your Angular filters to make them run faster and avoid nasty digest loops.

Here's the problem. You have an Angular filter that you're planning to use in an ng-repeat directive.

angular.module("TestApp.filters", []).filter("numbersToObjects", function(){
    //Takes an array of integers and returns an array of objects with numbers published under the 'val' property
    return function(numbers) {
       var numberObjects = [];
       for(var x = 0; x < numbers.length; x++){
          numberObjects.push({val: numbers[x] });
      }
      return numberObjects;
    }
})

Why do this? In my case, the reason was that the items in a collection you're planning to use ng-repeat on must all be unique. My array of integers had duplicate numbers in it. Turning the integers into objects would ensure uniqueness. Even if object[0].val and object[3].val were the same, the objects those properties attached to would still be unique.

<ul>
    <li ng-repeat="number in numbers | numbersToObjects">  </li>
</ul>

I thought I was being pretty clever, but unfortunately my solution caused another error. The thing is, the filter worked, but the error polluted the console log. It was the dreaded 10 $digest() iterations reached. Aborting! error. Why was it happening?

Here's what was going on. Each digest cycle, filters get executed, taking in some input and sending out some output. Angular knows it doesn't need to run another digest cycle if and only if the output of the filter on this digest cycle is identical to the output of the filter from the previous one (assuming the input is the same.) In other words, the digests repeat until things settle down.

Take this for example:

angular.module("TestApp", [])
  //GOOD FILTER: ELEPHANT === ELEPHANT
  .filter("capitalizeString", function(){
    return function(aString){
      return aString.toUpperCase();
    }
  })
  //BAD FILTER -- DO NOT DO THIS: 5.02 !== 5.05 (or whatever)
  .filter("addUniqueIdToNumber", function(){
    return function(numbers){
      console.log("Running filter");
      number += Math.random();
    }
  });

Send the string "elephant" into the first filter and it sends out "ELEPHANT" every time. Send the number 4 into the second filter and you get a digest loop because the output will always be a little different even if the input is the same.

It's worth thinking about what identity means in JavaScript when it comes to objects. Consider:

var object1 = {val: 5};
var object2 = {val: 5};
console.log(object1 === object2); //false.

Two objects can have identical properties, with identical values, and still not be identical. Now check this:

var object1 = {val: 5};
var object2 = object1;
object2.val = 10;
console.log(object1 === object2); //true
console.log(object1.val);          // 10

Although JavaScript doesn't have pointers, I think I understand pointers better after seeing examples like this one!

So the problem was that my filter would output different objects each time it was run, even if the input array never changed. [1, 2, 3] would yield [{val: 1}, {val: 2}, {val: 3}] every time the filter was run, but the output objects would be different every time (despite the properties being exactly the same.) Angular would detect this difference and repeat the digest, same as in my example with the unique ID filter.

The solution: Memoize the filter function

For identical input, a good function returns identical output. Unfortunately, if your function creates objects from its inputs, it will never return identical output for identical input.

Solution: underscore's memoize method. We basically want the filter function to keep a table of its outputs and inputs. If it gets an input it has seen before, we want it to return exactly the same output. Identical output for identical input will break the digest cycle. That's what _.memoize() does. Feed it a function, and it returns a version of that function that keeps track of inputs and outputs in the way I just described.

That way, the filter returns exactly the same objects it did the first time it was run, as long as the input to the filter was the same.

// The memoized filter function
angular.module("TestApp.filters", [])
  .filter("numbersToObjects", function(){
      //takes an array of integers and returns an array of objects with the numbers published under the specified property
      return _.memoize(function(numbers, propertyToPublishNumberUnder){
        propertyToPublishNumberUnder = propertyToPublishNumberUnder || "val";
        var objects = [], object;
        angular.forEach(numbers, function(number, index){
          object = {};
          object[propertyToPublishNumberUnder] = number
          objects.push(object);
        });
        return objects;  
      });
    //}
  }) 

Notes:

[1] If I remember right, filters run several times per digest cycle. That means they need to be light and not do any resource-intensive computations. However, if that is unavoidable, memoizing the computation-heavy function would be a good idea.

[2] I'm pretty sure JavaScript's notion of identity flagrantly flouts Leibniz's principle of the identity of indiscernables.

JavaScript and clean coding

\[ \dfrac{TeamDiscipline}{LanguageExpressiveness * NumberOfCollaborators} = Productivity\]

tl;dr An expressive, loosely-typed language, like JavaScript, can only be used in a large scale project if standards, however arbitrary, are rigorously and ruthlessly enforced.

Lately, I've been reading Robert C. Martin's book Clean Code. I've also been engaged in a long-running discussion with a very knowledgeable friend about the merits of Node.js compared to Java for large scale applications.

He's not a fan of Node. My guess is that Martin wouldn't like it much either. But I am unsure.

The problem seems to be this: JavaScript is a very expressive language. A lot can be done with just a few lines of code. It is also a loosely-typed language. It is, overall, not a very rigid language. To be maximally precise, if there is a problem to be solved, there will be dozens of ways to solve it in JavaScript, with some being very clearly worse than others but not one being unequivocally the best. The choice of using a functional style, a protypal object-oriented style, or aping a classical object-oriented style (using constructor functions to simulate classes, etc.) is left up to the developer.

Which means, in a collaborative environment, that it is left up to each and every developer.

Not only that, but even if a choice is made, nothing stops a developer from being inconsistent, from being functional in this part of the code, more classical over here, or even avoiding the OO features of JavaScript entirely to be boring and procedural. JavaScript allows you to do it.

This is a problem. Obviously, you'd have to be a really bad developer, or maybe just a wickedly devious one, to flip flop from style to style like this. No one would do that, right?

Sure. But a single devious or supremely incompetent developer is not the problem. The problem is a team of developers, none of whom is totally incompetent or devious, but each of which is sometimes less competent -- or, more precisely, less disciplined -- than he or she could be.

Spontaneous Order & A Million Acts of Care

Bringing me to Martin's book. I haven't finished it yet. I loved the forward; especially this quotation:

Quality is the result of a million selfless acts of care -- not just any great method that descends from the heavens.

The reference need not to a million acts of a single individual, but to many different acts by many different people. If care is lacking from one or two of those acts, the difference in quality may be negligible. If there is a pervasive lack of care, disaster will result.

How to explain this? F.A. Hayek described spontaneous order, the idea of an overall order emerging from the acts of individuals, most or all of whom did not seek to create that order. A million selfless acts of care add up to something that is more than the sum of all those choices.

Conversely, too many "isolated" uncaring acts can corrupt a process and undermine the final product, without any individual actor being grossly incompetent or malicious. All that is required is a sufficient number of uncaring acts, each isolated from all the others. The more collaborators there are, the more this can occur -- that is, unless team discipline prevents uncaring acts, or provides a fix for them before their effects multiply.

Let's just call this effect -- lots of individual acts making a product worse than any individual actor intended, and even making the product worse than the competence or incompetence of any individual actor can explain -- spontaneous disorder. For proof that spontaneous disorder exists, I invite the reader to witness a meeting of any academic committee.

Collaborators vs. team (or average) discipline -- that explains part of my equation. There comes a point where average discipline is insufficient to overcome the sheer entropic impact of the combined interactions of the actors. Nothing can be done about that except to maintain discipline, and for that, Martin's book is an excellent place to start.

Spontaneous Disorder In Spite of Discipline

But what about the other variable, language expressiveness?

It seems to me that language expressiveness, in the sense I earlier described it, can cause spontaneous disorder despite discipline. If there is a single way to solve a problem, then by definition that is the most efficient way to solve it. If there are multiple ways to solve a problem, but one is theoretically more efficient, then a sufficiently competent team will eventually arrive at that solution.

But if there are multiple ways to solve a problem, and none is clearly more efficient than the other, then the choice between the solutions will be somewhat arbitrary. The arbitrariness will lead to inconsistency. Why solve all problems of type X in the same way? Why not solve some problems of type X in one way, some in another? Will this inconsistently inevitably emerge, given a sufficiently expressive language? I am not sure. If so, that is a problem for JavaScript, because JavaScript is very expressive.

If I am right, large JavaScript projects will have a tendency toward spontaneous disorder, even with a disciplined team. This is because discipline is itself insufficient to provide answers to the questions an expressive, loosely-typed language raises. No amount of discipline -- none of the principles Martin articulates -- can provide a definitive answer to how a complex problem should be solved in JavaScript.

It is also clear that the severity of this problem scales up with the number of developers. More developers equals more arbitrariness, more decisions made with insufficient reason. One developer can be satisfied doing things one way, just because he has decided that is how things shall be done. Two developers may come to an agreement between them. Three or more? Must management ultimately decide?

That may, in fact, be the answer. When market forces are insufficient to provide an important good, there are those who believe the government ought to provide it. Similarly, when a million selfless acts of care cannot yield quality, then maybe it is time for a higher authority to set standards, no matter how arbitrary those standards are.