Ibis and Baboon

Chomsky, Norvig and Practical Machine Learning

2011-06-03T00:00:00+02:00

Is it more important for science to describe how things behave or to explain why things behave the way they do? This seems to be the question behind statements Noam Chomsky made at a symposium regarding machine learning and an article by Peter Norvig discussing Chomsky’s opinions.

Chomsky is highly critical of the commonly used statistical models for machine learning that focus on the “how”-part. He discounts the practical success of these models as unimportant for the advancement of science. His main goal – as far as I understand it – is to find the principles on which language is based.

Norvig argues in favour of statistical models and purely descriptive research as something well worth pursuing and not as rare in the history of science as Chomsky claims it is. He cites several examples from Chomsky’s publications that show a lack of knowledge about the capabilities of machine learning algorithms. Norvig’s most important argument – to my understanding – is that our current statistical models perform better in describing the reality of language than our current explanatory models.

Regardless of the advantages of statistical models, explanatory models definitely are easier for humans to talk and think about. And however our brains really learn languages, rules are how we teach them. Beyond the scientific considerations Norvig’s article and Chomsky’s statements focus on, I find this aspect relevant for the practical use of machine learning as well. Many applications or services that rely on machine learning suffer in usability, because you cannot really understand why they do what they do.

Spam filters are a good example for this problem. Originally, programs like Spamassasin only used explicit rules (e.g. does the text contain the word “viagra”) to determine whether a mail was spam. For each matching rule a certain number of points is added to the mail’s spam score. And usually the program adds a header where all the matching rules and the number of points resulting from it are listed (e.g. DRUGS_ERECTILE=0.282). These scores are helpful when looking why some mails where not classified correctly.

Spam detection was, however, greatly improved by the introduction of Bayesian filters. These probabilistic filters are trained with corpora of mails marked as spam or not-spam and calculate the probability of new mails being spam. In Spamassassin, this results in a single rather opaque score BAYES_99=3.5. The other descriptive scores are still there, but from a brief look through my recent spam, the Bayesian classifier contributes the most significant numbers for the mails that were correctly filtered.

The good news is that statistical models don’t always mean that you won’t get a good explanation. Amazon, for example, shows you why you get a certain recommendation. Below each recommended item, there is a link to “fix this recommendation”.

The page you get, shows items you bought or looked at that the recommender thought were similar to the one you now get as recommendation. The page also gives you ways to tell Amazon not to use these items for you in the future.

In applications with a machine learning component, giving some explanation or reasoning goes a long way to improve the usability.

On the News

2011-04-13T00:00:00+02:00

Mandy Brown wrote a really good post about the evolution her news-reading behaviour. She started out reading one newspaper and than transitioned to reading many different online news sources. I can absolutely relate to that. I also share the transition in the behavioural pattern of reading news: with the newspaper I read it once in the morning during breakfast; now I’m checking the news a day.

An than Mandy talks about expectations from the news she reads:

I want a reading experience that defends the news from the circus that online advertising creates. I want good storytelling and analysis, not naked facts. I want news that admits and defends its point of view (and acknowledges that there is a truth to be uncovered), not news that parrots the party line while making claims to objectivity. I want long essays on the events at Fukushima and the consequences for nuclear power going forward, not shrieking dispatches of each new fire or setback. I want a history of American engagement in Libya, putting the events of the past few weeks in context. I want twenty thousand words on the recession and its effects on the middle class, not another lone statistic about the unemployment rate. I want thoughtful, investigative journalism that exposes the ways in which our government is failing us, so that we can make it better.

I can say ‘yes’ to each of the points, except for the second item. I do want good storytelling and analysis, but I also want links to the naked facts. When I read about a planned increase in taxes for Diesel fuel where one quoted expert claims that preferring Diesel is bad from an environmental point of view, while car manufactures claim that Diesel is good for the environment, I want links to studies about the different properties of Diesel vs. gasoline with respect to environmental issues. When I read an op-ed about health care, where the author claims that the largest part of the health care costs for a person are incurred in the final year of their lives no matter how long the person lived, I want a link to a statistic supporting the claim. Bonus points for additional links to stats that do not agree and an explanation why they are less credible.

I have another addition to this wish list while we’re at it: I do want news that admits and defends its point of view, and I also want pointers to articles with different points of view.

Link: Punk Rock Languages

2011-03-07T00:00:00+01:00

Chris Adamson wrote a polemic about programming languages for the March 2011 issue of the PragPub Magazin that is both entertaining and thoughprovoking.

The natural appeal of the language is to write software with it, not to mess with the language itself—Solve your users’ problems rather than indulging your own programming fetishes.

Link: Presenting Like a Hacker

2011-03-03T00:00:00+01:00

After blogging like a hacker with Jekyll by Tom Preston-Werner, I recently came across a similar thing for presentations: Showoff by Scott Chacon. With it you can create presentation in Markdown and show them in a browser. This is particularly usefull when you have code in your presentations which is a real pain in other applications.

Link: Full Text RSS Feed

2011-03-02T00:00:00+01:00

Full Text RSS Feed is a cool simple web-service.

Love the ease of RSS, but hate when feeds don’t display the whole article, forcing you to click through just to read it?

I just hope it won’t get taken down by those who use the money from annoying ads to pay lawyers.

Literate Programming

2011-01-09T00:00:00+01:00

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

— Donald Knuth. “Literate Programming (1984)” in Literate Programming. CSLI, 1992, pg. 99.

Literate Programming is hardly a new idea and it was never popular enough that Wired or TechChrunch would have pronounced it dead at any given time. Recently however, a few tools appeared that implement some of the ideas from LP.

The first of these is Docco which generates HTML that shows the documentation alongside the code. The input is the commented source code, so that no tools are needed to convert it to be used as a program. With Knuth’s original LP, you wrote in a special kind of language – mixing your programming language with Latex – from which the source code could be extracted.

Inspired by Docco, Michael Fogus wrote Marginalia to work with Clojure code, extracting the text from both comments and function documentation.

These tools, however, do not have all the features to be considered literate programming in the original sense. Most notably missing is the ability to organize the source strictly by the flow of the text and not by the needs of the compiler to have stuff declared before they are used.

I tried out Marginalia and the literate-programming-lite-style for my clj-bookmarks library (it’s a client implementation for the Delicious and Pinboard APIs). The result is here and the code is on GitHub.

Although there are some minor bugs in Marginalia, I am mostly happy with the result. I might continue this experiment.

The Functional Elegance of Ring Middleware

2010-10-04T00:00:00+02:00

Higher-order functions are to me the most awe-inspiring feature of functional programming languages. However, like many incredibly elegant concepts, its greatness is not immediately obvious (recursion is another example that comes to mind). When I was exclusively working with imperative programming languages and came across the definition of higher-order functions, I thought the idea was in part trivial and in part irrelevant theoretical nonsense.

[…] Higher-order functions […] are functions which do at least one of the following:

take one or more functions as an input

output a function.

– Wikipedia

The first part, I could relate to. This is like passing a function pointer in C or an anonymous inner class in Java as e.g. event handlers. In both cases the syntax is so cumbersome that is feels like honest hard work, not like an awesome technique based on theory.

The second part – output a function – why would I want that? The typical examples are as helpful as fibonacci numbers are for explaining recursion to someone who is biased towards a narrow notion of practicality. They often go look something like this:

(defn plus-x [x]
  (fn [y] (+ x y)))

(def plus2 (plus-x 2))

(plus2 3)
=> 5

The function plus-x takes a parameter x and returns a function that takes a parameter y and returns x + y.

(If you didn’t believe me about the cumbersome syntax, here is the above example translated into Java. Related: Execution in the Kingdom of Nouns)

I’m telling you this, because I want to highlight a particularly elegant and practical example of higher-order functions: Ring middleware. Ring is a Clojure library for writing web apps. It gives you an abstraction on top of HTTP similar to Rack for Ruby or WSGI for Python. Higher-level frameworks/libraries such as Compojure, Moustache, or Sandbar are built on top of Ring.

A simple “Hello, World” with Ring looks like this (from the README):

(use 'ring.adapter.jetty)

(defn app [req]
  {:status  200
   :headers {"Content-Type" "text/html"}
   :body    "Hello World from Ring"})

(run-jetty app {:port 8080})

A ring handler (app in the example) is simply a function that takes a map representing the incoming request and returns a map representing the response. Such a handler can be given to an adapter (Jetty in this case) that deals with the actual HTTP connection and calls the handler function. Thus the adapter is a higher-order function of the kind I once thought to be trivial.

If we want to serve static files with this app, we can add middleware that wraps our app to look for requests to files in a given directory:

(use 'ring.middleware.file 'ring.adapter.jetty)

(defn hello [req]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "Hello World"})

(def app (wrap-file hello "public"))

(run-jetty app {:port 8080})

Here, we have a higher-order function of the second kind: wrap-file. hello is a handler function in its own right – it is equivalent to app in the previous example – but we do not pass it the the adapter directly. The result of wrap-file is a new handler function, it has to be, otherwise we couldn’t pass it to the adapter.

The implementation of the wrapper is simple and clean:

(defn wrap-file
  [app #^String root-path]
  (ensure-dir root-path)
  (fn [req]
    (if-not (= :get (:request-method req))
      (app req)
      (let [path (.substring (codec/url-decode (:uri req)) 1)]
        (or
          (response/file-response path
            {:root root-path :index-files? true :html-files? true})
          (app req))))))

First the function checks if the the directory is actually there by calling ensure-dir. The rest of the code in wrap-file is building the resulting handler function. If the request method is not GET, the inner handler is called. Otherwise, we extract the path and try to make a response out of that using file-response. If that returns nil, the inner handler is called.

Now, this is all nice and well, but the awesome elegance of this approach to middleware becomes apparent, when we realize that wrappers can be wrapped around other wrappers in arbitrary numbers:

(def app (wrap-params (wrap-file-info (wrap-file hello "public"))))

The resulting handler has a wrapper that makes the request parameters easier to process and one that adds content-type headers to the response.

As the prefix notation is not the most readable way to express such a chain of call, you would rather use the threading macro to write this:

(def app (-> hello
             (wrap-file "public")
             (wrap-file-info)
             (wrap-params)))

But that is just syntactic sugar, the beauty of this solution is possible, because it uses higher-order functions to great effect.

IDSC V: Accessing and Modifying Vectors

2010-07-03T00:00:00+02:00

In the previous post of the Immutable Data-Structure Canon we looked at vectors, their internal structure, how they are created, and how more elements are inserted. In this post we continue where we left off and examine the code used to access and remove values from a vector.

Accessing Elements

The elements of a vector are accessible by index. The way vectors are usually implemented, this is a constant time operation, as it only takes the calculation of the offset of a memory location. As we’ve seen, Clojure vectors are implemented as trees to allow for shared structures between persistent “modified” versions, so the access methods need to find their way around that structure.

Let’s say, for example, that we have a vector v with 1500 elements and we want to get the one at index 1101. There are three different ways to find an element in a vector by index: the nth function, the get function, and using the vector as a function. They differ in how they handle the vector being nil, the index being out of range, and whether they support a “not found” argument. For this discussion we’ll use nth; it returns nil if the vector is nil, throws an exception, if the index is out of range, but you can pass an optional argument that is returned, when the index is not found.

(nth v 1101)

Internally, this function is mapped to a call to the nth method in PersistentVector:

public Object nth(int i){
	Object[] node = arrayFor(i);
	return node[i & 0x01f];
}

The arrayFor method handles the lookup in the internal structure and returns on of the 32-element arrays where the elements are stored – either from one of the leaf nodes or from the tail (line 2). The index in that array is calculated by applying a bit-mask to the index passed into the method (line 3). For our example that local index is 13.

public Object[] arrayFor(int i){
	if(i >= 0 && i < cnt)
		{
		if(i >= tailoff())
			return tail;
		Node node = root;
		for(int level = shift; level > 0; level -= 5)
			node = (Node) node.array[(i >>> level) & 0x01f];
		return node.array;
		}
	throw new IndexOutOfBoundsException();
}

The arrayFor method checks that the index is valid and throws an exception if it is not (lines 2, 11). If the requested index is greater than the number of values in the tree, the tail array is returned (lines 4,5). In our example, this is not the case, we need to look into the tree.

As a tree with two layers (root and leaves) can hold 1024 elements, the tree in for a vector with 1500 elements needs three layers. The index we’re looking for is in the second subtree, as it’s greater than 1024. The field shift which holds a multiple of 5 proportional to the height of the tree is 10 in our case, so that we enter the loop in line 7 with 10 as level.

Inside the loop, we find the subtree holding our value by bit-shifting the index by the current level and applying a bit-mask to get it into the 32-element frame. The index for the subtree in our example is 1, as expected. In the second iteration of the loop, we look at the leaves, so that level is decremented to 5. This time, we find the node with index 2. The loop terminates here, as there is not level further down. The method returns the array attached to the node we found.

Summarizing all the index-offsets, we have:

the second subtree of the root (the first subtree contains 1024 elements),
the third leaf of that subtree (the leaves contain 32 elements each), and
the fourteenth element of the leaf (calculated in nth).

Thus we have the 1102nd element of the vector. Bingo!

Like the cons operation we saw last time, the number of steps necessary to find the nth element depends on the height of the tree, so the complexity here is once again O(log32 N).

Deleting Elements

To finish the discussion of vectors, let’s look at deleting elements. The only position where efficient deletions are possible is the end. This can be done using the pop function that is conveniently mapped to the pop method of PersistentVector:

public PersistentVector pop(){
	if(cnt == 0)
		throw new IllegalStateException("Can't pop empty vector");
	if(cnt == 1)
		return EMPTY.withMeta(meta());
	if(cnt-tailoff() > 1)
		{
		Object[] newTail = new Object[tail.length - 1];
		System.arraycopy(tail, 0, newTail, 0, newTail.length);
		return new PersistentVector(meta(), cnt - 1, shift, root, newTail);
		}
	Object[] newtail = arrayFor(cnt - 2);
	Node newroot = popTail(shift, root);
	int newshift = shift;
	if(newroot == null)
		{
		newroot = EMPTY_NODE;
		}
	if(shift > 5 && newroot.array[1] == null)
		{
		newroot = (Node) newroot.array[0];
		newshift -= 5;
		}
	return new PersistentVector(meta(), cnt - 1, newshift, newroot, newtail);
}

First, the edge-cases are handled: pop on an empty vector does not work (lines 2,3), and pop on a one-element vector returns an empty vector (lines 4,5). Then we look at the easy case that the tail holds more than one element (line 6); we just copy all tail elements except for the last one into a new array and use it to create a new vector that shares the tree with the original (lines 8-10).

The condition for the simple case is phrased so that we don’t use it on a tail with only one element. This is because we need to change the tree when the tail run empty; the last node becomes the new tail in that case.

Shrinking the tree starts with getting the array from the last node by calling the arrayFor method (line 12). Next, we call popTail to get a new root node for a tree without the last node (line 13). This leaves us with three cases:

The node we removed was the only node, so that our return vector gets an empty node as root (lines 15-18).
The tree has intermediate levels between the root and the leaves and the removed node was the only leaf in the second subtree, so that we can remove one layer (lines 19-23).
Otherwise the new tree still has the same height as the old tree.

Like the other vector operations we looked at, pop also has a complexity of O(log32 N), because the number of steps necessary is dependent on the height of the tree.

Summary

Vectors are implemented based on a tree and a tail array.
All arrays (both in the nodes of the tree and in the tail) have up to 32 elements.
The leaves always have exactly 32 elements in them.
Elements can added and removed efficiently at the end of the vector.
Addition, removal, and index lookup are O(log32 N), which is essentially constant time in practice.

IDSC IV: Creating and Growing Vectors

2010-06-28T00:00:00+02:00

Vectors provide constant time random access to any element referenced by an index. Like their fixed length cousins, arrays, they are usually implemented by storing their elements in consecutive memory locations. Such a strait forward implementation, however, doesn’t allow for immutability – at least not when we are interested in the performance characteristics, but in that case we could go with lists anyway.

This post is part of the immutable data-structure canon. While the implementation of immutable lists is relatively simple, immutable persistent vectors require quite a bit of work. I decided to split this topic into two parts: this post covers how vectors are created and how elements; we’ll also learn about the structure used to store the values. The second part to be posted next week, covers how vectors are accessed and how elements are modified and deleted.

Before we dive in, I need to correct an error in the previous parts of the series. Until now, I used “sequence” and “seq” synonymously as a generic term for data-structures that can hold multiple values. I realized now that the correct generic term is collection. Sequence in the Clojure context is a collection that a series of values without reordering them where values may or may not exist yet. seq specifically refers to an API for using collections with the funtions first and rest. Henceforth I shall use the right terminology. And now: vectors.

To achieve both immutability and performance, Clojure only stores limited segments of a vector in a row in memory. The overall structure used to represent the vector behind the scenes is a tree. Each of the tree’s nodes holds an array that contains a segment of the vector. Functions that “modify” a vector return a new object that shares all the nodes not affected by the operation with the original. Only the segments that are different get stored in separate node objects.

The implementation of vectors is in the Java class clojure.lang.PersistentVector. Instances of that class provide the API through which vectors are used and they hold a reference to the root of the tree. The nodes are implemented in the internal class Node.

Creating Vectors

To create a vector in the Java part of the language, you call the static method create from PersistentVector.java:

static public PersistentVector create(List items){
	TransientVector ret = EMPTY.asTransient();
	for(Object item : items)
		ret = ret.conj(item);
	return ret.persistent();
}

The actual work here is done by TransientVector, an internal class used for efficiently modifying vectors. The method starts by creating an empty TransientVector (line 2). The elements are inserted into it by one by calling conj (lines 3, 4). Finally, the resulting transient is converted to a persistent representation (line 5).

Transients are a feature of Clojure that allows you (and the language’s internals) to use mutable data-structures in performance critical parts. Transients are thread-save and they cannot be shared with other code. As we’re looking at immutable data-structures here, we won’t go into the details of the implementation here. The vector that we get from the call to persistent has the same structure we would get by starting with an empty persistent vector and adding the values with immutable operations. So let’s look at that.

Adding Elements

When elements are added to a PersistentVector, the cons method is called.

public PersistentVector cons(Object val){
	int i = cnt;
	//room in tail?
	if(cnt - tailoff() < 32)
		{
		Object[] newTail = new Object[tail.length + 1];
		System.arraycopy(tail, 0, newTail, 0, tail.length);
		newTail[tail.length] = val;
		return new PersistentVector(meta(), cnt + 1, shift, root, newTail);
		}
	//full tail, push into tree
	Node newroot;
	Node tailnode = new Node(root.edit,tail);
	int newshift = shift;
	//overflow root?
	if((cnt >>> 5) > (1 << shift))
		{
		newroot = new Node(root.edit);
		newroot.array[0] = root;
		newroot.array[1] = newPath(root.edit,shift, tailnode);
		newshift += 5;
		}
	else
		newroot = pushTail(shift, root, tailnode);
	return new PersistentVector(meta(), cnt + 1, newshift, newroot, new Object[]{val});
}

For some reason, the method starts with initializing a variable i that is never used (line 2). After that, we are confronted with the details of structure used to store the values.

Trees don’t have tails – not even the strange computer science kind that grows downward. The tail here is not part of the tree, it’s an array holding values that are not part of the tree. Up to 32 values end up in the tail, the rest goes into the tree. As the name suggests, the last elements of the vector get stored in the tail.

32 is a magic number for vectors: the leaves of the tree have exactly 32 values, the other nodes have up to 32 children, and the aforementioned 32-element tail. The tailoff method called in line 4 returns the greatest multiple of 32 that’s less than the current length, i.e. the number of elements in stored in the tree. The field cnt is the length of the vector.

When there is still room in the tail, the tail array is copied into a new array (lines 6, 7) and the new value is appended (line 8). The return value is a new PersistentVector instance with the new tail array, an incremented count, and otherwise the same properties as the original vector (line 9). In this case adding an element is a constant time operation.

Only when the vector to which we’re adding has exactly 32*n elements (with n being a positive integer), we need to deal with the tree. In that case, a new tree node to hold the original vector’s tail is created (line 13). Then we create a tree with the new node in it (lines 14-24), and we return a new instance with the updated tree, the incremented count, and a new tail array that only contains the added element (line 25).

When constructing the tree, we have two cases: if the tree is completely full at the current height, the resulting tree needs a new level (lines 18-21). Otherwise, we only need to find the right place for the new node (line 24).

The field shift is the height of the tree multiplied by 5. The height is represented in this form so that it can be used with the count and the capacity in bit-shift operations for additional efficiency. When we add the 33rd element, cnt is 32, as that is the value of the original vector. shift is 5, the initial value when there is only an empty root node. As a consequence, the root of the tree for the returned vector is constructed by calling pushTail.

private Node pushTail(int level, Node parent, Node tailnode){
	int subidx = ((cnt - 1) >>> level) & 0x01f;
	Node ret = new Node(parent.edit, parent.array.clone());
	Node nodeToInsert;
	if(level == 5)
		{
		nodeToInsert = tailnode;
		}
	else
		{
		Node child = (Node) parent.array[subidx];
		nodeToInsert = (child != null)?
		                pushTail(level-5,child, tailnode)
		                :newPath(root.edit,level-5, tailnode);
		}
	ret.array[subidx] = nodeToInsert;
	return ret;
}

The method starts by calculating the index where to add the new node depending on the length of the original vector and the level of the tree where we are looking to insert the node (line 2). In the example of adding the 33rd value, the index of the new node is 0, as this is the first node to be pushed into the root. When we add the 65th element – thus having filled up the tail for the second time – we get the index 1.

The return value of pushTail is a new node that clones the references to all the children of the original parent node, so that the child nodes are shared between the old and the new vector (line 3). If we are at the level where the child nodes are leaves, we directly insert the node we’re supposed to add at the calculated index of the returned node (lines 7, 16, 17).

When the tree is already higher, we cannot add the leaf node directly to the parent node. Instead we find the child node at the calculated index (line 11) and use the recursive nature of trees to call pushTree again for that subtree with a lower level (line 13). If no child node in the calculated place exists, we call newPath to create a new subtree of the appropriate hight that contains nodes with exactly one child down to our newly added leaf node (line 14). These operations only copy the subtree where the new value is added. Because of the 32-children condition these are at most O(log32 N) steps.

This was the part of the insertion process, when we can insert into the tree without adding another level. Back in the cons method, we still need to look at the case where we grow the tree in height (lines 18-21). Here we create an all new root for the returned vector (line 18) that gets the original vector’s root as the first child (line 19) and a new subtree of the right height with the new leaf node created by newPath as the second child (line 20). Finally, the field shift for the new vector is incremented by 5 to reflect the new height (line 21). As the number of steps for this operation is bounded by the height of the tree, the complexity here is also O(log32 N), which is consequently the complexity of the whole algorithm.

Summary

In this post we have mainly looked at adding elements to immutable vectors. The algorithm and data-structure described here is used for both creating and growing vectors, although the creation process uses optimizations based on mutable state which we did not discuss.

I want to close this post with an illustration of the structure using a before/after example. Let us assume we have a vector with 1056 elements and want to append another value calling cons.

The old vector (shown in blue) has a root node with 32 children each holding 32 values, and a tail with 32 more values. This means the tree is full at is current level. The new vector (shown in red) has another layer in its tree. The first child of the new root is the old root, thus sharing the structure of the old vector without copying. The second child of the new root is a node that has one leaf as child which contains a reference to the old tail – again, sharing not copying. The tail of the new vector contains a single value, the one we added.

IDSC III: Lazy Seqs

2010-06-10T00:00:00+02:00

Lazy evaluation is an important concept in functional programming. Running on the JVM, Clojure does not support general laziness, but it has a data-structure abstraction called lazy sequence that provides for many of the benefits of the more general strategy.

This post is a slight detour in the course of the immutable data-structure canon. In the previous installation, I described how some operations on lists return lazy sequences to retain immutability and performance characteristics. As this behavior is not specific to lists, and indeed fundamental to all Clojure data-structures, I decided that lazy seqs deserve a closer look.

The Rationale of Lazy Evaluation

In many cases it might not be necessary to use all the values in a data-structure. But it might be hard to determine what is required when the structure is defined, as the creation and the consumption could be in different parts of the program. Laziness allows you to write a general definition and later use only what you need without incurring the cost for needless computations.

Laziness also allows for an interesting special case: infinite sequences. You can specify the computation of elements for a series (e.g. Fibonacci numbers) and use parts of that series without problems, as long as you do not try to traverse the entire sequence.

Postponing Calculations

In a language with general support for lazy evaluation like Haskell, the compiler takes care that nothing gets evaluated before it is needed. In Clojure, however, eager evaluation is the norm. That means that when you call a function with the result of another function-call as a parameter, that second function gets evaluated immediately. In a lazy language, the second function would only be evaluated when the first function needs the respective parameter.

For this reason, lazy-seq (which as we saw last time is used to define lazy sequences), cannot be implemented as a function – the body of what we pass to it must not be evaluated at the time of definition. The solution is a construct often cited as the most powerful feature in Lisp: macros.

Macros are the extension mechanism of Clojure (and Lisp in general); they allow you as a programmer to add features to the language. This makes it possible to have a very small core language, but still provide for pleasant programming. Unlike functions, macros do not evaluate their arguments immediately. When Clojure comes across code that uses a macro it first expands the macro and replaces the code with the result before the compilation proceeds normally.

The definition of the lazy-seq macro in core.clj is surprisingly short:

(defmacro lazy-seq
  [& body]
  (list 'new 'clojure.lang.LazySeq
    (list* '^{:once true} fn* [] body)))

As macro expansion and the corresponding escaping rules are beyond the scope here, let it suffice to say that (lazy-seq BODY) would expand to

(new clojure.lang.LazySeq (fn* [] BODY))

The body that defines the seq is wrapped into an anonymous function and passed to the constructor of the LazySeq class. (new is a special form for Java interoperability that allows us to construct a Java object.) The main part of the implementation of lazy seqs is in Java, more specifically in the aforementioned class clojure.lang.LazySeq.

LazySeq has three fields:

fn: a function object to store the definition body,
sv: a generic object to store the computed value, and
s: a representation of the current view of the sequence.

When we construct a LazySeq instance, only the function object is assigned a value.

For performance reasons, LazySeq does not simply call the function object and return its result, when the contents are accessed. Instead the other two fields are used to do some caching.

Caching

To understand how the access works, let us look at the first method in LazySeq.java:

public Object first(){
	seq();
	if(s == null)
		return null;
	return s.first();
}

The seq method that is called in line 2, is the key to how access to lazy seqs works. It is also called from all the other access methods (count, more, cons, etc.).

final synchronized public ISeq seq(){
	sval();
	if(sv != null)
		{
		Object ls = sv;
		sv = null;
		while(ls instanceof LazySeq)
			{
			ls = ((LazySeq)ls).sval();
			}
		s = RT.seq(ls);
		}
	return s;
}

The sval method called from line 2 executes the function object and stores the result in the sv field. sval also takes care that the function is not executed more than once. In line 5 the calculated value is assigned to the temporary variable ls, and the field for the calculated value is set to null in line 6. Lines 7-10 resolve nested lazy seqs until we get a value of a different type assigned to the temporary variable ls.

The return value is computed and assigned to the field s in line 11. RT is the runtime class that provides the fundamental Clojure functions for the Java code implemented as static methods. The seq method in RT turns its parameter into a sequence if it implements the Iterable Java interface.

This computation happens only once. In subsequent calls to seq the cached value in field s is returned, as sv is set to null (line 6) so that the condition in line 3 is false and the block in lines 4-12 is not executed. The sval method does not set sv again in later invokations either.

The `concat` Example

To clarify what’s happening in seq, let us go through an example. We’ll use an abbreviated version of concat.

(defn concat
  [x y]
    (lazy-seq
      (let [s (seq x)]
        (if s
          (cons (first s) (concat (rest s) y)))
          y))))

What happens when we take the seq resulting from a concatenation and call first on it? Via the first method we get to seq which in turn calls sval, so that the body gets executed. The body of the lazy-seq use in concat is the call to cons (line 6). The result of cons is an instance of clojure.lang.Cons – an abstraction with list semantics, i.e. it has a head and a tail. The Cons is assigned to the sv field and – in the seq method – to the temporary variable ls (line 5). The loop for nested lazy seqs (lines 7-10) is skipped, because a Cons is not an instance of LazySeq.

In line 11, the Cons gets assigned to the field s (RT.seq does not do anything to it). Back to first. s is not null, so we call first on the Cons and return the result. The first element of the Cons is the first element of the concatenated seq, just as expected.

Accessing the tail of the seq, happens analogously; the tail of the Cons is another lazy seq due to the recursive definition of concat which we discussed last time.

Summary

Lazy sequences allow computations to be postponed. Even infinite sequences are possible
Lazy seqs are created using the lazy-seq macro
Using a macro avoids immediately evaluating the definition of the seq
The implementation of lazy seqs is in the Java class clojure.lang.LazySeq
Lazy seqs cache the results of the computation

The Design Of Design

2010-06-04T00:00:00+02:00

Mediocre design provably wastes the world’s resources, corrupts the environment, affects internatonal competitiveness. Design is important; teaching design is important.

Fred Brooks, “The Design of Design”. Page x

Fredrick Brooks, the author of the “Mythical Man Month” and the project manager of the IBM System/360 (the most successful mainframe computer series back in the day), has written a new book called “The Design of Design”. Like “The Mythical Man Month”, it is a collection of essays, but in this one focus is on how good design can be achieved.

While computer-related projects are a main source of examples in the book, Brooks also draws from experience in designing buildings and other projects. The book is aimed at designers and project managers of many kinds, but computer science is clearly Brooks’s home discipline.

The essays are by no means how-tos trying to teach you a particular design process. Nor are there any singularly novel ideas you have never heard before, if you have some background in designing software. The value of the book is that it makes you think about the different aspects of design. The texts are well written and opinionated, inviting you to look at the issues presented in them from angles that are different from what you might be used to. In the rest of this post, I summarize some ideas that resonated with me, but I really recommend reading the book.

Fred Brooks (c) sd&m

The book is organized in six parts, I follow that structure for my notes here (not all parts are mentioned, however).

Models of Designing

The hardest part of design is deciding what to design

Page 22

In the first part of the book, Brooks explains that he rejects the waterfall model, because it is impossible to know enough about requirements and other factors influencing the design up front.

The Rational Model, in any of its forms [the waterfall model being one of them], leads us to demand up-front statements of design requirements. It leads us to believe that such can be formulated. […]

The Waterfall Model is wrong and harmful; we must outgrow it.

Pages 33,34

This is old news to anyone sympathizing with agile styles of design, but it is interesting to hear it from someone who worked on mainframes for IBM in the 50ies and 60ies.

The rejection of the waterfall model raises two questions: (1) why are project plans still often based on it and (2) what is a better model?

There are two plausible answers to the first question. It seems obvious, clean and strait-forward. But more importantly, most projects involve some sort of contract between the designer and the customer, where the customer would not be to happy about an open-ended, pay by the hour, it’s-done-when-it’s-done deal.

Brooks proposes decoupling the contracts for design from the contracts for implementation, but concedes that this is not a complete solution especially in software, where the boundaries between design and construction are blurry.

So what about a better model? The purpose of a model for design is to serve as a means of teaching and as map to answer the question “where are we?” in the course of a project.

Brooks has no definitive and final answer to this question, but he is in favor of something based on the spiral model proposed by Barry Boehm. In this model, development repeatedly go through stages of planning, determining requirements and contraints, prototyping, risk analysis, and verification.

Collaboration and Telecollaboration

In the second part, Brooks questions the notion that collaboration and design in teams are good per se.

It is generally assumed that collaboration is, in and of itself, a “good thing.” “Plays well with others” is high praise from kindergarten onward. “All of us are smarter than any of us.” “The more participation in design, the better.” Now, these attractive propositions are far from self-evident. I will argue that they are not universally true.

Page 64

Although there are good reasons from collaborative design such as technological complexity and time contraints, the challenge is conceptual integrity. As a consequence, Brooks argues that a design team should always have one system architect who calls shots in favor of a consistent concept.

Design Perspectives

The third part of the book contains essays with ideas that are similar to those of agile development.

Brooks starts with an interesting perspective on design where he compares different approaches to the philosophical schools of empiricism and rationalism. Rationalists believe that given sufficient experience and careful consideration, you can come up with a perfect design. The empiricist view on the other hand is that anything you design, no matter how well considered, will be flawed. The flaws need to be fixed by trial and error.

Brooks is clearly in the empiricist camp:

Can I, by sufficient thought alone, design a complex object correctly? No; testing and iteration are in practice necessary. But careful thought helps.

Page 109

An important factor for the design is the model you have of your product’s user. While it is impractical to know everything about the users, Brooks argues that you should explicitly state the assumptions you make in great detail, even if some of them are wrong.

Wrong explicit assumptions are much better than vague ones. Wrong ones will perhaps be questioned; vague ones won’t.

Page 117

Reminding me of 37signals he supports the notion that constraints are your friends.

Since constraints are the designer’s friend, if the task originally seems unconstrained, first think harder about what is really desired, about the user and use models, and you will probably find some narrowing constraints, to the benefit of both designer and user.

Page 135

Brooks than goes on to examine esthetics and style in technical design. In his analysis “logical beauty” comes from parsimony, structural clarity, and consistency. Parsimony roughly means “accomplishing a great deal with few elements”. Parsimony alone, however, is not enough. Without any redundancy, as design can be cryptic and infective.

According to Brooks, “consistency underlies all principles of quality.”

A good architecture is consistent in the sense that, given partial knowledge of the system, one can predict the remainder.

Page 143

Three design principles are important for consistency: orthogonality (do not link what is independent), propriety (do not introduce what is immaterial), and generality (do not restrict what is inherent).

Great Designers

In the fifth part, Brooks examines the value of design processes, individual designers, and education in design. He maintains that “great designs come from great designers, not from great design processes.”

I believe that standard corporate product design processes do indeed work against truly great and innovative design.

Page 233

This does not mean, however, that he is rejects processes entirely. The key is some middle ground where processes prevent grave errors, but are not followed to the letter stifling creativity.

First and foremost, the top leader of the organization must passionately want innovative products with great designs

Page 238

Concerning education, the most important idea in the book is that you should learn by critiqued practice, instead of getting taught the basics before starting to make stuff. Another component in learning is teaching to others. A third part in the education of a designer should be studying exemplars – designs by other designers.

IDSC II: Lists

2010-05-27T00:00:00+02:00

Lists are the simplest data-structures that can easily grow. In this post, as part of the immutable data-structure canon, we’re going to look at how immutable lists are implemented in Clojure.

Let us review the definition of list in general. A list is either empty or has a head and a tail. The head contains a data element and the tail is another list. A simple recursive definition. The list (1 2 3) for example has 1 as its head and the list (2 3) as tail. The head of that list is 2 and the tail is the list (3). Finally, that list has 3 as head and the empty list as tail.

This structure is implemented as clojure.lang.PersistentList in the Java part of Clojure.

We can create a list using the list function

(def l1 (list 1 2 3))

We also saved the list in a binding named l1. The general convention (at least in Lisps) is that adding elements happens at the front of the list. To add an element, we use cons.

(def l2 (cons 42 l1))
l2
=> (42 1 2 3)
l1
=> (1 2 3)

cons returns a list with the added element, but l still contains only 1 2 3. This property is called persistence. Conceptually this is easy. We create a new list with 42 as the head and what we called l as the tail. l still refers to the same list as before.

Because the addition does not have to go through the existing list, this operation takes constant time.

Given the recursive definition of list, immutably and persistently removing elements from the front of the list is simple as well. Things get trickier when we concatenate two lists.

(def ls (concat l1 l2))
l1
=> (1 2 3)
l2
=> (42 1 2 3)
ls
=> (1 2 3 42 1 2 3)

In a mutable implementation, we could simply change the last element of the first list to refer to the second list as tail, but that would destroy the persistence of l1.

What is the performance expectation here? You might be tempted to say, that it is constant, because we only need to change one pointer. But remember, that we are talking about a singly-linked list. That means, we can only find the last element of l1 by traversing the whole list. Thus the operation takes linear time.

Back to our original question, how do we concatenate two lists without modifying them?

For Clojure the answer is both yes and no.

Clojure’s solution here could be described as “copying light”. We create a new data-structure that has new entries for all the elements, but each entry is only really created when it is used. This is called a lazy sequence.

A lazy sequence behaves like a normal collection. The special power of lazy seqs is that they don’t contain all the data elements, they only know how they are generated. This means that the values in the seq are only created when they are accessed, allowing expensive calculations to be postponed. Lazy seqs can even represent infinite data sets – as long as the data-structure is not traversed completely. We will look into this construct in greater depth in the next part of this series.

Back to our immutable lists. How do lazy seqs help with concatenating two lists? The concat function does not actually return a lists, it returns a lazy seq. When we go through the list, a new list entry is created. Thus for the whole list, we get a newly created entry of each element of both lists, resulting in linear complexity. The data elements, however, are not copied as we can be sure that they are also values and thus won’t change.

Here is the implementation of concat in core.clj:

(defn concat
  "Returns a lazy seq representing the concatenation of the elements in the supplied colls."
  ([] (lazy-seq nil))
  ([x] (lazy-seq x))
  ([x y]
    (lazy-seq
      (let [s (seq x)]
        (if s
          (if (chunked-seq? s)
            (chunk-cons (chunk-first s) (concat (chunk-rest s) y))
            (cons (first s) (concat (rest s) y)))
          y))))

(I’ve omitted the implementation of concatenating more than two lists, as that is not essential to this discussion.)

Let us walk through this code. There are three definitions of of the function here, the dispatch depends on arity (i.e. with how many parameters this function is called). Lines 3 and 4 handle the cases that concat is called with zero or one parameters the obvious way, by just returning a lazy seq with the same contents that were passed as parameters.

Starting in line 5, we have the “real” case for concat: two parameters. Line 6 tells us that the result is a lazy seq. Lines 7-12 are only executed when an element of the list is read – that’s the laziness.

Line 7 makes sure the rest of the function is working with a sequence type, and line 8 makes sure that the first list to concatenate is not nil, otherwise the second list is returned in line 12.

Chunked seqs are an optimization that we’ll ignore here, so we can ignore lines 9 and 10. That leaves us with line 11. Here, we take the head of the first list, and add it (using cons) to the result of a recursive call to concat passing the tail of the first list and the second list.

When we go through the above example of concatenating (1 2 3) and (42 1 2 3), we get a lazy seq back immediately, without the function looking at any of the elements. Only when we go through that result seq, lines 7-12 get invoked. For the first element, we get an entry that contains 1 as head and another lazy seq – the result of (concat (list 2 3) (list 42 1 2 3)) – as the tail. Getting the second happens the same way, as we resolve the second lazy seq. The third, ditto. After the third element, the first list passed to the inner call to concatis empty, so that the else part of the if in line 8 kicks in, and the result of the call is simply the second list (line 12).

In summary

Lists in clojure are immutable
By default, elements are added to the front of the list
The linked structure makes it easy to implement immutable adding of elements
Concatenating lists results in a lazy sequence where new list entries are created “on-demand”
The data elements are not copied, as they are immutable values

The Immutable Data-Structure Canon

2010-05-20T00:00:00+02:00

In the April issue of Communications of the ACM, George V. Neville-Neil (a.k.a. Kode Vicious) reviews the fundamental data-structures – array, list, tree, and hash table. Because of the importance of data-structures for all programming, he calls it the “data-structure canon” (requires subscription). Taking a cue from that article, I’m starting a series of posts about a special kind of implementation of the basic data-structures: the immutable lists, vectors, and maps in Clojure.

Clojure

Clojure is a Lisp dialect for the Java Virtual Machine with a focus on functional programming. It has strong support for clean multithreaded design and a key ingredient are immutable, persistent data-structures.

Immutability is important for clean multithreaded programs, because it allows you to separate values from identities. Rich Hickey, Clojure’s creator, defines these terms as follows:

By identity I mean a stable logical entity associated with a series of different values over time. Models need identity for the same reasons humans need identity – to represent the world. How could it work if identities like ‘today’ or ‘America’ had to represent a single constant value for all time? Note that by identities I don’t mean names (I call my mother Mom, but you wouldn’t).

So, for this discussion, an identity is an entity that has a state, which is its value at a point in time. And a value is something that doesn’t change. 42 doesn’t change. June 29th 2008 doesn’t change. Points don’t move, dates don’t change, no matter what some bad class libraries may cause you to believe. Even aggregates are values. The set of my favorite foods doesn’t change, i.e. if I prefer different foods in the future, that will be a different set.

When you have pure functions dealing with values, problems with multiple threads simply go away, because there is no danger of a value being changed from under a function’s backside. Of course, the world is not as easy as that, and programs need to model state that can change.

In Clojure, an identity can be in different states – i.e. have different values – over time. Identities are implemented as atomic references to values using so called Refs and Agents. But these constructs do not concern us in this series, I mentioned them only to give a background on the rationale for using immutable data-structures to model values.

Abstractly speaking, new values are functions of old values. For example, let us define a list and bind it to the name l1.

(def l1 (list 1 2 3))

When we want to add another element, we do not modify the existing list, but we call the function cons create new list with one more element.

(def l2 (cons 42 l1))
l2
=> (42 1 2 3)
l1
=> (1 2 3)

If we had passed l1 to a function in another thread, it could happily work on the initial three elements without being bothered by our addition of 42.

Your initial impression is probably that there is a lot of copying involved in the implementation of the immutable data-structures, but that ain’t so. Clojure’s data-structures maintain close to the same performance guarantees that their mutable counterparts make.

In the following posts, we look at the implementation of lists, vectors, and maps one by one.

How Hard Can It Be to Read a File?

2010-05-09T00:00:00+02:00

Reading code is good for you. Everyone knows this, but few actually follow the advice. It’s like healthy eating: Nobody would oppose it, but it is often ignored out of inertia. I’m certainly guilty of skipping my healthy dose of code reading in the past, but I’m going to make a habit out of it from now on. To support the process, I will write about the code I’m reading.

(c)iStockphoto.com/skodonnell

For the first exercise, I’m going to take a look at a simple bit of library code: reading a file into a string. I choose this example, because it turns out that reading a file with Java is more involved than I though it should be.

Coming from Ruby I’m used to being able to read the contents of a file into a string using one line of code like this:

str = File::read "bla.txt"

The simplest solution I found for Java is this

String str = "";
try {
	BufferedReader in = new BufferedReader(new FileReader("bla.txt"));
	String line;
	while((line = in.readLine()) != null) {
  		str += line + "\n";
	}
	in.close();
} catch(IOException e) {
}

Let us ignore the missing error handling and the inefficiency of concatenating strings for the moment. The number of lines is not of interest either – this not some “Java sucks”-rant, there are more than enough of that already.

The difficulty of this trivial task illustrates a concept I have been thinking about regularly lately: the Radius of Comprehension which I wrote about here. It is a property of a codebase defined as follows

If you are looking at a given fragment of code, how far away from that bit of the code do you need to have in your mind at that time in order to understand the fragment at hand?

So how far from the code in our examples do we have to have in our heads? In Ruby, we need to understand one method in one class. (I’m not including String here, as I consider that to be part of fundamental understanding of the language.) However, to actually find the read method, you need to look into the IO class, the parent of File. The names of the classes and the method are pretty obvious, so you quickly find them when looking through the documentation.

I’m going to make a naive attempt at quantifying the radius of comprehension here. The Ruby solution tentatively gets a score of three: two classes and one method each with meaningful names. I’ll talk about the merits of this quantification at the end of the post, but first let’s get the comparison with Java.

In the Java version, we have two classes, BufferedReader and FileReader, and one method, readLine. Having to loop through the file and concatenate all the lines makes the code more verbose and thus harder to read, but I wouldn’t say it makes it harder to understand. A source of confusion in this code is the indirect relationship between the FileReader and the actual reading. FileReader is a self-explanatory name, but BufferedReader? Having one reader and passing it to another reader from which you can actually read strings in your program increases the radius of comprehension by more than an obvious inheritance. As a consequence I assign the Java code a radius four: one for the method, one for the FileReader, and two for BufferedReader.

Error Handling

There is another factor I want to take into account here: error handling. In both examples an exception could be thrown, and we need to understand what kind of exception that is in order to write robust code. So I would add one to both radius scores.

In the Java code there is an additional complication: an error could occur when creating the readers or in the call to readLine. In the latter case, we should to close the reader to avoid leaking resources. In Ruby we don’t have to worry about closing anything. Understanding this additional error case in Java adds to the radius. Thus, with error handling we end up with: Ruby: 4, Java: 6.

Again, the intention of this post is not to prove that Java stinks and Ruby is great. I’m looking at a very limited use case here and actively ignore scenarios where the added complexity in Java’s library might be useful. The point is understand more about the radius of comprehension.

What I like about this metric is that the difference between the examples is a factor of 1.5 while the more straight forward metric of lines of code differs by a factor of 10. This score feels about right. The Java code is more complicated, but not hugely so.

Why Quantify the Radius of Comprehension?

While I’m satisfied with the result of my calculation here, the ad-hoc way I pulled the number from my pants is not where near scientific or generally useful. Maybe I can improve and formalize the process while reading more code. There is, however, another deeper question: Is it even worthwhile to try and formalize the radius of comprehension?

Mike Taylor, who invented the term, writes

I’m talking about a human issue here (and therefore, sadly, an all but impossible one to measure, though we know it when we see it)

While we may not be able to measure it given the vagueness of the definition, it might be possible to approximate it. Like any metric about code quality, it brings the danger of creating an arbitrary goal that is pursued at the expense of others, with a negative overall effect on quality. But I also see potential in quantifying the radius of comprehension.

Code always has two audiences: the compiler or interpreter and other programmers who have to use or maintain it. While we can reliably and immediately assess how well the machine understands our code, feedback on how well others understand it is rare and fuzzy. If we had a way to determine the radius of comprehension, we could judge how well other programmers can understand what we wrote – and take steps to improve it.

Ibis and Baboon

Chomsky, Norvig and Practical Machine Learning

On the News

Link: Punk Rock Languages

Link: Presenting Like a Hacker

Link: Full Text RSS Feed

Literate Programming

The Functional Elegance of Ring Middleware

IDSC V: Accessing and Modifying Vectors

Accessing Elements

Deleting Elements

Summary

IDSC IV: Creating and Growing Vectors

Creating Vectors

Adding Elements

Summary

IDSC III: Lazy Seqs

The Rationale of Lazy Evaluation

Postponing Calculations

Caching

The concat Example

Summary

The Design Of Design

Models of Designing

Collaboration and Telecollaboration

Design Perspectives

Great Designers

IDSC II: Lists

In summary

The Immutable Data-Structure Canon

Clojure

More on HTML5

How Hard Can It Be to Read a File?

Error Handling

Why Quantify the Radius of Comprehension?

The `concat` Example