The general approach I'm following code wise with that command language is to first get a code API to expose the capabilities of the system, then somehow plug the command language into that API thanks to a parser. It turns out that doing so in Common Lisp is really easy, and that you can get a compiler for free too, while at it. Let's see about that.

A very simple toy example

In this newsgroup article What is symbolic compoutation?, Pascal Bourguignon did propose a very simple piece of code:

(defparameter *additive-color-graph*
  '((red   (red white)   (green yellow) (blue magenta))
    (green (red yellow)  (green white)  (blue cyan))
    (blue  (red magenta) (green cyan)   (blue white))))

(defun symbolic-color-add (a b)
  (cadr (assoc a (cdr (assoc b *additive-color-graph*)))))

This is an example of symbolic computation, and we're going to build a little language to express the data and the code. Not that we would need to build one, mind you, more in order to have a really simple example leading us to the ahah moment you're now waiting for.

Before we dive into the main topic, you have to realize that the previous code example actually works: it's defining some data, using an implicit data structure composed by nesting lists together, and defines a function that knows how to sort out the data in that anonymous data structure so as to compound 2 colors together.

TOY-PARSER> (symbolic-color-add 'red 'green)
YELLOW

A command language and parser

I decided to go with the following language:

color red   +red white    +green yellow  +blue magenta
color green +red yellow   +green white   +blue cyan
color blue  +red magenta  +green cyan    +blue white

mix red and green

And here's how some of the parser looks like, using the esrap packrat lib:

(defrule color-name (and whitespaces (+ (alpha-char-p character)))
  (:destructure (ws name)
    (declare (ignore ws))               ; ignore whitespaces
    ;; CL symbols default to upper case.
    (intern (string-upcase (coerce name 'string)) :toy-parser)))

;;; parse string "+ red white"
(defrule color-mix (and whitespaces "+" color-name color-name)
  (:destructure (ws plus color-added color-obtained)
    (declare (ignore ws plus))          ; ignore whitespaces and keywords
    (list color-added color-obtained)))

;;; mix red and green
(defrule mix-two-colors (and kw-mix color-name kw-and color-name)
  (:destructure (mix c1 and c2)
    (declare (ignore mix and))          ; ignore keywords
    (list c1 c2)))

Those rules are not the whole parser, go have a look at the project on github if you want to see the whole code, it's called toy-parser over there. The main idea here is to show that when we parse a line from our little language, we produce the simplest possible structured data: in lisp that's symbols and lists.

The reason why it makes sense doing that is the next rule:

(defrule program (and colors mix-two-colors)
  (:destructure (graph (c1 c2))
    `(lambda ()
       (let ((*additive-color-graph* ',graph))
         (symbolic-color-add ',c1 ',c2)))))

This rule is the complex one to bind them all. It's using a quasiquote, a basic lisp syntax element allowing the programmer to very easily produce data that looks exactly like code. Let's see how it goes with a very simple example:

TOY-PARSER> (pprint (parse 'program
                           "color red +green yellow mix green and red"))

(LAMBDA NIL
  (LET ((*ADDITIVE-COLOR-GRAPH* '((RED (GREEN YELLOW)))))
    (SYMBOLIC-COLOR-ADD 'RED 'GREEN)))

The parser is producing structure (nested) data that really looks like lisp code, right? So maybe we can just run that code...

What about a compiler now?

Let's see about actually running the code:

TOY-PARSER> (let* ((code "color red +green yellow mix green and red")
                   (program (parse 'program code)))
              (compile nil program))
#<Anonymous Function #x3020027CF0EF>
NIL
NIL
TOY-PARSER> (let* ((code "color red +green yellow mix green and red")
                   (program (parse 'program code)))
              (funcall (compile nil program)))
YELLOW

So we have a string reprensing code in our very little language, and a parser that knows how to produce a nested list of atoms that looks like lisp code. And as we have lisp, we can actually compile that code at run-time with the same compiler that we used to produce our parser, and we can then funcall that function we just built.

Oh and the function is actually compiled down to native code, of course:

TOY-PARSER> (let* ((code "color red +green yellow mix red and green")
                   (program (parse 'program code))
                   (func    (compile nil program)))
              (time (loop repeat 1000 do (funcall func))))

(LOOP REPEAT 1000 DO (FUNCALL FUNC))
took 108 microseconds (0.000108 seconds) to run.
During that period, and with 4 available CPU cores,
     105 microseconds (0.000105 seconds) were spent in user mode
      13 microseconds (0.000013 seconds) were spent in system mode
NIL

Yeah, it took the whole of 108 microseconds to actually run the code generated by our own parser a thousand times, on my laptop. I can believe it's been compiled to native code, that seems like the right ballpark.

Conclusion

The toy-parser code is there on GitHub and you can actually load it using Quicklisp: clone the repository in ~/quicklisp/local-projects/ then (ql:quickload "toy-parser"), and play with it in (in-package :toy-parser).

The only thing I still want to say here is this: can your programming language of choice make it that easy?

We have a great line-up for this conference, which makes me proud to be able to be there. If you've ever been paying attention when using Emacs then you've already heard those names: Sacha Chua is frequently blogging about how she manages to improve her workflow thanks to Emacs Lisp, John Wiegley is a proficient Emacs contributor maybe best known for his ledger Emacs Mode, then we have Luke Gorrie who hacked up SLIME among other things, we also have Nic Ferrier who is starting a revolution in how to use Emacs Lisp with elnode. And more! Including Steve Yegge!

When using Common Lisp, you have an awesome interactive development environment where you can redefine function and objects while testing them. That means you don't have to quit the interpreter, reload the new version of the code and put the interactive test case together all over again after a change. Just evaluate the change in the interactive environement: functions are compiled incrementally over their previous definition, objects whose classes have changed are migrated live.

See, I just said objects and classes. Common Lisp comes with some advanced Object Oriented Programming facilities named CLOS and MOP where the Java and Python and C++ object models are just a subset of what you're being offered. Hint, those don't have Multiple Dispatch.

And you have a very sophisticated Condition System where Exceptions are just a subset of what you can do (hint: have a look a restarts and tell me you didn't wish your programming language of choice had them). And it continues that way for about any basic building bloc you might want to be using.

Loading data

Back to pgloader will you tell me. Right. I've been spending a couple of evening on hacking on the new version of pgloader in Common Lisp, and wanted to share some preliminary results.

The current status of the new pgloader still is pretty rough, if you're not used to develop in Common Lisp you might not find it ready for use yet. I'm still working on the internal APIs and trying to make something clean and easy to use for a developer, and then I will provide some external ways to play with it, user oriented. I missed that step once with the Python based version of the tool, I don't want to do the same errors again this time.

So here's a test run with the current pgloader, on a small enough data set of 226 MB of CSV files.

time python pgloader.py -R.. --summary -Tc ../pgloader.dbname.conf

Table name        |    duration |    size |  copy rows |     errors
====================================================================
aaaaaaaaaa_aaaa   |      2.148s |       - |      24595 |          0
bbbbbbbbbb_bbbb...|      0.609s |       - |        326 |          0
cccccccccc_cccc...|      2.868s |       - |      25126 |          0
dddddddddd_dddd...|      0.638s |       - |          8 |          0
eeeeeeeeee_eeee...|      2.874s |       - |      36825 |          0
ffffffffff_ffffff |      0.667s |       - |        624 |          0
gggggggggg_gggg...|      0.847s |       - |       5638 |          0
hhh_hhhhhhh       |      9.907s |       - |     120159 |          0
iii_iiiiiiiiiiiii |      0.574s |       - |        661 |          0
jjjjjjj           |      6.647s |       - |      30027 |          0
kkk_kkkkkkkkk     |      0.439s |       - |         12 |          0
lll_llllll        |      0.308s |       - |          4 |          0
mmmm_mmm          |      2.139s |       - |      29669 |          0
nnnn_nnnnnn       |      8.555s |       - |     100197 |          0
oooo_ooooo        |     13.781s |       - |      93555 |          0
pppp_ppppppp      |      8.275s |       - |      76457 |          0
qqqq_qqqqqqqqqqqq |      8.568s |       - |     126159 |          0
====================================================================
Total             |  01m09.902s |       - |     670042 |          0

Streaming data

With the new code in Common Lisp, I could benefit from real multi threading and higher level abstraction to make it easy to use: lparallel is a lib providing exactly what I need here, with workers and queues to communicate data in between them.

What I'm doing is that two threads are separated, one is reading the data from either a CSV file or a MySQL database directly, and pushing that data in the queue; while the other thread is pulling data from the queue and writing it into our PostgreSQL database.

CL-USER> (pgloader.csv:import-database "dbname"
            :csv-path-root "/path/to/csv/"
            :separator #\Tab
            :quote #\"
            :escape "\"\""
            :null-as ":null:")
                    table name       read   imported     errors       time
------------------------------  ---------  ---------  ---------  ---------
               aaaaaaaaaa_aaaa      24595      24595          0     0.995s
          bbbbbbbbbb_bbbbbbbbb        326        326          0     0.570s
       cccccccccc_cccccccccccc      25126      25126          0     1.461s
      dddddddddd_dddddddddd_dd          8          8          0     0.650s
eeeeeeeeee_eeeeeeeeee_eeeeeeee      36825      36825          0     1.664s
             ffffffffff_ffffff        624        624          0     0.707s
     gggggggggg_ggggg_gggggggg       5638       5638          0     0.655s
                   hhh_hhhhhhh     120159     120159          0     3.415s
             iii_iiiiiiiiiiiii        661        661          0     0.420s
                       jjjjjjj      30027      30027          0     2.743s
                 kkk_kkkkkkkkk         12         12          0     0.327s
                    lll_llllll          4          4          0     0.315s
                      mmmm_mmm      29669      29669          0     1.182s
                   nnnn_nnnnnn     100197     100197          0     2.206s
                    oooo_ooooo      93555      93555          0     9.683s
                  pppp_ppppppp      76457      76457          0     5.349s
             qqqq_qqqqqqqqqqqq     126159     126159          0     2.495s
------------------------------  ---------  ---------  ---------  ---------
             Total import time     670042     670042          0    34.836s
NIL

As you can see the control is still made for interactive developer usage, which is fine for now but will have to change down the road, when the APIs stabilize.

Now, let's compare to reading directly from MySQL:

CL-USER> (pgloader.mysql:stream-database "dbname")
                    table name       read   imported     errors       time
------------------------------  ---------  ---------  ---------  ---------
               aaaaaaaaaa_aaaa      24595      24595          0     0.887s
          bbbbbbbbbb_bbbbbbbbb        326        326          0     0.617s
       cccccccccc_cccccccccccc      25126      25126          0     1.497s
      dddddddddd_dddddddddd_dd          8          8          0     0.582s
eeeeeeeeee_eeeeeeeeee_eeeeeeee      36825      36825          0     1.697s
             ffffffffff_ffffff        624        624          0     0.748s
     gggggggggg_ggggg_gggggggg       5638       5638          0     0.923s
                   hhh_hhhhhhh     120159     120159          0     3.525s
             iii_iiiiiiiiiiiii        661        661          0     0.449s
                       jjjjjjj      30027      30027          0     2.546s
                 kkk_kkkkkkkkk         12         12          0     0.330s
                    lll_llllll          4          4          0     0.323s
                      mmmm_mmm      29669      29669          0     1.227s
                   nnnn_nnnnnn     100197     100197          0     2.489s
                    oooo_ooooo      93555      93555          0     9.148s
                  pppp_ppppppp      76457      76457          0     6.713s
             qqqq_qqqqqqqqqqqq     126159     126159          0     4.571s
------------------------------  ---------  ---------  ---------  ---------
          Total streaming time     670042     670042          0    38.272s
NIL

The streaming here is a tad slower than the importing from files. Now if you want to be fair when comparing those, you would have to take into account the time it takes to export the data out from its source. When doing that export/import dance, a quick test shows a timing of 1m4.745s. Now, if we do an export only test, it runs in 31.822s. So yes streaming is a good thing to have here.

Conclusion

We just got twice as fast as the python version.

Some will say that I'm not comparing fairly to the Python version of pgloader here, because I could have implemented the streaming facility in Python too. Well actually I did, the option are called section_threads and split_file_reading, that you can set so that a reader is pushing data into a set of queues and several workers are feeding each from its own queue. It didn't help with performances at all. Once again, read about the infamous Global Interpreter Lock to understand why not.

So actually it's a fair comparison here where the new code is twice as fast as the previous one, with only some hours of hacking and before spending any time on optimisation. Well, apart from using a producer, a consumer and a queue, which I almost had to have for streaming in between two database connections anyways.

In a recent migration project where we freed data from MySQL into PostgreSQL, we used pgloader again. But the loading time was not fast enough for the service downtime window that we had here. Indeed Python is not known for being the fastest solution around. It's easy to use and to ship to production, but sometimes you not only want to be able to be efficient when writing code, you also need the code to actually run fast too.

Faster data loading

So I began writing a little dedicated tool for that migration in Common Lisp which is growing on me as my personal answer to the burning question: python 2 or python 3? I find Common Lisp to offer an even more dynamic programming environment, an easier language to use, and the result often has performances characteristics way beyond what I can get with python. Between 5 times faster and 121 times faster in some quite stupid benchmark.

Here, with real data, my one shot attempt has been running more than twice as fast as the python version, after about a day of programming.

The other thing here is that I've tempted to get pgloader work in parallel, but at the time I didn't know about the Global Interpreter Lock that they didn't find how to remove in Python 3 still, by the way. So my threading attempts at making pgloader work in parallel are pretty useless.

Whereas in Common Lisp I can just use the lparallel lib, which exposes threading facilities and some queueing facilities as a mean to communicate data in between workers, and have my code easily work in parallel for real.

Compatibility

The only drawback that I can see here is that if you've been writing your own reformating modules in python for pgloader (yes you can implement your own reformating module for pgloader), then you would have to port it to Common Lisp. Shout me an email if that's your case.

Next version

So, I think we're going to have a pgloader 3 someday, that will be way faster than the current one, and bundle some more features: real parallel behavior, ability to fetch non local data (connecting to MySQL directly, or HTTP, S3, etc); and I'm thinking about offering a COPY like syntax to drive the loading too, while at it. Also, the ability to discover the set of data to load all by itself when you want to load a whole database: think of it as a special Migration mode of operations.

Some feature requests can't be solved easily when keeping the old .INI syntax cruft, so it's high time to implement some kind of a real command language. I have several ideas about those, in between the COPY syntax and the SQL*Loader configuration format, which is both clunky and quite powerful, too.

After a beginning in TCL and a complete rewrite in python in 2005, it looks like 2013 is going to be the year of pgloader 3, in Common Lisp!

So, here we go with a simple Common Lisp attempt. The Lost in scope article begins with defining a very simple function returning a boolean value, only true when it's not monday.

Monday is special

Keep in mind that the following example has been choosen to be simple yet offer a case of lexical binding shadowing. It looks convoluted. Focus on the day binding.

(defparameter *days*
  '(monday tuesday wednesday thursday friday saturday sunday)
  "List of days in the week")

(defun any-day-but-monday? (day)
  "Returns a generalized boolean, true unless DAY is 'monday"
  (member day (remove-if (lambda (day) (eq day 'monday)) *days*)))

So as you can see, in Common Lisp we just get away with a list of symbols rather than a string that we split to have a list of strings, or an array of strings, as in the examples with python and ruby.

Now, the generalized boolean is either nil to mean false, or anything else to mean true, and in that example the return value of member is a sub-list that begins where the member was found:

CL-USER> (any-day-but-monday? 'monday)
NIL

CL-USER> (any-day-but-monday? 'tuesday)
(TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY)

Oh, and as we work with Common Lisp, we're having a real REPL where to play directly with our code, no need to add interactive stanzas in the main program text file just to be able to play with it. In Emacs Slime we just use C-M-x on a form to have it available in the REPL, or C-c C-l to load the whole file we're working on.

So, we see that Common Lisp scoping rules are silently doing the right thing here. Within the remove-if call we define a lambda function taking a single parameter called day. It so happens that this parameter is shadowing the any-day-but-monday? function parameter, and that shadowing only happens in the lexical scope of the lambda we are creating. For a detailed discussion about that concept, I would refer you to the Scope and Extent chapter of Common Lisp the Language, 2nd Edition.

In Common Lisp we have both lexical scope and dynamic extents, and a variable defined with defparameter or defvar or that you otherwise declare special will have a dynamic extent. Hence this section title.

Closures

Now, the lost in scope article tries some more at finding a solution around the scoping rules of the python and ruby languages, where the developer can not easily instruct the language about the scoping rules he wants to be using in a case by case way, as far as I can see.

First, let's reproduce the problem by using a single variable that we bind in all the closures. Those are called callbacks in the original article, so I've kept using that name here.

(defparameter *callbacks-all-sunday*
    (loop
       for day in *days*
       collect (lambda () day))
  "loop binds DAY only once")

In that example, there's only a single variable day that we reuse throughout the loop construct, so that when the loop ends, we have a list of closures all refering to the same variable, and this variable, by the end of the loop, has sunday as its value.

CL-USER> (mapcar #'funcall *callbacks-all-sunday*)
(SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY)

Closures, take 2

Now, the way to have what we want here, that is a list of closures each having its own variable.

(defparameter *callbacks*
  (mapcar (lambda (day)
            ;; for each day, produce a separate closure
            ;; around its own lexical variable day
            (lambda () day))
          *days*)
  "A list of callbacks to return the current day...")

And there we go:

CL-USER> (mapcar #'funcall *callbacks*)
(MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY)

Conclusion

Scoping rules are very important in any programming language, functional or not, and must be well understood by programmers. I find that once again, that topic has received a very deep thinking in Common Lisp, and the language is giving all the options to its developers.

I want to stress that in Common Lisp the scope rules are very clearly defined in the standard documentation of the language. For instance, defun and let both introduce a lexical binding, defvar and defparameter introduce a dynamic variable.

Also, as a user of the language you have the ability to declare any variable as being special in order to introduce yourself a dynamic variable. In C you can declare some variables as being static, which is something else and frown with a very different set of problems.

Today I'm back on that topic and as I'm toying with Common Lisp I though it would be a good excuse to learn me some new tricks. As you can see from the earlier blog entry, last time I did attack the digits problem quite lightly. Let's try a better approach now.

(defun digits (n)
  "return the list of the digits of N"
  (nreverse
   (loop for x = n then r
      for (r d) = (multiple-value-list (truncate x 10))
      collect d
      until (zerop r))))

As you can see I wanted to use that facility I like very much, the for x = n then r way to handle first loop iteration differently from the next ones. But I've been hinted on #lisp that there's a much better way to write same code:

(defun integer-digits (integer)
  "stassats version"
  (nreverse
   (loop with remainder
      do (setf (values integer remainder) (truncate integer 10))
      collect remainder
      until (zerop integer))))

That code runs about twice as fast as the previous one and is easier to reason about. It's using setf and the form setf values, something nice to discover as it seems to be quite powerful. Let's see how to use it, even if it's really simple:

CL-USER> (integer-digits 12304501)
(1 2 3 0 4 5 0 1)

Let's move on to solving the Happy Numbers problem though:

(defun sum-of-squares-of-digits (integer)
  (loop with remainder
     do (setf (values integer remainder) (truncate integer 10))
     sum (* remainder remainder)
     until (zerop integer)))

(defun happy? (n &optional seen)
  "return true when n is a happy number"
  (let* ((happiness (sum-of-squares-of-digits n)))
    (cond ((eq 1 happiness)      t)
          ((memq happiness seen) nil)
          (t
           (happy? happiness (push happiness seen))))))

(defun find-happy-numbers (limit)
  "find all happy numbers from 1 to limit"
  (loop for n from 1 to limit when (happy? n) collect n))

And here's how it goes:

CL-USER> (find-happy-numbers 100)
(1 7 10 13 19 23 28 31 32 44 49 68 70 79 82 86 91 94 97 100)

CL-USER> (time (length (find-happy-numbers 1000000)))
(LENGTH (FIND-HAPPY-NUMBERS 1000000))
took 1,621,413 microseconds (1.621413 seconds) to run.
       116,474 microseconds (0.116474 seconds, 7.18%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     1,431,332 microseconds (1.431332 seconds) were spent in user mode
       145,941 microseconds (0.145941 seconds) were spent in system mode
 185,438,208 bytes of memory allocated.
 1 minor page faults, 0 major page faults, 0 swaps.
143071

Of course that code is much faster than the one I wrote before both in SQL and Emacs Lisp, the reason being that instead of writing the number into a string with (format t "~d" number) then subseq to get them one after the other, we're now using truncate.

Happy hacking!

Update

It turns out that to solve math related problem, some maths hindsight is helping. Who would have believed that? So if you want to easily get some more performances out of the previous code, just try that solution:

(defvar *depressed-squares* '(0 4 16 20 37 42 58 89 145)
  "see http://oeis.org/A039943")

(defun undepressed? (n)
  "same as happy?, using a static list of unhappy sums"
  (cond ((eq 1 n) t)
        ((member n *depressed-squares*) nil)
        (t
         (let ((h (sum-of-squares-of-digits n)))
           (undepressed? h)))))

(defun find-undepressed-numbers (limit)
  "find all happy numbers from 1 to limit"
  (loop for n from 1 to limit when (undepressed? n) collect n))

Time to compare:

CL-USER> (time (length (find-happy-numbers 1000000)))
(LENGTH (FIND-HAPPY-NUMBERS 1000000))
took 1,938,048 microseconds (1.938048 seconds) to run.
       290,902 microseconds (0.290902 seconds, 15.01%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     1,778,021 microseconds (1.778021 seconds) were spent in user mode
       140,862 microseconds (0.140862 seconds) were spent in system mode
 185,438,208 bytes of memory allocated.
 3,320 minor page faults, 0 major page faults, 0 swaps.
143071

CL-USER> (time (length (find-undepressed-numbers 1000000)))
(LENGTH (FIND-UNDEPRESSED-NUMBERS 1000000))
took 1,036,847 microseconds (1.036847 seconds) to run.
         5,372 microseconds (0.005372 seconds, 0.52%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     1,018,708 microseconds (1.018708 seconds) were spent in user mode
        16,982 microseconds (0.016982 seconds) were spent in system mode
 2,289,152 bytes of memory allocated.
143071
CL-USER>

To quote the article:

The first thing I always do when playing around with a new software platform is to write a concurrent "Hello World" program. The program works as follows: One active entity (e.g. thread, Erlang process, Goroutine) has to print "Hello " and another one "World!\n" with the two active entities synchronizing with each other so that the output always is "Hello World!\n".

Here's my try in Common Lisp using lparallel and some local nicknames, the whole 23 lines of it:

(defun say-hello (helloq worldq n)
  (dotimes (i n)
    (format t "Hello ")
    (lq:push-queue :say-world worldq)
    (lq:pop-queue helloq))
  (lq:push-queue :quit worldq))

(defun say-world (helloq worldq)
  (when (eq (lq:pop-queue worldq) :say-world)
    (format t "World!~%")
    (lq:push-queue :say-hello helloq)
    (say-world helloq worldq)))

(defun hello-world (n)
  (let* ((lp:*kernel*  (lp:make-kernel 2)) ; a new one each time, as we end it
         (channel      (lp:make-channel))
         (helloq       (lq:make-queue))
         (worldq       (lq:make-queue)))
    (lp:submit-task channel #'say-world helloq worldq)
    (lp:submit-task channel #'say-hello helloq worldq n)
    (lp:receive-result channel)
    (lp:receive-result channel)
    (lp:end-kernel)))

If you want to play locally with that code, I've been updating it to a github project named go-hello-world, even if it's coded in CL. See the package.lisp in there for how I did enable the local nicknames lp and lq for the lparallel packages.

Beware of the REPL

In a previous version of this very article, I said that sometimes I get an extra line feed in the output and I didn't understand why. Some great Common Lisp folks did hint me about that: it's the REPL output that get intermingled with the program output, and that's because the hello-world main function was returning before the thing is over.

I've added a receive-result call in it per worker so that it waits until the end of the program before returning to the REPL, and that indeed fixes it. A way to assert that is using the time macro, which was always intermingled with the output before. It's fixed now:

CL-USER> (time (go-hello-world:hello-world 1000))
Hello World!
...
Hello World!
(GO-HELLO-WORLD:HELLO-WORLD 1000)
took 27,886 microseconds (0.027886 seconds) to run.
      1,593 microseconds (0.001593 seconds, 5.71%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     23,246 microseconds (0.023246 seconds) were spent in user mode
     14,427 microseconds (0.014427 seconds) were spent in system mode
 4,272 bytes of memory allocated.
 10 minor page faults, 0 major page faults, 0 swaps.
(#<PROCESS lparallel kernel shutdown manager(62) [Reset] #x30200109F65D> ...)
CL-USER>

Conclusion

While Go language seems to bring very interesting things on the table, such as better compilation units and tools, I still think that the concurrency primitives at the core of it are easy to find in other places. Which is a good thing, as it means we know they work.

That also means that we don't need to accept Go syntax as the only way to properly solve that concurrency problem, I much prefer doing so with Common Lisp (lack of?) syntax myself.

Update

A previous version of this article was finished and published too quickly, and the conclusion was made from a buggy version of the program. It's all fixed now. Thanks a lot to people who contributed comments so that I could fix it, and thanks again to James M. Lawrence for lparallel!

With so many smart qualifiers you can only guess that I did love the challenge. The idea is to write the simplest code possible and see how smarter you need to be when you need perfs. Let's have a try!

local python results

Here's the code I did use to benchmark the python solution:

def sumrange(arg):
    return sum(xrange(arg))

def sumrange2(arg):
    x = i = 0
    while i < arg:
        x += i
        i += 1
    return x


import ctypes
ct_sumrange = ctypes.CDLL('/Users/dim/dev/CL/jiaroo/sumrange.so')

def sumrange_ctypes(arg):
    return ct_sumrange.sumrange(arg)

if __name__ == "__main__":
    import timeit
    t1 = timeit.Timer('import jiaroo; jiaroo.sumrange(10**10)')
    t2 = timeit.Timer('import jiaroo; jiaroo.sumrange2(10**10)')
    ct = timeit.Timer('import jiaroo; jiaroo.sumrange_ctypes(10**10)')

    print 'timing python sumrange(10**10)'
    print 'xrange: %5fs' % t1.timeit(1)
    print 'while:  %5fs' % t2.timeit(1)
    print 'ctypes: %5fs' % ct.timeit(1)

Oh. And the C code too, sorry about that.

#include <stdio.h>

int sumrange(int arg)
{
    int i, x;
    x = 0;

    for (i = 0; i < arg; i++) {
        x = x + i;
    }
    return x;
}

And here's how I did compile it. The author of the inspiring article insisted on stupid optimisation targets, I did follow him:

gcc -shared -Wl,-install_name,sumrange.so -o sumrange.so -fPIC sumrange.c -O0

And here's the result I did get out of it:

python jiaroo.py
timing python sumrange(10**10)
xrange: 927.039917s
while:  2377.291237s
ctypes: 5.297124s

Let's be fair, with -O2 we get much better results:

timing python sumrange(10**10)
ctypes: 1.065684s

Common Lisp to the rescue

So let's have a try in Common Lisp, will you ask me, right?

Here's the code I did use, you can see three different tries:

;;;; jiaroo.lisp
;;;
;;; See http://jiaaro.com/python-performance-the-easyish-way
;;;
;;; The goal here is to find out if CL needs to resort to C for very simple
;;; optimisation tricks like python apparently needs too, unless using pypy
;;; (to some extend).

(in-package #:jiaroo)

;;; "jiaroo" goes here. Hacks and glory await!

(defun sumrange-loop (max)
  "return the sum of numbers from 1 to MAX"
  (let ((sum 0))
    (declare (type (and unsigned-byte fixnum) max sum)
             (optimize speed))
    (loop for i fixnum from 1 to max do (incf sum i))))

(defun sumrange-dotimes (max)
  "return the sum of numbers from 1 to MAX"
  (let ((sum 0))
    (declare (type (and unsigned-byte fixnum) max sum)
             (optimize speed))
    (dotimes (i max sum)
      (incf sum i))))

(defun pk-sumrange (max)
  (declare (type (and unsigned-byte fixnum) max)
           (optimize speed))
  (let ((sum 0))
    (declare (type (and fixnum unsigned-byte) sum))
    (dotimes (i max sum)
      (setf sum (logand (+ sum i) most-positive-fixnum)))))

(defmacro timing (&body forms)
  "return both how much real time was spend in body and its result"
  (let ((start (gensym))
        (end (gensym))
        (result (gensym)))
    `(let* ((,start (get-internal-real-time))
            (,result (progn ,@forms))
            (,end (get-internal-real-time)))
       (values ,result (/ (- ,end ,start) internal-time-units-per-second)))))

(defun bench-sumrange (power)
  "print execution time of both the previous functions"
  (let* ((max (expt 10 power))
         (lp-time (multiple-value-bind (r s) (timing (sumrange-loop max)) s))
         (dt-time (multiple-value-bind (r s) (timing (sumrange-dotimes max)) s))
         (pk-time (multiple-value-bind (r s) (timing (pk-sumrange max)) s)))
    (format t "timing common lisp sumrange 10**~d~%" power)
    (format t "loop:       ~2,3fs ~%" lp-time)
    (format t "dotimes:    ~2,3fs ~%" dt-time)
    (format t "pk dotimes: ~2,3fs ~%" pk-time)))

And here's the results:

CL-USER> (bench-sumrange 10)
timing common lisp sumrange 10**10
loop:       11.213s
dotimes:    7.642s
pk dotimes: 22.185s
NIL

Discussion

So python is very slow. C is pretty fast. And Common Lisp just in the middle. Honnestly I expected better performances from my beloved Common Lisp here, but I didn't try very hard, by using Clozure Common Lisp which is not the quicker Common Lisp implementation around. For this very benchmark, if you're seeking speed use either Steel Bank Common Lisp or CLISP which is known to have a pretty fast bignums implementation (which you don't need in 64 bits in that game).

On the other hand, I think that having to go write a C plugin and deal with how to compile and deploy it in the middle of a python script is something to avoid. When using Common Lisp you don't need to resort to that for the runtime to get down from python xrange implementation at 927.039917s down to the dotimes implementation taking 7.642s. That's about 121 times faster.

So while C is even better, and while I would like a Common Lisp guru to show me how to get a better speed here, I still very much appreciate the solution here.

Let's see the winning source code in python and common lisp to compare the programmer side of things: how hard was it really to get 121 times faster?

def sumrange(arg):
    return sum(xrange(arg))

(defun sumrange-dotimes (max)
  "return the sum of numbers from 1 to MAX"
  (let ((sum 0))
    (declare (type (and unsigned-byte fixnum) max sum)
             (optimize speed))
    (dotimes (i max sum)
      (incf sum i))))

That's about it. Yes we can see some manual optimisation directives here, which are optimisation extra complexity. Not to the same level as bringing a compiled artifact that you need to build and deploy, though. Remember that you will need to know the full path where to find the sumrange.so file on the production system, in the optimised python case, so that's what we are comparing against.

Here's what happens without the optimisation, and with a smaller target:

CL-USER> (time (jiaroo:sumrange-dotimes (expt 10 9)))
(JIAROO:SUMRANGE-DOTIMES (EXPT 10 9))
took 722,592 microseconds (0.722592 seconds) to run.
During that period, and with 2 available CPU cores,
     714,709 microseconds (0.714709 seconds) were spent in user mode
       1,183 microseconds (0.001183 seconds) were spent in system mode
499999999500000000

CL-USER> (time (let ((sum 0)) (dotimes (i (expt 10 9) sum) (incf sum i))))
(LET ((SUM 0)) (DOTIMES (I (EXPT 10 9) SUM) (INCF SUM I)))
took 2,174,767 microseconds (2.174767 seconds) to run.
During that period, and with 2 available CPU cores,
     2,156,549 microseconds (2.156549 seconds) were spent in user mode
        10,225 microseconds (0.010225 seconds) were spent in system mode
499999999500000000

We get a 3 times speed-up from those 2 lines of lisp optimisation directives, which is pretty good. And it's exponential as I didn't have the patience to actually wait until the non optimised 10^10 run finished, I killed it.

Conclusion

That's a case here where I don't know how to reach C level of performances with Common Lisp, which could just be because I don't know yet how to do.

Still, getting a 121 times speedup when compared to the pure python version of the code is pretty good and encourages me to continue diving into Common Lisp.

The article is very well written and makes it easy to think that coming up with the code for such a solver is a very easy task, you apply some basic problem search principles and there you are. Which is partly true, in fact. Also, he uses python, and that means that a lot of trivial programming activities are not a concern anymore, such as memory management.

As I've been teaching myself Common Lisp for some weeks now I though I would like to read a lisp version of his code, and the article even has a section titled Translations. Unfortunately, no lisp version is available there. One might argue that Clojure is a decent enough lisp, but my current quest is all about Common Lisp really. So I had to write one myself.

CL-USER> (sudoku:print-puzzle
          (sudoku:solve-grid
"530070000600195000098000060800060003400803001700020006060000280000419005000080079"))
5 3 4 | 6 7 8 | 9 1 2
6 7 2 | 1 9 5 | 3 4 8
1 9 8 | 3 4 2 | 5 6 7
------+-------+------
8 5 9 | 7 6 1 | 4 2 3
4 2 6 | 8 5 3 | 7 9 1
7 1 3 | 9 2 4 | 8 5 6
------+-------+------
9 6 1 | 5 3 7 | 2 8 4
2 8 7 | 4 1 9 | 6 3 5
3 4 5 | 2 8 6 | 1 7 9
took 1,974 microseconds (0.001974 seconds) to run.
During that period, and with 2 available CPU cores,
     1,894 microseconds (0.001894 seconds) were spent in user mode
        88 microseconds (0.000088 seconds) were spent in system mode
 174,320 bytes of memory allocated.
#<SUDOKU::PUZZLE #x3020023BB9FD>

Comments on the python version

Norvig's article is very well written, I think. By that I mean that by reading it you're confident that you've understood the problem and how the solution is articulated, so you almost think you don't need to really try to understand the code, it's just an illustration of the text.

Well, not so much. When you want to port the exact same algorithm you have to understand exactly what the code is doing so that you're not implementing something else. All the more when, as I did, you want to use some other data structure.

My goal was not to rewrite the code as-is, but to try and come up with idiomatic lisp code implementing Norvig's solution. So rather than using strings and dictionaries (in lisp, they still call them a hash table) I've been using more natural data structures.

The python code is really uneasy to follow, full of functional programming veteran tricks. I mean avoiding exceptions and simply returning False whenever there's a problem, and using functions such as all and some to manage that. It's certainly working, it's not making the code any easier to read.

To summarize, that code looks like it's been written by someone smart who didn't want to spend more than a couple of hours on it, and did take all known trustworthy shortcuts he could to achieve that goal. Quality and readability certainly weren't the key motive. I've been quite deceived after reading a very good article.

Comments on the common lisp version

Keep in mind that I'm just a Common Lisp newbie. I've been told some good pieces of advice by knowledgeable people though, so with some luck my implementation is somewhat lispy enough.

So we start by defining some data structures and low-level functions to build up the more complex one, so that it's easier to read and debug. The sudoku puzzle is then a grid of digits and a grid of possible values in places where the digits are yet unknown.

The way to represent that 9x9 grid is with using make-array:

(make-array '(9 9)
            :element-type '(integer 0 9)
            :initial-element 0)

Then the possible values. I though about using a bit-vector (and actually I did implement it that way), then I've been told that the Common Lisp way to approach that is using 2-complement integer representation, as we have plenty of functions to operate numbers that way. I wouldn't believe that would make the code simpler, but in fact it really did, see:

CL-USER> #b111111111
511
CL-USER> (logcount #b111111111)
9
CL-USER> (logcount 511)
9
CL-USER> (logbitp 3 #b100100100)
NIL
CL-USER> (logbitp 2 #b100100100)
T
CL-USER> (format nil "~2r" (logxor #b111111111 (ash 1 4)))
"111101111"
CL-USER> (logbitp 4 (logxor #b111111111 (ash 1 4)))
NIL

With that in mind, we can write the following code:

(defun count-remaining-possible-values (possible-values)
  "How many possible values are left in there?"
  ;; we could raise an empty-values condition if we get 0...
  (logcount possible-values))

(defun first-set-value (possible-values)
  "Return the index of the first set value in POSSIBLE-VALUES."
  (+ 1 (floor (log possible-values 2))))

(defun only-possible-value-is? (possible-values value)
  "Return a generalized boolean which is true when the only value found in
   POSSIBLE-VALUES is VALUE"
  (and (logbitp (- value 1) possible-values)
       (= 1 (logcount possible-values))))

(defun list-all-possible-values (possible-values)
  "Return a list of all possible values to explore"
  (loop for i from 1 to 9
     when (logbitp (- i 1) possible-values)
     collect i))

(defun value-is-set? (possible-values value)
  "Return a generalized boolean which is true when given VALUE is possible
   in POSSIBLE-VALUES"
  (logbitp (- value 1) possible-values))

(defun unset-possible-value (possible-values value)
  "return an integer representing POSSIBLE-VALUES with VALUE unset"
  (logxor possible-values (ash 1 (- value 1))))

You can see here that I was also under the influence of a recent reading about making it obvious, or so called intentional programming, following what Joe Armstrong has to say about it:

Intentional programming is a name I give to a style of programming where the reader of a program can easily see what the programmer intended by their code. The intention of the code should be obvious from the names of the functions involved and not be inferred by analysing the structure of the code. (Reading the code should) precisely expresses the intention of the programmer—here no guesswork or program analysis is involved, we clearly read what was intended.

So there we go with function names such as count-remaining-possible-values, that will help when reading some more complex code, as in the following, the meat of the solution:

(defmethod eliminate ((puzzle puzzle) row col value)
  "Eliminate given VALUE from possible values in cell ROWxCOL of PUZZLE, and
   propagate when needed"
  (with-slots (grid values) puzzle
    ;; if already unset, work is already done
    (when (value-is-set? (aref values row col) value)
      ;; eliminate the value from the set of possible values
      (let* ((possible-values
              (unset-possible-value (aref values row col) value)))
        (setf (aref values row col) possible-values)

        ;; now if we're left with a single possible value
        (when (= 1 (count-remaining-possible-values possible-values))
          (let ((found-value (first-set-value possible-values)))
            ;; update the main grid
            (setf (aref grid row col) found-value)

            ;; eliminate that value we just found in all peers
            (eliminate-value-in-peers puzzle row col found-value)))

        ;; now check if any unit has a single possible place for that value
        (loop
           for (r . c)
           in (list-places-with-single-unit-solution puzzle row col value)
           do (assign puzzle r c value))))))

So that lisp code is quite verbose and at 389 lines almost doubles the 201 lines Norvig had. When clarity is part of the goal, that's hard to avoid, I hope I made a good case that this is not due to lisp being overly verbose by itself.

Comments on the development environment

Or why I even considered Common Lisp as an interesting language for that kind of exercise, and some more. I'll have to tell about re-sharding data live with 16 threads and 256 databases, all in CL, someday.

So I've been doing some Emacs Lisp development for a while now, and the part that makes that so much fun is the instant reward. You write some code in your editor, type a key chord (usually, that's C-M-x runs the command eval-defun) and your code is loaded up, ready to be tested. In Emacs Lisp the test can be simply using your editor and watching the new behavior taking place, or playing in the M-x ielm console. When the code is not ready, it crashes, and you're left in the interactive debugger, where you can use C-x C-e runs the command eval-last-sexp to evaluate any expression in your source and see its value in the current debug frame.

That way of working is a huge productivity boost, that I've been missing much when getting back to writing C code for PostgreSQL. I can't C-M-x the current function and go write some SQL to test it right away, I have to compile the whole source tree, then install the new binaries, then restart the test server and then open up a psql console to interact with the new code. Of course I could just make check and watch the results, but then if I attach a debugger it complains that the code on-disk is more recent than the code in the core dump.

What if you want Emacs Lisp integrated facilities and something made for general programming rather than suited to building a text editor? Don't get me wrong, you can probably find more production ready code in elisp than in many other languages, just because Emacs has been there for about 35 years. Editor targeted production code, though.

This integrated development cycle is all the same when you're using Common Lisp. The awesome Superior Lisp Interaction Mode for Emacs is providing exactly that experience. Just run M-x slime and then as you define your code you can C-M-x the function at point, see the compilation errors and warnings if any in the associated REPL, and just try your code. I tend to mostly play in the command line, it's possible to just use C-x C-e while typing too.

Performances

Of course we do care! After all the original article came with a quite detailed performance analysis with graphs and all. I won't be reproducing that, sorry. I'll just show you what penalty you get for using an older language specification, much more dynamic and with more features than python, and with a great, scratch that, awesome development environment.

Oh wait, that's the other way round, no penalty, it's actually so much faster!

Python version perfs

The results I got on my desktop machine are about twice as fast as in the original article, I guess newer machines and newer python have something to say for that:

  dim ~/dev/CL/sudoku python sudoku.dim.py
  All tests pass.
  Solved 50 of 50 easy puzzles (avg 0.01 secs (151 Hz), max 0.01 secs).
  Solved 95 of 95 hard puzzles (avg 0.02 secs (42 Hz), max 0.12 secs).
  Solved 11 of 11 hardest puzzles (avg 0.01 secs (115 Hz), max 0.01 secs).

That makes an average of (50*151 + 95*42 + 11*115) / (50+95+11) = 82Hz.

That seems pretty good, let's continue.

As you can see I've cut away the random puzzle part, that's because I was too lazy to implement that part, which didn't seem all that interesting to me. If you think that's a problem and need solving, I accept patches.

Common lisp version perfs

When using SBCL on the same machine, what I got was:

  (sudoku:solve-example-grids)
  Solved 50 of 50 easy puzzles (avg .0021 sec (471.7 Hz), max 0.015 secs).
  Solved 95 of 95 hard puzzles (avg .0022 sec (446.0 Hz), max 0.008 secs).
  Solved 11 of 11 hardest puzzles (avg .0018 sec (550.0 Hz), max 0.003 secs).

With the same way to compute the average, we now have 461.6Hz.

Now, that's between 3 times and more than 10 times faster than the python version (taken collection per collection), for a comparable effort, a much better development environment, and the same all dynamic no explicit compiling approach.

Conclusion

I guess I'm fond of Common Lisp, which I already saw coming (so did you, right?), and now I have some public article and code to share about why :)

The code is hosted at https://github.com/dimitri/sudoku if you're interested, with the necessary files to reproduce, some docs, etc.

Also, apart from using integers as bitfields, which I did more for being lispy than for performances, I did very little effort for optimizing the code. It's quite naive in this respect, yet allow me an average of 461.6Hz rather than 82Hz, that's 5.6 times faster average.

So yes, I will continue to invest some precious time in Common Lisp as a very good interactive scripting language, and maybe more than that.