The general approach I'm following code wise with that command language is to first get a code API to expose the capabilities of the system, then somehow plug the command language into that API thanks to a parser. It turns out that doing so in Common Lisp is really easy, and that you can get a compiler for free too, while at it. Let's see about that.

A very simple toy example

In this newsgroup article What is symbolic compoutation?, Pascal Bourguignon did propose a very simple piece of code:

(defparameter *additive-color-graph*
  '((red   (red white)   (green yellow) (blue magenta))
    (green (red yellow)  (green white)  (blue cyan))
    (blue  (red magenta) (green cyan)   (blue white))))

(defun symbolic-color-add (a b)
  (cadr (assoc a (cdr (assoc b *additive-color-graph*)))))

This is an example of symbolic computation, and we're going to build a little language to express the data and the code. Not that we would need to build one, mind you, more in order to have a really simple example leading us to the ahah moment you're now waiting for.

Before we dive into the main topic, you have to realize that the previous code example actually works: it's defining some data, using an implicit data structure composed by nesting lists together, and defines a function that knows how to sort out the data in that anonymous data structure so as to compound 2 colors together.

TOY-PARSER> (symbolic-color-add 'red 'green)
YELLOW

A command language and parser

I decided to go with the following language:

color red   +red white    +green yellow  +blue magenta
color green +red yellow   +green white   +blue cyan
color blue  +red magenta  +green cyan    +blue white

mix red and green

And here's how some of the parser looks like, using the esrap packrat lib:

(defrule color-name (and whitespaces (+ (alpha-char-p character)))
  (:destructure (ws name)
    (declare (ignore ws))               ; ignore whitespaces
    ;; CL symbols default to upper case.
    (intern (string-upcase (coerce name 'string)) :toy-parser)))

;;; parse string "+ red white"
(defrule color-mix (and whitespaces "+" color-name color-name)
  (:destructure (ws plus color-added color-obtained)
    (declare (ignore ws plus))          ; ignore whitespaces and keywords
    (list color-added color-obtained)))

;;; mix red and green
(defrule mix-two-colors (and kw-mix color-name kw-and color-name)
  (:destructure (mix c1 and c2)
    (declare (ignore mix and))          ; ignore keywords
    (list c1 c2)))

Those rules are not the whole parser, go have a look at the project on github if you want to see the whole code, it's called toy-parser over there. The main idea here is to show that when we parse a line from our little language, we produce the simplest possible structured data: in lisp that's symbols and lists.

The reason why it makes sense doing that is the next rule:

(defrule program (and colors mix-two-colors)
  (:destructure (graph (c1 c2))
    `(lambda ()
       (let ((*additive-color-graph* ',graph))
         (symbolic-color-add ',c1 ',c2)))))

This rule is the complex one to bind them all. It's using a quasiquote, a basic lisp syntax element allowing the programmer to very easily produce data that looks exactly like code. Let's see how it goes with a very simple example:

TOY-PARSER> (pprint (parse 'program
                           "color red +green yellow mix green and red"))

(LAMBDA NIL
  (LET ((*ADDITIVE-COLOR-GRAPH* '((RED (GREEN YELLOW)))))
    (SYMBOLIC-COLOR-ADD 'RED 'GREEN)))

The parser is producing structure (nested) data that really looks like lisp code, right? So maybe we can just run that code...

What about a compiler now?

Let's see about actually running the code:

TOY-PARSER> (let* ((code "color red +green yellow mix green and red")
                   (program (parse 'program code)))
              (compile nil program))
#<Anonymous Function #x3020027CF0EF>
NIL
NIL
TOY-PARSER> (let* ((code "color red +green yellow mix green and red")
                   (program (parse 'program code)))
              (funcall (compile nil program)))
YELLOW

So we have a string reprensing code in our very little language, and a parser that knows how to produce a nested list of atoms that looks like lisp code. And as we have lisp, we can actually compile that code at run-time with the same compiler that we used to produce our parser, and we can then funcall that function we just built.

Oh and the function is actually compiled down to native code, of course:

TOY-PARSER> (let* ((code "color red +green yellow mix red and green")
                   (program (parse 'program code))
                   (func    (compile nil program)))
              (time (loop repeat 1000 do (funcall func))))

(LOOP REPEAT 1000 DO (FUNCALL FUNC))
took 108 microseconds (0.000108 seconds) to run.
During that period, and with 4 available CPU cores,
     105 microseconds (0.000105 seconds) were spent in user mode
      13 microseconds (0.000013 seconds) were spent in system mode
NIL

Yeah, it took the whole of 108 microseconds to actually run the code generated by our own parser a thousand times, on my laptop. I can believe it's been compiled to native code, that seems like the right ballpark.

Conclusion

The toy-parser code is there on GitHub and you can actually load it using Quicklisp: clone the repository in ~/quicklisp/local-projects/ then (ql:quickload "toy-parser"), and play with it in (in-package :toy-parser).

The only thing I still want to say here is this: can your programming language of choice make it that easy?

A very localized example

We first need to find and import some data, and I found at the following place a CSV listing of french cities with coordinates and population and some numbers of interest for the exercise here.

To import the data set, we first need a table, then a COPY command:

CREATE TABLE lion1906 (
  insee       text,
  nom         text,
  altitude    integer,
  code_postal text,
  longitude   double precision,
  latitude    double precision,
  pop99       bigint,
  surface     double precision
);

\copy lion1906 from 'villes.csv' with csv header delimiter ';' encoding 'latin1'

With that data in place, we can find the 10 nearest towns of a random choosing of us, let's pick Villeurbanne which is in the region of Lyon.

   select code_postal, nom, pop99
     from lion1906
 order by point(longitude, latitude) <->
          (select point(longitude, latitude)
             from lion1906
            where nom = 'Villeurbanne')
    limit 10;

 code_postal |          nom           | pop99
-------------+------------------------+--------
 69100       | Villeurbanne           | 124215
 69300       | Caluire-et-Cuire       |  41233
 69120       | Vaulx-en-Velin         |  39154
 69580       | Sathonay-Camp          |   4336
 69140       | Rillieux-la-Pape       |  28367
 69000       | Lyon                   | 445452
 69500       | Bron                   |  37369
 69580       | Sathonay-Village       |   1693
 01700       | Neyron                 |   2157
 69660       | Collonges-au-Mont-d'Or |   3420
(10 rows)

We find Lyon in our list in there, and we want the query now to return only that one as it has the greatest number of inhabitants in the list:

with neighbours as (
   select code_postal, nom, pop99
     from lion1906
 order by point(longitude, latitude) <->
          (select point(longitude, latitude)
             from lion1906 where nom = 'Villeurbanne')
    limit 10
)
  select *
    from neighbours
order by pop99 desc
   limit 1;

 code_postal | nom  | pop99
-------------+------+--------
 69000       | Lyon | 445452
(1 row)

Well, thank you PostgreSQL, that was easy!

Note that you can actually index such queries, that's called a KNN index. PostgreSQL knows how to use some kind of indexes to fetch data matching an expression such as ORDER BY a <-> b, which allow you to consider a KNN search in your application.

Let's get worldwide

The real scope of our exercise is to associate every known town in the world with some big city around, so let's first fetch and import some worldwide data this time, from http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz.

CREATE TABLE maxmind_worldcities (
        country_code text,
        city_lower text,
        city_normal text,
        region_code text DEFAULT '',
        population INT DEFAULT '0',
        latitude float8 DEFAULT '0',
        longitude float8 DEFAULT '0'
);

\copy maxmind_worldcities FROM '/tmp/worldcitiespop.txt' WITH  DELIMITER ',' QUOTE E'\f' CSV HEADER ENCODING 'LATIN1';

alter table maxmind_worldcities add column loc point;
update maxmind_worldcities set loc = point(longitude, latitude);

This time you can see that I created an extra column with the location in there, so that I don't have to compute it each time I need it, like I did before.

Now is the time to test that data set and hopefully fetch the same result as before when we only had french cities loaded:

with neighbours as (
   select country_code, city_lower, population
     from maxmind_worldcities
    where population is not null
 order by loc <->
          (select loc
             from maxmind_worldcities
            where city_lower = 'villeurbanne')
   limit 10
)
   select * from neighbours order by population desc limit 1;

  country_code | city_lower | population
--------------+------------+------------
 fr           | lyon       |     463700
(1 row)

Ok, looks like we're all set for the real problem. Now we want to pick for each of those cities it's nearest neighboor, so here's how to do that:

create index on maxmind_worldcities(country_code, region_code, city_lower);
create index on maxmind_worldcities using gist(loc);

create table maxmind_neighbours as
  select country_code, region_code, city_lower,
         (with neighbours as (
             select country_code, city_lower, population
               from maxmind_worldcities
              where population is not null
                    and country_code = wc.country_code
                    and region_code = wc.region_code
           order by loc <-> wc.loc
             limit 10)
             select city_lower
               from neighbours
           order by population desc
              limit 1
         ) as neighbour
    from maxmind_worldcities wc ;

To be fair, I have to tell you that this query took almost 2 hours to complete on my laptop here, but as I'm doing that for friend and a blog article, I've been lazy and didn't try to optimise it. It could be using LATERAL for sure, I don't know if that would help very much with performances: I didn't try.

With that in hands we can now check some cities and their biggest neighbours, as in the following query:

select * from maxmind_neighbours where city_lower = 'villeurbanne';
 country_code | region_code |  city_lower  | neighbour
--------------+-------------+--------------+-----------
 fr           | B9          | villeurbanne | lyon
(1 row)

And looking for New-York City suburbs I did find a chinatown, which is a pretty common smaller town name apparently:

select * from maxmind_neighbours where city_lower = 'chinatown';
 country_code | region_code | city_lower |   neighbour
--------------+-------------+------------+---------------
 sb           | 08          | chinatown  | honiara
 us           | CA          | chinatown  | san francisco
 us           | DC          | chinatown  | washington
 us           | HI          | chinatown  | honolulu
 us           | IL          | chinatown  | chicago
 us           | MT          | chinatown  | missoula
 us           | NV          | chinatown  | reno
 us           | NY          | chinatown  | new york
(8 rows)

Big Cities in the big world

So, let's see how many smaller towns each of those random big cities have:

   select country_code, region_code, neighbour, count(*)
    from maxmind_neighbours
   where neighbour in ('london', 'new york', 'moscow',
                       'paris', 'tokyo', 'sao polo', 'chicago')
group by country_code, region_code, neighbour;
 country_code | region_code | neighbour | count
--------------+-------------+-----------+-------
 gb           | H9          | london    |     2
 jp           | 40          | tokyo     |   414
 us           | NY          | new york  |   131
 ca           | 08          | london    |    16
 ru           | 48          | moscow    |   245
 fr           | A8          | paris     |    16
 us           | IL          | chicago   |    13
(7 rows)

And now let's be fair and see where are the cities with the greatest number of towns nearby them, with the following query:

  select country_code, region_code, neighbour, count(*)
    from maxmind_neighbours
   where neighbour is not null
group by country_code, region_code, neighbour
order by 4 desc
   limit 25;

 country_code | region_code | neighbour  | count
--------------+-------------+------------+-------
 cn           | 03          | nanchang   | 16759
 cn           | 26          | xian       | 12864
 id           | 18          | kupang     | 10715
 cn           | 24          | taiyuan    | 10550
 mm           | 11          | taunggyi   | 10253
 id           | 38          | makasar    |  9471
 ir           | 15          | ahvaz      |  9461
 id           | 01          | banda aceh |  9161
 cn           | 14          | lasa       |  8841
 cn           | 15          | lanzhou    |  8618
 ir           | 29          | kerman     |  8579
 id           | 26          | medan      |  7787
 ir           | 04          | iranshahr  |  7249
 ir           | 07          | shiraz     |  7219
 ma           | 55          | agadir     |  7121
 ir           | 42          | mashhad    |  7107
 af           | 08          | gazni      |  7011
 ir           | 33          | tabriz     |  6586
 cn           | 01          | hefei      |  6521
 bd           | 81          | dhaka      |  6480
 ir           | 08          | rasht      |  6471
 id           | 17          | mataram    |  6467
 id           | 33          | cilegon    |  6287
 af           | 23          | qandahar   |  6213
 cn           | 07          | fuzhou     |  6089
(25 rows)

The conference has been packed with awesome really. Among the things that I'm going home with are new thoughts, tricks and tips, and new modes to use in Emacs.

The main new though is all about learning to program. That's a problem space in which I have a growing interest in, and the conference talk about arxana showed that it should be possible to build an environment where you can learn programming with the excuse of having fun with maths. And after talking about music and its notation and lilypond, it should even be possible to offer some interactive programming environment where not only you can play music live as Meta-Ex is doing, but where the other output of your program would be the updated music scores.

The main practical bits I'm going home with is redshank, A collection of code-wrangling Emacs macros mostly geared towards Common Lisp, but some are useful for other Lisp dialects, too. That complements paredit and allows you to do some reformating very easily.

Lastly, I'm back to giving the dark background environment a try now. I think I prefer the contrast and richer color sets of the default Emacs color theme, but the black window has some classy visual effect too. And with main-line the effect is quite awesome!

J'ai eu le plaisir de réaliser une présentation intitulée « The Need for Speed » dans laquelle on replace l'effort d'optimisation dans son contexte métier, afin de faire une étude des coûts et bénéfices et de savoir non seulement à quoi s'attendre mais aussi quand s'arrêter.

Merci à dalibo pour cette conférence !

A case where we want to apply the previous article approach is when replicating data with a trigger based solution, such as SkyTools and londiste. Well, maybe not in all cases, we need to have a amount of UPDATE trafic worthy of setting up the solution. As soon as we know we're getting to replay important enough batches of events, though, certainly using the batch update tricks makes sense.

It so happens that londiste 3 includes the capability to use handlers. Those are plugins written in python (like all the client side code from SkyTools) whose job is to handle the processing of the event batches. Several of them are included in the londiste sources, and one of them is named bulk.py.

Bulk loading data with londiste

To use set in londiste.ini:

handler_modules = londiste.handlers.bulk

then add table with one of those commands:

londiste3 add-table xx --handler="bulk"
londiste3 add-table xx --handler="bulk(method=X)"

The default method is 0, and the available methods are the following:

correct (0)

• inserts as COPY into table
• update as COPY into temp table and single UPDATE from there
• delete as COPY into temp table and single DELETE from there

delete (1)

• as correct, but update are done as DELETE then COPY

merged (2)

• as delete, but merge insert rows with update rows

Conclusion

Yes, by using that handler which is provided by default in londiste, you will apply the previous article tricks in your replication solution. And you can even choose to use that for only some of the tables you are replicating.

So when you need to UPDATE a bunch of rows from a given source, remember that you can actually use a JOIN in the update statement. Either the source of data is already in the database, in which case it's as simple as using the FROM clause in the update statement, or it's not, and we're getting back to that in a minute.

UPDATE FROM

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row.

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

As we're talking about performances, the trick here is to use the COPY protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches.

Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol.

Insert or Update

And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all DML queries, which means that you can do the following just fine:

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd t USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

That query here is updating all the rows that are known in both the target and the source and returns what we took from the source in the operation, so that we can do an anti-join in the next step of the query, where we're inserting any row that was not taken care of in the update part of the statement.

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Now, you will tell me that we just solved the UPSERT problem. Well what happens if more than one transaction is trying to do the WITH (UPDATE) INSERT dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong?

What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table.

There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical.

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock.

Conclusion

The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough PostgreSQL releases you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today!

So when you need to UPDATE a bunch of rows from a given source, remember that you can actually use a JOIN in the update statement. Either the source of data is already in the database, in which case it's as simple as using the FROM clause in the update statement, or it's not, and we're getting back to that in a minute.

UPDATE FROM

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row.

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

As we're talking about performances, the trick here is to use the COPY protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches.

Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol.

Insert or Update

And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all DML queries, which means that you can do the following just fine:

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd t USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

That query here is updating all the rows that are known in both the target and the source and returns what we took from the source in the operation, so that we can do an anti-join in the next step of the query, where we're inserting any row that was not taken care of in the update part of the statement.

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Now, you will tell me that we just solved the UPSERT problem. Well what happens if more than one transaction is trying to do the WITH (UPDATE) INSERT dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong?

What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table.

There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical.

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock.

Conclusion

The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough PostgreSQL releases you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today!

So when you need to UPDATE a bunch of rows from a given source, remember that you can actually use a JOIN in the update statement. Either the source of data is already in the database, in which case it's as simple as using the FROM clause in the update statement, or it's not, and we're getting back to that in a minute.

UPDATE FROM

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row.

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

As we're talking about performances, the trick here is to use the COPY protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches.

Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol.

Insert or Update

And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all DML queries, which means that you can do the following just fine:

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd t USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

That query here is updating all the rows that are known in both the target and the source and returns what we took from the source in the operation, so that we can do an anti-join in the next step of the query, where we're inserting any row that was not taken care of in the update part of the statement.

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Now, you will tell me that we just solved the UPSERT problem. Well what happens if more than one transaction is trying to do the WITH (UPDATE) INSERT dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong?

What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table.

There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical.

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock.

Conclusion

The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough PostgreSQL releases you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today!

So when you need to UPDATE a bunch of rows from a given source, remember that you can actually use a JOIN in the update statement. Either the source of data is already in the database, in which case it's as simple as using the FROM clause in the update statement, or it's not, and we're getting back to that in a minute.

UPDATE FROM

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row.

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

As we're talking about performances, the trick here is to use the COPY protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches.

Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol.

Insert or Update

And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all DML queries, which means that you can do the following just fine:

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

That query here is updating all the rows that are known in both the target and the source and returns what we took from the source in the operation, so that we can do an anti-join in the next step of the query, where we're inserting any row that was not taken care of in the update part of the statement.

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Now, you will tell me that we just solved the UPSERT problem. Well what happens if more than one transaction is trying to do the WITH (UPDATE) INSERT dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong?

What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table.

There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical.

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock.

Conclusion

The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough PostgreSQL releases you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today!

So when you need to UPDATE a bunch of rows from a given source, remember that you can actually use a JOIN in the update statement. Either the source of data is already in the database, in which case it's as simple as using the FROM clause in the update statement, or it's not, and we're getting back to that in a minute.

UPDATE FROM

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row.

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

As we're talking about performances, the trick here is to use the COPY protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches.

Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol.

Insert or Update

And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all DML queries, which means that you can do the following just fine:

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

That query here is updating all the rows that are known in both the target and the source and returns what we took from the source in the operation, so that we can do an anti-join in the next step of the query, where we're inserting any row that was not taken care of in the update part of the statement.

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Now, you will tell me that we just solved the UPSERT problem. Well what happens if more than one transaction is trying to do the WITH (UPDATE) INSERT dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong?

What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table.

There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical.

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock.

Conclusion

The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough PostgreSQL releases you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today!

We have a great line-up for this conference, which makes me proud to be able to be there. If you've ever been paying attention when using Emacs then you've already heard those names: Sacha Chua is frequently blogging about how she manages to improve her workflow thanks to Emacs Lisp, John Wiegley is a proficient Emacs contributor maybe best known for his ledger Emacs Mode, then we have Luke Gorrie who hacked up SLIME among other things, we also have Nic Ferrier who is starting a revolution in how to use Emacs Lisp with elnode. And more! Including Steve Yegge!

The first query here has the default level of magic in it, really. What happens is that each time we do an update of the HyperLogLog hash value, we update some data which are allowing us to compute its cardinality.

=> select date,
          #users as daily,
          pg_column_size(users) as bytes
     from daily_uniques
 order by date;
    date    |      daily       | bytes
------------+------------------+-------
 2013-02-22 | 401676.779509985 |  1287
 2013-02-23 | 660187.271908359 |  1287
 2013-02-24 | 869980.029947449 |  1287
 2013-02-25 | 580865.296677817 |  1287
 2013-02-26 | 240569.492722719 |  1287
(5 rows)

And has advertized the data is kept in a static sized data structure. The magic here all happens at hll_add() time, the function you have to call to update the data.

Now on to something way more magic!

=> select to_char(date, 'YYYY/MM') as month,
          round(#hll_union_agg(users)) as monthly
     from daily_uniques group by 1;
  month  | monthly
---------+---------
 2013/02 | 1960380
(1 row)

The HyperLogLog data structure is allowing the implementation of an union algorithm that will be able to compute how many unique values you happen to have registered in both one day and the next. Extended in its general form, and doing SQL, what you get is an aggregate that you can use in GROUP BY constructs and window functions. Did you read about them yet?

Installing postgresql-hll

It's as simple as CREATE EXTENSION hll; really, even if to get there you must have installed the package on your system. We did some packaging work for debian and the result should appear soon in a distro near you.

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

So in our example what you see is that we want to decipher how many unique IP addresses we saw, and we do that by first creating a hash of that source data then calling hll_add() with the current value and the hash result.

The current value must be initialized using hll_empty().

Concurrency

The most awake readers among you have already spotted that: using an UPDATE on the same row over and over again is a good recipe to kill any form of concurrency, so you don't want to do that on your production setup unless you don't care about those UPDATE waiting piling up in your system.

The idea is then to fill-in a queue of updates and asynchronously update the daily_uniques table from that queue, possibly using the hll_add_agg aggregate that the extension provides, so that you do only one update per batch of values to process.

∅: Empty Set and NULL

Now, what happens when the batch of new unique values you want to update from is itself empty? Well I would have expected hll_add_agg over an empty set to return an empty hll value, the same as returned by hll_empty(), but it turns out it's returning NULL instead.

And then hll_add(users, NULL) will happily return NULL. So the next UPDATE is cancelling all the previous work, which is not nice. We had to cater for that case explicitely in the UPDATE query that's working from the batch of new values to add to our current HyperLogLog hash entry, and I can't resist to show off one of the most awesome PostgreSQL features here: writable CTE.

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

That's how you protect against an empty set being turned into a NULL. I think the real fix would need to be included in postgresql-hll itself, in making it so that the hll_add_agg aggregate returns hll_empty() on an empty set, and I will report that bug (with that very article as the detailed explanation of it).

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

I coulnd't resist to show off two of my favorite SQL constructs in that example query here, which are the Common Table Expressions (or CTE) and window functions. If that over() clause reads strange to you, take a minute now and go read about it. Yes, do that now, we're waiting.

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

When using postgresql-hll you need to be careful not to kill your application concurrency abilities, and you need to protect yourself against the ∅ killer too. The other thing to keep in mind is that the numbers you get out of the hll technique are estimates within a given precision, and you might want to read some more about what it means for your intended usage of the feature.

Installing postgresql-hll

It's as simple as CREATE EXTENSION hll; really, even if to get there you must have installed the package on your system. We did some packaging work for debian and the result should appear soon in a distro near you.

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

So in our example what you see is that we want to decipher how many unique IP addresses we saw, and we do that by first creating a hash of that source data then calling hll_add() with the current value and the hash result.

The current value must be initialized using hll_empty().

Concurrency

The most awake readers among you have already spotted that: using an UPDATE on the same row over and over again is a good recipe to kill any form of concurrency, so you don't want to do that on your production setup unless you don't care about those UPDATE waiting piling up in your system.

The idea is then to fill-in a queue of updates and asynchronously update the daily_uniques table from that queue, possibly using the hll_add_agg aggregate that the extension provides, so that you do only one update per batch of values to process.

∅: Empty Set and NULL

Now, what happens when the batch of new unique values you want to update from is itself empty? Well I would have expected hll_add_agg over an empty set to return an empty hll value, the same as returned by hll_empty(), but it turns out it's returning NULL instead.

And then hll_add(users, NULL) will happily return NULL. So the next UPDATE is cancelling all the previous work, which is not nice. We had to cater for that case explicitely in the UPDATE query that's working from the batch of new values to add to our current HyperLogLog hash entry, and I can't resist to show off one of the most awesome PostgreSQL features here: writable CTE.

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

That's how you protect against an empty set being turned into a NULL. I think the real fix would need to be included in postgresql-hll itself, in making it so that the hll_add_agg aggregate returns hll_empty() on an empty set, and I will report that bug (with that very article as the detailed explanation of it).

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

I coulnd't resist to show off two of my favorite SQL constructs in that example query here, which are the Common Table Expressions (or CTE) and window functions. If that over() clause reads strange to you, take a minute now and go read about it. Yes, do that now, we're waiting.

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

When using postgresql-hll you need to be careful not to kill your application concurrency abilities, and you need to protect yourself against the ∅ killer too. The other thing to keep in mind is that the numbers you get out of the hll technique are estimates within a given precision, and you might want to read some more about what it means for your intended usage of the feature.

Installing postgresql-hll

It's as simple as CREATE EXTENSION hll; really, even if to get there you must have installed the package on your system. We did some packaging work for debian and the result should appear soon in a distro near you.

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

So in our example what you see is that we want to decipher how many unique IP addresses we saw, and we do that by first creating a hash of that source data then calling hll_add() with the current value and the hash result.

The current value must be initialized using hll_empty().

Concurrency

The most awake readers among you have already spotted that: using an UPDATE on the same row over and over again is a good recipe to kill any form of concurrency, so you don't want to do that on your production setup unless you don't care about those UPDATE waiting piling up in your system.

The idea is then to fill-in a queue of updates and asynchronously update the daily_uniques table from that queue, possibly using the hll_add_agg aggregate that the extension provides, so that you do only one update per batch of values to process.

∅: Empty Set and NULL

Now, what happens when the batch of new unique values you want to update from is itself empty? Well I would have expected hll_add_agg over an empty set to return an empty hll value, the same as returned by hll_empty(), but it turns out it's returning NULL instead.

And then hll_add(users, NULL) will happily return NULL. So the next UPDATE is cancelling all the previous work, which is not nice. We had to cater for that case explicitely in the UPDATE query that's working from the batch of new values to add to our current HyperLogLog hash entry, and I can't resist to show off one of the most awesome PostgreSQL features here: writable CTE.

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

That's how you protect against an empty set being turned into a NULL. I think the real fix would need to be included in postgresql-hll itself, in making it so that the hll_add_agg aggregate returns hll_empty() on an empty set, and I will report that bug (with that very article as the detailed explanation of it).

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

I coulnd't resist to show off two of my favorite SQL constructs in that example query here, which are the Common Table Expressions (or CTE) and window functions. If that over() clause reads strange to you, take a minute now and go read about it. Yes, do that now, we're waiting.

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

When using postgresql-hll you need to be careful not to kill your application concurrency abilities, and you need to protect yourself against the ∅ killer too. The other thing to keep in mind is that the numbers you get out of the hll technique are estimates within a given precision, and you might want to read some more about what it means for your intended usage of the feature.

Installing postgresql-hll

It's as simple as CREATE EXTENSION hll; really, even if to get there you must have installed the package on your system. We did some packaging work for debian and the result should appear soon in a distro near you.

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

So in our example what you see is that we want to decipher how many unique IP addresses we saw, and we do that by first creating a hash of that source data then calling hll_add() with the current value and the hash result.

The current value must be initialized using hll_empty().

Concurrency

The most awake readers among you have already spotted that: using an UPDATE on the same row over and over again is a good recipe to kill any form of concurrency, so you don't want to do that on your production setup unless you don't care about those UPDATE waiting piling up in your system.

The idea is then to fill-in a queue of updates and asynchronously update the daily_uniques table from that queue, possibly using the hll_add_agg aggregate that the extension provides, so that you do only one update per batch of values to process.

∅: Empty Set and NULL

Now, what happens when the batch of new unique values you want to update from is itself empty? Well I would have expected hll_add_agg over an empty set to return an empty hll value, the same as returned by hll_empty(), but it turns out it's returning NULL instead.

And then hll_add(users, NULL) will happily return NULL. So the next UPDATE is cancelling all the previous work, which is not nice. We had to cater for that case explicitely in the UPDATE query that's working from the batch of new values to add to our current HyperLogLog hash entry, and I can't resist to show off one of the most awesome PostgreSQL features here: writable CTE.

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

That's how you protect against an empty set being turned into a NULL. I think the real fix would need to be included in postgresql-hll itself, in making it so that the hll_add_agg aggregate returns hll_empty() on an empty set, and I will report that bug (with that very article as the detailed explanation of it).

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

I coulnd't resist to show off two of my favorite SQL constructs in that example query here, which are the Common Table Expressions (or CTE) and window functions. If that over() clause reads strange to you, take a minute now and go read about it. Yes, do that now, we're waiting.

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

When using postgresql-hll you need to be careful not to kill your application concurrency abilities, and you need to protect yourself against the ∅ killer too. The other thing to keep in mind is that the numbers you get out of the hll technique are estimates within a given precision, and you might want to read some more about what it means for your intended usage of the feature.

Installing postgresql-hll

It's as simple as CREATE EXTENSION hll; really, even if to get there you must have installed the package on your system. We did some packaging work for debian and the result should appear soon in a distro near you.

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

So in our example what you see is that we want to decipher how many unique IP addresses we saw, and we do that by first creating a hash of that source data then calling hll_add() with the current value and the hash result.

The current value must be initialized using hll_empty().

Concurrency

The most awake readers among you have already spotted that: using an UPDATE on the same row over and over again is a good recipe to kill any form of concurrency, so you don't want to do that on your production setup unless you don't care about those UPDATE waiting piling up in your system.

The idea is then to fill-in a queue of updates and asynchronously update the daily_uniques table from that queue, possibly using the hll_add_agg aggregate that the extension provides, so that you do only one update per batch of values to process.

∅: Empty Set and NULL

Now, what happens when the batch of new unique values you want to update from is itself empty? Well I would have expected hll_add_agg over an empty set to return an empty hll value, the same as returned by hll_empty(), but it turns out it's returning NULL instead.

And then hll_add(users, NULL) will happily return NULL. So the next UPDATE is cancelling all the previous work, which is not nice. We had to cater for that case explicitely in the UPDATE query that's working from the batch of new values to add to our current HyperLogLog hash entry, and I can't resist to show off one of the most awesome PostgreSQL features here: writable CTE.

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

That's how you protect against an empty set being turned into a NULL. I think the real fix would need to be included in postgresql-hll itself, in making it so that the hll_add_agg aggregate returns hll_empty() on an empty set, and I will report that bug (with that very article as the detailed explanation of it).

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

I coulnd't resist to show off two of my favorite SQL constructs in that example query here, which are the Common Table Expressions (or CTE) and window functions. If that over() clause reads strange to you, take a minute now and go read about it. Yes, do that now, we're waiting.

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

When using postgresql-hll you need to be careful not to kill your application concurrency abilities, and you need to protect yourself against the ∅ killer too. The other thing to keep in mind is that the numbers you get out of the hll technique are estimates within a given precision, and you might want to read some more about what it means for your intended usage of the feature.

When using Common Lisp, you have an awesome interactive development environment where you can redefine function and objects while testing them. That means you don't have to quit the interpreter, reload the new version of the code and put the interactive test case together all over again after a change. Just evaluate the change in the interactive environement: functions are compiled incrementally over their previous definition, objects whose classes have changed are migrated live.

See, I just said objects and classes. Common Lisp comes with some advanced Object Oriented Programming facilities named CLOS and MOP where the Java and Python and C++ object models are just a subset of what you're being offered. Hint, those don't have Multiple Dispatch.

And you have a very sophisticated Condition System where Exceptions are just a subset of what you can do (hint: have a look a restarts and tell me you didn't wish your programming language of choice had them). And it continues that way for about any basic building bloc you might want to be using.

Loading data

Back to pgloader will you tell me. Right. I've been spending a couple of evening on hacking on the new version of pgloader in Common Lisp, and wanted to share some preliminary results.

The current status of the new pgloader still is pretty rough, if you're not used to develop in Common Lisp you might not find it ready for use yet. I'm still working on the internal APIs and trying to make something clean and easy to use for a developer, and then I will provide some external ways to play with it, user oriented. I missed that step once with the Python based version of the tool, I don't want to do the same errors again this time.

So here's a test run with the current pgloader, on a small enough data set of 226 MB of CSV files.

time python pgloader.py -R.. --summary -Tc ../pgloader.dbname.conf

Table name        |    duration |    size |  copy rows |     errors
====================================================================
aaaaaaaaaa_aaaa   |      2.148s |       - |      24595 |          0
bbbbbbbbbb_bbbb...|      0.609s |       - |        326 |          0
cccccccccc_cccc...|      2.868s |       - |      25126 |          0
dddddddddd_dddd...|      0.638s |       - |          8 |          0
eeeeeeeeee_eeee...|      2.874s |       - |      36825 |          0
ffffffffff_ffffff |      0.667s |       - |        624 |          0
gggggggggg_gggg...|      0.847s |       - |       5638 |          0
hhh_hhhhhhh       |      9.907s |       - |     120159 |          0
iii_iiiiiiiiiiiii |      0.574s |       - |        661 |          0
jjjjjjj           |      6.647s |       - |      30027 |          0
kkk_kkkkkkkkk     |      0.439s |       - |         12 |          0
lll_llllll        |      0.308s |       - |          4 |          0
mmmm_mmm          |      2.139s |       - |      29669 |          0
nnnn_nnnnnn       |      8.555s |       - |     100197 |          0
oooo_ooooo        |     13.781s |       - |      93555 |          0
pppp_ppppppp      |      8.275s |       - |      76457 |          0
qqqq_qqqqqqqqqqqq |      8.568s |       - |     126159 |          0
====================================================================
Total             |  01m09.902s |       - |     670042 |          0

Streaming data

With the new code in Common Lisp, I could benefit from real multi threading and higher level abstraction to make it easy to use: lparallel is a lib providing exactly what I need here, with workers and queues to communicate data in between them.

What I'm doing is that two threads are separated, one is reading the data from either a CSV file or a MySQL database directly, and pushing that data in the queue; while the other thread is pulling data from the queue and writing it into our PostgreSQL database.

CL-USER> (pgloader.csv:import-database "dbname"
            :csv-path-root "/path/to/csv/"
            :separator #\Tab
            :quote #\"
            :escape "\"\""
            :null-as ":null:")
                    table name       read   imported     errors       time
------------------------------  ---------  ---------  ---------  ---------
               aaaaaaaaaa_aaaa      24595      24595          0     0.995s
          bbbbbbbbbb_bbbbbbbbb        326        326          0     0.570s
       cccccccccc_cccccccccccc      25126      25126          0     1.461s
      dddddddddd_dddddddddd_dd          8          8          0     0.650s
eeeeeeeeee_eeeeeeeeee_eeeeeeee      36825      36825          0     1.664s
             ffffffffff_ffffff        624        624          0     0.707s
     gggggggggg_ggggg_gggggggg       5638       5638          0     0.655s
                   hhh_hhhhhhh     120159     120159          0     3.415s
             iii_iiiiiiiiiiiii        661        661          0     0.420s
                       jjjjjjj      30027      30027          0     2.743s
                 kkk_kkkkkkkkk         12         12          0     0.327s
                    lll_llllll          4          4          0     0.315s
                      mmmm_mmm      29669      29669          0     1.182s
                   nnnn_nnnnnn     100197     100197          0     2.206s
                    oooo_ooooo      93555      93555          0     9.683s
                  pppp_ppppppp      76457      76457          0     5.349s
             qqqq_qqqqqqqqqqqq     126159     126159          0     2.495s
------------------------------  ---------  ---------  ---------  ---------
             Total import time     670042     670042          0    34.836s
NIL

As you can see the control is still made for interactive developer usage, which is fine for now but will have to change down the road, when the APIs stabilize.

Now, let's compare to reading directly from MySQL:

CL-USER> (pgloader.mysql:stream-database "dbname")
                    table name       read   imported     errors       time
------------------------------  ---------  ---------  ---------  ---------
               aaaaaaaaaa_aaaa      24595      24595          0     0.887s
          bbbbbbbbbb_bbbbbbbbb        326        326          0     0.617s
       cccccccccc_cccccccccccc      25126      25126          0     1.497s
      dddddddddd_dddddddddd_dd          8          8          0     0.582s
eeeeeeeeee_eeeeeeeeee_eeeeeeee      36825      36825          0     1.697s
             ffffffffff_ffffff        624        624          0     0.748s
     gggggggggg_ggggg_gggggggg       5638       5638          0     0.923s
                   hhh_hhhhhhh     120159     120159          0     3.525s
             iii_iiiiiiiiiiiii        661        661          0     0.449s
                       jjjjjjj      30027      30027          0     2.546s
                 kkk_kkkkkkkkk         12         12          0     0.330s
                    lll_llllll          4          4          0     0.323s
                      mmmm_mmm      29669      29669          0     1.227s
                   nnnn_nnnnnn     100197     100197          0     2.489s
                    oooo_ooooo      93555      93555          0     9.148s
                  pppp_ppppppp      76457      76457          0     6.713s
             qqqq_qqqqqqqqqqqq     126159     126159          0     4.571s
------------------------------  ---------  ---------  ---------  ---------
          Total streaming time     670042     670042          0    38.272s
NIL

The streaming here is a tad slower than the importing from files. Now if you want to be fair when comparing those, you would have to take into account the time it takes to export the data out from its source. When doing that export/import dance, a quick test shows a timing of 1m4.745s. Now, if we do an export only test, it runs in 31.822s. So yes streaming is a good thing to have here.

Conclusion

We just got twice as fast as the python version.

Some will say that I'm not comparing fairly to the Python version of pgloader here, because I could have implemented the streaming facility in Python too. Well actually I did, the option are called section_threads and split_file_reading, that you can set so that a reader is pushing data into a set of queues and several workers are feeding each from its own queue. It didn't help with performances at all. Once again, read about the infamous Global Interpreter Lock to understand why not.

So actually it's a fair comparison here where the new code is twice as fast as the previous one, with only some hours of hacking and before spending any time on optimisation. Well, apart from using a producer, a consumer and a queue, which I almost had to have for streaming in between two database connections anyways.

It's pretty simple actually, when you have the point at the beginning of a word or an identifier (containing numbers, dashes, underscores and other punctuation signs), you can select the whole of it in a single key chord!

The best thing is that if you press the same key chord again, it will expand to include the next expression. And that works in plain text and most programming languages where I've tried it, which is not so much recently. It does not depend that much on the programming language anyway.

The full general solution here is to use something like expand region, don't miss the Emacs Rocks Expand Region Episode, it's less than 3 minutes and you will want to install expand-region after that. For easy installing, of course you are already using el-get right?

Now, a friend just asked this morning how to select the current word even when the the point is currently in the middle of it. Going manually back to the beginning of it is no fun. I knew about thing-at-point and a little about how it works, but didn't find anything readily made for that use case (hint: it needs to be an interactive command).

Here's what I came up with, then:

(defun mha:select-current-word ()
  "Select the current word."
  (interactive)
  (beginning-of-thing 'symbol)
  (push-mark (point) nil t)
  (end-of-thing 'symbol)
  (exchange-point-and-mark))

(global-set-key (kbd "C-S-M-SPC") 'mha:select-current-word)

I picked C-M-S-SPC not because it's the easiest way to invoke the new command, but because to me it's a quite natural extension to the C-M-SPC that I use so often. Again, each time you want to select a identifier in some code of yours, you'd most certainly be better off using C-M-SPC.

Skytools 3.1.3 enters debian

First news is that Skytools 3.1.3 has been entering debian today (I hope that by the time you reach that URL, it's been updated to show information according to the news here, but I might be early). As there's current a debian freeze to release wheezy (and you can help squash some bugs), this version is only getting uploaded to experimental for now. Thanks to the tireless work of Christoph Berg though, this version is already available from apt.postgresql.org.

Upgrading to PGQ 3

The other news is that I've been testing live upgrade scenario where we want to upgrade from PGQ to PGQ3, and it works pretty well, and it's quite simple to achieve too. Here's how.

So the first thing is to shut down the current ticker process. Then we install the new packages, assuming that you did follow the step in the wiki pointed above, please go read apt.postgresql.org again now if needs be.

pgqadm.py ticker.ini -s
sudo apt-get install postgresql-9.1-pgq3 skytools3-ticker skytools3

The ticker is not running anymore, we have the right version of the software installed. Next step is to upgrade the database parts of PGQ:

psql -f /usr/share/skytools3/pgq.upgrade_2.1_to_3.0.sql ...
psql -1 -f /usr/share/postgresql/9.1/contrib/pgq.upgrade.sql ...

Of course replace those ... with options such as your actual connection string. I tend to always add -vON_ERROR_STOP=1 to all these commands, so that I don't depend on having the right .psqlrc on the particular server I'm connected to. Also remember that if you want to do that for more than one database, you need to actually run that pair of commands for each of them.

Now it's time to restart the new ticker. The main changes from the previous one is that it is now a C program called pgqd that knows how to tick for any number of databases, so that you only have to have one instance around per cluster now.

sudo /etc/init.d/skytools3 start
tail -f /var/log/skytools/pgqd.log

Those two commands are taking for granted that you did prepare the pgqd setup the debian and skytools way, by adding your config in /etc/skytools3/pgqd.ini and editing /etc/skytools.ini accordingly, so that it's automatically taken into account at machine boot.

Note that I did actually exercised the procedure above while running a pgbench test replicated with londiste. Of course the replication has been lagging a little while no ticker was running, and then it catched-up as fast as it could, in that case:

INFO {count: 245673, ignored: 0, duration: 422.104366064}

Happy Hacking!

So if you have any batch processing needs, remember to consider what PGQ has to offer. And yes if you're running some cron job to compute things out of the database for you, you are doing some batch processing.

Having had the opportunity to meet more people from those other development communities, I really think we should go and reach for them in their own conferences. About any PostgreSQL community member I've been talking about with about that idea seemed to agree and generally already was thinking the same thing. And most are already doing it, in fact...

I had the pleasure to run two conferences there, both in the PostgreSQL devroom.

Event Triggers

I'm currently in the middle of implementing Event Triggers for PostgreSQL and I have been for about the last 2 years. It's a quite complex feature to get right and so the patch itself is complex and large, which means the reviewing process is complex and takes time.

That also means that some parts of the design have already been redone completely at least 3 times, and that what got commited to the PostgreSQL code is nothing like what the design we decided should go in looks like. That's just a fact of life, maybe, but that makes for a very long development process.

We're now getting to the end of it though, and this talk is showing both where we want to go with Event Triggers, where we are now and what remains to be done for 9.3 if we want the feature to be any useful.

If you're interested into that development, have a look at the slide deck and possibly ask me some questions about what's not clear on the pgsql-hackers mailing list (preferably).

The other way to get summarized and clear information about Event Triggers is the wiki page by the same name: Event Triggers.

You will see that while a lot has been done (internal refactoring, adding new infrastructure and SQL level commands, and the minimum PLpgSQL support); a lot remains to be done where the code has already been submitted several times, following several designs directions given by careful review on hackers, and still we have some choices to make.

Implementing High-Availability

This talk is showing several ways to implement High Availability with PostgreSQL. The fact is that that term is overloaded already, and usually covers two very different things which are Service Availability and Data Availability.

In the talk, we're showing up several techniques that you can use to address different set of compromises in between scaling, load balancing, data availability and durability, and service availability. The first two points could seem unrelated to the main topic, but scaling often is a simple enough way to achieve service availability... until you need to think about sharding, that is.

So the talk is all about making compromises in between them and getting to an architecture able to implement the choosen compromises. While the talk has been pretty well received, it was delivered in a 50 mins slot where we usually take a whole day or three when addressing that problems at a customer's site.

Some parts of how to get to the right architecture for the compromises that are important for you can't be fully covered in that time slot, while still being able to actually present the techniques that we're using.

I think it might be useful to extract a single use-case or two from that talk then have a full 50 mins version reduced to a single or a couple of very clear compromises and how to achieve them in details, rather than trying to present a full range of techniques and how to use them in different scenarios.

FOSDEM

After having been talking with many people, it appears that for next year's edition I should be proposing a more general talk that aims at helping developpers in other communities (python, ruby, etc) discover what's in for them in PostgreSQL. This database is full of advanced features that are really easy to use, and the only problem when preparing such a talk is choosing the right subset...

If you're running a local developper user group and are interested into learning some more about how PostgreSQL can help you in a daily basis, please do get in touch with me and let's schedule a presentation together!

Having had the opportunity to meet more people from those other development communities, I really think we should go and reach for them in their own conferences. About any PostgreSQL community member I've been talking about with about that idea seemed to agree and generally already was thinking the same thing. And most are already doing it, in fact...

I had the pleasure to run two conferences there, both in the PostgreSQL devroom.

Event Triggers

I'm currently in the middle of implementing Event Triggers for PostgreSQL and I have been for about the last 2 years. It's a quite complex feature to get right and so the patch itself is complex and large, which means the reviewing process is complex and takes time.

That also means that some parts of the design have already been redone completely at least 3 times, and that what got commited to the PostgreSQL code is nothing like what the design we decided should go in looks like. That's just a fact of life, maybe, but that makes for a very long development process.

We're now getting to the end of it though, and this talk is showing both where we want to go with Event Triggers, where we are now and what remains to be done for 9.3 if we want the feature to be any useful.

If you're interested into that development, have a look at the slide deck and possibly ask me some questions about what's not clear on the pgsql-hackers mailing list (preferably).

The other way to get summarized and clear information about Event Triggers is the wiki page by the same name: Event Triggers.

You will see that while a lot has been done (internal refactoring, adding new infrastructure and SQL level commands, and the minimum PLpgSQL support); a lot remains to be done where the code has already been submitted several times, following several designs directions given by careful review on hackers, and still we have some choices to make.

Implementing High-Availability

This talk is showing several ways to implement High Availability with PostgreSQL. The fact is that that term is overloaded already, and usually covers two very different things which are Service Availability and Data Availability.

In the talk, we're showing up several techniques that you can use to address different set of compromises in between scaling, load balancing, data availability and durability, and service availability. The first two points could seem unrelated to the main topic, but scaling often is a simple enough way to achieve service availability... until you need to think about sharding, that is.

So the talk is all about making compromises in between them and getting to an architecture able to implement the choosen compromises. While the talk has been pretty well received, it was delivered in a 50 mins slot where we usually take a whole day or three when addressing that problems at a customer's site.

Some parts of how to get to the right architecture for the compromises that are important for you can't be fully covered in that time slot, while still being able to actually present the techniques that we're using.

I think it might be useful to extract a single use-case or two from that talk then have a full 50 mins version reduced to a single or a couple of very clear compromises and how to achieve them in details, rather than trying to present a full range of techniques and how to use them in different scenarios.

FOSDEM

After having been talking with many people, it appears that for next year's edition I should be proposing a more general talk that aims at helping developpers in other communities (python, ruby, etc) discover what's in for them in PostgreSQL. This database is full of advanced features that are really easy to use, and the only problem when preparing such a talk is choosing the right subset...

If you're running a local developper user group and are interested into learning some more about how PostgreSQL can help you in a daily basis, please do get in touch with me and let's schedule a presentation together!

Having had the opportunity to meet more people from those other development communities, I really think we should go and reach for them in their own conferences. About any PostgreSQL community member I've been talking about with about that idea seemed to agree and generally already was thinking the same thing. And most are already doing it, in fact...

I had the pleasure to run two conferences there, both in the PostgreSQL devroom.

Event Triggers

I'm currently in the middle of implementing Event Triggers for PostgreSQL and I have been for about the last 2 years. It's a quite complex feature to get right and so the patch itself is complex and large, which means the reviewing process is complex and takes time.

That also means that some parts of the design have already been redone completely at least 3 times, and that what got commited to the PostgreSQL code is nothing like what the design we decided should go in looks like. That's just a fact of life, maybe, but that makes for a very long development process.

We're now getting to the end of it though, and this talk is showing both where we want to go with Event Triggers, where we are now and what remains to be done for 9.3 if we want the feature to be any useful.

If you're interested into that development, have a look at the slide deck and possibly ask me some questions about what's not clear on the pgsql-hackers mailing list (preferably).

The other way to get summarized and clear information about Event Triggers is the wiki page by the same name: Event Triggers.

You will see that while a lot has been done (internal refactoring, adding new infrastructure and SQL level commands, and the minimum PLpgSQL support); a lot remains to be done where the code has already been submitted several times, following several designs directions given by careful review on hackers, and still we have some choices to make.

Implementing High-Availability

This talk is showing several ways to implement High Availability with PostgreSQL. The fact is that that term is overloaded already, and usually covers two very different things which are Service Availability and Data Availability.

In the talk, we're showing up several techniques that you can use to address different set of compromises in between scaling, load balancing, data availability and durability, and service availability. The first two points could seem unrelated to the main topic, but scaling often is a simple enough way to achieve service availability... until you need to think about sharding, that is.

So the talk is all about making compromises in between them and getting to an architecture able to implement the choosen compromises. While the talk has been pretty well received, it was delivered in a 50 mins slot where we usually take a whole day or three when addressing that problems at a customer's site.

Some parts of how to get to the right architecture for the compromises that are important for you can't be fully covered in that time slot, while still being able to actually present the techniques that we're using.

I think it might be useful to extract a single use-case or two from that talk then have a full 50 mins version reduced to a single or a couple of very clear compromises and how to achieve them in details, rather than trying to present a full range of techniques and how to use them in different scenarios.

FOSDEM

After having been talking with many people, it appears that for next year's edition I should be proposing a more general talk that aims at helping developpers in other communities (python, ruby, etc) discover what's in for them in PostgreSQL. This database is full of advanced features that are really easy to use, and the only problem when preparing such a talk is choosing the right subset...

If you're running a local developper user group and are interested into learning some more about how PostgreSQL can help you in a daily basis, please do get in touch with me and let's schedule a presentation together!

Turns out it's not true, because we still depend on past century technologies somehow. Not everybody will be looking at the schedule on the web using a connected mobile device (you know, you've heard of them, those tracking and surveillance devices, if you want to believe Stallman), and as the schedule gets printed on little paper sheets, it's unfortunately too late to change it now.

So it happens that I'll be speaking twice on Sunday and not at all on Friday.

And I'm presenting two talks over there that are both currently scheduled on the Sunday in the PostgreSQL devroom. We're talking about changing that though, so that one of those will in fact happen this Friday at the FOSDEM PGDay 2013, which has a different schedule, so consider watching for that.

One of those two talks is about Implementing High Availability (yes, with PostgreSQL). It's been quite well received in the places I had to chance to make it before (namely PGDay France and PG Conf Europe), and it's going to be a stripped down version of it so that it fits well in the 45 mins slot we have here.

The other talk is going to be about Event Triggers, a feature new in PostgreSQL 9.3 (due in september 2013, crossing fingers) and while the goal of that talk is to introduce what the feature is all about and a bunch of use cases that you can address by using it, it will certainly offer a peek into the PostgreSQL development cycle and community processes.

In a recent migration project where we freed data from MySQL into PostgreSQL, we used pgloader again. But the loading time was not fast enough for the service downtime window that we had here. Indeed Python is not known for being the fastest solution around. It's easy to use and to ship to production, but sometimes you not only want to be able to be efficient when writing code, you also need the code to actually run fast too.

Faster data loading

So I began writing a little dedicated tool for that migration in Common Lisp which is growing on me as my personal answer to the burning question: python 2 or python 3? I find Common Lisp to offer an even more dynamic programming environment, an easier language to use, and the result often has performances characteristics way beyond what I can get with python. Between 5 times faster and 121 times faster in some quite stupid benchmark.

Here, with real data, my one shot attempt has been running more than twice as fast as the python version, after about a day of programming.

The other thing here is that I've tempted to get pgloader work in parallel, but at the time I didn't know about the Global Interpreter Lock that they didn't find how to remove in Python 3 still, by the way. So my threading attempts at making pgloader work in parallel are pretty useless.

Whereas in Common Lisp I can just use the lparallel lib, which exposes threading facilities and some queueing facilities as a mean to communicate data in between workers, and have my code easily work in parallel for real.

Compatibility

The only drawback that I can see here is that if you've been writing your own reformating modules in python for pgloader (yes you can implement your own reformating module for pgloader), then you would have to port it to Common Lisp. Shout me an email if that's your case.

Next version

So, I think we're going to have a pgloader 3 someday, that will be way faster than the current one, and bundle some more features: real parallel behavior, ability to fetch non local data (connecting to MySQL directly, or HTTP, S3, etc); and I'm thinking about offering a COPY like syntax to drive the loading too, while at it. Also, the ability to discover the set of data to load all by itself when you want to load a whole database: think of it as a special Migration mode of operations.

Some feature requests can't be solved easily when keeping the old .INI syntax cruft, so it's high time to implement some kind of a real command language. I have several ideas about those, in between the COPY syntax and the SQL*Loader configuration format, which is both clunky and quite powerful, too.

After a beginning in TCL and a complete rewrite in python in 2005, it looks like 2013 is going to be the year of pgloader 3, in Common Lisp!

Migrating the schema

For the schema parts, I've been using mysql2pgsql with success for many years. This tool is not complete and will do only about 80% of the work. As I think that the schema should always be validated manually when doing a migration anyway, I happen to think that it's good news.

Getting the data out

Then for the data parts I keep on using pgloader. The data is never quite right, and the ability to filter out what you can't readily import in a reject file proves itself a a must have here. The problems you have in the exported MySQL data are quite serious:

First, date formating is not compatible with what PostgreSQL expects, sometimes using 20130117143218 instead of what we expect: 2013-01-17 14:32:18, and of course even when the format is right (that seems to depend on the MySQL server's version), you still have to transform the 0000-00-00 00:00:00 into NULL.

Before thinking about the usage of that particular date rather than using NULL when you don't have the information, you might want to remember that there's no year zero in the calendar, it's year 1 BC and then year 1.

Then, text encoding is often mixed up, even when the MySQL databases are said to be in latin1 or unicode, you somehow always end up finding texts in win1252 or some other code page in there.

And of course, MySQL provides no tool to export the data to CSV, so you have to come up with your own. The SELECT INTO OUTFILE command on the server produces non conforming CSV (\n can appear in non-escaped field contents), and while the mysql client manual page details that it outputs CSV when stdout is not a terminal, it won't even try to quote fields or escape \t when they appear in the data.

So, we use the mysqltocsv little script to export the data, and then use that data to feed pgloader.

Loading the data in

Now, we have to write down a configuration file for pgloader to know what to load and where to find the data. What about generating the file from the database schema instead, using the query in generate-pgloader-config.sql:

with reformat as (
   select relname, attnum, attname, typname,
          case typname
               when 'timestamptz'
               then attname || ':mynull:timestamp'
               when 'date'
               then attname || ':mynull:date'
           end as reformat
      from pg_class c
           join pg_namespace n on n.oid = c.relnamespace
           left join pg_attribute a on c.oid = a.attrelid
           join pg_type t on t.oid = a.atttypid
     where c.relkind = 'r'
           and attnum > 0
           and n.nspname = 'public'
),
 config_reformat as (
  select relname,
         '['||relname||']' || E'\n' ||
         'table = ' || relname || E' \n' ||
         'filename = /path/to/csv/' || relname || E'.csv\n' ||
         'format = csv' || E'\n' ||
         'field_sep = \t' || E'\n' ||
         'columns = *' || E' \n' ||
         'reformat = ' || array_to_string(array_agg(reformat), ', ')
         || E'\n' as config
    from reformat
   where reformat is not null
group by relname
),
 noreformat as (
   select relname, bool_and(reformat is null) as noreformating
     from reformat
 group by relname
),
 config_noreformat as (
  select relname,
         '['||relname||']' || E'\n' ||
         'table = ' || relname || E' \n' ||
         'filename = /path/to/csv/' || relname || E'.csv\n' ||
         'format = csv' || E'\n' ||
         'field_sep = \t' || E'\n' ||
         'columns = *' || E' \n'
         || E'\n' as config
    from reformat join noreformat using (relname)
   where noreformating
group by relname
),
allconfs as (
    select relname, config from config_reformat
 union all
    select relname, config from config_noreformat
)
select config
  from allconfs
 where relname not in ('tables', 'wedont', 'wantto', 'load')
 order by relname;

To work with the setup generated, you will have to prepend a global section for pgloader and to include a reformating module in python, that I named mynull.py:

# Author: Dimitri Fontaine <dimitri@2ndQuadrant.fr>
#
# pgloader mysql reformating module

def timestamp(reject, input):
    """ Reformat str as a PostgreSQL timestamp

    MySQL timestamps are ok this time:  2012-12-18 23:38:12
    But may contain the infamous all-zero date, where we want NULL.
    """
    if input == '0000-00-00 00:00:00':
        return None

    return input

def date(reject, input):
    """ date columns can also have '0000-00-00'"""
    if input == '0000-00-00':
        return None

    return input

Now you can launch pgloader and profit!

Conclusion

There are plenty of tools to assist you migrating away from MySQL and other databases. When you make that decision, you're not alone, and it's easy enough to find people to come and help you.

While MySQL is Open Source and is not a lock in from a licencing perspective, I still find it hard to swallow that there's no provided tools for getting data out in a sane format, and that so many little inconsistencies exist in the product with respect to data handling (try to have a NOT NULL column, then enjoy the default empty strings that have been put in there). So at this point, yes, I consider that moving to PostgreSQL is a way to free your data:

So, here we go with a simple Common Lisp attempt. The Lost in scope article begins with defining a very simple function returning a boolean value, only true when it's not monday.

Monday is special

Keep in mind that the following example has been choosen to be simple yet offer a case of lexical binding shadowing. It looks convoluted. Focus on the day binding.

(defparameter *days*
  '(monday tuesday wednesday thursday friday saturday sunday)
  "List of days in the week")

(defun any-day-but-monday? (day)
  "Returns a generalized boolean, true unless DAY is 'monday"
  (member day (remove-if (lambda (day) (eq day 'monday)) *days*)))

So as you can see, in Common Lisp we just get away with a list of symbols rather than a string that we split to have a list of strings, or an array of strings, as in the examples with python and ruby.

Now, the generalized boolean is either nil to mean false, or anything else to mean true, and in that example the return value of member is a sub-list that begins where the member was found:

CL-USER> (any-day-but-monday? 'monday)
NIL

CL-USER> (any-day-but-monday? 'tuesday)
(TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY)

Oh, and as we work with Common Lisp, we're having a real REPL where to play directly with our code, no need to add interactive stanzas in the main program text file just to be able to play with it. In Emacs Slime we just use C-M-x on a form to have it available in the REPL, or C-c C-l to load the whole file we're working on.

So, we see that Common Lisp scoping rules are silently doing the right thing here. Within the remove-if call we define a lambda function taking a single parameter called day. It so happens that this parameter is shadowing the any-day-but-monday? function parameter, and that shadowing only happens in the lexical scope of the lambda we are creating. For a detailed discussion about that concept, I would refer you to the Scope and Extent chapter of Common Lisp the Language, 2nd Edition.

In Common Lisp we have both lexical scope and dynamic extents, and a variable defined with defparameter or defvar or that you otherwise declare special will have a dynamic extent. Hence this section title.

Closures

Now, the lost in scope article tries some more at finding a solution around the scoping rules of the python and ruby languages, where the developer can not easily instruct the language about the scoping rules he wants to be using in a case by case way, as far as I can see.

First, let's reproduce the problem by using a single variable that we bind in all the closures. Those are called callbacks in the original article, so I've kept using that name here.

(defparameter *callbacks-all-sunday*
    (loop
       for day in *days*
       collect (lambda () day))
  "loop binds DAY only once")

In that example, there's only a single variable day that we reuse throughout the loop construct, so that when the loop ends, we have a list of closures all refering to the same variable, and this variable, by the end of the loop, has sunday as its value.

CL-USER> (mapcar #'funcall *callbacks-all-sunday*)
(SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY)

Closures, take 2

Now, the way to have what we want here, that is a list of closures each having its own variable.

(defparameter *callbacks*
  (mapcar (lambda (day)
            ;; for each day, produce a separate closure
            ;; around its own lexical variable day
            (lambda () day))
          *days*)
  "A list of callbacks to return the current day...")

And there we go:

CL-USER> (mapcar #'funcall *callbacks*)
(MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY)

Conclusion

Scoping rules are very important in any programming language, functional or not, and must be well understood by programmers. I find that once again, that topic has received a very deep thinking in Common Lisp, and the language is giving all the options to its developers.

I want to stress that in Common Lisp the scope rules are very clearly defined in the standard documentation of the language. For instance, defun and let both introduce a lexical binding, defvar and defparameter introduce a dynamic variable.

Also, as a user of the language you have the ability to declare any variable as being special in order to introduce yourself a dynamic variable. In C you can declare some variables as being static, which is something else and frown with a very different set of problems.

The solution we talked about is to use templates, and so I've been working on a patch to bring templates for extensions to PostgreSQL. As we're talking about 3 new system catalogs, that's a big patch in term of lines of code. In term of features though, it's quite an easy one.

Here's how it goes. Let's say you want to prepare the system to be able to CREATE EXTENSION pair; without having to install it as an OS package for which you would need to get root access on the server where your PostgreSQL instance is running, which is not always easy, and sometimes not a good idea.

Installing an extension template

With the template patch I just sent on the lists, what you can do is prepare a template with your extension's script and properties, then use it to install the extensions.

create template
   for extension pair default version '1.0'
  with (nosuperuser, norelocatable, schema public)
as $$
  CREATE TYPE pair AS ( k text, v text );

  CREATE OR REPLACE FUNCTION pair(anyelement, text)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

  CREATE OR REPLACE FUNCTION pair(text, anyelement)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

  CREATE OR REPLACE FUNCTION pair(anyelement, anyelement)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

  CREATE OR REPLACE FUNCTION pair(text, text)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair;';
$$;

Installing an extension from a template

With the template installed in the catalogs, now you can go and install your extension:

foo> create extension pair;
CREATE EXTENSION

foo> \dx pair
     List of installed extensions
 Name | Version | Schema | Description
------+---------+--------+-------------
 pair | 1.0     | public |
(1 row)

foo> \dx+ pair
     Objects in extension "pair"
          Object Description
--------------------------------------
 function pair(anyelement,anyelement)
 function pair(anyelement,text)
 function pair(text,anyelement)
 function pair(text,text)
 type pair
(5 rows)

The extension installation is now happening from the catalog templates rather than the file system, which means you didn't need to be root on the system where the server is running. Also note that this example above did happen when connected as the database owner, a user who is not the superuser. Requiring less privileges is always good news, right?

Managing upgrade scripts and extension update

Now that the extension is installed, you might want to update it with some new awesome features. Let's have a look at that.

Rather than make a new version of the extension package with the new files in there, then asking the operations team to make the new package available on the internal repositories then install them on the servers, you could now prepare and QA the new setup that way:

create template for extension pair from '1.0' to '1.1'
as $$
  CREATE OPERATOR ~> (LEFTARG = text,
                      RIGHTARG = anyelement,
                      PROCEDURE = pair);

  CREATE OPERATOR ~> (LEFTARG = anyelement,
                      RIGHTARG = text,
                      PROCEDURE = pair);

  CREATE OPERATOR ~> (LEFTARG = anyelement,
                      RIGHTARG = anyelement,
                      PROCEDURE = pair);

  CREATE OPERATOR ~> (LEFTARG = text,
                      RIGHTARG = text,
                      PROCEDURE = pair);
$$;

create template
   for extension pair from '1.1' to '1.2'
as $$
            comment on extension pair is 'Simple Key Value Text Type';
$$;

Of course it's not the most realistic example when you look at the content. In particular the 1.2 version that only adds a comment to the extension. I needed another version to test the automatic upgrade path with more than one step though, so here we go.

foo> alter extension pair update to '1.2';
ALTER EXTENSION

foo> \dx pair
             List of installed extensions
 Name | Version | Schema |        Description
------+---------+--------+----------------------------
 pair | 1.2     | public | Simple Key Value Text Type
(1 row)

foo> \dx+ pair
     Objects in extension "pair"
          Object Description
--------------------------------------
 function pair(anyelement,anyelement)
 function pair(anyelement,text)
 function pair(text,anyelement)
 function pair(text,text)
 operator ~>(anyelement,anyelement)
 operator ~>(anyelement,text)
 operator ~>(text,anyelement)
 operator ~>(text,text)
 type pair
(9 rows)

We did it!

Internals

Let's have a look at those new catalogs:

Here we go now:

foo> select * from pg_extension_control;
select * from pg_extension_control;
-[ RECORD 1 ]--+-------
ctlname        | pair
ctlowner       | 32926
ctldefault     | t
ctlrelocatable | f
ctlsuperuser   | f
ctlnamespace   | public
ctlversion     | 1.0
ctlrequires    |

foo> select * from pg_extension_template;
select * from pg_extension_template;
-[ RECORD 1 ]-----------------------------------------------------------------
tplname    | pair
tplowner   | 32926
tplversion | 1.0
tplscript  |
           |   CREATE TYPE pair AS ( k text, v text );
           |
           |   CREATE OR REPLACE FUNCTION pair(anyelement, text)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';
           |
           |   CREATE OR REPLACE FUNCTION pair(text, anyelement)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';
           |
           |   CREATE OR REPLACE FUNCTION pair(anyelement, anyelement)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';
           |
           |   CREATE OR REPLACE FUNCTION pair(text, text)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair;';
           |

foo> select * from pg_extension_uptmpl;
select * from pg_extension_uptmpl;
-[ RECORD 1 ]-------------------------------------------------------------
uptname   | pair
uptowner  | 32926
uptfrom   | 1.0
uptto     | 1.1
uptscript |
          |   CREATE OPERATOR ~> (LEFTARG = text,
          |                       RIGHTARG = anyelement,
          |                       PROCEDURE = pair);
          |
          |   CREATE OPERATOR ~> (LEFTARG = anyelement,
          |                       RIGHTARG = text,
          |                       PROCEDURE = pair);
          |
          |   CREATE OPERATOR ~> (LEFTARG = anyelement,
          |                       RIGHTARG = anyelement,
          |                       PROCEDURE = pair);
          |
          |   CREATE OPERATOR ~> (LEFTARG = text,
          |                       RIGHTARG = text,
          |                       PROCEDURE = pair);
          |
-[ RECORD 2 ]-------------------------------------------------------------
uptname   | pair
uptowner  | 32926
uptfrom   | 1.1
uptto     | 1.2
uptscript |
          |     comment on extension pair is 'Simple Key Value Text Type';
          |

As you can see there's nothing too complex here, it's quite straightforward. We need to separate away the creating templates from the updating templates because we need unique keys and we can't have that on NULL columns.

foo> \d pg_extension_template
\d pg_extension_template
Table "pg_catalog.pg_extension_template"
   Column   | Type | Modifiers
------------+------+-----------
 tplname    | name | not null
 tplowner   | oid  | not null
 tplversion | text |
 tplscript  | text |
Indexes:
    "pg_extension_template_name_version_index" UNIQUE, btree (tplname, tplversion)
    "pg_extension_template_oid_index" UNIQUE, btree (oid)

foo> \d pg_extension_uptmpl
\d pg_extension_uptmpl
Table "pg_catalog.pg_extension_uptmpl"
  Column   | Type | Modifiers
-----------+------+-----------
 uptname   | name | not null
 uptowner  | oid  | not null
 uptfrom   | text |
 uptto     | text |
 uptscript | text |
Indexes:
    "pg_extension_uptmpl_name_from_to_index" UNIQUE, btree (uptname, uptfrom, uptto)
    "pg_extension_uptmpl_oid_index" UNIQUE, btree (oid)

Next steps

Now that we have the basics in place, the patch is far from finished still. It needs pg_dump and psql support, support for the function pg_available_extension_versions(), implementing some ALTER TEMPLATE FOR EXTENSION commands for which I only sketched the syntax in the grammar, and some more infrastructure to be able to have ALTER OWNER and ALTER RENAME commands.

All that is pretty technical though, the real thing that patch needs is some quality review and maybe some adjustments. I would be surprised if it didn't need adjustments, really. Because the way the community works, we always need some. That's why the PostgreSQL product is so good!

Now that we have the Extension facility though, what we see is a growing number of users taking advantage of it for the purpose of managing in house procedural code and related objects. This code can be a bunch of PLpgSQL or plpython functions and as such you normaly create them directly from any application connection to PostgreSQL.

So the idea would be to allow creating Extensions fully from a SQL command, including the whole set of objects it contains. More than one approach are possible to reach that goal, each with downsides and advantages. We will see them later in that document.

Before that though, let's first review what the extension mechanism has to offer to its users when there's no contrib like module to manage.

A use case for next generation extensions

The only design goal of the 9.1 PostgreSQL Extension feature has been to support a proper dump & restore user experience when using contrib modules such as hstore or ltree. Building up on that, what do Extensions have to offer to non C developpers out there? In other words, what CREATE EXTENSION brings on the table that a bunch of loose objects does not? What problems can we now solve?

A way to phrase it is to say that Extensions are user defined CASCADE support. Extensions brings extensibility to the pg_depend PostgreSQL internal dependency tracking system that CASCADE is built on. From that angle, Extensions are a way to manage dependencies of SQL objects in a way that allow you to manage them as a single entity.

One of the existing problems this helps solving is the infamous lack of dependency tracking between function calls. Using Extensions when you deal with a set of functions acting as an API, you can at least protect that as a unit:

STATEMENT: drop function public.populate_record(anyelement,hstore);
    ERROR: cannot drop function populate_record(anyelement,hstore) because
           extension hstore requires it
     HINT: You can drop extension hstore instead.

And you also have a version number and tools integration to manage extensions, with psql \dx command and the equivalent feature in pgAdmin. Coming with your own version number management is not impossible, some do that already. Here it's integrated and the upgrade sequences are offered too (applying 1.1--1.2 then 1.2--1.3 automatically).

Let's just say that it's very easy to understand the traction our users feel towards leveraging Extensions features in order to properly manage their set of stored procedures and SQL objects.

The dump & restore experience

The common problem of all those proposals is very central to the whole idea of Extensions as we know them. The goal of building them as been to fix the restoring experience when using extensions in a database, and we managed to do that properly for contrib likes extensions.

When talking about Inline Extensions, the fly in the ointment is how to properly manage their pg_dump behavior. The principle we built for Extensions and that is almost unique to them is to omit them in the dump files. The only other objects that we filter out of the dump are the one installed at server initialisation times, when using initdb, to be found in the pg_catalog and information_schema systems' schema.

At restore time, the dump file contains the CREATE EXTENSION command so the PostgreSQL server will go fetch the control and script files on disk and process them, loading the database with the right set of SQL objects.

Now we're talking about Extensions which we would maybe want to dump the objects of, so that at restore time we don't need to find them from unknown external resources: the fact that the extension is Inline means that the PostgreSQL server has no way to know where its content is coming from.

The next proposals are trying to address that problem, with more or less success. So far none of them is entirely sastisfying to me, even if a clear temporary winner as emerged on the hackers mailing list, summarized in the in-catalog Extension Scripts and Control parameters (templates?) thread.

Inline Extension Proposals

Now, on to some proposals to make the best out of our all time favorite PostgreSQL feature, the only one that makes no sense at all by itself...

Starting from an empty extension

We already have the facility to add existing loose objects to an extension, and that's exactly what we use when we create an extension for the first time when it used not to be an extension before, with the CREATE EXTENSION ... FROM 'unpackaged'; command.

The hstore--unpackaged--1.0.sql file contains statements such as:

ALTER EXTENSION hstore ADD type hstore;
ALTER EXTENSION hstore ADD function hstore_in(cstring);
ALTER EXTENSION hstore ADD function hstore_out(hstore);
ALTER EXTENSION hstore ADD function hstore_recv(internal);
ALTER EXTENSION hstore ADD function hstore_send(hstore);

Opening CREATE EXTENSION so that it allows you to create a really empty extension would then allow you to fill-in as you need, with as many commands as you want to add objects to it. The control file properties would need to find their way in that design, that sure can be taken care of.

The main drawback here is that there's no separation anymore in between the extension author, the distribution means, the DBA and the database user. When you want to install a third party Extension using only SQL commands, you could do it with that scheme by using a big script full of one-liners commands.

So that if you screw up your copy/pasting session (well you should maybe reconsider your choice of tooling at this point, but that's another topic), you will end up with a perfectly valid Extension that does not contain what you wanted. As the end user, you have no clue about that until the first time using the extension fails.

CREATE EXTENSION AS

The next idea is to embed the Extension script itself in the command, so as to to get a cleaner command API (in my opinion at least) and a better error message when the paste is wrong. Of course it your paste problem happens to just be loosing a line in the middle of the script there is not so much I can do for you...

CREATE EXTENSION hstore
  WITH parameter = value, ...
AS $$
CREATE TYPE hstore;

CREATE FUNCTION hstore_in(cstring) RETURNS hstore
 AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_out(hstore) RETURNS cstring
AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_recv(internal) RETURNS hstore
AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_send(hstore) RETURNS bytea
AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE TYPE hstore (
        INTERNALLENGTH = -1, STORAGE = extended
        INPUT = hstore_in, OUTPUT = hstore_out,
        RECEIVE = hstore_recv, SEND = hstore_send);
$$;

I've actually proposed a patch to implement that, as you can see in the pg_dump --extension-script commit fest entry. As spoiled by the commit fest entry title, the main problem we have with Inline Extensions is their management in the seamless experience of dump & restore that we are so happy to have now. More about that later, though.

Extension Templates

Another idea is to continue working from control parameters and scripts to install and update extensions, but to have two different places where to find those. Either on the server's File System (when dealing with contribs and shared libraries, there's but a choice), or on the system catalogs.

The idea would then be to have some new specific TEMPLATE SQL Object that would be used to import or upload your control file and create and update scripts in the database, using nothing else than a SQL connection. Then at CREATE EXTENSION time the system would be able to work either from the file system or the template catalogs.

One obvious problem is how to deal with a unique namespace when we split the sources into the file system and the database, and when the file system is typically maintained by using apt-get or yum commands.

Then again I would actually prefer that mechanism better than the other proposals if the idea was to load the file system control and scripts files as TEMPLATEs themselves and then only operate Extensions from Templates. But doing that would mean getting back to the situation where we still are not able to devise a good, simple and robust pg_dump policy for extensions and templates.

Conclusion

I hope to be finding the right solution to my long term plan in this release development cycle, but it looks like the right challenge to address now is to find the right compromise instead. Using the Templates idea already brings a lot on the table, if not the whole set of features I would like to see.

What would be missing mainly would be the ability for an Extension to switch from being file based to being a template, either because the author decided to change the way he's shipping it, or because the user is switching from using the pgxn client to using proper system packages. I guess that's something we can see about later, though.

Another way to ask that question is saying that

So here it is, M-x ack:

;;; dim-ack.el --- Dimitri Fontaine
;;
;; http://stackoverflow.com/questions/2322389/ack-does-not-work-when-run-from-grep-find-in-emacs-on-windows

(defcustom ack-command (or (executable-find "ack")
                           (executable-find "ack-grep"))
  "Command to use to call ack, e.g. ack-grep under debian"
  :type 'file)

(defvar ack-command-line (concat ack-command " --nogroup --nocolor "))
(defvar ack-history nil)
(defvar ack-host-defaults-alist nil)

(defun ack ()
  "Like grep, but using ack-command as the default"
  (interactive)
  ; Make sure grep has been initialized
  (if (>= emacs-major-version 22)
      (require 'grep)
    (require 'compile))
  ; Close STDIN to keep ack from going into filter mode
  (let ((null-device (format "< %s" null-device))
        (grep-command ack-command-line)
        (grep-history ack-history)
        (grep-host-defaults-alist ack-host-defaults-alist))
    (call-interactively 'grep)
    (setq ack-history             grep-history
          ack-host-defaults-alist grep-host-defaults-alist)))

(provide 'dim-ack)

Enjoy!

Today I'm back on that topic and as I'm toying with Common Lisp I though it would be a good excuse to learn me some new tricks. As you can see from the earlier blog entry, last time I did attack the digits problem quite lightly. Let's try a better approach now.

(defun digits (n)
  "return the list of the digits of N"
  (nreverse
   (loop for x = n then r
      for (r d) = (multiple-value-list (truncate x 10))
      collect d
      until (zerop r))))

As you can see I wanted to use that facility I like very much, the for x = n then r way to handle first loop iteration differently from the next ones. But I've been hinted on #lisp that there's a much better way to write same code:

(defun integer-digits (integer)
  "stassats version"
  (nreverse
   (loop with remainder
      do (setf (values integer remainder) (truncate integer 10))
      collect remainder
      until (zerop integer))))

That code runs about twice as fast as the previous one and is easier to reason about. It's using setf and the form setf values, something nice to discover as it seems to be quite powerful. Let's see how to use it, even if it's really simple:

CL-USER> (integer-digits 12304501)
(1 2 3 0 4 5 0 1)

Let's move on to solving the Happy Numbers problem though:

(defun sum-of-squares-of-digits (integer)
  (loop with remainder
     do (setf (values integer remainder) (truncate integer 10))
     sum (* remainder remainder)
     until (zerop integer)))

(defun happy? (n &optional seen)
  "return true when n is a happy number"
  (let* ((happiness (sum-of-squares-of-digits n)))
    (cond ((eq 1 happiness)      t)
          ((memq happiness seen) nil)
          (t
           (happy? happiness (push happiness seen))))))

(defun find-happy-numbers (limit)
  "find all happy numbers from 1 to limit"
  (loop for n from 1 to limit when (happy? n) collect n))

And here's how it goes:

CL-USER> (find-happy-numbers 100)
(1 7 10 13 19 23 28 31 32 44 49 68 70 79 82 86 91 94 97 100)

CL-USER> (time (length (find-happy-numbers 1000000)))
(LENGTH (FIND-HAPPY-NUMBERS 1000000))
took 1,621,413 microseconds (1.621413 seconds) to run.
       116,474 microseconds (0.116474 seconds, 7.18%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     1,431,332 microseconds (1.431332 seconds) were spent in user mode
       145,941 microseconds (0.145941 seconds) were spent in system mode
 185,438,208 bytes of memory allocated.
 1 minor page faults, 0 major page faults, 0 swaps.
143071

Of course that code is much faster than the one I wrote before both in SQL and Emacs Lisp, the reason being that instead of writing the number into a string with (format t "~d" number) then subseq to get them one after the other, we're now using truncate.

Happy hacking!

Update

It turns out that to solve math related problem, some maths hindsight is helping. Who would have believed that? So if you want to easily get some more performances out of the previous code, just try that solution:

(defvar *depressed-squares* '(0 4 16 20 37 42 58 89 145)
  "see http://oeis.org/A039943")

(defun undepressed? (n)
  "same as happy?, using a static list of unhappy sums"
  (cond ((eq 1 n) t)
        ((member n *depressed-squares*) nil)
        (t
         (let ((h (sum-of-squares-of-digits n)))
           (undepressed? h)))))

(defun find-undepressed-numbers (limit)
  "find all happy numbers from 1 to limit"
  (loop for n from 1 to limit when (undepressed? n) collect n))

Time to compare:

CL-USER> (time (length (find-happy-numbers 1000000)))
(LENGTH (FIND-HAPPY-NUMBERS 1000000))
took 1,938,048 microseconds (1.938048 seconds) to run.
       290,902 microseconds (0.290902 seconds, 15.01%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     1,778,021 microseconds (1.778021 seconds) were spent in user mode
       140,862 microseconds (0.140862 seconds) were spent in system mode
 185,438,208 bytes of memory allocated.
 3,320 minor page faults, 0 major page faults, 0 swaps.
143071

CL-USER> (time (length (find-undepressed-numbers 1000000)))
(LENGTH (FIND-UNDEPRESSED-NUMBERS 1000000))
took 1,036,847 microseconds (1.036847 seconds) to run.
         5,372 microseconds (0.005372 seconds, 0.52%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     1,018,708 microseconds (1.018708 seconds) were spent in user mode
        16,982 microseconds (0.016982 seconds) were spent in system mode
 2,289,152 bytes of memory allocated.
143071
CL-USER>

Let me tell you upfront that I really dislike the whole idea of the vim golf challenge. I've been a user of both Emacs and Vim for many years, and finally decided to switch to living in Emacs; or if you prefer, climbing my way up from level 2 as in The Levels Of Emacs Proficiency. The reason why is that I found that in my case, using Vim would mean spending more time thinking about how to do some editing operation rather than the problem I wanted to solve by editing some text, most often code.

Of course, the main effect of Vim Golf is to make you focus even more on the wrong problem. There's still a good side of it though, which is that such challenges are good excuses to discover new features of your editor. So let's use that excuse to talk about some nice Emacs features.

The challenge

The previous image will lead you to a particular challenge where it's all about filling in an array with consecutive hexadecimal numbers written as 0xab, where you begin with a template containing only the 0x00 entry. The idea is of course to use the Vim feature that will increment the number at point, and is available through the C-a keystroke.

unsigned int hex[] = {
        0x00,
};

A first solution

Emacs does not ship with an increment-number-at-point, much less so with one that would support decimal, octal and hexadecimal and even automatically recognize 0x as a prefix meaning that the next number is hexadecimal. But Emacs ship with Emacs Keyboard Macros and those have a counter, so it's easy enough to fill in numbers from 1 to 255 that way: M-1 F3 F3 , F4 will register a macro where the counter starts at 1, and each time you hit F4 it will insert the current counter value, increment it and insert a coma. You want to do that 254 times, so you do C-u 2 5 4 F4 and Emacs just does that.

Now, to transform those decimal numbers into their hexadecimal representation, you can use advanced Emacs Regexp Replace features. Replace [0-9]+ with the result from the following Emacs Lisp code: \,(format "0x%02x" (string-to-number \&)). The \& in there will be replaced by the matching text, so that will do what we need here, turning 10 into 0x0a.

Let's get some better tools

We could do better, though. I happen to already use a key chord to duplicate the current line, and we would need a function to Increment Number At Point. Those I found over at EmacsWiki were not to my taste as they were not able to figure out easily which base to use. So here's a little Emacs Lisp example showing how to extend your favorite editor to have some Vim common features, which is why Emacs ships with Emacs Lisp in the first place.

(defun duplicate-current-line (&optional n)
  "duplicate current line, make more than 1 copy given a numeric argument"
  (interactive "p")
  (let ((nb (or n 1))
        (current-line (thing-at-point 'line)))
    (save-excursion
      ;; when on last line, insert a newline first
      (when (or (= 1 (forward-line 1)) (eq (point) (point-max)))
        (insert "\n"))

      ;; now insert as many time as requested
      (while (> n 0)
        (insert current-line)
        (decf n)))
    ;; now move down as many lines as we inserted
    (next-line nb)))

(global-set-key (kbd "C-S-d") 'duplicate-current-line)

(require 'cl)  ; destructuring-bind is found there

(defun dim:increment-number-at-point (&optional prefix)
  (interactive "p")
  (let* ((beg    (skip-chars-backward "0-9a-fA-F"))
         (hexa   (save-excursion (forward-char -2) (looking-at-p "0x")))
         ;; force the prefix to hexa (4) we see "0x" before the number
         (prefix (if hexa 4 prefix))
         (end    (re-search-forward "[0-9a-fA-F]+" nil t))
         (nstr   (match-string 0))
         (l      (- (match-end 0) (match-beginning 0)))
         (fmt    (format "%%0%d" l)))
    (message "PLOP: %d" prefix)
    (destructuring-bind (base format)
        (case prefix
          ((1)  '(10 "d"))              ; no command prefix, decimal
          ((4)  '(16 "x"))              ; C-u, hexadecimal
          ((16) '(8 "o")))              ; C-u C-u, octal
      (let* ((n   (string-to-number nstr base))
             (n+1 (+ n 1))
             (fmt (format "%s%s" fmt format)))
        (replace-match (format fmt n+1))))))

(global-set-key (kbd "C-c +") 'dim:increment-number-at-point)

So if you're using Emacs a lot but always found an excuse not to grasp Emacs Lisp, I hope that article could be an excuse for you to do so…

Another solution

Anyway, now that we are much better equipped, we can picture a better way to solve the problem. Instead of using a macro that inserts the next counter value, we can use one that duplicate current line, increment number at point (and figures out on its own that the number prefixed with 0x is hexadecimal), and does that 254 times more. Then it's all about reformatting the text so that if fits nicely on screen, and for that the command M-q runs the command fill-paragraph is exactly what we need. The command C-x f runs the command set-fill-column can be used to set the maximum column we allow Emacs to reach before going to the next line.

Our Golf then becomes a 19 steps solution if you start with the cursor at the ',' in the previous example:

C-x f 5 6 RET
F3 C-S-d C-c + F4
C-u 2 5 4 F4
C-SPC M-< C-n M-q

First, set the fill column, then register a macro (in between F3 and F4) that will duplicate current line (using C-S-d) then increment number at point (using C-c +). Third line, replay that macro 254 times (C-u 2 5 4 F4). Fourth line, select all those hexadecimal numbers and fill the paragraph they form correctly, so as to get:

All those tips for...

unsigned int hex[] = {
        0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
        0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
        0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
        0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
        0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27,
        0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,
        0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37,
        0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f,
        0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47,
        0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f,
        0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57,
        0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f,
        0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67,
        0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f,
        0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77,
        0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f,
        0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87,
        0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f,
        0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97,
        0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f,
        0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7,
        0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf,
        0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7,
        0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf,
        0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
        0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf,
        0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7,
        0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf,
        0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7,
        0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef,
        0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7,
        0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff,
};

Conclusion

Thanks to the excuse of that challenge we now have a generic facility to increment a number at point in different common bases, which is a nice building block for all kinds of Emacs Keyboard Macros. We also now have a function to duplicate the line at point, which is something I use very often myself.

More importantly, we've been refreshing our memory on how to use some advanced replacement facilities wherein you can actually use inline Emacs Lisp code as a replacement pattern, and for the most interested readers here we have a good excuse to learn some more about Emacs Lisp programming.

The main thing I want to say is that using Emacs Keyboard Macros is an interactive process: you don't have to pause your current activity to write some code in another language (here, that would be either Vim Script or Emacs Lisp) just to save a few minutes on a boring task.

How effective your are at solving that challenge, for me, is not at all about measure how many keystrokes you ended up using, it's all about being able to get some precious help from your working tools without having to stop focusing on the main problem you are solving.

I wouldn't ever get to write such Emacs Lisp code when doing that kind of editing once. I would only do that when I'm thinking I've just been doing a boring task by hand one time too many already. Like for example copying and pasting the pg_backend_pid() of the PostgreSQL backend I'm working with at the psql prompt so that I can attach gdb to it. I'll get back to talking about pgdevenv-el later!

Hope you did enjoy that article, whose goal is to help you while you're journeying in The Levels Of Emacs Proficiency.

Update

While looking at the docs for the Keyboard Macro Counter to check how to reset it without having to record the macro again, I just stumbled on this part of the docs: C-x C-k C-f runs the command kmacro-set-format. So another way to solve our problem with only facilities that come with a bare Emacs is the following:

C-x f 5 6 RET
C-x C-k C-f 0x%02x RET
C-1 F3 SPC F3 , F4
C-u 2 5 4 F4
DEL C-SPC C-a C-q

We're now at 30 keystrokes, so much more than previously, but it's stock Emacs features and that kmacro-set-format is a wonderful little tool you might as well have a need for in the future.

One of the things where Emacs really shines is that interactive development environment you get when working on some Emacs Lisp code. Evaluating an function as easy as a single key chord, and that will both compile in the function and load it in the running process. I can't tell you how many times I've been missing that ability when editing C code.

With PostgreSQL too we get a pretty interactive environment with the psql console application, or with pgAdmin. One feature from pgAdmin that I've often wished I had in psql is the ability to edit my query online and easily run it in the console, rather than either using the readline limited history editing features or launching a new editor process each time with \e. At the same time I would much prefer using my usual Emacs editor to actually edit the query.

If you've been reading that blog before you know what to expect. My solution to the stated problem is available in pgdevenv-el, an Emacs package aimed at helping PostgreSQL developers. Most of the features in there are geared toward the core backend developers, except for this one I want to talk about today (I'll blog about the other ones too I guess).

What you can see from that screenshot is that the selected query text has been sent to the psql buffer and exectuted over there. And that the psql buffer is echoing all queries sent to it. What you can not see straight from that picture is the interaction to get there. Well, I've been implementing some elisp features that I was missing.

First, movement: you can do C-M-a and C-M-e to navigate to the beginning and the end of the SQL query at point, like you do in C or in lisp in Emacs.

Then, selection: you can do C-M-h to select the SQL query at point, you don't have to navigate yourself, pgdev-sql-mode knows how to do that. Side note, pgdev-sql-mode is the name of the minor mode you need to activate in your SQL buffers to have the magic available.

Last but not least, evaluation: as when editing lisp code, you can now use C-M-x to send the current query text to an associated psql buffer.

The way to associate the psql buffer to an SQL buffer is currently done thanks to the other pgdevenv-el features that this blog post is not talking about, and the setup is addressed in the documentation: you have to let know pgdevenv-el where your PostgreSQL branches are installed locally so that it can prepare you a Shell buffer with PGDATA and PGPORT already set for you. And currently, for C-M-x to work you need to open the buffer yourself before hand, using C-c - n (to run the command pgdev-open-shell), and type psql in the Shell prompt.

What that means for me is that I can at least edit SQL (in PostgreSQL regression files and other places) in my usual Emacs buffer and actually refine it as I go until it does exactly what I need, without having to use the readline history editing or the \e command, which is not great when your Shell is in already running inside Emacs.

To quote the article:

The first thing I always do when playing around with a new software platform is to write a concurrent "Hello World" program. The program works as follows: One active entity (e.g. thread, Erlang process, Goroutine) has to print "Hello " and another one "World!\n" with the two active entities synchronizing with each other so that the output always is "Hello World!\n".

Here's my try in Common Lisp using lparallel and some local nicknames, the whole 23 lines of it:

(defun say-hello (helloq worldq n)
  (dotimes (i n)
    (format t "Hello ")
    (lq:push-queue :say-world worldq)
    (lq:pop-queue helloq))
  (lq:push-queue :quit worldq))

(defun say-world (helloq worldq)
  (when (eq (lq:pop-queue worldq) :say-world)
    (format t "World!~%")
    (lq:push-queue :say-hello helloq)
    (say-world helloq worldq)))

(defun hello-world (n)
  (let* ((lp:*kernel*  (lp:make-kernel 2)) ; a new one each time, as we end it
         (channel      (lp:make-channel))
         (helloq       (lq:make-queue))
         (worldq       (lq:make-queue)))
    (lp:submit-task channel #'say-world helloq worldq)
    (lp:submit-task channel #'say-hello helloq worldq n)
    (lp:receive-result channel)
    (lp:receive-result channel)
    (lp:end-kernel)))

If you want to play locally with that code, I've been updating it to a github project named go-hello-world, even if it's coded in CL. See the package.lisp in there for how I did enable the local nicknames lp and lq for the lparallel packages.

Beware of the REPL

In a previous version of this very article, I said that sometimes I get an extra line feed in the output and I didn't understand why. Some great Common Lisp folks did hint me about that: it's the REPL output that get intermingled with the program output, and that's because the hello-world main function was returning before the thing is over.

I've added a receive-result call in it per worker so that it waits until the end of the program before returning to the REPL, and that indeed fixes it. A way to assert that is using the time macro, which was always intermingled with the output before. It's fixed now:

CL-USER> (time (go-hello-world:hello-world 1000))
Hello World!
...
Hello World!
(GO-HELLO-WORLD:HELLO-WORLD 1000)
took 27,886 microseconds (0.027886 seconds) to run.
      1,593 microseconds (0.001593 seconds, 5.71%) of which was spent in GC.
During that period, and with 4 available CPU cores,
     23,246 microseconds (0.023246 seconds) were spent in user mode
     14,427 microseconds (0.014427 seconds) were spent in system mode
 4,272 bytes of memory allocated.
 10 minor page faults, 0 major page faults, 0 swaps.
(#<PROCESS lparallel kernel shutdown manager(62) [Reset] #x30200109F65D> ...)
CL-USER>

Conclusion

While Go language seems to bring very interesting things on the table, such as better compilation units and tools, I still think that the concurrency primitives at the core of it are easy to find in other places. Which is a good thing, as it means we know they work.

That also means that we don't need to accept Go syntax as the only way to properly solve that concurrency problem, I much prefer doing so with Common Lisp (lack of?) syntax myself.

Update

A previous version of this article was finished and published too quickly, and the conclusion was made from a buggy version of the program. It's all fixed now. Thanks a lot to people who contributed comments so that I could fix it, and thanks again to James M. Lawrence for lparallel!

That slide deck contains mainly SQL language, but some french too, rather than english. Sorry for the inconvenience if that's not something you can read. Get me to talk at an english developer friendly conference and I'll translate it for you! :)

The aim of that talk is to have people think about SQL as a real asset in their development tool set. SQL really should get compared to your application development language rather than your UI formating language, it's more like PHP or Python than it is like HTML.

So the whole talk is about showing off some advanced SQL features, all provided by default in released PostgreSQL versions. The main parts of the talk all come from an article in this blog: Reset Counter.

I did have the chance to speak several times at that event, and you can get the slides at my Conferences page that I try to keep up to date. I did one talk about Implementing High Availability that was about 2 hours long (a double slot), PGQ Cooperative Consumers that Marko Kreen copresented with me and the Large Scale MySQL Migration to PostgreSQL that I already presented before this year.

Next conference is in Lyon and will be in French, the talk is called Présentation de PostgreSQL. The audience is going to be composed of PHP developers interested to know more about PostgreSQL, I'll tell you how it goes!

prefix_range release 1.2.0

I'm sorry it took that long. It's now done, you can have prefix 1.2.0 from https://github.com/dimitri/prefix or if you want a tagged tarball then you can use this link: https://github.com/dimitri/prefix/tarball/v1.2.0.

The changelog is all about fixing an index search bug and updating the package to primarily be an extension for PostgreSQL 9.1 and 9.2. Of course older Major Versions are still supported (all of them since 8.1, but please first consider upgrading PostgreSQL) if you want to install it manually, using the prefix--1.2.0.sql file.

debian package

And thanks to Christoph Berg the debian package is already validated and has reached debian experimental. We don't target sid these days because debian is preparing a new stable release, so there's a freeze. I think. Anyway, take your prefix package from here if you need it: http://packages.debian.org/experimental/postgresql-9.1-prefix.

Range Types

If you step back a little there's an interesting question to answer here. Why isn't prefix_range and PostgreSQL Range Type? Given the names it seems like a pretty good candidate.

Well the thing is that to make a generic range type you need to have a total ordering on the range elements, and a distance function that tells you how far any two elements of a range are one from each other.

When talking about prefixes, I don't see how to do that. The prefix range ['abcd', 'abce') contains an infinity of elements, all the strings that begin with the letters abcd. I guess that coming with an ordering on text is possible, but what if any text element represents a prefix?

I mean that in our case, the elements would be of type prefix, and 'abcd' is a prefix of 'abcdefg'. The question I want to answer is that given a table with prefixes 'abcd', 'abce' and 'abcde' which row in there has the longest prefix matching the literal 'abcdef'.

I'm not seeing how to abuse the Range Types mechanism to implement that, so if you have some ideas please share them!

What's to solve

Say we store in a table entries from a counter that only increases and the time stamp when we did the measurement. So that when you read 30 then later 40 in fact that means we counted 10 more the second reading when compared to the first, in other words the first 30 are counted again in the second counter value, 40.

Now of course it's a real world counter. Think network traffic counter on a network interface, if you want something real to play with in your mind. So the counter will sometime reset and you will read measure sequences such as 40, 0, 20 if you happen to read just when the counter is reset, or most of the time that will look like 45, 25, 50.

The question we want to answer is, given a series of that counter measures, including some resets, what is the current logical value of the counter?

Given the sequence of measures 0, 10, 20, 30, 40, 0, 20, 30, 60 the result we want is 40 + 60, that is 100. Right?

Playing with some data

Let's model an hypothetical dataset easy enough to play with. What about just the previous example? We also need to time stamp the measurements, let's just use a tick for now, as it's easier to think about:

create table measures(tick int, nb int);

insert into measures
     values (1, 0), (2, 10), (3, 20), (4, 30), (5, 40),
            (6, 0), (7, 20), (8, 30), (9, 60);

Now that we have some data in a table to play with, let's try to find out the numbers we are interested in: we only want to keep the latest measure we read on the counter just before it wraps. That means values where the next one (in tick or time stamping order) is lesser than the current counter value.

As we are lucky enough to be playing with the awesome PostgreSQL which brings window functions on the table, we can easily implement just what we said in a readable way:

  select tick, nb,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
              else null
          end as max
    from measures
  window w as (order by tick);

The firt case is the exact translation of the problem as spelled in english in just the previous paragraph where we stated we want to keep the current counter value in case of a wraparound, so I guess it's easy enough to get at.

Then we have a couple of tricks in that query in order to massage the data as we want it. First, the last row of the output won't have a lead, that window function call is going to return NULL. In that case, we keep the current counter value as if we just did a wraparound. And finally, when there's no wraparound, we don't care about the data. Well, for the purpose of knowing the current logical value of the counter, that is.

And we get that encouraging result:

 tick | nb  | max
------+-----+-----
    1 |   0 |
    2 |  10 |
    3 |  20 |
    4 |  30 |
    5 |  40 |  40
    6 |   0 |
    7 |  20 |
    8 |  30 |
    9 |  60 |  60

As you see, we have been able to create a new column out of the dataset, and that new column only contains the data we are interested into.

Finding the current counter value

All we have to do now is sum this computed columns entries. Remember that the sum() aggregate function will simply discard nulls, so that we don't have to turn them into a bunch of 0.

with t(tops) as (
  select case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
              else null
          end as max
    from measures
  window w as (order by tick)
)
select sum(tops) from t;

And here's the expected result:

 sum
-----
 100

Now what about testing with another set of data or two, just to be sure that the counter is allowed to wrap more than once within our solution?

insert into measures
     values (10, 0), (11, 10), (12, 30), (13, 35), (14, 45),
            (15, 25), (16, 50), (17, 100), (18, 110);

Then we have:

with t(tops) as (
   select case when lead(nb) over w < nb then nb
               when lead(nb) over w is null then nb
               else null
           end as max
     from measures
   window w as (order by tick)
)
select sum(tops) from t;
 sum
-----
 255
(1 row)

All good!

Counter logical value over a given period

Now of course what we want is to find the logical value of the counter for a given day's or month's worth of measures. We then need to pay attention to the value of the counter at the start of our period so that we know to substract it from the logical sum over the period.

Here's an SQL version of the same sentence, applied to the period in between ticks 4 and 14, in a completely arbitrary choosing of mine:

with t as (
  select tick,
         first_value(nb) over w as first,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
              else null
          end as max
    from measures
   where tick >= 4 and tick < 14
  window w as (order by tick)
)
select sum(max) - min(first) as sum
 from t;

Here we are using the first_value() window function to retain it in the whole resultset of the Common Table Expression (the inner query introduced by the keyword WITH is called that way). And when doing the sum we're interested in at the outer level, we didn't forget to substract the first value: we need to use an aggregate here because we're doing a sum() aggregate at the same query level, and we have the same value in each row of the resultset, so we used min(), max() would have been as good.

Another important trick we're using in that query is how to express the date range. Never use between for that, as you would end up counting boundaries twice, and customer won't like your accounting process if you do that. Always use a combo of inclusive and exclusive boundaries comparison, as in that WHERE clause in the previous query.

Let's have a quick look at the raw data in that range, using another nice aggregate that PostgreSQL comes with:

select array_agg(nb) from measures where tick >= 4 and tick < 14;
           array_agg
-------------------------------
 {30,40,0,20,30,60,0,10,30,35}
(1 row)

And now, the logical counter value for that period is computed as the following value by the previous query:

 sum
-----
 105
(1 row)

We can verify it manually, we want 40 + 60 + 35 - 30, I think we're all good again. Don't forget we have to substract the first measure from the period!

Extending the problem

Another interesting problem, that we didn't have here but that I find interesting enough to extend this article, is finding the ranges of time (here, ticks) within which the counter didn't reset.

The query is more complex because we need to split the data into partitions, each partition containing data from the same counter series of measures without wrapping. The usual trick is to self-join our data set so that for each given row we have a set of rows from the same partition, we are going to instead use a correlated subquery to go fetch the next wraparound value:

with tops as (
  select tick, nb,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
             else null
         end as max
    from measures
  window w as (order by tick)
)
  select tick, nb, max,
         (select tick
            from tops t2
           where t2.tick >= t1.tick and max is not null
        order by t2.tick
           limit 1) as p
    from tops t1;

 tick | nb  | max | p
------+-----+-----+----
    1 |   0 |     |  5
    2 |  10 |     |  5
    3 |  20 |     |  5
    4 |  30 |     |  5
    5 |  40 |  40 |  5
    6 |   0 |     |  9
    7 |  20 |     |  9
    8 |  30 |     |  9
    9 |  60 |  60 |  9
   10 |   0 |     | 14
   11 |  10 |     | 14
   12 |  30 |     | 14
   13 |  35 |     | 14
   14 |  45 |  45 | 14
   15 |  25 |     | 18
   16 |  50 |     | 18
   17 | 100 |     | 18
   18 | 110 | 110 | 18
(18 rows)

With that as an input it's then possible to build ranges of ticks including non wrapping set of measures from our counter, and get for each range the logical value tat the counter had at the end of it:

with tops as (
  select tick, nb,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
             else null
         end as max
    from measures
  window w as (order by tick)
),
     parts as (
  select tick, nb, max,
         (select tick
            from tops t2
           where t2.tick >= t1.tick and max is not null
        order by t2.tick
           limit 1) as p
    from tops t1
),
     ranges as (
  select first_value(tick) over w as start,
         last_value(tick) over w as end,
         max(max) over w
    from parts
  window w as (partition by p order by tick)
)
select * from ranges where max is not null;

 start | end | max
-------+-----+-----
     1 |   5 |  40
     6 |   9 |  60
    10 |  14 |  45
    15 |  18 | 110
(4 rows)

Conclusion

What I hope to have shown here, apart from some window function tips and some nice use cases for common table expressions, is that as a developper adding SQL to your tool set is a very good idea.

You don't want to have several parts of your code dealing with a logical counter like this, because you want the reporting, accounting, quota, billing and other software to all agree on the values. And you most probably want to avoid to fetch a huge result set of data and process it in the application memory (it'd better fit) rather than just get back a single integer column single row resultset, right?

If you find this SQL example to be off the limits, it's a good sign that you need to improve on your skills so that SQL is a real asset of your developer multi languages multi paradygm talents.

As a PostgreSQL contributor though, the release of 9.2 mainly means to me that it's time to fully concentrate on preparing 9.3. The developement season of which as already begun, by the way, so some amount of work has already been done here.

The list of things I want to be working on for that next release is quite long, and looks more like a christmas list than anything else. Let's only talk about those things I might as well make happen rather than all the things I wish I was able to be delivering in a single release...

Event Triggers

We missed 9.2 for wanting to include too big a feature in one go, leading to too many choices to review and take decision about, for once, and also to some non optimal choices that had to be reconsidered. Thanks to PGCON in Ottawa earlier this year, I could meet in person with Robert Haas and we've been able to decide how to attack that big patch I had. The first step has been to commit in the PostgreSQL tree only infrastructure parts, on which we will be able to build the feature itself.

Infrastructure

What we already have today is the ability to run user defined function when some event occurs, and an event can only be a ddl_command_start as of now. Also the trigger itself must be written in PLpgSQL or PL/C, as the support for the other languages was not included from the patch.

That leaves some work to be done in the next months, right?

PL support

The user defined function will get some information from magic variables such as TG_EVENT and such. That allows easier integration of future information we want to add, without disrupting those existing triggers that you wrote (no API change), at the cost of having to write a specific integration per procedural language.

So one of the first things to do now is to take the support for the others PL that I had in my proposal and make a new patch with only that in there.

Fill-in more information

Then again, this first infrastructure part was all about being actually able to run a user function and left behind most of the information I would like the function to have. The information already there is the command tag, the event name and the parsetree that's only usable if you're writing your trigger in C, which we expect some users to be doing.

To supplement that, we're talking about the Object ID that has been the target of the event, the schema it leaves in when applicable, the Object Name, the Operation that's running (CREATE, ALTER, DROP), the Object Kind being the target of said operation (e.g. TABLE or FUNCTION), and the command string.

Publishing the Command String

Publishing the Command String here is not an easy task, because we have to rebuild a normalized version of it. Or maybe we can go with passing explicit context in which the command is running, such as the search_path.

Even with an explicit context that would be easy enough to SET back again (in a remote server where you would be replicating the DDL, say), it would be better to normalize the command string so as to remove extra spaces and make it easier to parse and process from a user defined function.

That part looks like where most of the work is going to happen in the next commit fests.

Events

The other big thing I want to be working on with respect to this feature is the event support, which is basically hard coded to be ddl_command_start in the current state of the 9.3 code.

We certainly will want to be able to run user defined function not only at the very beginning of a DDL command, but also just before it finishes so that the newly created object already exists, for example.

We might also be interested into supporting triggers on more than DDL, there I doubt we will see that happening in 9.3, as some people in the community would go crazy about complex use cases. Time is limited, and I think this is better kept open for the next release, as the way our beloved PostgreSQL works is by delivering reliable features: quality first.

Use cases

I'm always happy to hear about use cases for the features I'm working on, and this one has the potential to be covering a non trivial amount of them. I already can think of trigger based replication systems and some integrated extension network facilities. With your help we can give those the place they should have: early days use cases in a great collection.

Extensions

So yes, event triggers first use case for me is in relation with extensions. Surprise! There's still some more I want to do with extensions, so much that I could consider their implementation in 9.1 just an enabler. In 9.1 the game has been to offer the best support we could design for existing contrib modules, with a very strong angle toward clean support for dump and restore.

The typical contrib module exports in SQL a list of C coded functions, sometime supporting a new datatype, sometime a set of administration functions. It's quite rare that contrib modules are handling user data embedded in their SQL definition, and when it happens it's mostly configuration kind of data, such as with PostGIS.

Now we want to fully support extensions that are maintaining their own user data, or even those that are all about them. The main difficulty here is that our current design of dump and restore support is following a model where installing the same extesion in a new database is all covered by create extension foo;. This is a limited model of the reality, that we need to expand.

The first manifestation of those problems is in the SEQUENCE support in extensions, and that impacts one of my favorite extensions: PGQ.

PostgreSQL releases

PostgreSQL just released an awesome release with 9.2, where we get tremendous performance optimisations and truly innovative features, such as RANGE TYPE. How not to consider PostgreSQL as a part of your application stack, where to develop and host your features.

While users are enjoying the newer release, contributors are already preparing the next one, hard at work again!

Why El-Get is relevant

Emacs 24.1 is the first release that includes package.el, and it even allows the user to setup several sources where to fetch packages. Those sources, such as Marmalade, are hosting lots of third party code for Emacs. package.el makes it easy to install (partly) those software.

This is a very fine way of getting extra features in your Emacs installation, and one that is supported out of the box. For a package to be listed, its sources need to be prepared, and you need to rely on the central website you now depend on to be up and running and accessible.

El-Get is all about allowing you to easily cope with the still vast majority of Emacs Lisp extensions you can find out there, that is non packaged code that is only available on some more or less mainstream distribution method, ranging from EmacsWiki to github including bare HTTP personal hosting.

With El-Get, you fetch the package where it's located. There's no need for a central server to host packaged and released software, and it's easy to share your findings with friends, or even to publish any elisp code you write.

El-Get will also take care of final steps that package.el did choose not to support, such as including Info material in your info browser (remember C-h i runs the command info?), running ./configure && make for you, byte compiling the sources you just retrieved, adding the necessary autoload support, etc.

And of course one of the methods supported by El-Get is ELPA, known as Emacs Lisp Package Archive and implemented by package.el.

So definitely, you typically want both ELPA and El-Get.

El-Get 4.1 Changelog Summary

The new El-Get release is packed with features. It really is. I will only list some of them now:

• Plenty new recipes, we now have 590 of them managed in the El-Get source repository itself, and El-Get will download the current EmacsWiki list of emacs lisp files at install time too.
• The default installation and usage has been simplified a lot.
• More options are provided to setup El-Get packages, see el-get-user-package-directory for example.
• Part of the simplification, el-get-sources has been revisited and now serves only one goal.
• We dropped (el-get 'wait) which was a misconception and had been broken for a long time in the development version of El-Get.
• We made improvements in the error handling and in dealing with some corner cases that still happen often enough for users to report them. Please continue reporting them!
• More caching is done, with a better dependency tracking and status management.
• Enhanced notification support, from DBUS to growl.
• Support for checksums with a lot of source types.
• Completing our git support, shallow clones and submodules are there.
• Better support for github including the zip and tar releases.
• Ability to reload a package when it's been updated.
• Moar features

And most importantly, El-Get documentation is now almost complete and comes in the nice Info format I know you've been expecting for so long!

Using El-Get

Here's a quick summary of what using El-Get is like, for a new user in 4.1. If you're already using El-Get see the section about upgrading. To install El-Get you need to paste those lines to your *scratch* buffer then hit C-j after the last closing parenthesis:

(url-retrieve
 "https://raw.github.com/dimitri/el-get/master/el-get-install.el"
 (lambda (s)
   (goto-char (point-max))
   (eval-print-last-sexp)))

Then you can try M-x el-get-list-packages and browse through more than 2000 available packages. Mark the ones you want to install with i then type x to see El-Get fetch and install all those packages you just selected. Here's a summary of what's available to you in the M-x el-get-list-packages buffer:

Major Mode Bindings:
SPC             el-get-package-menu-mark-unmark
?               el-get-package-menu-describe
d               el-get-package-menu-mark-delete
g               el-get-package-menu-revert
h               el-get-package-menu-quick-help
i               el-get-package-menu-mark-install
u               el-get-package-menu-mark-update
x               el-get-package-menu-execute

Once a package is installed, El-Get will initialize it for you, and it will also do that step at every Emacs startup from there on, provided that you added some lines to your ~/.emacs initialization file, that look a lot like the previous *scratch* code you did paste:

;;
;; Here's a typical El-Get integration for your .emacs file:
;;
(add-to-list 'load-path "~/.emacs.d/el-get/el-get")
(setq el-get-user-package-directory "~/.emacs.d/packages.d/")

(unless (require 'el-get nil t)
  (with-current-buffer
      (url-retrieve-synchronously
       "https://raw.github.com/dimitri/el-get/master/el-get-install.el")
    (goto-char (point-max))
    (eval-print-last-sexp)))

(el-get 'sync)

Then you can add files named like init-<package>.el in the el-get-user-package-directory directory, those files will get loaded when El-Get initialize <package>.

You can also use M-x el-get-install if you want to bypass the full screen package listing, you will get completion on the package name.

Community and development

El-Get community grew to be a really cool place to be participating in these days, with core and recipe contributions from more than 130 different people already, and with 526 stars on github and 184 forks. I almost can't believe it!

git --no-pager shortlog -n -s | wc -l
     137
git --no-pager shortlog -n -s | head -10
   734  Dimitri Fontaine
   336  Ryan C. Thompson
   114  Julien Danjou
   110  Dave Abrahams
    73  Ryan Thompson
    72  Sébastien Gross
    42  Takafumi Arakaki
    27  Alex Ott
    25  Yakkala Yagnesh Raghava
    21  Rüdiger Sonderfeld

Now that we have something that looks like a core team forming up, I'm thinking about scheduling much more aggressive stable release. 4.1 has been very long in the making, I hope to now have a rapid release cycle leading us to 4.2 in quite a short time. As that's not an individual effort by any mean, though, only history will tell.

The roadmap

We have lots of ideas and some rough edges to address, so 4.1 is only a stop in the release history of El-Get. Next ideas include better error management in face of rare corner cases and in face of external events, like when you did rm -rf a directory holding an El-Get managed extension: we should mark it removed and clean up the autoloads that came from it.

Upgrading to 4.1

This item has received some treatment in the documentation. The basic idea is that el-get-sources is no longer what it used to be, it's now only an alternative source location for recipes, like it should always have been. Not that you can still override in there some properties that you want merged with an official recipe.

The new thing about el-get-sources is that it will no longer be the authoritative list of packages that El-Get manages. That list is not either given explicitly when calling the el-get function in your .emacs setup, or derived from the packages that are known installed on your system (like e.g. debian is doing).

Also, given that it took us so much time to brew 4.1 a lot of packages have changed either their hosting location or even switched their SCM. In such cases an automatic update of the recipe will no longer be possible, you might need to el-get-remove then el-get-install packages to get them back.

Conclusion

El-Get 4.1 is now ready for public consumption, don't be shy, we've been a lot of users running the development branch for a long time now, I'm running 4.0.7.6901194 while writing this post. 4.0 is the development version of what is now released as 4.1.

Many thanks to all who contributed to El-Get and to all our users, I'm very proud that together we worked out a very nice and complete tool!

With so many smart qualifiers you can only guess that I did love the challenge. The idea is to write the simplest code possible and see how smarter you need to be when you need perfs. Let's have a try!

local python results

Here's the code I did use to benchmark the python solution:

def sumrange(arg):
    return sum(xrange(arg))

def sumrange2(arg):
    x = i = 0
    while i < arg:
        x += i
        i += 1
    return x


import ctypes
ct_sumrange = ctypes.CDLL('/Users/dim/dev/CL/jiaroo/sumrange.so')

def sumrange_ctypes(arg):
    return ct_sumrange.sumrange(arg)

if __name__ == "__main__":
    import timeit
    t1 = timeit.Timer('import jiaroo; jiaroo.sumrange(10**10)')
    t2 = timeit.Timer('import jiaroo; jiaroo.sumrange2(10**10)')
    ct = timeit.Timer('import jiaroo; jiaroo.sumrange_ctypes(10**10)')

    print 'timing python sumrange(10**10)'
    print 'xrange: %5fs' % t1.timeit(1)
    print 'while:  %5fs' % t2.timeit(1)
    print 'ctypes: %5fs' % ct.timeit(1)

Oh. And the C code too, sorry about that.

#include <stdio.h>

int sumrange(int arg)
{
    int i, x;
    x = 0;

    for (i = 0; i < arg; i++) {
        x = x + i;
    }
    return x;
}

And here's how I did compile it. The author of the inspiring article insisted on stupid optimisation targets, I did follow him:

gcc -shared -Wl,-install_name,sumrange.so -o sumrange.so -fPIC sumrange.c -O0

And here's the result I did get out of it:

python jiaroo.py
timing python sumrange(10**10)
xrange: 927.039917s
while:  2377.291237s
ctypes: 5.297124s

Let's be fair, with -O2 we get much better results:

timing python sumrange(10**10)
ctypes: 1.065684s

Common Lisp to the rescue

So let's have a try in Common Lisp, will you ask me, right?

Here's the code I did use, you can see three different tries:

;;;; jiaroo.lisp
;;;
;;; See http://jiaaro.com/python-performance-the-easyish-way
;;;
;;; The goal here is to find out if CL needs to resort to C for very simple
;;; optimisation tricks like python apparently needs too, unless using pypy
;;; (to some extend).

(in-package #:jiaroo)

;;; "jiaroo" goes here. Hacks and glory await!

(defun sumrange-loop (max)
  "return the sum of numbers from 1 to MAX"
  (let ((sum 0))
    (declare (type (and unsigned-byte fixnum) max sum)
             (optimize speed))
    (loop for i fixnum from 1 to max do (incf sum i))))

(defun sumrange-dotimes (max)
  "return the sum of numbers from 1 to MAX"
  (let ((sum 0))
    (declare (type (and unsigned-byte fixnum) max sum)
             (optimize speed))
    (dotimes (i max sum)
      (incf sum i))))

(defun pk-sumrange (max)
  (declare (type (and unsigned-byte fixnum) max)
           (optimize speed))
  (let ((sum 0))
    (declare (type (and fixnum unsigned-byte) sum))
    (dotimes (i max sum)
      (setf sum (logand (+ sum i) most-positive-fixnum)))))

(defmacro timing (&body forms)
  "return both how much real time was spend in body and its result"
  (let ((start (gensym))
        (end (gensym))
        (result (gensym)))
    `(let* ((,start (get-internal-real-time))
            (,result (progn ,@forms))
            (,end (get-internal-real-time)))
       (values ,result (/ (- ,end ,start) internal-time-units-per-second)))))

(defun bench-sumrange (power)
  "print execution time of both the previous functions"
  (let* ((max (expt 10 power))
         (lp-time (multiple-value-bind (r s) (timing (sumrange-loop max)) s))
         (dt-time (multiple-value-bind (r s) (timing (sumrange-dotimes max)) s))
         (pk-time (multiple-value-bind (r s) (timing (pk-sumrange max)) s)))
    (format t "timing common lisp sumrange 10**~d~%" power)
    (format t "loop:       ~2,3fs ~%" lp-time)
    (format t "dotimes:    ~2,3fs ~%" dt-time)
    (format t "pk dotimes: ~2,3fs ~%" pk-time)))

And here's the results:

CL-USER> (bench-sumrange 10)
timing common lisp sumrange 10**10
loop:       11.213s
dotimes:    7.642s
pk dotimes: 22.185s
NIL

Discussion

So python is very slow. C is pretty fast. And Common Lisp just in the middle. Honnestly I expected better performances from my beloved Common Lisp here, but I didn't try very hard, by using Clozure Common Lisp which is not the quicker Common Lisp implementation around. For this very benchmark, if you're seeking speed use either Steel Bank Common Lisp or CLISP which is known to have a pretty fast bignums implementation (which you don't need in 64 bits in that game).

On the other hand, I think that having to go write a C plugin and deal with how to compile and deploy it in the middle of a python script is something to avoid. When using Common Lisp you don't need to resort to that for the runtime to get down from python xrange implementation at 927.039917s down to the dotimes implementation taking 7.642s. That's about 121 times faster.

So while C is even better, and while I would like a Common Lisp guru to show me how to get a better speed here, I still very much appreciate the solution here.

Let's see the winning source code in python and common lisp to compare the programmer side of things: how hard was it really to get 121 times faster?

def sumrange(arg):
    return sum(xrange(arg))

(defun sumrange-dotimes (max)
  "return the sum of numbers from 1 to MAX"
  (let ((sum 0))
    (declare (type (and unsigned-byte fixnum) max sum)
             (optimize speed))
    (dotimes (i max sum)
      (incf sum i))))

That's about it. Yes we can see some manual optimisation directives here, which are optimisation extra complexity. Not to the same level as bringing a compiled artifact that you need to build and deploy, though. Remember that you will need to know the full path where to find the sumrange.so file on the production system, in the optimised python case, so that's what we are comparing against.

Here's what happens without the optimisation, and with a smaller target:

CL-USER> (time (jiaroo:sumrange-dotimes (expt 10 9)))
(JIAROO:SUMRANGE-DOTIMES (EXPT 10 9))
took 722,592 microseconds (0.722592 seconds) to run.
During that period, and with 2 available CPU cores,
     714,709 microseconds (0.714709 seconds) were spent in user mode
       1,183 microseconds (0.001183 seconds) were spent in system mode
499999999500000000

CL-USER> (time (let ((sum 0)) (dotimes (i (expt 10 9) sum) (incf sum i))))
(LET ((SUM 0)) (DOTIMES (I (EXPT 10 9) SUM) (INCF SUM I)))
took 2,174,767 microseconds (2.174767 seconds) to run.
During that period, and with 2 available CPU cores,
     2,156,549 microseconds (2.156549 seconds) were spent in user mode
        10,225 microseconds (0.010225 seconds) were spent in system mode
499999999500000000

We get a 3 times speed-up from those 2 lines of lisp optimisation directives, which is pretty good. And it's exponential as I didn't have the patience to actually wait until the non optimised 10^10 run finished, I killed it.

Conclusion

That's a case here where I don't know how to reach C level of performances with Common Lisp, which could just be because I don't know yet how to do.

Still, getting a 121 times speedup when compared to the pure python version of the code is pretty good and encourages me to continue diving into Common Lisp.

This talk shares hindsights about the why and the how of that migration, what problems couldn't be solved without moving away and how the solution now looks. The tools used for migrating away the data, the methods the new architecture are detailed. And the new home, in the cloud!

Not that much later after that the European PostgreSQL community is giving us a very nice occasion to get to Prague with PostgreSQL Conference Europe 2012 (October 23-26). If you've been meaning to meet with the community, if you've been meaning to visit Prague someday, or any mix of those two very good reasons, think about booking that conference already.

The call for papers for pgconf.eu has been extended to August 7th, 2012. Consider sharing your hindsights too!

The article is very well written and makes it easy to think that coming up with the code for such a solver is a very easy task, you apply some basic problem search principles and there you are. Which is partly true, in fact. Also, he uses python, and that means that a lot of trivial programming activities are not a concern anymore, such as memory management.

As I've been teaching myself Common Lisp for some weeks now I though I would like to read a lisp version of his code, and the article even has a section titled Translations. Unfortunately, no lisp version is available there. One might argue that Clojure is a decent enough lisp, but my current quest is all about Common Lisp really. So I had to write one myself.

CL-USER> (sudoku:print-puzzle
          (sudoku:solve-grid
"530070000600195000098000060800060003400803001700020006060000280000419005000080079"))
5 3 4 | 6 7 8 | 9 1 2
6 7 2 | 1 9 5 | 3 4 8
1 9 8 | 3 4 2 | 5 6 7
------+-------+------
8 5 9 | 7 6 1 | 4 2 3
4 2 6 | 8 5 3 | 7 9 1
7 1 3 | 9 2 4 | 8 5 6
------+-------+------
9 6 1 | 5 3 7 | 2 8 4
2 8 7 | 4 1 9 | 6 3 5
3 4 5 | 2 8 6 | 1 7 9
took 1,974 microseconds (0.001974 seconds) to run.
During that period, and with 2 available CPU cores,
     1,894 microseconds (0.001894 seconds) were spent in user mode
        88 microseconds (0.000088 seconds) were spent in system mode
 174,320 bytes of memory allocated.
#<SUDOKU::PUZZLE #x3020023BB9FD>

Comments on the python version

Norvig's article is very well written, I think. By that I mean that by reading it you're confident that you've understood the problem and how the solution is articulated, so you almost think you don't need to really try to understand the code, it's just an illustration of the text.

Well, not so much. When you want to port the exact same algorithm you have to understand exactly what the code is doing so that you're not implementing something else. All the more when, as I did, you want to use some other data structure.

My goal was not to rewrite the code as-is, but to try and come up with idiomatic lisp code implementing Norvig's solution. So rather than using strings and dictionaries (in lisp, they still call them a hash table) I've been using more natural data structures.

The python code is really uneasy to follow, full of functional programming veteran tricks. I mean avoiding exceptions and simply returning False whenever there's a problem, and using functions such as all and some to manage that. It's certainly working, it's not making the code any easier to read.

To summarize, that code looks like it's been written by someone smart who didn't want to spend more than a couple of hours on it, and did take all known trustworthy shortcuts he could to achieve that goal. Quality and readability certainly weren't the key motive. I've been quite deceived after reading a very good article.

Comments on the common lisp version

Keep in mind that I'm just a Common Lisp newbie. I've been told some good pieces of advice by knowledgeable people though, so with some luck my implementation is somewhat lispy enough.

So we start by defining some data structures and low-level functions to build up the more complex one, so that it's easier to read and debug. The sudoku puzzle is then a grid of digits and a grid of possible values in places where the digits are yet unknown.

The way to represent that 9x9 grid is with using make-array:

(make-array '(9 9)
            :element-type '(integer 0 9)
            :initial-element 0)

Then the possible values. I though about using a bit-vector (and actually I did implement it that way), then I've been told that the Common Lisp way to approach that is using 2-complement integer representation, as we have plenty of functions to operate numbers that way. I wouldn't believe that would make the code simpler, but in fact it really did, see:

CL-USER> #b111111111
511
CL-USER> (logcount #b111111111)
9
CL-USER> (logcount 511)
9
CL-USER> (logbitp 3 #b100100100)
NIL
CL-USER> (logbitp 2 #b100100100)
T
CL-USER> (format nil "~2r" (logxor #b111111111 (ash 1 4)))
"111101111"
CL-USER> (logbitp 4 (logxor #b111111111 (ash 1 4)))
NIL

With that in mind, we can write the following code:

(defun count-remaining-possible-values (possible-values)
  "How many possible values are left in there?"
  ;; we could raise an empty-values condition if we get 0...
  (logcount possible-values))

(defun first-set-value (possible-values)
  "Return the index of the first set value in POSSIBLE-VALUES."
  (+ 1 (floor (log possible-values 2))))

(defun only-possible-value-is? (possible-values value)
  "Return a generalized boolean which is true when the only value found in
   POSSIBLE-VALUES is VALUE"
  (and (logbitp (- value 1) possible-values)
       (= 1 (logcount possible-values))))

(defun list-all-possible-values (possible-values)
  "Return a list of all possible values to explore"
  (loop for i from 1 to 9
     when (logbitp (- i 1) possible-values)
     collect i))

(defun value-is-set? (possible-values value)
  "Return a generalized boolean which is true when given VALUE is possible
   in POSSIBLE-VALUES"
  (logbitp (- value 1) possible-values))

(defun unset-possible-value (possible-values value)
  "return an integer representing POSSIBLE-VALUES with VALUE unset"
  (logxor possible-values (ash 1 (- value 1))))

You can see here that I was also under the influence of a recent reading about making it obvious, or so called intentional programming, following what Joe Armstrong has to say about it:

Intentional programming is a name I give to a style of programming where the reader of a program can easily see what the programmer intended by their code. The intention of the code should be obvious from the names of the functions involved and not be inferred by analysing the structure of the code. (Reading the code should) precisely expresses the intention of the programmer—here no guesswork or program analysis is involved, we clearly read what was intended.

So there we go with function names such as count-remaining-possible-values, that will help when reading some more complex code, as in the following, the meat of the solution:

(defmethod eliminate ((puzzle puzzle) row col value)
  "Eliminate given VALUE from possible values in cell ROWxCOL of PUZZLE, and
   propagate when needed"
  (with-slots (grid values) puzzle
    ;; if already unset, work is already done
    (when (value-is-set? (aref values row col) value)
      ;; eliminate the value from the set of possible values
      (let* ((possible-values
              (unset-possible-value (aref values row col) value)))
        (setf (aref values row col) possible-values)

        ;; now if we're left with a single possible value
        (when (= 1 (count-remaining-possible-values possible-values))
          (let ((found-value (first-set-value possible-values)))
            ;; update the main grid
            (setf (aref grid row col) found-value)

            ;; eliminate that value we just found in all peers
            (eliminate-value-in-peers puzzle row col found-value)))

        ;; now check if any unit has a single possible place for that value
        (loop
           for (r . c)
           in (list-places-with-single-unit-solution puzzle row col value)
           do (assign puzzle r c value))))))

So that lisp code is quite verbose and at 389 lines almost doubles the 201 lines Norvig had. When clarity is part of the goal, that's hard to avoid, I hope I made a good case that this is not due to lisp being overly verbose by itself.

Comments on the development environment

Or why I even considered Common Lisp as an interesting language for that kind of exercise, and some more. I'll have to tell about re-sharding data live with 16 threads and 256 databases, all in CL, someday.

So I've been doing some Emacs Lisp development for a while now, and the part that makes that so much fun is the instant reward. You write some code in your editor, type a key chord (usually, that's C-M-x runs the command eval-defun) and your code is loaded up, ready to be tested. In Emacs Lisp the test can be simply using your editor and watching the new behavior taking place, or playing in the M-x ielm console. When the code is not ready, it crashes, and you're left in the interactive debugger, where you can use C-x C-e runs the command eval-last-sexp to evaluate any expression in your source and see its value in the current debug frame.

That way of working is a huge productivity boost, that I've been missing much when getting back to writing C code for PostgreSQL. I can't C-M-x the current function and go write some SQL to test it right away, I have to compile the whole source tree, then install the new binaries, then restart the test server and then open up a psql console to interact with the new code. Of course I could just make check and watch the results, but then if I attach a debugger it complains that the code on-disk is more recent than the code in the core dump.

What if you want Emacs Lisp integrated facilities and something made for general programming rather than suited to building a text editor? Don't get me wrong, you can probably find more production ready code in elisp than in many other languages, just because Emacs has been there for about 35 years. Editor targeted production code, though.

This integrated development cycle is all the same when you're using Common Lisp. The awesome Superior Lisp Interaction Mode for Emacs is providing exactly that experience. Just run M-x slime and then as you define your code you can C-M-x the function at point, see the compilation errors and warnings if any in the associated REPL, and just try your code. I tend to mostly play in the command line, it's possible to just use C-x C-e while typing too.

Performances

Of course we do care! After all the original article came with a quite detailed performance analysis with graphs and all. I won't be reproducing that, sorry. I'll just show you what penalty you get for using an older language specification, much more dynamic and with more features than python, and with a great, scratch that, awesome development environment.

Oh wait, that's the other way round, no penalty, it's actually so much faster!

Python version perfs

The results I got on my desktop machine are about twice as fast as in the original article, I guess newer machines and newer python have something to say for that:

  dim ~/dev/CL/sudoku python sudoku.dim.py
  All tests pass.
  Solved 50 of 50 easy puzzles (avg 0.01 secs (151 Hz), max 0.01 secs).
  Solved 95 of 95 hard puzzles (avg 0.02 secs (42 Hz), max 0.12 secs).
  Solved 11 of 11 hardest puzzles (avg 0.01 secs (115 Hz), max 0.01 secs).

That makes an average of (50*151 + 95*42 + 11*115) / (50+95+11) = 82Hz.

That seems pretty good, let's continue.

As you can see I've cut away the random puzzle part, that's because I was too lazy to implement that part, which didn't seem all that interesting to me. If you think that's a problem and need solving, I accept patches.

Common lisp version perfs

When using SBCL on the same machine, what I got was:

  (sudoku:solve-example-grids)
  Solved 50 of 50 easy puzzles (avg .0021 sec (471.7 Hz), max 0.015 secs).
  Solved 95 of 95 hard puzzles (avg .0022 sec (446.0 Hz), max 0.008 secs).
  Solved 11 of 11 hardest puzzles (avg .0018 sec (550.0 Hz), max 0.003 secs).

With the same way to compute the average, we now have 461.6Hz.

Now, that's between 3 times and more than 10 times faster than the python version (taken collection per collection), for a comparable effort, a much better development environment, and the same all dynamic no explicit compiling approach.

Conclusion

I guess I'm fond of Common Lisp, which I already saw coming (so did you, right?), and now I have some public article and code to share about why :)

The code is hosted at https://github.com/dimitri/sudoku if you're interested, with the necessary files to reproduce, some docs, etc.

Also, apart from using integers as bitfields, which I did more for being lispy than for performances, I did very little effort for optimizing the code. It's quite naive in this respect, yet allow me an average of 461.6Hz rather than 82Hz, that's 5.6 times faster average.

So yes, I will continue to invest some precious time in Common Lisp as a very good interactive scripting language, and maybe more than that.

You might have guessed it already, I did talk about replication. Here's the slide deck I did use, it's in french, sorry if you don't grok that language.

The conference was very nice and did go smoothly, even if we were “only” 60 of us I had the pleasure to meet with different users with very different set of needs. Very happy to have been there!

In this case we're talking about the M-x compile and M-x recompile functions. My friend bound the former to <f11> and wanted that C-u f11 do a recompile with the exact same command line as the previous compile command.

Well, to be honest, I didn't know about M-x recompile until after I wrote the following function, made to trigger another compile with the last command used if using C-u.

(defvar cyb-compile-last-command nil)
(defvar cyb-compile-command-history nil)

(defun cyb-compile (arg)
  "Compile with given command, optionally recompile with last command"
  (interactive "P")
  (if arg
      (progn
        ;; arg given: compile with last command
        (unless cyb-compile-last-command
          (error "Can't recompile yet, no known last command"))
        (compile cyb-compile-last-command))
    ;; else branch, no arg given, ask for a command
    (let ((command
           (read-string
            "Compile with command: "
            "make -k" 'cyb-compile-command-history "make -k")))
      (setq cyb-compile-last-command command)
      (compile command))))

(global-set-key (kbd "<f11>") 'cyb-compile)

With that little Emacs Lisp code we're driving Emacs the way we want to be working, and that's great! You can see it was a quick hack in that if you wanted to use the function non interactively it would still prompt for the command to use to compile, when Emacs Lisp interactive special form would allow us to implement something way smarter here. Also if we wanted to spend some more time on that feature, we should probably tweak the error condition to be asking for the command rather than just complaining, that would certainly be more useful.

Exercise left to the reader, rewrite using recompile rather than reinventing it in a hurry! Beware of call-interactively though. Oh and fix the aforementioned infelicities, too.

To conclude, we see that writing Emacs Lisp code to fix a usability problem in a hurry is a great force of Emacs, and that we're provided with the necessary tool set so as to be able to reach completeness if we wanted to do so.

The in core replication project has been presented with slides titled Future In-Core Replication for PostgreSQL and got a very good reception. For instance, people implementing Slony (Jan Wieck, Christopher Browne and Steve Singer where here) appreciated the concepts here and where rather supportive of both the requirements and the design, and appreciated the very early demo and results that we had to show already, as a kind of a proof of concepts.

After those first two days, we could start the actual show. I had the honnor to present a migration use case entitled Large Scale MySQL Migration where we're speaking about going from MySQL to PostgreSQL, from 37 to 256 shards, moving more than 6TB of data including binary blobs that we had to process with pl/java. A quite involved migration project whose slides you now can read here:

I've heard that we should soon be able to enjoy audio and video recordings of the sessions, so if you couldn't make it this year for any reason, don't miss that, you will have loads of very interesting talks to virtually attend. I definitely will do that to catch-up with some talks I couldn't attend, having to pick one out of three is not an easy task, all the more when you add the providential hallway track.

Here we go. It used to be much more simple than that, so if you're still using PGQ from Skytools2, just jump to the next step.

Unregister Subconsumers

That query will figure out subconsumers in the system function pgq.get_consumer_info() and ask PGQ to please unregister them, losing events in the way, even events from batches that are currently active.

 with subconsumers as (
   select q1.queue_name,
          q2.consumer_name,
          substring(q1.consumer_name from '%.#"%#"' for '#') as subconsumer_name
    from (select *
            from pgq.get_consumer_info()
           where lag is null)
         as q1
    join (select *
           from pgq.get_consumer_info()
           where lag is not null)
         as q2
         on q1.queue_name = q2.queue_name
)
select *,
       pgq_coop.unregister_subconsumer(queue_name, consumer_name,
                                       subconsumer_name, 1)
 from subconsumers;

Unregister Consumers

Now that the first step is done, we have to unregister the main consumers, which is easy and what you already did before:

select queue_name, consumer_name,
       pgq.unregister_consumer(queue_name, consumer_name)
  from pgq.get_consumer_info();

Drop queues

And as we want to really clean up the mess, let's also drop the queues.

select queue_name, pgq.drop_queue(queue_name)
  from pgq.queue;

That calls for using PGQ, the jobs queue solution from Skytools, the power horse for Londiste. If PGQ is good enough to build a full trigger-based replication solution on top of it, certainly it's good enough for our custom processing, right? Well, you still need to check that your expectations are met, and that was happily the case in my implementation. It's a very common problem, and PGQ very often is a great solution to it.

As this implementation is PHP centric, we've been using libphp-pgq to drive our background workers. Using PGQ in PHP has been very easy to setup, the only trap being not to forget about running the ticker process.

It got interesting because of two elements. First, we're nor running a single database instance here but a bunch of them... make it 256 databases. Each of them having 5 queues to consume, that would be about 1280 consumer processes, distributed on 16 servers that's still 80 per server, so way too many. What we did instead is reuse the queue mover script found in the Skytools distribution and adapt it to forward the event of the 1280 source queues to only 5 destination queues. We then process the events from this single location.

Now it's easier to deal with, but we're not still exactly there. Of course, with so many sources, concentrating them all into the same place means that a single consumer is not able to process the events as fast as they are produced. That's where the cooperative consuming shines, it's very easy to turn your consumer into a cooperative one even on an existing and running queue, and that's what we did. So now we can choose how many workers we want per queue: one of them has 4 workers, another one see not so much activity and 1 worker still fits.

The queue mover script that knows how to subscribe to many queues from the same process is going to be contributed to Skytools proper, of course.

To be specific, only superusers are allowed to install C coded stored procedures, and that impacts a lot of very useful PostgreSQL extension: all those shiped in the contrib package are coded in C. Now, Heroku is not giving away superuser access to their hosted customers in order to limit the number of ways they can shoot themselves in the foot. And given PostgreSQL security model, being granted database owner is mostly good enough for day to day operation.

See Andrew's article Heroku, a really easy way to get a database in a hurry for more context about Heroku's offering here.

Mostly, but as we see, not completely good enough. How to arrange for a non superuser to be able to still install a C-coded extension in his own database? That's quite dangerous as any bug causing a crash would mean a PostgreSQL whole restart. So you not only want to empower CREATE EXTENSION to database owners, you also want to be able to review and explicitely white list the allowed extensions.

Here we go: pgextwlist is a PostgreSQL extensions implementing just that idea. You have to tweak local_preload_libraries so that it gets loaded automatically and early enough, and you have to provide for the list of authorized extensions in the extwlist.extensions setting.

Let's see a usage example, straight from the documentation:

dim=> select rolsuper from pg_roles where rolname = current_user;
select rolsuper from pg_roles where rolname = current_user;
 rolsuper
----------
 f
(1 row)

dim=> create extension hstore;
create extension hstore;
WARNING:  => is deprecated as an operator name
DETAIL:  This name may be disallowed altogether in future versions of PostgreSQL.
CREATE EXTENSION

dim=> create extension earthdistance;
create extension earthdistance;
ERROR:  extension "earthdistance" is not whitelisted
DETAIL: Installing the extension "earthdistance" failed, because it is not
        on the whitelist of user-installable extensions.
HINT: Your system administrator has allowed users to install certain
      extensions. SHOW extwlist.extensions;

dim=> \dx
\dx
                           List of installed extensions
  Name   | Version |   Schema   |                   Description
---------+---------+------------+--------------------------------------------------
 hstore  | 1.0     | public     | data type for storing sets of (key, value) pairs
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

dim=> drop extension hstore;
drop extension hstore;
DROP EXTENSION

As you can see, it allows non superusers to install an extension written in C.

J'ai dans cette présentation très rapide (5 minutes seulement) mentionné l'approche axiomatique de John McCarthy lorsqu'il a découvert le language, on peut en lire un peu plus sur le site de Paul Graham et son article The Roots of Lisp et le code associé, une implémentation du LISP de McCarthy en common lisp.

Merci à Duchess pour une bonne soirée où nous avons pu échanger nos points de vue et débattre des languages fonctionnels et objects, des différences entre Erlang et Haskell et Ruby, et de quelques autres sujets dérivés !

As the plugin is 300 lines of python code, it's not a good idea to just inline it here, so please grab it at pgbouncer_.

You might need to know that the script name once installed should follow the form pgbouncer_dbname_stats_requests or pgbouncer_dbname_pools, where of course dbname can contain any number of _ characters. This script supports quite old versions of pgbouncer that didn't accept the normal pq protocol, you did have to use psql to have any chance of getting the data from a script, you couldn't then just use a PostgreSQL driver such as psycopg2.

L'idée de ma présentation, que la plupart d'entre vous a loupé je suppose (en tout cas je n'avais qu'une petite poignée de français dans la salle, et j'espère avoir des lecteurs qui n'étaient pas à Amsterdam), l'idée est d'utiliser les mécanismes offerts par les extensions afin de maintenir le code PL que vous utilisez en production.

Il s'agit la plupart du temps de procédures qui implémentent une partie de la logique métier de vos applications, mais si proche des données que cela termine en base directement : c'est une bonne chose, en particulier depuis PostgreSQL 9.1. Cette version propose en effet une gestion assez complète des extensions.

Il s'agit de réaliser un empaquetage de vos procédures en suivant la documentation en ligne et son chapitre 35.15. Empaqueter des objets dans une extension. Une fois cela fait, il est alors possible de déployer votre ensemble de procédure stockée avec la commande CREATE EXTENSION mesprocs;, et ensuite la commande psql \dx vous permet de lister les extensions installées et leur numéro de version.

Les mises à jours sont également gérées avec une commande SQL dédiée, il s'agit alors de ALTER EXTENSION mesprocs UPDATE [TO version];. Il suffit de fournir des scripts intermédiaires nommés par exemple mesprocs--1.0--1.1.sql et mesprocs--1.1--1.2.sql et PostgreSQL saura comment passer de 1.0 à 1.1.

Voilà, vous savez presque tout de ma présentation à Amsterdam et vous pouvez retrouver le reste sur le support proposé en début d'article. Bien sûr je n'ai pas reproduit ici les questions intéressantes qui m'ont été posées, une bonne partie d'entre elles sont venues enrichir ma liste de Noël pour les extensions. Si vous voulez être sûr de trouver cela sous votre sapin, cependant, le meilleur moyen est encore de m'en parler : sponsoriser les développement Open Source est une belle démarche :)

Dave Page talked about SQL/MED under the title PostgreSQL at the center of your dataverse and detailed what to expert from a Foreign Data Wrapper in PostgreSQL 9.1, then how to write your own. Wherever you are currently managing your data, you can easily enough make it so that PostgreSQL integrates them by means of fetching them to answer your queries. Which means real time data federating: you don't copy data around, you remote access them when executing the query.

I might need to come up with new Foreign Data Wrappers in a not too distant future, now that I better grasp how much work it really is to do that, it appears to be a good migration strategy too:

  INSERT INTO real.table
       SELECT * FROM foreign.table;

Another discovery is that apparently PLv8 is ready for public consumption. Using it can lead to Heralding the Death of NoSQL, so use it with care.

In the presentation of Synchronous Replication and Durability Tuning we mainly saw that mixing synchronous and asynchronous transactions in your application is the key to real performances across the ocean, as the speed of the light is not infinite. From Baltimore to Amsterdam the latency can not be better than 100ms and that's not the same as instant nowadays.

Then again, depending on the number of concurrent queries to sync over the ocean link, the experimental setup was able to achieve several thousands of queries per second, which is validating the model we picked for sync rep and its implementation.

If you want to read the slides again at home, or if you could not be there for some reason, then most of the talks are now available online at the PostgreSQL Conference Europe Talks 2011 wiki page.

I tend to think best practice here begins with defining properly the backup plan you want to implement. It's quite a complex matter, so be sure to ask yourself about your needs: what do you want to be protected from?

The two main things to want to protect from are hardware loss (crash disaster, plane in the data center, fire, water flood, etc) and human error (UPDATE without a where clause). Replication is an answer to the former, archiving and dumps to the latter. You generally need both.

Often enough “backups” include WAL archiving and shipping and nightly or weekly base backups, with some retention and some scripts or procedures ready to setup Point In Time Recovery and recover some data without interfering with the WAL archiving and shipping. Of course with PostgreSQL 9.0 and 9.1, the WAL Shipping can be implemented with streaming replication and you can even have a Hot Standby. But for backups you still want archiving.

Mostly I still implement pg_dump -Fc nightly backups with a custom retention (for example, 1 backup a month kept 2 years, 1 backup a week kept 6 or 12 months, 1 backup a night kept 1 to 2 weeks), when the database size allow the pg_dump run to remain constrained in the maintenance window, if any.

Don't forget that while pg_dump runs, you can't roll out DDL changes to the production system any more, so you want to be careful about this maintenance window thing. When you have one.

Physical backups are not locking rollouts away, but they often suck a good deal of the IO bandwidth so you need to pick up a right timing to do them. That's how you can get to once a week base backup and WAL archiving.

If you can't pg_dump production, maybe you can have automated restore jobs from the physical backups that you then pg_dump -Fc, so that you still have that. That can come up handy, really: you can't test your major upgrade out of a physical backup.

Also, obviously, never consider your backup strategy implemented until you have either automated restores in place or a regular schedule to exercise them (staging instances, devel instances).

Then as far as the practical tools go, I tend to think that pg_staging is worth its installation complexity, and for WAL archiving and base backup I recommend walmgr from Skytools, that's a very handy tool. When using PostgreSQL 9.0 or 9.1, consider using walmgr3 so that it's behaving nice alongside streaming replication.

Je présenterai donc comment utiliser les extensions, le titre en anglais est Extensions are good for business logic, et l'idée est de voir comment exploiter les extensions afin de mieux gérer vos mises à jours en bases de données.

Le cycle de vie des bases de données en production inclue souvent l'utilisation d'une base de développement où le schéma évolue au rythme des besoins des développeurs, et de temps en temps on consolide une partie de ces modifications (dans des rollouts, scripts contenant principalement des DDL) afin de les déployer en production — si possible avec une étape intermédiaire en préproduction, tout de même.

Savoir ce qui est déployé en développement et comment en retirer le script à jouer en production peut être parfois fastidieu. Quand ce n'est pas le cas, c'est que le travail a été fait en amont, ce qui est le signe d'une bonne organisation, avec les surcoûts que l'on peut imaginer.

Les extensions telles que présentes dans PostgreSQL 9.1 vous permettent de mieux gérer ce genre de cas, en optimisant le surcoût : il ne disparaît pas, mais devient opérationnel plutôt que de rester une charge d'organisation.

Allez, je vous laisse maintenant, je dois me préparer pour la conférence :)

It's not the way I would typically approach scaling problems, and apparently I'm not alone on the Stored Procedures camp. Did you read this nice blog post Mythbusters: Stored Procedures Edition already? Well it happens in another land that where my comfort zone is, but still has some interesting things to say.

I won't try and address all of the myths they attack in a single article. Let's pick the scalability problems, the two of them I think about are code management and performances. We are quite well equiped for that in PostgreSQL, really.

For code maintainance we now have PostgreSQL Extensions, which allows you to pack all your procedures into separate extensions, and to maintain a version number and upgrade procedures for each of them. You can handle separate rollouts in development for going from 1.12 to 1.13 then 1.14 and after the developers tested it more completely and changed their mind again on the best API they want to work with, 1.15 which is stamped ok for production. At this point, ALTER EXTENSION UPGRADE will happily apply all the rollouts in sequence to upgrade from 1.12 straight to 1.15 in one go. And if you prefer to bake a special careful script to handle that big jump, you also can provide a specific extension--1.12--1.15.sql script.

Of course you're managing all those files with your favorite SCM, to answer to some other myth from the blog reference we are loosely following.

I wanted to talk about the other side of the scalability problem, which is the operations side of it. What happens when you need to scale the database in terms of its size and level of concurrent activity? PostgreSQL earned a very good reputation at being able to scale-up, what about scaling-out? Certainly, now that you're all down into Stored Procedure, it's going to be a very bad situation?

Well, in fact, you're then in a very good position here, thanks to PLproxy. This extension is a custom procedural language whose job is to handle a cluster of database shards that all expose the same PL API, and it's very good at doing that.

Stored Procedures are a very good tool to have, be sure to get comfortable enough with them so that you can choose exactly when to use them. If you're not sure about that, we at 2ndQuadrant will be happy to help you there!

I'll be presenting another talk about extensions, but this time I've geared up to use cases, with Extensions are good for business logic. The idea is not to talk about how to make PostgreSQL play fair with extensions including at dump and restore times, that's already done and I've been talking only too much about it. The idea this time is to figure out how much you get from this feature.

If you ever felt like something is missing in your processes between pushing rollouts in devel environments and refining them as developers are testing and preparing something for the live databases, then we have something for you here. Including how to easily compare state between production and development, but without having to guess or reverse engineer anything.

Yeah, extensions are all about getting even more professional! A great tool you'll be happy to master!

And now I need to prepare a damn good slide deck, right? See you there! :)

C'est l'endroit où aller pour apprendre comment le projet fonctionne, comprendre les impacts des nouvelles versions sur votre architecture, avoir une discussion technique pointue sur cette fonctionalité que vous voudriez voir arriver dans la prochaine version, ou simplement vous rendre compte de l'énergie formidable qui est insuflée dans ce projet !

Évidemment 2ndQuadrant sera de la partie, nous présenterons plusieurs de nos contributions PostgreSQL 9.1. Cela commencera avec la formation d'une journée complète de Greg, Performance From Start to Crash : si vous voulez apprendre comment aborder les performances d'un serveur PostgreSQL par le leader international du domaine, auteur du livre Bases de données PostgreSQL 9.0, réservez vite votre place !

Les présentation au format classique commencent le lendemain, et en trois jours la liste des présentation de notre équipe 2ndQuadrant est assez copieuse. Voyons cela.

Nous commençons avec Migration to PostgreSQL - a holistic view par Harald Armin Massa, qui propose un point de vue intéressant sur les raisons qui retiennent certaines migrations. Ensuite Gianni Ciolli présentera Look Out The Window Functions (and free your SQL) ou comment résoudre simplement des problèmes complexes lorsque l'on dispose d'outils avancés.

Une autre présentation à ne pas rater, Synchronous Replication and Durability Tuning détaille comment profiter au mieux de PostgreSQL 9.1 afin d'obtenir les garanties de durabilité des données souhaitées dans votre application. Et cette présentation est animée par Greg Smith et Simon Riggs. Ce dernier a développé la réplication synchrone, et Hot Standby avant cela. Vous ne trouverez personne au monde mieux placé pour faire cette présentation !

Les deux prochaines présentation de nos experts PostgreSQL, en continuant notre lecture du programme de pgconf.eu dans l'ordre, ont lieu au même moment. Le choix ne sera pas facile entre Improving VACUUM Suction par Greg à nouveau, et une comparaison de londiste 3 et slony 2.1 par Cédric Villemain, en français.

À suivre, Extensions are good for business logic que je vous présenterai moi-même, vous pouvez voir ma présentation sur la fiche qui porte mon nom : Dimitri Fontaine. Il s'agit d'une présentation en anglais qui détaille comment utiliser les extensions dans le cadre de la maintenance de la partie procédures stockées d'une application.

Et pour finir le deuxième jour des conférences 2ndQuadrant, vous pourrez apprendre avec Gianni comment Debugging complex SQL queries with writable CTEs, une fonctionnalité contribuée au projet par un autre consultant 2ndQuadrant, Marko Tiikkaja.

Et il reste encore une journée ! Nous ne mentons pas en disant que le programme est complet ! Le dernier jour de la conférence n'est pas le moins intéressant, j'espère que vous aurez su garder un peu d'énergie pour suivre…

Simon Riggs qui présentera sa vision de la PostgreSQL Roadmap pour les prochaines années. Ce n'est bien sûr que sa vision personnelle, mais lorsque l'on fait le bilan de ces 7 dernières années de contributions à PostgreSQL, on voit à quel point son opinion personnelle peut avoir du poids dans le développement du projet.

À suivre, la présentation de Greg sur son sujet de prédilection : Bottom-up Database Benchmarking. Tout ce que vous avez toujours voulu savoir sur les mesures de performances de vos bases de données, sans jamais oser le demander. Quelque chose dans ce style en tout cas :)

Bien sûr d'autres présentations sont disponibles et retiendront votre attention, ce billet vous présente seulement celles qui seront données par les experts PostgreSQL de 2ndQuadrant. En vous souhaitant bonne conférence à tous, j'espère avoir le plaisir de vous retrouver à Amsterdam le mois prochain !

WalMgr is the Skytools component that manages WAL shipping for you, and archiving too. It knows how to prepare your master and standby setup, how to take a base backup and push it to the standby's system, how to archive (at the satndby) master's WAL files as they are produced and have the standby restore from this archive.

What's new in walmgr from Skytools 3 is its support for Streaming Replication that made its way into PostgreSQL 9.0 and is even more useful in PostgreSQL 9.1 (better monitoring, synchronous replication option).

Getting ready

Now, I'm using debian here, and a build virtual machine where I'm doing the backporting work. As PostgreSQL 9.1 is now out, let's use that.

:~$ pg_lsclusters
Version Cluster   Port Status Owner    Data directory
8.4     main      5432 online postgres /var/lib/postgresql/8.4/main ...
9.0     main      5433 online postgres /var/lib/postgresql/9.0/main ...
9.1     main      5434 online postgres /var/lib/postgresql/9.1/main ...

After some editing of the configuration files (enabling hot standby and switching pg_hba.conf to trust for the sake of this example), we can see that the cluster is ready to be abused:

:~$ sudo pg_ctlcluster 9.1 main restart
:~$ psql --cluster 9.1/main  -U postgres \
-c "select name, setting from pg_settings where name in ('max_wal_senders', 'wal_level')"
      name       |   setting
-----------------+-------------
 max_wal_senders | 1
 wal_level       | hot_standby
(2 rows)

:~$ sudo mkdir -p /etc/walshipping/9.1/main /var/lib/postgresql/walshipping
:~$ sudo chown -R postgres:postgres /etc/walshipping /var/lib/postgresql/walshipping

:~$ ssh-keygen -t dsa
:~/.ssh$ cp id_dsa.pub authorized_keys
:~$ ssh localhost

So the order of operations is to prepare a standby, then have it restore from the archives, then activate the wal streaming and check that the setup allows the standby to switch back and forth between the streaming and the archives.

Setting walmgr

To prepare the standby, we will do a base backup of the master. That step is handled by walmgr, so we first need to set it up. Here's the sample master.ini file:

[walmgr]
job_name             = wal-master
logfile              = /var/log/postgresql/%(job_name)s.log
pidfile              = /var/run/postgresql/%(job_name)s.pid
use_skylog           = 0

master_db            = port=5434 dbname=template1
master_data          = /var/lib/postgresql/9.1/main/
master_config        = /etc/postgresql/9.1/main/postgresql.conf
master_bin           = /usr/lib/postgresql/9.1/bin

# set this only if you can afford database restarts during setup and stop.
master_restart_cmd   = pg_ctlcluster 9.1 main restart

slave = 127.0.0.1
slave_config = /etc/walshipping/9.1/main/standby.ini

walmgr_data          = /var/lib/postgresql/walshipping/9.1/main
completed_wals       = %(walmgr_data)s/logs.complete
partial_wals         = %(walmgr_data)s/logs.partial
full_backup          = %(walmgr_data)s/data.master
config_backup        = %(walmgr_data)s/config.backup

# syncdaemon update frequency
loop_delay           = 10.0
# use record based shipping available since 8.2
use_xlog_functions   = 0

# pass -z to rsync, useful on low bandwidth links
compression          = 0

# keep symlinks for pg_xlog and pg_log
keep_symlinks        = 1

# tell walmgr to set wal_level to hot_standby during setup
#hot_standby          = 1

# periodic sync
#command_interval     = 600
#periodic_command     = /var/lib/postgresql/walshipping/periodic.sh

And the /etc/walshipping/9.1/main/standby.ini companion:

[walmgr]
job_name             = wal-standby
logfile              = /var/log/postgresql/%(job_name)s.log
use_skylog           = 0

slave_data           = /var/lib/postgresql/9.1/standby
slave_bin            = /usr/lib/postgresql/9.1/bin
slave_stop_cmd       = pg_ctlcluster 9.1 standby stop
slave_start_cmd      = pg_ctlcluster 9.1 standby start
slave_config_dir     = /etc/postgresql/9.1/standby/

walmgr_data          = /var/lib/postgresql/walshipping/9.1/main
completed_wals       = %(walmgr_data)s/logs.complete
partial_wals         = %(walmgr_data)s/logs.partial
full_backup          = %(walmgr_data)s/data.master
config_backup        = %(walmgr_data)s/config.backup

backup_datadir       = no
keep_backups         = 0
# archive_command =

# primary database connect string for hot standby -- enabling
# this will cause the slave to be started in hot standby mode.
primary_conninfo     = host=127.0.0.1 port=5434 user=postgres

And let's get started:

:~$ cp standby.ini /etc/walshipping/9.1/main/

:~$ walmgr3 -v master.ini setup
2011-09-21 16:57:05,685 30450 INFO Configuring WAL archiving
2011-09-21 16:57:05,687 30450 DEBUG found 'archive_mode' in config -- enabling it
2011-09-21 16:57:05,687 30450 DEBUG found 'wal_level' in config -- setting to 'archive'
2011-09-21 16:57:05,688 30450 DEBUG modifying configuration: {'archive_mode': 'on', 'wal_level': 'archive', 'archive_command': '/usr/bin/walmgr3 /var/lib/postgresql/master.ini xarchive %p %f'}
2011-09-21 16:57:05,688 30450 DEBUG found parameter archive_mode with value ''off''
2011-09-21 16:57:05,690 30450 DEBUG found parameter wal_level with value ''minimal''
2011-09-21 16:57:05,690 30450 DEBUG found parameter archive_command with value ''''
2011-09-21 16:57:05,691 30450 INFO Restarting postmaster
2011-09-21 16:57:05,691 30450 DEBUG Execute cmd: 'pg_ctlcluster 9.1 main restart'
2011-09-21 16:57:09,404 30450 DEBUG Execute cmd: 'ssh' '-Tn' '-o' 'Batchmode=yes' '-o' 'StrictHostKeyChecking=no' '127.0.0.1' '/usr/bin/walmgr3' '/etc/walshipping/9.1/main/standby.ini' 'setup'
2011-09-21 16:57:09,712 30450 INFO Done

postgres@squeeze64:~$ walmgr3 master.ini backup
2011-09-21 17:00:17,259 30702 INFO Backup lock obtained.
2011-09-21 17:00:17,277 30692 INFO Execute SQL: select pg_start_backup('FullBackup'); [port=5434 dbname=template1]
2011-09-21 17:00:17,791 30712 INFO Removing expired backup directory: /var/lib/postgresql/walshipping/9.1/main/data.master
2011-09-21 17:00:18,200 30692 INFO Checking tablespaces
2011-09-21 17:00:18,202 30692 INFO pg_log does not exist, skipping
2011-09-21 17:00:18,259 30692 INFO Backup conf files from /etc/postgresql/9.1/main
2011-09-21 17:00:18,590 30731 INFO First useful WAL file is: 000000010000000200000092
2011-09-21 17:00:19,901 30759 INFO Backup lock released.
2011-09-21 17:00:19,919 30692 INFO Full backup successful

:~$ walmgr3 /etc/walshipping/9.1/main/standby.ini listbackups

List of backups:

Backup set      Timestamp                Label       First WAL
--------------- ------------------------ ----------- ------------------------
data.master     2011-09-21 17:00:17 CEST FullBackup  000000010000000200000092

Following articles will show how to manage that archive and how to go from that to an Hot Standby fed by either Streaming Replication or Archives.

New features

Among the features you will find dependencies management and M-x el-get-list-packages, that you should try as soon as possible. Of course, don't miss M-x el-get-self-update that eases the process somehow.

This shows the result of M-x el-get-list-packages. The packages that don't have a description are the one from emacswiki that doesn't provide a listing of the filename and the first line of the file (it usually follows the format ;;; filename.el --- description here). As we don't want to mirror the website just to be able to provide descriptions, we just don't have them now.

Another nice new feature, contributed by a user that wanted to self-learn elisp, is the el-get-user-package-directory support. Just place in there some init-my-package.el files, and when el-get wants to init the my-package package, it will load that file for you. That helps managing your setup, and I'm already using that in my own ~/.emacs.d/ repository.

Upgrading

The upgrading is to be done with some care, though, because you need to edit your packaging setup. The el-get-sources variable used to be both where to setup extra recipes and the list of packages you want to have installed, and several people rightfully insisted that I should change that. I've been slow to be convinced, but there it is, they were right.

So now, el-get works from the current status of packages and will init all those packages you have installed. Which means that you just M-x el-get-install a package and don't think about it anymore. If you need to override this behavior, it's still possible to do so by specifying the whole list of packages you want initialized (and installed if necessary) on the (el-get 'sync ...) call.

That later setup is useful if you want to share your el-get selection on several machines.

Nous commençons à lire des articles qui reprennent la nouvelle dans la presse française, et j'ai le plaisir de mentionner celui de programmez.com qui annonce « un système d'extensions inégalé ». En tant que développeur des Extensions dans PostgreSQL, je ne peux qu'être non seulement d'accord avec eux, mais aussi flatté :)

Bons tests à tous, et bonne mises à jour pour les plus chanceux !

Nous faisons face ici à un problème de sécurité très bien décrit dans le billet humoristique de Little Boby Tables, dont je vous recommande la lecture. L'idée est simple, la mise en place de contre mesure fourmille de pièges subtils, à moins d'utiliser la solution décrite ci-après.

Lorsque l'on envoie une requête SQL à PostgreSQL, celle-ci contient pêle-mêle un mélange de mots-clés SQL et de données utilisateurs. Dans la requête SELECT colname FROM table WHERE pk = 1234; l'élément 1234 est une donnée fournie à PostgreSQL. Lorsque l'on utilise d'autre types de données, on va parler de litéral, qui peut être ou non décoré. Un exemple ?

=# SELECT 'undecorated literal', pg_typeof('undecoreted literal'),
          date 'today', pg_typeof(date 'today');
      ?column?       | pg_typeof |    date    | pg_typeof
---------------------+-----------+------------+-----------
 undecorated literal | unknown   | 2011-09-07 | date
(1 row)

Outre l'aspect types de données (un litéral non décoré est de type unknown jusqu'à ce qu'une opération force son type, c'est ce qui permet d'avoir du polymorphisme dans PostgreSQL), nous voyons ici que PostgreSQL doit faire la différence entre le SQL lui-même et les paramètres qui le composent. Il sait bien sûr faire cela, il suffit d'encadrer les valeurs dans des simples guillemets ou bien d'utiliser la notation dite de dollar quoting. Mais si l'on ne prend pas de précautions, l'utilisateur peut terminer la séquence d'échappements depuis le champ de saisie du formulaire…

libpq est la librairie standard cliente de PostgreSQL et fourni des API de connexion et propose une fonction PGexecParams. Cette fonction expose un mécanisme disponible dans le protocole de communication de PostgreSQL lui-même : il est possible de faire parvenir le SQL et les données qu'il contient dans deux parties différentes du messages plutôt que de les mélanger. Ainsi, le serveur n'a plus du tout à deviner où commencent et où terminent les données dans la requête, il lui suffit de regarder dans le tableau séparé contenant les données quand il en a besoin.

Terminées les injections SQL !

Note : cette fonction est exposée dans la plupart des pilotes de connexion, et même en PHP, dont la popularité et l'exposition me poussent à donner une référence plus précise : utilisez pg_query_params, son intérêt n'est pas simplement syntaxique, il va jusque dans la définition des échanges de données entre le client (votre code PHP) et le serveur (PostgreSQL).

Nous faisons face ici à un problème de sécurité très bien décrit dans le billet humoristique de Little Boby Tables, dont je vous recommande la lecture. L'idée est simple, la mise en place de contre mesure fourmille de pièges subtils, à moins d'utiliser la solution décrite ci-après.

Lorsque l'on envoie une requête SQL à PostgreSQL, celle-ci contient pêle-mêle un mélange de mots-clés SQL et de données utilisateurs. Dans la requête SELECT colname FROM table WHERE pk = 1234; l'élément 1234 est une donnée fournie à PostgreSQL. Lorsque l'on utilise d'autre types de données, on va parler de litéral, qui peut être ou non décoré. Un exemple ?

=# SELECT 'undecorated literal', pg_typeof('undecoreted literal'),
          date 'today', pg_typeof(date 'today');
      ?column?       | pg_typeof |    date    | pg_typeof
---------------------+-----------+------------+-----------
 undecorated literal | unknown   | 2011-09-07 | date
(1 row)

Outre l'aspect types de données (un litéral non décoré est de type unknown jusqu'à ce qu'une opération force son type, c'est ce qui permet d'avoir du polymorphisme dans PostgreSQL), nous voyons ici que PostgreSQL doit faire la différence entre le SQL lui-même et les paramètres qui le composent. Il sait bien sûr faire cela, il suffit d'encadrer les valeurs dans des simples guillemets ou bien d'utiliser la notation dite de dollar quoting. Mais si l'on ne prend pas de précautions, l'utilisateur peut terminer la séquence d'échappements depuis le champ de saisie du formulaire…

libpq est la librairie standard cliente de PostgreSQL et fourni des API de connexion et propose une fonction PGexecParams. Cette fonction expose un mécanisme disponible dans le protocole de communication de PostgreSQL lui-même : il est possible de faire parvenir le SQL et les données qu'il contient dans deux parties différentes du messages plutôt que de les mélanger. Ainsi, le serveur n'a plus du tout à deviner où commencent et où terminent les données dans la requête, il lui suffit de regarder dans le tableau séparé contenant les données quand il en a besoin.

Terminées les injections SQL !

Note : cette fonction est exposée dans la plupart des pilotes de connexion, et même en PHP, que la popularité et l'exposition me poussent à donner une référence plus précise : utilisez pg_query_params, son intérêt n'est pas simplement syntaxique, il va jusque dans la définition des échanges de données entre le client (votre code PHP) et le serveur (PostgreSQL).

We're now thinking to support the i386 and amd64 architectures for lenny, squeeze, wheezy and sid, and also for maverick and natty, maybe oneiric too while at it.

It's still the very beginning of the effort, and it was triggered by the decision to move sid to 9.1. While it's a good decision in itself, I still hate to have to pick only one PostgreSQL version per debian stable release when we have all the technical support we need to be able to support all stable releases that upstream is willing to maintain. If you've been living under a rock, or if you couldn't care less about debian choices, the problem here for debian is ensuring security (and fixes) updates for PostgreSQL — they promise they will handle the job just fine in the social contract, and don't want to have to it without support from PostgreSQL if a debian stable release contains a deprecated PostgreSQL version.

That opens the door for PostgreSQL community to handle the packaging of its solutions as a service to its debian users. We intend to open with support for 8.4, 9.0 and 9.1, and maybe 8.3 too, as Christoph Berg is doing good progress on this front. See, it's teamwork here!

We still have more work to do, and setting up the build environment so that we are able to provide the packages for so much targets will indeed be interesting. Getting there, a step after another.

The pg_restore list is just a listing of one object per line of all objects contained into a custom dump, that is one made with pg_dump -Fc. You can then tweak this listing in order to comment out some objects (prepending a ; to the line where you find it), and give your hacked file back to pg_restore --use-list so that it will skip them.

What's pretty useful here, among other things, is that a table will have in fact more than one line in the listing. One is for the TABLE definition, another one for the TABLE DATA. So that pg_staging is able to provide you with options for only restoring some schemas, some schemas_nodata and even some tablename_nodata_regexp, to use directly the configuration options names.

How to do a very simple exclusion of some table's data when restoring a dump, will you ask me? There we go. Let's first prepare an environment, where I have only a PostgreSQL server running.

$ git clone git://github.com/dimitri/pg_staging.git
$ git clone git://github.com/dimitri/pgloader.git
$ for s in */*.sql; do psql -f $s; done
$ pg_dump -Fc > pgloader.dump

Now I have a dump with some nearly random SQL objects in it, let's filter out the tables named reformat and parallel from that. We will take the sample setup from the pg_staging project. Going the quick route, we will not even change the default sample database name that's used, which is postgres. After all, the catalog command of pg_staging that we're using here is a developer command, you're supposed to be using pg_staging for a lot more services that just this one.

$ cp pg_staging/pg_staging.ini .
$ (echo "schemas = public";
   echo "tablename_nodata_regexp = parallel,reformat") \
  >> pg_staging.ini
$ echo "catalog postgres pgloader.dump" \
   | python pg_staging/pg_staging.py -c pg_staging.ini
 ; Archive created at Mon Aug 29 17:17:49 2011
 ;
 ; [EDITED OUTPUT]
 ;
 ; Selected TOC Entries:
 ;
3; 2615 2200 SCHEMA - public postgres
1864; 0 0 COMMENT - SCHEMA public postgres
1536; 1259 174935 TABLE public parallel dimitri
1537; 1259 174943 TABLE public partial dimitri
1538; 1259 174951 TABLE public reformat dimitri
;1853; 0 174935 TABLE DATA public parallel dimitri
1854; 0 174943 TABLE DATA public partial dimitri
;1855; 0 174951 TABLE DATA public reformat dimitri
1834; 2606 174942 CONSTRAINT public parallel_pkey dimitri
1836; 2606 174950 CONSTRAINT public partial_pkey dimitri
1838; 2606 174955 CONSTRAINT public reformat_pkey dimitri

We can see that the objects indeed are skipped, now how to really go about the pg_restore is like that:

$ createdb foo
$ echo "catalog postgres pgloader.dump" \
 |python pg_staging/pg_staging.py -c pg_staging.ini > short.list
$ pg_restore -L short.list -d foo pgloader.dump

The little bonus with using pg_staging is that when filtering out a schema it will track all tables and triggers from that schema, and also the functions used in the trigger definition. Which is not as easy as it sounds, believe me!

The practical use case is when filtering out PGQ and Londiste, then the PGQ triggers will automatically be skipped by pg_staging rather than polluting the pg_restore logs because the CREATE TRIGGER command could not find the necessary implementation procedure.

Here are the slides from the CHAR(11) talk I made last month, about that very subject:

The new version comes with a lot of new features. PGQ now is able to duplicate the queue events from one node to the next, so that it's able to manage switching over. To do that we have three types of nodes now, root, branch and leaf. PGQ also supports cooperative consumers, meaning that you can share the processing load among many consumers, or workers.

Londiste now benefits from the switch over feature, and is packed with new little features like add <table> --create, the new --trigger-flags argument, and the new --handler thing (to do e.g. partial table replication). Let's not forget the much awaited execute <script> command that allows to include DDL commands into the replication stream, nor the parallel COPY support that will boost your initial setup.

walmgr in the new version behaves correctly when using PostgreSQL 9.0. Meaning that as soon as no more WAL files are available in the archives, it returns an error code to the archiver so that the server switches to streaming live from the primary_conninfo, then back to replaying the files from the archive if the connection were to fail, etc. All in all, it just works.

Details to follow here, stay tuned!

Rather than talking about what pgfincore is all about (A set of functions to manage pages in memory from PostgreSQL), I will talk about its packaging and support as a debian package. Here's the first example of a modern multi-version packaging I have to offer. pgfincore packaging supports building for 8.4 and 9.0 and 9.1 out of the box, even if the only binary you'll find in debian sid is the 9.0 one, as you can check on the pgfincore debian source package page.

Also, this is the first package I've done properly using the newer version of debhelper, which make the debian/rules file easier than ever. Let's have a look at it:

SRCDIR = $(CURDIR)
TARGET = $(CURDIR)/debian/pgfincore-%v
PKGVERS = $(shell dpkg-parsechangelog | awk -F '[:-]' '/^Version:/ { print substr($$2, 2) }')
EXCLUDE = --exclude-vcs --exclude=debian

include /usr/share/postgresql-common/pgxs_debian_control.mk

override_dh_auto_clean: debian/control
        pg_buildext clean $(SRCDIR) $(TARGET) "$(CFLAGS)"
        dh_clean

override_dh_auto_build:
        # build all supported version
        pg_buildext build $(SRCDIR) $(TARGET) "$(CFLAGS)"

override_dh_auto_install:
        # then install each of them
        for v in `pg_buildext supported-versions $(SRCDIR)`; do \
                dh_install -ppostgresql-$$v-pgfincore ;\
        done

orig: clean
        cd .. && tar czf pgfincore_$(PKGVERS).orig.tar.gz $(EXCLUDE) pgfincore

%:
        dh $@

The debian/rules file is known to be the corner stone of your debian packaging, and usually is the most complex part of it. It's a Makefile at its heart, and we can see that thanks to the debhelper magic it's not that complex to maintain anymore.

Then, this file is using support from a bunch of helpers command, each of them comes with its own man page and does a little part of the work. The overall idea around debhelper is that what it does covers 90% of the cases around, and it's not aiming for more. You have to override the parts where it defaults to being wrong.

Here for example the build system has to produce files for all three supported versions of PostgreSQL, which means invoking the same build system three time with some changes in the environment (mainly setting the PG_CONFIG variable correctly). But even for that we have a debian facility, that comes in the package postgresql-server-dev-all, called pg_buildext. As long as your extension build system is VPATH friendly, it's all automated.

Please read that last sentence another time. VPATH is the thing that allows Make to find your source tree somewhere in the system, not in the current working directory. That allows you to cleanly build the same sources in different build locations, which is exactly what we need here, and is cleanly supported by PGXS, the PostgreSQL Extension Building Infrastructure.

Which means that the main Makefile of pgfincore had to be simplified, and the code layout too. Some advances Make features such as $(wildcard ...) and all will not work here. See what we got at the end:

ifndef VPATH
SRCDIR = .
else
SRCDIR = $(VPATH)
endif

EXTENSION    = pgfincore
EXTVERSION   = $(shell grep default_version $(SRCDIR)/$(EXTENSION).control | \
               sed -e "s/default_version[[:space:]]*=[[:space:]]*'\([^']*\)'/\1/")

MODULES      = $(EXTENSION)
DATA         = sql/pgfincore.sql sql/uninstall_pgfincore.sql
DOCS         = doc/README.$(EXTENSION).rst

PG_CONFIG    = pg_config

PG91         = $(shell $(PG_CONFIG) --version | grep -qE "8\.|9\.0" && echo no || echo yes)

ifeq ($(PG91),yes)
all: pgfincore--$(EXTVERSION).sql

pgfincore--$(EXTVERSION).sql: sql/pgfincore.sql
        cp $< $@

DATA        = pgfincore--unpackaged--$(EXTVERSION).sql pgfincore--$(EXTVERSION).sql
EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql
endif

PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

deb:
        dh clean
        make -f debian/rules orig
        debuild -us -uc -sa

No more Make magic to find source files. Franckly though, when your sources are 1 c file and 2 sql files, you don't need that much magic anyway. You just want to believe that a single generic Makefile will happily build any project you throw at it, only requiring minor adjustment. Well, the reality is that you might need some more little adjustments if you want to benefit from VPATH building, and having the binaries for 8.4 and 9.0 and 9.1 built seemlessly in a simple loop. Like we have here for pgfincore.

Now the Makefile still contains a little bit of magic, in order to parse the extension version number from its control file and produce a script named accordingly. Then you'll notice a difference between the postgresql-9.1-pgfincore.install file and the postgresql-9.0-pgfincore.install. We're just not shipping the same files:

debian/pgfincore-9.0/pgfincore.so usr/lib/postgresql/9.0/lib
sql/pgfincore.sql usr/share/postgresql/9.0/contrib
sql/uninstall_pgfincore.sql usr/share/postgresql/9.0/contrib

As you can see here:

debian/pgfincore-9.1/pgfincore.so usr/lib/postgresql/9.1/lib
debian/pgfincore-9.1/pgfincore*.sql usr/share/postgresql/9.1/extension
sql/pgfincore--unpackaged--1.0.sql usr/share/postgresql/9.1/extension

So, now that we uncovered all the relevant magic, packaging and building your next extension so that it supports as many PostgreSQL major releases as you need to will be that easy.

For reference, you might need to also tweak /usr/share/postgresql-common/supported-versions so that it allows you to build for all those versions you claim to support in the debian/pgversions file.

$ sudo dpkg-divert \
--divert /usr/share/postgresql-common/supported-versions.distrib \
--rename /usr/share/postgresql-common/supported-versions

$ cat /usr/share/postgresql-common/supported-versions
#! /bin/bash

dpkg -l postgresql-server-dev-* \
| awk -F '[ -]' '/^ii/ && ! /server-dev-all/ {print $6}'

All of this will come pretty handy when we finally sit down and work on a way to provide binary packages for PostgreSQL and its extensions, and all supported versions of those at that. This very project is not dead, it's just sleeping some more.

En effet, l'utilisation d'échappements avec le caractère « anti-slash » n'est pas conforme au standard SQL. Le paramètre standard_conforming_strings permet de contrôler le comportement de PostgreSQL lorsqu'il lit une chaîne de caractère dans une requête SQL.

Voyons quelques exemples :

dimitri=# set standard_conforming_strings to true;
SET
dimitri=# select 'hop''';
 ?column?
----------
 hop'
(1 ligne)

dimitri=# select 'hop\'';
dimitri'# ';
 ?column?
----------
 hop\';

(1 ligne)

dimitri=# select E'hop\'';
 ?column?
----------
 hop'
(1 ligne)

dimitri=# set standard_conforming_strings to false;
SET
dimitri=# select E'hop\'';
 ?column?
----------
 hop'
(1 ligne)

dimitri=# select 'hop\'';
ATTENTION:  utilisation non standard de \' dans une chaîne littérale
LIGNE 1 : select 'hop\'';
                 ^
ASTUCE : Utilisez '' pour écrire des guillemets dans une chaîne ou utilisez
la syntaxe de chaîne d'échappement (E'...').
 ?column?
----------
 hop'
(1 ligne)

Il existe un moyen de forcer PostgreSQL à accepter l'utilisation d'échappements avec « anti-slash » indépendamment de la valeur de standard_conforming_strings, c'est la notation préfixée avec E. Il est recommandé de toujours l'utiliser dès lors que la chaîne de caractère contient des « anti-slash » utilisés comme échappement (du caractère simple guillemet en général).

Le paramètre escape_string_warning, enfin, permet de désactiver les avertissements tels que présentés dans le dernier exemple ci-dessus, lorsqu'il est positionné à off. Bien sûr, sa valeur par défaut est on.

Toute apparition de ce WARNING lorsque escape_string_warning est on signifie que votre application n'est pas prête à migrer à 9.1 avec son paramétrage par défaut. Il existe deux actions possible : changer le paramétrage de sa nouvelle valeur par défaut à sa précédente, ou bien corriger ses applications pour utiliser le préfixe E dès que cela est nécessaire.

L'utilisation de standard_conforming_strings à on présente un autre avantage au respect du standard SQL : la sécurité contre les injections. S'il n'est pas possible d'échapper le guillemet simple qui termine toute chaîne de caractère utilisateur, il devient compliqué de jouer au plus malin avec le parser. Le mieux ici reste bien sûr d'utiliser les requêtes paramétrées, à suivre dans un prochain article.

Well, the joy of Open Source & Free Software (pick your own poison). jglee1027 is this GitHub guy who did offer an implementation of said facility, and who added descriptions for almost all of the now 402 recipes that we have included with el-get.

Here's an image of what you get:

The packages with no description are fetched by M-x el-get-emacswiki-refresh which will not download all emacswiki content locally just so that it can parse the scripts's header and have a local description. Maybe it's time to ask for another page over there like emacswiki page index but containing the first line too.

For recipes we offer, this first line often looks like the following:

;;; 123-menu.el --- Simple menuing system, reminiscent of Lotus 123 in DOS

Of course some files over there are not following the stanza, but that would be good enough already.

All in all, I hope you enjoy M-x el-get-list-packages!

The basic situation where you need to do so is adding an origin field to your table. The value of that is not to be found in the data file itself, typically, but known in the pgloader setup. That could even be the filename you are importing data from.

In pgloader that's called a user defined column. Here's what the relevant examples/pgloader.conf setup looks like:

[udc]
table           = udc
format          = text
filename        = udc/udc.data
input_encoding  = 'latin1'
field_sep       = %
columns         = b:2, d:1, x:3, y:4
udc_c           = constant value
copy_columns    = b, c, d

And the data file is:

1%5%foo%bar
2%10%bar%toto
3%4%toto%titi
4%18%titi%baz
5%2%baz%foo

And here's what the loaded table looks like:

pgloader/examples$ pgloader -Tsc pgloader.conf udc
Table name        |    duration |    size |  copy rows |     errors
====================================================================
udc               |      0.201s |       - |          5 |          0

pgloader/examples$ psql --cluster 8.4/main pgloader -c "table udc"
 b  |       c        | d
----+----------------+---
  5 | constant value | 1
 10 | constant value | 2
  4 | constant value | 3
 18 | constant value | 4
  2 | constant value | 5
(5 rows)

Of course the configuration is not so straightforward as to process fields in the data file in the order that they appear, after all the examples/pgloader.conf are also a test suite.

Long story short: if you need to add some constant values into the target table you're loading data to, pgloader will help you there!

The basic situation where you need to do so is adding an origin field to your table. The value of that is not to be found in the data file itself, typically, but known in the pgloader setup. That could even be the filename you are importing data from.

In pgloader that's called a user defined column. Here's what the relevant examples/pgloader.conf setup looks like:

[udc]
table           = udc
format          = text
filename        = udc/udc.data
input_encoding  = 'latin1'
field_sep       = %
columns         = b:2, d:1, x:3, y:4
udc_c           = constant value
copy_columns    = b, c, d

And the data file is:

1%5%foo%bar
2%10%bar%toto
3%4%toto%titi
4%18%titi%baz
5%2%baz%foo

And here's what the loaded table looks like:

pgloader/examples$ pgloader -Tsc pgloader.conf udc
Table name        |    duration |    size |  copy rows |     errors
====================================================================
udc               |      0.201s |       - |          5 |          0

pgloader/examples$ psql --cluster 8.4/main pgloader -c "table udc"
 b  |       c        | d
----+----------------+---
  5 | constant value | 1
 10 | constant value | 2
  4 | constant value | 3
 18 | constant value | 4
  2 | constant value | 5
(5 rows)

Of course the configuration is not so straightforward as to process fields in the data file in the order that they appear, after all the examples/pgloader.conf are also a test suite.

Long story short: if you need to add some constant values into the target table you're loading data to, pgloader will help you there!

Some users are even starting Emacs often enough for the startup time to be a concern. With an emacs-uptime (yes it's a command, you can M-x emacs-uptime) of days to weeks (10 days, 17 hours, 45 minutes, 34 seconds as of this writing), it's not something I really care about much.

But I know that some el-get users still do care, and will use el-get-is-lazy and do all their Emacs tweaking as eval-after-load blocks. Trying to have an idea of how much a worst case startup with el-get is, I have added the following piece of elisp at the very end of my startup code:

(defun dim:notify-startup-done ()
  " notify user that Emacs is now ready"
  (el-get-notify
   "Emacs is ready."
   (format "The init sequence took %g seconds."
           (float-time (time-subtract after-init-time before-init-time)))))

(add-hook 'after-init-hook 'dim:notify-startup-done)

The el-get-notify function will adapt and either use the dbus implementation from Emacs 24, or notify.el from EmacsWiki (just M-x el-get-install it if you need it), or will use its own implementation of an Emacs Growl client (it's about 5 lines long), and baring all of that will use the message function.

The reason I say worst case is that I have a lot of packages to initialize at startup, and that I did absolutely no effort for this initializing to be quick. Still, my Emacs setup is taking about 20 seconds to boot. Pretty good I would say, for a weekly operation.

Some users are even starting Emacs often enough for the startup time to be a concern. With an emacs-uptime (yes it's a command, you can M-x emacs-uptime) of days to weeks (10 days, 17 hours, 45 minutes, 34 seconds as of this writing), it's not something I really care about much.

But I know that some el-get users still do care, and will use el-get-is-lazy and do all their Emacs tweaking as eval-after-load blocks. Trying to have an idea of how much a worst case startup with el-get is, I have added the following piece of elisp at the very end of my startup code:

(defun dim:notify-startup-done ()
  " notify user that Emacs is now ready"
  (el-get-notify
   "Emacs is ready."
   (format "The init sequence took %g seconds."
           (float-time (time-subtract after-init-time before-init-time)))))

(add-hook 'after-init-hook 'dim:notify-startup-done)

The el-get-notify function will adapt and either use the dbus implementation from Emacs 24, or notify.el from EmacsWiki (just M-x el-get-install it if you need it), or will use its own implementation of an Emacs Growl client (it's about 5 lines long), and baring all of that will use the message function.

The reason I say worst case is that I have a lot of packages to initialize at startup, and that I did absolutely no effort for this initializing to be quick. Still, my Emacs setup is taking about 20 seconds to boot. Pretty good I would say, for a weekly operation.

Here's what the pgloader documentation has to say about this reformat parameter: The value of this option is a comma separated list of columns to rewrite, which are a colon separated list of column name, reformat module name, reformat function name.

And here's the examples/pgloader.conf section that deals with reformat:

[reformat]
table           = reformat
format          = text
filename        = reformat/reformat.data
field_sep       = |
columns         = id, timestamp
reformat        = timestamp:mysql:timestamp

The documentation says some more about it, so check it out. Also, the reformat_path option (set either on the command line or in the configuration file) is used to find the python module implementing the reformat function. Please refer to the manual as to how to set it.

Now, obviously, for the reformat to happen we need to write some code. That's the whole point of the option: you need something very specific, you are in a position to write the 5 lines of code needed to make it happen, pgloader allows you to just do that. Of course, the code needs to be written in python here, so that you can even benefit from the parallel pgloader settings.

Let's see an reformat module exemple, as found in reformat/mysql.py in the pgloader sources:

# Author: Dimitri Fontaine <dim@tapoueh.org>
#
# pgloader mysql reformating module
#

def timestamp(reject, input):
    """ Reformat str as a PostgreSQL timestamp

    MySQL timestamps are like:  20041002152952
    We want instead this input: 2004-10-02 15:29:52
    """
    if len(input) != 14:
        e = "MySQL timestamp reformat input too short: %s" % input
        reject.log(e, input)

    year    = input[0:4]
    month   = input[4:6]
    day     = input[6:8]
    hour    = input[8:10]
    minute  = input[10:12]
    seconds = input[12:14]

    return '%s-%s-%s %s:%s:%s' % (year, month, day, hour, minute, seconds)

This reformat module will transform a timestamp representation as issued by certain versions of MySQL into something that PostgreSQL is able to read as a timestamp.

If you're in the camp that wants to write as little code as possible rather than easy to read and maintain code, I guess you could write it this way instead:

import re
def timestamp(reject, input):
    """ 20041002152952 -> 2004-10-02 15:29:52 """
    g = re.match(r"(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})", input)
    return '%s-%s-%s %s:%s:%s' % tuple([g.group(x+1) for x in range(6)])

Whenever you have an input file with data that PostgreSQL chokes upon, you can solve this problem from pgloader itself: no need to resort to scripting and a pipelines of awk (which I use a lot in other cases, don't get me wrong) or other tools. See, you finally have an excuse to Dive into Python!

When you want to benchmark your own application, to know how many more clients it can handle or how much gain you will see with some new shiny hardware, Tsung is the tool to use. It will allow you to record a number of sessions then replay them at high scale. pgfouine supports Tsung and is able to turn your PostgreSQL logs into Tsung sessions, too.

Tsung did get used in the video game world, their version of it is called uTsung, apparently using the uLink game development facilities. They even made a video demo of uTsung, that you might find interresting:

several files at a time

Parallelism is implemented in 3 different ways in pgloader. First, you can load more than one file at a time thanks to the max_parallel_sections parameter, that has to be setup in the global section of the file.

This setting is quite simple and already allows the most common use case.

several workers per file

The other use case is when you have huge files to load into the database. Then you want to be able to have more than one process reading the file at the same time. Using pgloader, you already did the compromise to load the whole content in more than one transaction, so there's no further drawback here about having those multiple transactions per file spread to more than one load worker.

There are basically two ways to split the work between several workers here, and both are implemented in pgloader.

N workers, N splits of the file

section_threads    = 4
split_file_reading = True

Setup this way, pgloader will launch 4 different threads (see the caveat section of this article). Each thread is then given a part of the input data file and will run the whole usual pgloader processing on its own. For this to work you need to be able to seek in the input stream, which might not always be convenient.

one reader, N workers

section_threads    = 4
split_file_reading = False
rrqueue_size       = 5000

With such a setup, pgloader will start 4 different worker threads that will receive the data input in an internal python queue. Another active thread will be responsible of reading the input file and filling the queues in a round robin fashion, but will hand all the processing of the data to each worker, of course.

how many threads?

If you're using a mix and match of max_parallel_sections and section_threads with split_file_reading set to True of False, it's uneasy to know exactly how many threads will run at any time in the loading. How to ascertain which section will run in parallel when it depends on the timing of the loading?

The advice here is the usual one, don't overestimate the capabilities of your system unless you are in a position to check before by doing trial runs.

caveat

Current implementation of all the parallelism in pgloader has been done with the python threading API. While this is easy enough to use when you want to exchange data between threads, it's suffering from the Global Interpreter Lock issue. This means that while the code is doing its processing in parallel, the runtime not so much. You might still benefit from the current implementation if you have hard to parse files, or custom reformat modules that are part of the loading bottleneck.

future

The solution would be to switch to using the newer python multiprocessing API, and some preliminary work has been done in pgloader to allow for that. If you're interested in real parallel bulk loading, contact-me!

several files at a time

Parallelism is implemented in 3 different ways in pgloader. First, you can load more than one file at a time thanks to the max_parallel_sections parameter, that has to be setup in the global section of the file.

This setting is quite simple and already allows the most common use case.

several workers per file

The other use case is when you have huge files to load into the database. Then you want to be able to have more than one process reading the file at the same time. Using pgloader, you already did the compromise to load the whole content in more than one transaction, so there's no further drawback here about having those multiple transactions per file spread to more than one load worker.

There are basically two ways to split the work between several workers here, and both are implemented in pgloader.

N workers, N splits of the file

section_threads    = 4
split_file_reading = True

Setup this way, pgloader will launch 4 different threads (see the caveat section of this article). Each thread is then given a part of the input data file and will run the whole usual pgloader processing on its own. For this to work you need to be able to seek in the input stream, which might not always be convenient.

one reader, N workers

section_threads    = 4
split_file_reading = False
rrqueue_size       = 5000

With such a setup, pgloader will start 4 different worker threads that will receive the data input in an internal python queue. Another active thread will be responsible of reading the input file and filling the queues in a round robin fashion, but will hand all the processing of the data to each worker, of course.

how many threads?

If you're using a mix and match of max_parallel_sections and section_threads with split_file_reading set to True of False, it's uneasy to know exactly how many threads will run at any time in the loading. How to ascertain which section will run in parallel when it depends on the timing of the loading?

The advice here is the usual one, don't overestimate the capabilities of your system unless you are in a position to check before by doing trial runs.

caveat

Current implementation of all the parallelism in pgloader has been done with the python threading API. While this is easy enough to use when you want to exchange data between threads, it's suffering from the Global Interpreter Lock issue. This means that while the code is doing its processing in parallel, the runtime not so much. You might still benefit from the current implementation if you have hard to parse files, or custom reformat modules that are part of the loading bottleneck.

future

The solution would be to switch to using the newer python multiprocessing API, and some preliminary work has been done in pgloader to allow for that. If you're interested in real parallel bulk loading, contact-me!

Bonne lecture !

This file is expected in the INI format, with a global section then one section per file you want to import. The global section defines some default options and how to connect to the PostgreSQL server.

The configuration setup is fully documented on the pgloader man page that you can even easily find online. As all unix style man pages, though, it's more a complete reference than introductory material. Let's review.

global section

Here's the global section of the examples/pgloader.conf file of the source files. Well, some options are debugger only options, really, so I changed their value so that what you see here is a better starting point.

[pgsql]
base = pgloader

log_file            = /tmp/pgloader.log
log_min_messages    = INFO
client_min_messages = WARNING

lc_messages         = C
pg_option_client_encoding = 'utf-8'
pg_option_standard_conforming_strings = on
pg_option_work_mem = 128MB

copy_every      = 15000

null         = ""
empty_string = "\ "

max_parallel_sections = 4

You don't see all the connection setup, here base was enough. You might need to setup host, port and user, and maybe even pass, too, to be able to connect to the PostgreSQL server.

The logging options allows you to set a file where to log all pgloader messages, that are categorized as either DEBUG, INFO, WARNING, ERROR or CRITICAL. The options log_min_messages and client_min_messages are another good idea stolen from PostgreSQL and allow you to setup the level of chatter you want to see on the interactive console (standard output and standard error streams) and on the log file.

Please note that the DEBUG level will produce more that 3 times as many data as the data file you're importing. If you're not a pgloader contributor or helping them, well, debug it, you want to avoid setting the log chatter to this value.

The client_encoding will be SET by pgloader on the PostgreSQL connection it establish. You can now even set any parameter you want by using the pg_option_parameter_name magic settings. Note that the command line option --pg-options (or -o for brevity) allows you to override that.

Then, the copy_every parameter is set to 5 in the examples, because the test files are containing less than 10 lines and we want to test several batches of commits when using them. So for your real loading, stick to default parameters (10 000 lines per COPY command), or more. You can play with this parameter, depending on the network (or local access) and disk system you're using you might see improvements by reducing it or enlarging it. There's no so much theory of operation as empirical testing and setting here. For a one-off operation, just remove the lines from the configuration.

The parameters null and empty_string are related to interpreting the data in the text or csv files you have, and the documentation is quite clear about them. Note that you have global setting and per-section setting too.

The last parameter of this example, max_parallel_sections, is detailed later in the article.

files section

After the global section come as many sections as you have file to load. Plus the template sections, that are only there so that you can share a bunch of parameters in more than one section. Picture a series of data file all of the same format, the only thing that will change is the filename. Use a template section in this case!

Let's see an example:

[simple_tmpl]
template     = True
format       = text
datestyle    = dmy
field_sep    = |
trailing_sep = True

[simple]
use_template    = simple_tmpl
table           = simple
filename        = simple/simple.data
columns         = a:1, b:3, c:2
skip_head_lines = 2

# those reject settings are defaults one
reject_log   = /tmp/simple.rej.log
reject_data  = /tmp/simple.rej

[partial]
table        = partial
format       = text
filename     = partial/partial.data
field_sep    = %
columns      = *
only_cols    = 1-3, 5

That's 2 of the examples from the examples/pgloader.conf file, in 3 sections so that we see one template example. Of course, having a single section using the template, it's just here for the example.

data file format

The most important setting that you have to care about is the file format. Your choice here is either text, csv or fixed. Mostly, what we are given nowadays is csv. You might remember having read that the nice thing about standards is that there's so many to choose from... well, the csv land is one where it's pretty hard to find different producers that understand it the same way.

So when you fail to have pgloader load your mostly csv files with a csv setup, it's time to consider using text instead. The text file format accept a lot of tunables to adapt to crazy situations, but is all python code when the python csv module is a C-coded module, more efficient.

If you're wondering what kind of format we're talking about here, here's the cluttered pgloader example for your reading pleasure, using ^ (carret) as the field separator:

1^some multi\
line text with\
newline escaping^and some other data following^
2^and another line^clean^
3^and\
a last multiline\
escaped line
with a missing\
escaping^just to test^
4^\ ^empty value^
5^^null value^
6^multi line\
escaped value\
\
with empty line\
embeded^last line^

And here's what we get by loading that:

pgloader/examples$ pgloader -c pgloader.conf -s cluttered
Table name        |    duration |    size |  copy rows |     errors
====================================================================
cluttered         |      0.193s |       - |          6 |          0

pgloader/examples$ psql pgloader -c "table cluttered;"
 a |               b               |        c
---+-------------------------------+------------------
 1 | and some other data following | some multi
                                   : line text with
                                   : newline escaping
 2 | clean                         | and another line
 3 | just to test                  | and
                                   : a last multiline
                                   : escaped line
                                   : with a missing
                                   : escaping
 4 | empty value                   |
 5 | null value                    |
 6 | last line                     | multi line
                                   : escaped value
                                   :
                                   : with empty line
                                   : embeded
(6 rows)

So when you have such kind of data, well, it might be that pgloader is still able to help you!

Please refer to the pgloader man page to know about each and every parameter that you can define and the values accepted, etc. And the fixed data format is to be used when you're not given a field separator but field positions in the file. Yes, we still encounter those from time to time. Who needs variable size storage, after all?

Now, a problem that I still had to solve was the colors used in the terminal. As I'm using the tango color theme for emacs, the default ANSI palette's blue color was not readable. Here's how to fix that:

   (require 'ansi-color)
   (setq ansi-color-names-vector
         (vector (frame-parameter nil 'background-color)
               "#f57900" "#8ae234" "#edd400" "#729fcf"
               "#ad7fa8" "cyan3" "#eeeeec")
         ansi-term-color-vector ansi-color-names-vector
         ansi-color-map (ansi-color-make-color-map))

Now your colors in an emacs terminal are easy to read, as you can see:

Hope you enjoy!

Now, a problem that I still had to solve was the colors used in the terminal. As I'm using the tango color theme for emacs, the default ANSI palette's blue color was not readable. Here's how to fix that:

   (require 'ansi-color)
   (setq ansi-color-names-vector
         (vector (frame-parameter nil 'background-color)
               "#f57900" "#8ae234" "#edd400" "#729fcf"
               "#ad7fa8" "cyan3" "#eeeeec")
         ansi-term-color-vector ansi-color-names-vector
         ansi-color-map (ansi-color-make-color-map))

Now your colors in an emacs terminal are easy to read, as you can see:

Hope you enjoy!

This file is expected in the INI format, with a global section then one section per file you want to import. The global section defines some default options and how to connect to the PostgreSQL server.

The configuration setup is fully documented on the pgloader man page that you can even easily find online. As all unix style man pages, though, it's more a complete reference than introductory material. Let's review.

global section

Here's the global section of the examples/pgloader.conf file of the source files. Well, some options are debugger only options, really, so I changed their value so that what you see here is a better starting point.

[pgsql]
base = pgloader

log_file            = /tmp/pgloader.log
log_min_messages    = INFO
client_min_messages = WARNING

lc_messages         = C
pg_option_client_encoding = 'utf-8'
pg_option_standard_conforming_strings = on
pg_option_work_mem = 128MB

copy_every      = 15000

null         = ""
empty_string = "\ "

max_parallel_sections = 4

You don't see all the connection setup, here base was enough. You might need to setup host, port and user, and maybe even pass, too, to be able to connect to the PostgreSQL server.

The logging options allows you to set a file where to log all pgloader messages, that are categorized as either DEBUG, INFO, WARNING, ERROR or CRITICAL. The options log_min_messages and client_min_messages are another good idea stolen from PostgreSQL and allow you to setup the level of chatter you want to see on the interactive console (standard output and standard error streams) and on the log file.

Please note that the DEBUG level will produce more that 3 times as many data as the data file you're importing. If you're not a pgloader contributor or helping them, well, debug it, you want to avoid setting the log chatter to this value.

The client_encoding will be SET by pgloader on the PostgreSQL connection it establish. You can now even set any parameter you want by using the pg_option_parameter_name magic settings. Note that the command line option --pg-options (or -o for brevity) allows you to override that.

Then, the copy_every parameter is set to 5 in the examples, because the test files are containing less than 10 lines and we want to test several batches of commits when using them. So for your real loading, stick to default parameters (10 000 lines per COPY command), or more. You can play with this parameter, depending on the network (or local access) and disk system you're using you might see improvements by reducing it or enlarging it. There's no so much theory of operation as empirical testing and setting here. For a one-off operation, just remove the lines from the configuration.

The parameters null and empty_string are related to interpreting the data in the text or csv files you have, and the documentation is quite clear about them. Note that you have global setting and per-section setting too.

The last parameter of this example, max_parallel_sections, is detailed later in the article.

files section

After the global section come as many sections as you have file to load. Plus the template sections, that are only there so that you can share a bunch of parameters in more than one section. Picture a series of data file all of the same format, the only thing that will change is the filename. Use a template section in this case!

Let's see an example:

[simple_tmpl]
template     = True
format       = text
datestyle    = dmy
field_sep    = |
trailing_sep = True

[simple]
use_template    = simple_tmpl
table           = simple
filename        = simple/simple.data
columns         = a:1, b:3, c:2
skip_head_lines = 2

# those reject settings are defaults one
reject_log   = /tmp/simple.rej.log
reject_data  = /tmp/simple.rej

[partial]
table        = partial
format       = text
filename     = partial/partial.data
field_sep    = %
columns      = *
only_cols    = 1-3, 5

That's 2 of the examples from the examples/pgloader.conf file, in 3 sections so that we see one template example. Of course, having a single section using the template, it's just here for the example.

data file format

The most important setting that you have to care about is the file format. Your choice here is either text, csv or fixed. Mostly, what we are given nowadays is csv. You might remember having read that the nice thing about standards is that there's so many to choose from... well, the csv land is one where it's pretty hard to find different producers that understand it the same way.

So when you fail to have pgloader load your mostly csv files with a csv setup, it's time to consider using text instead. The text file format accept a lot of tunables to adapt to crazy situations, but is all python code when the python csv module is a C-coded module, more efficient.

If you're wondering what kind of format we're talking about here, here's the cluttered pgloader example for your reading pleasure, using ^ (carret) as the field separator:

1^some multi\
line text with\
newline escaping^and some other data following^
2^and another line^clean^
3^and\
a last multiline\
escaped line
with a missing\
escaping^just to test^
4^\ ^empty value^
5^^null value^
6^multi line\
escaped value\
\
with empty line\
embeded^last line^

And here's what we get by loading that:

pgloader/examples$ pgloader -c pgloader.conf -s cluttered
Table name        |    duration |    size |  copy rows |     errors
====================================================================
cluttered         |      0.193s |       - |          6 |          0

pgloader/examples$ psql pgloader -c "table cluttered;"
 a |               b               |        c
---+-------------------------------+------------------
 1 | and some other data following | some multi
                                   : line text with
                                   : newline escaping
 2 | clean                         | and another line
 3 | just to test                  | and
                                   : a last multiline
                                   : escaped line
                                   : with a missing
                                   : escaping
 4 | empty value                   |
 5 | null value                    |
 6 | last line                     | multi line
                                   : escaped value
                                   :
                                   : with empty line
                                   : embeded
(6 rows)

So when you have such kind of data, well, it might be that pgloader is still able to help you!

Please refer to the pgloader man page to know about each and every parameter that you can define and the values accepted, etc. And the fixed data format is to be used when you're not given a field separator but field positions in the file. Yes, we still encounter those from time to time.

parallel processing

one reader, multiple workers

multiple workers, each reading

Here's a catalog query to help you there:

=> select *
->   from
->   (
(>   select 'previous parts' as schemaname, count(*)::text as tablename
(>     from pg_tables
(>    where schemaname not in ('pg_catalog','information_schema')
(>      and tablename like to_char(now(), '%YYYYMM')
(>
(>   union
(>
(>   select schemaname, substring(tablename,1,length(tablename)-6) || '201108'
(>     from pg_tables
(>    where schemaname not in ('pg_catalog','information_schema')
(>      and tablename like to_char(now(), '%YYYYMM')
(>
(>   except
(>
(>   select schemaname, tablename
(>     from pg_tables
(>    where schemaname not in ('pg_catalog','information_schema')
(>      and tablename like to_char(now() + interval '1 month', '%YYYYMM')
(>   ) as t
-> order by schemaname <> 'previous parts', schemaname;
   schemaname   |       tablename
----------------+------------------------
 previous parts | 1
 central        | stats_entrantes_201108
(2 rows)

As you see, our partitions are named _YYYYMM so that's it's easy to match them in our queries, but I guess about everyone does about the same here. Then the to_char expressions only allow to not enter manually '%201108' in the query text. And there's a trick so that we display how many partitions we have this month, adding a line to the result...

En attendant, bonne lecture !

installing pgloader

Either use the debian package or the one for your distribution of choice if you use another one. RedHat, CentOS, FreeBSD, OpenBSD and some more already include a binary package that you can use directly.

Or you could git clone https://github.com/dimitri/pgloader.git and go from there. As it's all python code, it runs fine interpreted from the source directory, you don't need to install it in a special place in your system.

setting up the test environment

To use them, please first create a pgloader database, then for each example the tables it needs, then issue the pgloader command:

$ createdb --encoding=utf-8 pgloader
$ cd examples
$ psql pgloader < simple/simple.sql
$ ../pgloader.py -Tvc pgloader.conf simple

If you want to load data from all examples, create tables for all of them first, then run pgloader without argument.

example description

The provided examples are:

• simple

  This dataset shows basic case, with trailing separator and data reordering.

• xzero

  Same as simple but using \0 as the null marker ()

• errors

  Same test, but with impossible dates. Should report some errors. If it does not report errors, check you're not using psycopg 1.1.21.

  Should report 3 errors out of 7 lines (4 updates).

• clob

  This dataset shows some text large object importing to PostgreSQL text datatype.

• cluttured

  A dataset with newline escaped and multi-line input (without quoting) Beware of data reordering, too.

• csv

  A dataset with csv delimiter ',' and quoting '"'.

• partial

  A dataset from which we only load some columns of the provided one.

• serial

  In this dataset the id field is ommited, it's a serial which will be automatically set by PostgreSQL while COPYing.

• reformat

  A timestamp column is formated the way MySQL dump its timestamp, which is not the same as the way PostgreSQL reads them. The reformat.mysql module is used to reformat the data on-the-fly.

• udc

  A used defined column test, where all file columns are not used but a new constant one, not found in the input datafile, is added while loading data.

running the import

You can launch all those pgloader tests in one run, provided you created the necessary tables:

 $ for sql in */*sql; do psql pgloader < $sql; done
 $ ../pgloader.py -Tsc pgloader.conf

  errors       WARNING  COPY error, trying to find on which line
  errors       WARNING  COPY data buffer saved in /tmp/errors.AhWvAv.pgloader
  errors       WARNING  COPY error recovery done (2/3) in 0.064s
  errors       WARNING  COPY error, trying to find on which line
  errors       WARNING  COPY data buffer saved in /tmp/errors.BclHtj.pgloader
  errors       WARNING  COPY error recovery done (1/1) in 0.054s
  errors       ERROR    3 errors found into [errors] data
  errors       ERROR    please read /tmp/errors.rej.log for errors log
  errors       ERROR    and /tmp/errors.rej for data still to process
  errors       ERROR    3 database errors occured
  reformat     WARNING  COPY error, trying to find on which line
  reformat     WARNING  COPY data buffer saved in /tmp/reformat.6P4WCD.pgloader
  reformat     WARNING  COPY error recovery done (1/4) in 0.034s
  reformat     ERROR    1 errors found into [reformat] data
  reformat     ERROR    please read /tmp/reformat.rej.log for errors log
  reformat     ERROR    and /tmp/reformat.rej for data still to process
  reformat     ERROR    1 database errors occured

  Table name        |    duration |    size |  copy rows |     errors
  ====================================================================
  allcols           |      0.025s |       - |          8 |          0
  clob              |      0.034s |       - |          7 |          0
  cluttered         |      0.061s |       - |          6 |          0
  csv               |      0.035s |       - |          6 |          0
  errors            |      0.113s |       - |          4 |          3
  fixed             |      0.045s |       - |          3 |          0
  partial           |      0.030s |       - |          7 |          0
  reformat          |      0.036s |       - |          4 |          1
  serial            |      0.029s |       - |          7 |          0
  simple            |      0.050s |       - |          7 |          0
  udc               |      0.020s |       - |          5 |          0
  ====================================================================
  Total             |      0.367s |       - |         64 |          4

Please note errors test should return 3 errors and reformat 1 error.

Hope you'll like it!

Hope you'll like it!

Les slides de l'ensemble des présentations devraient être publiés en ligne à terme, mais cela ne va pas pouvoir être fait aussi rapidement que nous le voudrions tous. Alors voici un peu de lecture en attendant la suite !

The slides for all the talks should eventually make their way to a central place, but expect some noticable delay here. Sorry about that, and have a good reading meanwhile!

As it was not that much work to implement, here's the whole of it:

;;;
;;; Breadcrumb support
;;;
(defun tapoueh-breadcrumb-to-current-page ()
  "Return a list of (name . link) from the index root page to current one"
  (let* ((current (muse-current-file))
         (cwd     (file-name-directory current))
         (project (muse-project-of-file current))
         (root    (muse-style-element :path (caddr project)))
         (path    (tapoueh-path-to-root))
         (dirs    (split-string (file-relative-name current root) "/")))
    ;; ("blog" "2011" "07" "13-back-from-char11.muse")
    (append
     (list (cons "/dev/dim" (concat path "index.html")))
     (loop for p in (butlast dirs)
           collect (cons p (format "%s%s/index.html" path p))
           do (setq path (concat path p "/"))))))

(defun tapoueh-insert-breadcrumb-div ()
  "The real HTML inserting"
  (insert "<div id=\"breadcrumb\">")
  (loop for (name . link) in (tapoueh-breadcrumb-to-current-page)
        do (insert (format "<a href=%s>%s</a>" link name) " / "))
  (insert "</div>\n"))

(defun tapoueh-insert-breadcrumb ()
  "Must run with current buffer being a muse article"
  (save-excursion
    (beginning-of-buffer)
    (when (tapoueh-extract-directive "author" (muse-current-file))
      (re-search-forward "<body>" nil t) ; find where the article content is
      (re-search-forward "<h2>" nil t)   ; that's the title line
      (beginning-of-line)
      (open-line 1)
      (tapoueh-insert-breadcrumb-div)

      (re-search-forward "<h2>" nil t 2) ; that's the TAG line
      (beginning-of-line)
      (open-line 1)
      (tapoueh-insert-breadcrumb-div))))

This code is now called in the :after function of my Muse project style, and it gets the work done.

This code is now called in the :after function of my Muse project style, and it gets the work done.

Tags

Emacs Muse

My setup is tentatively called tapoueh.el and browsable online. It consists of some tweaks on top of Muse, so that I can enjoy tags and proper rss support. By proper, I mean that I want to be able to produce as many topic RSS feeds from a single blog, and thanks to the tags support that's now what I have.

The RSS handling and the tagging system are adhoc code, and this very article begins like this:

#author Dimitri Fontaine
#title  Muse setup revised
#date   20110705-19:55
#tags   Emacs Muse

All the information for the site navigation are taken from there, and at long last the RSS I publish now contains proper URLs without abusing anchors, as in the previous link which is a compatibility page in case you had some bookmarks. The compat only works with javascript (did you know that anchors are not part of the URL that is sent to the server, so that you can't apply RedirectMatch or other tweaks?), but all it needs is 2 lines of code, so I guess that's not so bad.

var anchor = window.location.hash;
document.location.href=document.getElementById(anchor).href;

I hope you like the new setup as much as I do, even if I'm left with some debugging to do. That's the price to pay for doing it yourself I guess. But I still don't know of a ready to use solution (as in off the shelf) that meet my criteria for web publishing. More on that topic another time.

My setup is tentatively called tapoueh.el and browsable online. It consists of some tweaks on top of Muse, so that I can enjoy tags and proper rss support. By proper, I mean that I want to be able to produce as many topic RSS feeds from a single blog, and thanks to the tags support that's now what I have.

The RSS handling and the tagging system are adhoc code, and this very article begins like this:

#author Dimitri Fontaine
#title  Muse setup revised
#date   20110705-19:55
#tags   Emacs Muse

All the information for the site navigation are taken from there, and at long last the RSS I publish now contains proper URLs without abusing anchors, as in the previous link which is a compatibility page in case you had some bookmarks. The compat only works with javascript (did you know that anchors are not part of the URL that is sent to the server, so that you can't apply RedirectMatch or other tweaks?), but all it needs is 2 lines of code, so I guess that's not so bad.

var anchor = window.location.hash;
document.location.href=document.getElementById(anchor).href;

I hope you like the new setup as much as I do, even if I'm left with some debugging to do. That's the price to pay for doing it yourself I guess. But I still don't know of a ready to use solution (as in off the shelf) that meet my criteria for web publishing. More on that topic another time.

(require 'cl)
(loop for b being the buffers
      when (string-match "projects" (or (buffer-file-name b) ""))
      return t)

If you want to collect the list of buffers whose name matches your test, then replace return t by collect b and you're done. Really, this loop thing is worth learning.

You're back to enjoying el-get :)

Latest released version

el-get version 2.1 is available, with a boatload of features, including autoloads support, byte-compiling in an external clean room Emacs instance, custom support, lazy initialisation support (defering all init functions to eval-after-load), and multi repositories ELPA support.

Version numbering

Version String are now inspired by how Emacs itself numbers its versions. First is the major version number, then a dot, then the minor version number. The minor version number is 0 when still developping the next major version. So 3.0 is a developer release while 3.1 will be the next stable release.

Please note that this versioning policy has been picked while backing 1.2~dev, so 1.0 was a stable release in fact. Ah, history.

That's why I needed M-x mailq to display the mail queue and have some easy shortcuts in order to operate it (mainly f runs the command mailq-mode-flush, but per site and per id delivery are useful too).

Now, I also happen to setup outgoing mail routes to walk through an SSH tunnel, which thanks to both ~/.ssh/config and cssh (C-= runs the command cssh-term-remote-open, with completion) is a couple of keystrokes away to start. Well it still happens to me to forget about starting it, which causes mails to hold in a queue until I realise it's not delivered, which always take just about too long.

A solution I've been thinking about is to add a little flag in the modeline in my gnus *Group* and *Summary* buffers. The flag would show up as ✔ when no mail is queued and waiting for me to open the tunnel, or ✘ as soon as the queue is not empty. Here's what it looks like here:

Well I'm pretty happy with the setup. The flag is refreshed every minute, and here's as an example how I did setup mailq in my el-get-sources setup:

         (:name mailq
                :after (lambda () (mailq-modeline-display)))

I'm not sure how many of you dear readers are using a local MTA to deliver your mails, but well, the ones who do (or consider doing so) might even find this article useful!

A while ago I've been fixing and publishing pgsql-linum-format separately. That allows to number PL/whatever code lines when editing from Emacs, and it's something very useful to turn on when debugging.

The carrets on the fringe in the emacs window are the result of (setq-default indicate-buffer-boundaries 'left) and here it's just overloading the image somehow. But the idea is to just M-x linum-mode when you need it, at least that's my usage of it.

You can use el-get to easily get (then update) this little Emacs extension.

What you'll find is a very simple 128 lines Emacs user init file, based on el-get for external packages. A not so random selection of those is used, here's the list when you hide some details:

 '(el-get                       ; el-get is self-hosting
   escreen                      ; screen for emacs, C-\ C-h
   php-mode-improved            ; if you're into php...
   psvn                         ; M-x svn-status
   switch-window                ; takes over C-x o
   auto-complete                ; complete as you type with overlays
   emacs-goodies-el             ; the debian addons for emacs
   yasnippet                    ; powerful snippet mode
   zencoding-mode               ; http://www.emacswiki.org/emacs/ZenCoding
   (:name buffer-move           ; move buffers around in windows
   (:name smex                  ; a better (ido like) M-x
   (:name magit                 ; git meet emacs, and a binding
   (:name goto-last-change      ; move pointer back to last change

Another interresting thing to note in this kicker is a choice of some key bindings that are rather unusual (yet) I guess.

(global-set-key (kbd "C-x C-b") 'ido-switch-buffer)
(global-set-key (kbd "C-x C-c") 'ido-switch-buffer)
(global-set-key (kbd "C-x B") 'ibuffer)

Yes, you see that I've rebound C-x C-c to switching buffers. That key is really easy to use and I don't think that M-x kill-emacs deserves it. Keys that are so easy to use should be kept for frequent actions, and quiting emacs is a once-a-day to once-a-month action here. And you can still quit from the window manager button or from the menu or from M-x.

Also Mac users are not left behind, you will see some settings that either are adapted to the system (like choosing another font, keep displaying the menu-bar or not installing the darkish tango-color-mode on this system, where it renders poorly in my opinion), as you can see here: