tail -f /dev/dim

Nearest Big City

dim@tapoueh.org (Dimitri Fontaine) — Thu, 02 May 2013 11:34:00 +0200

In this article, we want to find the town with the greatest number of inhabitants near a given location.

A very localized example

We first need to find and import some data, and I found at the following place a CSV listing of french cities with coordinates and population and some numbers of interest for the exercise here.

To import the data set, we first need a table, then a COPY command:

CREATE TABLE lion1906 (
  insee       text,
  nom         text,
  altitude    integer,
  code_postal text,
  longitude   double precision,
  latitude    double precision,
  pop99       bigint,
  surface     double precision
);

\copy lion1906 from 'villes.csv' with csv header delimiter ';' encoding 'latin1'

With that data in place, we can find the 10 nearest towns of a random choosing of us, let's pick Villeurbanne which is in the region of Lyon.

   select code_postal, nom, pop99
     from lion1906
 order by point(longitude, latitude) <->
          (select point(longitude, latitude)
             from lion1906
            where nom = 'Villeurbanne')
    limit 10;

 code_postal |          nom           | pop99
-------------+------------------------+--------
 69100       | Villeurbanne           | 124215
 69300       | Caluire-et-Cuire       |  41233
 69120       | Vaulx-en-Velin         |  39154
 69580       | Sathonay-Camp          |   4336
 69140       | Rillieux-la-Pape       |  28367
 69000       | Lyon                   | 445452
 69500       | Bron                   |  37369
 69580       | Sathonay-Village       |   1693
 01700       | Neyron                 |   2157
 69660       | Collonges-au-Mont-d'Or |   3420
(10 rows)

We find Lyon in our list in there, and we want the query now to return only that one as it has the greatest number of inhabitants in the list:

with neighbours as (
   select code_postal, nom, pop99
     from lion1906
 order by point(longitude, latitude) <->
          (select point(longitude, latitude)
             from lion1906 where nom = 'Villeurbanne')
    limit 10
)
  select *
    from neighbours
order by pop99 desc
   limit 1;

 code_postal | nom  | pop99
-------------+------+--------
 69000       | Lyon | 445452
(1 row)

Well, thank you PostgreSQL, that was easy!

Note that you can actually index such queries, that's called a KNN index. PostgreSQL knows how to use some kind of indexes to fetch data matching an expression such as ORDER BY a <-> b, which allow you to consider a KNN search in your application.

Let's get worldwide

The real scope of our exercise is to associate every known town in the world with some big city around, so let's first fetch and import some worldwide data this time, from http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz.

CREATE TABLE maxmind_worldcities (
        country_code text,
        city_lower text,
        city_normal text,
        region_code text DEFAULT '',
        population INT DEFAULT '0',
        latitude float8 DEFAULT '0',
        longitude float8 DEFAULT '0'
);

\copy maxmind_worldcities FROM '/tmp/worldcitiespop.txt' WITH  DELIMITER ',' QUOTE E'\f' CSV HEADER ENCODING 'LATIN1';

alter table maxmind_worldcities add column loc point;
update maxmind_worldcities set loc = point(longitude, latitude);

This time you can see that I created an extra column with the location in there, so that I don't have to compute it each time I need it, like I did before.

Now is the time to test that data set and hopefully fetch the same result as before when we only had french cities loaded:

with neighbours as (
   select country_code, city_lower, population
     from maxmind_worldcities
    where population is not null
 order by loc <->
          (select loc
             from maxmind_worldcities
            where city_lower = 'villeurbanne')
   limit 10
)
   select * from neighbours order by population desc limit 1;

  country_code | city_lower | population
--------------+------------+------------
 fr           | lyon       |     463700
(1 row)

Ok, looks like we're all set for the real problem. Now we want to pick for each of those cities it's nearest neighboor, so here's how to do that:

create index on maxmind_worldcities(country_code, region_code, city_lower);
create index on maxmind_worldcities using gist(loc);

create table maxmind_neighbours as
  select country_code, region_code, city_lower,
         (with neighbours as (
             select country_code, city_lower, population
               from maxmind_worldcities
              where population is not null
                    and country_code = wc.country_code
                    and region_code = wc.region_code
           order by loc <-> wc.loc
             limit 10)
             select city_lower
               from neighbours
           order by population desc
              limit 1
         ) as neighbour
    from maxmind_worldcities wc ;

To be fair, I have to tell you that this query took almost 2 hours to complete on my laptop here, but as I'm doing that for friend and a blog article, I've been lazy and didn't try to optimise it. It could be using LATERAL for sure, I don't know if that would help very much with performances: I didn't try.

With that in hands we can now check some cities and their biggest neighbours, as in the following query:

select * from maxmind_neighbours where city_lower = 'villeurbanne';
 country_code | region_code |  city_lower  | neighbour
--------------+-------------+--------------+-----------
 fr           | B9          | villeurbanne | lyon
(1 row)

And looking for New-York City suburbs I did find a chinatown, which is a pretty common smaller town name apparently:

select * from maxmind_neighbours where city_lower = 'chinatown';
 country_code | region_code | city_lower |   neighbour
--------------+-------------+------------+---------------
 sb           | 08          | chinatown  | honiara
 us           | CA          | chinatown  | san francisco
 us           | DC          | chinatown  | washington
 us           | HI          | chinatown  | honolulu
 us           | IL          | chinatown  | chicago
 us           | MT          | chinatown  | missoula
 us           | NV          | chinatown  | reno
 us           | NY          | chinatown  | new york
(8 rows)

Big Cities in the big world

We might need to change some of our views

So, let's see how many smaller towns each of those random big cities have:

   select country_code, region_code, neighbour, count(*)
    from maxmind_neighbours
   where neighbour in ('london', 'new york', 'moscow',
                       'paris', 'tokyo', 'sao polo', 'chicago')
group by country_code, region_code, neighbour;
 country_code | region_code | neighbour | count
--------------+-------------+-----------+-------
 gb           | H9          | london    |     2
 jp           | 40          | tokyo     |   414
 us           | NY          | new york  |   131
 ca           | 08          | london    |    16
 ru           | 48          | moscow    |   245
 fr           | A8          | paris     |    16
 us           | IL          | chicago   |    13
(7 rows)

And now let's be fair and see where are the cities with the greatest number of towns nearby them, with the following query:

  select country_code, region_code, neighbour, count(*)
    from maxmind_neighbours
   where neighbour is not null
group by country_code, region_code, neighbour
order by 4 desc
   limit 25;

 country_code | region_code | neighbour  | count
--------------+-------------+------------+-------
 cn           | 03          | nanchang   | 16759
 cn           | 26          | xian       | 12864
 id           | 18          | kupang     | 10715
 cn           | 24          | taiyuan    | 10550
 mm           | 11          | taunggyi   | 10253
 id           | 38          | makasar    |  9471
 ir           | 15          | ahvaz      |  9461
 id           | 01          | banda aceh |  9161
 cn           | 14          | lasa       |  8841
 cn           | 15          | lanzhou    |  8618
 ir           | 29          | kerman     |  8579
 id           | 26          | medan      |  7787
 ir           | 04          | iranshahr  |  7249
 ir           | 07          | shiraz     |  7219
 ma           | 55          | agadir     |  7121
 ir           | 42          | mashhad    |  7107
 af           | 08          | gazni      |  7011
 ir           | 33          | tabriz     |  6586
 cn           | 01          | hefei      |  6521
 bd           | 81          | dhaka      |  6480
 ir           | 08          | rasht      |  6471
 id           | 17          | mataram    |  6467
 id           | 33          | cilegon    |  6287
 af           | 23          | qandahar   |  6213
 cn           | 07          | fuzhou     |  6089
(25 rows)

Bulk Replication

dim@tapoueh.org (Dimitri Fontaine) — Mon, 18 Mar 2013 14:54:00 +0100

In the previous article here we talked about how to properly update more than one row at a time, under the title Batch Update. We did consider performances, including network round trips, and did look at the behavior of our results when used concurrently.

A case where we want to apply the previous article approach is when replicating data with a trigger based solution, such as SkyTools and londiste. Well, maybe not in all cases, we need to have a amount of UPDATE trafic worthy of setting up the solution. As soon as we know we're getting to replay important enough batches of events, though, certainly using the batch update tricks makes sense.

It so happens that londiste 3 includes the capability to use handlers. Those are plugins written in python (like all the client side code from SkyTools) whose job is to handle the processing of the event batches. Several of them are included in the londiste sources, and one of them is named bulk.py.

Bulk loading data with londiste

To use set in londiste.ini:

handler_modules = londiste.handlers.bulk

then add table with one of those commands:

londiste3 add-table xx --handler="bulk"
londiste3 add-table xx --handler="bulk(method=X)"

The default method is 0, and the available methods are the following:

correct (0)

inserts as COPY into table
update as COPY into temp table and single UPDATE from there
delete as COPY into temp table and single DELETE from there

delete (1)

as correct, but update are done as DELETE then COPY

merged (2)

as delete, but merge insert rows with update rows

Conclusion

Yes, by using that handler which is provided by default in londiste, you will apply the previous article tricks in your replication solution. And you can even choose to use that for only some of the tables you are replicating.

Batch Update

dim@tapoueh.org (Dimitri Fontaine) — Fri, 15 Mar 2013 10:47:00 +0100

Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

Another kind of Batch to update

So when you need to UPDATE a bunch of rows from a given source, remember that you can actually use a JOIN in the update statement. Either the source of data is already in the database, in which case it's as simple as using the FROM clause in the update statement, or it's not, and we're getting back to that in a minute.

`UPDATE FROM`

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row.

What about that round trip?

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

As we're talking about performances, the trick here is to use the COPY protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches.

Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol.

Insert or Update

And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all DML queries, which means that you can do the following just fine:

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd t USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

That query here is updating all the rows that are known in both the target and the source and returns what we took from the source in the operation, so that we can do an anti-join in the next step of the query, where we're inserting any row that was not taken care of in the update part of the statement.

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Now, you will tell me that we just solved the UPSERT problem. Well what happens if more than one transaction is trying to do the WITH (UPDATE) INSERT dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong?

Concurrent processing

What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table.

There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical.

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock.

Conclusion

The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough PostgreSQL releases you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today!

Batch Update

dim@tapoueh.org (Dimitri Fontaine) — Fri, 15 Mar 2013 10:47:00 +0100

Another kind of Batch to update

`UPDATE FROM`

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

What about that round trip?

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

Insert or Update

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd t USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Concurrent processing

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

Conclusion

Batch Update

dim@tapoueh.org (Dimitri Fontaine) — Fri, 15 Mar 2013 10:47:00 +0100

Another kind of Batch to update

`UPDATE FROM`

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

What about that round trip?

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

Insert or Update

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd t USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Concurrent processing

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

Conclusion

Batch Update

dim@tapoueh.org (Dimitri Fontaine) — Fri, 15 Mar 2013 10:47:00 +0100

Another kind of Batch to update

`UPDATE FROM`

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

What about that round trip?

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

Insert or Update

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Concurrent processing

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

Conclusion

Batch Update

dim@tapoueh.org (Dimitri Fontaine) — Fri, 15 Mar 2013 10:47:00 +0100

Another kind of Batch to update

`UPDATE FROM`

It's all about using that FROM clause in an update statement, right?

    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that.

Preparing the Batch

What about that round trip?

Let's see how it goes:

CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t
   SET counter = t.counter + s.counter,
  FROM source s
 WHERE t.id = s.id

Insert or Update

WITH upd AS (
    UPDATE target t
       SET counter = t.counter + s.counter,
      FROM source s
     WHERE t.id = s.id
 RETURNING s.id
)
INSERT INTO target(id, counter)
     SELECT id, sum(counter)
       FROM source s LEFT JOIN upd USING(id)
      WHERE t.id IS NULL
   GROUP BY s.id
  RETURNING t.id

Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key.

Concurrency patterns

Concurrent processing

The other thing you can do, is force the concurrent transactions to serialize one after the other, using an explicit locking statement:

LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE;

Conclusion

Emacs Conference

dim@tapoueh.org (Dimitri Fontaine) — Mon, 04 Mar 2013 13:58:00 +0100

The Emacs Conference is happening, it's real, and it will take place at the end of this month in London. Check it out, and register at Emacs Conference Event Brite. It's free and there's still some availability.

It's all about Emacs, and it rocks!

We have a great line-up for this conference, which makes me proud to be able to be there. If you've ever been paying attention when using Emacs then you've already heard those names: Sacha Chua is frequently blogging about how she manages to improve her workflow thanks to Emacs Lisp, John Wiegley is a proficient Emacs contributor maybe best known for his ledger Emacs Mode, then we have Luke Gorrie who hacked up SLIME among other things, we also have Nic Ferrier who is starting a revolution in how to use Emacs Lisp with elnode. And more! Including Steve Yegge!

See you there in London.

HyperLogLog Unions

dim@tapoueh.org (Dimitri Fontaine) — Tue, 26 Feb 2013 12:44:00 +0100

In the article from yesterday we talked about PostgreSQL HyperLogLog with some details. The real magic of that extension has been skimmed over though, and needs another very small article all by itself, in case you missed it.

Which Set Operation do you want for counting unique values?

The first query here has the default level of magic in it, really. What happens is that each time we do an update of the HyperLogLog hash value, we update some data which are allowing us to compute its cardinality.

=> select date,
          #users as daily,
          pg_column_size(users) as bytes
     from daily_uniques
 order by date;
    date    |      daily       | bytes
------------+------------------+-------
 2013-02-22 | 401676.779509985 |  1287
 2013-02-23 | 660187.271908359 |  1287
 2013-02-24 | 869980.029947449 |  1287
 2013-02-25 | 580865.296677817 |  1287
 2013-02-26 | 240569.492722719 |  1287
(5 rows)

And has advertized the data is kept in a static sized data structure. The magic here all happens at hll_add() time, the function you have to call to update the data.

Now on to something way more magic!

Are those the aggregates you're looking for?

=> select to_char(date, 'YYYY/MM') as month,
          round(#hll_union_agg(users)) as monthly
     from daily_uniques group by 1;
  month  | monthly
---------+---------
 2013/02 | 1960380
(1 row)

The HyperLogLog data structure is allowing the implementation of an union algorithm that will be able to compute how many unique values you happen to have registered in both one day and the next. Extended in its general form, and doing SQL, what you get is an aggregate that you can use in GROUP BY constructs and window functions. Did you read about them yet?

PostgreSQL HyperLogLog

dim@tapoueh.org (Dimitri Fontaine) — Mon, 25 Feb 2013 10:23:00 +0100

If you've been following along at home the newer statistics developments, you might have heard about this new State of The Art Cardinality Estimation Algorithm called HyperLogLog. This technique is now available for PostgreSQL in the extension postgresql-hll available at https://github.com/aggregateknowledge/postgresql-hll and soon to be in debian.

How to Compute Cardinality?

Installing postgresql-hll

It's as simple as CREATE EXTENSION hll; really, even if to get there you must have installed the package on your system. We did some packaging work for debian and the result should appear soon in a distro near you.

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

So in our example what you see is that we want to decipher how many unique IP addresses we saw, and we do that by first creating a hash of that source data then calling hll_add() with the current value and the hash result.

The current value must be initialized using hll_empty().

Concurrency

The most awake readers among you have already spotted that: using an UPDATE on the same row over and over again is a good recipe to kill any form of concurrency, so you don't want to do that on your production setup unless you don't care about those UPDATE waiting piling up in your system.

The idea is then to fill-in a queue of updates and asynchronously update the daily_uniques table from that queue, possibly using the hll_add_agg aggregate that the extension provides, so that you do only one update per batch of values to process.

∅: Empty Set and NULL

Yes there's a unicode entry for that, ∅

Now, what happens when the batch of new unique values you want to update from is itself empty? Well I would have expected hll_add_agg over an empty set to return an empty hll value, the same as returned by hll_empty(), but it turns out it's returning NULL instead.

And then hll_add(users, NULL) will happily return NULL. So the next UPDATE is cancelling all the previous work, which is not nice. We had to cater for that case explicitely in the UPDATE query that's working from the batch of new values to add to our current HyperLogLog hash entry, and I can't resist to show off one of the most awesome PostgreSQL features here: writable CTE.

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

That's how you protect against an empty set being turned into a NULL. I think the real fix would need to be included in postgresql-hll itself, in making it so that the hll_add_agg aggregate returns hll_empty() on an empty set, and I will report that bug (with that very article as the detailed explanation of it).

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

I coulnd't resist to show off two of my favorite SQL constructs in that example query here, which are the Common Table Expressions (or CTE) and window functions. If that over() clause reads strange to you, take a minute now and go read about it. Yes, do that now, we're waiting.

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

The HyperLogLog DV estimator

When using postgresql-hll you need to be careful not to kill your application concurrency abilities, and you need to protect yourself against the ∅ killer too. The other thing to keep in mind is that the numbers you get out of the hll technique are estimates within a given precision, and you might want to read some more about what it means for your intended usage of the feature.

PostgreSQL HyperLogLog

dim@tapoueh.org (Dimitri Fontaine) — Mon, 25 Feb 2013 10:23:00 +0100

How to Compute Cardinality?

Installing postgresql-hll

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

The current value must be initialized using hll_empty().

Concurrency

∅: Empty Set and NULL

Yes there's a unicode entry for that, ∅

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

The HyperLogLog DV estimator

PostgreSQL HyperLogLog

dim@tapoueh.org (Dimitri Fontaine) — Mon, 25 Feb 2013 10:23:00 +0100

How to Compute Cardinality?

Installing postgresql-hll

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

The current value must be initialized using hll_empty().

Concurrency

∅: Empty Set and NULL

Yes there's a unicode entry for that, ∅

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

The HyperLogLog DV estimator

PostgreSQL HyperLogLog

dim@tapoueh.org (Dimitri Fontaine) — Mon, 25 Feb 2013 10:23:00 +0100

How to Compute Cardinality?

Installing postgresql-hll

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

The current value must be initialized using hll_empty().

Concurrency

∅: Empty Set and NULL

Yes there's a unicode entry for that, ∅

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

The HyperLogLog DV estimator

PostgreSQL HyperLogLog

dim@tapoueh.org (Dimitri Fontaine) — Mon, 25 Feb 2013 10:23:00 +0100

How to Compute Cardinality?

Installing postgresql-hll

Then you also need to keep your data in some table, straight from the documentation we can use that schema:

-- Create the destination table
CREATE TABLE daily_uniques (
DATE            DATE UNIQUE,
users           hll
);

Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement:

UPDATE daily_uniques
   SET users = hll_add(users, hll_hash_text('123.123.123.123'))
 WHERE date = current_date;

The current value must be initialized using hll_empty().

Concurrency

∅: Empty Set and NULL

Yes there's a unicode entry for that, ∅

WITH hll(agg) AS (
  SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch
)
  UPDATE daily_uniques
     SET users = CASE WHEN hll.agg IS NULL THEN users
                      ELSE hll_union(users, hll.agg)
                  END
    FROM hll
   WHERE date = current_date;

Using postgresql-hll

When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table:

with stats as (
  select date, #users as daily, #hll_union_agg(users) over() as total
    from daily_uniques
)
  select date,
         round(daily) as daily,
         round((daily/total*100)::numeric, 2) as percent
    from stats
order by date;
    date    | daily  | percent
------------+--------+---------
 2013-02-22 | 401677 |   25.19
 2013-02-23 | 660187 |   41.41
 2013-02-24 | 869980 |   54.56
 2013-02-25 | 154996 |    9.72
(4 rows)

The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low.

Conclusion

The HyperLogLog DV estimator

Playing with pgloader

dim@tapoueh.org (Dimitri Fontaine) — Tue, 12 Feb 2013 11:17:00 +0100

While making progress with both Event Triggers and Extension Templates, I needed to make a little break. My current keeping sane mental exercise seems to mainly involve using Common Lisp, a programming language that ships with about all the building blocks you need.

Yes, that old language brings so much on the table

When using Common Lisp, you have an awesome interactive development environment where you can redefine function and objects while testing them. That means you don't have to quit the interpreter, reload the new version of the code and put the interactive test case together all over again after a change. Just evaluate the change in the interactive environement: functions are compiled incrementally over their previous definition, objects whose classes have changed are migrated live.

See, I just said objects and classes. Common Lisp comes with some advanced Object Oriented Programming facilities named CLOS and MOP where the Java and Python and C++ object models are just a subset of what you're being offered. Hint, those don't have Multiple Dispatch.

And you have a very sophisticated Condition System where Exceptions are just a subset of what you can do (hint: have a look a restarts and tell me you didn't wish your programming language of choice had them). And it continues that way for about any basic building bloc you might want to be using.

Loading data

Back to pgloader will you tell me. Right. I've been spending a couple of evening on hacking on the new version of pgloader in Common Lisp, and wanted to share some preliminary results.

Playing with the loader

The current status of the new pgloader still is pretty rough, if you're not used to develop in Common Lisp you might not find it ready for use yet. I'm still working on the internal APIs and trying to make something clean and easy to use for a developer, and then I will provide some external ways to play with it, user oriented. I missed that step once with the Python based version of the tool, I don't want to do the same errors again this time.

So here's a test run with the current pgloader, on a small enough data set of 226 MB of CSV files.

time python pgloader.py -R.. --summary -Tc ../pgloader.dbname.conf

Table name        |    duration |    size |  copy rows |     errors
====================================================================
aaaaaaaaaa_aaaa   |      2.148s |       - |      24595 |          0
bbbbbbbbbb_bbbb...|      0.609s |       - |        326 |          0
cccccccccc_cccc...|      2.868s |       - |      25126 |          0
dddddddddd_dddd...|      0.638s |       - |          8 |          0
eeeeeeeeee_eeee...|      2.874s |       - |      36825 |          0
ffffffffff_ffffff |      0.667s |       - |        624 |          0
gggggggggg_gggg...|      0.847s |       - |       5638 |          0
hhh_hhhhhhh       |      9.907s |       - |     120159 |          0
iii_iiiiiiiiiiiii |      0.574s |       - |        661 |          0
jjjjjjj           |      6.647s |       - |      30027 |          0
kkk_kkkkkkkkk     |      0.439s |       - |         12 |          0
lll_llllll        |      0.308s |       - |          4 |          0
mmmm_mmm          |      2.139s |       - |      29669 |          0
nnnn_nnnnnn       |      8.555s |       - |     100197 |          0
oooo_ooooo        |     13.781s |       - |      93555 |          0
pppp_ppppppp      |      8.275s |       - |      76457 |          0
qqqq_qqqqqqqqqqqq |      8.568s |       - |     126159 |          0
====================================================================
Total             |  01m09.902s |       - |     670042 |          0

Streaming data

With the new code in Common Lisp, I could benefit from real multi threading and higher level abstraction to make it easy to use: lparallel is a lib providing exactly what I need here, with workers and queues to communicate data in between them.

What I'm doing is that two threads are separated, one is reading the data from either a CSV file or a MySQL database directly, and pushing that data in the queue; while the other thread is pulling data from the queue and writing it into our PostgreSQL database.

CL-USER> (pgloader.csv:import-database "dbname"
            :csv-path-root "/path/to/csv/"
            :separator #\Tab
            :quote #\"
            :escape "\"\""
            :null-as ":null:")
                    table name       read   imported     errors       time
------------------------------  ---------  ---------  ---------  ---------
               aaaaaaaaaa_aaaa      24595      24595          0     0.995s
          bbbbbbbbbb_bbbbbbbbb        326        326          0     0.570s
       cccccccccc_cccccccccccc      25126      25126          0     1.461s
      dddddddddd_dddddddddd_dd          8          8          0     0.650s
eeeeeeeeee_eeeeeeeeee_eeeeeeee      36825      36825          0     1.664s
             ffffffffff_ffffff        624        624          0     0.707s
     gggggggggg_ggggg_gggggggg       5638       5638          0     0.655s
                   hhh_hhhhhhh     120159     120159          0     3.415s
             iii_iiiiiiiiiiiii        661        661          0     0.420s
                       jjjjjjj      30027      30027          0     2.743s
                 kkk_kkkkkkkkk         12         12          0     0.327s
                    lll_llllll          4          4          0     0.315s
                      mmmm_mmm      29669      29669          0     1.182s
                   nnnn_nnnnnn     100197     100197          0     2.206s
                    oooo_ooooo      93555      93555          0     9.683s
                  pppp_ppppppp      76457      76457          0     5.349s
             qqqq_qqqqqqqqqqqq     126159     126159          0     2.495s
------------------------------  ---------  ---------  ---------  ---------
             Total import time     670042     670042          0    34.836s
NIL

As you can see the control is still made for interactive developer usage, which is fine for now but will have to change down the road, when the APIs stabilize.

Now, let's compare to reading directly from MySQL:

CL-USER> (pgloader.mysql:stream-database "dbname")
                    table name       read   imported     errors       time
------------------------------  ---------  ---------  ---------  ---------
               aaaaaaaaaa_aaaa      24595      24595          0     0.887s
          bbbbbbbbbb_bbbbbbbbb        326        326          0     0.617s
       cccccccccc_cccccccccccc      25126      25126          0     1.497s
      dddddddddd_dddddddddd_dd          8          8          0     0.582s
eeeeeeeeee_eeeeeeeeee_eeeeeeee      36825      36825          0     1.697s
             ffffffffff_ffffff        624        624          0     0.748s
     gggggggggg_ggggg_gggggggg       5638       5638          0     0.923s
                   hhh_hhhhhhh     120159     120159          0     3.525s
             iii_iiiiiiiiiiiii        661        661          0     0.449s
                       jjjjjjj      30027      30027          0     2.546s
                 kkk_kkkkkkkkk         12         12          0     0.330s
                    lll_llllll          4          4          0     0.323s
                      mmmm_mmm      29669      29669          0     1.227s
                   nnnn_nnnnnn     100197     100197          0     2.489s
                    oooo_ooooo      93555      93555          0     9.148s
                  pppp_ppppppp      76457      76457          0     6.713s
             qqqq_qqqqqqqqqqqq     126159     126159          0     4.571s
------------------------------  ---------  ---------  ---------  ---------
          Total streaming time     670042     670042          0    38.272s
NIL

The streaming here is a tad slower than the importing from files. Now if you want to be fair when comparing those, you would have to take into account the time it takes to export the data out from its source. When doing that export/import dance, a quick test shows a timing of 1m4.745s. Now, if we do an export only test, it runs in 31.822s. So yes streaming is a good thing to have here.

Conclusion

We just got twice as fast as the python version.

Some will say that I'm not comparing fairly to the Python version of pgloader here, because I could have implemented the streaming facility in Python too. Well actually I did, the option are called section_threads and split_file_reading, that you can set so that a reader is pushing data into a set of queues and several workers are feeding each from its own queue. It didn't help with performances at all. Once again, read about the infamous Global Interpreter Lock to understand why not.

So actually it's a fair comparison here where the new code is twice as fast as the previous one, with only some hours of hacking and before spending any time on optimisation. Well, apart from using a producer, a consumer and a queue, which I almost had to have for streaming in between two database connections anyways.

Live Upgrading PGQ

dim@tapoueh.org (Dimitri Fontaine) — Fri, 08 Feb 2013 15:52:00 +0100

Some skytools related new today, it's been a while. For those who where at my FOSDEM's talk about Implementing High Availability you might have heard that I really like working with PGQ. A new version has been released a while ago, and the most recent verion is now 3.1.3, as announced in the Skytools 3.1.3 email.

Upgrade time!

Skytools 3.1.3 enters debian

First news is that Skytools 3.1.3 has been entering debian today (I hope that by the time you reach that URL, it's been updated to show information according to the news here, but I might be early). As there's current a debian freeze to release wheezy (and you can help squash some bugs), this version is only getting uploaded to experimental for now. Thanks to the tireless work of Christoph Berg though, this version is already available from apt.postgresql.org.

Upgrading to PGQ 3

The other news is that I've been testing live upgrade scenario where we want to upgrade from PGQ to PGQ3, and it works pretty well, and it's quite simple to achieve too. Here's how.

So the first thing is to shut down the current ticker process. Then we install the new packages, assuming that you did follow the step in the wiki pointed above, please go read apt.postgresql.org again now if needs be.

pgqadm.py ticker.ini -s
sudo apt-get install postgresql-9.1-pgq3 skytools3-ticker skytools3

The ticker is not running anymore, we have the right version of the software installed. Next step is to upgrade the database parts of PGQ:

psql -f /usr/share/skytools3/pgq.upgrade_2.1_to_3.0.sql ...
psql -1 -f /usr/share/postgresql/9.1/contrib/pgq.upgrade.sql ...

Of course replace those ... with options such as your actual connection string. I tend to always add -vON_ERROR_STOP=1 to all these commands, so that I don't depend on having the right .psqlrc on the particular server I'm connected to. Also remember that if you want to do that for more than one database, you need to actually run that pair of commands for each of them.

Now it's time to restart the new ticker. The main changes from the previous one is that it is now a C program called pgqd that knows how to tick for any number of databases, so that you only have to have one instance around per cluster now.

sudo /etc/init.d/skytools3 start
tail -f /var/log/skytools/pgqd.log

Those two commands are taking for granted that you did prepare the pgqd setup the debian and skytools way, by adding your config in /etc/skytools3/pgqd.ini and editing /etc/skytools.ini accordingly, so that it's automatically taken into account at machine boot.

Note that I did actually exercised the procedure above while running a pgbench test replicated with londiste. Of course the replication has been lagging a little while no ticker was running, and then it catched-up as fast as it could, in that case:

INFO {count: 245673, ignored: 0, duration: 422.104366064}

Happy Hacking!

So if you have any batch processing needs, remember to consider what PGQ has to offer. And yes if you're running some cron job to compute things out of the database for you, you are doing some batch processing.

Yes, I did search for Transactional Batch Processing

Another Great FOSDEM

dim@tapoueh.org (Dimitri Fontaine) — Mon, 04 Feb 2013 09:55:00 +0100

This year's FOSDEM has been a great edition, in particular the FOSDEM PGDAY 2013 was a great way to begin a 3 days marathon of talking about PostgreSQL with people not only from our community but also from plenty other Open Source communities too: users!

PostgreSQL at FOSDEM made for a great event

Having had the opportunity to meet more people from those other development communities, I really think we should go and reach for them in their own conferences. About any PostgreSQL community member I've been talking about with about that idea seemed to agree and generally already was thinking the same thing. And most are already doing it, in fact...

I had the pleasure to run two conferences there, both in the PostgreSQL devroom.

Event Triggers

I'm currently in the middle of implementing Event Triggers for PostgreSQL and I have been for about the last 2 years. It's a quite complex feature to get right and so the patch itself is complex and large, which means the reviewing process is complex and takes time.

That also means that some parts of the design have already been redone completely at least 3 times, and that what got commited to the PostgreSQL code is nothing like what the design we decided should go in looks like. That's just a fact of life, maybe, but that makes for a very long development process.

We're now getting to the end of it though, and this talk is showing both where we want to go with Event Triggers, where we are now and what remains to be done for 9.3 if we want the feature to be any useful.

If you're interested into that development, have a look at the slide deck and possibly ask me some questions about what's not clear on the pgsql-hackers mailing list (preferably).

Event Triggers, The Real Mess™

The other way to get summarized and clear information about Event Triggers is the wiki page by the same name: Event Triggers.

You will see that while a lot has been done (internal refactoring, adding new infrastructure and SQL level commands, and the minimum PLpgSQL support); a lot remains to be done where the code has already been submitted several times, following several designs directions given by careful review on hackers, and still we have some choices to make.

Implementing High-Availability

This talk is showing several ways to implement High Availability with PostgreSQL. The fact is that that term is overloaded already, and usually covers two very different things which are Service Availability and Data Availability.

In the talk, we're showing up several techniques that you can use to address different set of compromises in between scaling, load balancing, data availability and durability, and service availability. The first two points could seem unrelated to the main topic, but scaling often is a simple enough way to achieve service availability... until you need to think about sharding, that is.

Implementing High Availability of Services and Data with PostgreSQL

So the talk is all about making compromises in between them and getting to an architecture able to implement the choosen compromises. While the talk has been pretty well received, it was delivered in a 50 mins slot where we usually take a whole day or three when addressing that problems at a customer's site.

Some parts of how to get to the right architecture for the compromises that are important for you can't be fully covered in that time slot, while still being able to actually present the techniques that we're using.

I think it might be useful to extract a single use-case or two from that talk then have a full 50 mins version reduced to a single or a couple of very clear compromises and how to achieve them in details, rather than trying to present a full range of techniques and how to use them in different scenarios.

FOSDEM

After having been talking with many people, it appears that for next year's edition I should be proposing a more general talk that aims at helping developpers in other communities (python, ruby, etc) discover what's in for them in PostgreSQL. This database is full of advanced features that are really easy to use, and the only problem when preparing such a talk is choosing the right subset...

If you're running a local developper user group and are interested into learning some more about how PostgreSQL can help you in a daily basis, please do get in touch with me and let's schedule a presentation together!

Another Great FOSDEM

dim@tapoueh.org (Dimitri Fontaine) — Mon, 04 Feb 2013 09:55:00 +0100

PostgreSQL at FOSDEM made for a great event

I had the pleasure to run two conferences there, both in the PostgreSQL devroom.

Event Triggers

If you're interested into that development, have a look at the slide deck and possibly ask me some questions about what's not clear on the pgsql-hackers mailing list (preferably).

Event Triggers, The Real Mess™

The other way to get summarized and clear information about Event Triggers is the wiki page by the same name: Event Triggers.

Implementing High-Availability

Implementing High Availability of Services and Data with PostgreSQL

FOSDEM

Another Great FOSDEM

dim@tapoueh.org (Dimitri Fontaine) — Mon, 04 Feb 2013 09:55:00 +0100

PostgreSQL at FOSDEM made for a great event

I had the pleasure to run two conferences there, both in the PostgreSQL devroom.

Event Triggers

If you're interested into that development, have a look at the slide deck and possibly ask me some questions about what's not clear on the pgsql-hackers mailing list (preferably).

Event Triggers, The Real Mess™

The other way to get summarized and clear information about Event Triggers is the wiki page by the same name: Event Triggers.

Implementing High-Availability

Implementing High Availability of Services and Data with PostgreSQL

FOSDEM

A Sunday at FOSDEM

dim@tapoueh.org (Dimitri Fontaine) — Wed, 30 Jan 2013 10:50:00 +0100

The previous article FOSDEM 2013 said to be careful with the PostgreSQL devroom schedule because one of my talks there might get swapped with a slot on the FOSDEM PGDay 2013 which happens this Friday and has been sold out anyway.

Turns out it's not true, because we still depend on past century technologies somehow. Not everybody will be looking at the schedule on the web using a connected mobile device (you know, you've heard of them, those tracking and surveillance devices, if you want to believe Stallman), and as the schedule gets printed on little paper sheets, it's unfortunately too late to change it now.

Those flyers are already printed on paper sheets, the schedule too

So it happens that I'll be speaking twice on Sunday and not at all on Friday.

FOSDEM 2013

dim@tapoueh.org (Dimitri Fontaine) — Tue, 29 Jan 2013 10:11:00 +0100

This year again I'm going to FOSDEM, and to the extra special PostgreSQL FOSDEM day. It will be the first time that I'm going to be at the event for the full week-end rather than just commuting in for the day.

I'm Going to the FOSDEM, hope to see you there!

And I'm presenting two talks over there that are both currently scheduled on the Sunday in the PostgreSQL devroom. We're talking about changing that though, so that one of those will in fact happen this Friday at the FOSDEM PGDay 2013, which has a different schedule, so consider watching for that.

One of those two talks is about Implementing High Availability (yes, with PostgreSQL). It's been quite well received in the places I had to chance to make it before (namely PGDay France and PG Conf Europe), and it's going to be a stripped down version of it so that it fits well in the 45 mins slot we have here.

The other talk is going to be about Event Triggers, a feature new in PostgreSQL 9.3 (due in september 2013, crossing fingers) and while the goal of that talk is to introduce what the feature is all about and a bunch of use cases that you can address by using it, it will certainly offer a peek into the PostgreSQL development cycle and community processes.

See you in Brussels!

pgloader: what's next?

dim@tapoueh.org (Dimitri Fontaine) — Mon, 28 Jan 2013 10:48:00 +0100

pgloader is a tool to help loading data into PostgreSQL, adding some error management to the COPY command. COPY is the fast way of loading data into PostgreSQL and is transaction safe. That means that if a single error appears within your bulk of data, you will have loaded none of it. pgloader will submit the data again in smaller chunks until it's able to isolate the bad from the good, and then the good is loaded in.

Not quite this kind of data loader

In a recent migration project where we freed data from MySQL into PostgreSQL, we used pgloader again. But the loading time was not fast enough for the service downtime window that we had here. Indeed Python is not known for being the fastest solution around. It's easy to use and to ship to production, but sometimes you not only want to be able to be efficient when writing code, you also need the code to actually run fast too.

Faster data loading

So I began writing a little dedicated tool for that migration in Common Lisp which is growing on me as my personal answer to the burning question: python 2 or python 3? I find Common Lisp to offer an even more dynamic programming environment, an easier language to use, and the result often has performances characteristics way beyond what I can get with python. Between 5 times faster and 121 times faster in some quite stupid benchmark.

Here, with real data, my one shot attempt has been running more than twice as fast as the python version, after about a day of programming.

See what's happening now?

The other thing here is that I've tempted to get pgloader work in parallel, but at the time I didn't know about the Global Interpreter Lock that they didn't find how to remove in Python 3 still, by the way. So my threading attempts at making pgloader work in parallel are pretty useless.

Whereas in Common Lisp I can just use the lparallel lib, which exposes threading facilities and some queueing facilities as a mean to communicate data in between workers, and have my code easily work in parallel for real.

Compatibility

The only drawback that I can see here is that if you've been writing your own reformating modules in python for pgloader (yes you can implement your own reformating module for pgloader), then you would have to port it to Common Lisp. Shout me an email if that's your case.

Next version

So, I think we're going to have a pgloader 3 someday, that will be way faster than the current one, and bundle some more features: real parallel behavior, ability to fetch non local data (connecting to MySQL directly, or HTTP, S3, etc); and I'm thinking about offering a COPY like syntax to drive the loading too, while at it. Also, the ability to discover the set of data to load all by itself when you want to load a whole database: think of it as a special Migration mode of operations.

Some feature requests can't be solved easily when keeping the old .INI syntax cruft, so it's high time to implement some kind of a real command language. I have several ideas about those, in between the COPY syntax and the SQL*Loader configuration format, which is both clunky and quite powerful, too.

After a beginning in TCL and a complete rewrite in python in 2005, it looks like 2013 is going to be the year of pgloader 3, in Common Lisp!

Automated Setup for pgloader

dim@tapoueh.org (Dimitri Fontaine) — Thu, 17 Jan 2013 14:32:00 +0100

Another day, another migration from MySQL to PostgreSQL... or at least that's how it feels sometimes. This time again I've been using some quite old scripts to help me do the migration.

That's how I feel for MySQL users

Migrating the schema

For the schema parts, I've been using mysql2pgsql with success for many years. This tool is not complete and will do only about 80% of the work. As I think that the schema should always be validated manually when doing a migration anyway, I happen to think that it's good news.

Getting the data out

Then for the data parts I keep on using pgloader. The data is never quite right, and the ability to filter out what you can't readily import in a reject file proves itself a a must have here. The problems you have in the exported MySQL data are quite serious:

Can I have my data please?

First, date formating is not compatible with what PostgreSQL expects, sometimes using 20130117143218 instead of what we expect: 2013-01-17 14:32:18, and of course even when the format is right (that seems to depend on the MySQL server's version), you still have to transform the 0000-00-00 00:00:00 into NULL.

Before thinking about the usage of that particular date rather than using NULL when you don't have the information, you might want to remember that there's no year zero in the calendar, it's year 1 BC and then year 1.

Then, text encoding is often mixed up, even when the MySQL databases are said to be in latin1 or unicode, you somehow always end up finding texts in win1252 or some other code page in there.

And of course, MySQL provides no tool to export the data to CSV, so you have to come up with your own. The SELECT INTO OUTFILE command on the server produces non conforming CSV (\n can appear in non-escaped field contents), and while the mysql client manual page details that it outputs CSV when stdout is not a terminal, it won't even try to quote fields or escape \t when they appear in the data.

So, we use the mysqltocsv little script to export the data, and then use that data to feed pgloader.

Loading the data in

Now, we have to write down a configuration file for pgloader to know what to load and where to find the data. What about generating the file from the database schema instead, using the query in generate-pgloader-config.sql:

with reformat as (
   select relname, attnum, attname, typname,
          case typname
               when 'timestamptz'
               then attname || ':mynull:timestamp'
               when 'date'
               then attname || ':mynull:date'
           end as reformat
      from pg_class c
           join pg_namespace n on n.oid = c.relnamespace
           left join pg_attribute a on c.oid = a.attrelid
           join pg_type t on t.oid = a.atttypid
     where c.relkind = 'r'
           and attnum > 0
           and n.nspname = 'public'
),
 config_reformat as (
  select relname,
         '['||relname||']' || E'\n' ||
         'table = ' || relname || E' \n' ||
         'filename = /path/to/csv/' || relname || E'.csv\n' ||
         'format = csv' || E'\n' ||
         'field_sep = \t' || E'\n' ||
         'columns = *' || E' \n' ||
         'reformat = ' || array_to_string(array_agg(reformat), ', ')
         || E'\n' as config
    from reformat
   where reformat is not null
group by relname
),
 noreformat as (
   select relname, bool_and(reformat is null) as noreformating
     from reformat
 group by relname
),
 config_noreformat as (
  select relname,
         '['||relname||']' || E'\n' ||
         'table = ' || relname || E' \n' ||
         'filename = /path/to/csv/' || relname || E'.csv\n' ||
         'format = csv' || E'\n' ||
         'field_sep = \t' || E'\n' ||
         'columns = *' || E' \n'
         || E'\n' as config
    from reformat join noreformat using (relname)
   where noreformating
group by relname
),
allconfs as (
    select relname, config from config_reformat
 union all
    select relname, config from config_noreformat
)
select config
  from allconfs
 where relname not in ('tables', 'wedont', 'wantto', 'load')
 order by relname;

To work with the setup generated, you will have to prepend a global section for pgloader and to include a reformating module in python, that I named mynull.py:

# Author: Dimitri Fontaine <dimitri@2ndQuadrant.fr>
#
# pgloader mysql reformating module

def timestamp(reject, input):
    """ Reformat str as a PostgreSQL timestamp

    MySQL timestamps are ok this time:  2012-12-18 23:38:12
    But may contain the infamous all-zero date, where we want NULL.
    """
    if input == '0000-00-00 00:00:00':
        return None

    return input

def date(reject, input):
    """ date columns can also have '0000-00-00'"""
    if input == '0000-00-00':
        return None

    return input

Now you can launch pgloader and profit!

Conclusion

There are plenty of tools to assist you migrating away from MySQL and other databases. When you make that decision, you're not alone, and it's easy enough to find people to come and help you.

While MySQL is Open Source and is not a lock in from a licencing perspective, I still find it hard to swallow that there's no provided tools for getting data out in a sane format, and that so many little inconsistencies exist in the product with respect to data handling (try to have a NOT NULL column, then enjoy the default empty strings that have been put in there). So at this point, yes, I consider that moving to PostgreSQL is a way to free your data:

Free your data!

Extensions Templates

dim@tapoueh.org (Dimitri Fontaine) — Tue, 08 Jan 2013 17:53:00 +0100

In a recent article titled Inline Extensions we detailed the problem of how to distribute an extension's package to a remote server without having access to its file system at all. The solution to that problem is non trivial, let's say. But thanks to the awesome PostgreSQL Community we finaly have some practical ideas on how to address the problem as discussed on pgsql-hackers, our development mailing list.

PostgreSQL is first an Awesome Community

The solution we talked about is to use templates, and so I've been working on a patch to bring templates for extensions to PostgreSQL. As we're talking about 3 new system catalogs, that's a big patch in term of lines of code. In term of features though, it's quite an easy one.

Here's how it goes. Let's say you want to prepare the system to be able to CREATE EXTENSION pair; without having to install it as an OS package for which you would need to get root access on the server where your PostgreSQL instance is running, which is not always easy, and sometimes not a good idea.

Installing an extension template

With the template patch I just sent on the lists, what you can do is prepare a template with your extension's script and properties, then use it to install the extensions.

create template
   for extension pair default version '1.0'
  with (nosuperuser, norelocatable, schema public)
as $$
  CREATE TYPE pair AS ( k text, v text );

  CREATE OR REPLACE FUNCTION pair(anyelement, text)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

  CREATE OR REPLACE FUNCTION pair(text, anyelement)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

  CREATE OR REPLACE FUNCTION pair(anyelement, anyelement)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

  CREATE OR REPLACE FUNCTION pair(text, text)
  RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair;';
$$;

Installing an extension from a template

With the template installed in the catalogs, now you can go and install your extension:

foo> create extension pair;
CREATE EXTENSION

foo> \dx pair
     List of installed extensions
 Name | Version | Schema | Description
------+---------+--------+-------------
 pair | 1.0     | public |
(1 row)

foo> \dx+ pair
     Objects in extension "pair"
          Object Description
--------------------------------------
 function pair(anyelement,anyelement)
 function pair(anyelement,text)
 function pair(text,anyelement)
 function pair(text,text)
 type pair
(5 rows)

The extension installation is now happening from the catalog templates rather than the file system, which means you didn't need to be root on the system where the server is running. Also note that this example above did happen when connected as the database owner, a user who is not the superuser. Requiring less privileges is always good news, right?

Managing upgrade scripts and extension update

Now that the extension is installed, you might want to update it with some new awesome features. Let's have a look at that.

Upload your Extension Update Scripts

Rather than make a new version of the extension package with the new files in there, then asking the operations team to make the new package available on the internal repositories then install them on the servers, you could now prepare and QA the new setup that way:

create template for extension pair from '1.0' to '1.1'
as $$
  CREATE OPERATOR ~> (LEFTARG = text,
                      RIGHTARG = anyelement,
                      PROCEDURE = pair);

  CREATE OPERATOR ~> (LEFTARG = anyelement,
                      RIGHTARG = text,
                      PROCEDURE = pair);

  CREATE OPERATOR ~> (LEFTARG = anyelement,
                      RIGHTARG = anyelement,
                      PROCEDURE = pair);

  CREATE OPERATOR ~> (LEFTARG = text,
                      RIGHTARG = text,
                      PROCEDURE = pair);
$$;

create template
   for extension pair from '1.1' to '1.2'
as $$
            comment on extension pair is 'Simple Key Value Text Type';
$$;

Of course it's not the most realistic example when you look at the content. In particular the 1.2 version that only adds a comment to the extension. I needed another version to test the automatic upgrade path with more than one step though, so here we go.

foo> alter extension pair update to '1.2';
ALTER EXTENSION

foo> \dx pair
             List of installed extensions
 Name | Version | Schema |        Description
------+---------+--------+----------------------------
 pair | 1.2     | public | Simple Key Value Text Type
(1 row)

foo> \dx+ pair
     Objects in extension "pair"
          Object Description
--------------------------------------
 function pair(anyelement,anyelement)
 function pair(anyelement,text)
 function pair(text,anyelement)
 function pair(text,text)
 operator ~>(anyelement,anyelement)
 operator ~>(anyelement,text)
 operator ~>(text,anyelement)
 operator ~>(text,text)
 type pair
(9 rows)

We did it!

Internals

Let's have a look at those new catalogs:

Oh, that's not quite the internals I expected...

Here we go now:

foo> select * from pg_extension_control;
select * from pg_extension_control;
-[ RECORD 1 ]--+-------
ctlname        | pair
ctlowner       | 32926
ctldefault     | t
ctlrelocatable | f
ctlsuperuser   | f
ctlnamespace   | public
ctlversion     | 1.0
ctlrequires    |

foo> select * from pg_extension_template;
select * from pg_extension_template;
-[ RECORD 1 ]-----------------------------------------------------------------
tplname    | pair
tplowner   | 32926
tplversion | 1.0
tplscript  |
           |   CREATE TYPE pair AS ( k text, v text );
           |
           |   CREATE OR REPLACE FUNCTION pair(anyelement, text)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';
           |
           |   CREATE OR REPLACE FUNCTION pair(text, anyelement)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';
           |
           |   CREATE OR REPLACE FUNCTION pair(anyelement, anyelement)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';
           |
           |   CREATE OR REPLACE FUNCTION pair(text, text)
           |   RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair;';
           |

foo> select * from pg_extension_uptmpl;
select * from pg_extension_uptmpl;
-[ RECORD 1 ]-------------------------------------------------------------
uptname   | pair
uptowner  | 32926
uptfrom   | 1.0
uptto     | 1.1
uptscript |
          |   CREATE OPERATOR ~> (LEFTARG = text,
          |                       RIGHTARG = anyelement,
          |                       PROCEDURE = pair);
          |
          |   CREATE OPERATOR ~> (LEFTARG = anyelement,
          |                       RIGHTARG = text,
          |                       PROCEDURE = pair);
          |
          |   CREATE OPERATOR ~> (LEFTARG = anyelement,
          |                       RIGHTARG = anyelement,
          |                       PROCEDURE = pair);
          |
          |   CREATE OPERATOR ~> (LEFTARG = text,
          |                       RIGHTARG = text,
          |                       PROCEDURE = pair);
          |
-[ RECORD 2 ]-------------------------------------------------------------
uptname   | pair
uptowner  | 32926
uptfrom   | 1.1
uptto     | 1.2
uptscript |
          |     comment on extension pair is 'Simple Key Value Text Type';
          |

As you can see there's nothing too complex here, it's quite straightforward. We need to separate away the creating templates from the updating templates because we need unique keys and we can't have that on NULL columns.

foo> \d pg_extension_template
\d pg_extension_template
Table "pg_catalog.pg_extension_template"
   Column   | Type | Modifiers
------------+------+-----------
 tplname    | name | not null
 tplowner   | oid  | not null
 tplversion | text |
 tplscript  | text |
Indexes:
    "pg_extension_template_name_version_index" UNIQUE, btree (tplname, tplversion)
    "pg_extension_template_oid_index" UNIQUE, btree (oid)

foo> \d pg_extension_uptmpl
\d pg_extension_uptmpl
Table "pg_catalog.pg_extension_uptmpl"
  Column   | Type | Modifiers
-----------+------+-----------
 uptname   | name | not null
 uptowner  | oid  | not null
 uptfrom   | text |
 uptto     | text |
 uptscript | text |
Indexes:
    "pg_extension_uptmpl_name_from_to_index" UNIQUE, btree (uptname, uptfrom, uptto)
    "pg_extension_uptmpl_oid_index" UNIQUE, btree (oid)

Next steps

Now that we have the basics in place, the patch is far from finished still. It needs pg_dump and psql support, support for the function pg_available_extension_versions(), implementing some ALTER TEMPLATE FOR EXTENSION commands for which I only sketched the syntax in the grammar, and some more infrastructure to be able to have ALTER OWNER and ALTER RENAME commands.

Warning: patch brewing here! Syntax and other key elements will change.

All that is pretty technical though, the real thing that patch needs is some quality review and maybe some adjustments. I would be surprised if it didn't need adjustments, really. Because the way the community works, we always need some. That's why the PostgreSQL product is so good!

Inline Extensions

dim@tapoueh.org (Dimitri Fontaine) — Thu, 13 Dec 2012 11:34:00 +0100

We've been having the CREATE EXTENSION feature in PostgreSQL for a couple of releases now, so let's talk about how to go from here. The first goal of the extension facility has been to allow for a clean dump and restore process of contrib modules. As such it's been tailored to the needs of deploying files on the file system because there's no escaping from that when you have to ship binary and executable files, those infamous .so, .dll or .dylib things.

Now that we have the Extension facility though, what we see is a growing number of users taking advantage of it for the purpose of managing in house procedural code and related objects. This code can be a bunch of PLpgSQL or plpython functions and as such you normaly create them directly from any application connection to PostgreSQL.

So the idea would be to allow creating Extensions fully from a SQL command, including the whole set of objects it contains. More than one approach are possible to reach that goal, each with downsides and advantages. We will see them later in that document.

Before that though, let's first review what the extension mechanism has to offer to its users when there's no contrib like module to manage.

A use case for next generation extensions

The only design goal of the 9.1 PostgreSQL Extension feature has been to support a proper dump & restore user experience when using contrib modules such as hstore or ltree. Building up on that, what do Extensions have to offer to non C developpers out there? In other words, what CREATE EXTENSION brings on the table that a bunch of loose objects does not? What problems can we now solve?

A Multi Functions Equipment, All Bundled Together

A way to phrase it is to say that Extensions are user defined CASCADE support. Extensions brings extensibility to the pg_depend PostgreSQL internal dependency tracking system that CASCADE is built on. From that angle, Extensions are a way to manage dependencies of SQL objects in a way that allow you to manage them as a single entity.

One of the existing problems this helps solving is the infamous lack of dependency tracking between function calls. Using Extensions when you deal with a set of functions acting as an API, you can at least protect that as a unit:

STATEMENT: drop function public.populate_record(anyelement,hstore);
    ERROR: cannot drop function populate_record(anyelement,hstore) because
           extension hstore requires it
     HINT: You can drop extension hstore instead.

And you also have a version number and tools integration to manage extensions, with psql \dx command and the equivalent feature in pgAdmin. Coming with your own version number management is not impossible, some do that already. Here it's integrated and the upgrade sequences are offered too (applying 1.1--1.2 then 1.2--1.3 automatically).

Let's just say that it's very easy to understand the traction our users feel towards leveraging Extensions features in order to properly manage their set of stored procedures and SQL objects.

The dump & restore experience

The common problem of all those proposals is very central to the whole idea of Extensions as we know them. The goal of building them as been to fix the restoring experience when using extensions in a database, and we managed to do that properly for contrib likes extensions.

A fly in the ointment

When talking about Inline Extensions, the fly in the ointment is how to properly manage their pg_dump behavior. The principle we built for Extensions and that is almost unique to them is to omit them in the dump files. The only other objects that we filter out of the dump are the one installed at server initialisation times, when using initdb, to be found in the pg_catalog and information_schema systems' schema.

At restore time, the dump file contains the CREATE EXTENSION command so the PostgreSQL server will go fetch the control and script files on disk and process them, loading the database with the right set of SQL objects.

Now we're talking about Extensions which we would maybe want to dump the objects of, so that at restore time we don't need to find them from unknown external resources: the fact that the extension is Inline means that the PostgreSQL server has no way to know where its content is coming from.

The next proposals are trying to address that problem, with more or less success. So far none of them is entirely sastisfying to me, even if a clear temporary winner as emerged on the hackers mailing list, summarized in the in-catalog Extension Scripts and Control parameters (templates?) thread.

Inline Extension Proposals

Now, on to some proposals to make the best out of our all time favorite PostgreSQL feature, the only one that makes no sense at all by itself...

Starting from an empty extension

We already have the facility to add existing loose objects to an extension, and that's exactly what we use when we create an extension for the first time when it used not to be an extension before, with the CREATE EXTENSION ... FROM 'unpackaged'; command.

The hstore--unpackaged--1.0.sql file contains statements such as:

ALTER EXTENSION hstore ADD type hstore;
ALTER EXTENSION hstore ADD function hstore_in(cstring);
ALTER EXTENSION hstore ADD function hstore_out(hstore);
ALTER EXTENSION hstore ADD function hstore_recv(internal);
ALTER EXTENSION hstore ADD function hstore_send(hstore);

Opening CREATE EXTENSION so that it allows you to create a really empty extension would then allow you to fill-in as you need, with as many commands as you want to add objects to it. The control file properties would need to find their way in that design, that sure can be taken care of.

Look me, an Empty Extension!

The main drawback here is that there's no separation anymore in between the extension author, the distribution means, the DBA and the database user. When you want to install a third party Extension using only SQL commands, you could do it with that scheme by using a big script full of one-liners commands.

So that if you screw up your copy/pasting session (well you should maybe reconsider your choice of tooling at this point, but that's another topic), you will end up with a perfectly valid Extension that does not contain what you wanted. As the end user, you have no clue about that until the first time using the extension fails.

CREATE EXTENSION AS

The next idea is to embed the Extension script itself in the command, so as to to get a cleaner command API (in my opinion at least) and a better error message when the paste is wrong. Of course it your paste problem happens to just be loosing a line in the middle of the script there is not so much I can do for you...

CREATE EXTENSION hstore
  WITH parameter = value, ...
AS $$
CREATE TYPE hstore;

CREATE FUNCTION hstore_in(cstring) RETURNS hstore
 AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_out(hstore) RETURNS cstring
AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_recv(internal) RETURNS hstore
AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_send(hstore) RETURNS bytea
AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE TYPE hstore (
        INTERNALLENGTH = -1, STORAGE = extended
        INPUT = hstore_in, OUTPUT = hstore_out,
        RECEIVE = hstore_recv, SEND = hstore_send);
$$;

An edited version of hstore--1.1.sql for vertical space concerns

I've actually proposed a patch to implement that, as you can see in the pg_dump --extension-script commit fest entry. As spoiled by the commit fest entry title, the main problem we have with Inline Extensions is their management in the seamless experience of dump & restore that we are so happy to have now. More about that later, though.

Extension Templates

Another idea is to continue working from control parameters and scripts to install and update extensions, but to have two different places where to find those. Either on the server's File System (when dealing with contribs and shared libraries, there's but a choice), or on the system catalogs.

We Already Have TEXT SEARCH TEMPLATE After All

The idea would then be to have some new specific TEMPLATE SQL Object that would be used to import or upload your control file and create and update scripts in the database, using nothing else than a SQL connection. Then at CREATE EXTENSION time the system would be able to work either from the file system or the template catalogs.

One obvious problem is how to deal with a unique namespace when we split the sources into the file system and the database, and when the file system is typically maintained by using apt-get or yum commands.

Then again I would actually prefer that mechanism better than the other proposals if the idea was to load the file system control and scripts files as TEMPLATEs themselves and then only operate Extensions from Templates. But doing that would mean getting back to the situation where we still are not able to devise a good, simple and robust pg_dump policy for extensions and templates.

Conclusion

I hope to be finding the right solution to my long term plan in this release development cycle, but it looks like the right challenge to address now is to find the right compromise instead. Using the Templates idea already brings a lot on the table, if not the whole set of features I would like to see.

PostgreSQL: Building on Solid Foundations

What would be missing mainly would be the ability for an Extension to switch from being file based to being a template, either because the author decided to change the way he's shipping it, or because the user is switching from using the pgxn client to using proper system packages. I guess that's something we can see about later, though.

Editing SQL

dim@tapoueh.org (Dimitri Fontaine) — Tue, 06 Nov 2012 09:55:00 +0100

It's hard to read my blog yet not know I'm using Emacs. It really is a great tool and has a lot to compare to PostgreSQL in terms of extensibility, documentation quality and community. And there's even a native implementation of the PostgreSQL Protocol written in Emacs Lisp.

One of the things where Emacs really shines is that interactive development environment you get when working on some Emacs Lisp code. Evaluating an function as easy as a single key chord, and that will both compile in the function and load it in the running process. I can't tell you how many times I've been missing that ability when editing C code.

With PostgreSQL too we get a pretty interactive environment with the psql console application, or with pgAdmin. One feature from pgAdmin that I've often wished I had in psql is the ability to edit my query online and easily run it in the console, rather than either using the readline limited history editing features or launching a new editor process each time with \e. At the same time I would much prefer using my usual Emacs editor to actually edit the query.

If you've been reading that blog before you know what to expect. My solution to the stated problem is available in pgdevenv-el, an Emacs package aimed at helping PostgreSQL developers. Most of the features in there are geared toward the core backend developers, except for this one I want to talk about today (I'll blog about the other ones too I guess).

What you can see from that screenshot is that the selected query text has been sent to the psql buffer and exectuted over there. And that the psql buffer is echoing all queries sent to it. What you can not see straight from that picture is the interaction to get there. Well, I've been implementing some elisp features that I was missing.

First, movement: you can do C-M-a and C-M-e to navigate to the beginning and the end of the SQL query at point, like you do in C or in lisp in Emacs.

Then, selection: you can do C-M-h to select the SQL query at point, you don't have to navigate yourself, pgdev-sql-mode knows how to do that. Side note, pgdev-sql-mode is the name of the minor mode you need to activate in your SQL buffers to have the magic available.

Last but not least, evaluation: as when editing lisp code, you can now use C-M-x to send the current query text to an associated psql buffer.

The way to associate the psql buffer to an SQL buffer is currently done thanks to the other pgdevenv-el features that this blog post is not talking about, and the setup is addressed in the documentation: you have to let know pgdevenv-el where your PostgreSQL branches are installed locally so that it can prepare you a Shell buffer with PGDATA and PGPORT already set for you. And currently, for C-M-x to work you need to open the buffer yourself before hand, using C-c - n (to run the command pgdev-open-shell), and type psql in the Shell prompt.

What that means for me is that I can at least edit SQL (in PostgreSQL regression files and other places) in my usual Emacs buffer and actually refine it as I go until it does exactly what I need, without having to use the readline history editing or the \e command, which is not great when your Shell is in already running inside Emacs.

PostgreSQL for developers

dim@tapoueh.org (Dimitri Fontaine) — Fri, 02 Nov 2012 16:22:00 +0100

As Guillaume says, we've been enjoying a great evening conference in Lyon 2 days ago, presenting PostgreSQL to developers. He did the first hour presenting the project and the main things you want to know to start using PostgreSQL in production, then I took the opportunity to be talking to developers to show off some SQL.

That slide deck contains mainly SQL language, but some french too, rather than english. Sorry for the inconvenience if that's not something you can read. Get me to talk at an english developer friendly conference and I'll translate it for you! :)

The aim of that talk is to have people think about SQL as a real asset in their development tool set. SQL really should get compared to your application development language rather than your UI formating language, it's more like PHP or Python than it is like HTML.

So the whole talk is about showing off some advanced SQL features, all provided by default in released PostgreSQL versions. The main parts of the talk all come from an article in this blog: Reset Counter.

Another awesome conf

dim@tapoueh.org (Dimitri Fontaine) — Tue, 30 Oct 2012 12:50:00 +0100

Last week was PostgreSQL Conference Europe 2012 in Prague, and it's been awesome. Many thanks to the organisers who did manage to host a very smooth conference with 290 attendees, including speakers. That means you kept walking into interesting people to talk to, and in particular the Hallway Track has been a giant success.

Photo by Oleg Bartunov

I did have the chance to speak several times at that event, and you can get the slides at my Conferences page that I try to keep up to date. I did one talk about Implementing High Availability that was about 2 hours long (a double slot), PGQ Cooperative Consumers that Marko Kreen copresented with me and the Large Scale MySQL Migration to PostgreSQL that I already presented before this year.

Next conference is in Lyon and will be in French, the talk is called Présentation de PostgreSQL. The audience is going to be composed of PHP developers interested to know more about PostgreSQL, I'll tell you how it goes!

Prefixes and Ranges

dim@tapoueh.org (Dimitri Fontaine) — Tue, 16 Oct 2012 10:47:00 +0200

It's been a long time since I last had some time to spend on the prefix PostgreSQL extension and its prefix_range data type. With PostgreSQL 9.2 out, some users wanted me to update the extension for that release, and hinted me that it was high time that I fix that old bug for which I already had a patch.

`prefix_range` release 1.2.0

I'm sorry it took that long. It's now done, you can have prefix 1.2.0 from https://github.com/dimitri/prefix or if you want a tagged tarball then you can use this link: https://github.com/dimitri/prefix/tarball/v1.2.0.

The changelog is all about fixing an index search bug and updating the package to primarily be an extension for PostgreSQL 9.1 and 9.2. Of course older Major Versions are still supported (all of them since 8.1, but please first consider upgrading PostgreSQL) if you want to install it manually, using the prefix--1.2.0.sql file.

debian package

And thanks to Christoph Berg the debian package is already validated and has reached debian experimental. We don't target sid these days because debian is preparing a new stable release, so there's a freeze. I think. Anyway, take your prefix package from here if you need it: http://packages.debian.org/experimental/postgresql-9.1-prefix.

Range Types

If you step back a little there's an interesting question to answer here. Why isn't prefix_range and PostgreSQL Range Type? Given the names it seems like a pretty good candidate.

Well the thing is that to make a generic range type you need to have a total ordering on the range elements, and a distance function that tells you how far any two elements of a range are one from each other.

When talking about prefixes, I don't see how to do that. The prefix range ['abcd', 'abce') contains an infinity of elements, all the strings that begin with the letters abcd. I guess that coming with an ordering on text is possible, but what if any text element represents a prefix?

I mean that in our case, the elements would be of type prefix, and 'abcd' is a prefix of 'abcdefg'. The question I want to answer is that given a table with prefixes 'abcd', 'abce' and 'abcde' which row in there has the longest prefix matching the literal 'abcdef'.

I'm not seeing how to abuse the Range Types mechanism to implement that, so if you have some ideas please share them!

Reset Counter

dim@tapoueh.org (Dimitri Fontaine) — Fri, 05 Oct 2012 09:44:00 +0200

I've been given a nice puzzle that I think is a good blog article opportunity, as it involves some thinking and window functions.

What's to solve

Say we store in a table entries from a counter that only increases and the time stamp when we did the measurement. So that when you read 30 then later 40 in fact that means we counted 10 more the second reading when compared to the first, in other words the first 30 are counted again in the second counter value, 40.

Now of course it's a real world counter. Think network traffic counter on a network interface, if you want something real to play with in your mind. So the counter will sometime reset and you will read measure sequences such as 40, 0, 20 if you happen to read just when the counter is reset, or most of the time that will look like 45, 25, 50.

The question we want to answer is, given a series of that counter measures, including some resets, what is the current logical value of the counter?

Given the sequence of measures 0, 10, 20, 30, 40, 0, 20, 30, 60 the result we want is 40 + 60, that is 100. Right?

Playing with some data

Let's model an hypothetical dataset easy enough to play with. What about just the previous example? We also need to time stamp the measurements, let's just use a tick for now, as it's easier to think about:

create table measures(tick int, nb int);

insert into measures
     values (1, 0), (2, 10), (3, 20), (4, 30), (5, 40),
            (6, 0), (7, 20), (8, 30), (9, 60);

Now that we have some data in a table to play with, let's try to find out the numbers we are interested in: we only want to keep the latest measure we read on the counter just before it wraps. That means values where the next one (in tick or time stamping order) is lesser than the current counter value.

As we are lucky enough to be playing with the awesome PostgreSQL which brings window functions on the table, we can easily implement just what we said in a readable way:

  select tick, nb,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
              else null
          end as max
    from measures
  window w as (order by tick);

The firt case is the exact translation of the problem as spelled in english in just the previous paragraph where we stated we want to keep the current counter value in case of a wraparound, so I guess it's easy enough to get at.

Then we have a couple of tricks in that query in order to massage the data as we want it. First, the last row of the output won't have a lead, that window function call is going to return NULL. In that case, we keep the current counter value as if we just did a wraparound. And finally, when there's no wraparound, we don't care about the data. Well, for the purpose of knowing the current logical value of the counter, that is.

And we get that encouraging result:

 tick | nb  | max
------+-----+-----
    1 |   0 |
    2 |  10 |
    3 |  20 |
    4 |  30 |
    5 |  40 |  40
    6 |   0 |
    7 |  20 |
    8 |  30 |
    9 |  60 |  60

As you see, we have been able to create a new column out of the dataset, and that new column only contains the data we are interested into.

Finding the current counter value

All we have to do now is sum this computed columns entries. Remember that the sum() aggregate function will simply discard nulls, so that we don't have to turn them into a bunch of 0.

with t(tops) as (
  select case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
              else null
          end as max
    from measures
  window w as (order by tick)
)
select sum(tops) from t;

And here's the expected result:

 sum
-----
 100

Now what about testing with another set of data or two, just to be sure that the counter is allowed to wrap more than once within our solution?

insert into measures
     values (10, 0), (11, 10), (12, 30), (13, 35), (14, 45),
            (15, 25), (16, 50), (17, 100), (18, 110);

Then we have:

with t(tops) as (
   select case when lead(nb) over w < nb then nb
               when lead(nb) over w is null then nb
               else null
           end as max
     from measures
   window w as (order by tick)
)
select sum(tops) from t;
 sum
-----
 255
(1 row)

All good!

Counter logical value over a given period

Now of course what we want is to find the logical value of the counter for a given day's or month's worth of measures. We then need to pay attention to the value of the counter at the start of our period so that we know to substract it from the logical sum over the period.

Here's an SQL version of the same sentence, applied to the period in between ticks 4 and 14, in a completely arbitrary choosing of mine:

with t as (
  select tick,
         first_value(nb) over w as first,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
              else null
          end as max
    from measures
   where tick >= 4 and tick < 14
  window w as (order by tick)
)
select sum(max) - min(first) as sum
 from t;

Here we are using the first_value() window function to retain it in the whole resultset of the Common Table Expression (the inner query introduced by the keyword WITH is called that way). And when doing the sum we're interested in at the outer level, we didn't forget to substract the first value: we need to use an aggregate here because we're doing a sum() aggregate at the same query level, and we have the same value in each row of the resultset, so we used min(), max() would have been as good.

Another important trick we're using in that query is how to express the date range. Never use between for that, as you would end up counting boundaries twice, and customer won't like your accounting process if you do that. Always use a combo of inclusive and exclusive boundaries comparison, as in that WHERE clause in the previous query.

Let's have a quick look at the raw data in that range, using another nice aggregate that PostgreSQL comes with:

select array_agg(nb) from measures where tick >= 4 and tick < 14;
           array_agg
-------------------------------
 {30,40,0,20,30,60,0,10,30,35}
(1 row)

And now, the logical counter value for that period is computed as the following value by the previous query:

 sum
-----
 105
(1 row)

We can verify it manually, we want 40 + 60 + 35 - 30, I think we're all good again. Don't forget we have to substract the first measure from the period!

Extending the problem

Another interesting problem, that we didn't have here but that I find interesting enough to extend this article, is finding the ranges of time (here, ticks) within which the counter didn't reset.

The query is more complex because we need to split the data into partitions, each partition containing data from the same counter series of measures without wrapping. The usual trick is to self-join our data set so that for each given row we have a set of rows from the same partition, we are going to instead use a correlated subquery to go fetch the next wraparound value:

with tops as (
  select tick, nb,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
             else null
         end as max
    from measures
  window w as (order by tick)
)
  select tick, nb, max,
         (select tick
            from tops t2
           where t2.tick >= t1.tick and max is not null
        order by t2.tick
           limit 1) as p
    from tops t1;

 tick | nb  | max | p
------+-----+-----+----
    1 |   0 |     |  5
    2 |  10 |     |  5
    3 |  20 |     |  5
    4 |  30 |     |  5
    5 |  40 |  40 |  5
    6 |   0 |     |  9
    7 |  20 |     |  9
    8 |  30 |     |  9
    9 |  60 |  60 |  9
   10 |   0 |     | 14
   11 |  10 |     | 14
   12 |  30 |     | 14
   13 |  35 |     | 14
   14 |  45 |  45 | 14
   15 |  25 |     | 18
   16 |  50 |     | 18
   17 | 100 |     | 18
   18 | 110 | 110 | 18
(18 rows)

With that as an input it's then possible to build ranges of ticks including non wrapping set of measures from our counter, and get for each range the logical value tat the counter had at the end of it:

with tops as (
  select tick, nb,
         case when lead(nb) over w < nb then nb
              when lead(nb) over w is null then nb
             else null
         end as max
    from measures
  window w as (order by tick)
),
     parts as (
  select tick, nb, max,
         (select tick
            from tops t2
           where t2.tick >= t1.tick and max is not null
        order by t2.tick
           limit 1) as p
    from tops t1
),
     ranges as (
  select first_value(tick) over w as start,
         last_value(tick) over w as end,
         max(max) over w
    from parts
  window w as (partition by p order by tick)
)
select * from ranges where max is not null;

 start | end | max
-------+-----+-----
     1 |   5 |  40
     6 |   9 |  60
    10 |  14 |  45
    15 |  18 | 110
(4 rows)

Conclusion

What I hope to have shown here, apart from some window function tips and some nice use cases for common table expressions, is that as a developper adding SQL to your tool set is a very good idea.

You don't want to have several parts of your code dealing with a logical counter like this, because you want the reporting, accounting, quota, billing and other software to all agree on the values. And you most probably want to avoid to fetch a huge result set of data and process it in the application memory (it'd better fit) rather than just get back a single integer column single row resultset, right?

If you find this SQL example to be off the limits, it's a good sign that you need to improve on your skills so that SQL is a real asset of your developer multi languages multi paradygm talents.

PostgreSQL 9.3

dim@tapoueh.org (Dimitri Fontaine) — Sat, 15 Sep 2012 18:43:00 +0200

PostgreSQL 9.2 is released! It's an awesome new release that I urge you to consider trying and adopting, an upgrade from even 9.1 should be very well worth it, as your hardware could suddenly be able to process a much higher load. Indeed, better performances mean more work done on the same budget, that's the name of the game!

As a PostgreSQL contributor though, the release of 9.2 mainly means to me that it's time to fully concentrate on preparing 9.3. The developement season of which as already begun, by the way, so some amount of work has already been done here.

The list of things I want to be working on for that next release is quite long, and looks more like a christmas list than anything else. Let's only talk about those things I might as well make happen rather than all the things I wish I was able to be delivering in a single release...

Event Triggers

We missed 9.2 for wanting to include too big a feature in one go, leading to too many choices to review and take decision about, for once, and also to some non optimal choices that had to be reconsidered. Thanks to PGCON in Ottawa earlier this year, I could meet in person with Robert Haas and we've been able to decide how to attack that big patch I had. The first step has been to commit in the PostgreSQL tree only infrastructure parts, on which we will be able to build the feature itself.

Infrastructure

What we already have today is the ability to run user defined function when some event occurs, and an event can only be a ddl_command_start as of now. Also the trigger itself must be written in PLpgSQL or PL/C, as the support for the other languages was not included from the patch.

That leaves some work to be done in the next months, right?

PL support

The user defined function will get some information from magic variables such as TG_EVENT and such. That allows easier integration of future information we want to add, without disrupting those existing triggers that you wrote (no API change), at the cost of having to write a specific integration per procedural language.

So one of the first things to do now is to take the support for the others PL that I had in my proposal and make a new patch with only that in there.

Fill-in more information

Then again, this first infrastructure part was all about being actually able to run a user function and left behind most of the information I would like the function to have. The information already there is the command tag, the event name and the parsetree that's only usable if you're writing your trigger in C, which we expect some users to be doing.

To supplement that, we're talking about the Object ID that has been the target of the event, the schema it leaves in when applicable, the Object Name, the Operation that's running (CREATE, ALTER, DROP), the Object Kind being the target of said operation (e.g. TABLE or FUNCTION), and the command string.

Publishing the Command String

Publishing the Command String here is not an easy task, because we have to rebuild a normalized version of it. Or maybe we can go with passing explicit context in which the command is running, such as the search_path.

Even with an explicit context that would be easy enough to SET back again (in a remote server where you would be replicating the DDL, say), it would be better to normalize the command string so as to remove extra spaces and make it easier to parse and process from a user defined function.

That part looks like where most of the work is going to happen in the next commit fests.

Events

The other big thing I want to be working on with respect to this feature is the event support, which is basically hard coded to be ddl_command_start in the current state of the 9.3 code.

We certainly will want to be able to run user defined function not only at the very beginning of a DDL command, but also just before it finishes so that the newly created object already exists, for example.

We might also be interested into supporting triggers on more than DDL, there I doubt we will see that happening in 9.3, as some people in the community would go crazy about complex use cases. Time is limited, and I think this is better kept open for the next release, as the way our beloved PostgreSQL works is by delivering reliable features: quality first.

Use cases

I'm always happy to hear about use cases for the features I'm working on, and this one has the potential to be covering a non trivial amount of them. I already can think of trigger based replication systems and some integrated extension network facilities. With your help we can give those the place they should have: early days use cases in a great collection.

Extensions

So yes, event triggers first use case for me is in relation with extensions. Surprise! There's still some more I want to do with extensions, so much that I could consider their implementation in 9.1 just an enabler. In 9.1 the game has been to offer the best support we could design for existing contrib modules, with a very strong angle toward clean support for dump and restore.

The typical contrib module exports in SQL a list of C coded functions, sometime supporting a new datatype, sometime a set of administration functions. It's quite rare that contrib modules are handling user data embedded in their SQL definition, and when it happens it's mostly configuration kind of data, such as with PostGIS.

Now we want to fully support extensions that are maintaining their own user data, or even those that are all about them. The main difficulty here is that our current design of dump and restore support is following a model where installing the same extesion in a new database is all covered by create extension foo;. This is a limited model of the reality, that we need to expand.

The first manifestation of those problems is in the SEQUENCE support in extensions, and that impacts one of my favorite extensions: PGQ.

PostgreSQL releases

PostgreSQL just released an awesome release with 9.2, where we get tremendous performance optimisations and truly innovative features, such as RANGE TYPE. How not to consider PostgreSQL as a part of your application stack, where to develop and host your features.

While users are enjoying the newer release, contributors are already preparing the next one, hard at work again!

Autumn 2012 Conferences

dim@tapoueh.org (Dimitri Fontaine) — Thu, 02 Aug 2012 01:08:00 +0200

The PostgreSQL community host a number of conferences all over the year, and the next ones I'm lucky enough to get to are approaching fast now. First, next month in September, we have Postgres Open in Chicago, where my talk about Large Scale Migration from MySQL to PostgreSQL has been selected!

This talk shares hindsights about the why and the how of that migration, what problems couldn't be solved without moving away and how the solution now looks. The tools used for migrating away the data, the methods the new architecture are detailed. And the new home, in the cloud!

Not that much later after that the European PostgreSQL community is giving us a very nice occasion to get to Prague with PostgreSQL Conference Europe 2012 (October 23-26). If you've been meaning to meet with the community, if you've been meaning to visit Prague someday, or any mix of those two very good reasons, think about booking that conference already.

The call for papers for pgconf.eu has been extended to August 7th, 2012. Consider sharing your hindsights too!

PGDay France 2012

dim@tapoueh.org (Dimitri Fontaine) — Fri, 08 Jun 2012 16:17:00 +0200

The french PostgreSQL Conference, pgday.fr, was yesterday in Lyon. We had a very good time and a great schedule with a single track packed with 7 talks, addressing a diverse set of PostgreSQL related topics, from GIS to fuzzy logic, including replication.

You might have guessed it already, I did talk about replication. Here's the slide deck I did use, it's in french, sorry if you don't grok that language.

The conference was very nice and did go smoothly, even if we were “only” 60 of us I had the pleasure to meet with different users with very different set of needs. Very happy to have been there!

Back From PgCon

dim@tapoueh.org (Dimitri Fontaine) — Thu, 24 May 2012 09:40:00 +0200

Last week was the annual PostgreSQL Hackers gathering in Canada, thanks to the awesome pgcon conference. This year's issue has been packed with good things, beginning with the Cluster Summit then followed the next day by the Developer Meeting just followed (yes, in the same day) with the In Core Replication Meeting. That was a packed shedule!

The in core replication project has been presented with slides titled Future In-Core Replication for PostgreSQL and got a very good reception. For instance, people implementing Slony (Jan Wieck, Christopher Browne and Steve Singer where here) appreciated the concepts here and where rather supportive of both the requirements and the design, and appreciated the very early demo and results that we had to show already, as a kind of a proof of concepts.

After those first two days, we could start the actual show. I had the honnor to present a migration use case entitled Large Scale MySQL Migration where we're speaking about going from MySQL to PostgreSQL, from 37 to 256 shards, moving more than 6TB of data including binary blobs that we had to process with pl/java. A quite involved migration project whose slides you now can read here:

I've heard that we should soon be able to enjoy audio and video recordings of the sessions, so if you couldn't make it this year for any reason, don't miss that, you will have loads of very interesting talks to virtually attend. I definitely will do that to catch-up with some talks I couldn't attend, having to pick one out of three is not an easy task, all the more when you add the providential hallway track.

Clean PGQ Subconsumers

dim@tapoueh.org (Dimitri Fontaine) — Thu, 26 Apr 2012 15:05:00 +0200

Now that you're all using the wonders of Cooperative Consumers to help you efficiently and reliably implement your business constraints and offload them from the main user transactions, you're reaching a point where you have to clean up your development environment (because that's what happens to development environments, right?), and you want a way to start again from a clean empty place.

Here we go. It used to be much more simple than that, so if you're still using PGQ from Skytools2, just jump to the next step.

Unregister Subconsumers

That query will figure out subconsumers in the system function pgq.get_consumer_info() and ask PGQ to please unregister them, losing events in the way, even events from batches that are currently active.

 with subconsumers as (
   select q1.queue_name,
          q2.consumer_name,
          substring(q1.consumer_name from '%.#"%#"' for '#') as subconsumer_name
    from (select *
            from pgq.get_consumer_info()
           where lag is null)
         as q1
    join (select *
           from pgq.get_consumer_info()
           where lag is not null)
         as q2
         on q1.queue_name = q2.queue_name
)
select *,
       pgq_coop.unregister_subconsumer(queue_name, consumer_name,
                                       subconsumer_name, 1)
 from subconsumers;

Unregister Consumers

Now that the first step is done, we have to unregister the main consumers, which is easy and what you already did before:

select queue_name, consumer_name,
       pgq.unregister_consumer(queue_name, consumer_name)
  from pgq.get_consumer_info();

Drop queues

And as we want to really clean up the mess, let's also drop the queues.

select queue_name, pgq.drop_queue(queue_name)
  from pgq.queue;

PGQ Coop Consumers

dim@tapoueh.org (Dimitri Fontaine) — Mon, 12 Mar 2012 14:43:00 +0100

While working a new PostgreSQL architecture for an high scale project that used to be in the top 10 of internet popular web sites (in terms of visitors), I needed to be able to off load some processing from the main path: that's called a batch job. This needs to be transactional: don't run the job if we did rollback; the transaction, process all events that were part of the same transaction in the same transaction, etc.

That calls for using PGQ, the jobs queue solution from Skytools, the power horse for Londiste. If PGQ is good enough to build a full trigger-based replication solution on top of it, certainly it's good enough for our custom processing, right? Well, you still need to check that your expectations are met, and that was happily the case in my implementation. It's a very common problem, and PGQ very often is a great solution to it.

As this implementation is PHP centric, we've been using libphp-pgq to drive our background workers. Using PGQ in PHP has been very easy to setup, the only trap being not to forget about running the ticker process.

It got interesting because of two elements. First, we're nor running a single database instance here but a bunch of them... make it 256 databases. Each of them having 5 queues to consume, that would be about 1280 consumer processes, distributed on 16 servers that's still 80 per server, so way too many. What we did instead is reuse the queue mover script found in the Skytools distribution and adapt it to forward the event of the 1280 source queues to only 5 destination queues. We then process the events from this single location.

Now it's easier to deal with, but we're not still exactly there. Of course, with so many sources, concentrating them all into the same place means that a single consumer is not able to process the events as fast as they are produced. That's where the cooperative consuming shines, it's very easy to turn your consumer into a cooperative one even on an existing and running queue, and that's what we did. So now we can choose how many workers we want per queue: one of them has 4 workers, another one see not so much activity and 1 worker still fits.

The queue mover script that knows how to subscribe to many queues from the same process is going to be contributed to Skytools proper, of course.

Extension White Listing

dim@tapoueh.org (Dimitri Fontaine) — Thu, 08 Mar 2012 14:25:00 +0100

PostgreSQL 9.1 includes proper extension support, as you might well know if you ever read this very blog here. Some hosting facilities are playing with PostgreSQL at big scale (hello Heroku!) and still meet with small caveats making their life uneasy.

To be specific, only superusers are allowed to install C coded stored procedures, and that impacts a lot of very useful PostgreSQL extension: all those shiped in the contrib package are coded in C. Now, Heroku is not giving away superuser access to their hosted customers in order to limit the number of ways they can shoot themselves in the foot. And given PostgreSQL security model, being granted database owner is mostly good enough for day to day operation.

See Andrew's article Heroku, a really easy way to get a database in a hurry for more context about Heroku's offering here.

Mostly, but as we see, not completely good enough. How to arrange for a non superuser to be able to still install a C-coded extension in his own database? That's quite dangerous as any bug causing a crash would mean a PostgreSQL whole restart. So you not only want to empower CREATE EXTENSION to database owners, you also want to be able to review and explicitely white list the allowed extensions.

Here we go: pgextwlist is a PostgreSQL extensions implementing just that idea. You have to tweak local_preload_libraries so that it gets loaded automatically and early enough, and you have to provide for the list of authorized extensions in the extwlist.extensions setting.

Let's see a usage example, straight from the documentation:

dim=> select rolsuper from pg_roles where rolname = current_user;
select rolsuper from pg_roles where rolname = current_user;
 rolsuper
----------
 f
(1 row)

dim=> create extension hstore;
create extension hstore;
WARNING:  => is deprecated as an operator name
DETAIL:  This name may be disallowed altogether in future versions of PostgreSQL.
CREATE EXTENSION

dim=> create extension earthdistance;
create extension earthdistance;
ERROR:  extension "earthdistance" is not whitelisted
DETAIL: Installing the extension "earthdistance" failed, because it is not
        on the whitelist of user-installable extensions.
HINT: Your system administrator has allowed users to install certain
      extensions. SHOW extwlist.extensions;

dim=> \dx
\dx
                           List of installed extensions
  Name   | Version |   Schema   |                   Description
---------+---------+------------+--------------------------------------------------
 hstore  | 1.0     | public     | data type for storing sets of (key, value) pairs
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

dim=> drop extension hstore;
drop extension hstore;
DROP EXTENSION

As you can see, it allows non superusers to install an extension written in C.

pgbouncer munin plugin

dim@tapoueh.org (Dimitri Fontaine) — Wed, 16 Nov 2011 14:00:00 +0100

It seems that if you search for a munin plugin for pgbouncer it's easy enough to reach an old page of mine with an old version of my plugin, and a broken link. Let's remedy that by publishing here the newer version of the plugin. To be honest, I though it already made its way into the official munin 1.4 set of plugins, but I've not been following closely enough.

As the plugin is 300 lines of python code, it's not a good idea to just inline it here, so please grab it at pgbouncer_.

You might need to know that the script name once installed should follow the form pgbouncer_dbname_stats_requests or pgbouncer_dbname_pools, where of course dbname can contain any number of _ characters. This script supports quite old versions of pgbouncer that didn't accept the normal pq protocol, you did have to use psql to have any chance of getting the data from a script, you couldn't then just use a PostgreSQL driver such as psycopg2.

Back From Amsterdam

dim@tapoueh.org (Dimitri Fontaine) — Wed, 26 Oct 2011 10:08:00 +0200

Another great conference took place last week, PostgreSQL Conference Europe 2011 was in Amsterdam and plenty of us PostgreSQL geeks were too. I attended to lot of talks and did learn some more about our project, its community and its features, but more than that it was a perfect occasion to meet with the community.

Dave Page talked about SQL/MED under the title PostgreSQL at the center of your dataverse and detailed what to expert from a Foreign Data Wrapper in PostgreSQL 9.1, then how to write your own. Wherever you are currently managing your data, you can easily enough make it so that PostgreSQL integrates them by means of fetching them to answer your queries. Which means real time data federating: you don't copy data around, you remote access them when executing the query.

I might need to come up with new Foreign Data Wrappers in a not too distant future, now that I better grasp how much work it really is to do that, it appears to be a good migration strategy too:

  INSERT INTO real.table
       SELECT * FROM foreign.table;

Another discovery is that apparently PLv8 is ready for public consumption. Using it can lead to Heralding the Death of NoSQL, so use it with care.

In the presentation of Synchronous Replication and Durability Tuning we mainly saw that mixing synchronous and asynchronous transactions in your application is the key to real performances across the ocean, as the speed of the light is not infinite. From Baltimore to Amsterdam the latency can not be better than 100ms and that's not the same as instant nowadays.

Then again, depending on the number of concurrent queries to sync over the ocean link, the experimental setup was able to achieve several thousands of queries per second, which is validating the model we picked for sync rep and its implementation.

If you want to read the slides again at home, or if you could not be there for some reason, then most of the talks are now available online at the PostgreSQL Conference Europe Talks 2011 wiki page.

Implementing backups

dim@tapoueh.org (Dimitri Fontaine) — Wed, 12 Oct 2011 22:22:00 +0200

I've been asked about my opinion on backup strategy and best practices, and it so happens that I have some kind of an opinion on the matter.

I tend to think best practice here begins with defining properly the backup plan you want to implement. It's quite a complex matter, so be sure to ask yourself about your needs: what do you want to be protected from?

The two main things to want to protect from are hardware loss (crash disaster, plane in the data center, fire, water flood, etc) and human error (UPDATE without a where clause). Replication is an answer to the former, archiving and dumps to the latter. You generally need both.

Often enough “backups” include WAL archiving and shipping and nightly or weekly base backups, with some retention and some scripts or procedures ready to setup Point In Time Recovery and recover some data without interfering with the WAL archiving and shipping. Of course with PostgreSQL 9.0 and 9.1, the WAL Shipping can be implemented with streaming replication and you can even have a Hot Standby. But for backups you still want archiving.

Mostly I still implement pg_dump -Fc nightly backups with a custom retention (for example, 1 backup a month kept 2 years, 1 backup a week kept 6 or 12 months, 1 backup a night kept 1 to 2 weeks), when the database size allow the pg_dump run to remain constrained in the maintenance window, if any.

Don't forget that while pg_dump runs, you can't roll out DDL changes to the production system any more, so you want to be careful about this maintenance window thing. When you have one.

Physical backups are not locking rollouts away, but they often suck a good deal of the IO bandwidth so you need to pick up a right timing to do them. That's how you can get to once a week base backup and WAL archiving.

If you can't pg_dump production, maybe you can have automated restore jobs from the physical backups that you then pg_dump -Fc, so that you still have that. That can come up handy, really: you can't test your major upgrade out of a physical backup.

Also, obviously, never consider your backup strategy implemented until you have either automated restores in place or a regular schedule to exercise them (staging instances, devel instances).

Then as far as the practical tools go, I tend to think that pg_staging is worth its installation complexity, and for WAL archiving and base backup I recommend walmgr from Skytools, that's a very handy tool. When using PostgreSQL 9.0 or 9.1, consider using walmgr3 so that it's behaving nice alongside streaming replication.

Scaling Stored Procedures

dim@tapoueh.org (Dimitri Fontaine) — Thu, 06 Oct 2011 18:23:00 +0200

In the news recently stored procedures where used as an excuse for moving away logic from the database layer to application layer, and to migrate away from a powerful technology to a simpler one, now that there's no logic anymore in the database.

It's not the way I would typically approach scaling problems, and apparently I'm not alone on the Stored Procedures camp. Did you read this nice blog post Mythbusters: Stored Procedures Edition already? Well it happens in another land that where my comfort zone is, but still has some interesting things to say.

I won't try and address all of the myths they attack in a single article. Let's pick the scalability problems, the two of them I think about are code management and performances. We are quite well equiped for that in PostgreSQL, really.

For code maintainance we now have PostgreSQL Extensions, which allows you to pack all your procedures into separate extensions, and to maintain a version number and upgrade procedures for each of them. You can handle separate rollouts in development for going from 1.12 to 1.13 then 1.14 and after the developers tested it more completely and changed their mind again on the best API they want to work with, 1.15 which is stamped ok for production. At this point, ALTER EXTENSION UPGRADE will happily apply all the rollouts in sequence to upgrade from 1.12 straight to 1.15 in one go. And if you prefer to bake a special careful script to handle that big jump, you also can provide a specific extension--1.12--1.15.sql script.

Of course you're managing all those files with your favorite SCM, to answer to some other myth from the blog reference we are loosely following.

I wanted to talk about the other side of the scalability problem, which is the operations side of it. What happens when you need to scale the database in terms of its size and level of concurrent activity? PostgreSQL earned a very good reputation at being able to scale-up, what about scaling-out? Certainly, now that you're all down into Stored Procedure, it's going to be a very bad situation?

Well, in fact, you're then in a very good position here, thanks to PLproxy. This extension is a custom procedural language whose job is to handle a cluster of database shards that all expose the same PL API, and it's very good at doing that.

Stored Procedures are a very good tool to have, be sure to get comfortable enough with them so that you can choose exactly when to use them. If you're not sure about that, we at 2ndQuadrant will be happy to help you there!

See you in Amsterdam

dim@tapoueh.org (Dimitri Fontaine) — Tue, 04 Oct 2011 14:25:00 +0200

The next PostgreSQL conference is approaching very fast now, I hope you have your ticket already: it's a very promissing event! If you want some help in deciding whether to register or not, just have another look at the schedule. Pick the talks you want to see. It's hard, given how packed with good ones the schedule is. When you're mind is all set, review the list. Registered?

I'll be presenting another talk about extensions, but this time I've geared up to use cases, with Extensions are good for business logic. The idea is not to talk about how to make PostgreSQL play fair with extensions including at dump and restore times, that's already done and I've been talking only too much about it. The idea this time is to figure out how much you get from this feature.

If you ever felt like something is missing in your processes between pushing rollouts in devel environments and refining them as developers are testing and preparing something for the live databases, then we have something for you here. Including how to easily compare state between production and development, but without having to guess or reverse engineer anything.

Yeah, extensions are all about getting even more professional! A great tool you'll be happy to master!

And now I need to prepare a damn good slide deck, right? See you there! :)

Skytools3: walmgr

dim@tapoueh.org (Dimitri Fontaine) — Wed, 21 Sep 2011 17:21:00 +0200

Let's begin the Skytools 3 documentation effort, which is long overdue. The code is waiting for you over at github, and is stable and working. Why is it still in release candidate status, I hear you asking? Well because it's missing updated documentation.

WalMgr is the Skytools component that manages WAL shipping for you, and archiving too. It knows how to prepare your master and standby setup, how to take a base backup and push it to the standby's system, how to archive (at the satndby) master's WAL files as they are produced and have the standby restore from this archive.

What's new in walmgr from Skytools 3 is its support for Streaming Replication that made its way into PostgreSQL 9.0 and is even more useful in PostgreSQL 9.1 (better monitoring, synchronous replication option).

Getting ready

Now, I'm using debian here, and a build virtual machine where I'm doing the backporting work. As PostgreSQL 9.1 is now out, let's use that.

:~$ pg_lsclusters
Version Cluster   Port Status Owner    Data directory
8.4     main      5432 online postgres /var/lib/postgresql/8.4/main ...
9.0     main      5433 online postgres /var/lib/postgresql/9.0/main ...
9.1     main      5434 online postgres /var/lib/postgresql/9.1/main ...

After some editing of the configuration files (enabling hot standby and switching pg_hba.conf to trust for the sake of this example), we can see that the cluster is ready to be abused:

:~$ sudo pg_ctlcluster 9.1 main restart
:~$ psql --cluster 9.1/main  -U postgres \
-c "select name, setting from pg_settings where name in ('max_wal_senders', 'wal_level')"
      name       |   setting
-----------------+-------------
 max_wal_senders | 1
 wal_level       | hot_standby
(2 rows)

:~$ sudo mkdir -p /etc/walshipping/9.1/main /var/lib/postgresql/walshipping
:~$ sudo chown -R postgres:postgres /etc/walshipping /var/lib/postgresql/walshipping

:~$ ssh-keygen -t dsa
:~/.ssh$ cp id_dsa.pub authorized_keys
:~$ ssh localhost

So the order of operations is to prepare a standby, then have it restore from the archives, then activate the wal streaming and check that the setup allows the standby to switch back and forth between the streaming and the archives.

Setting walmgr

To prepare the standby, we will do a base backup of the master. That step is handled by walmgr, so we first need to set it up. Here's the sample master.ini file:

[walmgr]
job_name             = wal-master
logfile              = /var/log/postgresql/%(job_name)s.log
pidfile              = /var/run/postgresql/%(job_name)s.pid
use_skylog           = 0

master_db            = port=5434 dbname=template1
master_data          = /var/lib/postgresql/9.1/main/
master_config        = /etc/postgresql/9.1/main/postgresql.conf
master_bin           = /usr/lib/postgresql/9.1/bin

# set this only if you can afford database restarts during setup and stop.
master_restart_cmd   = pg_ctlcluster 9.1 main restart

slave = 127.0.0.1
slave_config = /etc/walshipping/9.1/main/standby.ini

walmgr_data          = /var/lib/postgresql/walshipping/9.1/main
completed_wals       = %(walmgr_data)s/logs.complete
partial_wals         = %(walmgr_data)s/logs.partial
full_backup          = %(walmgr_data)s/data.master
config_backup        = %(walmgr_data)s/config.backup

# syncdaemon update frequency
loop_delay           = 10.0
# use record based shipping available since 8.2
use_xlog_functions   = 0

# pass -z to rsync, useful on low bandwidth links
compression          = 0

# keep symlinks for pg_xlog and pg_log
keep_symlinks        = 1

# tell walmgr to set wal_level to hot_standby during setup
#hot_standby          = 1

# periodic sync
#command_interval     = 600
#periodic_command     = /var/lib/postgresql/walshipping/periodic.sh

And the /etc/walshipping/9.1/main/standby.ini companion:

[walmgr]
job_name             = wal-standby
logfile              = /var/log/postgresql/%(job_name)s.log
use_skylog           = 0

slave_data           = /var/lib/postgresql/9.1/standby
slave_bin            = /usr/lib/postgresql/9.1/bin
slave_stop_cmd       = pg_ctlcluster 9.1 standby stop
slave_start_cmd      = pg_ctlcluster 9.1 standby start
slave_config_dir     = /etc/postgresql/9.1/standby/

walmgr_data          = /var/lib/postgresql/walshipping/9.1/main
completed_wals       = %(walmgr_data)s/logs.complete
partial_wals         = %(walmgr_data)s/logs.partial
full_backup          = %(walmgr_data)s/data.master
config_backup        = %(walmgr_data)s/config.backup

backup_datadir       = no
keep_backups         = 0
# archive_command =

# primary database connect string for hot standby -- enabling
# this will cause the slave to be started in hot standby mode.
primary_conninfo     = host=127.0.0.1 port=5434 user=postgres

And let's get started:

:~$ cp standby.ini /etc/walshipping/9.1/main/

:~$ walmgr3 -v master.ini setup
2011-09-21 16:57:05,685 30450 INFO Configuring WAL archiving
2011-09-21 16:57:05,687 30450 DEBUG found 'archive_mode' in config -- enabling it
2011-09-21 16:57:05,687 30450 DEBUG found 'wal_level' in config -- setting to 'archive'
2011-09-21 16:57:05,688 30450 DEBUG modifying configuration: {'archive_mode': 'on', 'wal_level': 'archive', 'archive_command': '/usr/bin/walmgr3 /var/lib/postgresql/master.ini xarchive %p %f'}
2011-09-21 16:57:05,688 30450 DEBUG found parameter archive_mode with value ''off''
2011-09-21 16:57:05,690 30450 DEBUG found parameter wal_level with value ''minimal''
2011-09-21 16:57:05,690 30450 DEBUG found parameter archive_command with value ''''
2011-09-21 16:57:05,691 30450 INFO Restarting postmaster
2011-09-21 16:57:05,691 30450 DEBUG Execute cmd: 'pg_ctlcluster 9.1 main restart'
2011-09-21 16:57:09,404 30450 DEBUG Execute cmd: 'ssh' '-Tn' '-o' 'Batchmode=yes' '-o' 'StrictHostKeyChecking=no' '127.0.0.1' '/usr/bin/walmgr3' '/etc/walshipping/9.1/main/standby.ini' 'setup'
2011-09-21 16:57:09,712 30450 INFO Done

postgres@squeeze64:~$ walmgr3 master.ini backup
2011-09-21 17:00:17,259 30702 INFO Backup lock obtained.
2011-09-21 17:00:17,277 30692 INFO Execute SQL: select pg_start_backup('FullBackup'); [port=5434 dbname=template1]
2011-09-21 17:00:17,791 30712 INFO Removing expired backup directory: /var/lib/postgresql/walshipping/9.1/main/data.master
2011-09-21 17:00:18,200 30692 INFO Checking tablespaces
2011-09-21 17:00:18,202 30692 INFO pg_log does not exist, skipping
2011-09-21 17:00:18,259 30692 INFO Backup conf files from /etc/postgresql/9.1/main
2011-09-21 17:00:18,590 30731 INFO First useful WAL file is: 000000010000000200000092
2011-09-21 17:00:19,901 30759 INFO Backup lock released.
2011-09-21 17:00:19,919 30692 INFO Full backup successful

:~$ walmgr3 /etc/walshipping/9.1/main/standby.ini listbackups

List of backups:

Backup set      Timestamp                Label       First WAL
--------------- ------------------------ ----------- ------------------------
data.master     2011-09-21 17:00:17 CEST FullBackup  000000010000000200000092

Following articles will show how to manage that archive and how to go from that to an Hot Standby fed by either Streaming Replication or Archives.

PostgreSQL and debian

dim@tapoueh.org (Dimitri Fontaine) — Mon, 05 Sep 2011 17:14:00 +0200

After talking about it for a very long time, work finally did begin! I'm talking about the apt.postgresql.org build system that will allow us, in the long run, to propose debian versions of binary packages for PostgreSQL and its extensions, compiled for a bunch of debian and ubuntu versions.

We're now thinking to support the i386 and amd64 architectures for lenny, squeeze, wheezy and sid, and also for maverick and natty, maybe oneiric too while at it.

It's still the very beginning of the effort, and it was triggered by the decision to move sid to 9.1. While it's a good decision in itself, I still hate to have to pick only one PostgreSQL version per debian stable release when we have all the technical support we need to be able to support all stable releases that upstream is willing to maintain. If you've been living under a rock, or if you couldn't care less about debian choices, the problem here for debian is ensuring security (and fixes) updates for PostgreSQL — they promise they will handle the job just fine in the social contract, and don't want to have to it without support from PostgreSQL if a debian stable release contains a deprecated PostgreSQL version.

That opens the door for PostgreSQL community to handle the packaging of its solutions as a service to its debian users. We intend to open with support for 8.4, 9.0 and 9.1, and maybe 8.3 too, as Christoph Berg is doing good progress on this front. See, it's teamwork here!

We still have more work to do, and setting up the build environment so that we are able to provide the packages for so much targets will indeed be interesting. Getting there, a step after another.

pg_restore -L & pg_staging

dim@tapoueh.org (Dimitri Fontaine) — Mon, 29 Aug 2011 18:05:00 +0200

On the PostgreSQL Hackers mailing lists, Andrew Dunstan just proposed some new options for pg_dump and pg_restore to ease our lives. One of the answers was talking about some scripts available to exploit the pg_restore listing that you play with using options -l and -L, or the long name versions --list and --use-list. The pg_staging tool allows you to easily exploit those lists too.

The pg_restore list is just a listing of one object per line of all objects contained into a custom dump, that is one made with pg_dump -Fc. You can then tweak this listing in order to comment out some objects (prepending a ; to the line where you find it), and give your hacked file back to pg_restore --use-list so that it will skip them.

What's pretty useful here, among other things, is that a table will have in fact more than one line in the listing. One is for the TABLE definition, another one for the TABLE DATA. So that pg_staging is able to provide you with options for only restoring some schemas, some schemas_nodata and even some tablename_nodata_regexp, to use directly the configuration options names.

How to do a very simple exclusion of some table's data when restoring a dump, will you ask me? There we go. Let's first prepare an environment, where I have only a PostgreSQL server running.

$ git clone git://github.com/dimitri/pg_staging.git
$ git clone git://github.com/dimitri/pgloader.git
$ for s in */*.sql; do psql -f $s; done
$ pg_dump -Fc > pgloader.dump

Now I have a dump with some nearly random SQL objects in it, let's filter out the tables named reformat and parallel from that. We will take the sample setup from the pg_staging project. Going the quick route, we will not even change the default sample database name that's used, which is postgres. After all, the catalog command of pg_staging that we're using here is a developer command, you're supposed to be using pg_staging for a lot more services that just this one.

$ cp pg_staging/pg_staging.ini .
$ (echo "schemas = public";
   echo "tablename_nodata_regexp = parallel,reformat") \
  >> pg_staging.ini
$ echo "catalog postgres pgloader.dump" \
   | python pg_staging/pg_staging.py -c pg_staging.ini
 ; Archive created at Mon Aug 29 17:17:49 2011
 ;
 ; [EDITED OUTPUT]
 ;
 ; Selected TOC Entries:
 ;
3; 2615 2200 SCHEMA - public postgres
1864; 0 0 COMMENT - SCHEMA public postgres
1536; 1259 174935 TABLE public parallel dimitri
1537; 1259 174943 TABLE public partial dimitri
1538; 1259 174951 TABLE public reformat dimitri
;1853; 0 174935 TABLE DATA public parallel dimitri
1854; 0 174943 TABLE DATA public partial dimitri
;1855; 0 174951 TABLE DATA public reformat dimitri
1834; 2606 174942 CONSTRAINT public parallel_pkey dimitri
1836; 2606 174950 CONSTRAINT public partial_pkey dimitri
1838; 2606 174955 CONSTRAINT public reformat_pkey dimitri

We can see that the objects indeed are skipped, now how to really go about the pg_restore is like that:

$ createdb foo
$ echo "catalog postgres pgloader.dump" \
 |python pg_staging/pg_staging.py -c pg_staging.ini > short.list
$ pg_restore -L short.list -d foo pgloader.dump

The little bonus with using pg_staging is that when filtering out a schema it will track all tables and triggers from that schema, and also the functions used in the trigger definition. Which is not as easy as it sounds, believe me!

The practical use case is when filtering out PGQ and Londiste, then the PGQ triggers will automatically be skipped by pg_staging rather than polluting the pg_restore logs because the CREATE TRIGGER command could not find the necessary implementation procedure.

Skytools, version 3

dim@tapoueh.org (Dimitri Fontaine) — Fri, 26 Aug 2011 21:30:00 +0200

You can find skytools3 in debian experimental already, it's in release candidate status. What's missing is the documentation, so here's an idea: I'm going to make a blog post series about skytools next features, how to use them, what they are good for, etc. This first article of the series will just list what are those new features.

Here are the slides from the CHAR(11) talk I made last month, about that very subject:

The new version comes with a lot of new features. PGQ now is able to duplicate the queue events from one node to the next, so that it's able to manage switching over. To do that we have three types of nodes now, root, branch and leaf. PGQ also supports cooperative consumers, meaning that you can share the processing load among many consumers, or workers.

Londiste now benefits from the switch over feature, and is packed with new little features like add

--create, the new --trigger-flags argument, and the new --handler thing (to do e.g. partial table replication). Let's not forget the much awaited execute