tapoueh.org

<?xml version="1.0" encoding="utf-8"?> <rss version="2.0">

<channel> <title>tail -f /dev/dim</title> <link>http://tapoueh.org/index.html</link> <description>Dimitri Fontaine's blog</description> <language>en-us</language> <generator>Emacs Muse</generator> <item> <title>from Parsing to Compiling</title> <link>http://tapoueh.org/blog/2013/05/13-from-parser-to-compiler.html</link> <description><![CDATA[Last week came with two bank hollidays in a row, and I took the opportunity to design a command language for <a href="../../../pgsql/pgloader.html">pgloader</a>. While doing that, I unexpectedly stumbled accross a very nice AHAH! moment, and I now want to share it with you, dear reader.

<center> <img src="../../../images/lightbulb.gif" alt=""> </center> <center> AHAH, you'll see! </center> The general approach I'm following code wise with that command language is to first get a code API to expose the capabilities of the system, then somehow plug the command language into that API thanks to a parser. It turns out that doing so in Common Lisp is really easy, and that you can get a compiler for free too, while at it. Let's see about that. <h3>A very simple toy example</h3> In this newsgroup article <a href="https://groups.google.com/forum/?fromgroups=#!topic/comp.lang.lisp/JJxTBqf7scU">What is symbolic compoutation?</a>, <a href="http://informatimago.com/">Pascal Bourguignon</a> did propose a very simple piece of code: <pre class="src"> (defparameter *additive-color-graph*

'((red (red white) (green yellow) (blue magenta)) (green (red yellow) (green white) (blue cyan)) (blue (red magenta) (green cyan) (blue white))))

(defun symbolic-color-add (a b)

(cadr (assoc a (cdr (assoc b additive-color-graph))))) </pre>

This is an example of symbolic computation, and we're going to build a little language to express the data and the code. Not that we would need to build one, mind you, more in order to have a really simple example leading us to the ahah moment you're now waiting for. Before we dive into the main topic, you have to realize that the previous code example actually works: it's defining some data, using an implicit data structure composed by nesting lists together, and defines a function that knows how to sort out the data in that anonymous data structure so as to compound 2 colors together. <pre class="src"> TOY-PARSER> (symbolic-color-add 'red 'green) YELLOW </pre> <h3>A command language and parser</h3> I decided to go with the following language: <pre class="src"> color red +red white +green yellow +blue magenta color green +red yellow +green white +blue cyan color blue +red magenta +green cyan +blue white

mix red and green </pre>

And here's how some of the parser looks like, using the <a href="http://nikodemus.github.io/esrap/">esrap</a> packrat lib: <pre class="src"> (defrule color-name (and whitespaces (+ (alpha-char-p character)))

(:destructure (ws name) (declare (ignore ws)) ; ignore whitespaces ;; CL symbols default to upper case. (intern (string-upcase (coerce name 'string)) :toy-parser)))

;;; parse string "+ red white" (defrule color-mix (and whitespaces "+" color-name color-name)

(:destructure (ws plus color-added color-obtained) (declare (ignore ws plus)) ; ignore whitespaces and keywords (list color-added color-obtained)))

;;; mix red and green (defrule mix-two-colors (and kw-mix color-name kw-and color-name)

(:destructure (mix c1 and c2) (declare (ignore mix and)) ; ignore keywords (list c1 c2))) </pre>

Those rules are not the whole parser, go have a look at the project on github if you want to see the whole code, it's called <a href="https://github.com/dimitri/toy-parser">toy-parser</a> over there. The main idea here is to show that when we parse a line from our little language, we produce the simplest possible structured data: in lisp that's symbols and lists. The reason why it makes sense doing that is the next rule: <center> <img src="../../../images/the-one-ring.jpg" alt=""> </center> <center> The one grammar rule to bind them all </center> <pre class="src"> (defrule program (and colors mix-two-colors)

(:destructure (graph (c1 c2)) `(lambda () (let ((additive-color-graph ',graph)) (symbolic-color-add ',c1 ',c2))))) </pre>

This rule is the complex one to bind them all. It's using a quasiquote, a basic lisp syntax element allowing the programmer to very easily produce data that looks exactly like code. Let's see how it goes with a very simple example: <pre class="src"> TOY-PARSER> (pprint (parse 'program

"color red +green yellow mix green and red"))

(LAMBDA NIL

(LET ((ADDITIVE-COLOR-GRAPH '((RED (GREEN YELLOW))))) (SYMBOLIC-COLOR-ADD 'RED 'GREEN)))

</pre> The parser is producing structure (nested) data that really looks like lisp code, right? So maybe we can just run that code... <h3>What about a compiler now?</h3> <center> <img src="../../../images/aha.jpg" alt=""> </center> <center> Here is my AHAH moment! </center> Let's see about actually running the code: <pre class="src"> TOY-PARSER> (let* ((code "color red +green yellow mix green and red")

(program (parse 'program code))) (compile nil program)) Function #x3020027CF0EF> NIL NIL TOY-PARSER> (let* ((code "color red +green yellow mix green and red") (program (parse 'program code))) (funcall (compile nil program))) YELLOW </pre>

So we have a string reprensing code in our very little language, and a parser that know how to produce a nested list of atoms that looks like lisp code. And as we have lisp, we can actually compile that code at run-time with the same compiler that we used to produce our parser, and we can then funcall that function we just built. Oh and the function is actually compiled down to native code, of course: <pre class="src"> TOY-PARSER> (let* ((code "color red +green yellow mix red and green")

(program (parse 'program code)) (func (compile nil program))) (time (loop repeat 1000 do (funcall func))))

(LOOP REPEAT 1000 DO (FUNCALL FUNC)) took 108 microseconds (0.000108 seconds) to run. During that period, and with 4 available CPU cores,

105 microseconds (0.000105 seconds) were spent in user mode 13 microseconds (0.000013 seconds) were spent in system mode NIL </pre>

Yeah, it took the whole of 108 microseconds to actually run the code generated by our own parser a thousand times, on my laptop. I can believe it's been compiled to native code, that seems like the right ballpark. <h3>Conclusion</h3> The <a href="https://github.com/dimitri/toy-parser">toy-parser</a> code is there on GitHub and you can actually load it using <a href="http://www.quicklisp.org/">Quicklisp</a>: clone the repository in ~/quicklisp/local-projects/ then (ql:quickload "toy-parser"), and play with it in (in-package :toy-parser). The only thing I still want to say here is this: can your programming language of choice make it that easy? ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 13 May 2013 11:08:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/05/13-from-parser-to-compiler.html</guid> </item> <item> <title>Nearest Big City</title> <link>http://tapoueh.org/blog/2013/05/02-nearest-big-city.html</link> <description><![CDATA[In this article, we want to find the town with the greatest number of inhabitants near a given location.

<center> <img src="../../../images/global_accessibility-640.png" alt=""> </center> <h3>A very localized example</h3> We first need to find and import some data, and I found at the following place a <a href="http://www.lion1906.com/Pages/francais/utile/telechargements.html">CSV listing of french cities with coordinates and population</a> and some numbers of interest for the exercise here. To import the data set, we first need a table, then a COPY command: <pre class="src"> CREATE TABLE lion1906 (

insee text, nom text, altitude integer, code_postal text, longitude double precision, latitude double precision, pop99 bigint, surface double precision );

\copy lion1906 from 'villes.csv' with csv header delimiter ';' encoding 'latin1' </pre>

With that data in place, we can find the 10 nearest towns of a random choosing of us, let's pick Villeurbanne which is in the region of Lyon. <pre class="src">

select code_postal, nom, pop99 from lion1906 order by point(longitude, latitude) <-> (select point(longitude, latitude) from lion1906 where nom = 'Villeurbanne') limit 10;

code_postal

nom

pop99

————-+————————+———

</span> 69100	Villeurbanne	124215
69300	Caluire-et-Cuire	41233
69120	Vaulx-en-Velin	39154
69580	Sathonay-Camp	4336
69140	Rillieux-la-Pape	28367
69000	Lyon	445452
69500	Bron	37369
69580	Sathonay-Village	1693
01700	Neyron	2157
69660	Collonges-au-Mont-d<span style="color: #73d216;">'Or	3420

(10 rows) </pre>

We find Lyon in our list in there, and we want the query now to return only that one as it has the greatest number of inhabitants in the list: <pre class="src"> with neighbours as (

select code_postal, nom, pop99 from lion1906 order by point(longitude, latitude) <-> (select point(longitude, latitude) from lion1906 where nom = 'Villeurbanne') limit 10 ) select * from neighbours order by pop99 desc limit 1;

code_postal

nom

pop99

————-+——+———

69000

Lyon

445452

(1 row) </pre>

Well, thank you PostgreSQL, that was easy! Note that you can actually index such queries, that's called a KNN index. PostgreSQL knows how to use some kind of indexes to fetch data matching an expression such as ORDER BY a <-> b, which allow you to consider a KNN search in your application. <h3>Let's get worldwide</h3> The real scope of our exercise is to associate every known town in the world with some big city around, so let's first fetch and import some worldwide data this time, from <a href="maxmind">http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz</a>. <center> <img src="../../../images/map_nearest_city_01.gif" alt=""> </center> <pre class="src"> CREATE TABLE maxmind_worldcities (

country_code text, city_lower text, city_normal text, region_code text DEFAULT '', population INT DEFAULT '0', latitude float8 DEFAULT '0', longitude float8 DEFAULT '0' );

\copy maxmind_worldcities FROM '/tmp/worldcitiespop.txt' WITH DELIMITER ',' QUOTE E'\f' CSV HEADER ENCODING 'LATIN1';

alter table maxmind_worldcities add column loc point; update maxmind_worldcities set loc = point(longitude, latitude); </pre>

This time you can see that I created an extra column with the location in there, so that I don't have to compute it each time I need it, like I did before. Now is the time to test that data set and hopefully fetch the same result as before when we only had french cities loaded: <pre class="src"> with neighbours as (

select country_code, city_lower, population from maxmind_worldcities where population is not null order by loc <-> (select loc from maxmind_worldcities where city_lower = 'villeurbanne') limit 10 ) select * from neighbours order by population desc limit 1;

country_code

city_lower

population

—————+————+————

fr

lyon

463700

(1 row) </pre>

Ok, looks like we're all set for the real problem. Now we want to pick for each of those cities it's nearest neighboor, so here's how to do that: <pre class="src"> create index on maxmind_worldcities(country_code, region_code, city_lower); create index on maxmind_worldcities using gist(loc);

create table maxmind_neighbours as

select country_code, region_code, city_lower, (with neighbours as ( select country_code, city_lower, population from maxmind_worldcities where population is not null and country_code = wc.country_code and region_code = wc.region_code order by loc <-> wc.loc limit 10) select city_lower from neighbours order by population desc limit 1 ) as neighbour from maxmind_worldcities wc ; </pre>

To be fair, I have to tell you that this query took almost 2 hours to complete on my laptop here, but as I'm doing that for friend and a blog article, I've been lazy and didn't try to optimise it. It could be using LATERAL for sure, I don't know if that would help very much with performances: I didn't try. With that in hands we can now check some cities and their biggest neighbours, as in the following query: <pre class="src"> select * from maxmind_neighbours where city_lower = 'villeurbanne';

country_code

region_code

city_lower

neighbour

—————+————-+—————+————

fr

villeurbanne

lyon

(1 row) </pre>

And looking for New-York City suburbs I did find a chinatown, which is a pretty common smaller town name apparently: <pre class="src"> select * from maxmind_neighbours where city_lower = 'chinatown';

country_code

region_code

city_lower

neighbour

—————+————-+————+—————

</span> sb	08	chinatown	honiara
us	CA	chinatown	san francisco
us	DC	chinatown	washington
us	HI	chinatown	honolulu
us	IL	chinatown	chicago
us	MT	chinatown	missoula
us	NV	chinatown	reno
us	NY	chinatown	<span style="color: #729fcf;">new</span> york

(8 rows) </pre>

<h3>Big Cities in the big world</h3> <center> <img src="../../../images/Old-Photos-of-Big-Cities-21.jpg" alt=""> </center> <center> We might need to change some of our views </center> So, let's see how many smaller towns each of those random big cities have: <pre class="src">

select country_code, region_code, neighbour, count() from maxmind_neighbours where neighbour in ('london', 'new york', 'moscow', 'paris', 'tokyo', 'sao polo', 'chicago') group by country_code, region_code, neighbour;

country_code region_code neighbour count

—————+————-+————+——-

 gb H9 london 2

jp 40 tokyo 414

us NY new york 131

ca 08 london 16

ru 48 moscow 245

fr A8 paris 16

us IL chicago 13

(7 rows) </pre>
And now let's be fair and see where are the cities with the greatest number of towns nearby them, with the following query: <pre class="src">

select country_code, region_code, neighbour, count() from maxmind_neighbours where neighbour is not null group by country_code, region_code, neighbour order by 4 desc limit 25;

country_code

region_code

neighbour

count

—————+————-+————+——-

</span> cn	03	nanchang	16759
cn	26	xian	12864
<span style="color: #729fcf;">id</span>	18	kupang	10715
cn	24	taiyuan	10550
mm	11	taunggyi	10253
<span style="color: #729fcf;">id</span>	38	makasar	9471
ir	15	ahvaz	9461
<span style="color: #729fcf;">id</span>	01	banda aceh	9161
cn	14	lasa	8841
cn	15	lanzhou	8618
ir	29	kerman	8579
<span style="color: #729fcf;">id</span>	26	medan	7787
ir	04	iranshahr	7249
ir	07	shiraz	7219
ma	55	agadir	7121
ir	42	mashhad	7107
af	08	gazni	7011
ir	33	tabriz	6586
cn	01	hefei	6521
bd	81	dhaka	6480
ir	08	rasht	6471
<span style="color: #729fcf;">id</span>	17	mataram	6467
<span style="color: #729fcf;">id</span>	33	cilegon	6287
af	23	qandahar	6213
cn	07	fuzhou	6089

(25 rows) </pre> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 02 May 2013 11:34:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/05/02-nearest-big-city.html</guid> </item> <item> <title>Emacs Conference</title> <link>http://tapoueh.org/blog/2013/04/02-Emacs-Conference.html</link> <description><![CDATA[Yes it did happen, for real, in London: the <a href="http://emacsconf.herokuapp.com/">Emacs Conference</a>. It was easter week-end. Yet the conference managed to have more than 60 people meet together and spend a full day talking about <a href="http://www.gnu.org/software/emacs/">Emacs</a>. If you weren't there, a live stream was available and soon enough (wait for about two weeks) the video material will be published, as <a href="http://sachachua.com/blog/">sacha</a> is working on it.

<center> <img src="../../../images/toplap-small.png" alt=""> </center> The conference has been packed with awesome really. Among the things that I'm going home with are new thoughts, tricks and tips, and new modes to use in Emacs. The main new though is all about learning to program. That's a problem space in which I have a growing interest in, and the conference talk about arxana showed that it should be possible to build an environment where you can learn programming with the excuse of having fun with maths. And after talking about music and its notation and <a href="http://www.lilypond.org/">lilypond</a>, it should even be possible to offer some interactive programming environment where not only you can play music live as <a href="http://meta-ex.com/">Meta-Ex</a> is doing, but where the other output of your program would be the updated music scores. The main practical bits I'm going home with is <a href="http://www.foldr.org/~michaelw/projects/redshank">redshank</a>, A collection of code-wrangling Emacs macros mostly geared towards Common Lisp, but some are useful for other Lisp dialects, too. That complements <a href="http://mumble.net/~campbell/emacs/paredit.el">paredit</a> and allows you to do some reformating very easily. Lastly, I'm back to giving the dark background environment a try now. I think I prefer the contrast and richer color sets of the default Emacs color theme, but the black window has some classy visual effect too. And with <a href="https://github.com/jasonm23/emacs-mainline">main-line</a> the effect is quite awesome! <center> <img src="../../../images/Emacs-Tango-2-Main-Line.png" alt=""> </center> <center> Look at that! </center> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 02 Apr 2013 09:56:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/04/02-Emacs-Conference.html</guid> </item> <item> <title>The Need For Speed</title> <link>http://tapoueh.org/blog/2013/03/29-the-need-for-speed.html</link> <description><![CDATA[Hier se tenait la <a href="http://www.postgresql-sessions.org/en/5/start">cinquième édition</a> de la conférence organisée par dalibo, où des intervenants extérieurs sont régulièrement invités. Le thème hier était à la fois clair et très vaste : la performance.

J'ai eu le plaisir de réaliser une présentation intitulée « The Need for Speed » dans laquelle on replace l'effort d'optimisation dans son contexte métier, afin de faire une étude des coûts et bénéfices et de savoir non seulement à quoi s'attendre mais aussi quand s'arrêter. <center> <a class="image-link" href="../../../images/confs/the_need_for_speed.pdf"> <img src="../../../images/confs/the_need_for_speed-3.png"></a> </center> Merci à dalibo pour cette conférence ! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 29 Mar 2013 09:49:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/29-the-need-for-speed.html</guid> </item> <item> <title>Bulk Replication</title> <link>http://tapoueh.org/blog/2013/03/18-bulk-replication.html</link> <description><![CDATA[In the previous article here we talked about how to properly update more than one row at a time, under the title <a href="http://tapoueh.org/blog/2013/03/15-batch-update.html">Batch Update</a>. We did consider performances, including network round trips, and did look at the behavior of our results when used concurrently.

<center> <img src="../../../images/clock-key.jpg" alt=""> </center> A case where we want to apply the previous article approach is when replicating data with a trigger based solution, such as <a href="http://wiki.postgresql.org/wiki/SkyTools">SkyTools</a> and <a href="https://github.com/markokr/skytools">londiste</a>. Well, maybe not in all cases, we need to have a amount of UPDATE trafic worthy of setting up the solution. As soon as we know we're getting to replay important enough batches of events, though, certainly using the batch update tricks makes sense. It so happens that londiste 3 includes the capability to use handlers. Those are plugins written in python (like all the client side code from SkyTools) whose job is to handle the processing of the event batches. Several of them are included in the <a href="https://github.com/markokr/skytools/tree/master/python/londiste">londiste sources</a>, and one of them is named bulk.py. <h3>Bulk loading data with londiste</h3> To use set in londiste.ini: <pre class="src"> handler_modules = londiste.handlers.bulk </pre> then add table with one of those commands: <pre class="src"> londiste3 add-table xx —handler="bulk" londiste3 add-table xx —handler="bulk(method=X)" </pre> The default method is 0, and the available methods are the following: correct (0) <ul> <li>inserts as COPY into table</li> <li>update as COPY into temp table and single UPDATE from there</li> <li>delete as COPY into temp table and single DELETE from there</li> </ul> delete (1) <ul> <li>as correct, but update are done as DELETE then COPY</li> </ul> merged (2) <ul> <li>as delete, but merge insert rows with update rows</li> </ul> <h3>Conclusion</h3> <center> <img src="../../../images/londiste.jpg" alt=""> </center> Yes, by using that handler which is provided by default in londiste, you will apply the previous article tricks in your replication solution. And you can even choose to use that for only some of the tables you are replicating. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 18 Mar 2013 14:54:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/18-bulk-replication.html</guid> </item> <item> <title>Batch Update</title> <link>http://tapoueh.org/blog/2013/03/15-batch-update.html</link> <description><![CDATA[Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

<center> <img src="../../../images/Home-Brewing.jpg" alt=""> </center> <center> Another kind of Batch to update </center> So when you need to UPDATE a bunch of rows from a given source, remember that you can actually use a JOIN in the update statement. Either the source of data is already in the database, in which case it's as simple as using the FROM clause in the update statement, or it's not, and we're getting back to that in a minute. <h3>UPDATE FROM</h3> It's all about using that FROM clause in an update statement, right? <pre class="src">

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that. <h3>Preparing the Batch</h3> Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row. <center> <img src="../../../images/round-trip.png" alt=""> </center> <center> What about that round trip? </center> Let's see how it goes: <pre class="src"> CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t

SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

As we're talking about performances, the trick here is to use the <a href="http://www.postgresql.org/docs/9.2/static/sql-copy.html">COPY</a> protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches. Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol. <h3>Insert or Update</h3> And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all <a href="http://www.postgresql.org/docs/9.2/static/dml.html">DML</a> queries, which means that you can do the following just fine: <pre class="src"> WITH upd AS (

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id RETURNING s.id ) INSERT INTO target(id, counter) SELECT id, sum(counter) FROM source s LEFT JOIN upd t USING(id) WHERE t.id IS NULL GROUP BY s.id RETURNING t.id </pre>

That query here is updating all the rows that are known in both the target and the source and returns what we took from the source in the operation, so that we can do an anti-join in the next step of the query, where we're inserting any row that was not taken care of in the update part of the statement. Note that when the batch gets to bigger size it's usually better to join against the target table in the INSERT statement, because that will have an index on the join key. <h3>Concurrency patterns</h3> Now, you will tell me that we just solved the UPSERT problem. Well what happens if more than one transaction is trying to do the

WITH (UPDATE)
INSERT

dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong? <center> <img src="../../../images/gophermegaphones.480.jpg" alt=""> </center> <center> Concurrent processing </center> What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table. There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical. The other thing you can do, is force the concurrent transactions to serialize one after the other, using an <a href="http://www.postgresql.org/docs/9.2/static/explicit-locking.html">explicit locking</a> statement: <pre class="src"> LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE; </pre> That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock. <h3>Conclusion</h3> <center> <a class="image-link" href="http://www.flickr.com/photos/asquarephotography/6841106459/in/photostream/"> <img src="../../../images/stack-of-old-books.jpg"></a> </center> The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough <a href="http://www.postgresql.org/about/featurematrix/">PostgreSQL releases</a> you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 15 Mar 2013 10:47:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/15-batch-update.html</guid> </item> <item> <title>Batch Update</title> <link>http://tapoueh.org/blog/2013/03/15-batch-update.html</link> <description><![CDATA[Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

COPY source FROM STDIN;

UPDATE target t

SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id RETURNING s.id ) INSERT INTO target(id, counter) SELECT id, sum(counter) FROM source s LEFT JOIN upd t USING(id) WHERE t.id IS NULL GROUP BY s.id RETURNING t.id </pre>

WITH (UPDATE)
INSERT

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 15 Mar 2013 10:47:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/15-batch-update.html</guid> </item> <item> <title>Batch Update</title> <link>http://tapoueh.org/blog/2013/03/15-batch-update.html</link> <description><![CDATA[Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that. <h3>Preparing the Batch</h3> Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row. <center> <img src="../../../images/round-trip.png" alt=""> </center> <center> What about that round trip? </center> Let's see how it goes: <pre class="src"> CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t

SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

As we're talking about performances, the trick here is to use the <a href="http://www.postgresql.org/docs/9.2/static/sql-copy.html">COPY</a> protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches. Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol. <h3>Insert or Update</h3> And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all <a href="http://www.postgresql.org/docs/9.2/static/dml.html">DML</a> queries, which means that you can do the following just fine: <pre class="src"> WITH upd AS (

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id RETURNING s.id ) INSERT INTO target(id, counter) SELECT id, sum(counter) FROM source s LEFT JOIN upd t USING(id) WHERE t.id IS NULL GROUP BY s.id RETURNING t.id </pre>

WITH (UPDATE)
INSERT

dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong? <center> <img src="../../../images/gophermegaphones.480.jpg" alt=""> </center> <center> Concurrent processing </center> What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table. There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical. The other thing you can do, is force the concurrent transactions to serialize one after the other, using an <a href="http://www.postgresql.org/docs/9.2/static/explicit-locking.html">explicit locking</a> statement: <pre class="src"> LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE; </pre> That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock. <h3>Conclusion</h3> <center> <a class="image-link" href="http://www.flickr.com/photos/asquarephotography/6841106459/in/photostream/"> <img src="../../../images/stack-of-old-books.jpg"></a> </center> The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough <a href="http://www.postgresql.org/about/featurematrix/">PostgreSQL releases</a> you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 15 Mar 2013 10:47:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/15-batch-update.html</guid> </item> <item> <title>Batch Update</title> <link>http://tapoueh.org/blog/2013/03/15-batch-update.html</link> <description><![CDATA[Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that. <h3>Preparing the Batch</h3> Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row. <center> <img src="../../../images/round-trip.png" alt=""> </center> <center> What about that round trip? </center> Let's see how it goes: <pre class="src"> CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t

SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

As we're talking about performances, the trick here is to use the <a href="http://www.postgresql.org/docs/9.2/static/sql-copy.html">COPY</a> protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches. Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol. <h3>Insert or Update</h3> And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all <a href="http://www.postgresql.org/docs/9.2/static/dml.html">DML</a> queries, which means that you can do the following just fine: <pre class="src"> WITH upd AS (

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id RETURNING s.id ) INSERT INTO target(id, counter) SELECT id, sum(counter) FROM source s LEFT JOIN upd USING(id) WHERE t.id IS NULL GROUP BY s.id RETURNING t.id </pre>

WITH (UPDATE)
INSERT

dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong? <center> <img src="../../../images/gophermegaphones.480.jpg" alt=""> </center> <center> Concurrent processing </center> What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table. There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical. The other thing you can do, is force the concurrent transactions to serialize one after the other, using an <a href="http://www.postgresql.org/docs/9.2/static/explicit-locking.html">explicit locking</a> statement: <pre class="src"> LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE; </pre> That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock. <h3>Conclusion</h3> <center> <a class="image-link" href="http://www.flickr.com/photos/asquarephotography/6841106459/in/photostream/"> <img src="../../../images/stack-of-old-books.jpg"></a> </center> The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough <a href="http://www.postgresql.org/about/featurematrix/">PostgreSQL releases</a> you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 15 Mar 2013 10:47:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/15-batch-update.html</guid> </item> <item> <title>Batch Update</title> <link>http://tapoueh.org/blog/2013/03/15-batch-update.html</link> <description><![CDATA[Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

Using that, you can actually update thousands of rows in our target table in a single statement, and you can't really get faster than that. <h3>Preparing the Batch</h3> Now, if you happen to have the source data in your application process' memory, the previous bits is not doing you any good, you think. Well, the trick is that pushing your in-memory data into the database and then joining against the now local source of data is generally faster than looping in the application and having to do a whole network round trip per row. <center> <img src="../../../images/round-trip.png" alt=""> </center> <center> What about that round trip? </center> Let's see how it goes: <pre class="src"> CREATE TEMP TABLE source(LIKE target INCLUDING ALL) ON COMMIT DROP;

COPY source FROM STDIN;

UPDATE target t

SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id </pre>

As we're talking about performances, the trick here is to use the <a href="http://www.postgresql.org/docs/9.2/static/sql-copy.html">COPY</a> protocol to fill in the temporary table we just create to hold our data. So we're now sending the whole data set in a temporary location in the database, then using that as the UPDATE source. And that's way faster than doing a separate UPDATE statement per row in your batch, even for small batches. Also, rather than using the SQL COPY command, you might want to look up the docs of the PostgreSQL driver you are currently using in your application, it certainly includes some higher level facilities to deal with pushing the data into the streaming protocol. <h3>Insert or Update</h3> And now sometime some of the rows in the batch have to be updated while some others are new and must be inserted. How do you do that? Well, PostgreSQL 9.1 brings on the table WITH support for all <a href="http://www.postgresql.org/docs/9.2/static/dml.html">DML</a> queries, which means that you can do the following just fine: <pre class="src"> WITH upd AS (

UPDATE target t SET counter = t.counter + s.counter, FROM source s WHERE t.id = s.id RETURNING s.id ) INSERT INTO target(id, counter) SELECT id, sum(counter) FROM source s LEFT JOIN upd USING(id) WHERE t.id IS NULL GROUP BY s.id RETURNING t.id </pre>

WITH (UPDATE)
INSERT

dance at the same time? It's a single statement, so it's a single snapshot. What can go wrong? <center> <img src="../../../images/gophermegaphones.480.jpg" alt=""> </center> <center> Concurrent processing </center> What happens is that as soon as the concurrent sources contain some data for the same primary key, you get a duplicate key error on the insert. As both the transactions are concurrent, they are seeing the same target table where the new data does not exists, and both will conclude that they need to INSERT the new data into the target table. There are two things that you can do to avoid the problem. The first thing is to make it so that you're doing only one batch update at any time, by architecting your application around that constraint. That's the most effective way around the problem, but not the most practical. The other thing you can do, is force the concurrent transactions to serialize one after the other, using an <a href="http://www.postgresql.org/docs/9.2/static/explicit-locking.html">explicit locking</a> statement: <pre class="src"> LOCK TABLE target IN SHARE ROW EXCLUSIVE MODE; </pre> That lock level is not automatically acquired by any PostgreSQL command, so the only way it helps you is when you're doing that for every transaction you want to serialize. When you know you're not at risk (that is, when not playing the insert or update dance), you can omit taking that lock. <h3>Conclusion</h3> <center> <a class="image-link" href="http://www.flickr.com/photos/asquarephotography/6841106459/in/photostream/"> <img src="../../../images/stack-of-old-books.jpg"></a> </center> The SQL language has its quirks, that's true. It's been made for efficient data processing, and with recent enough <a href="http://www.postgresql.org/about/featurematrix/">PostgreSQL releases</a> you even have some advanced pipelining facilities included in the language. Properly learning how to make the most out of that old component of your programming stack still makes a lot of sense today! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 15 Mar 2013 10:47:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/15-batch-update.html</guid> </item> <item> <title>Emacs Conference</title> <link>http://tapoueh.org/blog/2013/03/04-Emacs-Conference.html</link> <description><![CDATA[The <a href="http://emacsconf.herokuapp.com/">Emacs Conference</a> is happening, it's real, and it will take place at the end of this month in London. Check it out, and register at <a href="http://emacsconf.eventbrite.co.uk/">Emacs Conference Event Brite</a>. It's free and there's still some availability.

<center> <img src="../../../images/emacs-rocks-logo.png" alt=""> </center> <center> It's all about Emacs, and it rocks! </center> We have a great line-up for this conference, which makes me proud to be able to be there. If you've ever been paying attention when using <a href="http://www.gnu.org/software/emacs/">Emacs</a> then you've already heard those names: <a href="http://sachachua.com/blog/">Sacha Chua</a> is frequently blogging about how she manages to improve her workflow thanks to <a href="http://www.gnu.org/software/emacs/emacs-lisp-intro/">Emacs Lisp</a>, <a href="https://github.com/jwiegley">John Wiegley</a> is a proficient Emacs contributor maybe best known for his <a href="https://github.com/ledger/ledger">ledger</a> Emacs Mode, then we have <a href="http://www.lukego.com/">Luke Gorrie</a> who hacked up <a href="http://wingolog.org/archives/2006/01/02/slime">SLIME</a> among other things, we also have <a href="http://nic.ferrier.me.uk/">Nic Ferrier</a> who is starting a revolution in how to use Emacs Lisp with <a href="http://elnode.org/">elnode</a>. And more! Including <a href="http://en.wikipedia.org/wiki/Steve_Yegge">Steve Yegge</a>! <center> See you there in London. </center> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 04 Mar 2013 13:58:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/03/04-Emacs-Conference.html</guid> </item> <item> <title>HyperLogLog Unions</title> <link>http://tapoueh.org/blog/2013/02/26-hll-union.html</link> <description><![CDATA[In the article from yesterday we talked about <a href="http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html">PostgreSQL HyperLogLog</a> with some details. The real magic of that extension has been skimmed over though, and needs another very small article all by itself, in case you missed it.

<center> <img src="../../../images/SetOperations.480.png" alt=""> </center> <center> Which Set Operation do you want for counting unique values? </center> The first query here has the default level of magic in it, really. What happens is that each time we do an update of the HyperLogLog hash value, we update some data which are allowing us to compute its cardinality. <pre class="src"> > select date,

#users as daily, pg_column_size(users) as bytes from daily_uniques order by date;

date

daily

bytes

————+——————+——-

</span> 2013-02-22	401676.779509985	1287
2013-02-23	660187.271908359	1287
2013-02-24	869980.029947449	1287
2013-02-25	580865.296677817	1287
2013-02-26	240569.492722719	1287

(5 rows) </pre>

And has advertized the data is kept in a static sized data structure. The magic here all happens at hll_add() time, the function you have to call to update the data. Now on to something way more magic! <center> <img src="../../../images/aggregates2.jpg" alt=""> </center> <center> Are those the aggregates you're looking for? </center> <pre class="src"> > select to_char(date, 'YYYY/MM') as month,

round(#hll_union_agg(users)) as monthly from daily_uniques group by 1;

month

monthly

———+———

2013/02

1960380

(1 row) </pre>

The HyperLogLog data structure is allowing the implementation of an union algorithm that will be able to compute how many unique values you happen to have registered in both one day and the next. Extended in its general form, and doing SQL, what you get is an aggregate that you can use in GROUP BY constructs and <a href="http://www.postgresql.org/docs/9.2/static/tutorial-window.html">window functions</a>. Did you read about them yet? ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 26 Feb 2013 12:44:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/26-hll-union.html</guid> </item> <item> <title>PostgreSQL HyperLogLog</title> <link>http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</link> <description><![CDATA[If you've been following along at home the newer statistics developments, you might have heard about this new <a href="http://research.google.com/pubs/pub40671.html">State of The Art Cardinality Estimation Algorithm</a> called <a href="http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/">HyperLogLog</a>. This technique is now available for PostgreSQL in the extension <a href="http://blog.aggregateknowledge.com/2013/02/04/open-source-release-postgresql-hll/">postgresql-hll</a> available at <a href="https://github.com/aggregateknowledge/postgresql-hll">https://github.com/aggregateknowledge/postgresql-hll</a> and soon to be in debian.

<center> <img src="../../../images/cardinality1.jpg" alt=""> </center> <center> How to Compute Cardinality? </center> <h3>Installing postgresql-hll</h3> It's as simple as CREATE EXTENSION hll; really, even if to get there you must have installed the package on your system. We did some packaging work for debian and the result should appear soon in a distro near you. Then you also need to keep your data in some table, straight from the documentation we can use that schema: <pre class="src"> — Create the destination table CREATE TABLE daily_uniques ( DATE DATE UNIQUE, users hll ); </pre> Then to add some data for which you want to know the cardinality of, it's as simple as in the following UPDATE statement: <pre class="src"> UPDATE daily_uniques

SET users = hll_add(users, hll_hash_text('123.123.123.123')) WHERE date = current_date; </pre>

So in our example what you see is that we want to decipher how many unique IP addresses we saw, and we do that by first creating a hash of that source data then calling hll_add() with the current value and the hash result. The current value must be initialized using hll_empty(). <h3>Concurrency</h3> The most awake readers among you have already spotted that: using an UPDATE on the same row over and over again is a good recipe to kill any form of concurrency, so you don't want to do that on your production setup unless you don't care about those UPDATE waiting piling up in your system. The idea is then to fill-in a queue of updates and asynchronously update the daily_uniques table from that queue, possibly using the hll_add_agg aggregate that the extension provides, so that you do only one update per batch of values to process. <h3>∅: Empty Set and NULL</h3> <center> <img src="../../../images/EmptySet_L.gif" alt=""> </center> <center> Yes there's a <a href="http://www.unicodemap.org/details/0x2205/index.html">unicode</a> entry for that, ∅ </center> Now, what happens when the batch of new unique values you want to update from is itself empty? Well I would have expected hll_add_agg over an empty set to return an empty hll value, the same as returned by hll_empty(), but it turns out it's returning NULL instead. And then hll_add(users, NULL) will happily return NULL. So the next UPDATE is cancelling all the previous work, which is not nice. We had to cater for that case explicitely in the UPDATE query that's working from the batch of new values to add to our current HyperLogLog hash entry, and I can't resist to show off one of the most awesome PostgreSQL features here: writable CTE. <pre class="src"> WITH hll(agg) AS (

SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch ) UPDATE daily_uniques SET users = CASE WHEN hll.agg IS NULL THEN users ELSE hll_union(users, hll.agg) END FROM hll WHERE date = current_date; </pre>

That's how you protect against an empty set being turned into a NULL. I think the real fix would need to be included in postgresql-hll itself, in making it so that the hll_add_agg aggregate returns hll_empty() on an empty set, and I will report that bug (with that very article as the detailed explanation of it). <h3>Using postgresql-hll</h3> When using postgresql-hll on the production system, we were able to get some good looking numbers from our daily_uniques table: <pre class="src"> with stats as (

select date, #users as daily, #hll_union_agg(users) over() as total from daily_uniques ) select date, round(daily) as daily, round((daily/total*100)::numeric, 2) as percent from stats order by date;

date

daily

percent

————+———+———

</span> 2013-02-22	401677	25.19
2013-02-23	660187	41.41
2013-02-24	869980	54.56
2013-02-25	154996	9.72

(4 rows) </pre>

I coulnd't resist to show off two of my favorite SQL constructs in that example query here, which are the <a href="http://www.postgresql.org/docs/9.2/static/queries-with.html">Common Table Expressions</a> (or CTE) and <a href="http://www.postgresql.org/docs/9.2/static/tutorial-window.html">window functions</a>. If that over() clause reads strange to you, take a minute now and go read about it. Yes, do that now, we're waiting. The data here is showing that we did setup the facility in the middle of the first day, and that the morning's activity is quite low. <h3>Conclusion</h3> <center> <img src="../../../images/hll-dv-estimator.png" alt=""> </center> <center> The <a href="http://blog.aggregateknowledge.com/author/wwkae/">HyperLogLog DV estimator</a> </center> When using postgresql-hll you need to be careful not to kill your application concurrency abilities, and you need to protect yourself against the ∅ killer too. The other thing to keep in mind is that the numbers you get out of the hll technique are estimates within a given precision, and you might want to read some more about what it means for your intended usage of the feature. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 25 Feb 2013 10:23:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</guid> </item> <item> <title>PostgreSQL HyperLogLog</title> <link>http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</link> <description><![CDATA[If you've been following along at home the newer statistics developments, you might have heard about this new <a href="http://research.google.com/pubs/pub40671.html">State of The Art Cardinality Estimation Algorithm</a> called <a href="http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/">HyperLogLog</a>. This technique is now available for PostgreSQL in the extension <a href="http://blog.aggregateknowledge.com/2013/02/04/open-source-release-postgresql-hll/">postgresql-hll</a> available at <a href="https://github.com/aggregateknowledge/postgresql-hll">https://github.com/aggregateknowledge/postgresql-hll</a> and soon to be in debian.

SET users = hll_add(users, hll_hash_text('123.123.123.123')) WHERE date = current_date; </pre>

SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch ) UPDATE daily_uniques SET users = CASE WHEN hll.agg IS NULL THEN users ELSE hll_union(users, hll.agg) END FROM hll WHERE date = current_date; </pre>

select date, #users as daily, #hll_union_agg(users) over() as total from daily_uniques ) select date, round(daily) as daily, round((daily/total*100)::numeric, 2) as percent from stats order by date;

date

daily

percent

————+———+———

</span> 2013-02-22	401677	25.19
2013-02-23	660187	41.41
2013-02-24	869980	54.56
2013-02-25	154996	9.72

(4 rows) </pre>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 25 Feb 2013 10:23:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</guid> </item> <item> <title>PostgreSQL HyperLogLog</title> <link>http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</link> <description><![CDATA[If you've been following along at home the newer statistics developments, you might have heard about this new <a href="http://research.google.com/pubs/pub40671.html">State of The Art Cardinality Estimation Algorithm</a> called <a href="http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/">HyperLogLog</a>. This technique is now available for PostgreSQL in the extension <a href="http://blog.aggregateknowledge.com/2013/02/04/open-source-release-postgresql-hll/">postgresql-hll</a> available at <a href="https://github.com/aggregateknowledge/postgresql-hll">https://github.com/aggregateknowledge/postgresql-hll</a> and soon to be in debian.

SET users = hll_add(users, hll_hash_text('123.123.123.123')) WHERE date = current_date; </pre>

SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch ) UPDATE daily_uniques SET users = CASE WHEN hll.agg IS NULL THEN users ELSE hll_union(users, hll.agg) END FROM hll WHERE date = current_date; </pre>

select date, #users as daily, #hll_union_agg(users) over() as total from daily_uniques ) select date, round(daily) as daily, round((daily/total*100)::numeric, 2) as percent from stats order by date;

date

daily

percent

————+———+———

</span> 2013-02-22	401677	25.19
2013-02-23	660187	41.41
2013-02-24	869980	54.56
2013-02-25	154996	9.72

(4 rows) </pre>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 25 Feb 2013 10:23:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</guid> </item> <item> <title>PostgreSQL HyperLogLog</title> <link>http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</link> <description><![CDATA[If you've been following along at home the newer statistics developments, you might have heard about this new <a href="http://research.google.com/pubs/pub40671.html">State of The Art Cardinality Estimation Algorithm</a> called <a href="http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/">HyperLogLog</a>. This technique is now available for PostgreSQL in the extension <a href="http://blog.aggregateknowledge.com/2013/02/04/open-source-release-postgresql-hll/">postgresql-hll</a> available at <a href="https://github.com/aggregateknowledge/postgresql-hll">https://github.com/aggregateknowledge/postgresql-hll</a> and soon to be in debian.

SET users = hll_add(users, hll_hash_text('123.123.123.123')) WHERE date = current_date; </pre>

SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch ) UPDATE daily_uniques SET users = CASE WHEN hll.agg IS NULL THEN users ELSE hll_union(users, hll.agg) END FROM hll WHERE date = current_date; </pre>

select date, #users as daily, #hll_union_agg(users) over() as total from daily_uniques ) select date, round(daily) as daily, round((daily/total*100)::numeric, 2) as percent from stats order by date;

date

daily

percent

————+———+———

</span> 2013-02-22	401677	25.19
2013-02-23	660187	41.41
2013-02-24	869980	54.56
2013-02-25	154996	9.72

(4 rows) </pre>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 25 Feb 2013 10:23:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</guid> </item> <item> <title>PostgreSQL HyperLogLog</title> <link>http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</link> <description><![CDATA[If you've been following along at home the newer statistics developments, you might have heard about this new <a href="http://research.google.com/pubs/pub40671.html">State of The Art Cardinality Estimation Algorithm</a> called <a href="http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/">HyperLogLog</a>. This technique is now available for PostgreSQL in the extension <a href="http://blog.aggregateknowledge.com/2013/02/04/open-source-release-postgresql-hll/">postgresql-hll</a> available at <a href="https://github.com/aggregateknowledge/postgresql-hll">https://github.com/aggregateknowledge/postgresql-hll</a> and soon to be in debian.

SET users = hll_add(users, hll_hash_text('123.123.123.123')) WHERE date = current_date; </pre>

SELECT hll_add_agg(hll_hash_text(value)) FROM new_batch ) UPDATE daily_uniques SET users = CASE WHEN hll.agg IS NULL THEN users ELSE hll_union(users, hll.agg) END FROM hll WHERE date = current_date; </pre>

select date, #users as daily, #hll_union_agg(users) over() as total from daily_uniques ) select date, round(daily) as daily, round((daily/total*100)::numeric, 2) as percent from stats order by date;

date

daily

percent

————+———+———

</span> 2013-02-22	401677	25.19
2013-02-23	660187	41.41
2013-02-24	869980	54.56
2013-02-25	154996	9.72

(4 rows) </pre>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 25 Feb 2013 10:23:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/25-postgresql-hyperloglog.html</guid> </item> <item> <title>Playing with pgloader</title> <link>http://tapoueh.org/blog/2013/02/12-playing-with-pgloader.html</link> <description><![CDATA[While making progress with both <a href="http://wiki.postgresql.org/wiki/Event_Triggers">Event Triggers</a> and <a href="http://tapoueh.org/blog/2013/01/08-Extensions-Templates.html">Extension Templates</a>, I needed to make a little break. My current keeping sane mental exercise seems to mainly involve using Common Lisp, a programming language that ships with about all the building blocks you need.

<center> <img src="../../../images/made-with-lisp.png" alt=""> </center> <center> Yes, that old language brings so much on the table </center> When using Common Lisp, you have an awesome interactive development environment where you can redefine function and objects while testing them. That means you don't have to quit the interpreter, reload the new version of the code and put the interactive test case together all over again after a change. Just evaluate the change in the interactive environement: functions are compiled incrementally over their previous definition, objects whose classes have changed are migrated live. See, I just said objects and classes. Common Lisp comes with some advanced Object Oriented Programming facilities named <a href="http://www.aiai.ed.ac.uk/~jeff/clos-guide.html">CLOS</a> and <a href="http://www.alu.org/mop/index.html">MOP</a> where the Java and Python and C++ object models are just a subset of what you're being offered. Hint, those don't have <a href="http://en.wikipedia.org/wiki/Multiple_dispatch">Multiple Dispatch</a>. And you have a very sophisticated <a href="http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html">Condition System</a> where Exceptions are just a subset of what you can do (hint: have a look a <a href="http://www.gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html#restarts">restarts</a> and tell me you didn't wish your programming language of choice had them). And it continues that way for about any basic building bloc you might want to be using. <h3>Loading data</h3> Back to <a href="http://tapoueh.org/pgsql/pgloader.html">pgloader</a> will you tell me. Right. I've been spending a couple of evening on hacking on the new version of pgloader in Common Lisp, and wanted to share some preliminary results. <center> <img src="../../../images/toy-loader.320.jpg" alt=""> </center> <center> Playing with the loader </center> The current status of the new pgloader still is pretty rough, if you're not used to develop in Common Lisp you might not find it ready for use yet. I'm still working on the internal APIs and trying to make something clean and easy to use for a developer, and then I will provide some external ways to play with it, user oriented. I missed that step once with the Python based version of the tool, I don't want to do the same errors again this time. So here's a test run with the current pgloader, on a small enough data set of 226 MB of CSV files. <pre class="src"> time python pgloader.py -R.. —summary -Tc ../pgloader.dbname.conf

Table name

duration

size

copy rows

errors

====================================================================

aaaaaaaaaa_aaaa	2.148s	-	24595	0
bbbbbbbbbb_bbbb...\| 0.609s	-	326	0
cccccccccc_cccc...\| 2.868s	-	25126	0
dddddddddd_dddd...\| 0.638s	-	8	0
eeeeeeeeee_eeee...\| 2.874s	-	36825	0
ffffffffff_ffffff	0.667s	-	624	0
gggggggggg_gggg...\| 0.847s	-	5638	0
hhh_hhhhhhh	9.907s	-	120159	0
iii_iiiiiiiiiiiii	0.574s	-	661	0
jjjjjjj	6.647s	-	30027	0
kkk_kkkkkkkkk	0.439s	-	12	0
lll_llllll	0.308s	-	4	0
mmmm_mmm	2.139s	-	29669	0
nnnn_nnnnnn	8.555s	-	100197	0
oooo_ooooo	13.781s	-	93555	0
pppp_ppppppp	8.275s	-	76457	0
qqqq_qqqqqqqqqqqq	8.568s	-	126159	0

====================================================================

Total

01m09.902s

670042

</pre> <h3>Streaming data</h3> With the new code in Common Lisp, I could benefit from real multi threading and higher level abstraction to make it easy to use: <a href="http://lparallel.org/">lparallel</a> is a lib providing exactly what I need here, with workers and queues to communicate data in between them. What I'm doing is that two threads are separated, one is reading the data from either a CSV file or a MySQL database directly, and pushing that data in the queue; while the other thread is pulling data from the queue and writing it into our <a href="http://www.postgresql.org/">PostgreSQL</a> database. <pre class="src"> CL-USER> (pgloader.csv:import-database "dbname"

:csv-path-root "/path/to/csv/" :separator #\Tab :quote #\" :escape "\"\"" :null-as ":null:") table name read imported errors time

——— ——— ——— ——— aaaaaaaaaa_aaaa 24595 24595 0 0.995s bbbbbbbbbb_bbbbbbbbb 326 326 0 0.570s cccccccccc_cccccccccccc 25126 25126 0 1.461s dddddddddd_dddddddddd_dd 8 8 0 0.650s eeeeeeeeee_eeeeeeeeee_eeeeeeee 36825 36825 0 1.664s ffffffffff_ffffff 624 624 0 0.707s gggggggggg_ggggg_gggggggg 5638 5638 0 0.655s hhh_hhhhhhh 120159 120159 0 3.415s iii_iiiiiiiiiiiii 661 661 0 0.420s jjjjjjj 30027 30027 0 2.743s kkk_kkkkkkkkk 12 12 0 0.327s lll_llllll 4 4 0 0.315s mmmm_mmm 29669 29669 0 1.182s nnnn_nnnnnn 100197 100197 0 2.206s oooo_ooooo 93555 93555 0 9.683s pppp_ppppppp 76457 76457 0 5.349s qqqq_qqqqqqqqqqqq 126159 126159 0 2.495s

——— ——— ——— ——— Total import time 670042 670042 0 34.836s NIL </pre>

As you can see the control is still made for interactive developer usage, which is fine for now but will have to change down the road, when the APIs stabilize. Now, let's compare to reading directly from MySQL: <pre class="src"> CL-USER> (pgloader.mysql:stream-database "dbname")

table name read imported errors time

——— ——— ——— ——— aaaaaaaaaa_aaaa 24595 24595 0 0.887s bbbbbbbbbb_bbbbbbbbb 326 326 0 0.617s cccccccccc_cccccccccccc 25126 25126 0 1.497s dddddddddd_dddddddddd_dd 8 8 0 0.582s eeeeeeeeee_eeeeeeeeee_eeeeeeee 36825 36825 0 1.697s ffffffffff_ffffff 624 624 0 0.748s gggggggggg_ggggg_gggggggg 5638 5638 0 0.923s hhh_hhhhhhh 120159 120159 0 3.525s iii_iiiiiiiiiiiii 661 661 0 0.449s jjjjjjj 30027 30027 0 2.546s kkk_kkkkkkkkk 12 12 0 0.330s lll_llllll 4 4 0 0.323s mmmm_mmm 29669 29669 0 1.227s nnnn_nnnnnn 100197 100197 0 2.489s oooo_ooooo 93555 93555 0 9.148s pppp_ppppppp 76457 76457 0 6.713s qqqq_qqqqqqqqqqqq 126159 126159 0 4.571s

——— ——— ——— ——— Total streaming time 670042 670042 0 38.272s NIL </pre>

The streaming here is a tad slower than the importing from files. Now if you want to be fair when comparing those, you would have to take into account the time it takes to export the data out from its source. When doing that export/import dance, a quick test shows a timing of 1m4.745s. Now, if we do an export only test, it runs in 31.822s. So yes streaming is a good thing to have here. <h3>Conclusion</h3> We just got twice as fast as the python version. Some will say that I'm not comparing fairly to the Python version of pgloader here, because I could have implemented the streaming facility in Python too. Well actually I did, the option are called <a href="http://tapoueh.org/pgsql/pgloader.html#sec13">section_threads</a> and <a href="http://tapoueh.org/pgsql/pgloader.html#sec15">split_file_reading</a>, that you can set so that a reader is pushing data into a set of queues and several workers are feeding each from its own queue. It didn't help with performances at all. Once again, read about the infamous <a href="http://docs.python.org/3/c-api/init.html#threads">Global Interpreter Lock</a> to understand why not. <center> <img src="../../../images/lisplogo_flag_128.png" alt=""> </center> So actually it's a fair comparison here where the new code is twice as fast as the previous one, with only some hours of hacking and before spending any time on optimisation. Well, apart from using a producer, a consumer and a queue, which I almost had to have for streaming in between two database connections anyways. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 12 Feb 2013 11:17:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/12-playing-with-pgloader.html</guid> </item> <item> <title>Marking whole word</title> <link>http://tapoueh.org/blog/2013/02/04-Emacs-mark-word.html</link> <description><![CDATA[I've discovered recently another Emacs facility that I since then use several times a day, and I wonder how I did without it before: C-M-SPC runs the command mark-sexp.

<center> <img src="../../../images/sexp.gif" alt=""> </center> <center> Well, mark-sexp apparently is related to the Sex Pistols </center> It's pretty simple actually, when you have the point at the beginning of a word or an identifier (containing numbers, dashes, underscores and other punctuation signs), you can select the whole of it in a single key chord! The best thing is that if you press the same key chord again, it will expand to include the next expression. And that works in plain text and most programming languages where I've tried it, which is not so much recently. It does not depend that much on the programming language anyway. The full general solution here is to use something like <a href="https://github.com/magnars/expand-region.el">expand region</a>, don't miss the <a href="http://emacsrocks.com/e09.html">Emacs Rocks Expand Region Episode</a>, it's less than 3 minutes and you will want to install expand-region after that. For easy installing, of course you are already using <a href="http://tapoueh.org/emacs/el-get.html">el-get</a> right? Now, a friend just asked this morning how to select the current word even when the the point is currently in the middle of it. Going manually back to the beginning of it is no fun. I knew about thing-at-point and a little about how it works, but didn't find anything readily made for that use case (hint: it needs to be an interactive command). Here's what I came up with, then: <pre class="src"> (defun mha:select-current-word ()

"Select the current word." (interactive) (beginning-of-thing 'symbol) (push-mark (point) nil t) (end-of-thing 'symbol) (exchange-point-and-mark))

(global-set-key (kbd "C-S-M-SPC") 'mha:select-current-word) </pre>

I picked C-M-S-SPC not because it's the easiest way to invoke the new command, but because to me it's a quite natural extension to the C-M-SPC that I use so often. Again, each time you want to select a identifier in some code of yours, you'd most certainly be better off using C-M-SPC. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 08 Feb 2013 17:15:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/04-Emacs-mark-word.html</guid> </item> <item> <title>Live Upgrading PGQ</title> <link>http://tapoueh.org/blog/2013/02/08-PGQ-Live-Upgrade.html</link> <description><![CDATA[Some <a href="http://skytools.projects.pgfoundry.org/skytools-3.0/doc/">skytools</a> related new today, it's been a while. For those who where at my <a href="http://tapoueh.org/blog/2013/02/04-Another-great-FOSDEM.html">FOSDEM's talk</a> about <a href="https://fosdem.org/2013/schedule/event/postgresql_implementing_high_availability/">Implementing High Availability</a> you might have heard that I really like working with <a href="http://wiki.postgresql.org/wiki/Skytools#PgQ">PGQ</a>. A new version has been released a while ago, and the most recent verion is now 3.1.3, as announced in the <a href="http://www.postgresql.org/message-id/CACMqXCLD2je5VFqUCzjwC2s5QQVYLe6-4awJaRvqLSBEVw8_MQ@mail.gmail.com">Skytools 3.1.3</a> email.

<center> <img src="../../../images/software-upgrade.320.png" alt=""> </center> <center> Upgrade time! </center> <h3>Skytools 3.1.3 enters debian</h3> First news is that Skytools 3.1.3 has been entering <a href="http://packages.debian.org/search?keywords=skytools3">debian</a> today (I hope that by the time you reach that URL, it's been updated to show information according to the news here, but I might be early). As there's current a debian freeze to release wheezy (and you can help <a href="http://www.debian.org/News/2012/20121110">squash some bugs</a>), this version is only getting uploaded to experimental for now. Thanks to the tireless work of <a href="http://www.df7cb.de/blog/2012/apt.postgresql.org.html">Christoph Berg</a> though, this version is already available from <a href="https://wiki.postgresql.org/wiki/Apt">apt.postgresql.org</a>. <h3>Upgrading to PGQ 3</h3> The other news is that I've been testing live upgrade scenario where we want to upgrade from PGQ to PGQ3, and it works pretty well, and it's quite simple to achieve too. Here's how. So the first thing is to shut down the current ticker process. Then we install the new packages, assuming that you did follow the step in the wiki pointed above, please go read <a href="https://wiki.postgresql.org/wiki/Apt">apt.postgresql.org</a> again now if needs be. <pre class="src"> pgqadm.py ticker.ini -s sudo apt-get install postgresql-9.1-pgq3 skytools3-ticker skytools3 </pre> The ticker is not running anymore, we have the right version of the software installed. Next step is to upgrade the database parts of PGQ: <pre class="src"> psql -f /usr/share/skytools3/pgq.upgrade_2.1_to_3.0.sql ... psql -1 -f /usr/share/postgresql/9.1/contrib/pgq.upgrade.sql ... </pre> Of course replace those ... with options such as your actual connection string. I tend to always add -vON_ERROR_STOP=1 to all these commands, so that I don't depend on having the right .psqlrc on the particular server I'm connected to. Also remember that if you want to do that for more than one database, you need to actually run that pair of commands for each of them. Now it's time to restart the new ticker. The main changes from the previous one is that it is now a C program called pgqd that knows how to tick for any number of databases, so that you only have to have one instance around per cluster now. <pre class="src"> sudo /etc/init.d/skytools3 start tail -f /var/log/skytools/pgqd.log </pre> Those two commands are taking for granted that you did prepare the pgqd setup the debian and skytools way, by adding your config in /etc/skytools3/pgqd.ini and editing /etc/skytools.ini accordingly, so that it's automatically taken into account at machine boot. Note that I did actually exercised the procedure above while running a <a href="http://www.postgresql.org/docs/9.2/static/pgbench.html">pgbench</a> test replicated with londiste. Of course the replication has been lagging a little while no ticker was running, and then it catched-up as fast as it could, in that case: <pre class="src"> INFO {count: 245673, ignored: 0, duration: 422.104366064} </pre> <h3>Happy Hacking!</h3> So if you have any batch processing needs, remember to consider what PGQ has to offer. And yes if you're running some cron job to compute things out of the database for you, you are doing some batch processing. <center> <img src="../../../images/hayseed.jpg" alt=""> </center> <center> Yes, I did search for Transactional Batch Processing </center> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 08 Feb 2013 15:52:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/08-PGQ-Live-Upgrade.html</guid> </item> <item> <title>Another Great FOSDEM</title> <link>http://tapoueh.org/blog/2013/02/04-Another-great-FOSDEM.html</link> <description><![CDATA[This year's FOSDEM has been a great edition, in particular the <a href="http://fosdem2013.pgconf.eu/">FOSDEM PGDAY 2013</a> was a great way to begin a 3 days marathon of talking about PostgreSQL with people not only from our community but also from plenty other Open Source communities too: users!

<center> <a class="image-link" href="https://fosdem.org/2013/"> <img src="../../../images/fosdem-logo.png"></a><a class="image-link" href="http://fosdem2013.pgconf.eu/"> <img src="../../../images/postgresql-elephant.small.png"></a> </center> <center> PostgreSQL at FOSDEM made for a great event </center> Having had the opportunity to meet more people from those other development communities, I really think we should go and reach for them in their own conferences. About any PostgreSQL community member I've been talking about with about that idea seemed to agree and generally already was thinking the same thing. And most are already doing it, in fact... I had the pleasure to run two conferences there, both in the <a href="https://fosdem.org/2013/schedule/track/postgresql/">PostgreSQL devroom</a>. <h3>Event Triggers</h3> I'm currently in the middle of implementing Event Triggers for PostgreSQL and I have been for about the last 2 years. It's a quite complex feature to get right and so the patch itself is complex and large, which means the reviewing process is complex and takes time. That also means that some parts of the design have already been redone completely at least 3 times, and that what got commited to the PostgreSQL code is nothing like what the design we decided should go in looks like. That's just a fact of life, maybe, but that makes for a very long development process. We're now getting to the end of it though, and this talk is showing both where we want to go with Event Triggers, where we are now and what remains to be done for 9.3 if we want the feature to be any useful. If you're interested into that development, have a look at the slide deck and possibly ask me some questions about what's not clear on the <a href="http://www.postgresql.org/list/pgsql-hackers/">pgsql-hackers</a> mailing list (preferably). <center> <a class="image-link" href="../../../images/confs/Fosdem2013_Event_Triggers.pdf"> <img src="../../../images/confs/Fosdem2013_Event_Triggers.png"></a> </center> <center> Event Triggers, The Real Mess™ </center> The other way to get summarized and clear information about Event Triggers is the wiki page by the same name: <a href="http://wiki.postgresql.org/wiki/Event_Triggers">Event Triggers</a>. You will see that while a lot has been done (internal refactoring, adding new infrastructure and SQL level commands, and the minimum PLpgSQL support); a lot remains to be done where the code has already been submitted several times, following several designs directions given by careful review on hackers, and still we have some choices to make. <h3>Implementing High-Availability</h3> This talk is showing several ways to implement High Availability with PostgreSQL. The fact is that that term is overloaded already, and usually covers two very different things which are Service Availability and Data Availability. In the talk, we're showing up several techniques that you can use to address different set of compromises in between scaling, load balancing, data availability and durability, and service availability. The first two points could seem unrelated to the main topic, but scaling often is a simple enough way to achieve service availability... until you need to think about sharding, that is. <center> <a class="image-link" href="../../../images/confs/Fosdem2013_High_Availability.pdf"> <img src="../../../images/confs/Fosdem2013_High_Availability.640.png"></a> </center> <center> Implementing High Availability of Services and Data with PostgreSQL </center> So the talk is all about making compromises in between them and getting to an architecture able to implement the choosen compromises. While the talk has been pretty well received, it was delivered in a 50 mins slot where we usually take a whole day or three when addressing that problems at a customer's site. Some parts of how to get to the right architecture for the compromises that are important for you can't be fully covered in that time slot, while still being able to actually present the techniques that we're using. I think it might be useful to extract a single use-case or two from that talk then have a full 50 mins version reduced to a single or a couple of very clear compromises and how to achieve them in details, rather than trying to present a full range of techniques and how to use them in different scenarios. <h3>FOSDEM</h3> After having been talking with many people, it appears that for next year's edition I should be proposing a more general talk that aims at helping developpers in other communities (python, ruby, etc) discover what's in for them in PostgreSQL. This database is full of advanced features that are really easy to use, and the only problem when preparing such a talk is choosing the right subset... If you're running a local developper user group and are interested into learning some more about how PostgreSQL can help you in a daily basis, please do get in touch with me and let's schedule a presentation together! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 04 Feb 2013 09:55:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/04-Another-great-FOSDEM.html</guid> </item> <item> <title>Another Great FOSDEM</title> <link>http://tapoueh.org/blog/2013/02/04-Another-great-FOSDEM.html</link> <description><![CDATA[This year's FOSDEM has been a great edition, in particular the <a href="http://fosdem2013.pgconf.eu/">FOSDEM PGDAY 2013</a> was a great way to begin a 3 days marathon of talking about PostgreSQL with people not only from our community but also from plenty other Open Source communities too: users!

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 04 Feb 2013 09:55:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/02/04-Another-great-FOSDEM.html</guid> </item> <item> <title>A Sunday at FOSDEM</title> <link>http://tapoueh.org/blog/2013/01/30-A-Sunday-at-FOSDEM.html</link> <description><![CDATA[The previous article <a href="29-FOSDEM-2013.html">FOSDEM 2013</a> said to be careful with the <a href="https://fosdem.org/2013/schedule/track/postgresql/">PostgreSQL devroom schedule</a> because one of my talks there might get swapped with a slot on the <a href="http://fosdem2013.pgconf.eu/">FOSDEM PGDay 2013</a> which happens this Friday and has been sold out anyway.

Turns out it's not true, because we still depend on past century technologies somehow. Not everybody will be looking at the schedule on the web using a connected mobile device (you know, you've heard of them, those tracking and surveillance devices, if you want to believe <a href="http://stallman.org/rms-lifestyle.html">Stallman</a>), and as the schedule gets printed on little paper sheets, it's unfortunately too late to change it now. <center> <a class="image-link" href="https://fosdem.org/2013/"> <img src="../../../images/fosdem.png"></a> </center> <center> Those flyers are already printed on paper sheets, the schedule too </center> So it happens that I'll be speaking twice on Sunday and not at all on Friday. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 30 Jan 2013 10:50:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/01/30-A-Sunday-at-FOSDEM.html</guid> </item> <item> <title>FOSDEM 2013</title> <link>http://tapoueh.org/blog/2013/01/29-FOSDEM-2013.html</link> <description><![CDATA[This year again I'm going to <a href="https://fosdem.org/2013/">FOSDEM</a>, and to the extra special <a href="http://fosdem2013.pgconf.eu/">PostgreSQL FOSDEM day</a>. It will be the first time that I'm going to be at the event for the full week-end rather than just commuting in for the day.

<center> <a class="image-link" href="https://fosdem.org/2013/"> <img src="../../../images/fosdem.png"></a> </center> <center> I'm Going to the FOSDEM, hope to see you there! </center> And I'm presenting two talks over there that are both currently scheduled on the Sunday in the <a href="https://fosdem.org/2013/schedule/track/postgresql/">PostgreSQL devroom</a>. We're talking about changing that though, so that one of those will in fact happen this Friday at the <a href="http://www.postgresql.eu/events/schedule/fosdem2013/">FOSDEM PGDay 2013</a>, which has a different schedule, so consider watching for that. One of those two talks is about <a href="https://fosdem.org/2013/schedule/event/postgresql_implementing_high_availability/">Implementing High Availability</a> (yes, with PostgreSQL). It's been quite well received in the places I had to chance to make it before (namely PGDay France and PG Conf Europe), and it's going to be a stripped down version of it so that it fits well in the 45 mins slot we have here. The other talk is going to be about <a href="https://fosdem.org/2013/schedule/event/postgresql_event_triggers/">Event Triggers</a>, a feature new in PostgreSQL 9.3 (due in september 2013, crossing fingers) and while the goal of that talk is to introduce what the feature is all about and a bunch of use cases that you can address by using it, it will certainly offer a peek into the PostgreSQL development cycle and community processes. <center> <img src="../../../images/belgium-beers.jpg" alt=""> </center> <center> See you in Brussels! </center> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 29 Jan 2013 10:11:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/01/29-FOSDEM-2013.html</guid> </item> <item> <title>pgloader: what's next?</title> <link>http://tapoueh.org/blog/2013/01/28-pgloader-future.html</link> <description><![CDATA[<a href="../../../pgsql/pgloader.html">pgloader</a> is a tool to help loading data into <a href="http://www.postgresql.org/">PostgreSQL</a>, adding some error management to the <a href="http://www.postgresql.org/docs/9.2/interactive/sql-copy.html">COPY</a> command. COPY is the fast way of loading data into PostgreSQL and is transaction safe. That means that if a single error appears within your bulk of data, you will have loaded none of it. pgloader will submit the data again in smaller chunks until it's able to isolate the bad from the good, and then the good is loaded in.

<center> <img src="../../../images/PDL_Adapter-250.png" alt=""> </center> <center> Not quite this kind of data loader </center> In a recent migration project where we freed data from MySQL into PostgreSQL, we used pgloader again. But the loading time was not fast enough for the service downtime window that we had here. Indeed <a href="http://www.python.org/">Python</a> is not known for being the fastest solution around. It's easy to use and to ship to production, but sometimes you not only want to be able to be efficient when writing code, you also need the code to actually run fast too. <h3>Faster data loading</h3> So I began writing a little dedicated tool for that migration in <a href="http://cliki.net/">Common Lisp</a> which is growing on me as my personal answer to the burning question: python 2 or python 3? I find Common Lisp to offer an even more dynamic programming environment, an easier language to use, and the result often has performances characteristics way beyond what I can get with python. Between <a href="http://tapoueh.org/blog/2012/07/10-solving-sudoku.html">5 times faster</a> and <a href="http://tapoueh.org/blog/2012/08/20-performance-the-easiest-way.html">121 times faster</a> in some quite stupid benchmark. Here, with real data, my one shot attempt has been running more than twice as fast as the python version, after about a day of programming. <center> <img src="../../../images/lisp-python.png" alt=""> </center> <center> See what's happening now? </center> The other thing here is that I've tempted to get pgloader work in parallel, but at the time I didn't know about the <a href="http://docs.python.org/3/c-api/init.html#threads">Global Interpreter Lock</a> that they didn't find how to remove in Python 3 still, by the way. So my threading attempts at making pgloader work in parallel are pretty useless. Whereas in Common Lisp I can just use the <a href="http://lparallel.org/">lparallel</a> lib, which exposes threading facilities and some queueing facilities as a mean to communicate data in between workers, and have my code easily work in parallel for real. <h3>Compatibility</h3> The only drawback that I can see here is that if you've been writing your own reformating modules in python for pgloader (yes you can <a href="http://tapoueh.org/pgsql/pgloader.html#sec21">implement your own reformating module for pgloader</a>), then you would have to port it to Common Lisp. Shout me an email if that's your case. <h3>Next version</h3> So, I think we're going to have a pgloader 3 someday, that will be way faster than the current one, and bundle some more features: real parallel behavior, ability to fetch non local data (connecting to MySQL directly, or HTTP, S3, etc); and I'm thinking about offering a COPY like syntax to drive the loading too, while at it. Also, the ability to discover the set of data to load all by itself when you want to load a whole database: think of it as a special Migration mode of operations. Some feature requests can't be solved easily when keeping the old .INI syntax cruft, so it's high time to implement some kind of a real command language. I have several ideas about those, in between the COPY syntax and the SQL*Loader configuration format, which is both clunky and quite powerful, too. After a beginning in TCL and a complete rewrite in python in 2005, it looks like 2013 is going to be the year of pgloader 3, in Common Lisp! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 28 Jan 2013 10:48:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/01/28-pgloader-future.html</guid> </item> <item> <title>Automated Setup for pgloader</title> <link>http://tapoueh.org/blog/2013/01/17-pgloader-auto-setup.html</link> <description><![CDATA[Another day, another migration from MySQL to <a href="http://www.postgresql.org/">PostgreSQL</a>... or at least that's how it feels sometimes. This time again I've been using some quite old scripts to help me do the migration.

<center> <img src="../../../images/dauphin-logo.jpg" alt=""> </center> <center> That's how I feel for MySQL users </center> <h3>Migrating the schema</h3> For the schema parts, I've been using <a href="http://pgfoundry.org/projects/mysql2pgsql/">mysql2pgsql</a> with success for many years. This tool is not complete and will do only about 80% of the work. As I think that the schema should always be validated manually when doing a migration anyway, I happen to think that it's good news. <h3>Getting the data out</h3> Then for the data parts I keep on using <a href="../../../pgsql/pgloader.html">pgloader</a>. The data is never quite right, and the ability to filter out what you can't readily import in a reject file proves itself a a must have here. The problems you have in the exported MySQL data are quite serious: <center> <img src="../../../images/data-unlocked.320.png" alt=""> </center> <center> Can I have my data please? </center> First, date formating is not compatible with what PostgreSQL expects, sometimes using 20130117143218 instead of what we expect:

2013-01-17
14:32:18

, and of course even when the format is right (that seems to depend on the MySQL server's version), you still have to transform the

0000-00-00
00:00:00

into NULL. <blockquote> Before thinking about the usage of that particular date rather than using NULL when you don't have the information, you might want to remember that there's no <a href="http://en.wikipedia.org/wiki/0_(year)">year zero</a> in the calendar, it's year 1 BC and then year 1. </blockquote> Then, text encoding is often mixed up, even when the MySQL databases are said to be in latin1 or unicode, you somehow always end up finding texts in win1252 or some other code page in there. And of course, MySQL provides no tool to export the data to CSV, so you have to come up with your own. The SELECT INTO OUTFILE command on the server produces non conforming CSV (\n can appear in non-escaped field contents), and while the mysql client manual page details that it outputs CSV when stdout is not a terminal, it won't even try to quote fields or escape \t when they appear in the data. So, we use the <a href="https://github.com/slardiere/mysqltocsv">mysqltocsv</a> little script to export the data, and then use that data to feed <a href="../../../pgsql/pgloader.html">pgloader</a>. <h3>Loading the data in</h3> Now, we have to write down a configuration file for pgloader to know what to load and where to find the data. What about generating the file from the database schema instead, using the query in <a href="generate-pgloader-config.sql">generate-pgloader-config.sql</a>: <pre class="src"> with reformat as (

select relname, attnum, attname, typname, case typname when 'timestamptz'

<span style="color: #7f007f;">then</span> attname	<span style="color: #bc8f8f;">':mynull:timestamp'</span>

when 'date'

<span style="color: #7f007f;">then</span> attname	<span style="color: #bc8f8f;">':mynull:date'</span>

end as reformat from pg_class c join pg_namespace n on n.oid = c.relnamespace left join pg_attribute a on c.oid = a.attrelid join pg_type t on t.oid = a.atttypid where c.relkind = 'r' and attnum > 0 and n.nspname = 'public' ), config_reformat as ( select relname,

<span style="color: #bc8f8f;">'['</span>\|\|relname\|\|<span style="color: #bc8f8f;">']'</span>	E<span style="color: #bc8f8f;">'\n'</span>
<span style="color: #bc8f8f;">'table = '</span>	relname	E<span style="color: #bc8f8f;">' \n'</span>
<span style="color: #bc8f8f;">'filename = /path/to/csv/'</span>	relname	E<span style="color: #bc8f8f;">'.csv\n'</span>
<span style="color: #bc8f8f;">'format = csv'</span>	E<span style="color: #bc8f8f;">'\n'</span>
<span style="color: #bc8f8f;">'field_sep = \t'</span>	E<span style="color: #bc8f8f;">'\n'</span>
<span style="color: #bc8f8f;">'columns = '</span>	E<span style="color: #bc8f8f;">' \n'</span>
<span style="color: #bc8f8f;">'reformat = '</span>	array_to_string(<span style="color: #da70d6;">array_agg</span>(reformat), <span style="color: #bc8f8f;">', '</span>)
	E<span style="color: #bc8f8f;">'\n'</span> <span style="color: #7f007f;">as</span> config

from reformat where reformat is not null group by relname ), noreformat as ( select relname, bool_and(reformat is null) as noreformating from reformat group by relname ), config_noreformat as ( select relname,

<span style="color: #bc8f8f;">'['</span>\|\|relname\|\|<span style="color: #bc8f8f;">']'</span>	E<span style="color: #bc8f8f;">'\n'</span>
<span style="color: #bc8f8f;">'table = '</span>	relname	E<span style="color: #bc8f8f;">' \n'</span>
<span style="color: #bc8f8f;">'filename = /path/to/csv/'</span>	relname	E<span style="color: #bc8f8f;">'.csv\n'</span>
<span style="color: #bc8f8f;">'format = csv'</span>	E<span style="color: #bc8f8f;">'\n'</span>
<span style="color: #bc8f8f;">'field_sep = \t'</span>	E<span style="color: #bc8f8f;">'\n'</span>
<span style="color: #bc8f8f;">'columns = '</span>	E<span style="color: #bc8f8f;">' \n'</span>
	E<span style="color: #bc8f8f;">'\n'</span> <span style="color: #7f007f;">as</span> config

from reformat join noreformat using (relname) where noreformating group by relname ), allconfs as ( select relname, config from config_reformat union all select relname, config from config_noreformat ) select config from allconfs where relname not in ('tables', 'wedont', 'wantto', 'load') order by relname; </pre>

To work with the setup generated, you will have to prepend a global section for pgloader and to include a reformating module in python, that I named <a href="mynull.py">mynull.py</a>: <pre class="src"> # Author: Dimitri Fontaine <<a href="mailto:dimitri@2ndQuadrant.fr">dimitri@2ndQuadrant.fr</a>> # # pgloader mysql reformating module def timestamp(reject, input):

""" Reformat str as a PostgreSQL timestamp

MySQL timestamps are ok this time: 2012-12-18 23:38:12 But may contain the infamous all-zero date, where we want NULL. """ if input == '0000-00-00 00:00:00': return None

return input

def date(reject, input):

""" date columns can also have '0000-00-00'""" if input == '0000-00-00': return None

return input </pre>

Now you can launch pgloader and profit! <h3>Conclusion</h3> There are plenty of tools to assist you migrating away from MySQL and other databases. When you make that decision, you're not alone, and it's easy enough to find people to come and help you. While MySQL is Open Source and is not a lock in from a licencing perspective, I still find it hard to swallow that there's no provided tools for getting data out in a sane format, and that so many little inconsistencies exist in the product with respect to data handling (try to have a NOT NULL column, then enjoy the default empty strings that have been put in there). So at this point, yes, I consider that moving to <a href="http://www.postgresql.org/">PostgreSQL</a> is a way to free your data: <center> <img src="../../../images/free-our-open-data.jpg" alt=""> </center> <center> Free your data! </center> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 17 Jan 2013 14:32:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/01/17-pgloader-auto-setup.html</guid> </item> <item> <title>Lost in scope</title> <link>http://tapoueh.org/blog/2013/01/09-Lost-in-scope.html</link> <description><![CDATA[Thanks to <a href="https://twitter.com/mickael/status/288795520179240962">Mickael</a> on twitter I got to read an article about loosing scope with some common programming languages. As the blog article <a href="https://my.smeuh.org/al/blog/lost-in-scope">Lost in scope</a> references functional programming languages and plays with both Javascript and Erlang, I though I had to try it out with Common Lisp too.

<center> <img src="../../../images/lambda.png" alt=""> </center> <center> Let's have fun with lambda! </center> So, here we go with a simple Common Lisp attempt. The Lost in scope article begins with defining a very simple function returning a boolean value, only true when it's not monday. <h3>Monday is special</h3> Keep in mind that the following example has been choosen to be simple yet offer a case of lexical binding shadowing. It looks convoluted. Focus on the day binding. <pre class="src"> (defparameter *days*

'(monday tuesday wednesday thursday friday saturday sunday) "List of days in the week")

(defun any-day-but-monday? (day)

"Returns a generalized boolean, true unless DAY is 'monday" (member day (remove-if (lambda (day) (eq day 'monday)) days))) </pre>

So as you can see, in Common Lisp we just get away with a list of symbols rather than a string that we split to have a list of strings, or an array of strings, as in the examples with python and ruby. Now, the generalized boolean is either nil to mean false, or anything else to mean true, and in that example the return value of <a href="http://www.lispworks.com/documentation/HyperSpec/Body/a_member.htm">member</a> is a sub-list that begins where the member was found: <pre class="src"> CL-USER> (any-day-but-monday? 'monday) NIL

CL-USER> (any-day-but-monday? 'tuesday) (TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY) </pre>

Oh, and as we work with Common Lisp, we're having a real <a href="http://www.gigamonkeys.com/book/lather-rinse-repeat-a-tour-of-the-repl.html">REPL</a> where to play directly with our code, no need to add interactive stanzas in the main program text file just to be able to play with it. In <a href="http://common-lisp.net/project/slime/">Emacs Slime</a> we just use C-M-x on a form to have it available in the REPL, or C-c C-l to load the whole file we're working on. So, we see that Common Lisp scoping rules are silently doing the right thing here. Within the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_rm_rm.htm">remove-if</a> call we define a lambda function taking a single parameter called day. It so happens that this parameter is shadowing the any-day-but-monday? function parameter, and that shadowing only happens in the lexical scope of the lambda we are creating. For a detailed discussion about that concept, I would refer you to the <a href="http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html">Scope and Extent</a> chapter of Common Lisp the Language, 2nd Edition. In Common Lisp we have both lexical scope and dynamic extents, and a variable defined with defparameter or defvar or that you otherwise <a href="http://www.lispworks.com/documentation/HyperSpec/Body/s_declar.htm">declare</a> special will have a dynamic extent. Hence this section title. <h3>Closures</h3> Now, the <a href="https://my.smeuh.org/al/blog/lost-in-scope">lost in scope</a> article tries some more at finding a solution around the scoping rules of the python and ruby languages, where the developer can not easily instruct the language about the scoping rules he wants to be using in a case by case way, as far as I can see. First, let's reproduce the problem by using a single variable that we bind in all the closures. Those are called callbacks in the original article, so I've kept using that name here. <center> <img src="../../../images/callback.jpg" alt=""> </center> <pre class="src"> (defparameter *callbacks-all-sunday*

(loop for day in days collect (lambda () day)) "loop binds DAY only once") </pre>

In that example, there's only a single variable day that we reuse throughout the loop construct, so that when the loop ends, we have a list of closures all refering to the same variable, and this variable, by the end of the loop, has sunday as its value. <pre class="src"> CL-USER> (mapcar #'funcall callbacks-all-sunday) (SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY SUNDAY) </pre> <h3>Closures, take 2</h3> Now, the way to have what we want here, that is a list of closures each having its own variable. <pre class="src">

(defparameter *callbacks*

(mapcar (lambda (day) ;; for each day, produce a separate closure ;; around its own lexical variable day (lambda () day)) days) "A list of callbacks to return the current day...") </pre>

And there we go: <pre class="src"> CL-USER> (mapcar #'funcall callbacks) (MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY SUNDAY) </pre> <h3>Conclusion</h3> Scoping rules are very important in any programming language, functional or not, and must be well understood by programmers. I find that once again, that topic has received a very deep thinking in Common Lisp, and the language is giving all the options to its developers. <center> <img src="../../../images/scope.png" alt=""> </center> <center> What are your language of choice scoping rules? </center> I want to stress that in Common Lisp the scope rules are very clearly defined in the standard documentation of the language. For instance, defun and let both introduce a lexical binding, defvar and defparameter introduce a dynamic variable. Also, as a user of the language you have the ability to declare any variable as being special in order to introduce yourself a dynamic variable. In C you can declare some variables as being static, which is something else and frown with a very different set of problems. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 09 Jan 2013 11:07:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/01/09-Lost-in-scope.html</guid> </item> <item> <title>Extensions Templates</title> <link>http://tapoueh.org/blog/2013/01/08-Extensions-Templates.html</link> <description><![CDATA[In a recent article titled <a href="../../2012/12/13-Inline-Extensions.html">Inline Extensions</a> we detailed the problem of how to distribute an extension's package to a remote server without having access to its file system at all. The solution to that problem is non trivial, let's say. But thanks to the awesome <a href="http://www.postgresql.org/community/">PostgreSQL Community</a> we finaly have some practical ideas on how to address the problem as discussed on <a href="http://archives.postgresql.org/pgsql-hackers/">pgsql-hackers</a>, our development mailing list.

<center> <img src="../../../images/community.jpg" alt=""> PostgreSQL is first an Awesome Community </center> The solution we talked about is to use templates, and so I've been working on a patch to bring templates for extensions to PostgreSQL. As we're talking about 3 new system catalogs, that's a big patch in term of lines of code. In term of features though, it's quite an easy one. Here's how it goes. Let's say you want to prepare the system to be able to CREATE EXTENSION pair; without having to install it as an OS package for which you would need to get root access on the server where your PostgreSQL instance is running, which is not always easy, and sometimes not a good idea. <h3>Installing an extension template</h3> With the <a href="http://www.postgresql.org/message-id/m2wqvoha0p.fsf%402ndQuadrant.fr">template patch</a> I just sent on the lists, what you can do is prepare a template with your extension's script and properties, then use it to install the extensions. <pre class="src"> create template

for extension pair default version '1.0' with (nosuperuser, norelocatable, schema public) as $$ CREATE TYPE pair AS ( k text, v text );

CREATE OR REPLACE FUNCTION pair(anyelement, text) RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

CREATE OR REPLACE FUNCTION pair(text, anyelement) RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

CREATE OR REPLACE FUNCTION pair(anyelement, anyelement) RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

CREATE OR REPLACE FUNCTION pair(text, text) RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair;'; $$; </pre>

<h3>Installing an extension from a template</h3> With the template installed in the catalogs, now you can go and install your extension: <pre class="src"> foo> create extension pair; CREATE EXTENSION

foo> \dx pair

List of installed extensions

Name

Version

Schema

Description

——+———+———+————-

pair

1.0

public

(1 row)

foo> \dx+ pair

Objects in extension "pair" Object Description ————————————— function pair(anyelement,anyelement) function pair(anyelement,text) function pair(text,anyelement) function pair(text,text) type pair (5 rows) </pre>

The extension installation is now happening from the catalog templates rather than the file system, which means you didn't need to be root on the system where the server is running. Also note that this example above did happen when connected as the database owner, a user who is not the superuser. Requiring less privileges is always good news, right? <h3>Managing upgrade scripts and extension update</h3> Now that the extension is installed, you might want to update it with some new awesome features. Let's have a look at that. <center> <img src="../../../images/extension-update.png" alt=""> </center> <center> Upload your Extension Update Scripts </center> Rather than make a new version of the extension package with the new files in there, then asking the operations team to make the new package available on the internal repositories then install them on the servers, you could now prepare and QA the new setup that way: <pre class="src"> create template for extension pair from '1.0' to '1.1' as $$

CREATE OPERATOR ~> (LEFTARG = text, RIGHTARG = anyelement, PROCEDURE = pair);

CREATE OPERATOR ~> (LEFTARG = anyelement, RIGHTARG = text, PROCEDURE = pair);

CREATE OPERATOR ~> (LEFTARG = anyelement, RIGHTARG = anyelement, PROCEDURE = pair);

CREATE OPERATOR ~> (LEFTARG = text, RIGHTARG = text, PROCEDURE = pair); $$;

create template

for extension pair from '1.1' to '1.2' as $$ comment on extension pair is 'Simple Key Value Text Type'; $$; </pre>

Of course it's not the most realistic example when you look at the content. In particular the 1.2 version that only adds a comment to the extension. I needed another version to test the automatic upgrade path with more than one step though, so here we go. <pre class="src"> foo> alter extension pair update to '1.2'; ALTER EXTENSION

foo> \dx pair

List of installed extensions

Name

Version

Schema

Description

——+———+———+—————————-

pair

1.2

public

Simple Key Value Text Type

(1 row)

foo> \dx+ pair

Objects in extension "pair" Object Description ————————————— function pair(anyelement,anyelement) function pair(anyelement,text) function pair(text,anyelement) function pair(text,text) operator ~>(anyelement,anyelement) operator ~>(anyelement,text) operator ~>(text,anyelement) operator ~>(text,text) type pair (9 rows) </pre>

We did it! <h3>Internals</h3> Let's have a look at those new catalogs: <center> <img src="../../../images/octopus-anatomy.jpg" alt=""> </center> <center> Oh, that's not quite the internals I expected... </center> Here we go now: <pre class="src"> foo> select * from pg_extension_control; select * from pg_extension_control; -[ RECORD 1 ]—+——-

ctlname	pair
ctlowner	32926
ctldefault	t
ctlrelocatable	f
ctlsuperuser	f
ctlnamespace	public
ctlversion	1.0
ctlrequires

foo> select * from pg_extension_template; select * from pg_extension_template; -[ RECORD 1 ]——————————————————————

tplname	pair
tplowner	32926
tplversion	1.0
tplscript
	CREATE TYPE pair AS ( k text, v text );

	CREATE OR REPLACE FUNCTION pair(anyelement, text)
	RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

	CREATE OR REPLACE FUNCTION pair(text, anyelement)
	RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

	CREATE OR REPLACE FUNCTION pair(anyelement, anyelement)
	RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair';

	CREATE OR REPLACE FUNCTION pair(text, text)
	RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::pair;';

foo> select * from pg_extension_uptmpl; select * from pg_extension_uptmpl; -[ RECORD 1 ]————————————————————-

uptname	pair
uptowner	32926
uptfrom	1.0
uptto	1.1
uptscript
	CREATE OPERATOR ~> (LEFTARG = text,
	RIGHTARG = anyelement,
	PROCEDURE = pair);

	CREATE OPERATOR ~> (LEFTARG = anyelement,
	RIGHTARG = text,
	PROCEDURE = pair);

	CREATE OPERATOR ~> (LEFTARG = anyelement,
	RIGHTARG = anyelement,
	PROCEDURE = pair);

	CREATE OPERATOR ~> (LEFTARG = text,
	RIGHTARG = text,
	PROCEDURE = pair);

-[ RECORD 2 ]————————————————————-

uptname	pair
uptowner	32926
uptfrom	1.1
uptto	1.2
uptscript
	comment on extension pair is 'Simple Key Value Text Type';

</pre> As you can see there's nothing too complex here, it's quite straightforward. We need to separate away the creating templates from the updating templates because we need unique keys and we can't have that on NULL columns. <pre class="src"> foo> \d pg_extension_template \d pg_extension_template Table "pg_catalog.pg_extension_template"

Column

Type

Modifiers

+——+————

tplname	name	not null
tplowner	oid	not null
tplversion	text
tplscript	text

Indexes:

"pg_extension_template_name_version_index" UNIQUE, btree (tplname, tplversion) "pg_extension_template_oid_index" UNIQUE, btree (oid)

foo> \d pg_extension_uptmpl \d pg_extension_uptmpl Table "pg_catalog.pg_extension_uptmpl"

Column

Type

Modifiers

+——+————

uptname	name	not null
uptowner	oid	not null
uptfrom	text
uptto	text
uptscript	text

Indexes:

"pg_extension_uptmpl_name_from_to_index" UNIQUE, btree (uptname, uptfrom, uptto) "pg_extension_uptmpl_oid_index" UNIQUE, btree (oid) </pre>

<h3>Next steps</h3> Now that we have the basics in place, the patch is far from finished still. It needs pg_dump and psql support, support for the function pg_available_extension_versions(), implementing some

ALTER TEMPLATE FOR
EXTENSION

commands for which I only sketched the syntax in the grammar, and some more infrastructure to be able to have ALTER OWNER and ALTER RENAME commands. <center> <img src="../../../images/patch-brewing.jpg" alt=""> </center> <center> Warning: patch brewing here! Syntax and other key elements will change. </center> All that is pretty technical though, the real thing that patch needs is some quality review and maybe some adjustments. I would be surprised if it didn't need adjustments, really. Because the way the community works, we always need some. That's why the PostgreSQL product is so good! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 08 Jan 2013 17:53:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2013/01/08-Extensions-Templates.html</guid> </item> <item> <title>Inline Extensions</title> <link>http://tapoueh.org/blog/2012/12/13-Inline-Extensions.html</link> <description><![CDATA[We've been having the CREATE EXTENSION feature in <a href="http://www.postgresql.org/">PostgreSQL</a> for a couple of releases now, so let's talk about how to go from here. The first goal of the extension facility has been to allow for a clean dump and restore process of <a href="http://www.postgresql.org/docs/9.2/static/contrib.html">contrib</a> modules. As such it's been tailored to the needs of deploying files on the file system because there's no escaping from that when you have to ship binary and executable files, those infamous .so, .dll or .dylib things.

<center> <img src="../../../images/dylibbundler.png" alt=""> </center> Now that we have the Extension facility though, what we see is a growing number of users taking advantage of it for the purpose of managing in house procedural code and related objects. This code can be a bunch of <a href="http://www.postgresql.org/docs/9.2/static/plpgsql.html">PLpgSQL</a> or <a href="http://www.postgresql.org/docs/9.2/static/plpython.html">plpython</a> functions and as such you normaly create them directly from any application connection to PostgreSQL. So the idea would be to allow creating Extensions fully from a SQL command, including the whole set of objects it contains. More than one approach are possible to reach that goal, each with downsides and advantages. We will see them later in that document. Before that though, let's first review what the extension mechanism has to offer to its users when there's no contrib like module to manage. <h3>A use case for next generation extensions</h3> The only design goal of the 9.1 PostgreSQL Extension feature has been to support a proper dump & restore user experience when using contrib modules such as hstore or ltree. Building up on that, what do Extensions have to offer to non C developpers out there? In other words, what CREATE EXTENSION brings on the table that a bunch of loose objects does not? What problems can we now solve? <center> <img src="../../../images/multi_function_equipment.jpg" alt=""> </center> <center> A Multi Functions Equipment, All Bundled Together </center> A way to phrase it is to say that Extensions are user defined CASCADE support. Extensions brings extensibility to the pg_depend PostgreSQL internal dependency tracking system that CASCADE is built on. From that angle, Extensions are a way to manage dependencies of SQL objects in a way that allow you to manage them as a single entity. One of the existing problems this helps solving is the infamous lack of dependency tracking between function calls. Using Extensions when you deal with a set of functions acting as an API, you can at least protect that as a unit: <pre class="src"> STATEMENT: drop function public.populate_record(anyelement,hstore);

ERROR: cannot drop function populate_record(anyelement,hstore) because extension hstore requires it HINT: You can drop extension hstore instead. </pre>

And you also have a version number and tools integration to manage extensions, with psql \dx command and the equivalent feature in <a href="http://www.pgadmin.org/">pgAdmin</a>. Coming with your own version number management is not impossible, some do that already. Here it's integrated and the upgrade sequences are offered too (applying 1.1--1.2 then 1.2--1.3 automatically). Let's just say that it's very easy to understand the traction our users feel towards leveraging Extensions features in order to properly manage their set of stored procedures and SQL objects. <h3>The dump & restore experience</h3> The common problem of all those proposals is very central to the whole idea of Extensions as we know them. The goal of building them as been to fix the restoring experience when using extensions in a database, and we managed to do that properly for contrib likes extensions. <center> <img src="../../../images/fly.tn.png" alt=""> </center> <center> A fly in the ointment </center> When talking about Inline Extensions, the fly in the ointment is how to properly manage their pg_dump behavior. The principle we built for Extensions and that is almost unique to them is to omit them in the dump files. The only other objects that we filter out of the dump are the one installed at server initialisation times, when using <a href="http://www.postgresql.org/docs/9.2/static/app-initdb.html">initdb</a>, to be found in the pg_catalog and information_schema systems' schema. At restore time, the dump file contains the CREATE EXTENSION command so the PostgreSQL server will go fetch the control and script files on disk and process them, loading the database with the right set of SQL objects. Now we're talking about Extensions which we would maybe want to dump the objects of, so that at restore time we don't need to find them from unknown external resources: the fact that the extension is Inline means that the PostgreSQL server has no way to know where its content is coming from. The next proposals are trying to address that problem, with more or less success. So far none of them is entirely sastisfying to me, even if a clear temporary winner as emerged on the hackers mailing list, summarized in the <a href="http://archives.postgresql.org/message-id/m2fw3judug.fsf@2ndQuadrant.fr">in-catalog Extension Scripts and Control parameters (templates?)</a> thread. <h3>Inline Extension Proposals</h3> Now, on to some proposals to make the best out of our all time favorite PostgreSQL feature, the only one that makes no sense at all by itself... <h4>Starting from an empty extension</h4> We already have the facility to add existing loose objects to an extension, and that's exactly what we use when we create an extension for the first time when it used not to be an extension before, with the

CREATE EXTENSION
... FROM 'unpackaged';

command. The hstore--unpackaged--1.0.sql file contains statements such as: <pre class="src"> ALTER EXTENSION hstore ADD type hstore; ALTER EXTENSION hstore ADD function hstore_in(cstring); ALTER EXTENSION hstore ADD function hstore_out(hstore); ALTER EXTENSION hstore ADD function hstore_recv(internal); ALTER EXTENSION hstore ADD function hstore_send(hstore); </pre> Opening CREATE EXTENSION so that it allows you to create a really empty extension would then allow you to fill-in as you need, with as many commands as you want to add objects to it. The control file properties would need to find their way in that design, that sure can be taken care of. <center> <img src="../../../images/empty-extension.jpg" alt=""> </center> <center> Look me, an Empty Extension! </center> The main drawback here is that there's no separation anymore in between the extension author, the distribution means, the DBA and the database user. When you want to install a third party Extension using only SQL commands, you could do it with that scheme by using a big script full of one-liners commands. So that if you screw up your copy/pasting session (well you should maybe reconsider your choice of tooling at this point, but that's another topic), you will end up with a perfectly valid Extension that does not contain what you wanted. As the end user, you have no clue about that until the first time using the extension fails. <h4>CREATE EXTENSION AS</h4> The next idea is to embed the Extension script itself in the command, so as to to get a cleaner command API (in my opinion at least) and a better error message when the paste is wrong. Of course it your paste problem happens to just be loosing a line in the middle of the script there is not so much I can do for you... <pre class="src"> CREATE EXTENSION hstore

WITH parameter = value, ... AS $$ CREATE TYPE hstore;

CREATE FUNCTION hstore_in(cstring) RETURNS hstore

AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_out(hstore) RETURNS cstring AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_recv(internal) RETURNS hstore AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE FUNCTION hstore_send(hstore) RETURNS bytea AS 'MODULE_PATHNAME' LANGUAGE C STRICT IMMUTABLE;

CREATE TYPE hstore (

INTERNALLENGTH = -1, STORAGE = extended INPUT = hstore_in, OUTPUT = hstore_out, RECEIVE = hstore_recv, SEND = hstore_send); $$; </pre> <center> An edited version of hstore--1.1.sql for vertical space concerns </center>

I've actually proposed a patch to implement that, as you can see in the <a href="https://commitfest.postgresql.org/action/patch_view?id=981">pg_dump —extension-script</a> commit fest entry. As spoiled by the commit fest entry title, the main problem we have with Inline Extensions is their management in the seamless experience of dump & restore that we are so happy to have now. More about that later, though. <h4>Extension Templates</h4> Another idea is to continue working from control parameters and scripts to install and update extensions, but to have two different places where to find those. Either on the server's File System (when dealing with contribs and shared libraries, there's but a choice), or on the system catalogs. <center> <img src="../../../images/templates.png" alt=""> </center> <center> We Already Have TEXT SEARCH TEMPLATE After All </center> The idea would then be to have some new specific TEMPLATE SQL Object that would be used to import or upload your control file and create and update scripts in the database, using nothing else than a SQL connection. Then at CREATE EXTENSION time the system would be able to work either from the file system or the template catalogs. One obvious problem is how to deal with a unique namespace when we split the sources into the file system and the database, and when the file system is typically maintained by using apt-get or yum commands. Then again I would actually prefer that mechanism better than the other proposals if the idea was to load the file system control and scripts files as TEMPLATEs themselves and then only operate Extensions from Templates. But doing that would mean getting back to the situation where we still are not able to devise a good, simple and robust pg_dump policy for extensions and templates. <h3>Conclusion</h3> I hope to be finding the right solution to my long term plan in this release development cycle, but it looks like the right challenge to address now is to find the right compromise instead. Using the Templates idea already brings a lot on the table, if not the whole set of features I would like to see. <center> <img src="../../../images/building-blocks.jpg" alt=""> </center> <center> PostgreSQL: Building on Solid Foundations </center> What would be missing mainly would be the ability for an Extension to switch from being file based to being a template, either because the author decided to change the way he's shipping it, or because the user is switching from using the <a href="http://pgxnclient.projects.pgfoundry.org/">pgxn client</a> to using proper system packages. I guess that's something we can see about later, though. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 13 Dec 2012 11:34:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/12/13-Inline-Extensions.html</guid> </item> <item> <title>Trigger Parameters</title> <link>http://tapoueh.org/blog/2012/12/06-parametrized-triggers.html</link> <description><![CDATA[We have a not too active <a href="http://archives.postgresql.org/pgsql-fr-generale/2012-12/index.php">postgresql-fr-generale</a> mailing list where some interesting questions are asked by our subscribers. This article comes from such a question about how to deal with trigger parameters, which are nice to have, but static.

<center> <img src="../../../images/trigger-wheels.big.jpg" alt=""> </center> Another way to ask that question is saying that ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 06 Dec 2012 11:10:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/12/06-parametrized-triggers.html</guid> </item> <item> <title>M-x ack</title> <link>http://tapoueh.org/blog/2012/11/22-Emacs-Ack-Mode.html</link> <description><![CDATA[I've been asked about how to integrate the <a href="http://betterthangrep.com/">ack</a> tool (you know, the one that is better than grep) into Emacs today. Again. And I just realized that I didn't blog about my solution. That might explain why I keep getting asked about it after all...

So here it is, M-x ack: <pre class="src"> ;;; dim-ack.el — Dimitri Fontaine ;; ;; http://stackoverflow.com/questions/2322389/ack-does-not-work-when-run-from-grep-find-in-emacs-on-windows (defcustom ack-command (or (executable-find "ack")

(executable-find "ack-grep")) "Command to use to call ack, e.g. ack-grep under debian" :type 'file)

(defvar ack-command-line (concat ack-command " —nogroup —nocolor ")) (defvar ack-history nil) (defvar ack-host-defaults-alist nil)

(defun ack ()

"Like grep, but using ack-command as the default" (interactive) ; Make sure grep has been initialized (if (>= emacs-major-version 22) (require 'grep) (require 'compile)) ; Close STDIN to keep ack from going into filter mode (let ((null-device (format "< %s" null-device)) (grep-command ack-command-line) (grep-history ack-history) (grep-host-defaults-alist ack-host-defaults-alist)) (call-interactively 'grep) (setq ack-history grep-history ack-host-defaults-alist grep-host-defaults-alist)))

(provide 'dim-ack) </pre>

Enjoy! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 22 Nov 2012 17:36:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/11/22-Emacs-Ack-Mode.html</guid> </item> <item> <title>CL Happy Numbers</title> <link>http://tapoueh.org/blog/2012/11/20-CL-Happy-Numbers.html</link> <description><![CDATA[A while ago I stumbled upon <a href="http://tapoueh.org/blog/2010/08/30-happy-numbers.html">Happy Numbers</a> as explained in <a href="http://programmingpraxis.com/2010/07/23/happy-numbers/">programming praxis</a>, and offered an implementation of them in SQL and in Emacs Lisp. Yeah, I know. Why not, though?

<center> <img src="../../../images/happy-numbers.png" alt=""> </center> Today I'm back on that topic and as I'm toying with Common Lisp I though it would be a good excuse to learn me some new tricks. As you can see from the earlier blog entry, last time I did attack the digits problem quite lightly. Let's try a better approach now. <pre class="src"> (defun digits (n)

"return the list of the digits of N" (nreverse (loop for x = n then r for (r d) = (multiple-value-list (truncate x 10)) collect d until (zerop r)))) </pre>

As you can see I wanted to use that facility I like very much, the

for
x = n then r

way to handle first loop iteration differently from the next ones. But I've been hinted on #lisp that there's a much better way to write same code: <pre class="src"> (defun integer-digits (integer)

"stassats version" (nreverse (loop with remainder do (setf (values integer remainder) (truncate integer 10)) collect remainder until (zerop integer)))) </pre>

That code runs about twice as fast as the previous one and is easier to reason about. It's using setf and the form <a href="http://www.lispworks.com/documentation/lw51/CLHS/Body/f_values.htm">setf values</a>, something nice to discover as it seems to be quite powerful. Let's see how to use it, even if it's really simple: <pre class="src"> CL-USER> (integer-digits 12304501) (1 2 3 0 4 5 0 1) </pre> Let's move on to solving the Happy Numbers problem though: <pre class="src"> (defun sum-of-squares-of-digits (integer)

(loop with remainder do (setf (values integer remainder) (truncate integer 10)) sum (* remainder remainder) until (zerop integer)))

(defun happy? (n &optional seen)

"return true when n is a happy number" (let* ((happiness (sum-of-squares-of-digits n))) (cond ((eq 1 happiness) t) ((memq happiness seen) nil) (t (happy? happiness (push happiness seen))))))

(defun find-happy-numbers (limit)

"find all happy numbers from 1 to limit" (loop for n from 1 to limit when (happy? n) collect n)) </pre>

And here's how it goes: <pre class="src"> CL-USER> (find-happy-numbers 100) (1 7 10 13 19 23 28 31 32 44 49 68 70 79 82 86 91 94 97 100)

CL-USER> (time (length (find-happy-numbers 1000000))) (LENGTH (FIND-HAPPY-NUMBERS 1000000)) took 1,621,413 microseconds (1.621413 seconds) to run.

116,474 microseconds (0.116474 seconds, 7.18%) of which was spent in GC. During that period, and with 4 available CPU cores, 1,431,332 microseconds (1.431332 seconds) were spent in user mode 145,941 microseconds (0.145941 seconds) were spent in system mode 185,438,208 bytes of memory allocated. 1 minor page faults, 0 major page faults, 0 swaps. 143071 </pre>

Of course that code is much faster than the one I wrote before both in SQL and Emacs Lisp, the reason being that instead of writing the number into a string with (format t "~d" number) then <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_subseq.htm">subseq</a> to get them one after the other, we're now using <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_floorc.htm">truncate</a>. Happy hacking! <h3>Update</h3> It turns out that to solve math related problem, some maths hindsight is helping. Who would have believed that? So if you want to easily get some more performances out of the previous code, just try that solution: <pre class="src"> (defvar *depressed-squares* '(0 4 16 20 37 42 58 89 145)

"see http://oeis.org/A039943")

(defun undepressed? (n)

"same as happy?, using a static list of unhappy sums" (cond ((eq 1 n) t) ((member n depressed-squares) nil) (t (let ((h (sum-of-squares-of-digits n))) (undepressed? h)))))

(defun find-undepressed-numbers (limit)

"find all happy numbers from 1 to limit" (loop for n from 1 to limit when (undepressed? n) collect n)) </pre>

Time to compare: <pre class="src"> CL-USER> (time (length (find-happy-numbers 1000000))) (LENGTH (FIND-HAPPY-NUMBERS 1000000)) took 1,938,048 microseconds (1.938048 seconds) to run.

290,902 microseconds (0.290902 seconds, 15.01%) of which was spent in GC. During that period, and with 4 available CPU cores, 1,778,021 microseconds (1.778021 seconds) were spent in user mode 140,862 microseconds (0.140862 seconds) were spent in system mode 185,438,208 bytes of memory allocated. 3,320 minor page faults, 0 major page faults, 0 swaps. 143071

CL-USER> (time (length (find-undepressed-numbers 1000000))) (LENGTH (FIND-UNDEPRESSED-NUMBERS 1000000)) took 1,036,847 microseconds (1.036847 seconds) to run.

5,372 microseconds (0.005372 seconds, 0.52%) of which was spent in GC. During that period, and with 4 available CPU cores, 1,018,708 microseconds (1.018708 seconds) were spent in user mode 16,982 microseconds (0.016982 seconds) were spent in system mode 2,289,152 bytes of memory allocated. 143071 CL-USER> </pre> ]]></description> <author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 20 Nov 2012 18:20:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/11/20-CL-Happy-Numbers.html</guid> </item> <item> <title>About Vimgolf</title> <link>http://tapoueh.org/blog/2012/11/06-About-vimgolf.html</link> <description><![CDATA[Following some tweet I found myself desultory watching an episode of the awesome <a href="http://vimeo.com/channels/222837">VimGolf in Emacs</a> video series by <a href="http://vimeo.com/timvisher">Tim Visher</a>. Those series are about picking some challenge from <a href="http://vimgolf.com/">vimgolf</a> and implementing it with our favorite editor instead. Because <a href="http://emacsrocks.com/">Emacs Rocks</a> guys.

<center> <a class="image-link" href="http://emacsrocks.com/"> <img src="../../../images/emacs-rocks-logo.png"></a> </center> Let me tell you upfront that I really dislike the whole idea of the vim golf challenge. I've been a user of both Emacs and Vim for many years, and finally decided to switch to living in Emacs; or if you prefer, climbing my way up from level 2 as in <a href="http://blog.vivekhaldar.com/post/3996068979/the-levels-of-emacs-proficiency">The Levels Of Emacs Proficiency</a>. The reason why is that I found that in my case, using Vim would mean spending more time thinking about how to do some editing operation rather than the problem I wanted to solve by editing some text, most often code. Of course, the main effect of Vim Golf is to make you focus even more on the wrong problem. There's still a good side of it though, which is that such challenges are good excuses to discover new features of your editor. So let's use that excuse to talk about some nice Emacs features. <center> <a class="image-link" href="http://vimgolf.com/challenges/4dd3e19aec9eb6000100000d"> <img src="../../../images/vim_golf_logo.png"></a> </center> <center> Vim Golf Challenge: Complete the hex array data </center> <h3>The challenge</h3> The previous image will lead you to a particular challenge where it's all about filling in an array with consecutive hexadecimal numbers written as 0xab, where you begin with a template containing only the 0x00 entry. The idea is of course to use the Vim feature that will increment the number at point, and is available through the C-a keystroke. <pre class="src"> unsigned int hex[] = {

0x00, }; </pre>

<h3>A first solution</h3> Emacs does not ship with an increment-number-at-point, much less so with one that would support decimal, octal and hexadecimal and even automatically recognize 0x as a prefix meaning that the next number is hexadecimal. But Emacs ship with <a href="http://www.gnu.org/software/emacs/manual/html_node/emacs/Keyboard-Macros.html">Emacs Keyboard Macros</a> and those have a counter, so it's easy enough to fill in numbers from 1 to 255 that way: M-1 F3 F3 , F4 will register a macro where the counter starts at 1, and each time you hit F4 it will insert the current counter value, increment it and insert a coma. You want to do that 254 times, so you do C-u 2 5 4 F4 and Emacs just does that. Now, to transform those decimal numbers into their hexadecimal representation, you can use advanced <a href="http://www.gnu.org/software/emacs/manual/html_node/emacs/Regexp-Replace.html">Emacs Regexp Replace</a> features. Replace [0-9]+ with the result from the following Emacs Lisp code:

\,(format
&quot;0x%02x&quot; (string-to-number \&amp;))

. The \& in there will be replaced by the matching text, so that will do what we need here, turning 10 into 0x0a. <h3>Let's get some better tools</h3> We could do better, though. I happen to already use a key chord to duplicate the current line, and we would need a function to <a href="http://www.emacswiki.org/emacs/IncrementNumber">Increment Number At Point</a>. Those I found over at <a href="http://www.emacswiki.org/">EmacsWiki</a> were not to my taste as they were not able to figure out easily which base to use. So here's a little Emacs Lisp example showing how to extend your favorite editor to have some Vim common features, which is why Emacs ships with Emacs Lisp in the first place. <pre class="src"> (defun duplicate-current-line (&optional n)

"duplicate current line, make more than 1 copy given a numeric argument" (interactive "p") (let ((nb (or n 1)) (current-line (thing-at-point 'line))) (save-excursion ;; when on last line, insert a newline first (when (or (= 1 (forward-line 1)) (eq (point) (point-max))) (insert "\n"))

;; now insert as many time as requested (while (> n 0) (insert current-line) (decf n))) ;; now move down as many lines as we inserted (next-line nb)))

(global-set-key (kbd "C-S-d") 'duplicate-current-line) </pre>

<center> <a class="image-link" href="http://lisperati.com/"> <img src="../../../images/emacs-on-toaster.jpg"></a> </center> <pre class="src"> (require 'cl) ; destructuring-bind is found there (defun dim:increment-number-at-point (&optional prefix)

(interactive "p") (let* ((beg (skip-chars-backward "0-9a-fA-F")) (hexa (save-excursion (forward-char -2) (looking-at-p "0x"))) ;; force the prefix to hexa (4) we see "0x" before the number (prefix (if hexa 4 prefix)) (end (re-search-forward "[0-9a-fA-F]+" nil t)) (nstr (match-string 0)) (l (- (match-end 0) (match-beginning 0))) (fmt (format "%%0%d" l))) (message "PLOP: %d" prefix) (destructuring-bind (base format) (case prefix ((1) '(10 "d")) ; no command prefix, decimal ((4) '(16 "x")) ; C-u, hexadecimal ((16) '(8 "o"))) ; C-u C-u, octal (let* ((n (string-to-number nstr base)) (n+1 (+ n 1)) (fmt (format "%s%s" fmt format))) (replace-match (format fmt n+1))))))

(global-set-key (kbd "C-c +") 'dim:increment-number-at-point) </pre>

<blockquote> So if you're using Emacs a lot but always found an excuse not to grasp Emacs Lisp, I hope that article could be an excuse for you to do so… </blockquote> <h3>Another solution</h3> Anyway, now that we are much better equipped, we can picture a better way to solve the problem. Instead of using a macro that inserts the next counter value, we can use one that duplicate current line, increment number at point (and figures out on its own that the number prefixed with 0x is hexadecimal), and does that 254 times more. Then it's all about reformatting the text so that if fits nicely on screen, and for that the command

M-q runs
the command fill-paragraph

is exactly what we need. The command

C-x f runs
the command set-fill-column

can be used to set the maximum column we allow Emacs to reach before going to the next line. Our Golf then becomes a 19 steps solution if you start with the cursor at the ',' in the previous example: <pre class="src"> C-x f 5 6 RET F3 C-S-d C-c + F4 C-u 2 5 4 F4 C-SPC M-< C-n M-q </pre> First, set the fill column, then register a macro (in between F3 and F4) that will duplicate current line (using C-S-d) then increment number at point (using C-c +). Third line, replay that macro 254 times (C-u 2 5 4 F4). Fourth line, select all those hexadecimal numbers and fill the paragraph they form correctly, so as to get: <h3>All those tips for...</h3> <pre class="src"> unsigned int hex[] = {

0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, 0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, 0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, 0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, 0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, 0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, }; </pre>

<h3>Conclusion</h3> Thanks to the excuse of that challenge we now have a generic facility to increment a number at point in different common bases, which is a nice building block for all kinds of Emacs Keyboard Macros. We also now have a function to duplicate the line at point, which is something I use very often myself. More importantly, we've been refreshing our memory on how to use some advanced replacement facilities wherein you can actually use inline Emacs Lisp code as a replacement pattern, and for the most interested readers here we have a good excuse to learn some more about Emacs Lisp programming. The main thing I want to say is that using Emacs Keyboard Macros is an interactive process: you don't have to pause your current activity to write some code in another language (here, that would be either Vim Script or Emacs Lisp) just to save a few minutes on a boring task. How effective your are at solving that challenge, for me, is not at all about measure how many keystrokes you ended up using, it's all about being able to get some precious help from your working tools without having to stop focusing on the main problem you are solving. I wouldn't ever get to write such Emacs Lisp code when doing that kind of editing once. I would only do that when I'm thinking I've just been doing a boring task by hand one time too many already. Like for example copying and pasting the pg_backend_pid() of the <a href="http://www.postgresql.org/">PostgreSQL</a> backend I'm working with at the psql prompt so that I can attach gdb to it. I'll get back to talking about <a href="https://github.com/dimitri/pgdevenv-el">pgdevenv-el</a> later! Hope you did enjoy that article, whose goal is to help you while you're journeying in <a href="http://blog.vivekhaldar.com/post/3996068979/the-levels-of-emacs-proficiency">The Levels Of Emacs Proficiency</a>. <h3>Update</h3> While looking at the docs for the <a href="http://www.gnu.org/software/emacs/manual/html_node/emacs/Keyboard-Macro-Counter.html#Keyboard-Macro-Counter">Keyboard Macro Counter</a> to check how to reset it without having to record the macro again, I just stumbled on this part of the docs: C-x C-k C-f runs the command kmacro-set-format. So another way to solve our problem with only facilities that come with a bare Emacs is the following: <pre class="src"> C-x f 5 6 RET C-x C-k C-f 0x%02x RET C-1 F3 SPC F3 , F4 C-u 2 5 4 F4 DEL C-SPC C-a C-q </pre> We're now at 30 keystrokes, so much more than previously, but it's stock Emacs features and that kmacro-set-format is a wonderful little tool you might as well have a need for in the future. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Sun, 11 Nov 2012 20:52:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/11/06-About-vimgolf.html</guid> </item> <item> <title>Editing SQL</title> <link>http://tapoueh.org/blog/2012/11/06-Interactive-SQL.html</link> <description><![CDATA[It's hard to read my blog yet not know I'm using <a href="http://www.gnu.org/software/emacs/#Platforms">Emacs</a>. It really is a great tool and has a lot to compare to <a href="http://www.postgresql.org/">PostgreSQL</a> in terms of extensibility, documentation quality and community. And there's even a native implementation of the <a href="http://www.postgresql.org/docs/current/static/protocol.html">PostgreSQL Protocol</a> written in <a href="http://www.gnu.org/software/emacs/emacs-lisp-intro/">Emacs Lisp</a>.

<center> <a class="image-link" href="http://www.online-marketwatch.com/pgel/pg.html"> <img src="../../../images/pg-el.png"></a> </center> One of the things where Emacs really shines is that interactive development environment you get when working on some Emacs Lisp code. Evaluating an function as easy as a single key chord, and that will both compile in the function and load it in the running process. I can't tell you how many times I've been missing that ability when editing C code. With PostgreSQL too we get a pretty interactive environment with the <a href="http://www.postgresql.org/docs/current/static/app-psql.html">psql</a> console application, or with <a href="http://www.pgadmin.org/">pgAdmin</a>. One feature from pgAdmin that I've often wished I had in psql is the ability to edit my query online and easily run it in the console, rather than either using the readline limited history editing features or launching a new editor process each time with \e. At the same time I would much prefer using my usual Emacs editor to actually edit the query. If you've been reading that blog before you know what to expect. My solution to the stated problem is available in <a href="https://github.com/dimitri/pgdevenv-el">pgdevenv-el</a>, an Emacs package aimed at helping PostgreSQL developers. Most of the features in there are geared toward the core backend developers, except for this one I want to talk about today (I'll blog about the other ones too I guess). <center> <img src="../../../images/pgdevenv-el-eval-sql.png" alt=""> </center> What you can see from that screenshot is that the selected query text has been sent to the psql buffer and exectuted over there. And that the psql buffer is echoing all queries sent to it. What you can not see straight from that picture is the interaction to get there. Well, I've been implementing some elisp features that I was missing. First, movement: you can do C-M-a and C-M-e to navigate to the beginning and the end of the SQL query at point, like you do in C or in lisp in Emacs. Then, selection: you can do C-M-h to select the SQL query at point, you don't have to navigate yourself, <a href="https://github.com/dimitri/pgdevenv-el">pgdev-sql-mode</a> knows how to do that. Side note, pgdev-sql-mode is the name of the minor mode you need to activate in your SQL buffers to have the magic available. Last but not least, evaluation: as when editing lisp code, you can now use C-M-x to send the current query text to an associated psql buffer. The way to associate the psql buffer to an SQL buffer is currently done thanks to the other pgdevenv-el features that this blog post is not talking about, and the setup is addressed in the documentation: you have to let know pgdevenv-el where your PostgreSQL branches are installed locally so that it can prepare you a Shell buffer with PGDATA and PGPORT already set for you. And currently, for C-M-x to work you need to open the buffer yourself before hand, using C-c - n (to run the command pgdev-open-shell), and type psql in the Shell prompt. What that means for me is that I can at least edit SQL (in PostgreSQL regression files and other places) in my usual Emacs buffer and actually refine it as I go until it does exactly what I need, without having to use the readline history editing or the \e command, which is not great when your Shell is in already running inside Emacs. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 06 Nov 2012 09:55:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/11/06-Interactive-SQL.html</guid> </item> <item> <title>Concurrent Hello</title> <link>http://tapoueh.org/blog/2012/11/04-Concurrent-Hello.html</link> <description><![CDATA[Thanks to <a href="https://twitter.com/mickael/status/265191809100181504">Mickael</a> on twitter I ran into that article about implementing a very basic Hello World! program as a way to get into a new concurrent language or facility. The original article, titled <a href="http://himmele.blogspot.de/2012/11/concurrent-hello-world-in-go-erlang.html">Concurrent Hello World in Go, Erlang and C++</a> is all about getting to know <a href="http://golang.org/">The Go Programming Language</a> better.

To quote the article: <blockquote> The first thing I always do when playing around with a new software platform is to write a concurrent "Hello World" program. The program works as follows: One active entity (e.g. thread, Erlang process, Goroutine) has to print "Hello " and another one "World!\n" with the two active entities synchronizing with each other so that the output always is "Hello World!\n". </blockquote> Here's my try in <a href="http://cliki.net/">Common Lisp</a> using <a href="http://lparallel.org/">lparallel</a> and some local nicknames, the whole 23 lines of it: <pre class="src"> (defun say-hello (helloq worldq n)

(dotimes (i n) (format t "Hello ") (lq:push-queue :say-world worldq) (lq:pop-queue helloq)) (lq:push-queue :quit worldq))

(defun say-world (helloq worldq)

(when (eq (lq:pop-queue worldq) :say-world) (format t "World!~%") (lq:push-queue :say-hello helloq) (say-world helloq worldq)))

(defun hello-world (n)

(let* ((lp:*kernel* (lp:make-kernel 2)) ; a new one each time, as we end it (channel (lp:make-channel)) (helloq (lq:make-queue)) (worldq (lq:make-queue))) (lp:submit-task channel #'say-world helloq worldq) (lp:submit-task channel #'say-hello helloq worldq n) (lp:receive-result channel) (lp:receive-result channel) (lp:end-kernel))) </pre>

If you want to play locally with that code, I've been updating it to a github project named <a href="https://github.com/dimitri/go-hello-world">go-hello-world</a>, even if it's coded in CL. See the package.lisp in there for how I did enable the local nicknames lp and lq for the lparallel packages. <h3>Beware of the REPL</h3> In a previous version of this very article, I said that sometimes I get an extra line feed in the output and I didn't understand why. Some great Common Lisp folks did hint me about that: it's the REPL output that get intermingled with the program output, and that's because the hello-world main function was returning before the thing is over. I've added a receive-result call in it per worker so that it waits until the end of the program before returning to the REPL, and that indeed fixes it. A way to assert that is using the time macro, which was always intermingled with the output before. It's fixed now: <pre class="src"> CL-USER> (time (go-hello-world:hello-world 1000)) Hello World! ... Hello World! (GO-HELLO-WORLD:HELLO-WORLD 1000) took 27,886 microseconds (0.027886 seconds) to run.

1,593 microseconds (0.001593 seconds, 5.71%) of which was spent in GC. During that period, and with 4 available CPU cores, 23,246 microseconds (0.023246 seconds) were spent in user mode 14,427 microseconds (0.014427 seconds) were spent in system mode 4,272 bytes of memory allocated. 10 minor page faults, 0 major page faults, 0 swaps. ( lparallel kernel shutdown manager(62) [Reset] #x30200109F65D> ...) CL-USER> </pre>

<h3>Conclusion</h3> While Go language seems to bring very interesting things on the table, such as better compilation units and tools, I still think that the concurrency primitives at the core of it are easy to find in other places. Which is a good thing, as it means we know they work. That also means that we don't need to accept Go syntax as the only way to properly solve that concurrency problem, I much prefer doing so with Common Lisp (lack of?) syntax myself. <h3>Update</h3> A previous version of this article was finished and published too quickly, and the conclusion was made from a buggy version of the program. It's all fixed now. Thanks a lot to people who contributed comments so that I could fix it, and thanks again to James M. Lawrence for <a href="http://lparallel.org/">lparallel</a>! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Sun, 04 Nov 2012 23:04:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/11/04-Concurrent-Hello.html</guid> </item> <item> <title>PostgreSQL for developers</title> <link>http://tapoueh.org/blog/2012/11/02-Conference-AFUP-Lyon.html</link> <description><![CDATA[As <a href="http://blog.guillaume.lelarge.info/index.php/post/2012/11/01/Conf%C3%A9rence-%C3%A0-l-AFUP-Lyon">Guillaume</a> says, we've been enjoying a great evening conference in Lyon 2 days ago, presenting PostgreSQL to developers. He did the first hour presenting the project and the main things you want to know to start using <a href="http://www.postgresql.org/">PostgreSQL</a> in production, then I took the opportunity to be talking to developers to show off some SQL.

<center> <a class="image-link" href="../../../images/confs/developper-avec-pgsql.pdf"> <img src="../../../images/confs/developper-avec-pgsql-0.png"></a> </center> That slide deck contains mainly SQL language, but some french too, rather than english. Sorry for the inconvenience if that's not something you can read. Get me to talk at an english developer friendly conference and I'll translate it for you! :) The aim of that talk is to have people think about SQL as a real asset in their development tool set. SQL really should get compared to your application development language rather than your UI formating language, it's more like PHP or Python than it is like HTML. So the whole talk is about showing off some advanced SQL features, all provided by default in released PostgreSQL versions. The main parts of the talk all come from an article in this blog: <a href="../10/05-reset-counter.html">Reset Counter</a>. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 02 Nov 2012 16:22:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/11/02-Conference-AFUP-Lyon.html</guid> </item> <item> <title>Another awesome conf</title> <link>http://tapoueh.org/blog/2012/10/30-Prague-Lyon.html</link> <description><![CDATA[Last week was <a href="http://2012.pgconf.eu/">PostgreSQL Conference Europe 2012</a> in Prague, and it's been awesome. Many thanks to the organisers who did manage to host a very smooth conference with 290 attendees, including speakers. That means you kept walking into interesting people to talk to, and in particular the Hallway Track has been a giant success.

<center> <a class="image-link" href="http://www.flickr.com/photos/obartunov/8128604476/lightbox/"> <img src="../../../images/prague.jpg"></a> </center> <center> Photo by <a href="http://www.sai.msu.su/~megera/">Oleg Bartunov</a> </center> I did have the chance to speak several times at that event, and you can get the slides at my <a href="../../../conferences.html">Conferences</a> page that I try to keep up to date. I did one talk about <a href="http://www.postgresql.eu/events/schedule/pgconfeu2012/session/318-implementing-high-availability/">Implementing High Availability</a> that was about 2 hours long (a double slot), <a href="http://www.postgresql.eu/events/schedule/pgconfeu2012/session/373-lightning-talks/">PGQ Cooperative Consumers</a> that Marko Kreen copresented with me and the <a href="http://www.postgresql.eu/events/schedule/pgconfeu2012/session/317-large-scale-mysql-migration-to-postgresql/">Large Scale MySQL Migration to PostgreSQL</a> that I already presented before this year. <center> <a class="image-link" href="../../../images/high-availability.pdf"> <img src="../../../images/high-availability.png"></a> </center> Next conference is in Lyon and will be in French, the talk is called <a href="http://lyon.afup.org/2012/10/17/presentation-de-postgresql-31102012-a-19h30/">Présentation de PostgreSQL</a>. The audience is going to be composed of PHP developers interested to know more about PostgreSQL, I'll tell you how it goes! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 30 Oct 2012 12:50:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/10/30-Prague-Lyon.html</guid> </item> <item> <title>Prefixes and Ranges</title> <link>http://tapoueh.org/blog/2012/10/16-prefix-update.html</link> <description><![CDATA[It's been a long time since I last had some time to spend on the <a href="http://tapoueh.org/pgsql/prefix.html">prefix</a> PostgreSQL extension and its prefix_range data type. With PostgreSQL 9.2 out, some users wanted me to update the extension for that release, and hinted me that it was high time that I fix that old bug for which I already had a patch.

<center> <img src="../../../images/Prefix-Pro-Blend.jpg" alt=""> </center> <h3>prefix_range release 1.2.0</h3> I'm sorry it took that long. It's now done, you can have prefix 1.2.0 from <a href="https://github.com/dimitri/prefix">https://github.com/dimitri/prefix</a> or if you want a tagged tarball then you can use this link: <a href="https://github.com/dimitri/prefix/tarball/v1.2.0">https://github.com/dimitri/prefix/tarball/v1.2.0</a>. The changelog is all about fixing an index search bug and updating the package to primarily be an extension for PostgreSQL 9.1 and 9.2. Of course older Major Versions are still supported (all of them since 8.1, but please first consider upgrading PostgreSQL) if you want to install it manually, using the prefix--1.2.0.sql file. <h3>debian package</h3> And thanks to <a href="http://www.df7cb.de/">Christoph Berg</a> the debian package is already validated and has reached debian experimental. We don't target sid these days because debian is preparing a new stable release, so there's a freeze. I think. Anyway, take your prefix package from here if you need it: <a href="http://packages.debian.org/experimental/postgresql-9.1-prefix">http://packages.debian.org/experimental/postgresql-9.1-prefix</a>. <h3>Range Types</h3> If you step back a little there's an interesting question to answer here. Why isn't prefix_range and <a href="http://www.postgresql.org/docs/9.2/static/rangetypes.html">PostgreSQL Range Type</a>? Given the names it seems like a pretty good candidate. Well the thing is that to make a generic range type you need to have a total ordering on the range elements, and a distance function that tells you how far any two elements of a range are one from each other. When talking about prefixes, I don't see how to do that. The prefix range ['abcd', 'abce') contains an infinity of elements, all the strings that begin with the letters abcd. I guess that coming with an ordering on text is possible, but what if any text element represents a prefix? I mean that in our case, the elements would be of type prefix, and 'abcd' is a prefix of 'abcdefg'. The question I want to answer is that given a table with prefixes 'abcd', 'abce' and 'abcde' which row in there has the longest prefix matching the literal 'abcdef'. I'm not seeing how to abuse the Range Types mechanism to implement that, so if you have some ideas please share them! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 16 Oct 2012 10:47:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/10/16-prefix-update.html</guid> </item> <item> <title>Reset Counter</title> <link>http://tapoueh.org/blog/2012/10/05-reset-counter.html</link> <description><![CDATA[I've been given a nice puzzle that I think is a good blog article opportunity, as it involves some thinking and window functions.

<h3>What's to solve</h3> Say we store in a table entries from a counter that only increases and the time stamp when we did the measurement. So that when you read 30 then later 40 in fact that means we counted 10 more the second reading when compared to the first, in other words the first 30 are counted again in the second counter value, 40. <center> <a class="image-link" href="http://xkcd.com/363/"> <img src="../../../images/reset.png"></a> </center> Now of course it's a real world counter. Think network traffic counter on a network interface, if you want something real to play with in your mind. So the counter will sometime reset and you will read measure sequences such as 40, 0, 20 if you happen to read just when the counter is reset, or most of the time that will look like 45, 25, 50. The question we want to answer is, given a series of that counter measures, including some resets, what is the current logical value of the counter? Given the sequence of measures 0, 10, 20, 30, 40, 0, 20, 30, 60 the result we want is 40 + 60, that is 100. Right? <h3>Playing with some data</h3> Let's model an hypothetical dataset easy enough to play with. What about just the previous example? We also need to time stamp the measurements, let's just use a tick for now, as it's easier to think about: <pre class="src"> create table measures(tick int, nb int);

insert into measures

values (1, 0), (2, 10), (3, 20), (4, 30), (5, 40), (6, 0), (7, 20), (8, 30), (9, 60); </pre>

Now that we have some data in a table to play with, let's try to find out the numbers we are interested in: we only want to keep the latest measure we read on the counter just before it wraps. That means values where the next one (in tick or time stamping order) is lesser than the current counter value. As we are lucky enough to be playing with the awesome <a href="http://www.postgresql.org/">PostgreSQL</a> which brings <a href="http://www.postgresql.org/docs/9.2/static/tutorial-window.html">window functions</a> on the table, we can easily implement just what we said in a readable way: <pre class="src">

select tick, nb, case when lead(nb) over w < nb then nb when lead(nb) over w is null then nb else null end as max from measures window w as (order by tick); </pre>

The firt case is the exact translation of the problem as spelled in english in just the previous paragraph where we stated we want to keep the current counter value in case of a wraparound, so I guess it's easy enough to get at. <center> <img src="../../../images/reset-circuit-thumbnail.jpg" alt=""> </center> Then we have a couple of tricks in that query in order to massage the data as we want it. First, the last row of the output won't have a lead, that window function call is going to return NULL. In that case, we keep the current counter value as if we just did a wraparound. And finally, when there's no wraparound, we don't care about the data. Well, for the purpose of knowing the current logical value of the counter, that is. And we get that encouraging result: <pre class="src">

tick

max

+——+——

1	0
2	10
3	20
4	30
5	40	40
6	0
7	20
8	30
9	60	60

</pre> As you see, we have been able to create a new column out of the dataset, and that new column only contains the data we are interested into. <h3>Finding the current counter value</h3> All we have to do now is sum this computed columns entries. Remember that the sum() aggregate function will simply discard nulls, so that we don't have to turn them into a bunch of 0. <pre class="src"> with t(tops) as (

select case when lead(nb) over w < nb then nb when lead(nb) over w is null then nb else null end as max from measures window w as (order by tick) ) select sum(tops) from t; </pre>

<center> <img src="../../../images/reset-elect.jpg" alt=""> </center> And here's the expected result: <pre class="src">

sum —— 100 </pre>

Now what about testing with another set of data or two, just to be sure that the counter is allowed to wrap more than once within our solution? <pre class="src"> insert into measures

values (10, 0), (11, 10), (12, 30), (13, 35), (14, 45), (15, 25), (16, 50), (17, 100), (18, 110); </pre>

Then we have: <pre class="src"> with t(tops) as (

select case when lead(nb) over w < nb then nb when lead(nb) over w is null then nb else null end as max from measures window w as (order by tick) ) select sum(tops) from t; sum —— 255 (1 row) </pre>

All good! <h3>Counter logical value over a given period</h3> Now of course what we want is to find the logical value of the counter for a given day's or month's worth of measures. We then need to pay attention to the value of the counter at the start of our period so that we know to substract it from the logical sum over the period. <center> <img src="../../../images/reset-coin-counter.jpg" alt=""> </center> Here's an SQL version of the same sentence, applied to the period in between ticks 4 and 14, in a completely arbitrary choosing of mine: <pre class="src"> with t as (

select tick, first_value(nb) over w as first, case when lead(nb) over w < nb then nb when lead(nb) over w is null then nb else null end as max from measures where tick >= 4 and tick < 14 window w as (order by tick) ) select sum(max) - min(first) as sum from t; </pre>

Here we are using the first_value() window function to retain it in the whole resultset of the Common Table Expression (the inner query introduced by the keyword WITH is called that way). And when doing the sum we're interested in at the outer level, we didn't forget to substract the first value: we need to use an aggregate here because we're doing a sum() aggregate at the same query level, and we have the same value in each row of the resultset, so we used min(), max() would have been as good. Another important trick we're using in that query is how to express the date range. Never use between for that, as you would end up counting boundaries twice, and customer won't like your accounting process if you do that. Always use a combo of inclusive and exclusive boundaries comparison, as in that WHERE clause in the previous query. Let's have a quick look at the raw data in that range, using another nice aggregate that PostgreSQL comes with: <pre class="src"> select array_agg(nb) from measures where tick >= 4 and tick < 14;

array_agg ——————————- {30,40,0,20,30,60,0,10,30,35} (1 row) </pre>

And now, the logical counter value for that period is computed as the following value by the previous query: <pre class="src">

sum
105 (1 row) </pre>

We can verify it manually, we want 40 + 60 + 35 - 30, I think we're all good again. Don't forget we have to substract the first measure from the period! <h3>Extending the problem</h3> Another interesting problem, that we didn't have here but that I find interesting enough to extend this article, is finding the ranges of time (here, ticks) within which the counter didn't reset. <center> <img src="../../../images/reset-A2a.jpg" alt=""> </center> The query is more complex because we need to split the data into partitions, each partition containing data from the same counter series of measures without wrapping. The usual trick is to self-join our data set so that for each given row we have a set of rows from the same partition, we are going to instead use a correlated subquery to go fetch the next wraparound value: <pre class="src"> with tops as (

select tick, nb, case when lead(nb) over w < nb then nb when lead(nb) over w is null then nb else null end as max from measures window w as (order by tick) ) select tick, nb, max, (select tick from tops t2 where t2.tick >= t1.tick and max is not null order by t2.tick limit 1) as p from tops t1;

tick

max

——+——+——+—-

</span> 1	0		5
2	10		5
3	20		5
4	30		5
5	40	40	5
6	0		9
7	20		9
8	30		9
9	60	60	9
10	0		14
11	10		14
12	30		14
13	35		14
14	45	45	14
15	25		18
16	50		18
17	100		18
18	110	110	18

(18 rows) </pre>

With that as an input it's then possible to build ranges of ticks including non wrapping set of measures from our counter, and get for each range the logical value tat the counter had at the end of it: <pre class="src"> with tops as (

select tick, nb, case when lead(nb) over w < nb then nb when lead(nb) over w is null then nb else null end as max from measures window w as (order by tick) ), parts as ( select tick, nb, max, (select tick from tops t2 where t2.tick >= t1.tick and max is not null order by t2.tick limit 1) as p from tops t1 ), ranges as ( select first_value(tick) over w as start, last_value(tick) over w as end, max(max) over w from parts window w as (partition by p order by tick) ) select * from ranges where max is not null;

start

end

max

——-+——+——

</span> 1	5	40
6	9	60
10	14	45
15	18	110

(4 rows) </pre>

<h3>Conclusion</h3> What I hope to have shown here, apart from some window function tips and some nice use cases for common table expressions, is that as a developper adding SQL to your tool set is a very good idea. <center> <img src="../../../images/skill-set.jpg" alt=""> </center> You don't want to have several parts of your code dealing with a logical counter like this, because you want the reporting, accounting, quota, billing and other software to all agree on the values. And you most probably want to avoid to fetch a huge result set of data and process it in the application memory (it'd better fit) rather than just get back a single integer column single row resultset, right? If you find this SQL example to be off the limits, it's a good sign that you need to improve on your skills so that SQL is a real asset of your developer multi languages multi paradygm talents. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 05 Oct 2012 09:44:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/10/05-reset-counter.html</guid> </item> <item> <title>PostgreSQL 9.3</title> <link>http://tapoueh.org/blog/2012/09/15-PostgreSQL-9.3.html</link> <description><![CDATA[<a href="http://www.postgresql.org/">PostgreSQL 9.2</a> is released! It's an awesome new release that I urge you to consider trying and adopting, an upgrade from even 9.1 should be very well worth it, as your hardware could suddenly be able to process a much higher load. Indeed, better performances mean more work done on the same budget, that's the name of the game!

As a PostgreSQL contributor though, the release of 9.2 mainly means to me that it's time to fully concentrate on preparing 9.3. The developement season of which as already begun, by the way, so some amount of work has already been done here. <center> <img src="../../../images/event-trigger.jpg" alt=""> </center> The list of things I want to be working on for that next release is quite long, and looks more like a christmas list than anything else. Let's only talk about those things I might as well make happen rather than all the things I wish I was able to be delivering in a single release... <h3>Event Triggers</h3> We missed 9.2 for wanting to include too big a feature in one go, leading to too many choices to review and take decision about, for once, and also to some non optimal choices that had to be reconsidered. Thanks to <a href="../06/24-back-from-pgcon.html">PGCON</a> in Ottawa earlier this year, I could meet in person with Robert Haas and we've been able to decide how to attack that big patch I had. The first step has been to commit in the PostgreSQL tree only infrastructure parts, on which we will be able to build the feature itself. <h4>Infrastructure</h4> What we already have today is the ability to run user defined function when some event occurs, and an event can only be a ddl_command_start as of now. Also the trigger itself must be written in PLpgSQL or PL/C, as the support for the other languages was not included from the patch. That leaves some work to be done in the next months, right? <h4>PL support</h4> The user defined function will get some information from magic variables such as TG_EVENT and such. That allows easier integration of future information we want to add, without disrupting those existing triggers that you wrote (no API change), at the cost of having to write a specific integration per procedural language. So one of the first things to do now is to take the support for the others PL that I had in my proposal and make a new patch with only that in there. <h4>Fill-in more information</h4> Then again, this first infrastructure part was all about being actually able to run a user function and left behind most of the information I would like the function to have. The information already there is the command tag, the event name and the parsetree that's only usable if you're writing your trigger in C, which we expect some users to be doing. To supplement that, we're talking about the Object ID that has been the target of the event, the schema it leaves in when applicable, the

Object
Name

, the Operation that's running (CREATE, ALTER, DROP), the Object Kind being the target of said operation (e.g. TABLE or FUNCTION), and the

command
string

. <h4>Publishing the Command String</h4> Publishing the Command String here is not an easy task, because we have to rebuild a normalized version of it. Or maybe we can go with passing explicit context in which the command is running, such as the search_path. Even with an explicit context that would be easy enough to SET back again (in a remote server where you would be replicating the DDL, say), it would be better to normalize the command string so as to remove extra spaces and make it easier to parse and process from a user defined function. That part looks like where most of the work is going to happen in the next commit fests. <h4>Events</h4> The other big thing I want to be working on with respect to this feature is the event support, which is basically hard coded to be ddl_command_start in the current state of the 9.3 code. We certainly will want to be able to run user defined function not only at the very beginning of a DDL command, but also just before it finishes so that the newly created object already exists, for example. We might also be interested into supporting triggers on more than DDL, there I doubt we will see that happening in 9.3, as some people in the community would go crazy about complex use cases. Time is limited, and I think this is better kept open for the next release, as the way our beloved PostgreSQL works is by delivering reliable features: quality first. <h4>Use cases</h4> I'm always happy to hear about use cases for the features I'm working on, and this one has the potential to be covering a non trivial amount of them. I already can think of trigger based replication systems and some integrated extension network facilities. With your help we can give those the place they should have: early days use cases in a great collection. <h3>Extensions</h3> <center> <img src="../../../images/extensions-cords.jpg" alt=""> </center> So yes, event triggers first use case for me is in relation with extensions. Surprise! There's still some more I want to do with extensions, so much that I could consider their implementation in 9.1 just an enabler. In 9.1 the game has been to offer the best support we could design for existing contrib modules, with a very strong angle toward clean support for dump and restore. The typical contrib module exports in SQL a list of C coded functions, sometime supporting a new datatype, sometime a set of administration functions. It's quite rare that contrib modules are handling user data embedded in their SQL definition, and when it happens it's mostly configuration kind of data, such as with <a href="TODO:%20add%20the%20link">PostGIS</a>. Now we want to fully support extensions that are maintaining their own user data, or even those that are all about them. The main difficulty here is that our current design of dump and restore support is following a model where installing the same extesion in a new database is all covered by create extension foo;. This is a limited model of the reality, that we need to expand. The first manifestation of those problems is in the SEQUENCE support in extensions, and that impacts one of my favorite extensions: <a href="http://wiki.postgresql.org/wiki/Skytools">PGQ</a>. <h3>PostgreSQL releases</h3> <a href="http://www.postgresql.org/">PostgreSQL</a> just released an awesome release with 9.2, where we get tremendous performance optimisations and truly innovative features, such as RANGE TYPE. How not to consider PostgreSQL as a part of your application stack, where to develop and host your features. While users are enjoying the newer release, contributors are already preparing the next one, hard at work again! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Sat, 15 Sep 2012 18:43:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/09/15-PostgreSQL-9.3.html</guid> </item> <item> <title>El-Get 4.1 is out</title> <link>http://tapoueh.org/blog/2012/08/28-el-get-new-stable-release.html</link> <description><![CDATA[Please welcome the new stable version of <a href="https://github.com/dimitri/el-get#readme">El-Get</a>, the much awaited version 4.1 has now been branched for your pleasure. It's packed with lots of features to make your life easy, comes with a Info documentation book and even has a logo. That's no joke, I found one, at least:

<center> <img src="../../../images/el-get.big.png" alt=""> </center> <h3>Why El-Get is relevant</h3> Emacs 24.1 is the first release that includes package.el, and it even allows the user to setup several sources where to fetch packages. Those sources, such as <a href="http://marmalade-repo.org/">Marmalade</a>, are hosting lots of third party code for Emacs. package.el makes it easy to install (partly) those software. This is a very fine way of getting extra features in your Emacs installation, and one that is supported out of the box. For a package to be listed, its sources need to be prepared, and you need to rely on the central website you now depend on to be up and running and accessible. El-Get is all about allowing you to easily cope with the still vast majority of Emacs Lisp extensions you can find out there, that is non packaged code that is only available on some more or less mainstream distribution method, ranging from <a href="http://emacswiki.org/">EmacsWiki</a> to <a href="http://github.com/">github</a> including bare HTTP personal hosting. With El-Get, you fetch the package where it's located. There's no need for a central server to host packaged and released software, and it's easy to share your findings with friends, or even to publish any elisp code you write. El-Get will also take care of final steps that package.el did choose not to support, such as including Info material in your info browser (remember

C-h
i runs the command info

?), running ./configure && make for you, byte compiling the sources you just retrieved, adding the necessary <a href="http://www.gnu.org/software/emacs/emacs-lisp-intro/html_node/Autoload.html">autoload</a> support, etc. And of course one of the methods supported by El-Get is ELPA, known as Emacs Lisp Package Archive and implemented by package.el. So definitely, you typically want both ELPA and <a href="http://github.com/dimitri/el-get">El-Get</a>. <h3>El-Get 4.1 Changelog Summary</h3> The new El-Get release is packed with features. It really is. I will only list some of them now: <ul> <li>Plenty new recipes, we now have 590 of them managed in the El-Get source repository itself, and El-Get will download the current <a href="http://emacswiki.org/">EmacsWiki</a> list of emacs lisp files at install time too.</li> <li>The default installation and usage has been simplified a lot.</li> <li>More options are provided to setup El-Get packages, see el-get-user-package-directory for example.</li> <li>Part of the simplification, el-get-sources has been revisited and now serves only one goal.</li> <li>We dropped (el-get 'wait) which was a misconception and had been broken for a long time in the development version of El-Get.</li> <li>We made improvements in the error handling and in dealing with some corner cases that still happen often enough for users to report them. Please continue reporting them!</li> <li>More caching is done, with a better dependency tracking and status management.</li> <li>Enhanced notification support, from DBUS to growl.</li> <li>Support for checksums with a lot of source types.</li> <li>Completing our git support, shallow clones and submodules are there.</li> <li>Better support for github including the zip and tar releases.</li> <li>Ability to reload a package when it's been updated.</li> <li>Moar features</li> </ul> And most importantly, El-Get documentation is now almost complete and comes in the nice Info format I know you've been expecting for so long! <h3>Using El-Get</h3> Here's a quick summary of what using El-Get is like, for a new user in 4.1. If you're already using El-Get see the section about upgrading. To install El-Get you need to paste those lines to your *scratch* buffer then hit C-j after the last closing parenthesis: <pre class="src"> (url-retrieve

"https://raw.github.com/dimitri/el-get/master/el-get-install.el" (lambda (s) (goto-char (point-max)) (eval-print-last-sexp))) </pre>

Then you can try M-x el-get-list-packages and browse through more than 2000 available packages. Mark the ones you want to install with i then type x to see El-Get fetch and install all those packages you just selected. Here's a summary of what's available to you in the M-x el-get-list-packages buffer: <pre class="src"> Major Mode Bindings: SPC el-get-package-menu-mark-unmark ? el-get-package-menu-describe d el-get-package-menu-mark-delete g el-get-package-menu-revert h el-get-package-menu-quick-help i el-get-package-menu-mark-install u el-get-package-menu-mark-update x el-get-package-menu-execute </pre> Once a package is installed, El-Get will initialize it for you, and it will also do that step at every Emacs startup from there on, provided that you added some lines to your ~/.emacs initialization file, that look a lot like the previous *scratch* code you did paste: <pre class="src"> ;; ;; Here's a typical El-Get integration for your .emacs file: ;; (add-to-list 'load-path "~/.emacs.d/el-get/el-get") (setq el-get-user-package-directory "~/.emacs.d/packages.d/")

(unless (require 'el-get nil t)

(with-current-buffer (url-retrieve-synchronously "https://raw.github.com/dimitri/el-get/master/el-get-install.el") (goto-char (point-max)) (eval-print-last-sexp)))

(el-get 'sync) </pre>

Then you can add files named like init-<package>.el in the el-get-user-package-directory directory, those files will get loaded when El-Get initialize <package>. You can also use M-x el-get-install if you want to bypass the full screen package listing, you will get completion on the package name. <h3>Community and development</h3> El-Get community grew to be a really cool place to be participating in these days, with core and recipe contributions from more than 130 different people already, and with 526 stars on github and 184 forks. I almost can't believe it! <pre class="src">

git —no-pager shortlog -n -s

wc -l

137

git —no-pager shortlog -n -s

head -10

734 Dimitri Fontaine 336 Ryan C. Thompson 114 Julien Danjou 110 Dave Abrahams 73 Ryan Thompson 72 Sébastien Gross 42 Takafumi Arakaki 27 Alex Ott 25 Yakkala Yagnesh Raghava 21 Rüdiger Sonderfeld </pre>

Now that we have something that looks like a core team forming up, I'm thinking about scheduling much more aggressive stable release. 4.1 has been very long in the making, I hope to now have a rapid release cycle leading us to 4.2 in quite a short time. As that's not an individual effort by any mean, though, only history will tell. <h3>The roadmap</h3> We have lots of ideas and some rough edges to address, so 4.1 is only a stop in the release history of El-Get. Next ideas include better error management in face of rare corner cases and in face of external events, like when you did rm -rf a directory holding an El-Get managed extension: we should mark it removed and clean up the autoloads that came from it. <h3>Upgrading to 4.1</h3> This item has received some treatment in the documentation. The basic idea is that el-get-sources is no longer what it used to be, it's now only an alternative source location for recipes, like it should always have been. Not that you can still override in there some properties that you want merged with an official recipe. The new thing about el-get-sources is that it will no longer be the authoritative list of packages that El-Get manages. That list is not either given explicitly when calling the el-get function in your .emacs setup, or derived from the packages that are known installed on your system (like e.g. debian is doing). Also, given that it took us so much time to brew 4.1 a lot of packages have changed either their hosting location or even switched their SCM. In such cases an automatic update of the recipe will no longer be possible, you might need to el-get-remove then el-get-install packages to get them back. <h3>Conclusion</h3> El-Get 4.1 is now ready for public consumption, don't be shy, we've been a lot of users running the development branch for a long time now, I'm running 4.0.7.6901194 while writing this post. 4.0 is the development version of what is now released as 4.1. Many thanks to all who contributed to El-Get and to all our users, I'm very proud that together we worked out a very nice and complete tool! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 28 Aug 2012 11:43:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/08/28-el-get-new-stable-release.html</guid> </item> <item> <title>Fast and stupid?</title> <link>http://tapoueh.org/blog/2012/08/20-performance-the-easiest-way.html</link> <description><![CDATA[I stumbled onto an interesting article about performance when using python, called <a href="http://jiaaro.com/python-performance-the-easyish-way">Python performance the easy(ish) way</a>, where the author tries to get the bet available performances out of the dumbiest possible python code, trying to solve a very simple and stupid problem.

With so many smart qualifiers you can only guess that I did love the challenge. The idea is to write the simplest code possible and see how smarter you need to be when you need perfs. Let's have a try! <h3>local python results</h3> Here's the code I did use to benchmark the python solution: <pre class="src"> def sumrange(arg):

return sum(xrange(arg))

def sumrange2(arg):

x = i = 0 while i < arg: x += i i += 1 return x

import ctypes ct_sumrange = ctypes.CDLL('/Users/dim/dev/CL/jiaroo/sumrange.so')

def sumrange_ctypes(arg):

return ct_sumrange.sumrange(arg)

if __name__ == "__main__":

import timeit t1 = timeit.Timer('import jiaroo; jiaroo.sumrange(10**10)') t2 = timeit.Timer('import jiaroo; jiaroo.sumrange2(10**10)') ct = timeit.Timer('import jiaroo; jiaroo.sumrange_ctypes(10**10)')

print 'timing python sumrange(10**10)' print 'xrange: %5fs' % t1.timeit(1) print 'while: %5fs' % t2.timeit(1) print 'ctypes: %5fs' % ct.timeit(1) </pre>

Oh. And the C code too, sorry about that. <pre class="src"> #include <stdio.h>

int sumrange(int arg) {

int i, x; x = 0;

for (i = 0; i < arg; i++) { x = x + i; } return x; } </pre>

And here's how I did compile it. The author of the inspiring article insisted on stupid optimisation targets, I did follow him: <pre class="src"> gcc -shared -Wl,-install_name,sumrange.so -o sumrange.so -fPIC sumrange.c -O0 </pre> And here's the result I did get out of it: <pre class="src"> python jiaroo.py timing python sumrange(10**10) xrange: 927.039917s while: 2377.291237s ctypes: 5.297124s </pre> Let's be fair, with -O2 we get much better results: <pre class="src"> timing python sumrange(10**10) ctypes: 1.065684s </pre> <h3>Common Lisp to the rescue</h3> So let's have a try in Common Lisp, will you ask me, right? Here's the code I did use, you can see three different tries: <pre class="src"> ;;;; jiaroo.lisp ;;; ;;; See http://jiaaro.com/python-performance-the-easyish-way ;;; ;;; The goal here is to find out if CL needs to resort to C for very simple ;;; optimisation tricks like python apparently needs too, unless using pypy ;;; (to some extend). (in-package #:jiaroo)

;;; "jiaroo" goes here. Hacks and glory await! (defun sumrange-loop (max)

"return the sum of numbers from 1 to MAX" (let ((sum 0)) (declare (type (and unsigned-byte fixnum) max sum) (optimize speed)) (loop for i fixnum from 1 to max do (incf sum i))))

(defun sumrange-dotimes (max)

"return the sum of numbers from 1 to MAX" (let ((sum 0)) (declare (type (and unsigned-byte fixnum) max sum) (optimize speed)) (dotimes (i max sum) (incf sum i))))

(defun pk-sumrange (max)

(declare (type (and unsigned-byte fixnum) max) (optimize speed)) (let ((sum 0)) (declare (type (and fixnum unsigned-byte) sum)) (dotimes (i max sum) (setf sum (logand (+ sum i) most-positive-fixnum)))))

(defmacro timing (&body forms)

"return both how much real time was spend in body and its result" (let ((start (gensym)) (end (gensym)) (result (gensym))) `(let* ((,start (get-internal-real-time)) (,result (progn ,@forms)) (,end (get-internal-real-time))) (values ,result (/ (- ,end ,start) internal-time-units-per-second)))))

(defun bench-sumrange (power)

"print execution time of both the previous functions" (let* ((max (expt 10 power)) (lp-time (multiple-value-bind (r s) (timing (sumrange-loop max)) s)) (dt-time (multiple-value-bind (r s) (timing (sumrange-dotimes max)) s)) (pk-time (multiple-value-bind (r s) (timing (pk-sumrange max)) s))) (format t "timing common lisp sumrange 10**~d~%" power) (format t "loop: ~2,3fs ~%" lp-time) (format t "dotimes: ~2,3fs ~%" dt-time) (format t "pk dotimes: ~2,3fs ~%" pk-time))) </pre>

And here's the results: <pre class="src"> CL-USER> (bench-sumrange 10) timing common lisp sumrange 10**10 loop: 11.213s dotimes: 7.642s pk dotimes: 22.185s NIL </pre> <h3>Discussion</h3> So python is very slow. C is pretty fast. And Common Lisp just in the middle. Honnestly I expected better performances from my beloved Common Lisp here, but I didn't try very hard, by using <a href="http://ccl.clozure.com/">Clozure Common Lisp</a> which is not the quicker Common Lisp implementation around. For this very benchmark, if you're seeking speed use either <a href="http://sbcl.org/">Steel Bank Common Lisp</a> or <a href="http://www.clisp.org/">CLISP</a> which is known to have a pretty fast bignums implementation (which you don't need in 64 bits in that game). On the other hand, I think that having to go write a C plugin and deal with how to compile and deploy it in the middle of a python script is something to avoid. When using Common Lisp you don't need to resort to that for the runtime to get down from python xrange implementation at 927.039917s down to the dotimes implementation taking 7.642s. That's about 121 times faster. So while C is even better, and while I would like a Common Lisp guru to show me how to get a better speed here, I still very much appreciate the solution here. Let's see the winning source code in python and common lisp to compare the programmer side of things: how hard was it really to get 121 times faster? <pre class="src"> def sumrange(arg):

return sum(xrange(arg)) </pre>

<pre class="src"> (defun sumrange-dotimes (max)

"return the sum of numbers from 1 to MAX" (let ((sum 0)) (declare (type (and unsigned-byte fixnum) max sum) (optimize speed)) (dotimes (i max sum) (incf sum i)))) </pre>

That's about it. Yes we can see some manual optimisation directives here, which are optimisation extra complexity. Not to the same level as bringing a compiled artifact that you need to build and deploy, though. Remember that you will need to know the full path where to find the sumrange.so file on the production system, in the optimised python case, so that's what we are comparing against. Here's what happens without the optimisation, and with a smaller target: <pre class="src"> CL-USER> (time (jiaroo:sumrange-dotimes (expt 10 9))) (JIAROO:SUMRANGE-DOTIMES (EXPT 10 9)) took 722,592 microseconds (0.722592 seconds) to run. During that period, and with 2 available CPU cores,

714,709 microseconds (0.714709 seconds) were spent in user mode 1,183 microseconds (0.001183 seconds) were spent in system mode 499999999500000000

CL-USER> (time (let ((sum 0)) (dotimes (i (expt 10 9) sum) (incf sum i)))) (LET ((SUM 0)) (DOTIMES (I (EXPT 10 9) SUM) (INCF SUM I))) took 2,174,767 microseconds (2.174767 seconds) to run. During that period, and with 2 available CPU cores,

2,156,549 microseconds (2.156549 seconds) were spent in user mode 10,225 microseconds (0.010225 seconds) were spent in system mode 499999999500000000 </pre>

We get a 3 times speed-up from those 2 lines of lisp optimisation directives, which is pretty good. And it's exponential as I didn't have the patience to actually wait until the non optimised 10^10 run finished, I killed it. <h3>Conclusion</h3> That's a case here where I don't know how to reach C level of performances with Common Lisp, which could just be because I don't know yet how to do. Still, getting a 121 times speedup when compared to the pure python version of the code is pretty good and encourages me to continue diving into Common Lisp. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 22 Aug 2012 16:05:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/08/20-performance-the-easiest-way.html</guid> </item> <item> <title>Autumn 2012 Conferences</title> <link>http://tapoueh.org/blog/2012/08/01-autumn-conferences.html</link> <description><![CDATA[The <a href="http://www.postgresql.org/">PostgreSQL</a> community host a number of <a href="../../../conferences.html">conferences</a> all over the year, and the next ones I'm lucky enough to get to are approaching fast now. First, next month in September, we have <a href="http://postgresopen.org/2012/home/">Postgres Open</a> in Chicago, where my talk about <a href="http://tapoueh.org/blog/2012/05/24-back-from-pgcon.html">Large Scale Migration from MySQL to PostgreSQL</a> has been selected!

<center> <img src="../../../images/autumn-leave-480.jpg" alt=""> </center> This talk shares hindsights about the why and the how of that migration, what problems couldn't be solved without moving away and how the solution now looks. The tools used for migrating away the data, the methods the new architecture are detailed. And the new home, in the cloud! Not that much later after that the European PostgreSQL community is giving us a very nice occasion to get to Prague with <a href="http://2012.pgconf.eu/">PostgreSQL Conference Europe 2012</a> (October 23-26). If you've been meaning to meet with the community, if you've been meaning to visit Prague someday, or any mix of those two very good reasons, think about booking that conference already. The <a href="http://2012.pgconf.eu/callforpapers/">call for papers for pgconf.eu</a> has been extended to August 7th, 2012. Consider sharing your hindsights too! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 02 Aug 2012 01:08:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/08/01-autumn-conferences.html</guid> </item> <item> <title>Solving Every Sudoku Puzzle</title> <link>http://tapoueh.org/blog/2012/07/10-solving-sudoku.html</link> <description><![CDATA[<a href="http://norvig.com/">Peter Norvig</a> published a while ago a very nice article titled <a href="http://norvig.com/sudoku.html">Solving Every Sudoku Puzzle</a> wherein he presents a programmatic approach to solving that puzzle game.

<center> <a class="image-link" href="http://en.wikipedia.org/wiki/Sudoku"> <img src="../../../images/sudoku.png"></a> </center> The article is very well written and makes it easy to think that coming up with the code for such a solver is a very easy task, you apply some basic problem search principles and there you are. Which is partly true, in fact. Also, he uses python, and that means that a lot of trivial programming activities are not a concern anymore, such as memory management. As I've been teaching myself <a href="http://www.cliki.net/Common%20Lisp">Common Lisp</a> for some weeks now I though I would like to read a lisp version of his code, and the article even has a section titled Translations. Unfortunately, no lisp version is available there. One might argue that <a href="http://clojure.org/">Clojure</a> is a decent enough lisp, but my current quest is all about Common Lisp really. So I had to write one myself. <pre class="src"> CL-USER> (sudoku:print-puzzle

(sudoku:solve-grid "530070000600195000098000060800060003400803001700020006060000280000419005000080079"))

5 3 4	6 7 8	9 1 2
6 7 2	1 9 5	3 4 8
1 9 8	3 4 2	5 6 7

+——-+——

8 5 9	7 6 1	4 2 3
4 2 6	8 5 3	7 9 1
7 1 3	9 2 4	8 5 6

+——-+——

9 6 1	5 3 7	2 8 4
2 8 7	4 1 9	6 3 5
3 4 5	2 8 6	1 7 9

took 1,974 microseconds (0.001974 seconds) to run. During that period, and with 2 available CPU cores,

1,894 microseconds (0.001894 seconds) were spent in user mode 88 microseconds (0.000088 seconds) were spent in system mode 174,320 bytes of memory allocated. </pre>

<h3>Comments on the python version</h3> Norvig's article is very well written, I think. By that I mean that by reading it you're confident that you've understood the problem and how the solution is articulated, so you almost think you don't need to really try to understand the code, it's just an illustration of the text. Well, not so much. When you want to port the exact same algorithm you have to understand exactly what the code is doing so that you're not implementing something else. All the more when, as I did, you want to use some other data structure. My goal was not to rewrite the code as-is, but to try and come up with idiomatic lisp code implementing Norvig's solution. So rather than using strings and dictionaries (in lisp, they still call them a <a href="http://www.lispworks.com/documentation/lw50/CLHS/Body/f_mk_has.htm">hash table</a>) I've been using more natural data structures. The python code is really uneasy to follow, full of functional programming veteran tricks. I mean avoiding exceptions and simply returning False whenever there's a problem, and using functions such as all and some to manage that. It's certainly working, it's not making the code any easier to read. To summarize, that code looks like it's been written by someone smart who didn't want to spend more than a couple of hours on it, and did take all known trustworthy shortcuts he could to achieve that goal. Quality and readability certainly weren't the key motive. I've been quite deceived after reading a very good article. <h3>Comments on the common lisp version</h3> Keep in mind that I'm just a Common Lisp newbie. I've been told some good pieces of advice by knowledgeable people though, so with some luck my implementation is somewhat lispy enough. So we start by defining some data structures and low-level functions to build up the more complex one, so that it's easier to read and debug. The sudoku puzzle is then a grid of digits and a grid of possible values in places where the digits are yet unknown. The way to represent that 9x9 grid is with using <a href="http://www.lispworks.com/documentation/lw51/CLHS/Body/f_mk_ar.htm">make-array</a>: <pre class="src"> (make-array '(9 9)

:element-type '(integer 0 9) :initial-element 0) </pre>

Then the possible values. I though about using a bit-vector (and actually I did implement it that way), then I've been told that the Common Lisp way to approach that is using <a href="http://psg.com/~dlamkins/sl/chapter18.html">2-complement integer representation</a>, as we have plenty of functions to operate numbers that way. I wouldn't believe that would make the code simpler, but in fact it really did, see: <pre class="src"> CL-USER> #b111111111 511 CL-USER> (logcount #b111111111) 9 CL-USER> (logcount 511) 9 CL-USER> (logbitp 3 #b100100100) NIL CL-USER> (logbitp 2 #b100100100) T CL-USER> (format nil "~2r" (logxor #b111111111 (ash 1 4))) "111101111" CL-USER> (logbitp 4 (logxor #b111111111 (ash 1 4))) NIL </pre> With that in mind, we can write the following code: <pre class="src"> (defun count-remaining-possible-values (possible-values)

"How many possible values are left in there?" ;; we could raise an empty-values condition if we get 0... (logcount possible-values))

(defun first-set-value (possible-values)

"Return the index of the first set value in POSSIBLE-VALUES." (+ 1 (floor (log possible-values 2))))

(defun only-possible-value-is? (possible-values value)

"Return a generalized boolean which is true when the only value found in POSSIBLE-VALUES is VALUE" (and (logbitp (- value 1) possible-values) (= 1 (logcount possible-values))))

(defun list-all-possible-values (possible-values)

"Return a list of all possible values to explore" (loop for i from 1 to 9 when (logbitp (- i 1) possible-values) collect i))

(defun value-is-set? (possible-values value)

"Return a generalized boolean which is true when given VALUE is possible in POSSIBLE-VALUES" (logbitp (- value 1) possible-values))

(defun unset-possible-value (possible-values value)

"return an integer representing POSSIBLE-VALUES with VALUE unset" (logxor possible-values (ash 1 (- value 1)))) </pre>

You can see here that I was also under the influence of a recent reading about <a href="http://gar1t.com/blog/2012/06/10/solving-embarrassingly-obvious-problems-in-erlang/">making it obvious</a>, or so called <a href="http://dieswaytoofast.blogspot.fr/2012/07/erlang-why-so-many-seemingly-identical.html">intentional programming</a>, following what <a href="http://armstrongonsoftware.blogspot.fr/">Joe Armstrong</a> has to say about it: <blockquote> Intentional programming is a name I give to a style of programming where the reader of a program can easily see what the programmer intended by their code. The intention of the code should be obvious from the names of the functions involved and not be inferred by analysing the structure of the code. (Reading the code should) precisely expresses the intention of the programmer—here no guesswork or program analysis is involved, we clearly read what was intended. </blockquote> So there we go with function names such as count-remaining-possible-values, that will help when reading some more complex code, as in the following, the meat of the solution: <pre class="src"> (defmethod eliminate ((puzzle puzzle) row col value)

"Eliminate given VALUE from possible values in cell ROWxCOL of PUZZLE, and propagate when needed" (with-slots (grid values) puzzle ;; if already unset, work is already done (when (value-is-set? (aref values row col) value) ;; eliminate the value from the set of possible values (let* ((possible-values (unset-possible-value (aref values row col) value))) (setf (aref values row col) possible-values)

;; now if we're left with a single possible value (when (= 1 (count-remaining-possible-values possible-values)) (let ((found-value (first-set-value possible-values))) ;; update the main grid (setf (aref grid row col) found-value)

;; eliminate that value we just found in all peers (eliminate-value-in-peers puzzle row col found-value)))

;; now check if any unit has a single possible place for that value (loop for (r . c) in (list-places-with-single-unit-solution puzzle row col value) do (assign puzzle r c value)))))) </pre>

So that lisp code is quite verbose and at 389 lines almost doubles the 201 lines Norvig had. When clarity is part of the goal, that's hard to avoid, I hope I made a good case that this is not due to lisp being overly verbose by itself. <h3>Comments on the development environment</h3> Or why I even considered Common Lisp as an interesting language for that kind of exercise, and some more. I'll have to tell about re-sharding data live with 16 threads and 256 databases, all in CL, someday. So I've been doing some Emacs Lisp development for a while now, and the part that makes that so much fun is the instant reward. You write some code in your editor, type a key chord (usually, that's

C-M-x runs the command
eval-defun

) and your code is loaded up, ready to be tested. In Emacs Lisp the test can be simply using your editor and watching the new behavior taking place, or playing in the M-x ielm console. When the code is not ready, it crashes, and you're left in the interactive debugger, where you can use C-x C-e runs the command eval-last-sexp to evaluate any expression in your source and see its value in the current debug frame. That way of working is a huge productivity boost, that I've been missing much when getting back to writing C code for PostgreSQL. I can't C-M-x the current function and go write some SQL to test it right away, I have to compile the whole source tree, then install the new binaries, then restart the test server and then open up a psql console to interact with the new code. Of course I could just make check and watch the results, but then if I attach a debugger it complains that the code on-disk is more recent than the code in the core dump. What if you want Emacs Lisp integrated facilities and something made for general programming rather than suited to building a text editor? Don't get me wrong, you can probably find more production ready code in elisp than in many other languages, just because Emacs has been there for about 35 years. Editor targeted production code, though. This integrated development cycle is all the same when you're using Common Lisp. The awesome <a href="http://common-lisp.net/project/slime/">Superior Lisp Interaction Mode for Emacs</a> is providing exactly that experience. Just run M-x slime and then as you define your code you can C-M-x the function at point, see the compilation errors and warnings if any in the associated REPL, and just try your code. I tend to mostly play in the command line, it's possible to just use C-x C-e while typing too. <h3>Performances</h3> Of course we do care! After all the original article came with a quite detailed performance analysis with graphs and all. I won't be reproducing that, sorry. I'll just show you what penalty you get for using an older language specification, much more dynamic and with more features than python, and with a great, scratch that, awesome development environment. Oh wait, that's the other way round, no penalty, it's actually so much faster! <h4>Python version perfs</h4> The results I got on my desktop machine are about twice as fast as in the original article, I guess newer machines and newer python have something to say for that: <pre class="src">

dim ~/dev/CL/sudoku python sudoku.dim.py All tests pass. Solved 50 of 50 easy puzzles (avg 0.01 secs (151 Hz), max 0.01 secs). Solved 95 of 95 hard puzzles (avg 0.02 secs (42 Hz), max 0.12 secs). Solved 11 of 11 hardest puzzles (avg 0.01 secs (115 Hz), max 0.01 secs). </pre>

That makes an average of

(50*151 + 95*42 + 11*115) / (50+95+11) =
82Hz

. That seems pretty good, let's continue. As you can see I've cut away the random puzzle part, that's because I was too lazy to implement that part, which didn't seem all that interesting to me. If you think that's a problem and need solving, I accept patches. <h4>Common lisp version perfs</h4> When using <a href="http://sbcl.org/">SBCL</a> on the same machine, what I got was: <pre class="src">

(sudoku:solve-example-grids) Solved 50 of 50 easy puzzles (avg .0021 sec (471.7 Hz), max 0.015 secs). Solved 95 of 95 hard puzzles (avg .0022 sec (446.0 Hz), max 0.008 secs). Solved 11 of 11 hardest puzzles (avg .0018 sec (550.0 Hz), max 0.003 secs). </pre>

With the same way to compute the average, we now have 461.6Hz. Now, that's between 3 times and more than 10 times faster than the python version (taken collection per collection), for a comparable effort, a much better development environment, and the same all dynamic no explicit compiling approach. <h3>Conclusion</h3> I guess I'm fond of Common Lisp, which I already saw coming (so did you, right?), and now I have some public article and code to share about why :) The code is hosted at <a href="https://github.com/dimitri/sudoku">https://github.com/dimitri/sudoku</a> if you're interested, with the necessary files to reproduce, some docs, etc. Also, apart from using integers as bitfields, which I did more for being lispy than for performances, I did very little effort for optimizing the code. It's quite naive in this respect, yet allow me an average of 461.6Hz rather than 82Hz, that's 5.6 times faster average. So yes, I will continue to invest some precious time in Common Lisp as a very good interactive scripting language, and maybe more than that. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 10 Jul 2012 20:37:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/07/10-solving-sudoku.html</guid> </item> <item> <title>PGDay France 2012</title> <link>http://tapoueh.org/blog/2012/06/08-pgdayfr-lyon.html</link> <description><![CDATA[The french PostgreSQL Conference, <a href="http://www.pgday.fr/programme">pgday.fr</a>, was yesterday in Lyon. We had a very good time and a great schedule with a single track packed with 7 talks, addressing a diverse set of PostgreSQL related topics, from GIS to fuzzy logic, including replication.

You might have guessed it already, I did talk about replication. Here's the slide deck I did use, it's in french, sorry if you don't grok that language. <center> <a class="image-link" href="../../../images/confs/PGDay_2012_Replications.pdf"> <img src="../../../images/confs/PGDay_2012_Replications.png"></a> </center> The conference was very nice and did go smoothly, even if we were “only” 60 of us I had the pleasure to meet with different users with very different set of needs. Very happy to have been there! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 08 Jun 2012 16:17:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/06/08-pgdayfr-lyon.html</guid> </item> <item> <title>M-x recompile</title> <link>http://tapoueh.org/blog/2012/06/01-emacs-recompile.html</link> <description><![CDATA[A friend of mine just asked me for advice to tweak some Emacs features, and I think that's really typical of using Emacs: rather than getting used to the way things are shipped to you, when using Emacs, you start wanting to adapt the tools to the way you want things to be working instead. And you can call that the awesome!

In this case we're talking about the M-x compile and M-x recompile functions. My friend bound the former to <f11> and wanted that C-u f11 do a recompile with the exact same command line as the previous compile command. Well, to be honest, I didn't know about M-x recompile until after I wrote the following function, made to trigger another compile with the last command used if using C-u. <pre class="src"> (defvar cyb-compile-last-command nil) (defvar cyb-compile-command-history nil)

(defun cyb-compile (arg)

"Compile with given command, optionally recompile with last command" (interactive "P") (if arg (progn ;; arg given: compile with last command (unless cyb-compile-last-command (error "Can't recompile yet, no known last command")) (compile cyb-compile-last-command)) ;; else branch, no arg given, ask for a command (let ((command (read-string "Compile with command: " "make -k" 'cyb-compile-command-history "make -k"))) (setq cyb-compile-last-command command) (compile command))))

(global-set-key (kbd "<f11>") 'cyb-compile) </pre>

With that little Emacs Lisp code we're driving Emacs the way we want to be working, and that's great! You can see it was a quick hack in that if you wanted to use the function non interactively it would still prompt for the command to use to compile, when Emacs Lisp interactive special form would allow us to implement something way smarter here. Also if we wanted to spend some more time on that feature, we should probably tweak the error condition to be asking for the command rather than just complaining, that would certainly be more useful. Exercise left to the reader, rewrite using recompile rather than reinventing it in a hurry! Beware of call-interactively though. Oh and fix the aforementioned infelicities, too. To conclude, we see that writing Emacs Lisp code to fix a usability problem in a hurry is a great force of Emacs, and that we're provided with the necessary tool set so as to be able to reach completeness if we wanted to do so. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 01 Jun 2012 18:45:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/06/01-emacs-recompile.html</guid> </item> <item> <title>Back From PgCon</title> <link>http://tapoueh.org/blog/2012/05/24-back-from-pgcon.html</link> <description><![CDATA[Last week was the annual PostgreSQL Hackers gathering in Canada, thanks to the awesome <a href="http://www.pgcon.org/">pgcon</a> conference. This year's issue has been packed with good things, beginning with the <a href="http://wiki.postgresql.org/wiki/PgCon2012CanadaClusterSummit">Cluster Summit</a> then followed the next day by the <a href="http://wiki.postgresql.org/wiki/PgCon_2012_Developer_Meeting">Developer Meeting</a> just followed (yes, in the same day) with the <a href="http://wiki.postgresql.org/wiki/PgCon2012CanadaInCoreReplicationMeeting">In Core Replication Meeting</a>. That was a packed shedule!

<center> <img src="../../../images/in-core-replication.jpg" alt=""> </center> The in core replication project has been presented with slides titled <a href="http://wiki.postgresql.org/images/7/75/BDR_Presentation_PGCon2012.pdf">Future In-Core Replication for PostgreSQL</a> and got a very good reception. For instance, people implementing <a href="http://slony.info/">Slony</a> (Jan Wieck, Christopher Browne and Steve Singer where here) appreciated the concepts here and where rather supportive of both the requirements and the design, and appreciated the very early demo and results that we had to show already, as a kind of a proof of concepts. After those first two days, we could start the actual show. I had the honnor to present a migration use case entitled <a href="http://www.pgcon.org/2012/schedule/events/431.en.html">Large Scale MySQL Migration</a> where we're speaking about going from MySQL to PostgreSQL, from 37 to 256 shards, moving more than 6TB of data including binary blobs that we had to process with pl/java. A quite involved migration project whose slides you now can read here: <center> <a class="image-link" href="../../../images/fotolog.pdf"> <img src="../../../images/fotolog.jpg"></a> </center> I've heard that we should soon be able to enjoy audio and video recordings of the sessions, so if you couldn't make it this year for any reason, don't miss that, you will have loads of very interesting talks to virtually attend. I definitely will do that to catch-up with some talks I couldn't attend, having to pick one out of three is not an easy task, all the more when you add the providential hallway track. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 24 May 2012 09:40:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/05/24-back-from-pgcon.html</guid> </item> <item> <title>Clean PGQ Subconsumers</title> <link>http://tapoueh.org/blog/2012/04/26-unregister-subconsumers.html</link> <description><![CDATA[Now that you're all using the wonders of <a href="../03/12-PGQ-Cooperative-Consumers.html">Cooperative Consumers</a> to help you efficiently and reliably implement your business constraints and offload them from the main user transactions, you're reaching a point where you have to clean up your development environment (because that's what happens to development environments, right?), and you want a way to start again from a clean empty place.

<center> <img src="../../../images/drop-queue.png" alt=""> </center> Here we go. It used to be much more simple than that, so if you're still using PGQ from Skytools2, just jump to the next step. <h3>Unregister Subconsumers</h3> That query will figure out subconsumers in the system function pgq.get_consumer_info() and ask PGQ to please unregister them, losing events in the way, even events from batches that are currently active. <pre class="src">

with subconsumers as ( select q1.queue_name, q2.consumer_name, substring(q1.consumer_name from '%.#"%#"' for '#') as subconsumer_name from (select * from pgq.get_consumer_info() where lag is null) as q1 join (select * from pgq.get_consumer_info() where lag is not null) as q2 on q1.queue_name = q2.queue_name ) select , pgq_coop.unregister_subconsumer(queue_name, consumer_name, subconsumer_name, 1) from subconsumers; </pre>

<h3>Unregister Consumers</h3> Now that the first step is done, we have to unregister the main consumers, which is easy and what you already did before: <pre class="src"> select queue_name, consumer_name,
pgq.unregister_consumer(queue_name, consumer_name) from pgq.get_consumer_info(); </pre>
<h3>Drop queues</h3> And as we want to really clean up the mess, let's also drop the queues. <pre class="src"> select queue_name, pgq.drop_queue(queue_name)

from pgq.queue; </pre> ]]></description> <author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 26 Apr 2012 15:05:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/04/26-unregister-subconsumers.html</guid> </item> <item> <title>PGQ Coop Consumers</title> <link>http://tapoueh.org/blog/2012/03/12-PGQ-Cooperative-Consumers.html</link> <description><![CDATA[While working a new <a href="http://www.postgresql.org/">PostgreSQL</a> architecture for an high scale project that used to be in the top 10 of internet popular web sites (in terms of visitors), I needed to be able to off load some processing from the main path: that's called a batch job. This needs to be transactional: don't run the job if we did rollback; the transaction, process all events that were part of the same transaction in the same transaction, etc.

<center> <img src="../../../images/workers.jpg" alt=""> </center> That calls for using <a href="http://wiki.postgresql.org/wiki/PGQ_Tutorial">PGQ</a>, the jobs queue solution from <a href="http://wiki.postgresql.org/wiki/Skytools">Skytools</a>, the power horse for <a href="http://wiki.postgresql.org/wiki/Londiste_Tutorial">Londiste</a>. If PGQ is good enough to build a full trigger-based replication solution on top of it, certainly it's good enough for our custom processing, right? Well, you still need to check that your expectations are met, and that was happily the case in my implementation. It's a very common problem, and PGQ very often is a great solution to it. As this implementation is PHP centric, we've been using <a href="https://github.com/dimitri/libphp-pgq">libphp-pgq</a> to drive our background workers. Using PGQ in PHP has been very easy to setup, the only trap being not to forget about running the ticker process. It got interesting because of two elements. First, we're nor running a single database instance here but a bunch of them... make it 256 databases. Each of them having 5 queues to consume, that would be about 1280 consumer processes, distributed on 16 servers that's still 80 per server, so way too many. What we did instead is reuse the <a href="https://github.com/markokr/skytools/blob/master/scripts/queue_mover.py">queue mover</a> script found in the Skytools distribution and adapt it to forward the event of the 1280 source queues to only 5 destination queues. We then process the events from this single location. Now it's easier to deal with, but we're not still exactly there. Of course, with so many sources, concentrating them all into the same place means that a single consumer is not able to process the events as fast as they are produced. That's where the cooperative consuming shines, it's very easy to turn your consumer into a cooperative one even on an existing and running queue, and that's what we did. So now we can choose how many workers we want per queue: one of them has 4 workers, another one see not so much activity and 1 worker still fits. <center> <img src="../../../images/coop-workers.jpeg" alt=""> </center> The queue mover script that knows how to subscribe to many queues from the same process is going to be contributed to Skytools proper, of course. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 12 Mar 2012 14:43:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/03/12-PGQ-Cooperative-Consumers.html</guid> </item> <item> <title>Extension White Listing</title> <link>http://tapoueh.org/blog/2012/03/08-extension-white-listing.html</link> <description><![CDATA[PostgreSQL 9.1 includes proper extension support, as you might well know if you ever read this very blog here. Some hosting facilities are playing with PostgreSQL at big scale (hello <a href="https://postgres.heroku.com/blog">Heroku</a>!) and still meet with small caveats making their life uneasy.

To be specific, only superusers are allowed to install C coded stored procedures, and that impacts a lot of very useful PostgreSQL extension: all those shiped in the contrib package are coded in C. Now, <a href="https://postgres.heroku.com/blog">Heroku</a> is not giving away superuser access to their hosted customers in order to limit the number of ways they can shoot themselves in the foot. And given PostgreSQL security model, being granted database owner is mostly good enough for day to day operation. <blockquote> See Andrew's article <a href="http://people.planetpostgresql.org/andrew/index.php?/archives/259-Heroku,-a-really-easy-way-to-get-a-database-in-a-hurry..html">Heroku, a really easy way to get a database in a hurry</a> for more context about Heroku's offering here. </blockquote> Mostly, but as we see, not completely good enough. How to arrange for a non superuser to be able to still install a C-coded extension in his own database? That's quite dangerous as any bug causing a crash would mean a PostgreSQL whole restart. So you not only want to empower CREATE EXTENSION to database owners, you also want to be able to review and explicitely white list the allowed extensions. Here we go: <a href="https://github.com/dimitri/pgextwlist">pgextwlist</a> is a PostgreSQL extensions implementing just that idea. You have to tweak local_preload_libraries so that it gets loaded automatically and early enough, and you have to provide for the list of authorized extensions in the extwlist.extensions setting. Let's see a usage example, straight from the documentation: <pre class="src"> dim=> select rolsuper from pg_roles where rolname = current_user; select rolsuper from pg_roles where rolname = current_user;

rolsuper ———- f (1 row)

dim=> create extension hstore; create extension hstore; WARNING: > is deprecated as an operator name DETAIL: This name may be disallowed altogether in future versions of PostgreSQL. CREATE EXTENSION dim> create extension earthdistance; create extension earthdistance; ERROR: extension "earthdistance" is not whitelisted DETAIL: Installing the extension "earthdistance" failed, because it is not

on the whitelist of user-installable extensions. HINT: Your system administrator has allowed users to install certain extensions. SHOW extwlist.extensions;

dim=> \dx \dx

List of installed extensions

Name Version Schema Description

———+———+————+—————————————————

 hstore 1.0 public data type for storing sets of (key, value) pairs

plpgsql 1.0 pg_catalog PL/pgSQL procedural language

(2 rows)

dim=> drop extension hstore; drop extension hstore; DROP EXTENSION </pre>
As you can see, it allows non superusers to install an extension written in C. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 08 Mar 2012 14:25:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/03/08-extension-white-listing.html</guid> </item> <item> <title>Battle Language à la Marmite</title> <link>http://tapoueh.org/blog/2012/03/01-duchessfr-battle-language.html</link> <description><![CDATA[J'ai eu la chance hier soir de participer à la <a href="http://jduchess.org/duchess-france/blog/battle-language-a-la-marmite/">Battle Language à la Marmite</a>, où j'avais proposé de parler de <a href="http://www.emacswiki.org/emacs/EmacsLisp">Emacs Lisp</a>, proposition qui s'est transformée en porte-étendard de la grande famille <a href="http://www.lisp.org/index.html">Lisp</a>. J'ai utilisé avec plaisir certains contenu de <a href="http://www.lisperati.com/">Lisperati</a> dans ma présentation et je vous recommande le détour sur ce site !

<center> <a class="image-link" href="../../../images/confs/elisp.pdf"> <img src="../../../images/confs/elisp-1.png"></a> </center> J'ai dans cette présentation très rapide (5 minutes seulement) mentionné l'approche axiomatique de John McCarthy lorsqu'il a découvert le language, on peut en lire un peu plus sur le site de Paul Graham et son article <a href="http://www.paulgraham.com/rootsoflisp.html">The Roots of Lisp</a> et le code associé, une <a href="http://lib.store.yahoo.net/lib/paulgraham/jmc.lisp">implémentation du LISP de McCarthy en common lisp</a>. Merci à <a href="http://jduchess.org/">Duchess</a> pour une bonne soirée où nous avons pu échanger nos points de vue et débattre des languages fonctionnels et objects, des différences entre Erlang et Haskell et Ruby, et de quelques autres sujets dérivés ! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 01 Mar 2012 14:49:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2012/03/01-duchessfr-battle-language.html</guid> </item> <item> <title>pgbouncer munin plugin</title> <link>http://tapoueh.org/blog/2011/11/16-pgbouncer-munin.html</link> <description><![CDATA[It seems that if you search for a <a href="http://munin-monitoring.org/">munin</a> plugin for <a href="http://wiki.postgresql.org/wiki/PgBouncer">pgbouncer</a> it's easy enough to reach an old page of mine with an old version of my plugin, and a broken link. Let's remedy that by publishing here the newer version of the plugin. To be honest, I though it already made its way into the official munin 1.4 set of plugins, but I've not been following closely enough.

<center> <img src="../../../images/bouncing_elephant.gif" alt=""> </center> As the plugin is 300 lines of python code, it's not a good idea to just inline it here, so please grab it at <a href="../../../resources/pgbouncer_">pgbouncer_</a>. You might need to know that the script name once installed should follow the form pgbouncer_dbname_stats_requests or pgbouncer_dbname_pools, where of course dbname can contain any number of _ characters. This script supports quite old versions of pgbouncer that didn't accept the normal pq protocol, you did have to use psql to have any chance of getting the data from a script, you couldn't then just use a PostgreSQL driver such as <a href="http://initd.org/psycopg/">psycopg2</a>. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 16 Nov 2011 14:00:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/11/16-pgbouncer-munin.html</guid> </item> <item> <title>Extensions en simple SQL</title> <link>http://tapoueh.org/blog/2011/10/31-extensions-sql.html</link> <description><![CDATA[La <a href="http://2011.pgconf.eu/">conférence européenne à Amsterdam</a> était un très bon évènement de la communauté, avec une organisation impeccable dans un hôtel accueillant. J'ai eu le plaisir d'y parler des extensions et de leur usage dans le cadre du développement applicatif « interne », sous le titre <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/138-extensions-are-good-for-business-logic/">Extensions are good for business logic</a>.

<center> <a class="image-link" href="http://wiki.postgresql.org/images/f/f1/Using-extensions.pdf"> <img src="../../../images/using-extensions-10.png"></a> </center> L'idée de ma présentation, que la plupart d'entre vous a loupé je suppose (en tout cas je n'avais qu'une petite poignée de français dans la salle, et j'espère avoir des lecteurs qui n'étaient pas à Amsterdam), l'idée est d'utiliser les mécanismes offerts par les extensions afin de maintenir le code PL que vous utilisez en production. Il s'agit la plupart du temps de procédures qui implémentent une partie de la logique métier de vos applications, mais si proche des données que cela termine en base directement : c'est une bonne chose, en particulier depuis PostgreSQL 9.1. Cette version propose en effet une gestion assez complète des extensions. Il s'agit de réaliser un empaquetage de vos procédures en suivant la documentation en ligne et son chapitre <a href="http://docs.postgresqlfr.org/9.1/extend-extensions.html">35.15. Empaqueter des objets dans une extension</a>. Une fois cela fait, il est alors possible de déployer votre ensemble de procédure stockée avec la commande CREATE EXTENSION mesprocs;, et ensuite la commande psql \dx vous permet de lister les extensions installées et leur numéro de version. Les mises à jours sont également gérées avec une commande SQL dédiée, il s'agit alors de ALTER EXTENSION mesprocs UPDATE [TO version];. Il suffit de fournir des scripts intermédiaires nommés par exemple mesprocs--1.0--1.1.sql et mesprocs--1.1--1.2.sql et PostgreSQL saura comment passer de 1.0 à 1.1. Voilà, vous savez presque tout de ma présentation à Amsterdam et vous pouvez retrouver le reste sur le support proposé en début d'article. Bien sûr je n'ai pas reproduit ici les questions intéressantes qui m'ont été posées, une bonne partie d'entre elles sont venues enrichir ma liste de Noël pour les extensions. Si vous voulez être sûr de trouver cela sous votre sapin, cependant, le meilleur moyen est encore de m'en parler : sponsoriser les développement Open Source est une belle démarche :) ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 31 Oct 2011 14:22:00 +0100</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/10/31-extensions-sql.html</guid> </item> <item> <title>Back From Amsterdam</title> <link>http://tapoueh.org/blog/2011/10/26-back-from-amsterdam.html</link> <description><![CDATA[Another great conference took place last week, <a href="http://2011.pgconf.eu/">PostgreSQL Conference Europe 2011</a> was in Amsterdam and plenty of us PostgreSQL geeks were too. I attended to lot of talks and did learn some more about our project, its community and its features, but more than that it was a perfect occasion to meet with the community.

<center> <img src="../../../images/ams-conf-room.jpg" alt=""> </center> <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/2-dave-page/">Dave Page</a> talked about SQL/MED under the title <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/146-postgresql-at-the-center-of-your-dataverse/">PostgreSQL at the center of your dataverse</a> and detailed what to expert from a Foreign Data Wrapper in PostgreSQL 9.1, then how to write your own. Wherever you are currently managing your data, you can easily enough make it so that PostgreSQL integrates them by means of fetching them to answer your queries. Which means real time data federating: you don't copy data around, you remote access them when executing the query. I might need to come up with new Foreign Data Wrappers in a not too distant future, now that I better grasp how much work it really is to do that, it appears to be a good migration strategy too: <pre class="src">

INSERT INTO real.table SELECT FROM foreign.table; </pre>

Another discovery is that apparently <a href="http://code.google.com/p/plv8js/wiki/PLV8">PLv8</a> is ready for public consumption. Using it can lead to <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/174-heralding-the-death-of-nosql/">Heralding the Death of NoSQL</a>, so use it with care. In the presentation of <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/156-synchronous-replication-and-durability-tuning/">Synchronous Replication and Durability Tuning</a> we mainly saw that mixing synchronous and asynchronous transactions in your application is the key to real performances across the ocean, as the speed of the light is not infinite. From Baltimore to Amsterdam the latency can not be better than 100ms and that's not the same as instant nowadays. Then again, depending on the number of concurrent queries to sync over the ocean link, the experimental setup was able to achieve several thousands of queries per second, which is validating the model we picked for sync rep and its implementation. If you want to read the slides again at home, or if you could not be there for some reason, then most of the talks are now available online at the <a href="http://wiki.postgresql.org/wiki/PostgreSQL_Conference_Europe_Talks_2011">PostgreSQL Conference Europe Talks 2011</a> wiki page. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 26 Oct 2011 10:08:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/10/26-back-from-amsterdam.html</guid> </item> <item> <title>Implementing backups</title> <link>http://tapoueh.org/blog/2011/10/12-backup-strategy.html</link> <description><![CDATA[I've been asked about my opinion on backup strategy and best practices, and it so happens that I have some kind of an opinion on the matter.

I tend to think best practice here begins with defining properly the backup plan you want to implement. It's quite a complex matter, so be sure to ask yourself about your needs: what do you want to be protected from? <center> <img src="../../../images/online-backup.jpg" alt=""> </center> The two main things to want to protect from are hardware loss (crash disaster, plane in the data center, fire, water flood, etc) and human error (UPDATE without a where clause). Replication is an answer to the former, archiving and dumps to the latter. You generally need both. Often enough “backups” include WAL archiving and shipping and nightly or weekly base backups, with some retention and some scripts or procedures ready to setup <a href="http://www.postgresql.org/docs/9.1/static/continuous-archiving.html">Point In Time Recovery</a> and recover some data without interfering with the WAL archiving and shipping. Of course with PostgreSQL 9.0 and 9.1, the WAL Shipping can be implemented with streaming replication and you can even have a Hot Standby. But for backups you still want archiving. Mostly I still implement pg_dump -Fc nightly backups with a custom retention (for example, 1 backup a month kept 2 years, 1 backup a week kept 6 or 12 months, 1 backup a night kept 1 to 2 weeks), when the database size allow the pg_dump run to remain constrained in the maintenance window, if any. Don't forget that while pg_dump runs, you can't roll out DDL changes to the production system any more, so you want to be careful about this maintenance window thing. When you have one. Physical backups are not locking rollouts away, but they often suck a good deal of the IO bandwidth so you need to pick up a right timing to do them. That's how you can get to once a week base backup and WAL archiving. If you can't pg_dump production, maybe you can have automated restore jobs from the physical backups that you then pg_dump -Fc, so that you still have that. That can come up handy, really: you can't test your major upgrade out of a physical backup. Also, obviously, never consider your backup strategy implemented until you have either automated restores in place or a regular schedule to exercise them (staging instances, devel instances). Then as far as the practical tools go, I tend to think that <a href="http://tapoueh.org/pgsql/pgstaging.html">pg_staging</a> is worth its installation complexity, and for WAL archiving and base backup I recommend <a href="http://skytools.projects.postgresql.org/doc/walmgr.html">walmgr</a> from <a href="http://wiki.postgresql.org/wiki/SkyTools">Skytools</a>, that's a very handy tool. When using PostgreSQL 9.0 or 9.1, consider using <a href="http://packages.debian.org/experimental/skytools3-walmgr">walmgr3</a> so that it's behaving nice alongside streaming replication. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 12 Oct 2011 22:22:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/10/12-backup-strategy.html</guid> </item> <item> <title>Extensions, applications</title> <link>http://tapoueh.org/blog/2011/10/10-extensions-applicatives.html</link> <description><![CDATA[La <a href="http://2011.pgconf.eu/">conférence PostgreSQL annuelle en Europe</a> a lieu la semaine prochaine à Amsterdam, et j'espère que vous avez déjà vos billets, car cette édition s'annonce comme un très bon millésime !

Je présenterai donc comment utiliser les extensions, le titre en anglais est <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/138-extensions-are-good-for-business-logic/">Extensions are good for business logic</a>, et l'idée est de voir comment exploiter les extensions afin de mieux gérer vos mises à jours en bases de données. Le cycle de vie des bases de données en production inclue souvent l'utilisation d'une base de développement où le schéma évolue au rythme des besoins des développeurs, et de temps en temps on consolide une partie de ces modifications (dans des rollouts, scripts contenant principalement des DDL) afin de les déployer en production — si possible avec une étape intermédiaire en préproduction, tout de même. Savoir ce qui est déployé en développement et comment en retirer le script à jouer en production peut être parfois fastidieu. Quand ce n'est pas le cas, c'est que le travail a été fait en amont, ce qui est le signe d'une bonne organisation, avec les surcoûts que l'on peut imaginer. Les <a href="http://www.postgresql.org/docs/9.1/static/extend-extensions.html">extensions</a> telles que présentes dans PostgreSQL 9.1 vous permettent de mieux gérer ce genre de cas, en optimisant le surcoût : il ne disparaît pas, mais devient opérationnel plutôt que de rester une charge d'organisation. Allez, je vous laisse maintenant, je dois me préparer pour la conférence :) ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 10 Oct 2011 10:35:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/10/10-extensions-applicatives.html</guid> </item> <item> <title>Scaling Stored Procedures</title> <link>http://tapoueh.org/blog/2011/10/06-scaling-with-stored-procedures.html</link> <description><![CDATA[In the news recently stored procedures where used as an excuse for moving away logic from the database layer to application layer, and to migrate away from a powerful technology to a simpler one, now that there's no logic anymore in the database.

It's not the way I would typically approach scaling problems, and apparently I'm not alone on the Stored Procedures camp. Did you read this nice blog post <a href="http://ora-00001.blogspot.com/2011/07/mythbusters-stored-procedures-edition.html">Mythbusters: Stored Procedures Edition</a> already? Well it happens in another land that where my comfort zone is, but still has some interesting things to say. I won't try and address all of the myths they attack in a single article. Let's pick the scalability problems, the two of them I think about are code management and performances. We are quite well equiped for that in PostgreSQL, really. For code maintainance we now have <a href="http://www.postgresql.org/docs/9.1/static/extend-extensions.html">PostgreSQL Extensions</a>, which allows you to pack all your procedures into separate extensions, and to maintain a version number and upgrade procedures for each of them. You can handle separate rollouts in development for going from 1.12 to 1.13 then 1.14 and after the developers tested it more completely and changed their mind again on the best API they want to work with, 1.15 which is stamped ok for production. At this point, ALTER EXTENSION UPGRADE will happily apply all the rollouts in sequence to upgrade from 1.12 straight to 1.15 in one go. And if you prefer to bake a special careful script to handle that big jump, you also can provide a specific extension--1.12--1.15.sql script. Of course you're managing all those files with your favorite SCM, to answer to some other myth from the blog reference we are loosely following. <center> <a class="image-link" href="http://postgresqlrussia.org/articles/view/131"> <img src="../../../images/Moskva_DB_Tools.v3.png"></a> </center> I wanted to talk about the other side of the scalability problem, which is the operations side of it. What happens when you need to scale the database in terms of its size and level of concurrent activity? PostgreSQL earned a very good reputation at being able to scale-up, what about scaling-out? Certainly, now that you're all down into Stored Procedure, it's going to be a very bad situation? Well, in fact, you're then in a very good position here, thanks to <a href="http://wiki.postgresql.org/wiki/PL/Proxy">PLproxy</a>. This extension is a custom procedural language whose job is to handle a cluster of database shards that all expose the same PL API, and it's very good at doing that. Stored Procedures are a very good tool to have, be sure to get comfortable enough with them so that you can choose exactly when to use them. If you're not sure about that, we at <a href="http://www.2ndquadrant.com/">2ndQuadrant</a> will be happy to help you there! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 06 Oct 2011 18:23:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/10/06-scaling-with-stored-procedures.html</guid> </item> <item> <title>See you in Amsterdam</title> <link>http://tapoueh.org/blog/2011/10/04-see-you-in-Amsterdam.html</link> <description><![CDATA[The next <a href="http://2011.pgconf.eu/">PostgreSQL conference</a> is approaching very fast now, I hope you have your ticket already: it's a very promissing event! If you want some help in deciding whether to register or not, just have another look at <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/">the schedule</a>. Pick the talks you want to see. It's hard, given how packed with good ones the schedule is. When you're mind is all set, review the list. Registered?

I'll be presenting another talk about extensions, but this time I've geared up to use cases, with <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/138-extensions-are-good-for-business-logic/">Extensions are good for business logic</a>. The idea is not to talk about how to make PostgreSQL play fair with extensions including at dump and restore times, that's already done and I've been talking only too much about it. The idea this time is to figure out how much you get from this feature. If you ever felt like something is missing in your processes between pushing rollouts in devel environments and refining them as developers are testing and preparing something for the live databases, then we have something for you here. Including how to easily compare state between production and development, but without having to guess or reverse engineer anything. Yeah, extensions are all about getting even more professional! A great tool you'll be happy to master! And now I need to prepare a damn good slide deck, right? See you there! :) ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 04 Oct 2011 14:25:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/10/04-see-you-in-Amsterdam.html</guid> </item> <item> <title>PostgreSQL à Amsterdam</title> <link>http://tapoueh.org/blog/2011/09/27-pgconf-eu.html</link> <description><![CDATA[Dans moins d'un mois se tient la conférence européenne PostgreSQL, <a href="http://2011.pgconf.eu/">pgconf.eu</a>. Il s'agit de quatre jours consacrés à votre SGBD préféré, où vous pourrez rencontrer la communauté européenne, consituée d'utilisateurs, d'entreprises de toutes tailles, de développeurs, de participants en tout genre.

C'est l'endroit où aller pour apprendre comment le projet fonctionne, comprendre les impacts des nouvelles versions sur votre architecture, avoir une discussion technique pointue sur cette fonctionalité que vous voudriez voir arriver dans la prochaine version, ou simplement vous rendre compte de l'énergie formidable qui est insuflée dans ce projet ! Évidemment <a href="http://2ndquadrant.fr/">2ndQuadrant</a> sera de la partie, nous présenterons plusieurs de nos <a href="http://www.2ndquadrant.com/fr/les-fonctionnalites-de-postgresql-91/">contributions PostgreSQL 9.1</a>. Cela commencera avec la formation d'une journée complète de <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/81-greg-smith/">Greg</a>, <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/162-performance-from-start-to-crash/">Performance From Start to Crash</a> : si vous voulez apprendre comment aborder les performances d'un serveur PostgreSQL par le leader international du domaine, auteur du livre <a href="http://www.amazon.fr/Bases-donn%C3%A9es-PostgreSQL-Gregory-Smith/dp/274402483X/ref=sr_1_1?ie=UTF8&qid=1316183931&sr=8-1">Bases de données PostgreSQL 9.0</a>, réservez vite votre place ! Les présentation au format classique commencent le lendemain, et en trois jours la liste des présentation de notre <a href="http://www.2ndquadrant.com/fr/profil-de-lequipe/">équipe 2ndQuadrant</a> est assez copieuse. Voyons cela. Nous commençons avec <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/144-migration-to-postgresql-a-holistic-view/">Migration to PostgreSQL - a holistic view</a> par <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/78-harald-armin-massa/">Harald Armin Massa</a>, qui propose un point de vue intéressant sur les raisons qui retiennent certaines migrations. Ensuite <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/34-gianni-ciolli/">Gianni Ciolli</a> présentera <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/159-look-out-the-window-functions-and-free-your-sql/">Look Out The Window Functions (and free your SQL)</a> ou comment résoudre simplement des problèmes complexes lorsque l'on dispose d'outils avancés. Une autre présentation à ne pas rater, <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/156-synchronous-replication-and-durability-tuning/">Synchronous Replication and Durability Tuning</a> détaille comment profiter au mieux de PostgreSQL 9.1 afin d'obtenir les garanties de durabilité des données souhaitées dans votre application. Et cette présentation est animée par <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/81-greg-smith/">Greg Smith</a> et <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/17-simon-riggs/">Simon Riggs</a>. Ce dernier a développé la réplication synchrone, et Hot Standby avant cela. Vous ne trouverez personne au monde mieux placé pour faire cette présentation ! Les deux prochaines présentation de nos <a href="http://expert-postgresql.fr/">experts PostgreSQL</a>, en continuant notre lecture du <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/">programme de pgconf.eu</a> dans l'ordre, ont lieu au même moment. Le choix ne sera pas facile entre <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/158-improving-vacuum-suction/">Improving VACUUM Suction</a> par Greg à nouveau, et une comparaison de <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/183-londiste-3-et-slony-21/">londiste 3 et slony 2.1</a> par <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/57-cedric-villemain/">Cédric Villemain</a>, en français. À suivre, <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/138-extensions-are-good-for-business-logic/">Extensions are good for business logic</a> que je vous présenterai moi-même, vous pouvez voir ma présentation sur la fiche qui porte mon nom : <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/14-dimitri-fontaine/">Dimitri Fontaine</a>. Il s'agit d'une présentation en anglais qui détaille comment utiliser les extensions dans le cadre de la maintenance de la partie procédures stockées d'une application. Et pour finir le deuxième jour des conférences 2ndQuadrant, vous pourrez apprendre avec Gianni comment <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/160-debugging-complex-sql-queries-with-writable-ctes/">Debugging complex SQL queries with writable CTEs</a>, une fonctionnalité contribuée au projet par un autre consultant <a href="http://www.2ndquadrant.com/fr/contact/">2ndQuadrant</a>, Marko Tiikkaja. Et il reste encore une journée ! Nous ne mentons pas en disant que le programme est complet ! Le dernier jour de la conférence n'est pas le moins intéressant, j'espère que vous aurez su garder un peu d'énergie pour suivre… <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/speaker/17-simon-riggs/">Simon Riggs</a> qui présentera sa vision de la <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/199-postgresql-roadmap/">PostgreSQL Roadmap</a> pour les prochaines années. Ce n'est bien sûr que sa vision personnelle, mais lorsque l'on fait le bilan de ces 7 dernières années de <a href="http://www.2ndquadrant.com/fr/histoire-postgresql/">contributions à PostgreSQL</a>, on voit à quel point son opinion personnelle peut avoir du poids dans le développement du projet. À suivre, la présentation de Greg sur son sujet de prédilection : <a href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/157-bottom-up-database-benchmarking/">Bottom-up Database Benchmarking</a>. Tout ce que vous avez toujours voulu savoir sur les mesures de performances de vos bases de données, sans jamais oser le demander. Quelque chose dans ce style en tout cas :) Bien sûr d'autres présentations sont disponibles et retiendront votre attention, ce billet vous présente seulement celles qui seront données par les <a href="http://expert-postgresql.fr/">experts PostgreSQL</a> de <a href="http://www.2ndquadrant.com/fr/expertise-postgresql/">2ndQuadrant</a>. En vous souhaitant bonne conférence à tous, j'espère avoir le plaisir de vous retrouver à Amsterdam le mois prochain ! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 27 Sep 2011 11:10:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/09/27-pgconf-eu.html</guid> </item> <item> <title>Skytools3: walmgr</title> <link>http://tapoueh.org/blog/2011/09/21-skytools-walmgr-part-1.html</link> <description><![CDATA[Let's begin the <a href="http://wiki.postgresql.org/wiki/SkyTools">Skytools 3</a> documentation effort, which is long overdue. The code is waiting for you over at <a href="https://github.com/markokr/skytools">github</a>, and is stable and working. Why is it still in release candidate status, I hear you asking? Well because it's missing updated documentation.

<a href="http://packages.debian.org/experimental/skytools3-walmgr">WalMgr</a> is the Skytools component that manages WAL shipping for you, and archiving too. It knows how to prepare your master and standby setup, how to take a base backup and push it to the standby's system, how to archive (at the satndby) master's WAL files as they are produced and have the standby restore from this archive. What's new in walmgr from Skytools 3 is its support for Streaming Replication that made its way into PostgreSQL 9.0 and is even more useful in PostgreSQL 9.1 (better monitoring, synchronous replication option). <h2>Getting ready</h2> Now, I'm using debian here, and a build virtual machine where I'm doing the backporting work. As <a href="http://www.postgresql.org/about/news.1349">PostgreSQL 9.1</a> is now out, let's use that. <pre class="src"> :~$ pg_lsclusters Version Cluster Port Status Owner Data directory 8.4 main 5432 online postgres /var/lib/postgresql/8.4/main ... 9.0 main 5433 online postgres /var/lib/postgresql/9.0/main ... 9.1 main 5434 online postgres /var/lib/postgresql/9.1/main ... </pre> After some editing of the configuration files (enabling hot standby and switching pg_hba.conf to trust for the sake of this example), we can see that the cluster is ready to be abused: <pre class="src"> :~$ sudo pg_ctlcluster 9.1 main restart :~$ psql —cluster 9.1/main -U postgres \ -c "select name, setting from pg_settings where name in ('max_wal_senders', 'wal_level')"

name

setting

+————-

max_wal_senders	1
wal_level	hot_standby

(2 rows)

:~$ sudo mkdir -p /etc/walshipping/9.1/main /var/lib/postgresql/walshipping :~$ sudo chown -R postgres:postgres /etc/walshipping /var/lib/postgresql/walshipping

:~$ ssh-keygen -t dsa :~/.ssh$ cp id_dsa.pub authorized_keys :~$ ssh localhost </pre>

So the order of operations is to prepare a standby, then have it restore from the archives, then activate the wal streaming and check that the setup allows the standby to switch back and forth between the streaming and the archives. <h2>Setting walmgr</h2> To prepare the standby, we will do a base backup of the master. That step is handled by walmgr, so we first need to set it up. Here's the sample master.ini file: <pre class="src"> [walmgr] job_name = wal-master logfile = /var/log/postgresql/%(job_name)s.log pidfile = /var/run/postgresql/%(job_name)s.pid use_skylog = 0

master_db = port=5434 dbname=template1 master_data = /var/lib/postgresql/9.1/main/ master_config = /etc/postgresql/9.1/main/postgresql.conf master_bin = /usr/lib/postgresql/9.1/bin

# set this only if you can afford database restarts during setup and stop. master_restart_cmd = pg_ctlcluster 9.1 main restart

slave = 127.0.0.1 slave_config = /etc/walshipping/9.1/main/standby.ini

walmgr_data = /var/lib/postgresql/walshipping/9.1/main completed_wals = %(walmgr_data)s/logs.complete partial_wals = %(walmgr_data)s/logs.partial full_backup = %(walmgr_data)s/data.master config_backup = %(walmgr_data)s/config.backup

# syncdaemon update frequency loop_delay = 10.0 # use record based shipping available since 8.2 use_xlog_functions = 0

# pass -z to rsync, useful on low bandwidth links compression = 0

# keep symlinks for pg_xlog and pg_log keep_symlinks = 1

# tell walmgr to set wal_level to hot_standby during setup #hot_standby = 1 # periodic sync #command_interval = 600 #periodic_command = /var/lib/postgresql/walshipping/periodic.sh </pre>

And the /etc/walshipping/9.1/main/standby.ini companion: <pre class="src"> [walmgr] job_name = wal-standby logfile = /var/log/postgresql/%(job_name)s.log use_skylog = 0

slave_data = /var/lib/postgresql/9.1/standby slave_bin = /usr/lib/postgresql/9.1/bin slave_stop_cmd = pg_ctlcluster 9.1 standby stop slave_start_cmd = pg_ctlcluster 9.1 standby start slave_config_dir = /etc/postgresql/9.1/standby/

backup_datadir = no keep_backups = 0 # archive_command = # primary database connect string for hot standby — enabling # this will cause the slave to be started in hot standby mode. primary_conninfo = host=127.0.0.1 port=5434 user=postgres </pre>

And let's get started: <pre class="src"> :~$ cp standby.ini /etc/walshipping/9.1/main/

:~$ walmgr3 -v master.ini setup 2011-09-21 16:57:05,685 30450 INFO Configuring WAL archiving 2011-09-21 16:57:05,687 30450 DEBUG found 'archive_mode' in config — enabling it 2011-09-21 16:57:05,687 30450 DEBUG found 'wal_level' in config — setting to 'archive' 2011-09-21 16:57:05,688 30450 DEBUG modifying configuration: {'archive_mode': 'on', 'wal_level': 'archive', 'archive_command': '/usr/bin/walmgr3 /var/lib/postgresql/master.ini xarchive %p %f'} 2011-09-21 16:57:05,688 30450 DEBUG found parameter archive_mode with value ''off'' 2011-09-21 16:57:05,690 30450 DEBUG found parameter wal_level with value ''minimal'' 2011-09-21 16:57:05,690 30450 DEBUG found parameter archive_command with value '''' 2011-09-21 16:57:05,691 30450 INFO Restarting postmaster 2011-09-21 16:57:05,691 30450 DEBUG Execute cmd: 'pg_ctlcluster 9.1 main restart' 2011-09-21 16:57:09,404 30450 DEBUG Execute cmd: 'ssh' '-Tn' '-o' 'Batchmode=yes' '-o' 'StrictHostKeyChecking=no' '127.0.0.1' '/usr/bin/walmgr3' '/etc/walshipping/9.1/main/standby.ini' 'setup' 2011-09-21 16:57:09,712 30450 INFO Done

postgres@squeeze64:~$ walmgr3 master.ini backup 2011-09-21 17:00:17,259 30702 INFO Backup lock obtained. 2011-09-21 17:00:17,277 30692 INFO Execute SQL: select pg_start_backup('FullBackup'); [port=5434 dbname=template1] 2011-09-21 17:00:17,791 30712 INFO Removing expired backup directory: /var/lib/postgresql/walshipping/9.1/main/data.master 2011-09-21 17:00:18,200 30692 INFO Checking tablespaces 2011-09-21 17:00:18,202 30692 INFO pg_log does not exist, skipping 2011-09-21 17:00:18,259 30692 INFO Backup conf files from /etc/postgresql/9.1/main 2011-09-21 17:00:18,590 30731 INFO First useful WAL file is: 000000010000000200000092 2011-09-21 17:00:19,901 30759 INFO Backup lock released. 2011-09-21 17:00:19,919 30692 INFO Full backup successful

:~$ walmgr3 /etc/walshipping/9.1/main/standby.ini listbackups

List of backups:

Backup set Timestamp Label First WAL

———————— ———— ———————— data.master 2011-09-21 17:00:17 CEST FullBackup 000000010000000200000092 </pre>

Following articles will show how to manage that archive and how to go from that to an Hot Standby fed by either Streaming Replication or Archives. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 21 Sep 2011 17:21:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/09/21-skytools-walmgr-part-1.html</guid> </item> <item> <title>el-get-3.1</title> <link>http://tapoueh.org/blog/2011/09/16-el-get-3.1.html</link> <description><![CDATA[The <a href="https://github.com/dimitri/el-get">el-get</a> project releases its new stable version, 3.1. This new release fixes bugs, add a host of new recipes (we have 420 of them and counting) and some nice new features too. You really want to upgrade.

<h2>New features</h2> Among the features you will find dependencies management and

M-x
el-get-list-packages

, that you should try as soon as possible. Of course, don't miss M-x el-get-self-update that eases the process somehow. <center> <img src="../../../images/emacs-el-get-list-packages.png" alt=""> </center> This shows the result of M-x el-get-list-packages. The packages that don't have a description are the one from <a href="http://www.emacswiki.org/cgi-bin/wiki?action=index;match=%5C.(el|tar)(%5C.gz)%3F%24">emacswiki</a> that doesn't provide a listing of the filename and the first line of the file (it usually follows the format ;;; filename.el --- description here). As we don't want to mirror the website just to be able to provide descriptions, we just don't have them now. Another nice new feature, contributed by a user that wanted to self-learn <a href="http://www.gnu.org/software/emacs/manual/html_node/elisp/index.html">elisp</a>, is the el-get-user-package-directory support. Just place in there some init-my-package.el files, and when el-get wants to init the my-package package, it will load that file for you. That helps managing your setup, and I'm already using that in my own ~/.emacs.d/ repository. <h2>Upgrading</h2> The upgrading is to be done with some care, though, because you need to edit your packaging setup. The el-get-sources variable used to be both where to setup extra recipes and the list of packages you want to have installed, and several people rightfully insisted that I should change that. I've been slow to be convinced, but there it is, they were right. So now, <a href="http://www.emacswiki.org/emacs/el-get">el-get</a> works from the current status of packages and will init all those packages you have installed. Which means that you just

M-x
el-get-install

a package and don't think about it anymore. If you need to override this behavior, it's still possible to do so by specifying the whole list of packages you want initialized (and installed if necessary) on the (el-get 'sync ...) call. That later setup is useful if you want to share your el-get selection on several machines. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 16 Sep 2011 14:13:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/09/16-el-get-3.1.html</guid> </item> <item> <title>PostgreSQL 9.1</title> <link>http://tapoueh.org/blog/2011/09/19-sortie-de-9.1.html</link> <description><![CDATA[<a href="http://www.postgresql.org/about/news.1349">PostgreSQL 9.1</a> est dans les bacs ! Vous n'avez pas encore cette nouvelle version en production ? Pas encore évalué pourquoi vous devriez envisager de migrer à cette version ? Il existe beaucoup de bonnes raisons de passer à cette version, et peu de pièges.

Nous commençons à lire des articles qui reprennent la nouvelle dans la presse française, et j'ai le plaisir de mentionner celui de <a href="http://www.programmez.com/actualites.php?titre_actu=Sortie-de-PostgreSQL-91-!&id_actu=10190">programmez.com</a> qui annonce « un système d'extensions inégalé ». En tant que développeur des <a href="http://www.postgresql.org/docs/9.1/static/extend-extensions.html">Extensions</a> dans PostgreSQL, je ne peux qu'être non seulement d'accord avec eux, mais aussi flatté :) Bons tests à tous, et bonne mises à jour pour les plus chanceux ! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 14 Sep 2011 10:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/09/19-sortie-de-9.1.html</guid> </item> <item> <title>Éviter les injections SQL</title> <link>http://tapoueh.org/blog/2011/09/07-eviter-les-injections-sql.html</link> <description><![CDATA[Nous avons parlé la dernière fois les règles d'<a href="http://tapoueh.org/blog/2011/08/18-echappements-de-chaine.html">échappement de chaînes</a> avec PostgreSQL, et mentionné qu'utiliser ces techniques afin de protéger les données insérées dans les requêtes SQL n'était pas une bonne idée dans la mesure où PostgreSQL offre une fonctionnalité bien plus adaptée.

Nous faisons face ici à un problème de sécurité très bien décrit dans le billet humoristique de <a href="http://xkcd.com/327/">Little Boby Tables</a>, dont je vous recommande la lecture. L'idée est simple, la mise en place de contre mesure fourmille de pièges subtils, à moins d'utiliser la solution décrite ci-après. <center> <img src="http://imgs.xkcd.com/comics/exploits_of_a_mom.png" alt=""> </center> Lorsque l'on envoie une requête SQL à PostgreSQL, celle-ci contient pêle-mêle un mélange de mots-clés SQL et de données utilisateurs. Dans la requête SELECT colname FROM table WHERE pk = 1234; l'élément 1234 est une donnée fournie à PostgreSQL. Lorsque l'on utilise d'autre types de données, on va parler de litéral, qui peut être ou non décoré. Un exemple ? <pre class="src"> # SELECT 'undecorated literal', pg_typeof('undecoreted literal'),

date 'today', pg_typeof(date 'today');

?column?

pg_typeof

date

pg_typeof

———————+————+————+————

undecorated literal

unknown

2011-09-07

date

(1 row) </pre>

Outre l'aspect types de données (un litéral non décoré est de type unknown jusqu'à ce qu'une opération force son type, c'est ce qui permet d'avoir du polymorphisme dans PostgreSQL), nous voyons ici que PostgreSQL doit faire la différence entre le SQL lui-même et les paramètres qui le composent. Il sait bien sûr faire cela, il suffit d'encadrer les valeurs dans des simples guillemets ou bien d'utiliser la notation dite de <a href="http://docs.postgresqlfr.org/9.0/sql-syntax.html#sql-syntax-dollar-quoting">dollar quoting</a>. Mais si l'on ne prend pas de précautions, l'utilisateur peut terminer la séquence d'échappements depuis le champ de saisie du formulaire… <a href="http://docs.postgresql.fr/9.1/libpq.html">libpq</a> est la librairie standard cliente de PostgreSQL et fourni des API de connexion et propose une fonction <a href="http://docs.postgresql.fr/9.1/libpq-exec.html#libpq-pqexecparams">PGexecParams</a>. Cette fonction expose un mécanisme disponible dans le protocole de communication de PostgreSQL lui-même : il est possible de faire parvenir le SQL et les données qu'il contient dans deux parties différentes du messages plutôt que de les mélanger. Ainsi, le serveur n'a plus du tout à deviner où commencent et où terminent les données dans la requête, il lui suffit de regarder dans le tableau séparé contenant les données quand il en a besoin. Terminées les injections SQL ! Note : cette fonction est exposée dans la plupart des pilotes de connexion, et même en PHP, dont la popularité et l'exposition me poussent à donner une référence plus précise : utilisez <a href="http://fr2.php.net/manual/en/function.pg-query-params.php">pg_query_params</a>, son intérêt n'est pas simplement syntaxique, il va jusque dans la définition des échanges de données entre le client (votre code PHP) et le serveur (PostgreSQL). ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 07 Sep 2011 11:36:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/09/07-eviter-les-injections-sql.html</guid> </item> <item> <title>Éviter les injections SQL</title> <link>http://tapoueh.org/blog/2011/09/07-requete-parametree.html</link> <description><![CDATA[Nous avons parlé la dernière fois les règles d'<a href="http://tapoueh.org/blog/2011/08/18-echappements-de-chaine.html">échappement de chaînes</a> avec PostgreSQL, et mentionné qu'utiliser ces techniques afin de protéger les données insérées dans les requêtes SQL n'était pas une bonne idée dans la mesure où PostgreSQL offre une fonctionnalité bien plus adaptée.

date 'today', pg_typeof(date 'today');

?column?

pg_typeof

date

pg_typeof

———————+————+————+————

undecorated literal

unknown

2011-09-07

date

(1 row) </pre>

Outre l'aspect types de données (un litéral non décoré est de type unknown jusqu'à ce qu'une opération force son type, c'est ce qui permet d'avoir du polymorphisme dans PostgreSQL), nous voyons ici que PostgreSQL doit faire la différence entre le SQL lui-même et les paramètres qui le composent. Il sait bien sûr faire cela, il suffit d'encadrer les valeurs dans des simples guillemets ou bien d'utiliser la notation dite de <a href="http://docs.postgresqlfr.org/9.0/sql-syntax.html#sql-syntax-dollar-quoting">dollar quoting</a>. Mais si l'on ne prend pas de précautions, l'utilisateur peut terminer la séquence d'échappements depuis le champ de saisie du formulaire… <a href="http://docs.postgresql.fr/9.1/libpq.html">libpq</a> est la librairie standard cliente de PostgreSQL et fourni des API de connexion et propose une fonction <a href="http://docs.postgresql.fr/9.1/libpq-exec.html#libpq-pqexecparams">PGexecParams</a>. Cette fonction expose un mécanisme disponible dans le protocole de communication de PostgreSQL lui-même : il est possible de faire parvenir le SQL et les données qu'il contient dans deux parties différentes du messages plutôt que de les mélanger. Ainsi, le serveur n'a plus du tout à deviner où commencent et où terminent les données dans la requête, il lui suffit de regarder dans le tableau séparé contenant les données quand il en a besoin. Terminées les injections SQL ! Note : cette fonction est exposée dans la plupart des pilotes de connexion, et même en PHP, que la popularité et l'exposition me poussent à donner une référence plus précise : utilisez <a href="http://fr2.php.net/manual/en/function.pg-query-params.php">pg_query_params</a>, son intérêt n'est pas simplement syntaxique, il va jusque dans la définition des échanges de données entre le client (votre code PHP) et le serveur (PostgreSQL). ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 07 Sep 2011 11:36:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/09/07-requete-parametree.html</guid> </item> <item> <title>PostgreSQL and debian</title> <link>http://tapoueh.org/blog/2011/09/05-apt-postgresql-org.html</link> <description><![CDATA[After talking about it for a very long time, work finally did begin! I'm talking about the <a href="https://github.com/dimitri/apt.postgresql.org">apt.postgresql.org</a> build system that will allow us, in the long run, to propose debian versions of binary packages for <a href="http://www.postgresql.org/">PostgreSQL</a> and its extensions, compiled for a bunch of debian and ubuntu versions.

We're now thinking to support the i386 and amd64 architectures for lenny, squeeze, wheezy and sid, and also for maverick and natty, maybe oneiric too while at it. It's still the very beginning of the effort, and it was triggered by the decision to move sid to 9.1. While it's a good decision in itself, I still hate to have to pick only one PostgreSQL version per debian stable release when we have all the technical support we need to be able to support all stable releases that upstream is willing to maintain. If you've been living under a rock, or if you couldn't care less about debian choices, the problem here for debian is ensuring security (and fixes) updates for PostgreSQL — they promise they will handle the job just fine in the social contract, and don't want to have to it without support from PostgreSQL if a debian stable release contains a deprecated PostgreSQL version. That opens the door for PostgreSQL community to handle the packaging of its solutions as a service to its debian users. We intend to open with support for 8.4, 9.0 and 9.1, and maybe 8.3 too, as <a href="http://qa.debian.org/developer.php?login=myon">Christoph Berg</a> is doing good progress on this front. See, it's teamwork here! We still have more work to do, and setting up the build environment so that we are able to provide the packages for so much targets will indeed be interesting. Getting there, a step after another. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 05 Sep 2011 17:14:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/09/05-apt-postgresql-org.html</guid> </item> <item> <title>pg_restore -L & pg_staging</title> <link>http://tapoueh.org/blog/2011/08/29-pgstaging-and-pgrestore-listing.html</link> <description><![CDATA[On the <a href="http://archives.postgresql.org/pgsql-hackers">PostgreSQL Hackers</a> mailing lists, <a href="http://people.planetpostgresql.org/andrew/">Andrew Dunstan</a> just proposed some new options for pg_dump and pg_restore to ease our lives. One of the answers was talking about some scripts available to exploit the <a href="http://www.postgresql.org/docs/9.0/static/app-pgrestore.html">pg_restore</a> listing that you play with using options -l and -L, or the long name versions --list and --use-list. The <a href="../../../pgsql/pgstaging.html">pg_staging</a> tool allows you to easily exploit those lists too.

The pg_restore list is just a listing of one object per line of all objects contained into a custom dump, that is one made with pg_dump -Fc. You can then tweak this listing in order to comment out some objects (prepending a ; to the line where you find it), and give your hacked file back to

pg_restore
--use-list

so that it will skip them. What's pretty useful here, among other things, is that a table will have in fact more than one line in the listing. One is for the TABLE definition, another one for the TABLE DATA. So that pg_staging is able to provide you with options for only restoring some schemas, some schemas_nodata and even some tablename_nodata_regexp, to use directly the configuration options names. How to do a very simple exclusion of some table's data when restoring a dump, will you ask me? There we go. Let's first prepare an environment, where I have only a <a href="http://www.postgresql.org/">PostgreSQL</a> server running. <pre class="src"> $ git clone git://github.com/dimitri/pg_staging.git $ git clone git://github.com/dimitri/pgloader.git $ for s in /.sql; do psql -f $s; done $ pg_dump -Fc > pgloader.dump </pre> Now I have a dump with some nearly random SQL objects in it, let's filter out the tables named reformat and parallel from that. We will take the sample setup from the pg_staging project. Going the quick route, we will not even change the default sample database name that's used, which is postgres. After all, the catalog command of pg_staging that we're using here is a developer command, you're supposed to be using pg_staging for a lot more services that just this one. <pre class="src"> $ cp pg_staging/pg_staging.ini . $ (echo "schemas = public";

echo "tablename_nodata_regexp = parallel,reformat") \ >> pg_staging.ini $ echo "catalog postgres pgloader.dump" \

python pg_staging/pg_staging.py -c pg_staging.ini

; Archive created at Mon Aug 29 17:17:49 2011 ; ; [EDITED OUTPUT] ; ; Selected TOC Entries: ; 3; 2615 2200 SCHEMA - public postgres 1864; 0 0 COMMENT - SCHEMA public postgres 1536; 1259 174935 TABLE public parallel dimitri 1537; 1259 174943 TABLE public partial dimitri 1538; 1259 174951 TABLE public reformat dimitri ;1853; 0 174935 TABLE DATA public parallel dimitri 1854; 0 174943 TABLE DATA public partial dimitri ;1855; 0 174951 TABLE DATA public reformat dimitri 1834; 2606 174942 CONSTRAINT public parallel_pkey dimitri 1836; 2606 174950 CONSTRAINT public partial_pkey dimitri 1838; 2606 174955 CONSTRAINT public reformat_pkey dimitri </pre>

We can see that the objects indeed are skipped, now how to really go about the pg_restore is like that: <pre class="src"> $ createdb foo $ echo "catalog postgres pgloader.dump" \

|python pg_staging/pg_staging.py -c pg_staging.ini > short.list $ pg_restore -L short.list -d foo pgloader.dump </pre>

The little bonus with using pg_staging is that when filtering out a schema it will track all tables and triggers from that schema, and also the functions used in the trigger definition. Which is not as easy as it sounds, believe me! The practical use case is when filtering out PGQ and Londiste, then the PGQ triggers will automatically be skipped by pg_staging rather than polluting the pg_restore logs because the CREATE TRIGGER command could not find the necessary implementation procedure. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 29 Aug 2011 18:05:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/29-pgstaging-and-pgrestore-listing.html</guid> </item> <item> <title>Skytools, version 3</title> <link>http://tapoueh.org/blog/2011/08/26-skytools3.html</link> <description><![CDATA[You can find <a href="http://packages.debian.org/source/experimental/skytools3">skytools3</a> in debian experimental already, it's in release candidate status. What's missing is the documentation, so here's an idea: I'm going to make a blog post series about <a href="https://github.com/markokr/skytools">skytools</a> next features, how to use them, what they are good for, etc. This first article of the series will just list what are those new features.

Here are the slides from the <a href="http://www.char11.org/">CHAR(11)</a> talk I made last month, about that very subject: <center> <a class="image-link" href="../../../images/confs/CHAR_2011_Skytools3.pdf"> <img src="../../../images/confs/CHAR_2011_Skytools3.png"></a> </center> The new version comes with a lot of new features. PGQ now is able to duplicate the queue events from one node to the next, so that it's able to manage switching over. To do that we have three types of nodes now, root, branch and leaf. PGQ also supports cooperative consumers, meaning that you can share the processing load among many consumers, or workers. Londiste now benefits from the switch over feature, and is packed with new little features like add <table> --create, the new --trigger-flags argument, and the new --handler thing (to do e.g. partial table replication). Let's not forget the much awaited execute <script> command that allows to include DDL commands into the replication stream, nor the parallel COPY support that will boost your initial setup. walmgr in the new version behaves correctly when using <a href="http://www.postgresql.org">PostgreSQL</a> 9.0. Meaning that as soon as no more WAL files are available in the archives, it returns an error code to the archiver so that the server switches to streaming live from the primary_conninfo, then back to replaying the files from the archive if the connection were to fail, etc. All in all, it just works. Details to follow here, stay tuned! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 26 Aug 2011 21:30:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/26-skytools3.html</guid> </item> <item> <title>pgfincore in debian</title> <link>http://tapoueh.org/blog/2011/08/19-pgfincore-in-debian.html</link> <description><![CDATA[As of pretty recently, <a href="http://villemain.org/projects/pgfincore">pgfincore</a> is now in debian, as you can see on its <a href="http://packages.debian.org/sid/postgresql-9.0-pgfincore">postgresql-9.0-pgfincore</a> page. The reason why it entered the <a href="http://www.debian.org/">debian</a> archives is that it reached the 1.0 release!

Rather than talking about what pgfincore is all about (A set of functions to manage pages in memory from PostgreSQL), I will talk about its packaging and support as a debian package. Here's the first example of a modern multi-version packaging I have to offer. <a href="https://github.com/dimitri/pgfincore/tree/master/debian">pgfincore packaging</a> supports building for 8.4 and 9.0 and 9.1 out of the box, even if the only binary you'll find in debian sid is the 9.0 one, as you can check on the <a href="http://packages.debian.org/source/sid/pgfincore">pgfincore debian source package</a> page. Also, this is the first package I've done properly using the newer version of <a href="http://kitenet.net/~joey/code/debhelper/">debhelper</a>, which make the <a href="https://github.com/dimitri/pgfincore/blob/master/debian/rules">debian/rules</a> file easier than ever. Let's have a look at it: <pre class="src"> SRCDIR = $(CURDIR) TARGET = $(CURDIR)/debian/pgfincore-%v

PKGVERS = $(shell dpkg-parsechangelog

awk -F '[:-]' '/^Version:/ { print substr($$2, 2) }')

EXCLUDE = —exclude-vcs —exclude=debian

include /usr/share/postgresql-common/pgxs_debian_control.mk

override_dh_auto_clean: debian/control

pg_buildext clean $(SRCDIR) $(TARGET) "$(CFLAGS)" dh_clean

override_dh_auto_build: # build all supported version pg_buildext build $(SRCDIR) $(TARGET) "$(CFLAGS)"

override_dh_auto_install: # then install each of them for v in `pg_buildext supported-versions $(SRCDIR)`; do \

dh_install -ppostgresql-$$v-pgfincore ;\ done

orig: clean

cd .. && tar czf pgfincore_$(PKGVERS).orig.tar.gz $(EXCLUDE) pgfincore

%:

dh $@ </pre>

The debian/rules file is known to be the corner stone of your debian packaging, and usually is the most complex part of it. It's a Makefile at its heart, and we can see that thanks to the debhelper magic it's not that complex to maintain anymore. Then, this file is using support from a bunch of helpers command, each of them comes with its own man page and does a little part of the work. The overall idea around debhelper is that what it does covers 90% of the cases around, and it's not aiming for more. You have to override the parts where it defaults to being wrong. Here for example the build system has to produce files for all three supported versions of <a href="http://www.postgresql.org/">PostgreSQL</a>, which means invoking the same build system three time with some changes in the environment (mainly setting the PG_CONFIG variable correctly). But even for that we have a debian facility, that comes in the package <a href="http://packages.debian.org/sid/postgresql-server-dev-all">postgresql-server-dev-all</a>, called pg_buildext. As long as your extension build system is VPATH friendly, it's all automated. Please read that last sentence another time. VPATH is the thing that allows Make to find your source tree somewhere in the system, not in the current working directory. That allows you to cleanly build the same sources in different build locations, which is exactly what we need here, and is cleanly supported by <a href="http://www.postgresql.org/docs/9.1/static/extend-pgxs.html">PGXS</a>, the <a href="http://www.postgresql.org/docs/9.1/static/extend-pgxs.html">PostgreSQL Extension Building Infrastructure</a>. Which means that the main Makefile of pgfincore had to be simplified, and the code layout too. Some advances Make features such as $(wildcard ...) and all will not work here. See what we got at the end: <pre class="src"> ifndef VPATH SRCDIR = . else SRCDIR = $(VPATH) endif

EXTENSION = pgfincore

EXTVERSION = $(shell grep default_version $(SRCDIR)/$(EXTENSION).control

sed -e "s/default_version:space:*=:space:*'$[^']*$'/\1/")

MODULES = $(EXTENSION) DATA = sql/pgfincore.sql sql/uninstall_pgfincore.sql DOCS = doc/README.$(EXTENSION).rst

PG_CONFIG = pg_config

<span style="color: #b8860b;">PG91</span> = $(<span style="color: #b8860b;">shell</span> $(<span style="color: #b8860b;">PG_CONFIG</span>) —version	grep -qE <span style="color: #bc8f8f;">"8\.\|9\.0"</span> && echo no	echo yes)

ifeq ($(PG91),yes) all: pgfincore—$(EXTVERSION).sql

pgfincore—$(EXTVERSION).sql: sql/pgfincore.sql

cp $< $@

DATA = pgfincore—unpackaged—$(EXTVERSION).sql pgfincore—$(EXTVERSION).sql EXTRA_CLEAN = sql/$(EXTENSION)—$(EXTVERSION).sql endif

PGXS := $(shell $(PG_CONFIG) —pgxs) include $(PGXS)

deb:

dh clean make -f debian/rules orig debuild -us -uc -sa </pre>

No more Make magic to find source files. Franckly though, when your sources are 1 c file and 2 sql files, you don't need that much magic anyway. You just want to believe that a single generic Makefile will happily build any project you throw at it, only requiring minor adjustment. Well, the reality is that you might need some more little adjustments if you want to benefit from VPATH building, and having the binaries for 8.4 and 9.0 and 9.1 built seemlessly in a simple loop. Like we have here for pgfincore. Now the Makefile still contains a little bit of magic, in order to parse the extension version number from its control file and produce a script named accordingly. Then you'll notice a difference between the <a href="https://github.com/dimitri/pgfincore/blob/master/debian/postgresql-9.1-pgfincore.install">postgresql-9.1-pgfincore.install</a> file and the <a href="https://github.com/dimitri/pgfincore/blob/master/debian/postgresql-9.0-pgfincore.install">postgresql-9.0-pgfincore.install</a>. We're just not shipping the same files: <pre class="src"> debian/pgfincore-9.0/pgfincore.so usr/lib/postgresql/9.0/lib sql/pgfincore.sql usr/share/postgresql/9.0/contrib sql/uninstall_pgfincore.sql usr/share/postgresql/9.0/contrib </pre> As you can see here: <pre class="src"> debian/pgfincore-9.1/pgfincore.so usr/lib/postgresql/9.1/lib debian/pgfincore-9.1/pgfincore*.sql usr/share/postgresql/9.1/extension sql/pgfincore—unpackaged—1.0.sql usr/share/postgresql/9.1/extension </pre> So, now that we uncovered all the relevant magic, packaging and building your next extension so that it supports as many PostgreSQL major releases as you need to will be that easy. For reference, you might need to also tweak /usr/share/postgresql-common/supported-versions so that it allows you to build for all those versions you claim to support in the <a href="https://github.com/dimitri/pgfincore/blob/master/debian/pgversions">debian/pgversions</a> file. <pre class="src"> $ sudo dpkg-divert \ —divert /usr/share/postgresql-common/supported-versions.distrib \ —rename /usr/share/postgresql-common/supported-versions

$ cat /usr/share/postgresql-common/supported-versions /bin/bash

dpkg -l postgresql-server-dev-* \ | awk -F '[ -]' '/^ii/ && ! /server-dev-all/ {print $6}' </pre>

All of this will come pretty handy when we finally sit down and work on a way to provide binary packages for PostgreSQL and its extensions, and all supported versions of those at that. This very project is not dead, it's just sleeping some more. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 19 Aug 2011 23:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/19-pgfincore-in-debian.html</guid> </item> <item> <title>Échappement de chaînes</title> <link>http://tapoueh.org/blog/2011/08/18-echappements-de-chaine.html</link> <description><![CDATA[Parmis les nouveautés de la <a href="http://www.postgresql.org/about/news.1331">prochaine version</a> de <a href="http://www.postgresql.org/">PostgreSQL</a>, la fameuse 9.1, il faut signaler le changement de valeur par défaut de la variable standard_conforming_strings, qui passe à vraie.

En effet, l'utilisation d'échappements avec le caractère « anti-slash » n'est pas conforme au standard SQL. Le paramètre standard_conforming_strings permet de contrôler le comportement de PostgreSQL lorsqu'il lit une chaîne de caractère dans une requête SQL. Voyons quelques exemples : <pre class="src"> dimitri=# set standard_conforming_strings to true; SET dimitri=# select 'hop''';

?column?
hop' (1 ligne)

dimitri=# select 'hop\''; dimitri'# ';

?column?
hop\';

(1 ligne)

dimitri=# select E'hop\'';

?column?
hop' (1 ligne)

dimitri=# set standard_conforming_strings to false; SET dimitri=# select E'hop\'';

?column?
hop' (1 ligne)

dimitri=# select 'hop\''; ATTENTION: utilisation non standard de \' dans une chaîne littérale LIGNE 1 : select 'hop\'';

^ ASTUCE : Utilisez '' pour écrire des guillemets dans une chaîne ou utilisez la syntaxe de chaîne d'échappement (E'...'). ?column?

hop' (1 ligne) </pre>

Il existe un moyen de forcer PostgreSQL à accepter l'utilisation d'échappements avec « anti-slash » indépendamment de la valeur de standard_conforming_strings, c'est la notation préfixée avec E. Il est recommandé de toujours l'utiliser dès lors que la chaîne de caractère contient des « anti-slash » utilisés comme échappement (du caractère simple guillemet en général). Le paramètre escape_string_warning, enfin, permet de désactiver les avertissements tels que présentés dans le dernier exemple ci-dessus, lorsqu'il est positionné à off. Bien sûr, sa valeur par défaut est on. Toute apparition de ce WARNING lorsque escape_string_warning est on signifie que votre application n'est pas prête à migrer à 9.1 avec son paramétrage par défaut. Il existe deux actions possible : changer le paramétrage de sa nouvelle valeur par défaut à sa précédente, ou bien corriger ses applications pour utiliser le préfixe E dès que cela est nécessaire. L'utilisation de standard_conforming_strings à on présente un autre avantage au respect du standard SQL : la sécurité contre les injections. S'il n'est pas possible d'échapper le guillemet simple qui termine toute chaîne de caractère utilisateur, il devient compliqué de jouer au plus malin avec le parser. Le mieux ici reste bien sûr d'utiliser les requêtes paramétrées, à suivre dans un prochain article. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 18 Aug 2011 19:01:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/18-echappements-de-chaine.html</guid> </item> <item> <title>el-get-list-packages</title> <link>http://tapoueh.org/blog/2011/08/18-el-get-list-packages.html</link> <description><![CDATA[From the first days of <a href="../../../emacs/el-get.html">el-get</a> is was quite clear for me that we would reach a point where users would want a nice listing including descriptions of the packages, and a major mode allowing you to select packages to install, remove and update. It was also quite clear that I was not much interested into doing it myself, even if I would appreciate having it done.

Well, the joy of Open Source & Free Software (pick your own poison). <a href="https://github.com/jglee1027">jglee1027</a> is this GitHub guy who did offer an implementation of said facility, and who added descriptions for almost all of the now 402 recipes that we have included with <a href="../../../emacs/el-get.html">el-get</a>. Here's an image of what you get: <center> <img src="../../../images/emacs-el-get-list-packages.png" alt=""> </center> The packages with no description are fetched by M-x el-get-emacswiki-refresh which will not download all <a href="http://emacswiki.org">emacswiki</a> content locally just so that it can parse the scripts's header and have a local description. Maybe it's time to ask for another page over there like <a href="http://www.emacswiki.org/cgi-bin/wiki?action=index;match=%5C.(el%7Ctar)(%5C.gz)%3F%24">emacswiki page index</a> but containing the first line too. For recipes we offer, this first line often looks like the following: <pre class="src"> ;;; 123-menu.el — Simple menuing system, reminiscent of Lotus 123 in DOS </pre> Of course some files over there are not following the stanza, but that would be good enough already. All in all, I hope you enjoy M-x el-get-list-packages! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 18 Aug 2011 18:10:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/18-el-get-list-packages.html</guid> </item> <item> <title>Tutoriel pgloader</title> <link>http://tapoueh.org/blog/2011/08/15-tutoriel-pgloader.html</link> <description><![CDATA[En reprenant le contenu des articles de la série sur <a href="http://tapoueh.org/pgsql/pgloader.html">pgloader</a>, j'ai pris le temps de compiler un tutoriel complet, en anglais. Si j'en crois les quelques mails que je reçois régulièrement au sujet de pgloader depuis quelques années maintenant, cela devrait aider les nouveaux utilisateurs. ]]></description> <author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 15 Aug 2011 15:39:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/15-tutoriel-pgloader.html</guid> </item> <item> <title>pgloader tutorial</title> <link>http://tapoueh.org/blog/2011/08/15-pgloader-tutorial.html</link> <description><![CDATA[To finish up the pgloader series, I've compiled all the information into a single page, the long awaited <a href="http://tapoueh.org/pgsql/pgloader.html#sec5">pgloader tutorial</a>. That should help lots of users to get started with pgloader. ]]></description> <author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 15 Aug 2011 15:33:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/15-pgloader-tutorial.html</guid> </item> <item> <title>pgloader constant cols</title> <link>http://tapoueh.org/blog/2011/08/12-pgloader-udc.html</link> <description><![CDATA[The previous articles in the <a href="../../../pgsql/pgloader.html">pgloader</a> series detailed <a href="http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html">How To Use PgLoader</a> then <a href="http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html">How to Setup pgloader</a>, then what to expect from a <a href="http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html">parallel pgloader</a> setup, and then <a href="http://tapoueh.org/blog/2011/08/05-reformating-modules-for-pgloader.html">pgloader reformating</a>. Another need you might encounter when you get to use <a href="../../../pgsql/pgloader.html">pgloader</a> is adding constant values into a table's column.

The basic situation where you need to do so is adding an origin field to your table. The value of that is not to be found in the data file itself, typically, but known in the pgloader setup. That could even be the filename you are importing data from. In <a href="../../../pgsql/pgloader.html">pgloader</a> that's called a user defined column. Here's what the relevant <a href="https://github.com/dimitri/pgloader/blob/master/examples/pgloader.conf">examples/pgloader.conf</a> setup looks like: <pre class="src"> [udc] table = udc format = text filename = udc/udc.data input_encoding = 'latin1' field_sep = % columns = b:2, d:1, x:3, y:4 udc_c = constant value copy_columns = b, c, d </pre> And the data file is: <pre class="src"> 1%5%foo%bar 2%10%bar%toto 3%4%toto%titi 4%18%titi%baz 5%2%baz%foo </pre> And here's what the loaded table looks like: <pre class="src"> pgloader/examples$ pgloader -Tsc pgloader.conf udc

Table name

duration

size

copy rows

errors

====================================================================

udc

0.201s

pgloader/examples$ psql —cluster 8.4/main pgloader -c "table udc"

+—————-+—

5	constant value	1
10	constant value	2
4	constant value	3
18	constant value	4
2	constant value	5

(5 rows) </pre>

Of course the configuration is not so straightforward as to process fields in the data file in the order that they appear, after all the <a href="https://github.com/dimitri/pgloader/blob/master/examples/pgloader.conf">examples/pgloader.conf</a> are also a test suite. Long story short: if you need to add some constant values into the target table you're loading data to, <a href="../../../pgsql/pgloader.html">pgloader</a> will help you there! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 12 Aug 2011 11:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/12-pgloader-udc.html</guid> </item> <item> <title>pgloader constant cols</title> <link>http://tapoueh.org/blog/2011/08/12-pgloader-udc.html</link> <description><![CDATA[The previous articles in the <a href="../../../pgsql/pgloader.html">pgloader</a> series detailed <a href="http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html">How To Use PgLoader</a> then <a href="http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html">How to Setup pgloader</a>, then what to expect from a <a href="http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html">parallel pgloader</a> setup, and then <a href="http://tapoueh.org/blog/2011/08/05-reformating-modules-for-pgloader.html">pgloader reformating</a>. Another need you might encounter when you get to use <a href="../../../pgsql/pgloader.html">pgloader</a> is adding constant values into a table's column.

The basic situation where you need to do so is adding an origin field to your table. The value of that is not to be found in the data file itself, typically, but known in the pgloader setup. That could even be the filename you are importing data from. In <a href="../../../pgsql/pgloader.html">pgloader</a> that's called a user defined column. Here's what the relevant <a href="https://github.com/dimitri/pgloader/blob/master/examples/pgloader.conf">examples/pgloader.conf</a> setup looks like: <pre class="src"> [udc] table = udc format = text filename = udc/udc.data input_encoding = 'latin1' field_sep = % columns = b:2, d:1, x:3, y:4 udc_c = constant value copy_columns = b, c, d </pre> And the data file is: <pre class="src"> 1%5%foo%bar 2%10%bar%toto 3%4%toto%titi 4%18%titi%baz 5%2%baz%foo </pre> And here's what the loaded table looks like: <pre class="src"> pgloader/examples$ pgloader -Tsc pgloader.conf udc

Table name

duration

size

copy rows

errors

====================================================================

udc

0.201s

pgloader/examples$ psql —cluster 8.4/main pgloader -c "table udc"

+—————-+—

5	constant value	1
10	constant value	2
4	constant value	3
18	constant value	4
2	constant value	5

(5 rows) </pre>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 12 Aug 2011 11:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/12-pgloader-udc.html</guid> </item> <item> <title>Emacs Startup</title> <link>http://tapoueh.org/blog/2011/08/blog/2011/08/06-emacs-startup-notification.html</link> <description><![CDATA[Using <a href="http://www.gnu.org/software/emacs/">Emacs</a> we get to manage a larger and larger setup file (either ~/.emacs or ~/.emacs.d/init.el), sometime with lots of dependencies, and some sub-files thanks to the load function or the provide and require mechanism.

Some users are even starting Emacs often enough for the startup time to be a concern. With an emacs-uptime (yes it's a command, you can

M-x
emacs-uptime

) of days to weeks (10 days, 17 hours, 45 minutes, 34 seconds as of this writing), it's not something I really care about much. But I know that some <a href="http://tapoueh.org/emacs/el-get.html">el-get</a> users still do care, and will use el-get-is-lazy and do all their Emacs tweaking as eval-after-load blocks. Trying to have an idea of how much a worst case startup with <a href="http://www.emacswiki.org/emacs/el-get">el-get</a> is, I have added the following piece of elisp at the very end of my startup code: <pre class="src"> (defun dim:notify-startup-done ()

" notify user that Emacs is now ready" (el-get-notify "Emacs is ready." (format "The init sequence took %g seconds." (float-time (time-subtract after-init-time before-init-time)))))

(add-hook 'after-init-hook 'dim:notify-startup-done) </pre>

The el-get-notify function will adapt and either use the dbus implementation from Emacs 24, or <a href="http://www.emacswiki.org/emacs/notify.el">notify.el</a> from <a href="http://www.emacswiki.org/">EmacsWiki</a> (just M-x el-get-install it if you need it), or will use its own implementation of an Emacs <a href="http://growl.info/">Growl</a> client (it's about 5 lines long), and baring all of that will use the message function. The reason I say worst case is that I have a lot of packages to initialize at startup, and that I did absolutely no effort for this initializing to be quick. Still, my Emacs setup is taking about 20 seconds to boot. Pretty good I would say, for a weekly operation. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Sat, 06 Aug 2011 14:58:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/blog/2011/08/06-emacs-startup-notification.html</guid> </item> <item> <title>Emacs Startup</title> <link>http://tapoueh.org/blog/2011/08/06-emacs-startup-notification.html</link> <description><![CDATA[Using <a href="http://www.gnu.org/software/emacs/">Emacs</a> we get to manage a larger and larger setup file (either ~/.emacs or ~/.emacs.d/init.el), sometime with lots of dependencies, and some sub-files thanks to the load function or the provide and require mechanism.

Some users are even starting Emacs often enough for the startup time to be a concern. With an emacs-uptime (yes it's a command, you can

M-x
emacs-uptime

" notify user that Emacs is now ready" (el-get-notify "Emacs is ready." (format "The init sequence took %g seconds." (float-time (time-subtract after-init-time before-init-time)))))

(add-hook 'after-init-hook 'dim:notify-startup-done) </pre>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Sat, 06 Aug 2011 14:58:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/06-emacs-startup-notification.html</guid> </item> <item> <title>pgloader reformating</title> <link>http://tapoueh.org/blog/2011/08/05-reformating-modules-for-pgloader.html</link> <description><![CDATA[Back to our series about <a href="../../../pgsql/pgloader.html">pgloader</a>. The previous articles detailed <a href="http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html">How To Use PgLoader</a> then <a href="http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html">How to Setup pgloader</a>, then what to expect from a <a href="http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html">parallel pgloader</a> setup. This article will detail how to reformat input columns so that what <a href="http://www.postgresql.org/">PostgreSQL</a> sees is not what's in the data file, but the result of a transformation from this data into something acceptable as an input for the target data type.

Here's what the <a href="http://pgloader.projects.postgresql.org/">pgloader documentation</a> has to say about this reformat parameter: The value of this option is a comma separated list of columns to rewrite, which are a colon separated list of column name, reformat module name, reformat function name. And here's the <a href="https://github.com/dimitri/pgloader/blob/master/examples/pgloader.conf">examples/pgloader.conf</a> section that deals with reformat: <pre class="src"> [reformat] table = reformat format = text filename = reformat/reformat.data

field_sep =

columns = id, timestamp reformat = timestamp:mysql:timestamp </pre>

The documentation says some more about it, so check it out. Also, the reformat_path option (set either on the command line or in the configuration file) is used to find the python module implementing the reformat function. Please refer to the manual as to how to set it. Now, obviously, for the reformat to happen we need to write some code. That's the whole point of the option: you need something very specific, you are in a position to write the 5 lines of code needed to make it happen, <a href="http://tapoueh.org/pgsql/pgloader.html">pgloader</a> allows you to just do that. Of course, the code needs to be written in python here, so that you can even benefit from the <a href="http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html">parallel pgloader</a> settings. Let's see an reformat module exemple, as found in <a href="https://github.com/dimitri/pgloader/blob/master/reformat/mysql.py">reformat/mysql.py</a> in the pgloader sources: <pre class="src"> # Author: Dimitri Fontaine <<a href="mailto:dim@tapoueh.org">dim@tapoueh.org</a>> # # pgloader mysql reformating module # def timestamp(reject, input):

""" Reformat str as a PostgreSQL timestamp

MySQL timestamps are like: 20041002152952 We want instead this input: 2004-10-02 15:29:52 """ if len(input) != 14: e = "MySQL timestamp reformat input too short: %s" % input reject.log(e, input)

year = input[0:4] month = input[4:6] day = input[6:8] hour = input[8:10] minute = input[10:12] seconds = input[12:14]

return '%s-%s-%s %s:%s:%s' % (year, month, day, hour, minute, seconds) </pre>

This reformat module will transform a timestamp representation as issued by certain versions of MySQL into something that PostgreSQL is able to read as a timestamp. If you're in the camp that wants to write as little code as possible rather than easy to read and maintain code, I guess you could write it this way instead: <pre class="src"> import re def timestamp(reject, input):

""" 20041002152952 -> 2004-10-02 15:29:52 """ g = re.match(r"(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})", input) return '%s-%s-%s %s:%s:%s' % tuple([g.group(x+1) for x in range(6)]) </pre>

Whenever you have an input file with data that PostgreSQL chokes upon, you can solve this problem from <a href="http://tapoueh.org/pgsql/pgloader.html">pgloader</a> itself: no need to resort to scripting and a pipelines of <a href="http://www.gnu.org/software/gawk/manual/gawk.html">awk</a> (which I use a lot in other cases, don't get me wrong) or other tools. See, you finally have an excuse to <a href="http://diveintopython.org/">Dive into Python</a>! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 05 Aug 2011 11:30:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/05-reformating-modules-for-pgloader.html</guid> </item> <item> <title>Reformater avec pgloader</title> <link>http://tapoueh.org/blog/2011/08/05-reformater-avec-pgloader.html</link> <description><![CDATA[Dans la série de nos articles sur <a href="http://tapoueh.org/tags/pgloader.html">pgloader</a>, le dernier venu détaille comment utiliser la fonction de reformatage de cet outil. Dans le cadre d'utilisation d'un <a href="http://fr.wikipedia.org/wiki/Extract_Transform_Load">ETL</a>, cela est assimilé à la phase Transform, ce qui fait de pgloader une solution simple pour vos besoins d'ETL. ]]></description> <author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 05 Aug 2011 11:26:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/05-reformater-avec-pgloader.html</guid> </item> <item> <title>See Tsung in action</title> <link>http://tapoueh.org/blog/2011/08/02-see-tsung-in-action.html</link> <description><![CDATA[<a href="http://tsung.erlang-projects.org/">Tsung</a> is an open-source multi-protocol distributed load testing tool and a mature project. It's been available for about 10 years and is built with the <a href="http://www.erlang.org/">Erlang</a> system. It supports several protocols, including the <a href="http://www.postgresql.org/">PostgreSQL</a> one.

When you want to benchmark your own application, to know how many more clients it can handle or how much gain you will see with some new shiny hardware, <a href="http://tsung.erlang-projects.org/">Tsung</a> is the tool to use. It will allow you to record a number of sessions then replay them at high scale. <a href="http://pgfouine.projects.postgresql.org/tsung.html">pgfouine</a> supports Tsung and is able to turn your PostgreSQL logs into Tsung sessions, too. Tsung did get used in the video game world, their version of it is called <a href="http://www.developer.unitypark3d.com/tools/utsung/">uTsung</a>, apparently using the <a href="http://www.developer.unitypark3d.com/index.html">uLink</a> game development facilities. They even made a video demo of uTsung, that you might find interresting: <blockquote> <a class="image-link" href="http://www.youtube.com/watch?v=rxBhqIP_7ls"> <img src="../../../images/utsung-demo.png"></a> </blockquote> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 02 Aug 2011 10:30:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/02-see-tsung-in-action.html</guid> </item> <item> <title>Parallel pgloader</title> <link>http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html</link> <description><![CDATA[This article continues the series that began with <a href="http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html">How To Use PgLoader</a> then detailed <a href="http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html">How to Setup pgloader</a>. We have some more fine points to talk about here, today's article is about loading your data in parallel with <a href="../../../pgsql/pgloader.html">pgloader</a>.

<h2>several files at a time</h2> Parallelism is implemented in 3 different ways in pgloader. First, you can load more than one file at a time thanks to the max_parallel_sections parameter, that has to be setup in the global section of the file. This setting is quite simple and already allows the most common use case. <h2>several workers per file</h2> The other use case is when you have huge files to load into the database. Then you want to be able to have more than one process reading the file at the same time. Using <a href="../../../pgsql/pgloader.html">pgloader</a>, you already did the compromise to load the whole content in more than one transaction, so there's no further drawback here about having those multiple transactions per file spread to more than one load worker. There are basically two ways to split the work between several workers here, and both are implemented in pgloader. <h3>N workers, N splits of the file</h3> <pre class="src"> section_threads = 4 split_file_reading = True </pre> Setup this way, <a href="../../../pgsql/pgloader.html">pgloader</a> will launch 4 different threads (see the caveat section of this article). Each thread is then given a part of the input data file and will run the whole usual pgloader processing on its own. For this to work you need to be able to seek in the input stream, which might not always be convenient. <h3>one reader, N workers</h3> <pre class="src"> section_threads = 4 split_file_reading = False rrqueue_size = 5000 </pre> With such a setup, <a href="../../../pgsql/pgloader.html">pgloader</a> will start 4 different worker threads that will receive the data input in an internal <a href="http://docs.python.org/library/collections.html#deque-objects">python queue</a>. Another active thread will be responsible of reading the input file and filling the queues in a round robin fashion, but will hand all the processing of the data to each worker, of course. <h3>how many threads?</h3> If you're using a mix and match of max_parallel_sections and section_threads with split_file_reading set to True of False, it's uneasy to know exactly how many threads will run at any time in the loading. How to ascertain which section will run in parallel when it depends on the timing of the loading? The advice here is the usual one, don't overestimate the capabilities of your system unless you are in a position to check before by doing trial runs. <h2>caveat</h2> Current implementation of all the parallelism in <a href="../../../pgsql/pgloader.html">pgloader</a> has been done with the <a href="http://docs.python.org/library/threading.html">python threading</a> API. While this is easy enough to use when you want to exchange data between threads, it's suffering from the <a href="http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock">Global Interpreter Lock</a> issue. This means that while the code is doing its processing in parallel, the runtime not so much. You might still benefit from the current implementation if you have hard to parse files, or custom reformat modules that are part of the loading bottleneck. <h2>future</h2> The solution would be to switch to using the newer <a href="http://docs.python.org/library/multiprocessing.html">python multiprocessing</a> API, and some preliminary work has been done in pgloader to allow for that. If you're interested in real parallel bulk loading, <a href="dim%20(at)%20tapoueh%20(dot)%20org">contact-me</a>! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 01 Aug 2011 12:15:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html</guid> </item> <item> <title>Parallel pgloader</title> <link>http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html</link> <description><![CDATA[This article continues the series that began with <a href="http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html">How To Use PgLoader</a> then detailed <a href="http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html">How to Setup pgloader</a>. We have some more fine points to talk about here, today's article is about loading your data in parallel with <a href="../../../pgsql/pgloader.html">pgloader</a>.

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Mon, 01 Aug 2011 12:05:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html</guid> </item> <item> <title>Configurer pgloader</title> <link>http://tapoueh.org/blog/2011/07/29-configurer-pgloader.html</link> <description><![CDATA[Je viens de publier un billet en anglais intitulé <a href="http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html">How to Setup pgloader</a>, qui complète l'écriture en cours d'un <a href="http://tapoueh.org/pgsql/pgloader.html">tutoriel pgloader</a> plus complet. Une fois de plus, je n'ai pas pris le temps de traduire cet article en français avant de savoir si cela vous intéresse, ô lecteurs. Si c'est le cas il suffit de me l'indiquer par mail (ou courriel, après tout) pour que j'ajoute cela dans ma TODO liste.

Bonne lecture ! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 29 Jul 2011 15:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/29-configurer-pgloader.html</guid> </item> <item> <title>How to Setup pgloader</title> <link>http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html</link> <description><![CDATA[In a previous article we detailed <a href="http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html">how to use pgloader</a>, let's now see how to write the pgloader.conf that instructs <a href="http://tapoueh.org/pgsql/pgloader.html">pgloader</a> about what to do.

This file is expected in the INI format, with a global section then one section per file you want to import. The global section defines some default options and how to connect to the <a href="http://tapoueh.org/pgsql/index.html">PostgreSQL</a> server. The configuration setup is fully documented on the <a href="http://pgloader.projects.postgresql.org/">pgloader man page</a> that you can even easily find online. As all unix style man pages, though, it's more a complete reference than introductory material. Let's review. <h2>global section</h2> Here's the global section of the <a href="https://github.com/dimitri/pgloader/blob/master/examples/pgloader.conf">examples/pgloader.conf</a> file of the source files. Well, some options are debugger only options, really, so I changed their value so that what you see here is a better starting point. <pre class="src"> [pgsql] base = pgloader

log_file = /tmp/pgloader.log log_min_messages = INFO client_min_messages = WARNING

lc_messages = C pg_option_client_encoding = 'utf-8' pg_option_standard_conforming_strings = on pg_option_work_mem = 128MB

copy_every = 15000

null = "" empty_string = "\ "

max_parallel_sections = 4 </pre>

You don't see all the connection setup, here base was enough. You might need to setup host, port and user, and maybe even pass, too, to be able to connect to the PostgreSQL server. The logging options allows you to set a file where to log all pgloader messages, that are categorized as either DEBUG, INFO, WARNING, ERROR or CRITICAL. The options log_min_messages and client_min_messages are another good idea stolen from <a href="http://www.postgresql.org/">PostgreSQL</a> and allow you to setup the level of chatter you want to see on the interactive console (standard output and standard error streams) and on the log file. Please note that the DEBUG level will produce more that 3 times as many data as the data file you're importing. If you're not a pgloader contributor or helping them, well, debug it, you want to avoid setting the log chatter to this value. The client_encoding will be <a href="http://www.postgresql.org/docs/current/static/sql-set.html">SET</a> by <a href="http://tapoueh.org/pgsql/pgloader.html">pgloader</a> on the PostgreSQL connection it establish. You can now even set any parameter you want by using the pg_option_parameter_name magic settings. Note that the command line option --pg-options (or -o for brevity) allows you to override that. Then, the copy_every parameter is set to 5 in the examples, because the test files are containing less than 10 lines and we want to test several batches of commits when using them. So for your real loading, stick to default parameters (10 000 lines per COPY command), or more. You can play with this parameter, depending on the network (or local access) and disk system you're using you might see improvements by reducing it or enlarging it. There's no so much theory of operation as empirical testing and setting here. For a one-off operation, just remove the lines from the configuration. The parameters null and empty_string are related to interpreting the data in the text or csv files you have, and the documentation is quite clear about them. Note that you have global setting and per-section setting too. The last parameter of this example, max_parallel_sections, is detailed later in the article. <h2>files section</h2> After the global section come as many sections as you have file to load. Plus the template sections, that are only there so that you can share a bunch of parameters in more than one section. Picture a series of data file all of the same format, the only thing that will change is the filename. Use a template section in this case! Let's see an example: <pre class="src"> [simple_tmpl] template = True format = text datestyle = dmy

field_sep =

trailing_sep = True

[simple] use_template = simple_tmpl table = simple filename = simple/simple.data columns = a:1, b:3, c:2 skip_head_lines = 2

# those reject settings are defaults one reject_log = /tmp/simple.rej.log reject_data = /tmp/simple.rej

[partial] table = partial format = text filename = partial/partial.data field_sep = % columns = * only_cols = 1-3, 5 </pre>

That's 2 of the examples from the <a href="https://github.com/dimitri/pgloader/blob/master/examples/pgloader.conf">examples/pgloader.conf</a> file, in 3 sections so that we see one template example. Of course, having a single section using the template, it's just here for the example. <h2>data file format</h2> The most important setting that you have to care about is the file format. Your choice here is either text, csv or fixed. Mostly, what we are given nowadays is csv. You might remember having read that the nice thing about standards is that there's so many to choose from... well, the csv land is one where it's pretty hard to find different producers that understand it the same way. So when you fail to have pgloader load your mostly csv files with a csv setup, it's time to consider using text instead. The text file format accept a lot of tunables to adapt to crazy situations, but is all python code when the <a href="http://docs.python.org/library/csv.html">python csv module</a> is a C-coded module, more efficient. If you're wondering what kind of format we're talking about here, here's the <a href="https://github.com/dimitri/pgloader/blob/master/examples/cluttered/cluttered.data">cluttered pgloader example</a> for your reading pleasure, using ^ (carret) as the field separator: <pre class="src"> 1^some multi\ line text with\ newline escaping^and some other data following^ 2^and another line^clean^ 3^and\ a last multiline\ escaped line with a missing\ escaping^just to test^ 4^\ ^empty value^ 5^^null value^ 6^multi line\ escaped value\ \ with empty line\ embeded^last line^ </pre> And here's what we get by loading that: <pre class="src"> pgloader/examples$ pgloader -c pgloader.conf -s cluttered

Table name

duration

size

copy rows

errors

====================================================================

cluttered

0.193s

pgloader/examples$ psql pgloader -c "table cluttered;"

—+——————————-+——————

and some other data following

some multi

: line text with : newline escaping

2	clean	and another line
3	just to test	and

: a last multiline : escaped line : with a missing : escaping

4	empty value
5	null value
6	last line	multi line

: escaped value : : with empty line : embeded (6 rows) </pre>

So when you have such kind of data, well, it might be that pgloader is still able to help you! Please refer to the <a href="http://pgloader.projects.postgresql.org/">pgloader man page</a> to know about each and every parameter that you can define and the values accepted, etc. And the fixed data format is to be used when you're not given a field separator but field positions in the file. Yes, we still encounter those from time to time. Who needs variable size storage, after all? ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 29 Jul 2011 15:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html</guid> </item> <item> <title>Emacs ANSI colors</title> <link>http://tapoueh.org/blog/2011/07/blog/2011/07/29-emacs-ansi-colors.html</link> <description><![CDATA[<a href="http://tapoueh.org/emacs/index.html">Emacs</a> comes with a pretty good implementation of a terminal emulator, M-x term. Well not that good actually, but given what I use it for, it's just what I need. Particulary if you add to that my <a href="http://tapoueh.org/emacs/cssh.html">cssh</a> tool, so that connecting with ssh to a remote host is just a =C-= runs the command cssh-term-remote-open away, and completes on the host name thanks to ~/.ssh/known_hosts.

Now, a problem that I still had to solve was the colors used in the terminal. As I'm using the tango color theme for emacs, the default ANSI palette's blue color was not readable. Here's how to fix that: <pre class="src">

(require 'ansi-color) (setq ansi-color-names-vector (vector (frame-parameter nil 'background-color) "#f57900" "#8ae234" "#edd400" "#729fcf" "#ad7fa8" "cyan3" "#eeeeec") ansi-term-color-vector ansi-color-names-vector ansi-color-map (ansi-color-make-color-map)) </pre>

Now your colors in an emacs terminal are easy to read, as you can see: <blockquote> <img src="../../../images/emacs-tango-term-colors.png" alt=""> </blockquote> Hope you enjoy! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 29 Jul 2011 10:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/blog/2011/07/29-emacs-ansi-colors.html</guid> </item> <item> <title>Emacs ANSI colors</title> <link>http://tapoueh.org/blog/2011/07/29-emacs-ansi-colors.html</link> <description><![CDATA[<a href="http://tapoueh.org/emacs/index.html">Emacs</a> comes with a pretty good implementation of a terminal emulator, M-x term. Well not that good actually, but given what I use it for, it's just what I need. Particulary if you add to that my <a href="http://tapoueh.org/emacs/cssh.html">cssh</a> tool, so that connecting with ssh to a remote host is just a =C-= runs the command cssh-term-remote-open away, and completes on the host name thanks to ~/.ssh/known_hosts.

(require 'ansi-color) (setq ansi-color-names-vector (vector (frame-parameter nil 'background-color) "#f57900" "#8ae234" "#edd400" "#729fcf" "#ad7fa8" "cyan3" "#eeeeec") ansi-term-color-vector ansi-color-names-vector ansi-color-map (ansi-color-make-color-map)) </pre>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 29 Jul 2011 10:00:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/29-emacs-ansi-colors.html</guid> </item> <item> <title>How to Setup pgloader</title> <link>http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html</link> <description><![CDATA[In a previous article we detailed <a href="http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html">how to use pgloader</a>, let's now see how to write the pgloader.conf that instructs <a href="http://tapoueh.org/pgsql/pgloader.html">pgloader</a> about what to do.

log_file = /tmp/pgloader.log log_min_messages = INFO client_min_messages = WARNING

copy_every = 15000

max_parallel_sections = 4 </pre>

field_sep =

trailing_sep = True

That's 2 of the examples from the <a href="https://github.com/dimitri/pgloader/blob/master/examples/pgloader.conf">examples/pgloader.conf</a> file, in 3 sections so that we see one template example. Of course, having a single section using the template, it's just here for the example. <h3>data file format</h3> The most important setting that you have to care about is the file format. Your choice here is either text, csv or fixed. Mostly, what we are given nowadays is csv. You might remember having read that the nice thing about standards is that there's so many to choose from... well, the csv land is one where it's pretty hard to find different producers that understand it the same way. So when you fail to have pgloader load your mostly csv files with a csv setup, it's time to consider using text instead. The text file format accept a lot of tunables to adapt to crazy situations, but is all python code when the <a href="http://docs.python.org/library/csv.html">python csv module</a> is a C-coded module, more efficient. If you're wondering what kind of format we're talking about here, here's the <a href="https://github.com/dimitri/pgloader/blob/master/examples/cluttered/cluttered.data">cluttered pgloader example</a> for your reading pleasure, using ^ (carret) as the field separator: <pre class="src"> 1^some multi\ line text with\ newline escaping^and some other data following^ 2^and another line^clean^ 3^and\ a last multiline\ escaped line with a missing\ escaping^just to test^ 4^\ ^empty value^ 5^^null value^ 6^multi line\ escaped value\ \ with empty line\ embeded^last line^ </pre> And here's what we get by loading that: <pre class="src"> pgloader/examples$ pgloader -c pgloader.conf -s cluttered

Table name

duration

size

copy rows

errors

====================================================================

cluttered

0.193s

pgloader/examples$ psql pgloader -c "table cluttered;"

—+——————————-+——————

and some other data following

some multi

: line text with : newline escaping

2	clean	and another line
3	just to test	and

: a last multiline : escaped line : with a missing : escaping

4	empty value
5	null value
6	last line	multi line

: escaped value : : with empty line : embeded (6 rows) </pre>

So when you have such kind of data, well, it might be that pgloader is still able to help you! Please refer to the <a href="http://pgloader.projects.postgresql.org/">pgloader man page</a> to know about each and every parameter that you can define and the values accepted, etc. And the fixed data format is to be used when you're not given a field separator but field positions in the file. Yes, we still encounter those from time to time. <h2>parallel processing</h2> <h3>one reader, multiple workers</h3> <h3>multiple workers, each reading</h3> ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 29 Jul 2011 09:57:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html</guid> </item> <item> <title>Next month partitions</title> <link>http://tapoueh.org/blog/2011/07/27-check-parts-for-next-month.html</link> <description><![CDATA[When you do partition your tables monthly, then comes the question of when to create next partitions. I tend to create them just the week before next month and I have some nice <a href="http://www.nagios.org/">nagios</a> scripts to alert me in case I've forgotten to do so. How to check that by hand in the end of a month?

Here's a catalog query to help you there: <pre class="src">

&gt; select *
-&gt;   from
-&gt;   (
(&gt;   select <span style

"color: #ad7fa8; font-style: italic;">'previous parts' as schemaname, count()::text as tablename (> from pg_tables (> where schemaname not in ('pg_catalog','information_schema') (> and tablename like to_char(now(), '%YYYYMM') (> (> union (>

(> select schemaname, substring(tablename,1,length(tablename)-6) '201108'

(> from pg_tables (> where schemaname not in ('pg_catalog','information_schema') (> and tablename like to_char(now(), '%YYYYMM') (> (> except (> (> select schemaname, tablename (> from pg_tables (> where schemaname not in ('pg_catalog','information_schema') (> and tablename like to_char(now() + interval '1 month', '%YYYYMM') (> ) as t -> order by schemaname <> 'previous parts', schemaname;

schemaname tablename

—————-+————————

 previous parts 1

central stats_entrantes_201108

(2 rows) </pre>
As you see, our partitions are named _YYYYMM so that's it's easy to match them in our queries, but I guess about everyone does about the same here. Then the to_char expressions only allow to not enter manually '%201108' in the query text. And there's a trick so that we display how many partitions we have this month, adding a line to the result... ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 27 Jul 2011 22:35:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/27-check-parts-for-next-month.html</guid> </item> <item> <title>Comment Utiliser pgloader</title> <link>http://tapoueh.org/blog/2011/07/22-comment-utiliser-pgloader.html</link> <description><![CDATA[C'est une question qui revient régulièrement, et à laquelle je pensais avoir apporté une réponse satisfaisante avec <a href="https://github.com/dimitri/pgloader/tree/master/examples">les exemples pgloader</a>. Ce document ressemble un peu à un tutoriel, en anglais, et je l'ai détaillé dans l'article <a href="22-how-to-use-pgloader.html">how to use pgloader</a> sur ce même site, en anglais. Si la demande est suffisante, je le traduirai en français.

En attendant, bonne lecture ! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 22 Jul 2011 13:48:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/22-comment-utiliser-pgloader.html</guid> </item> <item> <title>How To Use PgLoader</title> <link>http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html</link> <description><![CDATA[This question about <a href="../../../pgsql/pgloader.html">pgloader</a> usage coms in quite frequently, and I think the examples <a href="https://github.com/dimitri/pgloader/tree/master/examples">README</a> goes a long way in answering it. It's not exactly a tutorial but is almost there. Let me paste it here for reference:

<h2>installing pgloader</h2> Either use the <a href="http://packages.debian.org/source/pgloader">debian package</a> or the one for your distribution of choice if you use another one. RedHat, CentOS, FreeBSD, OpenBSD and some more already include a binary package that you can use directly. Or you could git clone https://github.com/dimitri/pgloader.git and go from there. As it's all python code, it runs fine interpreted from the source directory, you don't need to install it in a special place in your system. <h2>setting up the test environment</h2> To use them, please first create a pgloader database, then for each example the tables it needs, then issue the pgloader command: <pre class="src"> $ createdb —encoding=utf-8 pgloader $ cd examples $ psql pgloader < simple/simple.sql $ ../pgloader.py -Tvc pgloader.conf simple </pre> If you want to load data from all examples, create tables for all of them first, then run pgloader without argument. <h2>example description</h2> The provided examples are: <ul> <li>simple This dataset shows basic case, with trailing separator and data reordering.</li> <li>xzero Same as simple but using \0 as the null marker ()</li> <li>errors Same test, but with impossible dates. Should report some errors. If it does not report errors, check you're not using psycopg 1.1.21. Should report 3 errors out of 7 lines (4 updates).</li> <li>clob This dataset shows some text large object importing to PostgreSQL text datatype.</li> <li>cluttured A dataset with newline escaped and multi-line input (without quoting) Beware of data reordering, too.</li> <li>csv A dataset with csv delimiter ',' and quoting '"'.</li> <li>partial A dataset from which we only load some columns of the provided one.</li> <li>serial In this dataset the id field is ommited, it's a serial which will be automatically set by PostgreSQL while COPYing.</li> <li>reformat A timestamp column is formated the way MySQL dump its timestamp, which is not the same as the way PostgreSQL reads them. The reformat.mysql module is used to reformat the data on-the-fly.</li> <li>udc A used defined column test, where all file columns are not used but a new constant one, not found in the input datafile, is added while loading data.</li> </ul> <h2>running the import</h2> You can launch all those pgloader tests in one run, provided you created the necessary tables: <pre class="src">

$ for sql in /*sql; do psql pgloader < $sql; done $ ../pgloader.py -Tsc pgloader.conf

errors WARNING COPY error, trying to find on which line errors WARNING COPY data buffer saved in /tmp/errors.AhWvAv.pgloader errors WARNING COPY error recovery done (2/3) in 0.064s errors WARNING COPY error, trying to find on which line errors WARNING COPY data buffer saved in /tmp/errors.BclHtj.pgloader errors WARNING COPY error recovery done (1/1) in 0.054s errors ERROR 3 errors found into [errors] data errors ERROR please read /tmp/errors.rej.log for errors log errors ERROR and /tmp/errors.rej for data still to process errors ERROR 3 database errors occured reformat WARNING COPY error, trying to find on which line reformat WARNING COPY data buffer saved in /tmp/reformat.6P4WCD.pgloader reformat WARNING COPY error recovery done (1/4) in 0.034s reformat ERROR 1 errors found into [reformat] data reformat ERROR please read /tmp/reformat.rej.log for errors log reformat ERROR and /tmp/reformat.rej for data still to process reformat ERROR 1 database errors occured

Table name

duration

size

copy rows

errors

====================================================================

allcols	0.025s	-	8	0
clob	0.034s	-	7	0
cluttered	0.061s	-	6	0
csv	0.035s	-	6	0
errors	0.113s	-	4	3
fixed	0.045s	-	3	0
partial	0.030s	-	7	0
reformat	0.036s	-	4	1
serial	0.029s	-	7	0
simple	0.050s	-	7	0
udc	0.020s	-	5	0

====================================================================

Total

0.367s

</pre> Please note errors test should return 3 errors and reformat 1 error. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Fri, 22 Jul 2011 13:38:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html</guid> </item> <item> <title>Emacs Cheat Sheet</title> <link>http://tapoueh.org/blog/2011/07/blog/2011/07/20-emacs-cheat-sheet.html</link> <description><![CDATA[I stumbled upon the following cheat sheet for <a href="http://www.gnu.org/software/emacs/">Emacs</a> yesterday, and it's worth sharing. I already learnt or discovered again some nice default chords, like for example C-x C-o runs the command delete-blank-lines and C-M-o runs the command split-line. I guess I will use the later one a lot.

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 20 Jul 2011 10:44:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/blog/2011/07/20-emacs-cheat-sheet.html</guid> </item> <item> <title>Emacs Cheat Sheet</title> <link>http://tapoueh.org/blog/2011/07/20-emacs-cheat-sheet.html</link> <description><![CDATA[I stumbled upon the following cheat sheet for <a href="http://www.gnu.org/software/emacs/">Emacs</a> yesterday, and it's worth sharing. I already learnt or discovered again some nice default chords, like for example C-x C-o runs the command delete-blank-lines and C-M-o runs the command split-line. I guess I will use the later one a lot.

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 20 Jul 2011 10:44:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/20-emacs-cheat-sheet.html</guid> </item> <item> <title>Skytools3 : les slides</title> <link>http://tapoueh.org/blog/2011/07/19-skytools3-slides.html</link> <description><![CDATA[La conférence <a href="http://char11.org/">CHAR(11)</a> étant maintenant terminée, il est d'usage de publier les slides utilisés. J'ai présenté <a href="http://wiki.postgresql.org/wiki/SkyTools">Skytools</a> 3.0 dont la prochaine version sera publiée dès que j'aurais eu le temps de terminer de revoir (en fait principalement d'écrire) la documentation.

<center> <a class="image-link" href="../../../images/skytools3.pdf"> <img src="../../../images/skytools3-0.png"></a> </center> Les slides de l'ensemble des présentations devraient être publiés en ligne à terme, mais cela ne va pas pouvoir être fait aussi rapidement que nous le voudrions tous. Alors voici un peu de lecture en attendant la suite ! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 19 Jul 2011 14:39:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/19-skytools3-slides.html</guid> </item> <item> <title>Skytools3 talk Slides</title> <link>http://tapoueh.org/blog/2011/07/19-skytools3-talk-slides.html</link> <description><![CDATA[In case you're wondering, here are the slides from the <a href="http://char11.org/">CHAR(11)</a> talk I gave, about <a href="http://wiki.postgresql.org/wiki/SkyTools">Skytools</a> 3.0, soon to be released. That means as soon as I have enough time available to polish (or write) the documentation.

<center> <a class="image-link" href="../../../images/skytools3.pdf"> <img src="../../../images/skytools3-0.png"></a> </center> The slides for all the talks should eventually make their way to a central place, but expect some noticable delay here. Sorry about that, and have a good reading meanwhile! ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Tue, 19 Jul 2011 14:24:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/19-skytools3-talk-slides.html</guid> </item> <item> <title>Elisp Breadcrumbs</title> <link>http://tapoueh.org/blog/2011/07/blog/2011/07/14-elisp-breadcrumbs.html</link> <description><![CDATA[A <a href="http://en.wikipedia.org/wiki/Breadcrumb_(navigation)">breadcrumb</a> is a navigation aid. I just added one to this website, so that it gets easier to browse from any article to its local and parents indexes and back to <a href="../../../index.html">/dev/dim</a>, the root webpage of this site.

As it was not that much work to implement, here's the whole of it: <pre class="src"> ;;; ;;; Breadcrumb support ;;; (defun tapoueh-breadcrumb-to-current-page ()

"Return a list of (name . link) from the index root page to current one" (let* ((current (muse-current-file)) (cwd (file-name-directory current)) (project (muse-project-of-file current)) (root (muse-style-element :path (caddr project))) (path (tapoueh-path-to-root)) (dirs (split-string (file-relative-name current root) "/"))) ;; ("blog" "2011" "07" "13-back-from-char11.muse") (append (list (cons "/dev/dim" (concat path "index.html"))) (loop for p in (butlast dirs) collect (cons p (format "%s%s/index.html" path p)) do (setq path (concat path p "/"))))))

(defun tapoueh-insert-breadcrumb-div ()

"The real HTML inserting" (insert "<div id=\"breadcrumb\">") (loop for (name . link) in (tapoueh-breadcrumb-to-current-page) do (insert (format "<a href=%s>%s</a>" link name) " / ")) (insert "</div>\n"))

(defun tapoueh-insert-breadcrumb ()

"Must run with current buffer being a muse article" (save-excursion (beginning-of-buffer) (when (tapoueh-extract-directive "author" (muse-current-file)) (re-search-forward "<body>" nil t) ; find where the article content is (re-search-forward "<h2>" nil t) ; that's the title line (beginning-of-line) (open-line 1) (tapoueh-insert-breadcrumb-div)

(re-search-forward "<h2>" nil t 2) ; that's the TAG line (beginning-of-line) (open-line 1) (tapoueh-insert-breadcrumb-div)))) </pre>

This code is now called in the :after function of my <a href="http://www.emacswiki.org/emacs/EmacsMuse">Muse</a> project style, and it gets the work done. ]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 14 Jul 2011 18:44:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/blog/2011/07/14-elisp-breadcrumbs.html</guid> </item> <item> <title>Elisp Breadcrumbs</title> <link>http://tapoueh.org/blog/2011/07/14-elisp-breadcrumbs.html</link> <description><![CDATA[ (open-line 1) (tapoueh-insert-breadcrumb-div)))) </pre>

This code is now called in the :after function of my <a href="http://www.emacswiki.org/emacs/EmacsMuse">Muse</a> project style, and it gets the work done. <h2>Tags</h2> <a href="../../../tags/emacs.html">Emacs</a> <a href="../../../tags/muse.html">Muse</a> </div>

]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Thu, 14 Jul 2011 18:44:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/14-elisp-breadcrumbs.html</guid> </item> <item> <title>De retour de CHAR(11)</title> <link>http://tapoueh.org/blog/2011/07/13-de-retour-de-char11.html</link> <description><![CDATA[h1>De retour de CHAR(11)</h1>
<a href=../../../index.html>/dev/dim</a> / <a href=../../../blog/index.html>blog</a> / <a href=../../../blog/2011/index.html>2011</a> / <a href=../../../blog/2011/07/index.html>07</a> /
Wednesday, July 13 2011, 17:30 </div>
Quelle meilleure occupation dans le train du retour de <a href="http://char11.org/schedule">CHAR(11)</a> que de se faire reporteur pour l'occasion ? En réalité, dormir serait une idée tant les soirées se sont prolongées !

Nous avons eu le plaisir d'écouter Jan Wieck présenter un historique simplifié de la réplication avec <a href="http://www.postgresql.org/">PostgreSQL</a>. Étant lui-même l'un des pionniers du domaine, son point de vue est des plus intéressants. Il a parlé de l'évolution des solutions de réplication, et je ne peux m'empêcher de penser que par bien des côtés <a href="http://wiki.postgresql.org/wiki/SKytools">Skytools</a> est une évolution de <a href="http://slony.info/">Slony</a> — Jan, auteur de Slony, semblait d'accord avec cela. En effet Skytools est né de limitations de Slony. Certaines d'entre elles existent toujours, comme l'absence de séparation entre la couche de queuing et la couche de réplication elle-même, et certaines ont été résolues depuis, comme les difficultés à subir de fortes charges en écriture. Et puis les deux solutions partagent même une partie de leur implémentation, depuis PostgreSQL 8.3, avec les types de données txid et <a href="http://www.postgresql.org/docs/8.3/interactive/functions-info.html#FUNCTIONS-TXID-SNAPSHOT">txid_snapshot</a>. Bien sûr, l'objectif de Skytools est d'avoir une solution la plus simple possible, parfaitement adapée à un ensemble de cas d'utilisation précis et bornés, alors que Slony essaye de résoudre automatiquement les problèmes les plus difficiles du domaine, au prix d'une interface très complexe. Bien sûr, Jan a pris le temps de comparer objectivement ces solutions de réplication avec la solution intégrée dans PostgreSQL, Streaming Replication et Hot Standby. Nous avions déjà la réplication binaire asynchrone, PostgreSQL 9.1 nous apporte la réplication synchrone avec un contrôle par transaction. <a href="http://database-explorer.blogspot.com/">Simon Riggs</a>, auteur de la fonctionalité, a insisté sur l'innovation que cela représente : aucun autre projet ne permet de contrôler la garantie de durabilité des données avec une granularité aussi souple et précise ! <a href="http://projects.2ndquadrant.com/repmgr">repmgr</a> est une solution d'administration de cluster animés avec Hot Standby et Streaming Replication (synchrone ou non). Son fonctionnement a été détaillé par Greg Smith et Cédric Villemain. Le premier a montré comment mettre au point une architecture permettant de répartir la charge en lecture, et le second comment obtenir un système tolérant aux pannes grâce au failover automatique intégré dans repmgr. Cette solution innovante a été mise au point en grande partie par 2ndQuadrant France, nous l'avons déjà estampillée production ready. <a href="http://www.hagander.net/">Magnus Hagander</a> a beaucoup travaillé sur le protocole de streaming utilisé pour la réplication intégrée dans PostgreSQL 9.1, ainsi que sur les outils qui exploitent ce protocole. Il a naturellement présenté cela, et l'idée d'un proxy relayant le flux binaire des journaux de transaction est revenue dans les discutions (nous avions déjà envisagé cela en 2010, l'article en anglais <a href="../../2010/05/27-back-from-pgcon2010.html">Back from PgCon2010</a> contient quelques éléments sur le sujet). Avec la réplication synchrone, il devient possible de concevoir des architectures avancées, robustes et versatiles — le proxy pourrait maintenant s'occuper à la fois des archives et des serveurs standby. <a href="http://database-explorer.blogspot.com/">Simon Riggs</a> nous a ensuite proposé une rétrospective des 7 dernières années de travail qu'il a réalisé avec PostgreSQL, de l'implémentation du Point in Time Recovery à la réplication synchrone, en passant par Hot Standby. Ce que nous avons dans PostgreSQL 9.0 correspond déjà à ce qu'Oracle propose de plus avancé en terme de durabitilé des données, et 9.1 permet de franchir l'étape suivante. Cela ne freine en rien Simon qui parlait déjà des projets à venir pour les 10 prochaines années. Enfin, <a href="http://www.heroku.com/">Heroku</a> nous a présenté leur incroyable entreprise. Ils ont aujourd'hui plus de 150 000 instances de PostgreSQL en production, démontrant que notre SGBD préféré est prêt pour les hébergeurs. Heroku est en train de concevoir et réaliser une solution prête à l'emploi pour le fameux Cloud si difficile à définir. Ici, il s'agit d'être capable d'ajouter des nouveaux réplicas en lecture seule à la volée pour encaisser les pics de trafic, créer des instances de développement d'un clic, etc. Cet article ne couvre qu'une petite sélection des sujets abordés à la conférence, je compte sur <a href="http://blog.guillaume.lelarge.info/">Guillaume</a> pour lui aussi vous parler de <a href="http://char11.org/schedule">CHAR(11)</a>, mais il faudra peut être attendre son retour des <a href="http://2011.rmll.info/">RMLL</a> (quelle énergie !). <h2>Tags</h2> <a href="../../../tags/postgresqlfr.html">PostgreSQLFr</a> <a href="../../../tags/conferences.html">Conferences</a> <a href="../../../tags/skytools.html">Skytools</a>

]]></description>

<author>dim@tapoueh.org (Dimitri Fontaine)</author> <pubDate>Wed, 13 Jul 2011 17:30:00 +0200</pubDate> <guid isPermaLink="true">http://tapoueh.org/blog/2011/07/13-de-retour-de-char11.html</guid> </item> <item> <title>Back From CHAR(11)</title> <link>http://tapoueh.org/blog/2011/07/13-back-from-char11.html</link> <description><![CDATA[h1>Back From CHAR(11)</h1>
<a href=../../../index.html>/dev/dim</a> / <a href=../../../blog/index.html>blog</a> / <a href=../../../blog/2011/index.html>2011</a> / <a href=../../../blog/2011/07/index.html>07</a> /
Wednesday, July 13 2011, 17:15 </div>
<a href="http://char11.org/schedule">CHAR(11)</a> finished somewhen in the night leading to today, if you consider the social events to be part of it, which I definitely do. This conference has been a very good one, both on the organisation side of things and of course for its content.

It began with a perspective about the evolution of replication solutions, by Jan Wieck himself. In some way <a href="http://wiki.postgresql.org/wiki/SKytools">Skytools</a> is an evolution of <a href="http://slony.info/">Slony</a>, in the sense that it reuses the same concepts, a part of the design, and even share bits of the implementation (like the <a href="http://www.postgresql.org/docs/8.3/interactive/functions-info.html#FUNCTIONS-TXID-SNAPSHOT">txid_snapshot</a> datatype that were added in PostgreSQL 8.3). The evolution occured in choosing a subset of the features of Slony and then simplifying the user interface as much as possible. And with Skytools 3.0, those features that were removed but still are useful to solve real-life problems are now available too. Of course the talk did approach the other replication solutions (not just the trigger based ones), and did compare <a href="http://wiki.postgresql.org/wiki/Setting_up_RServ_with_PostgreSQL_7.0.3">RServ</a> to <a href="http://bucardo.org/">Bucardo</a> for example. And then all those were compared to the <a href="http://www.postgresql.org/">PostgreSQL</a> core replication facilities, which are quite a different animal. It was a really nice keynote here, preparing the audience minds to make the most out of all the other talks. I will not review all the talks in details, as I'm pretty sure some other attendees will turn into reporters themselves: scaling the write load! Still <a href="http://projects.2ndquadrant.com/repmgr">repmgr</a> got its share of attention. <a href="http://www.2ndquadrant.com/books/postgresql-9-0-high-performance/">Greg Smith</a> and <a href="http://www.2ndquadrant.fr/">Cédric Villemain</a> did present both how to do read scaling and auto failover management with this tool, going into fine details about how it works internally and how to best design your services architecture for maximum data availibility. The question and answers section led to insist on the fact that you can not have data availibility with less than 3 production nodes. <a href="http://www.hagander.net/">Magnus Hagander</a> detailed how flexible the core protocol support for replication (and streaming) really is. That flexibility means that you can quite easily talk this protocol from any application, and the idea of a wal proxy did pop out again (see <a href="../../2010/05/27-back-from-pgcon2010.html">Back from PgCon2010</a> article for my first mentionning of the idea). The main difference is that we now have synchronous replication support, so that the proxy could be trusted both for archiving and serving standbys. Of course <a href="http://database-explorer.blogspot.com/">Simon</a> still has lots of ideas about next 10 years of replication oriented projects for core PostgreSQL code, and his talk nicely summarized the previous 7 years. Future is bright, and guess what, it's beginning today! We also heard about <a href="http://www.heroku.com/">Heroku</a>, and these guys are doing crazy impressive things. Like running 150 000 PostgreSQL instances, for example, showing that you can actually use our prefered database server in the hosting business. I expect that the maturing solution and tool sets providing data availibility are soon to be a game changer here. What they are doing is designing a flexible data architecture with strong guarantees (no data loss). The cloud elasticity is reaching out from the stateless services, and those guys are making it happen now. May you live in interresting times! <h2>Tags</h2> <a href="../../../tags/postgresql.html">PostgreSQL</a> <a href="../../../tags/skytools.html">skytools</a> <a href="../../../tags/conferences.html">Conferences</a>

$ sudo dpkg-divert \ —divert /usr/share/postgresql-common/supported-versions.distrib \ —rename /usr/share/postgresql-common/supported-versions

dpkg -l postgresql-server-dev-* \ | awk -F '[ -]' '/^ii/ && ! /server-dev-all/ {print $6}' </pre>

$ cat debian/postgresql-9.1-ip4r.install debian/ip4r-9.1/ip4r.so usr/lib/postgresql/9.1/lib ip4r.control usr/share/postgresql/9.1/extension debian/ip4r-9.1/ip4r.sql usr/share/postgresql/9.1/extension

$ cat debian/postgresql-9.1-ip4r.links usr/share/postgresql/9.1/extension/ip4r.sql usr/share/postgresql/9.1/extension/ip4r—1.05.sql </pre>

$ psql -U postgres —cluster 9.1/main -c 'create extension ip4r;' CREATE EXTENSION </pre>

# \o /tmp/create.sql

# select 'CREATE VIEW ' || views || E' AS \n'

# \o
</pre>

<p>Replace <code>SCHEMA.TABLENAME</code> and <code>COLUMN_NAME</code> with your targets here and the
first query should give you one row per candidate view.  Well if you're not
using the <code>\o</code> trick, that is — if you do, check out the generated file
instead, with <code>\! cat /tmp/drop.sql</code> for example.</p>

<p>Please note that this catalog query is not accurate, as it will select as a
candidate any view that will by chance both depend on the target table and
contain the <code>column_name</code> in its text definition.  So either filter out the
candidates properly (by proper proof reading then another <code>WHERE</code> clause), or
just accept that you might <code>DROP</code> then <code>CREATE</code> again more <em>views</em> than need be.</p>

<p>If you need some more details about the <code>\t \o</code> sequence you might be
interested in this older article about <a href

"http://tapoueh.org/articles/blog/_Resetting_sequences._All_of_them,_please!.html">resetting sequences</a>. <h2>Tags</h2> <a href="../../../tags/postgresql.html">PostgreSQL</a> <a href="../../../tags/catalogs.html">catalogs</a>

(when (string-match "apple-darwin" system-configuration)

(when (string-match "apple-darwin" system-configuration)

(defvar dim-desktop-file-readahead-list

(defun dim-desktop-get-readahead-file-list (&optional filename dir)

;; This will not work because the hook is run before to add the buffers into ;; the desktop file. ;; ;;(add-hook 'desktop-save-hook 'dim-desktop-get-readahead-file-list) ;; so instead, advise the function (defadvice desktop-save (after desktop-save-readahead activate)

(provide 'dim-desktop) </pre>

# just readahead the emacs desktop files # this file listing is maintained directly from Emacs itself readahead ~/.emacs.desktop.readahead </pre>

(defvar dim-desktop-file-readahead-list

(defun dim-desktop-get-readahead-file-list (&optional filename dir)

;; This will not work because the hook is run before to add the buffers into ;; the desktop file. ;; ;;(add-hook 'desktop-save-hook 'dim-desktop-get-readahead-file-list) ;; so instead, advise the function (defadvice desktop-save (after desktop-save-readahead activate)

(provide 'dim-desktop) </pre>

# just readahead the emacs desktop files # this file listing is maintained directly from Emacs itself readahead ~/.emacs.desktop.readahead </pre>

;; visual settings (setq inhibit-splash-screen t) (menu-bar-mode -1) (tool-bar-mode -1) (scroll-bar-mode -1)

;; Use the clipboard, pretty please, so that copy/paste "works" (setq x-select-enable-clipboard t)

(set-frame-font "Monospace-10")

;; suivre les changements exterieurs sur les fichiers (global-auto-revert-mode 1)

;; pour les couleurs dans M-x shell (autoload 'ansi-color-for-comint-mode-on "ansi-color" nil t) (add-hook 'shell-mode-hook 'ansi-color-for-comint-mode-on)

;; S-fleches pour changer de fenêtre (windmove-default-keybindings) (setq windmove-wrap-around t)

;; find-file-at-point quand ça a du sens (setq ffap-machine-p-known 'accept) ; no pinging (setq ffap-url-regexp nil) ; disable URL features in ffap (setq ffap-ftp-regexp nil) ; disable FTP features in ffap (define-key global-map (kbd "C-x C-f") 'find-file-at-point)

(require 'ibuffer) (global-set-key "\C-x\C-b" 'ibuffer)

;; use iswitchb-mode for C-x b (iswitchb-mode)

;; I can't remember having meant to use C-z as suspend-frame (global-set-key (kbd "C-z") 'undo)

;; winner-mode pour revenir sur le layout précédent C-c <left> (winner-mode 1)

;; dired-x pour C-x C-j (require 'dired-x)

;; full screen (defun fullscreen ()

1812; 1262 18497 DATABASE - exts dim 1; 3996 18498 EXTENSION - pg_trgm 1813; 0 0 COMMENT - EXTENSION pg_trgm 6; 2615 2200 SCHEMA - public dim 1814; 0 0 COMMENT - SCHEMA public dim 1815; 0 0 ACL - public dim 320; 2612 11602 PROCEDURAL LANGUAGE - plpgsql dim 1521; 1259 18543 TABLE public test dim 1809; 0 18543 TABLE DATA public test dim 1808; 1259 18549 INDEX public idx_test_name dim </pre>

dim=# select get_nth_dow_of_month(1, 4, '2010-09-12');

(el-get) ; that could/should be (el-get 'sync) (provide 'dim-packages) </pre>

("auctex.el" "auto-complete-etags.el" "auto-complete-extension.el" "auto-complete.el" "auto-install.el" "autopair.el" "bbdb.el" "blender-python-mode.el" "color-theme-twilight.el" "color-theme.el" "cssh.el" "django-mode.el" "el-get.el" "emacs-w3m.el" "emacschrome.el" "emms.el" "ensime.el" "erc-highlight-nicknames.el" "erc-track-score.el" "escreen.el" "filladapt.el" "flyguess.el" "gist.el" "google-maps.el" "google-weather.el" "goto-last-change.el" "haskell-mode.el" "highlight-parentheses.el" "hl-sexp.el" "levenshtein.el" "magit.el" "mailq.el" "maxframe.el" "multi-term.el" "muse-blog.el" "nognus.el" "nterm.el" "nxhtml.el" "offlineimap.el" "package.el" "popup-kill-ring.el" "pos-tip.el" "pov-mode.el" "psvn.el" "pymacs.el" "rainbow-mode.el" "rcirc-groups.el" "rinari.el" "ropemacs.el" "rt-liberation.el" "scratch.el" "session.el" "sicp.el" "smex.el" "switch-window.el" "textile-mode.el" "todochiku.el" "twitter.el" "twittering-mode.el" "undo-tree.el" "verbiste.el" "vimpulse-surround.el" "vimpulse.el" "vkill.el" "xcscope.el" "xml-rpc-el.el" "yasnippet.el") </pre>

(el-get) ; that could/should be (el-get 'sync) (provide 'dim-packages) </pre>

("auctex.el" "auto-complete-etags.el" "auto-complete-extension.el" "auto-complete.el" "auto-install.el" "autopair.el" "bbdb.el" "blender-python-mode.el" "color-theme-twilight.el" "color-theme.el" "cssh.el" "django-mode.el" "el-get.el" "emacs-w3m.el" "emacschrome.el" "emms.el" "ensime.el" "erc-highlight-nicknames.el" "erc-track-score.el" "escreen.el" "filladapt.el" "flyguess.el" "gist.el" "google-maps.el" "google-weather.el" "goto-last-change.el" "haskell-mode.el" "highlight-parentheses.el" "hl-sexp.el" "levenshtein.el" "magit.el" "mailq.el" "maxframe.el" "multi-term.el" "muse-blog.el" "nognus.el" "nterm.el" "nxhtml.el" "offlineimap.el" "package.el" "popup-kill-ring.el" "pos-tip.el" "pov-mode.el" "psvn.el" "pymacs.el" "rainbow-mode.el" "rcirc-groups.el" "rinari.el" "ropemacs.el" "rt-liberation.el" "scratch.el" "session.el" "sicp.el" "smex.el" "switch-window.el" "textile-mode.el" "todochiku.el" "twitter.el" "twittering-mode.el" "undo-tree.el" "verbiste.el" "vimpulse-surround.el" "vimpulse.el" "vkill.el" "xcscope.el" "xml-rpc-el.el" "yasnippet.el") </pre>

smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache smtp_tls_loglevel = 2 smtp_use_tls = yes smtp_tls_security_level = may </pre>

# explain analyze select x
                     from generate_series(1, 10000) as t(x)
                    where happy(x);
                      QUERY PLAN
<span style

"color: #b22222;">———————————————————— Function Scan on generate_series t

(defun find-happy-numbers (&optional limit)

# explain analyze select x
                     from generate_series(1, 10000) as t(x)
                    where happy(x);
                      QUERY PLAN
<span style

"color: #888a85;">———————————————————— Function Scan on generate_series t

(defun find-happy-numbers (&optional limit)

# select ('101011001011100110010110'::varbit &lt;&lt; 16)::bit(8);
   bit
----------
 10010110
(1 row)

# select * from TEMP VERSION OF THE FUNCTION FOR TESTING

ts_column
<span style="color: #ad7fa8; font-style: italic;">' FROM (SELECT ('</span>
quote_literal(OLD)	<span style="color: #ad7fa8; font-style: italic;">'::'</span>	TG_RELID::regclass
<span style="color: #ad7fa8; font-style: italic;">').*) as n'</span>

adminpack	\|	Administrative functions for PostgreSQL
auto_username	\|	functions for tracking who changed a table
autoinc	\|	functions for autoincrementing fields
btree_gin	\|	GIN support for common types BTree operators
btree_gist	\|	GiST support for common types BTree operators
chkpass	\|	Store crypt()ed passwords
citext	\|	case-insensitive character string type
cube	\|	data type for representing multidimensional cubes
dblink	\|	connect to other PostgreSQL databases from within a database
dict_int	\|	example of an add-on dictionary template for full-text search
dict_xsyn	\|	example of an add-on dictionary template for full-text search
earthdistance	\|	calculating great circle distances on the surface of the Earth
fuzzystrmatch	\|	determine similarities and distance between strings
hstore	\|	storing sets of key/value pairs
int_aggregate	\|	integer aggregator and an enumerator (obsolete)
intarray	\|	one-dimensional arrays of integers: functions, operators, index support
isn	\|	data types for the international product numbering standards
lo	\|	managing Large Objects
ltree	\|	data type for hierarchical tree-like structure
moddatetime	\|	functions for tracking last modification time
pageinspect	\|	inspect the contents of database pages at a low level
pg_buffercache	\|	examine the shared buffer cache in real time
pg_freespacemap	\|	examine the free space map (FSM)
pg_stat_statements	\|	tracking execution statistics of all SQL statements executed
pg_trgm	\|	determine the similarity of text, with indexing support
pgcrypto	\|	cryptographic functions
pgrowlocks	\|	show row locking information for a specified table
pgstattuple	\|	obtain tuple-level statistics
prefix	\|	Prefix Match Indexing
refint	\|	functions for implementing referential integrity
seg	\|	data type for representing line segments, or floating point intervals
tablefunc	\|	various functions that return tables, including crosstab(text sql)
test_parser	\|	example of a custom parser for full-text search
timetravel	\|	functions for implementing time travel
tsearch2	\|	backwards-compatible text search functionality (pre-8.3)
unaccent	\|	text search dictionary that removes accents

pg_extension	3996	18498	extension pg_trgm
pg_proc	1255	18499	function set_limit(real)
pg_proc	1255	18500	function show_limit()
pg_proc	1255	18501	function show_trgm(text)
pg_proc	1255	18502	function similarity(text,text)
pg_proc	1255	18503	function similarity_op(text,text)
pg_operator	2617	18504	operator %(text,text)
pg_type	1247	18505	type gtrgm
pg_proc	1255	18506	function gtrgm_in(cstring)
pg_proc	1255	18507	function gtrgm_out(gtrgm)
pg_type	1247	18508	type gtrgm[]
pg_proc	1255	18509	function gtrgm_consistent(internal,text,integer,oid,internal)
pg_proc	1255	18510	function gtrgm_compress(internal)
pg_proc	1255	18511	function gtrgm_decompress(internal)
pg_proc	1255	18512	function gtrgm_penalty(internal,internal,internal)
pg_proc	1255	18513	function gtrgm_picksplit(internal,internal)
pg_proc	1255	18514	function gtrgm_union(bytea,internal)
pg_proc	1255	18515	function gtrgm_same(gtrgm,gtrgm,internal)
pg_opfamily	2753	18516	operator family gist_trgm_ops for access method gist
pg_opclass	2616	18517	operator class gist_trgm_ops for access method gist
pg_amop	2602	18518	operator 1 %(text,text) of operator family gist_trgm_ops for access method gist
pg_amproc	2603	18519	function 1 gtrgm_consistent(internal,text,integer,oid,internal) of operator family gist_trgm_ops for access method gist
pg_amproc	2603	18520	function 2 gtrgm_union(bytea,internal) of operator family gist_trgm_ops for access method gist
pg_amproc	2603	18521	function 3 gtrgm_compress(internal) of operator family gist_trgm_ops for access method gist
pg_amproc	2603	18522	function 4 gtrgm_decompress(internal) of operator family gist_trgm_ops for access method gist
pg_amproc	2603	18523	function 5 gtrgm_penalty(internal,internal,internal) of operator family gist_trgm_ops for access method gist
pg_amproc	2603	18524	function 6 gtrgm_picksplit(internal,internal) of operator family gist_trgm_ops for access method gist
pg_amproc	2603	18525	function 7 gtrgm_same(gtrgm,gtrgm,internal) of operator family gist_trgm_ops for access method gist
pg_proc	1255	18526	function gin_extract_trgm(text,internal)
pg_proc	1255	18527	function gin_extract_trgm(text,internal,smallint,internal,internal)
pg_proc	1255	18528	function gin_trgm_consistent(internal,smallint,text,integer,internal,internal)
pg_opfamily	2753	18529	operator family gin_trgm_ops for access method gin
pg_opclass	2616	18530	operator class gin_trgm_ops for access method gin
pg_amop	2602	18531	operator 1 %(text,text) of operator family gin_trgm_ops for access method gin
pg_amproc	2603	18532	function 1 btint4cmp(integer,integer) of operator family gin_trgm_ops for access method gin
pg_amproc	2603	18533	function 2 gin_extract_trgm(text,internal) of operator family gin_trgm_ops for access method gin
pg_amproc	2603	18534	function 3 gin_extract_trgm(text,internal,smallint,internal,internal) of operator family gin_trgm_ops for access method gin
pg_amproc	2603	18535	function 4 gin_trgm_consistent(internal,smallint,text,integer,internal,internal) of operator family gin_trgm_ops for access method gin

doc/src/sgml/extend.sgml	46 ++
doc/src/sgml/ref/allfiles.sgml	2
doc/src/sgml/ref/create_extension.sgml	95 ++++
doc/src/sgml/ref/drop_extension.sgml	115 +++++
doc/src/sgml/reference.sgml	2
src/backend/access/transam/xlog.c	95 —-
src/backend/catalog/Makefile	1
src/backend/catalog/dependency.c	25 +
src/backend/catalog/heap.c	9
src/backend/catalog/objectaddress.c	14
src/backend/catalog/pg_aggregate.c	7
src/backend/catalog/pg_conversion.c	7
src/backend/catalog/pg_namespace.c	13
src/backend/catalog/pg_operator.c	7
src/backend/catalog/pg_proc.c	7
src/backend/catalog/pg_type.c	8
src/backend/commands/Makefile	3
src/backend/commands/comment.c	6
src/backend/commands/extension.c	688 +++++++++++++++++++++++++++++++++
src/backend/commands/foreigncmds.c	19
src/backend/commands/functioncmds.c	7
src/backend/commands/opclasscmds.c	13
src/backend/commands/proclang.c	7
src/backend/commands/tsearchcmds.c	25 +
src/backend/nodes/copyfuncs.c	22 +
src/backend/nodes/equalfuncs.c	18
src/backend/parser/gram.y	51 ++
src/backend/tcop/utility.c	27 +
src/backend/utils/adt/genfile.c	193 +++++++++
src/backend/utils/init/postinit.c	3
src/backend/utils/misc/Makefile	2
src/backend/utils/misc/cfparser.c	113 +++++
src/backend/utils/misc/guc-file.l	26 -
src/backend/utils/misc/guc.c	160 ++++++-
src/bin/pg_dump/common.c	6
src/bin/pg_dump/pg_dump.c	520 ++++++++++++++++++++++—
src/bin/pg_dump/pg_dump.h	10
src/bin/pg_dump/pg_dump_sort.c	7
src/bin/psql/command.c	3
src/bin/psql/describe.c	45 ++
src/bin/psql/describe.h	3
src/bin/psql/help.c	1
src/include/catalog/dependency.h	1
src/include/catalog/indexing.h	6
src/include/catalog/pg_extension.h	61 ++
src/include/catalog/pg_proc.h	13
src/include/catalog/toasting.h	1
src/include/commands/extension.h	54 ++
src/include/nodes/nodes.h	2
src/include/nodes/parsenodes.h	20
src/include/parser/kwlist.h	1
src/include/utils/builtins.h	4
src/include/utils/cfparser.h	18
src/include/utils/guc.h	11
src/makefiles/pgxs.mk	21 -

</span> 7	{}
49	{49}
97	{49,97}
130	{49,97,130}
10	{49,97,130,10}
1	{49,97,130,10,1}

</span> gb	H9	london	2
jp	40	tokyo	414
us	NY	<span style="color: #729fcf;">new</span> york	131
ca	08	london	16
ru	48	moscow	245
fr	A8	paris	16
us	IL	chicago	13

</span> hstore	1.0	public	data type for storing sets of (key, value) pairs
plpgsql	1.0	pg_catalog	PL/pgSQL procedural language

0	10101100101111010001100011011011	acbd18db
1	01001100110000101111100001011100	4cc2f85c
2	11101101111011110110010101001111	edef654f
3	11001100110001001010010011011000	ccc4a4d8

Muse setup revised