tail -f /dev/dim http://tapoueh.org/index.html Dimitri Fontaine's blog en-us Emacs Muse pgbouncer munin plugin http://tapoueh.org/blog/2011/11/16-pgbouncer-munin.html It seems that if you search for a munin plugin for pgbouncer it's easy enough to reach an old page of mine with an old version of my plugin, and a broken link. Let's remedy that by publishing here the newer version of the plugin. To be honest, I though it already made its way into the official munin 1.4 set of plugins, but I've not been following closely enough.

As the plugin is 300 lines of python code, it's not a good idea to just inline it here, so please grab it at pgbouncer_.

You might need to know that the script name once installed should follow the form pgbouncer_dbname_stats_requests or pgbouncer_dbname_pools, where of course dbname can contain any number of _ characters. This script supports quite old versions of pgbouncer that didn't accept the normal pq protocol, you did have to use psql to have any chance of getting the data from a script, you couldn't then just use a PostgreSQL driver such as psycopg2.

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 16 Nov 2011 14:00:00 +0100 http://tapoueh.org/blog/2011/11/16-pgbouncer-munin.html Extensions en simple SQL http://tapoueh.org/blog/2011/10/31-extensions-sql.html La conférence européenne à Amsterdam était un très bon évènement de la communauté, avec une organisation impeccable dans un hôtel accueillant. J'ai eu le plaisir d'y parler des extensions et de leur usage dans le cadre du développement applicatif « interne », sous le titre Extensions are good for business logic.

L'idée de ma présentation, que la plupart d'entre vous a loupé je suppose (en tout cas je n'avais qu'une petite poignée de français dans la salle, et j'espère avoir des lecteurs qui n'étaient pas à Amsterdam), l'idée est d'utiliser les mécanismes offerts par les extensions afin de maintenir le code PL que vous utilisez en production.

Il s'agit la plupart du temps de procédures qui implémentent une partie de la logique métier de vos applications, mais si proche des données que cela termine en base directement : c'est une bonne chose, en particulier depuis PostgreSQL 9.1. Cette version propose en effet une gestion assez complète des extensions.

Il s'agit de réaliser un empaquetage de vos procédures en suivant la documentation en ligne et son chapitre 35.15. Empaqueter des objets dans une extension. Une fois cela fait, il est alors possible de déployer votre ensemble de procédure stockée avec la commande CREATE EXTENSION mesprocs;, et ensuite la commande psql \dx vous permet de lister les extensions installées et leur numéro de version.

Les mises à jours sont également gérées avec une commande SQL dédiée, il s'agit alors de ALTER EXTENSION mesprocs UPDATE [TO version];. Il suffit de fournir des scripts intermédiaires nommés par exemple mesprocs--1.0--1.1.sql et mesprocs--1.1--1.2.sql et PostgreSQL saura comment passer de 1.0 à 1.1.

Voilà, vous savez presque tout de ma présentation à Amsterdam et vous pouvez retrouver le reste sur le support proposé en début d'article. Bien sûr je n'ai pas reproduit ici les questions intéressantes qui m'ont été posées, une bonne partie d'entre elles sont venues enrichir ma liste de Noël pour les extensions. Si vous voulez être sûr de trouver cela sous votre sapin, cependant, le meilleur moyen est encore de m'en parler : sponsoriser les développement Open Source est une belle démarche :)

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 31 Oct 2011 14:22:00 +0100 http://tapoueh.org/blog/2011/10/31-extensions-sql.html Back From Amsterdam http://tapoueh.org/blog/2011/10/26-back-from-amsterdam.html Another great conference took place last week, PostgreSQL Conference Europe 2011 was in Amsterdam and plenty of us PostgreSQL geeks were too. I attended to lot of talks and did learn some more about our project, its community and its features, but more than that it was a perfect occasion to meet with the community.

Dave Page talked about SQL/MED under the title PostgreSQL at the center of your dataverse and detailed what to expert from a Foreign Data Wrapper in PostgreSQL 9.1, then how to write your own. Wherever you are currently managing your data, you can easily enough make it so that PostgreSQL integrates them by means of fetching them to answer your queries. Which means real time data federating: you don't copy data around, you remote access them when executing the query.

I might need to come up with new Foreign Data Wrappers in a not too distant future, now that I better grasp how much work it really is to do that, it appears to be a good migration strategy too:

  INSERT INTO real.table
       SELECT * FROM foreign.table;

Another discovery is that apparently PLv8 is ready for public consumption. Using it can lead to Heralding the Death of NoSQL, so use it with care.

In the presentation of Synchronous Replication and Durability Tuning we mainly saw that mixing synchronous and asynchronous transactions in your application is the key to real performances across the ocean, as the speed of the light is not infinite. From Baltimore to Amsterdam the latency can not be better than 100ms and that's not the same as instant nowadays.

Then again, depending on the number of concurrent queries to sync over the ocean link, the experimental setup was able to achieve several thousands of queries per second, which is validating the model we picked for sync rep and its implementation.

If you want to read the slides again at home, or if you could not be there for some reason, then most of the talks are now available online at the PostgreSQL Conference Europe Talks 2011 wiki page.

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 26 Oct 2011 10:08:00 +0200 http://tapoueh.org/blog/2011/10/26-back-from-amsterdam.html Implementing backups http://tapoueh.org/blog/2011/10/12-backup-strategy.html I've been asked about my opinion on backup strategy and best practices, and it so happens that I have some kind of an opinion on the matter.

I tend to think best practice here begins with defining properly the backup plan you want to implement. It's quite a complex matter, so be sure to ask yourself about your needs: what do you want to be protected from?

The two main things to want to protect from are hardware loss (crash disaster, plane in the data center, fire, water flood, etc) and human error (UPDATE without a where clause). Replication is an answer to the former, archiving and dumps to the latter. You generally need both.

Often enough “backups” include WAL archiving and shipping and nightly or weekly base backups, with some retention and some scripts or procedures ready to setup Point In Time Recovery and recover some data without interfering with the WAL archiving and shipping. Of course with PostgreSQL 9.0 and 9.1, the WAL Shipping can be implemented with streaming replication and you can even have a Hot Standby. But for backups you still want archiving.

Mostly I still implement pg_dump -Fc nightly backups with a custom retention (for example, 1 backup a month kept 2 years, 1 backup a week kept 6 or 12 months, 1 backup a night kept 1 to 2 weeks), when the database size allow the pg_dump run to remain constrained in the maintenance window, if any.

Don't forget that while pg_dump runs, you can't roll out DDL changes to the production system any more, so you want to be careful about this maintenance window thing. When you have one.

Physical backups are not locking rollouts away, but they often suck a good deal of the IO bandwidth so you need to pick up a right timing to do them. That's how you can get to once a week base backup and WAL archiving.

If you can't pg_dump production, maybe you can have automated restore jobs from the physical backups that you then pg_dump -Fc, so that you still have that. That can come up handy, really: you can't test your major upgrade out of a physical backup.

Also, obviously, never consider your backup strategy implemented until you have either automated restores in place or a regular schedule to exercise them (staging instances, devel instances).

Then as far as the practical tools go, I tend to think that pg_staging is worth its installation complexity, and for WAL archiving and base backup I recommend walmgr from Skytools, that's a very handy tool. When using PostgreSQL 9.0 or 9.1, consider using walmgr3 so that it's behaving nice alongside streaming replication.

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 12 Oct 2011 22:22:00 +0200 http://tapoueh.org/blog/2011/10/12-backup-strategy.html Extensions, applications http://tapoueh.org/blog/2011/10/10-extensions-applicatives.html La conférence PostgreSQL annuelle en Europe a lieu la semaine prochaine à Amsterdam, et j'espère que vous avez déjà vos billets, car cette édition s'annonce comme un très bon millésime !

Je présenterai donc comment utiliser les extensions, le titre en anglais est Extensions are good for business logic, et l'idée est de voir comment exploiter les extensions afin de mieux gérer vos mises à jours en bases de données.

Le cycle de vie des bases de données en production inclue souvent l'utilisation d'une base de développement où le schéma évolue au rythme des besoins des développeurs, et de temps en temps on consolide une partie de ces modifications (dans des rollouts, scripts contenant principalement des DDL) afin de les déployer en production — si possible avec une étape intermédiaire en préproduction, tout de même.

Savoir ce qui est déployé en développement et comment en retirer le script à jouer en production peut être parfois fastidieu. Quand ce n'est pas le cas, c'est que le travail a été fait en amont, ce qui est le signe d'une bonne organisation, avec les surcoûts que l'on peut imaginer.

Les extensions telles que présentes dans PostgreSQL 9.1 vous permettent de mieux gérer ce genre de cas, en optimisant le surcoût : il ne disparaît pas, mais devient opérationnel plutôt que de rester une charge d'organisation.

Allez, je vous laisse maintenant, je dois me préparer pour la conférence :)

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 10 Oct 2011 10:35:00 +0200 http://tapoueh.org/blog/2011/10/10-extensions-applicatives.html Scaling Stored Procedures http://tapoueh.org/blog/2011/10/06-scaling-with-stored-procedures.html In the news recently stored procedures where used as an excuse for moving away logic from the database layer to application layer, and to migrate away from a powerful technology to a simpler one, now that there's no logic anymore in the database.

It's not the way I would typically approach scaling problems, and apparently I'm not alone on the Stored Procedures camp. Did you read this nice blog post Mythbusters: Stored Procedures Edition already? Well it happens in another land that where my comfort zone is, but still has some interesting things to say.

I won't try and address all of the myths they attack in a single article. Let's pick the scalability problems, the two of them I think about are code management and performances. We are quite well equiped for that in PostgreSQL, really.

For code maintainance we now have PostgreSQL Extensions, which allows you to pack all your procedures into separate extensions, and to maintain a version number and upgrade procedures for each of them. You can handle separate rollouts in development for going from 1.12 to 1.13 then 1.14 and after the developers tested it more completely and changed their mind again on the best API they want to work with, 1.15 which is stamped ok for production. At this point, ALTER EXTENSION UPGRADE will happily apply all the rollouts in sequence to upgrade from 1.12 straight to 1.15 in one go. And if you prefer to bake a special careful script to handle that big jump, you also can provide a specific extension--1.12--1.15.sql script.

Of course you're managing all those files with your favorite SCM, to answer to some other myth from the blog reference we are loosely following.

I wanted to talk about the other side of the scalability problem, which is the operations side of it. What happens when you need to scale the database in terms of its size and level of concurrent activity? PostgreSQL earned a very good reputation at being able to scale-up, what about scaling-out? Certainly, now that you're all down into Stored Procedure, it's going to be a very bad situation?

Well, in fact, you're then in a very good position here, thanks to PLproxy. This extension is a custom procedural language whose job is to handle a cluster of database shards that all expose the same PL API, and it's very good at doing that.

Stored Procedures are a very good tool to have, be sure to get comfortable enough with them so that you can choose exactly when to use them. If you're not sure about that, we at 2ndQuadrant will be happy to help you there!

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 06 Oct 2011 18:23:00 +0200 http://tapoueh.org/blog/2011/10/06-scaling-with-stored-procedures.html See you in Amsterdam http://tapoueh.org/blog/2011/10/04-see-you-in-Amsterdam.html The next PostgreSQL conference is approaching very fast now, I hope you have your ticket already: it's a very promissing event! If you want some help in deciding whether to register or not, just have another look at the schedule. Pick the talks you want to see. It's hard, given how packed with good ones the schedule is. When you're mind is all set, review the list. Registered?

I'll be presenting another talk about extensions, but this time I've geared up to use cases, with Extensions are good for business logic. The idea is not to talk about how to make PostgreSQL play fair with extensions including at dump and restore times, that's already done and I've been talking only too much about it. The idea this time is to figure out how much you get from this feature.

If you ever felt like something is missing in your processes between pushing rollouts in devel environments and refining them as developers are testing and preparing something for the live databases, then we have something for you here. Including how to easily compare state between production and development, but without having to guess or reverse engineer anything.

Yeah, extensions are all about getting even more professional! A great tool you'll be happy to master!

And now I need to prepare a damn good slide deck, right? See you there! :)

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 04 Oct 2011 14:25:00 +0200 http://tapoueh.org/blog/2011/10/04-see-you-in-Amsterdam.html PostgreSQL à Amsterdam http://tapoueh.org/blog/2011/09/27-pgconf-eu.html Dans moins d'un mois se tient la conférence européenne PostgreSQL, pgconf.eu. Il s'agit de quatre jours consacrés à votre SGBD préféré, où vous pourrez rencontrer la communauté européenne, consituée d'utilisateurs, d'entreprises de toutes tailles, de développeurs, de participants en tout genre.

C'est l'endroit où aller pour apprendre comment le projet fonctionne, comprendre les impacts des nouvelles versions sur votre architecture, avoir une discussion technique pointue sur cette fonctionalité que vous voudriez voir arriver dans la prochaine version, ou simplement vous rendre compte de l'énergie formidable qui est insuflée dans ce projet !

Évidemment 2ndQuadrant sera de la partie, nous présenterons plusieurs de nos contributions PostgreSQL 9.1. Cela commencera avec la formation d'une journée complète de Greg, Performance From Start to Crash : si vous voulez apprendre comment aborder les performances d'un serveur PostgreSQL par le leader international du domaine, auteur du livre Bases de données PostgreSQL 9.0, réservez vite votre place !

Les présentation au format classique commencent le lendemain, et en trois jours la liste des présentation de notre équipe 2ndQuadrant est assez copieuse. Voyons cela.

Nous commençons avec Migration to PostgreSQL - a holistic view par Harald Armin Massa, qui propose un point de vue intéressant sur les raisons qui retiennent certaines migrations. Ensuite Gianni Ciolli présentera Look Out The Window Functions (and free your SQL) ou comment résoudre simplement des problèmes complexes lorsque l'on dispose d'outils avancés.

Une autre présentation à ne pas rater, Synchronous Replication and Durability Tuning détaille comment profiter au mieux de PostgreSQL 9.1 afin d'obtenir les garanties de durabilité des données souhaitées dans votre application. Et cette présentation est animée par Greg Smith et Simon Riggs. Ce dernier a développé la réplication synchrone, et Hot Standby avant cela. Vous ne trouverez personne au monde mieux placé pour faire cette présentation !

Les deux prochaines présentation de nos experts PostgreSQL, en continuant notre lecture du programme de pgconf.eu dans l'ordre, ont lieu au même moment. Le choix ne sera pas facile entre Improving VACUUM Suction par Greg à nouveau, et une comparaison de londiste 3 et slony 2.1 par Cédric Villemain, en français.

À suivre, Extensions are good for business logic que je vous présenterai moi-même, vous pouvez voir ma présentation sur la fiche qui porte mon nom : Dimitri Fontaine. Il s'agit d'une présentation en anglais qui détaille comment utiliser les extensions dans le cadre de la maintenance de la partie procédures stockées d'une application.

Et pour finir le deuxième jour des conférences 2ndQuadrant, vous pourrez apprendre avec Gianni comment Debugging complex SQL queries with writable CTEs, une fonctionnalité contribuée au projet par un autre consultant 2ndQuadrant, Marko Tiikkaja.

Et il reste encore une journée ! Nous ne mentons pas en disant que le programme est complet ! Le dernier jour de la conférence n'est pas le moins intéressant, j'espère que vous aurez su garder un peu d'énergie pour suivre…

Simon Riggs qui présentera sa vision de la PostgreSQL Roadmap pour les prochaines années. Ce n'est bien sûr que sa vision personnelle, mais lorsque l'on fait le bilan de ces 7 dernières années de contributions à PostgreSQL, on voit à quel point son opinion personnelle peut avoir du poids dans le développement du projet.

À suivre, la présentation de Greg sur son sujet de prédilection : Bottom-up Database Benchmarking. Tout ce que vous avez toujours voulu savoir sur les mesures de performances de vos bases de données, sans jamais oser le demander. Quelque chose dans ce style en tout cas :)

Bien sûr d'autres présentations sont disponibles et retiendront votre attention, ce billet vous présente seulement celles qui seront données par les experts PostgreSQL de 2ndQuadrant. En vous souhaitant bonne conférence à tous, j'espère avoir le plaisir de vous retrouver à Amsterdam le mois prochain !

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 27 Sep 2011 11:10:00 +0200 http://tapoueh.org/blog/2011/09/27-pgconf-eu.html Skytools3: walmgr http://tapoueh.org/blog/2011/09/21-skytools-walmgr-part-1.html Let's begin the Skytools 3 documentation effort, which is long overdue. The code is waiting for you over at github, and is stable and working. Why is it still in release candidate status, I hear you asking? Well because it's missing updated documentation.

WalMgr is the Skytools component that manages WAL shipping for you, and archiving too. It knows how to prepare your master and standby setup, how to take a base backup and push it to the standby's system, how to archive (at the satndby) master's WAL files as they are produced and have the standby restore from this archive.

What's new in walmgr from Skytools 3 is its support for Streaming Replication that made its way into PostgreSQL 9.0 and is even more useful in PostgreSQL 9.1 (better monitoring, synchronous replication option).

Getting ready

Now, I'm using debian here, and a build virtual machine where I'm doing the backporting work. As PostgreSQL 9.1 is now out, let's use that.

:~$ pg_lsclusters
Version Cluster   Port Status Owner    Data directory
8.4     main      5432 online postgres /var/lib/postgresql/8.4/main ...
9.0     main      5433 online postgres /var/lib/postgresql/9.0/main ...
9.1     main      5434 online postgres /var/lib/postgresql/9.1/main ...

After some editing of the configuration files (enabling hot standby and switching pg_hba.conf to trust for the sake of this example), we can see that the cluster is ready to be abused:

:~$ sudo pg_ctlcluster 9.1 main restart
:~$ psql --cluster 9.1/main  -U postgres \
-c "select name, setting from pg_settings where name in ('max_wal_senders', 'wal_level')"
      name       |   setting
-----------------+-------------
 max_wal_senders | 1
 wal_level       | hot_standby
(2 rows)

:~$ sudo mkdir -p /etc/walshipping/9.1/main /var/lib/postgresql/walshipping
:~$ sudo chown -R postgres:postgres /etc/walshipping /var/lib/postgresql/walshipping

:~$ ssh-keygen -t dsa
:~/.ssh$ cp id_dsa.pub authorized_keys
:~$ ssh localhost

So the order of operations is to prepare a standby, then have it restore from the archives, then activate the wal streaming and check that the setup allows the standby to switch back and forth between the streaming and the archives.

Setting walmgr

To prepare the standby, we will do a base backup of the master. That step is handled by walmgr, so we first need to set it up. Here's the sample master.ini file:

[walmgr]
job_name             = wal-master
logfile              = /var/log/postgresql/%(job_name)s.log
pidfile              = /var/run/postgresql/%(job_name)s.pid
use_skylog           = 0

master_db            = port=5434 dbname=template1
master_data          = /var/lib/postgresql/9.1/main/
master_config        = /etc/postgresql/9.1/main/postgresql.conf
master_bin           = /usr/lib/postgresql/9.1/bin

# set this only if you can afford database restarts during setup and stop.
master_restart_cmd   = pg_ctlcluster 9.1 main restart

slave = 127.0.0.1
slave_config = /etc/walshipping/9.1/main/standby.ini

walmgr_data          = /var/lib/postgresql/walshipping/9.1/main
completed_wals       = %(walmgr_data)s/logs.complete
partial_wals         = %(walmgr_data)s/logs.partial
full_backup          = %(walmgr_data)s/data.master
config_backup        = %(walmgr_data)s/config.backup

# syncdaemon update frequency
loop_delay           = 10.0
# use record based shipping available since 8.2
use_xlog_functions   = 0

# pass -z to rsync, useful on low bandwidth links
compression          = 0

# keep symlinks for pg_xlog and pg_log
keep_symlinks        = 1

# tell walmgr to set wal_level to hot_standby during setup
#hot_standby          = 1

# periodic sync
#command_interval     = 600
#periodic_command     = /var/lib/postgresql/walshipping/periodic.sh

And the /etc/walshipping/9.1/main/standby.ini companion:

[walmgr]
job_name             = wal-standby
logfile              = /var/log/postgresql/%(job_name)s.log
use_skylog           = 0

slave_data           = /var/lib/postgresql/9.1/standby
slave_bin            = /usr/lib/postgresql/9.1/bin
slave_stop_cmd       = pg_ctlcluster 9.1 standby stop
slave_start_cmd      = pg_ctlcluster 9.1 standby start
slave_config_dir     = /etc/postgresql/9.1/standby/

walmgr_data          = /var/lib/postgresql/walshipping/9.1/main
completed_wals       = %(walmgr_data)s/logs.complete
partial_wals         = %(walmgr_data)s/logs.partial
full_backup          = %(walmgr_data)s/data.master
config_backup        = %(walmgr_data)s/config.backup

backup_datadir       = no
keep_backups         = 0
# archive_command =

# primary database connect string for hot standby -- enabling
# this will cause the slave to be started in hot standby mode.
primary_conninfo     = host=127.0.0.1 port=5434 user=postgres

And let's get started:

:~$ cp standby.ini /etc/walshipping/9.1/main/

:~$ walmgr3 -v master.ini setup
2011-09-21 16:57:05,685 30450 INFO Configuring WAL archiving
2011-09-21 16:57:05,687 30450 DEBUG found 'archive_mode' in config -- enabling it
2011-09-21 16:57:05,687 30450 DEBUG found 'wal_level' in config -- setting to 'archive'
2011-09-21 16:57:05,688 30450 DEBUG modifying configuration: {'archive_mode': 'on', 'wal_level': 'archive', 'archive_command': '/usr/bin/walmgr3 /var/lib/postgresql/master.ini xarchive %p %f'}
2011-09-21 16:57:05,688 30450 DEBUG found parameter archive_mode with value ''off''
2011-09-21 16:57:05,690 30450 DEBUG found parameter wal_level with value ''minimal''
2011-09-21 16:57:05,690 30450 DEBUG found parameter archive_command with value ''''
2011-09-21 16:57:05,691 30450 INFO Restarting postmaster
2011-09-21 16:57:05,691 30450 DEBUG Execute cmd: 'pg_ctlcluster 9.1 main restart'
2011-09-21 16:57:09,404 30450 DEBUG Execute cmd: 'ssh' '-Tn' '-o' 'Batchmode=yes' '-o' 'StrictHostKeyChecking=no' '127.0.0.1' '/usr/bin/walmgr3' '/etc/walshipping/9.1/main/standby.ini' 'setup'
2011-09-21 16:57:09,712 30450 INFO Done

postgres@squeeze64:~$ walmgr3 master.ini backup
2011-09-21 17:00:17,259 30702 INFO Backup lock obtained.
2011-09-21 17:00:17,277 30692 INFO Execute SQL: select pg_start_backup('FullBackup'); [port=5434 dbname=template1]
2011-09-21 17:00:17,791 30712 INFO Removing expired backup directory: /var/lib/postgresql/walshipping/9.1/main/data.master
2011-09-21 17:00:18,200 30692 INFO Checking tablespaces
2011-09-21 17:00:18,202 30692 INFO pg_log does not exist, skipping
2011-09-21 17:00:18,259 30692 INFO Backup conf files from /etc/postgresql/9.1/main
2011-09-21 17:00:18,590 30731 INFO First useful WAL file is: 000000010000000200000092
2011-09-21 17:00:19,901 30759 INFO Backup lock released.
2011-09-21 17:00:19,919 30692 INFO Full backup successful

:~$ walmgr3 /etc/walshipping/9.1/main/standby.ini listbackups

List of backups:

Backup set      Timestamp                Label       First WAL
--------------- ------------------------ ----------- ------------------------
data.master     2011-09-21 17:00:17 CEST FullBackup  000000010000000200000092

Following articles will show how to manage that archive and how to go from that to an Hot Standby fed by either Streaming Replication or Archives.

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 21 Sep 2011 17:21:00 +0200 http://tapoueh.org/blog/2011/09/21-skytools-walmgr-part-1.html el-get-3.1 http://tapoueh.org/blog/2011/09/16-el-get-3.1.html The el-get project releases its new stable version, 3.1. This new release fixes bugs, add a host of new recipes (we have 420 of them and counting) and some nice new features too. You really want to upgrade.

New features

Among the features you will find dependencies management and M-x el-get-list-packages, that you should try as soon as possible. Of course, don't miss M-x el-get-self-update that eases the process somehow.

This shows the result of M-x el-get-list-packages. The packages that don't have a description are the one from emacswiki that doesn't provide a listing of the filename and the first line of the file (it usually follows the format ;;; filename.el --- description here). As we don't want to mirror the website just to be able to provide descriptions, we just don't have them now.

Another nice new feature, contributed by a user that wanted to self-learn elisp, is the el-get-user-package-directory support. Just place in there some init-my-package.el files, and when el-get wants to init the my-package package, it will load that file for you. That helps managing your setup, and I'm already using that in my own ~/.emacs.d/ repository.

Upgrading

The upgrading is to be done with some care, though, because you need to edit your packaging setup. The el-get-sources variable used to be both where to setup extra recipes and the list of packages you want to have installed, and several people rightfully insisted that I should change that. I've been slow to be convinced, but there it is, they were right.

So now, el-get works from the current status of packages and will init all those packages you have installed. Which means that you just M-x el-get-install a package and don't think about it anymore. If you need to override this behavior, it's still possible to do so by specifying the whole list of packages you want initialized (and installed if necessary) on the (el-get 'sync ...) call.

That later setup is useful if you want to share your el-get selection on several machines.

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 16 Sep 2011 14:13:00 +0200 http://tapoueh.org/blog/2011/09/16-el-get-3.1.html PostgreSQL 9.1 http://tapoueh.org/blog/2011/09/19-sortie-de-9.1.html PostgreSQL 9.1 est dans les bacs ! Vous n'avez pas encore cette nouvelle version en production ? Pas encore évalué pourquoi vous devriez envisager de migrer à cette version ? Il existe beaucoup de bonnes raisons de passer à cette version, et peu de pièges.

Nous commençons à lire des articles qui reprennent la nouvelle dans la presse française, et j'ai le plaisir de mentionner celui de programmez.com qui annonce « un système d'extensions inégalé ». En tant que développeur des Extensions dans PostgreSQL, je ne peux qu'être non seulement d'accord avec eux, mais aussi flatté :)

Bons tests à tous, et bonne mises à jour pour les plus chanceux !

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 14 Sep 2011 10:00:00 +0200 http://tapoueh.org/blog/2011/09/19-sortie-de-9.1.html Éviter les injections SQL http://tapoueh.org/blog/2011/09/07-eviter-les-injections-sql.html Nous avons parlé la dernière fois les règles d'échappement de chaînes avec PostgreSQL, et mentionné qu'utiliser ces techniques afin de protéger les données insérées dans les requêtes SQL n'était pas une bonne idée dans la mesure où PostgreSQL offre une fonctionnalité bien plus adaptée.

Nous faisons face ici à un problème de sécurité très bien décrit dans le billet humoristique de Little Boby Tables, dont je vous recommande la lecture. L'idée est simple, la mise en place de contre mesure fourmille de pièges subtils, à moins d'utiliser la solution décrite ci-après.

Lorsque l'on envoie une requête SQL à PostgreSQL, celle-ci contient pêle-mêle un mélange de mots-clés SQL et de données utilisateurs. Dans la requête SELECT colname FROM table WHERE pk = 1234; l'élément 1234 est une donnée fournie à PostgreSQL. Lorsque l'on utilise d'autre types de données, on va parler de litéral, qui peut être ou non décoré. Un exemple ?

=# SELECT 'undecorated literal', pg_typeof('undecoreted literal'),
          date 'today', pg_typeof(date 'today');
      ?column?       | pg_typeof |    date    | pg_typeof
---------------------+-----------+------------+-----------
 undecorated literal | unknown   | 2011-09-07 | date
(1 row)

Outre l'aspect types de données (un litéral non décoré est de type unknown jusqu'à ce qu'une opération force son type, c'est ce qui permet d'avoir du polymorphisme dans PostgreSQL), nous voyons ici que PostgreSQL doit faire la différence entre le SQL lui-même et les paramètres qui le composent. Il sait bien sûr faire cela, il suffit d'encadrer les valeurs dans des simples guillemets ou bien d'utiliser la notation dite de dollar quoting. Mais si l'on ne prend pas de précautions, l'utilisateur peut terminer la séquence d'échappements depuis le champ de saisie du formulaire…

libpq est la librairie standard cliente de PostgreSQL et fourni des API de connexion et propose une fonction PGexecParams. Cette fonction expose un mécanisme disponible dans le protocole de communication de PostgreSQL lui-même : il est possible de faire parvenir le SQL et les données qu'il contient dans deux parties différentes du messages plutôt que de les mélanger. Ainsi, le serveur n'a plus du tout à deviner où commencent et où terminent les données dans la requête, il lui suffit de regarder dans le tableau séparé contenant les données quand il en a besoin.

Terminées les injections SQL !

Note : cette fonction est exposée dans la plupart des pilotes de connexion, et même en PHP, dont la popularité et l'exposition me poussent à donner une référence plus précise : utilisez pg_query_params, son intérêt n'est pas simplement syntaxique, il va jusque dans la définition des échanges de données entre le client (votre code PHP) et le serveur (PostgreSQL).

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 07 Sep 2011 11:36:00 +0200 http://tapoueh.org/blog/2011/09/07-eviter-les-injections-sql.html Éviter les injections SQL http://tapoueh.org/blog/2011/09/07-requete-parametree.html Nous avons parlé la dernière fois les règles d'échappement de chaînes avec PostgreSQL, et mentionné qu'utiliser ces techniques afin de protéger les données insérées dans les requêtes SQL n'était pas une bonne idée dans la mesure où PostgreSQL offre une fonctionnalité bien plus adaptée.

Nous faisons face ici à un problème de sécurité très bien décrit dans le billet humoristique de Little Boby Tables, dont je vous recommande la lecture. L'idée est simple, la mise en place de contre mesure fourmille de pièges subtils, à moins d'utiliser la solution décrite ci-après.

Lorsque l'on envoie une requête SQL à PostgreSQL, celle-ci contient pêle-mêle un mélange de mots-clés SQL et de données utilisateurs. Dans la requête SELECT colname FROM table WHERE pk = 1234; l'élément 1234 est une donnée fournie à PostgreSQL. Lorsque l'on utilise d'autre types de données, on va parler de litéral, qui peut être ou non décoré. Un exemple ?

=# SELECT 'undecorated literal', pg_typeof('undecoreted literal'),
          date 'today', pg_typeof(date 'today');
      ?column?       | pg_typeof |    date    | pg_typeof
---------------------+-----------+------------+-----------
 undecorated literal | unknown   | 2011-09-07 | date
(1 row)

Outre l'aspect types de données (un litéral non décoré est de type unknown jusqu'à ce qu'une opération force son type, c'est ce qui permet d'avoir du polymorphisme dans PostgreSQL), nous voyons ici que PostgreSQL doit faire la différence entre le SQL lui-même et les paramètres qui le composent. Il sait bien sûr faire cela, il suffit d'encadrer les valeurs dans des simples guillemets ou bien d'utiliser la notation dite de dollar quoting. Mais si l'on ne prend pas de précautions, l'utilisateur peut terminer la séquence d'échappements depuis le champ de saisie du formulaire…

libpq est la librairie standard cliente de PostgreSQL et fourni des API de connexion et propose une fonction PGexecParams. Cette fonction expose un mécanisme disponible dans le protocole de communication de PostgreSQL lui-même : il est possible de faire parvenir le SQL et les données qu'il contient dans deux parties différentes du messages plutôt que de les mélanger. Ainsi, le serveur n'a plus du tout à deviner où commencent et où terminent les données dans la requête, il lui suffit de regarder dans le tableau séparé contenant les données quand il en a besoin.

Terminées les injections SQL !

Note : cette fonction est exposée dans la plupart des pilotes de connexion, et même en PHP, que la popularité et l'exposition me poussent à donner une référence plus précise : utilisez pg_query_params, son intérêt n'est pas simplement syntaxique, il va jusque dans la définition des échanges de données entre le client (votre code PHP) et le serveur (PostgreSQL).

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 07 Sep 2011 11:36:00 +0200 http://tapoueh.org/blog/2011/09/07-requete-parametree.html PostgreSQL and debian http://tapoueh.org/blog/2011/09/05-apt-postgresql-org.html After talking about it for a very long time, work finally did begin! I'm talking about the apt.postgresql.org build system that will allow us, in the long run, to propose debian versions of binary packages for PostgreSQL and its extensions, compiled for a bunch of debian and ubuntu versions.

We're now thinking to support the i386 and amd64 architectures for lenny, squeeze, wheezy and sid, and also for maverick and natty, maybe oneiric too while at it.

It's still the very beginning of the effort, and it was triggered by the decision to move sid to 9.1. While it's a good decision in itself, I still hate to have to pick only one PostgreSQL version per debian stable release when we have all the technical support we need to be able to support all stable releases that upstream is willing to maintain. If you've been living under a rock, or if you couldn't care less about debian choices, the problem here for debian is ensuring security (and fixes) updates for PostgreSQL — they promise they will handle the job just fine in the social contract, and don't want to have to it without support from PostgreSQL if a debian stable release contains a deprecated PostgreSQL version.

That opens the door for PostgreSQL community to handle the packaging of its solutions as a service to its debian users. We intend to open with support for 8.4, 9.0 and 9.1, and maybe 8.3 too, as Christoph Berg is doing good progress on this front. See, it's teamwork here!

We still have more work to do, and setting up the build environment so that we are able to provide the packages for so much targets will indeed be interesting. Getting there, a step after another.

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 05 Sep 2011 17:14:00 +0200 http://tapoueh.org/blog/2011/09/05-apt-postgresql-org.html pg_restore -L & pg_staging http://tapoueh.org/blog/2011/08/29-pgstaging-and-pgrestore-listing.html On the PostgreSQL Hackers mailing lists, Andrew Dunstan just proposed some new options for pg_dump and pg_restore to ease our lives. One of the answers was talking about some scripts available to exploit the pg_restore listing that you play with using options -l and -L, or the long name versions --list and --use-list. The pg_staging tool allows you to easily exploit those lists too.

The pg_restore list is just a listing of one object per line of all objects contained into a custom dump, that is one made with pg_dump -Fc. You can then tweak this listing in order to comment out some objects (prepending a ; to the line where you find it), and give your hacked file back to pg_restore --use-list so that it will skip them.

What's pretty useful here, among other things, is that a table will have in fact more than one line in the listing. One is for the TABLE definition, another one for the TABLE DATA. So that pg_staging is able to provide you with options for only restoring some schemas, some schemas_nodata and even some tablename_nodata_regexp, to use directly the configuration options names.

How to do a very simple exclusion of some table's data when restoring a dump, will you ask me? There we go. Let's first prepare an environment, where I have only a PostgreSQL server running.

$ git clone git://github.com/dimitri/pg_staging.git
$ git clone git://github.com/dimitri/pgloader.git
$ for s in */*.sql; do psql -f $s; done
$ pg_dump -Fc > pgloader.dump

Now I have a dump with some nearly random SQL objects in it, let's filter out the tables named reformat and parallel from that. We will take the sample setup from the pg_staging project. Going the quick route, we will not even change the default sample database name that's used, which is postgres. After all, the catalog command of pg_staging that we're using here is a developer command, you're supposed to be using pg_staging for a lot more services that just this one.

$ cp pg_staging/pg_staging.ini .
$ (echo "schemas = public";
   echo "tablename_nodata_regexp = parallel,reformat") \
  >> pg_staging.ini
$ echo "catalog postgres pgloader.dump" \
   | python pg_staging/pg_staging.py -c pg_staging.ini
 ; Archive created at Mon Aug 29 17:17:49 2011
 ;
 ; [EDITED OUTPUT]
 ;
 ; Selected TOC Entries:
 ;
3; 2615 2200 SCHEMA - public postgres
1864; 0 0 COMMENT - SCHEMA public postgres
1536; 1259 174935 TABLE public parallel dimitri
1537; 1259 174943 TABLE public partial dimitri
1538; 1259 174951 TABLE public reformat dimitri
;1853; 0 174935 TABLE DATA public parallel dimitri
1854; 0 174943 TABLE DATA public partial dimitri
;1855; 0 174951 TABLE DATA public reformat dimitri
1834; 2606 174942 CONSTRAINT public parallel_pkey dimitri
1836; 2606 174950 CONSTRAINT public partial_pkey dimitri
1838; 2606 174955 CONSTRAINT public reformat_pkey dimitri

We can see that the objects indeed are skipped, now how to really go about the pg_restore is like that:

$ createdb foo
$ echo "catalog postgres pgloader.dump" \
 |python pg_staging/pg_staging.py -c pg_staging.ini > short.list
$ pg_restore -L short.list -d foo pgloader.dump

The little bonus with using pg_staging is that when filtering out a schema it will track all tables and triggers from that schema, and also the functions used in the trigger definition. Which is not as easy as it sounds, believe me!

The practical use case is when filtering out PGQ and Londiste, then the PGQ triggers will automatically be skipped by pg_staging rather than polluting the pg_restore logs because the CREATE TRIGGER command could not find the necessary implementation procedure.

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 29 Aug 2011 18:05:00 +0200 http://tapoueh.org/blog/2011/08/29-pgstaging-and-pgrestore-listing.html Skytools, version 3 http://tapoueh.org/blog/2011/08/26-skytools3.html You can find skytools3 in debian experimental already, it's in release candidate status. What's missing is the documentation, so here's an idea: I'm going to make a blog post series about skytools next features, how to use them, what they are good for, etc. This first article of the series will just list what are those new features.

Here are the slides from the CHAR(11) talk I made last month, about that very subject:

The new version comes with a lot of new features. PGQ now is able to duplicate the queue events from one node to the next, so that it's able to manage switching over. To do that we have three types of nodes now, root, branch and leaf. PGQ also supports cooperative consumers, meaning that you can share the processing load among many consumers, or workers.

Londiste now benefits from the switch over feature, and is packed with new little features like add <table> --create, the new --trigger-flags argument, and the new --handler thing (to do e.g. partial table replication). Let's not forget the much awaited execute <script> command that allows to include DDL commands into the replication stream, nor the parallel COPY support that will boost your initial setup.

walmgr in the new version behaves correctly when using PostgreSQL 9.0. Meaning that as soon as no more WAL files are available in the archives, it returns an error code to the archiver so that the server switches to streaming live from the primary_conninfo, then back to replaying the files from the archive if the connection were to fail, etc. All in all, it just works.

Details to follow here, stay tuned!

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 26 Aug 2011 21:30:00 +0200 http://tapoueh.org/blog/2011/08/26-skytools3.html pgfincore in debian http://tapoueh.org/blog/2011/08/19-pgfincore-in-debian.html As of pretty recently, pgfincore is now in debian, as you can see on its postgresql-9.0-pgfincore page. The reason why it entered the debian archives is that it reached the 1.0 release!

Rather than talking about what pgfincore is all about (A set of functions to manage pages in memory from PostgreSQL), I will talk about its packaging and support as a debian package. Here's the first example of a modern multi-version packaging I have to offer. pgfincore packaging supports building for 8.4 and 9.0 and 9.1 out of the box, even if the only binary you'll find in debian sid is the 9.0 one, as you can check on the pgfincore debian source package page.

Also, this is the first package I've done properly using the newer version of debhelper, which make the debian/rules file easier than ever. Let's have a look at it:

SRCDIR = $(CURDIR)
TARGET = $(CURDIR)/debian/pgfincore-%v
PKGVERS = $(shell dpkg-parsechangelog | awk -F '[:-]' '/^Version:/ { print substr($$2, 2) }')
EXCLUDE = --exclude-vcs --exclude=debian

include /usr/share/postgresql-common/pgxs_debian_control.mk

override_dh_auto_clean: debian/control
        pg_buildext clean $(SRCDIR) $(TARGET) "$(CFLAGS)"
        dh_clean

override_dh_auto_build:
        # build all supported version
        pg_buildext build $(SRCDIR) $(TARGET) "$(CFLAGS)"

override_dh_auto_install:
        # then install each of them
        for v in `pg_buildext supported-versions $(SRCDIR)`; do \
                dh_install -ppostgresql-$$v-pgfincore ;\
        done

orig: clean
        cd .. && tar czf pgfincore_$(PKGVERS).orig.tar.gz $(EXCLUDE) pgfincore

%:
        dh $@

The debian/rules file is known to be the corner stone of your debian packaging, and usually is the most complex part of it. It's a Makefile at its heart, and we can see that thanks to the debhelper magic it's not that complex to maintain anymore.

Then, this file is using support from a bunch of helpers command, each of them comes with its own man page and does a little part of the work. The overall idea around debhelper is that what it does covers 90% of the cases around, and it's not aiming for more. You have to override the parts where it defaults to being wrong.

Here for example the build system has to produce files for all three supported versions of PostgreSQL, which means invoking the same build system three time with some changes in the environment (mainly setting the PG_CONFIG variable correctly). But even for that we have a debian facility, that comes in the package postgresql-server-dev-all, called pg_buildext. As long as your extension build system is VPATH friendly, it's all automated.

Please read that last sentence another time. VPATH is the thing that allows Make to find your source tree somewhere in the system, not in the current working directory. That allows you to cleanly build the same sources in different build locations, which is exactly what we need here, and is cleanly supported by PGXS, the PostgreSQL Extension Building Infrastructure.

Which means that the main Makefile of pgfincore had to be simplified, and the code layout too. Some advances Make features such as $(wildcard ...) and all will not work here. See what we got at the end:

ifndef VPATH
SRCDIR = .
else
SRCDIR = $(VPATH)
endif

EXTENSION    = pgfincore
EXTVERSION   = $(shell grep default_version $(SRCDIR)/$(EXTENSION).control | \
               sed -e "s/default_version[[:space:]]*=[[:space:]]*'\([^']*\)'/\1/")

MODULES      = $(EXTENSION)
DATA         = sql/pgfincore.sql sql/uninstall_pgfincore.sql
DOCS         = doc/README.$(EXTENSION).rst

PG_CONFIG    = pg_config

PG91         = $(shell $(PG_CONFIG) --version | grep -qE "8\.|9\.0" && echo no || echo yes)

ifeq ($(PG91),yes)
all: pgfincore--$(EXTVERSION).sql

pgfincore--$(EXTVERSION).sql: sql/pgfincore.sql
        cp $< $@

DATA        = pgfincore--unpackaged--$(EXTVERSION).sql pgfincore--$(EXTVERSION).sql
EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql
endif

PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

deb:
        dh clean
        make -f debian/rules orig
        debuild -us -uc -sa

No more Make magic to find source files. Franckly though, when your sources are 1 c file and 2 sql files, you don't need that much magic anyway. You just want to believe that a single generic Makefile will happily build any project you throw at it, only requiring minor adjustment. Well, the reality is that you might need some more little adjustments if you want to benefit from VPATH building, and having the binaries for 8.4 and 9.0 and 9.1 built seemlessly in a simple loop. Like we have here for pgfincore.

Now the Makefile still contains a little bit of magic, in order to parse the extension version number from its control file and produce a script named accordingly. Then you'll notice a difference between the postgresql-9.1-pgfincore.install file and the postgresql-9.0-pgfincore.install. We're just not shipping the same files:

debian/pgfincore-9.0/pgfincore.so usr/lib/postgresql/9.0/lib
sql/pgfincore.sql usr/share/postgresql/9.0/contrib
sql/uninstall_pgfincore.sql usr/share/postgresql/9.0/contrib

As you can see here:

debian/pgfincore-9.1/pgfincore.so usr/lib/postgresql/9.1/lib
debian/pgfincore-9.1/pgfincore*.sql usr/share/postgresql/9.1/extension
sql/pgfincore--unpackaged--1.0.sql usr/share/postgresql/9.1/extension

So, now that we uncovered all the relevant magic, packaging and building your next extension so that it supports as many PostgreSQL major releases as you need to will be that easy.

For reference, you might need to also tweak /usr/share/postgresql-common/supported-versions so that it allows you to build for all those versions you claim to support in the debian/pgversions file.

$ sudo dpkg-divert \
--divert /usr/share/postgresql-common/supported-versions.distrib \
--rename /usr/share/postgresql-common/supported-versions

$ cat /usr/share/postgresql-common/supported-versions
#! /bin/bash

dpkg -l postgresql-server-dev-* \
| awk -F '[ -]' '/^ii/ && ! /server-dev-all/ {print $6}'

All of this will come pretty handy when we finally sit down and work on a way to provide binary packages for PostgreSQL and its extensions, and all supported versions of those at that. This very project is not dead, it's just sleeping some more.

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 19 Aug 2011 23:00:00 +0200 http://tapoueh.org/blog/2011/08/19-pgfincore-in-debian.html Échappement de chaînes http://tapoueh.org/blog/2011/08/18-echappements-de-chaine.html Parmis les nouveautés de la prochaine version de PostgreSQL, la fameuse 9.1, il faut signaler le changement de valeur par défaut de la variable standard_conforming_strings, qui passe à vraie.

En effet, l'utilisation d'échappements avec le caractère « anti-slash » n'est pas conforme au standard SQL. Le paramètre standard_conforming_strings permet de contrôler le comportement de PostgreSQL lorsqu'il lit une chaîne de caractère dans une requête SQL.

Voyons quelques exemples :

dimitri=# set standard_conforming_strings to true;
SET
dimitri=# select 'hop''';
 ?column?
----------
 hop'
(1 ligne)

dimitri=# select 'hop\'';
dimitri'# ';
 ?column?
----------
 hop\';

(1 ligne)

dimitri=# select E'hop\'';
 ?column?
----------
 hop'
(1 ligne)

dimitri=# set standard_conforming_strings to false;
SET
dimitri=# select E'hop\'';
 ?column?
----------
 hop'
(1 ligne)

dimitri=# select 'hop\'';
ATTENTION:  utilisation non standard de \' dans une chaîne littérale
LIGNE 1 : select 'hop\'';
                 ^
ASTUCE : Utilisez '' pour écrire des guillemets dans une chaîne ou utilisez
la syntaxe de chaîne d'échappement (E'...').
 ?column?
----------
 hop'
(1 ligne)

Il existe un moyen de forcer PostgreSQL à accepter l'utilisation d'échappements avec « anti-slash » indépendamment de la valeur de standard_conforming_strings, c'est la notation préfixée avec E. Il est recommandé de toujours l'utiliser dès lors que la chaîne de caractère contient des « anti-slash » utilisés comme échappement (du caractère simple guillemet en général).

Le paramètre escape_string_warning, enfin, permet de désactiver les avertissements tels que présentés dans le dernier exemple ci-dessus, lorsqu'il est positionné à off. Bien sûr, sa valeur par défaut est on.

Toute apparition de ce WARNING lorsque escape_string_warning est on signifie que votre application n'est pas prête à migrer à 9.1 avec son paramétrage par défaut. Il existe deux actions possible : changer le paramétrage de sa nouvelle valeur par défaut à sa précédente, ou bien corriger ses applications pour utiliser le préfixe E dès que cela est nécessaire.

L'utilisation de standard_conforming_strings à on présente un autre avantage au respect du standard SQL : la sécurité contre les injections. S'il n'est pas possible d'échapper le guillemet simple qui termine toute chaîne de caractère utilisateur, il devient compliqué de jouer au plus malin avec le parser. Le mieux ici reste bien sûr d'utiliser les requêtes paramétrées, à suivre dans un prochain article.

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 18 Aug 2011 19:01:00 +0200 http://tapoueh.org/blog/2011/08/18-echappements-de-chaine.html el-get-list-packages http://tapoueh.org/blog/2011/08/18-el-get-list-packages.html From the first days of el-get is was quite clear for me that we would reach a point where users would want a nice listing including descriptions of the packages, and a major mode allowing you to select packages to install, remove and update. It was also quite clear that I was not much interested into doing it myself, even if I would appreciate having it done.

Well, the joy of Open Source & Free Software (pick your own poison). jglee1027 is this GitHub guy who did offer an implementation of said facility, and who added descriptions for almost all of the now 402 recipes that we have included with el-get.

Here's an image of what you get:

The packages with no description are fetched by M-x el-get-emacswiki-refresh which will not download all emacswiki content locally just so that it can parse the scripts's header and have a local description. Maybe it's time to ask for another page over there like emacswiki page index but containing the first line too.

For recipes we offer, this first line often looks like the following:

;;; 123-menu.el --- Simple menuing system, reminiscent of Lotus 123 in DOS

Of course some files over there are not following the stanza, but that would be good enough already.

All in all, I hope you enjoy M-x el-get-list-packages!

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 18 Aug 2011 18:10:00 +0200 http://tapoueh.org/blog/2011/08/18-el-get-list-packages.html Tutoriel pgloader http://tapoueh.org/blog/2011/08/15-tutoriel-pgloader.html En reprenant le contenu des articles de la série sur pgloader, j'ai pris le temps de compiler un tutoriel complet, en anglais. Si j'en crois les quelques mails que je reçois régulièrement au sujet de pgloader depuis quelques années maintenant, cela devrait aider les nouveaux utilisateurs.

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 15 Aug 2011 15:39:00 +0200 http://tapoueh.org/blog/2011/08/15-tutoriel-pgloader.html pgloader tutorial http://tapoueh.org/blog/2011/08/15-pgloader-tutorial.html To finish up the pgloader series, I've compiled all the information into a single page, the long awaited pgloader tutorial. That should help lots of users to get started with pgloader.

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 15 Aug 2011 15:33:00 +0200 http://tapoueh.org/blog/2011/08/15-pgloader-tutorial.html pgloader constant cols http://tapoueh.org/blog/2011/08/12-pgloader-udc.html The previous articles in the pgloader series detailed How To Use PgLoader then How to Setup pgloader, then what to expect from a parallel pgloader setup, and then pgloader reformating. Another need you might encounter when you get to use pgloader is adding constant values into a table's column.

The basic situation where you need to do so is adding an origin field to your table. The value of that is not to be found in the data file itself, typically, but known in the pgloader setup. That could even be the filename you are importing data from.

In pgloader that's called a user defined column. Here's what the relevant examples/pgloader.conf setup looks like:

[udc]
table           = udc
format          = text
filename        = udc/udc.data
input_encoding  = 'latin1'
field_sep       = %
columns         = b:2, d:1, x:3, y:4
udc_c           = constant value
copy_columns    = b, c, d

And the data file is:

1%5%foo%bar
2%10%bar%toto
3%4%toto%titi
4%18%titi%baz
5%2%baz%foo

And here's what the loaded table looks like:

pgloader/examples$ pgloader -Tsc pgloader.conf udc
Table name        |    duration |    size |  copy rows |     errors
====================================================================
udc               |      0.201s |       - |          5 |          0

pgloader/examples$ psql --cluster 8.4/main pgloader -c "table udc"
 b  |       c        | d
----+----------------+---
  5 | constant value | 1
 10 | constant value | 2
  4 | constant value | 3
 18 | constant value | 4
  2 | constant value | 5
(5 rows)

Of course the configuration is not so straightforward as to process fields in the data file in the order that they appear, after all the examples/pgloader.conf are also a test suite.

Long story short: if you need to add some constant values into the target table you're loading data to, pgloader will help you there!

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 12 Aug 2011 11:00:00 +0200 http://tapoueh.org/blog/2011/08/12-pgloader-udc.html pgloader constant cols http://tapoueh.org/blog/2011/08/12-pgloader-udc.html The previous articles in the pgloader series detailed How To Use PgLoader then How to Setup pgloader, then what to expect from a parallel pgloader setup, and then pgloader reformating. Another need you might encounter when you get to use pgloader is adding constant values into a table's column.

The basic situation where you need to do so is adding an origin field to your table. The value of that is not to be found in the data file itself, typically, but known in the pgloader setup. That could even be the filename you are importing data from.

In pgloader that's called a user defined column. Here's what the relevant examples/pgloader.conf setup looks like:

[udc]
table           = udc
format          = text
filename        = udc/udc.data
input_encoding  = 'latin1'
field_sep       = %
columns         = b:2, d:1, x:3, y:4
udc_c           = constant value
copy_columns    = b, c, d

And the data file is:

1%5%foo%bar
2%10%bar%toto
3%4%toto%titi
4%18%titi%baz
5%2%baz%foo

And here's what the loaded table looks like:

pgloader/examples$ pgloader -Tsc pgloader.conf udc
Table name        |    duration |    size |  copy rows |     errors
====================================================================
udc               |      0.201s |       - |          5 |          0

pgloader/examples$ psql --cluster 8.4/main pgloader -c "table udc"
 b  |       c        | d
----+----------------+---
  5 | constant value | 1
 10 | constant value | 2
  4 | constant value | 3
 18 | constant value | 4
  2 | constant value | 5
(5 rows)

Of course the configuration is not so straightforward as to process fields in the data file in the order that they appear, after all the examples/pgloader.conf are also a test suite.

Long story short: if you need to add some constant values into the target table you're loading data to, pgloader will help you there!

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 12 Aug 2011 11:00:00 +0200 http://tapoueh.org/blog/2011/08/12-pgloader-udc.html Emacs Startup http://tapoueh.org/blog/2011/08/blog/2011/08/06-emacs-startup-notification.html Using Emacs we get to manage a larger and larger setup file (either ~/.emacs or ~/.emacs.d/init.el), sometime with lots of dependencies, and some sub-files thanks to the load function or the provide and require mechanism.

Some users are even starting Emacs often enough for the startup time to be a concern. With an emacs-uptime (yes it's a command, you can M-x emacs-uptime) of days to weeks (10 days, 17 hours, 45 minutes, 34 seconds as of this writing), it's not something I really care about much.

But I know that some el-get users still do care, and will use el-get-is-lazy and do all their Emacs tweaking as eval-after-load blocks. Trying to have an idea of how much a worst case startup with el-get is, I have added the following piece of elisp at the very end of my startup code:

(defun dim:notify-startup-done ()
  " notify user that Emacs is now ready"
  (el-get-notify
   "Emacs is ready."
   (format "The init sequence took %g seconds."
           (float-time (time-subtract after-init-time before-init-time)))))

(add-hook 'after-init-hook 'dim:notify-startup-done)

The el-get-notify function will adapt and either use the dbus implementation from Emacs 24, or notify.el from EmacsWiki (just M-x el-get-install it if you need it), or will use its own implementation of an Emacs Growl client (it's about 5 lines long), and baring all of that will use the message function.

The reason I say worst case is that I have a lot of packages to initialize at startup, and that I did absolutely no effort for this initializing to be quick. Still, my Emacs setup is taking about 20 seconds to boot. Pretty good I would say, for a weekly operation.

]]> dim@tapoueh.org (Dimitri Fontaine) Sat, 06 Aug 2011 14:58:00 +0200 http://tapoueh.org/blog/2011/08/blog/2011/08/06-emacs-startup-notification.html Emacs Startup http://tapoueh.org/blog/2011/08/06-emacs-startup-notification.html Using Emacs we get to manage a larger and larger setup file (either ~/.emacs or ~/.emacs.d/init.el), sometime with lots of dependencies, and some sub-files thanks to the load function or the provide and require mechanism.

Some users are even starting Emacs often enough for the startup time to be a concern. With an emacs-uptime (yes it's a command, you can M-x emacs-uptime) of days to weeks (10 days, 17 hours, 45 minutes, 34 seconds as of this writing), it's not something I really care about much.

But I know that some el-get users still do care, and will use el-get-is-lazy and do all their Emacs tweaking as eval-after-load blocks. Trying to have an idea of how much a worst case startup with el-get is, I have added the following piece of elisp at the very end of my startup code:

(defun dim:notify-startup-done ()
  " notify user that Emacs is now ready"
  (el-get-notify
   "Emacs is ready."
   (format "The init sequence took %g seconds."
           (float-time (time-subtract after-init-time before-init-time)))))

(add-hook 'after-init-hook 'dim:notify-startup-done)

The el-get-notify function will adapt and either use the dbus implementation from Emacs 24, or notify.el from EmacsWiki (just M-x el-get-install it if you need it), or will use its own implementation of an Emacs Growl client (it's about 5 lines long), and baring all of that will use the message function.

The reason I say worst case is that I have a lot of packages to initialize at startup, and that I did absolutely no effort for this initializing to be quick. Still, my Emacs setup is taking about 20 seconds to boot. Pretty good I would say, for a weekly operation.

]]> dim@tapoueh.org (Dimitri Fontaine) Sat, 06 Aug 2011 14:58:00 +0200 http://tapoueh.org/blog/2011/08/06-emacs-startup-notification.html pgloader reformating http://tapoueh.org/blog/2011/08/05-reformating-modules-for-pgloader.html Back to our series about pgloader. The previous articles detailed How To Use PgLoader then How to Setup pgloader, then what to expect from a parallel pgloader setup. This article will detail how to reformat input columns so that what PostgreSQL sees is not what's in the data file, but the result of a transformation from this data into something acceptable as an input for the target data type.

Here's what the pgloader documentation has to say about this reformat parameter: The value of this option is a comma separated list of columns to rewrite, which are a colon separated list of column name, reformat module name, reformat function name.

And here's the examples/pgloader.conf section that deals with reformat:

[reformat]
table           = reformat
format          = text
filename        = reformat/reformat.data
field_sep       = |
columns         = id, timestamp
reformat        = timestamp:mysql:timestamp

The documentation says some more about it, so check it out. Also, the reformat_path option (set either on the command line or in the configuration file) is used to find the python module implementing the reformat function. Please refer to the manual as to how to set it.

Now, obviously, for the reformat to happen we need to write some code. That's the whole point of the option: you need something very specific, you are in a position to write the 5 lines of code needed to make it happen, pgloader allows you to just do that. Of course, the code needs to be written in python here, so that you can even benefit from the parallel pgloader settings.

Let's see an reformat module exemple, as found in reformat/mysql.py in the pgloader sources:

# Author: Dimitri Fontaine <dim@tapoueh.org>
#
# pgloader mysql reformating module
#

def timestamp(reject, input):
    """ Reformat str as a PostgreSQL timestamp

    MySQL timestamps are like:  20041002152952
    We want instead this input: 2004-10-02 15:29:52
    """
    if len(input) != 14:
        e = "MySQL timestamp reformat input too short: %s" % input
        reject.log(e, input)

    year    = input[0:4]
    month   = input[4:6]
    day     = input[6:8]
    hour    = input[8:10]
    minute  = input[10:12]
    seconds = input[12:14]

    return '%s-%s-%s %s:%s:%s' % (year, month, day, hour, minute, seconds)

This reformat module will transform a timestamp representation as issued by certain versions of MySQL into something that PostgreSQL is able to read as a timestamp.

If you're in the camp that wants to write as little code as possible rather than easy to read and maintain code, I guess you could write it this way instead:

import re
def timestamp(reject, input):
    """ 20041002152952 -> 2004-10-02 15:29:52 """
    g = re.match(r"(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})", input)
    return '%s-%s-%s %s:%s:%s' % tuple([g.group(x+1) for x in range(6)])

Whenever you have an input file with data that PostgreSQL chokes upon, you can solve this problem from pgloader itself: no need to resort to scripting and a pipelines of awk (which I use a lot in other cases, don't get me wrong) or other tools. See, you finally have an excuse to Dive into Python!

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 05 Aug 2011 11:30:00 +0200 http://tapoueh.org/blog/2011/08/05-reformating-modules-for-pgloader.html Reformater avec pgloader http://tapoueh.org/blog/2011/08/05-reformater-avec-pgloader.html Dans la série de nos articles sur pgloader, le dernier venu détaille comment utiliser la fonction de reformatage de cet outil. Dans le cadre d'utilisation d'un ETL, cela est assimilé à la phase Transform, ce qui fait de pgloader une solution simple pour vos besoins d'ETL.

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 05 Aug 2011 11:26:00 +0200 http://tapoueh.org/blog/2011/08/05-reformater-avec-pgloader.html See Tsung in action http://tapoueh.org/blog/2011/08/02-see-tsung-in-action.html Tsung is an open-source multi-protocol distributed load testing tool and a mature project. It's been available for about 10 years and is built with the Erlang system. It supports several protocols, including the PostgreSQL one.

When you want to benchmark your own application, to know how many more clients it can handle or how much gain you will see with some new shiny hardware, Tsung is the tool to use. It will allow you to record a number of sessions then replay them at high scale. pgfouine supports Tsung and is able to turn your PostgreSQL logs into Tsung sessions, too.

Tsung did get used in the video game world, their version of it is called uTsung, apparently using the uLink game development facilities. They even made a video demo of uTsung, that you might find interresting:

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 02 Aug 2011 10:30:00 +0200 http://tapoueh.org/blog/2011/08/02-see-tsung-in-action.html Parallel pgloader http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html This article continues the series that began with How To Use PgLoader then detailed How to Setup pgloader. We have some more fine points to talk about here, today's article is about loading your data in parallel with pgloader.

several files at a time

Parallelism is implemented in 3 different ways in pgloader. First, you can load more than one file at a time thanks to the max_parallel_sections parameter, that has to be setup in the global section of the file.

This setting is quite simple and already allows the most common use case.

several workers per file

The other use case is when you have huge files to load into the database. Then you want to be able to have more than one process reading the file at the same time. Using pgloader, you already did the compromise to load the whole content in more than one transaction, so there's no further drawback here about having those multiple transactions per file spread to more than one load worker.

There are basically two ways to split the work between several workers here, and both are implemented in pgloader.

N workers, N splits of the file

section_threads    = 4
split_file_reading = True

Setup this way, pgloader will launch 4 different threads (see the caveat section of this article). Each thread is then given a part of the input data file and will run the whole usual pgloader processing on its own. For this to work you need to be able to seek in the input stream, which might not always be convenient.

one reader, N workers

section_threads    = 4
split_file_reading = False
rrqueue_size       = 5000

With such a setup, pgloader will start 4 different worker threads that will receive the data input in an internal python queue. Another active thread will be responsible of reading the input file and filling the queues in a round robin fashion, but will hand all the processing of the data to each worker, of course.

how many threads?

If you're using a mix and match of max_parallel_sections and section_threads with split_file_reading set to True of False, it's uneasy to know exactly how many threads will run at any time in the loading. How to ascertain which section will run in parallel when it depends on the timing of the loading?

The advice here is the usual one, don't overestimate the capabilities of your system unless you are in a position to check before by doing trial runs.

caveat

Current implementation of all the parallelism in pgloader has been done with the python threading API. While this is easy enough to use when you want to exchange data between threads, it's suffering from the Global Interpreter Lock issue. This means that while the code is doing its processing in parallel, the runtime not so much. You might still benefit from the current implementation if you have hard to parse files, or custom reformat modules that are part of the loading bottleneck.

future

The solution would be to switch to using the newer python multiprocessing API, and some preliminary work has been done in pgloader to allow for that. If you're interested in real parallel bulk loading, contact-me!

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 01 Aug 2011 12:15:00 +0200 http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html Parallel pgloader http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html This article continues the series that began with How To Use PgLoader then detailed How to Setup pgloader. We have some more fine points to talk about here, today's article is about loading your data in parallel with pgloader.

several files at a time

Parallelism is implemented in 3 different ways in pgloader. First, you can load more than one file at a time thanks to the max_parallel_sections parameter, that has to be setup in the global section of the file.

This setting is quite simple and already allows the most common use case.

several workers per file

The other use case is when you have huge files to load into the database. Then you want to be able to have more than one process reading the file at the same time. Using pgloader, you already did the compromise to load the whole content in more than one transaction, so there's no further drawback here about having those multiple transactions per file spread to more than one load worker.

There are basically two ways to split the work between several workers here, and both are implemented in pgloader.

N workers, N splits of the file

section_threads    = 4
split_file_reading = True

Setup this way, pgloader will launch 4 different threads (see the caveat section of this article). Each thread is then given a part of the input data file and will run the whole usual pgloader processing on its own. For this to work you need to be able to seek in the input stream, which might not always be convenient.

one reader, N workers

section_threads    = 4
split_file_reading = False
rrqueue_size       = 5000

With such a setup, pgloader will start 4 different worker threads that will receive the data input in an internal python queue. Another active thread will be responsible of reading the input file and filling the queues in a round robin fashion, but will hand all the processing of the data to each worker, of course.

how many threads?

If you're using a mix and match of max_parallel_sections and section_threads with split_file_reading set to True of False, it's uneasy to know exactly how many threads will run at any time in the loading. How to ascertain which section will run in parallel when it depends on the timing of the loading?

The advice here is the usual one, don't overestimate the capabilities of your system unless you are in a position to check before by doing trial runs.

caveat

Current implementation of all the parallelism in pgloader has been done with the python threading API. While this is easy enough to use when you want to exchange data between threads, it's suffering from the Global Interpreter Lock issue. This means that while the code is doing its processing in parallel, the runtime not so much. You might still benefit from the current implementation if you have hard to parse files, or custom reformat modules that are part of the loading bottleneck.

future

The solution would be to switch to using the newer python multiprocessing API, and some preliminary work has been done in pgloader to allow for that. If you're interested in real parallel bulk loading, contact-me!

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 01 Aug 2011 12:05:00 +0200 http://tapoueh.org/blog/2011/08/01-parallel-pgloader.html Configurer pgloader http://tapoueh.org/blog/2011/07/29-configurer-pgloader.html Je viens de publier un billet en anglais intitulé How to Setup pgloader, qui complète l'écriture en cours d'un tutoriel pgloader plus complet. Une fois de plus, je n'ai pas pris le temps de traduire cet article en français avant de savoir si cela vous intéresse, ô lecteurs. Si c'est le cas il suffit de me l'indiquer par mail (ou courriel, après tout) pour que j'ajoute cela dans ma TODO liste.

Bonne lecture !

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 29 Jul 2011 15:00:00 +0200 http://tapoueh.org/blog/2011/07/29-configurer-pgloader.html How to Setup pgloader http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html In a previous article we detailed how to use pgloader, let's now see how to write the pgloader.conf that instructs pgloader about what to do.

This file is expected in the INI format, with a global section then one section per file you want to import. The global section defines some default options and how to connect to the PostgreSQL server.

The configuration setup is fully documented on the pgloader man page that you can even easily find online. As all unix style man pages, though, it's more a complete reference than introductory material. Let's review.

global section

Here's the global section of the examples/pgloader.conf file of the source files. Well, some options are debugger only options, really, so I changed their value so that what you see here is a better starting point.

[pgsql]
base = pgloader

log_file            = /tmp/pgloader.log
log_min_messages    = INFO
client_min_messages = WARNING

lc_messages         = C
pg_option_client_encoding = 'utf-8'
pg_option_standard_conforming_strings = on
pg_option_work_mem = 128MB

copy_every      = 15000

null         = ""
empty_string = "\ "

max_parallel_sections = 4

You don't see all the connection setup, here base was enough. You might need to setup host, port and user, and maybe even pass, too, to be able to connect to the PostgreSQL server.

The logging options allows you to set a file where to log all pgloader messages, that are categorized as either DEBUG, INFO, WARNING, ERROR or CRITICAL. The options log_min_messages and client_min_messages are another good idea stolen from PostgreSQL and allow you to setup the level of chatter you want to see on the interactive console (standard output and standard error streams) and on the log file.

Please note that the DEBUG level will produce more that 3 times as many data as the data file you're importing. If you're not a pgloader contributor or helping them, well, debug it, you want to avoid setting the log chatter to this value.

The client_encoding will be SET by pgloader on the PostgreSQL connection it establish. You can now even set any parameter you want by using the pg_option_parameter_name magic settings. Note that the command line option --pg-options (or -o for brevity) allows you to override that.

Then, the copy_every parameter is set to 5 in the examples, because the test files are containing less than 10 lines and we want to test several batches of commits when using them. So for your real loading, stick to default parameters (10 000 lines per COPY command), or more. You can play with this parameter, depending on the network (or local access) and disk system you're using you might see improvements by reducing it or enlarging it. There's no so much theory of operation as empirical testing and setting here. For a one-off operation, just remove the lines from the configuration.

The parameters null and empty_string are related to interpreting the data in the text or csv files you have, and the documentation is quite clear about them. Note that you have global setting and per-section setting too.

The last parameter of this example, max_parallel_sections, is detailed later in the article.

files section

After the global section come as many sections as you have file to load. Plus the template sections, that are only there so that you can share a bunch of parameters in more than one section. Picture a series of data file all of the same format, the only thing that will change is the filename. Use a template section in this case!

Let's see an example:

[simple_tmpl]
template     = True
format       = text
datestyle    = dmy
field_sep    = |
trailing_sep = True

[simple]
use_template    = simple_tmpl
table           = simple
filename        = simple/simple.data
columns         = a:1, b:3, c:2
skip_head_lines = 2

# those reject settings are defaults one
reject_log   = /tmp/simple.rej.log
reject_data  = /tmp/simple.rej

[partial]
table        = partial
format       = text
filename     = partial/partial.data
field_sep    = %
columns      = *
only_cols    = 1-3, 5

That's 2 of the examples from the examples/pgloader.conf file, in 3 sections so that we see one template example. Of course, having a single section using the template, it's just here for the example.

data file format

The most important setting that you have to care about is the file format. Your choice here is either text, csv or fixed. Mostly, what we are given nowadays is csv. You might remember having read that the nice thing about standards is that there's so many to choose from... well, the csv land is one where it's pretty hard to find different producers that understand it the same way.

So when you fail to have pgloader load your mostly csv files with a csv setup, it's time to consider using text instead. The text file format accept a lot of tunables to adapt to crazy situations, but is all python code when the python csv module is a C-coded module, more efficient.

If you're wondering what kind of format we're talking about here, here's the cluttered pgloader example for your reading pleasure, using ^ (carret) as the field separator:

1^some multi\
line text with\
newline escaping^and some other data following^
2^and another line^clean^
3^and\
a last multiline\
escaped line
with a missing\
escaping^just to test^
4^\ ^empty value^
5^^null value^
6^multi line\
escaped value\
\
with empty line\
embeded^last line^

And here's what we get by loading that:

pgloader/examples$ pgloader -c pgloader.conf -s cluttered
Table name        |    duration |    size |  copy rows |     errors
====================================================================
cluttered         |      0.193s |       - |          6 |          0

pgloader/examples$ psql pgloader -c "table cluttered;"
 a |               b               |        c
---+-------------------------------+------------------
 1 | and some other data following | some multi
                                   : line text with
                                   : newline escaping
 2 | clean                         | and another line
 3 | just to test                  | and
                                   : a last multiline
                                   : escaped line
                                   : with a missing
                                   : escaping
 4 | empty value                   |
 5 | null value                    |
 6 | last line                     | multi line
                                   : escaped value
                                   :
                                   : with empty line
                                   : embeded
(6 rows)

So when you have such kind of data, well, it might be that pgloader is still able to help you!

Please refer to the pgloader man page to know about each and every parameter that you can define and the values accepted, etc. And the fixed data format is to be used when you're not given a field separator but field positions in the file. Yes, we still encounter those from time to time. Who needs variable size storage, after all?

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 29 Jul 2011 15:00:00 +0200 http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html Emacs ANSI colors http://tapoueh.org/blog/2011/07/blog/2011/07/29-emacs-ansi-colors.html Emacs comes with a pretty good implementation of a terminal emulator, M-x term. Well not that good actually, but given what I use it for, it's just what I need. Particulary if you add to that my cssh tool, so that connecting with ssh to a remote host is just a =C-= runs the command cssh-term-remote-open away, and completes on the host name thanks to ~/.ssh/known_hosts.

Now, a problem that I still had to solve was the colors used in the terminal. As I'm using the tango color theme for emacs, the default ANSI palette's blue color was not readable. Here's how to fix that:

   (require 'ansi-color)
   (setq ansi-color-names-vector
         (vector (frame-parameter nil 'background-color)
               "#f57900" "#8ae234" "#edd400" "#729fcf"
               "#ad7fa8" "cyan3" "#eeeeec")
         ansi-term-color-vector ansi-color-names-vector
         ansi-color-map (ansi-color-make-color-map))

Now your colors in an emacs terminal are easy to read, as you can see:

Hope you enjoy!

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 29 Jul 2011 10:00:00 +0200 http://tapoueh.org/blog/2011/07/blog/2011/07/29-emacs-ansi-colors.html Emacs ANSI colors http://tapoueh.org/blog/2011/07/29-emacs-ansi-colors.html Emacs comes with a pretty good implementation of a terminal emulator, M-x term. Well not that good actually, but given what I use it for, it's just what I need. Particulary if you add to that my cssh tool, so that connecting with ssh to a remote host is just a =C-= runs the command cssh-term-remote-open away, and completes on the host name thanks to ~/.ssh/known_hosts.

Now, a problem that I still had to solve was the colors used in the terminal. As I'm using the tango color theme for emacs, the default ANSI palette's blue color was not readable. Here's how to fix that:

   (require 'ansi-color)
   (setq ansi-color-names-vector
         (vector (frame-parameter nil 'background-color)
               "#f57900" "#8ae234" "#edd400" "#729fcf"
               "#ad7fa8" "cyan3" "#eeeeec")
         ansi-term-color-vector ansi-color-names-vector
         ansi-color-map (ansi-color-make-color-map))

Now your colors in an emacs terminal are easy to read, as you can see:

Hope you enjoy!

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 29 Jul 2011 10:00:00 +0200 http://tapoueh.org/blog/2011/07/29-emacs-ansi-colors.html How to Setup pgloader http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html In a previous article we detailed how to use pgloader, let's now see how to write the pgloader.conf that instructs pgloader about what to do.

This file is expected in the INI format, with a global section then one section per file you want to import. The global section defines some default options and how to connect to the PostgreSQL server.

The configuration setup is fully documented on the pgloader man page that you can even easily find online. As all unix style man pages, though, it's more a complete reference than introductory material. Let's review.

global section

Here's the global section of the examples/pgloader.conf file of the source files. Well, some options are debugger only options, really, so I changed their value so that what you see here is a better starting point.

[pgsql]
base = pgloader

log_file            = /tmp/pgloader.log
log_min_messages    = INFO
client_min_messages = WARNING

lc_messages         = C
pg_option_client_encoding = 'utf-8'
pg_option_standard_conforming_strings = on
pg_option_work_mem = 128MB

copy_every      = 15000

null         = ""
empty_string = "\ "

max_parallel_sections = 4

You don't see all the connection setup, here base was enough. You might need to setup host, port and user, and maybe even pass, too, to be able to connect to the PostgreSQL server.

The logging options allows you to set a file where to log all pgloader messages, that are categorized as either DEBUG, INFO, WARNING, ERROR or CRITICAL. The options log_min_messages and client_min_messages are another good idea stolen from PostgreSQL and allow you to setup the level of chatter you want to see on the interactive console (standard output and standard error streams) and on the log file.

Please note that the DEBUG level will produce more that 3 times as many data as the data file you're importing. If you're not a pgloader contributor or helping them, well, debug it, you want to avoid setting the log chatter to this value.

The client_encoding will be SET by pgloader on the PostgreSQL connection it establish. You can now even set any parameter you want by using the pg_option_parameter_name magic settings. Note that the command line option --pg-options (or -o for brevity) allows you to override that.

Then, the copy_every parameter is set to 5 in the examples, because the test files are containing less than 10 lines and we want to test several batches of commits when using them. So for your real loading, stick to default parameters (10 000 lines per COPY command), or more. You can play with this parameter, depending on the network (or local access) and disk system you're using you might see improvements by reducing it or enlarging it. There's no so much theory of operation as empirical testing and setting here. For a one-off operation, just remove the lines from the configuration.

The parameters null and empty_string are related to interpreting the data in the text or csv files you have, and the documentation is quite clear about them. Note that you have global setting and per-section setting too.

The last parameter of this example, max_parallel_sections, is detailed later in the article.

files section

After the global section come as many sections as you have file to load. Plus the template sections, that are only there so that you can share a bunch of parameters in more than one section. Picture a series of data file all of the same format, the only thing that will change is the filename. Use a template section in this case!

Let's see an example:

[simple_tmpl]
template     = True
format       = text
datestyle    = dmy
field_sep    = |
trailing_sep = True

[simple]
use_template    = simple_tmpl
table           = simple
filename        = simple/simple.data
columns         = a:1, b:3, c:2
skip_head_lines = 2

# those reject settings are defaults one
reject_log   = /tmp/simple.rej.log
reject_data  = /tmp/simple.rej

[partial]
table        = partial
format       = text
filename     = partial/partial.data
field_sep    = %
columns      = *
only_cols    = 1-3, 5

That's 2 of the examples from the examples/pgloader.conf file, in 3 sections so that we see one template example. Of course, having a single section using the template, it's just here for the example.

data file format

The most important setting that you have to care about is the file format. Your choice here is either text, csv or fixed. Mostly, what we are given nowadays is csv. You might remember having read that the nice thing about standards is that there's so many to choose from... well, the csv land is one where it's pretty hard to find different producers that understand it the same way.

So when you fail to have pgloader load your mostly csv files with a csv setup, it's time to consider using text instead. The text file format accept a lot of tunables to adapt to crazy situations, but is all python code when the python csv module is a C-coded module, more efficient.

If you're wondering what kind of format we're talking about here, here's the cluttered pgloader example for your reading pleasure, using ^ (carret) as the field separator:

1^some multi\
line text with\
newline escaping^and some other data following^
2^and another line^clean^
3^and\
a last multiline\
escaped line
with a missing\
escaping^just to test^
4^\ ^empty value^
5^^null value^
6^multi line\
escaped value\
\
with empty line\
embeded^last line^

And here's what we get by loading that:

pgloader/examples$ pgloader -c pgloader.conf -s cluttered
Table name        |    duration |    size |  copy rows |     errors
====================================================================
cluttered         |      0.193s |       - |          6 |          0

pgloader/examples$ psql pgloader -c "table cluttered;"
 a |               b               |        c
---+-------------------------------+------------------
 1 | and some other data following | some multi
                                   : line text with
                                   : newline escaping
 2 | clean                         | and another line
 3 | just to test                  | and
                                   : a last multiline
                                   : escaped line
                                   : with a missing
                                   : escaping
 4 | empty value                   |
 5 | null value                    |
 6 | last line                     | multi line
                                   : escaped value
                                   :
                                   : with empty line
                                   : embeded
(6 rows)

So when you have such kind of data, well, it might be that pgloader is still able to help you!

Please refer to the pgloader man page to know about each and every parameter that you can define and the values accepted, etc. And the fixed data format is to be used when you're not given a field separator but field positions in the file. Yes, we still encounter those from time to time.

parallel processing

one reader, multiple workers

multiple workers, each reading

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 29 Jul 2011 09:57:00 +0200 http://tapoueh.org/blog/2011/07/29-how-to-setup-pgloader.html Next month partitions http://tapoueh.org/blog/2011/07/27-check-parts-for-next-month.html When you do partition your tables monthly, then comes the question of when to create next partitions. I tend to create them just the week before next month and I have some nice nagios scripts to alert me in case I've forgotten to do so. How to check that by hand in the end of a month?

Here's a catalog query to help you there:

=> select *
->   from
->   (
(>   select 'previous parts' as schemaname, count(*)::text as tablename
(>     from pg_tables
(>    where schemaname not in ('pg_catalog','information_schema')
(>      and tablename like to_char(now(), '%YYYYMM')
(>
(>   union
(>
(>   select schemaname, substring(tablename,1,length(tablename)-6) || '201108'
(>     from pg_tables
(>    where schemaname not in ('pg_catalog','information_schema')
(>      and tablename like to_char(now(), '%YYYYMM')
(>
(>   except
(>
(>   select schemaname, tablename
(>     from pg_tables
(>    where schemaname not in ('pg_catalog','information_schema')
(>      and tablename like to_char(now() + interval '1 month', '%YYYYMM')
(>   ) as t
-> order by schemaname <> 'previous parts', schemaname;
   schemaname   |       tablename
----------------+------------------------
 previous parts | 1
 central        | stats_entrantes_201108
(2 rows)

As you see, our partitions are named _YYYYMM so that's it's easy to match them in our queries, but I guess about everyone does about the same here. Then the to_char expressions only allow to not enter manually '%201108' in the query text. And there's a trick so that we display how many partitions we have this month, adding a line to the result...

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 27 Jul 2011 22:35:00 +0200 http://tapoueh.org/blog/2011/07/27-check-parts-for-next-month.html Comment Utiliser pgloader http://tapoueh.org/blog/2011/07/22-comment-utiliser-pgloader.html C'est une question qui revient régulièrement, et à laquelle je pensais avoir apporté une réponse satisfaisante avec les exemples pgloader. Ce document ressemble un peu à un tutoriel, en anglais, et je l'ai détaillé dans l'article how to use pgloader sur ce même site, en anglais. Si la demande est suffisante, je le traduirai en français.

En attendant, bonne lecture !

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 22 Jul 2011 13:48:00 +0200 http://tapoueh.org/blog/2011/07/22-comment-utiliser-pgloader.html How To Use PgLoader http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html This question about pgloader usage coms in quite frequently, and I think the examples README goes a long way in answering it. It's not exactly a tutorial but is almost there. Let me paste it here for reference:

installing pgloader

Either use the debian package or the one for your distribution of choice if you use another one. RedHat, CentOS, FreeBSD, OpenBSD and some more already include a binary package that you can use directly.

Or you could git clone https://github.com/dimitri/pgloader.git and go from there. As it's all python code, it runs fine interpreted from the source directory, you don't need to install it in a special place in your system.

setting up the test environment

To use them, please first create a pgloader database, then for each example the tables it needs, then issue the pgloader command:

$ createdb --encoding=utf-8 pgloader
$ cd examples
$ psql pgloader < simple/simple.sql
$ ../pgloader.py -Tvc pgloader.conf simple

If you want to load data from all examples, create tables for all of them first, then run pgloader without argument.

example description

The provided examples are:

  • simple

    This dataset shows basic case, with trailing separator and data reordering.

  • xzero

    Same as simple but using \0 as the null marker ()

  • errors

    Same test, but with impossible dates. Should report some errors. If it does not report errors, check you're not using psycopg 1.1.21.

    Should report 3 errors out of 7 lines (4 updates).

  • clob

    This dataset shows some text large object importing to PostgreSQL text datatype.

  • cluttured

    A dataset with newline escaped and multi-line input (without quoting) Beware of data reordering, too.

  • csv

    A dataset with csv delimiter ',' and quoting '"'.

  • partial

    A dataset from which we only load some columns of the provided one.

  • serial

    In this dataset the id field is ommited, it's a serial which will be automatically set by PostgreSQL while COPYing.

  • reformat

    A timestamp column is formated the way MySQL dump its timestamp, which is not the same as the way PostgreSQL reads them. The reformat.mysql module is used to reformat the data on-the-fly.

  • udc

    A used defined column test, where all file columns are not used but a new constant one, not found in the input datafile, is added while loading data.

running the import

You can launch all those pgloader tests in one run, provided you created the necessary tables:

 $ for sql in */*sql; do psql pgloader < $sql; done
 $ ../pgloader.py -Tsc pgloader.conf

  errors       WARNING  COPY error, trying to find on which line
  errors       WARNING  COPY data buffer saved in /tmp/errors.AhWvAv.pgloader
  errors       WARNING  COPY error recovery done (2/3) in 0.064s
  errors       WARNING  COPY error, trying to find on which line
  errors       WARNING  COPY data buffer saved in /tmp/errors.BclHtj.pgloader
  errors       WARNING  COPY error recovery done (1/1) in 0.054s
  errors       ERROR    3 errors found into [errors] data
  errors       ERROR    please read /tmp/errors.rej.log for errors log
  errors       ERROR    and /tmp/errors.rej for data still to process
  errors       ERROR    3 database errors occured
  reformat     WARNING  COPY error, trying to find on which line
  reformat     WARNING  COPY data buffer saved in /tmp/reformat.6P4WCD.pgloader
  reformat     WARNING  COPY error recovery done (1/4) in 0.034s
  reformat     ERROR    1 errors found into [reformat] data
  reformat     ERROR    please read /tmp/reformat.rej.log for errors log
  reformat     ERROR    and /tmp/reformat.rej for data still to process
  reformat     ERROR    1 database errors occured

  Table name        |    duration |    size |  copy rows |     errors
  ====================================================================
  allcols           |      0.025s |       - |          8 |          0
  clob              |      0.034s |       - |          7 |          0
  cluttered         |      0.061s |       - |          6 |          0
  csv               |      0.035s |       - |          6 |          0
  errors            |      0.113s |       - |          4 |          3
  fixed             |      0.045s |       - |          3 |          0
  partial           |      0.030s |       - |          7 |          0
  reformat          |      0.036s |       - |          4 |          1
  serial            |      0.029s |       - |          7 |          0
  simple            |      0.050s |       - |          7 |          0
  udc               |      0.020s |       - |          5 |          0
  ====================================================================
  Total             |      0.367s |       - |         64 |          4

Please note errors test should return 3 errors and reformat 1 error.

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 22 Jul 2011 13:38:00 +0200 http://tapoueh.org/blog/2011/07/22-how-to-use-pgloader.html Emacs Cheat Sheet http://tapoueh.org/blog/2011/07/blog/2011/07/20-emacs-cheat-sheet.html I stumbled upon the following cheat sheet for Emacs yesterday, and it's worth sharing. I already learnt or discovered again some nice default chords, like for example C-x C-o runs the command delete-blank-lines and C-M-o runs the command split-line. I guess I will use the later one a lot.

Hope you'll like it!

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 20 Jul 2011 10:44:00 +0200 http://tapoueh.org/blog/2011/07/blog/2011/07/20-emacs-cheat-sheet.html Emacs Cheat Sheet http://tapoueh.org/blog/2011/07/20-emacs-cheat-sheet.html I stumbled upon the following cheat sheet for Emacs yesterday, and it's worth sharing. I already learnt or discovered again some nice default chords, like for example C-x C-o runs the command delete-blank-lines and C-M-o runs the command split-line. I guess I will use the later one a lot.

Hope you'll like it!

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 20 Jul 2011 10:44:00 +0200 http://tapoueh.org/blog/2011/07/20-emacs-cheat-sheet.html Skytools3 : les slides http://tapoueh.org/blog/2011/07/19-skytools3-slides.html La conférence CHAR(11) étant maintenant terminée, il est d'usage de publier les slides utilisés. J'ai présenté Skytools 3.0 dont la prochaine version sera publiée dès que j'aurais eu le temps de terminer de revoir (en fait principalement d'écrire) la documentation.

Les slides de l'ensemble des présentations devraient être publiés en ligne à terme, mais cela ne va pas pouvoir être fait aussi rapidement que nous le voudrions tous. Alors voici un peu de lecture en attendant la suite !

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 19 Jul 2011 14:39:00 +0200 http://tapoueh.org/blog/2011/07/19-skytools3-slides.html Skytools3 talk Slides http://tapoueh.org/blog/2011/07/19-skytools3-talk-slides.html In case you're wondering, here are the slides from the CHAR(11) talk I gave, about Skytools 3.0, soon to be released. That means as soon as I have enough time available to polish (or write) the documentation.

The slides for all the talks should eventually make their way to a central place, but expect some noticable delay here. Sorry about that, and have a good reading meanwhile!

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 19 Jul 2011 14:24:00 +0200 http://tapoueh.org/blog/2011/07/19-skytools3-talk-slides.html Elisp Breadcrumbs http://tapoueh.org/blog/2011/07/blog/2011/07/14-elisp-breadcrumbs.html A breadcrumb is a navigation aid. I just added one to this website, so that it gets easier to browse from any article to its local and parents indexes and back to /dev/dim, the root webpage of this site.

As it was not that much work to implement, here's the whole of it:

;;;
;;; Breadcrumb support
;;;
(defun tapoueh-breadcrumb-to-current-page ()
  "Return a list of (name . link) from the index root page to current one"
  (let* ((current (muse-current-file))
         (cwd     (file-name-directory current))
         (project (muse-project-of-file current))
         (root    (muse-style-element :path (caddr project)))
         (path    (tapoueh-path-to-root))
         (dirs    (split-string (file-relative-name current root) "/")))
    ;; ("blog" "2011" "07" "13-back-from-char11.muse")
    (append
     (list (cons "/dev/dim" (concat path "index.html")))
     (loop for p in (butlast dirs)
           collect (cons p (format "%s%s/index.html" path p))
           do (setq path (concat path p "/"))))))

(defun tapoueh-insert-breadcrumb-div ()
  "The real HTML inserting"
  (insert "<div id=\"breadcrumb\">")
  (loop for (name . link) in (tapoueh-breadcrumb-to-current-page)
        do (insert (format "<a href=%s>%s</a>" link name) " / "))
  (insert "</div>\n"))

(defun tapoueh-insert-breadcrumb ()
  "Must run with current buffer being a muse article"
  (save-excursion
    (beginning-of-buffer)
    (when (tapoueh-extract-directive "author" (muse-current-file))
      (re-search-forward "<body>" nil t) ; find where the article content is
      (re-search-forward "<h2>" nil t)   ; that's the title line
      (beginning-of-line)
      (open-line 1)
      (tapoueh-insert-breadcrumb-div)

      (re-search-forward "<h2>" nil t 2) ; that's the TAG line
      (beginning-of-line)
      (open-line 1)
      (tapoueh-insert-breadcrumb-div))))

This code is now called in the :after function of my Muse project style, and it gets the work done.

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 14 Jul 2011 18:44:00 +0200 http://tapoueh.org/blog/2011/07/blog/2011/07/14-elisp-breadcrumbs.html Elisp Breadcrumbs http://tapoueh.org/blog/2011/07/14-elisp-breadcrumbs.html

This code is now called in the :after function of my Muse project style, and it gets the work done.

Tags

Emacs Muse

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 14 Jul 2011 18:44:00 +0200 http://tapoueh.org/blog/2011/07/14-elisp-breadcrumbs.html De retour de CHAR(11) http://tapoueh.org/blog/2011/07/13-de-retour-de-char11.html De retour de CHAR(11)
Wednesday, July 13 2011, 17:30

Quelle meilleure occupation dans le train du retour de CHAR(11) que de se faire reporteur pour l'occasion ? En réalité, dormir serait une idée tant les soirées se sont prolongées !

Nous avons eu le plaisir d'écouter Jan Wieck présenter un historique simplifié de la réplication avec PostgreSQL. Étant lui-même l'un des pionniers du domaine, son point de vue est des plus intéressants. Il a parlé de l'évolution des solutions de réplication, et je ne peux m'empêcher de penser que par bien des côtés Skytools est une évolution de Slony — Jan, auteur de Slony, semblait d'accord avec cela.

En effet Skytools est né de limitations de Slony. Certaines d'entre elles existent toujours, comme l'absence de séparation entre la couche de queuing et la couche de réplication elle-même, et certaines ont été résolues depuis, comme les difficultés à subir de fortes charges en écriture. Et puis les deux solutions partagent même une partie de leur implémentation, depuis PostgreSQL 8.3, avec les types de données txid et txid_snapshot. Bien sûr, l'objectif de Skytools est d'avoir une solution la plus simple possible, parfaitement adapée à un ensemble de cas d'utilisation précis et bornés, alors que Slony essaye de résoudre automatiquement les problèmes les plus difficiles du domaine, au prix d'une interface très complexe.

Bien sûr, Jan a pris le temps de comparer objectivement ces solutions de réplication avec la solution intégrée dans PostgreSQL, Streaming Replication et Hot Standby. Nous avions déjà la réplication binaire asynchrone, PostgreSQL 9.1 nous apporte la réplication synchrone avec un contrôle par transaction. Simon Riggs, auteur de la fonctionalité, a insisté sur l'innovation que cela représente : aucun autre projet ne permet de contrôler la garantie de durabilité des données avec une granularité aussi souple et précise !

repmgr est une solution d'administration de cluster animés avec Hot Standby et Streaming Replication (synchrone ou non). Son fonctionnement a été détaillé par Greg Smith et Cédric Villemain. Le premier a montré comment mettre au point une architecture permettant de répartir la charge en lecture, et le second comment obtenir un système tolérant aux pannes grâce au failover automatique intégré dans repmgr. Cette solution innovante a été mise au point en grande partie par 2ndQuadrant France, nous l'avons déjà estampillée production ready.

Magnus Hagander a beaucoup travaillé sur le protocole de streaming utilisé pour la réplication intégrée dans PostgreSQL 9.1, ainsi que sur les outils qui exploitent ce protocole. Il a naturellement présenté cela, et l'idée d'un proxy relayant le flux binaire des journaux de transaction est revenue dans les discutions (nous avions déjà envisagé cela en 2010, l'article en anglais Back from PgCon2010 contient quelques éléments sur le sujet). Avec la réplication synchrone, il devient possible de concevoir des architectures avancées, robustes et versatiles — le proxy pourrait maintenant s'occuper à la fois des archives et des serveurs standby.

Simon Riggs nous a ensuite proposé une rétrospective des 7 dernières années de travail qu'il a réalisé avec PostgreSQL, de l'implémentation du Point in Time Recovery à la réplication synchrone, en passant par Hot Standby. Ce que nous avons dans PostgreSQL 9.0 correspond déjà à ce qu'Oracle propose de plus avancé en terme de durabitilé des données, et 9.1 permet de franchir l'étape suivante. Cela ne freine en rien Simon qui parlait déjà des projets à venir pour les 10 prochaines années.

Enfin, Heroku nous a présenté leur incroyable entreprise. Ils ont aujourd'hui plus de 150 000 instances de PostgreSQL en production, démontrant que notre SGBD préféré est prêt pour les hébergeurs. Heroku est en train de concevoir et réaliser une solution prête à l'emploi pour le fameux Cloud si difficile à définir. Ici, il s'agit d'être capable d'ajouter des nouveaux réplicas en lecture seule à la volée pour encaisser les pics de trafic, créer des instances de développement d'un clic, etc.

Cet article ne couvre qu'une petite sélection des sujets abordés à la conférence, je compte sur Guillaume pour lui aussi vous parler de CHAR(11), mais il faudra peut être attendre son retour des RMLL (quelle énergie !).

Tags

PostgreSQLFr Conferences Skytools

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 13 Jul 2011 17:30:00 +0200 http://tapoueh.org/blog/2011/07/13-de-retour-de-char11.html Back From CHAR(11) http://tapoueh.org/blog/2011/07/13-back-from-char11.html Back From CHAR(11)
Wednesday, July 13 2011, 17:15

CHAR(11) finished somewhen in the night leading to today, if you consider the social events to be part of it, which I definitely do. This conference has been a very good one, both on the organisation side of things and of course for its content.

It began with a perspective about the evolution of replication solutions, by Jan Wieck himself. In some way Skytools is an evolution of Slony, in the sense that it reuses the same concepts, a part of the design, and even share bits of the implementation (like the txid_snapshot datatype that were added in PostgreSQL 8.3). The evolution occured in choosing a subset of the features of Slony and then simplifying the user interface as much as possible. And with Skytools 3.0, those features that were removed but still are useful to solve real-life problems are now available too.

Of course the talk did approach the other replication solutions (not just the trigger based ones), and did compare RServ to Bucardo for example. And then all those were compared to the PostgreSQL core replication facilities, which are quite a different animal. It was a really nice keynote here, preparing the audience minds to make the most out of all the other talks.

I will not review all the talks in details, as I'm pretty sure some other attendees will turn into reporters themselves: scaling the write load!

Still repmgr got its share of attention. Greg Smith and Cédric Villemain did present both how to do read scaling and auto failover management with this tool, going into fine details about how it works internally and how to best design your services architecture for maximum data availibility. The question and answers section led to insist on the fact that you can not have data availibility with less than 3 production nodes.

Magnus Hagander detailed how flexible the core protocol support for replication (and streaming) really is. That flexibility means that you can quite easily talk this protocol from any application, and the idea of a wal proxy did pop out again (see Back from PgCon2010 article for my first mentionning of the idea). The main difference is that we now have synchronous replication support, so that the proxy could be trusted both for archiving and serving standbys.

Of course Simon still has lots of ideas about next 10 years of replication oriented projects for core PostgreSQL code, and his talk nicely summarized the previous 7 years. Future is bright, and guess what, it's beginning today!

We also heard about Heroku, and these guys are doing crazy impressive things. Like running 150 000 PostgreSQL instances, for example, showing that you can actually use our prefered database server in the hosting business. I expect that the maturing solution and tool sets providing data availibility are soon to be a game changer here. What they are doing is designing a flexible data architecture with strong guarantees (no data loss). The cloud elasticity is reaching out from the stateless services, and those guys are making it happen now.

May you live in interresting times!

Tags

PostgreSQL skytools Conferences

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 13 Jul 2011 17:15:00 +0200 http://tapoueh.org/blog/2011/07/13-back-from-char11.html Muse setup revised http://tapoueh.org/blog/2011/07/blog/2011/07/05-muse-setup-revised.html Most of you are probably reading my posts directly in their RSS reader tools (mine is gnus thanks to the Gwene service), so you probably missed it, but I just pushed a whole new version of my website, still using Emacs Muse as the engine.

My setup is tentatively called tapoueh.el and browsable online. It consists of some tweaks on top of Muse, so that I can enjoy tags and proper rss support. By proper, I mean that I want to be able to produce as many topic RSS feeds from a single blog, and thanks to the tags support that's now what I have.

The RSS handling and the tagging system are adhoc code, and this very article begins like this:

#author Dimitri Fontaine
#title  Muse setup revised
#date   20110705-19:55
#tags   Emacs Muse

All the information for the site navigation are taken from there, and at long last the RSS I publish now contains proper URLs without abusing anchors, as in the previous link which is a compatibility page in case you had some bookmarks. The compat only works with javascript (did you know that anchors are not part of the URL that is sent to the server, so that you can't apply RedirectMatch or other tweaks?), but all it needs is 2 lines of code, so I guess that's not so bad.

var anchor = window.location.hash;
document.location.href=document.getElementById(anchor).href;

I hope you like the new setup as much as I do, even if I'm left with some debugging to do. That's the price to pay for doing it yourself I guess. But I still don't know of a ready to use solution (as in off the shelf) that meet my criteria for web publishing. More on that topic another time.

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 05 Jul 2011 19:55:00 +0200 http://tapoueh.org/blog/2011/07/blog/2011/07/05-muse-setup-revised.html Prêt pour CHAR(11) ? http://tapoueh.org/blog/2011/07/04-pret-pour-char11.html Prêt pour CHAR(11) ?
Monday, July 04 2011, 20:15

La semaine prochaine déjà se tient CHAR(11), la conférence spécialisée sur le Clustering, la Haute Disponibilité et la Réplication avec PostgreSQL. C'est en Europe, à Cambridge cette fois, et c'est en anglais même si plusieurs compatriotes seront dans l'assistance.

Si vous n'avez pas encore jeté un œil au programme, je vous encourage à le faire. Même si vous n'aviez pas prévu de venir… parce qu'il y a de quoi vous faire changer d'avis !

Il est déjà difficile de suivre les listes de diffusions PostgreSQL en anglais, pour une simple question de temps, mais parfois la barrière de la langue peut également jouer. Alors si vous n'aviez pas bien suivi, je me permets de préciser qui sont les principaux intervenants à cette conférence.

Jan Wieck assure la première intervention avec un rétrospectif des solutions de réplication pour PostgreSQL. Il a initié Slony et continue d'être très actif dans son architecture et son développement.

Greg Smith, un collègue chez 2ndQuadrant, est monsieur performances « bas niveau » : sa spécialité est de tirer le meilleur de votre matériel, de votre configuration serveur, de PostgreSQL lui-même, et des requêtes que vous lui soumettez. Son livre PostgreSQL High Performance est un incontournable, à ce titre traduit en français.

Nous avons ensuite Magnus Hagander qui a rejoint récemment la core team (l'organisation centrale du projet), et qui contribue depuis plus de 10 ans au code de PostgreSQL.

Simon Riggs, lui aussi un de nos collègues, a réalisé le PITR, l'archivage des journaux de transactions, la réplication asynchrone et pour la prochaine version de PostgreSQL, la réplication synchrone.

Hannu Krosing (devinez où il travaille ?) a conçu l'architecture (et les outils) qui permettent à Skype d'annoncer une « scalability » infinie, en tout cas annoncée pour supporter jusqu'à 1 milliard d'utilisateurs.

Koichi Suzuki dirige les efforts du produit prometteur PostgreS-XC, un bel exemple de collaboration entre différents acteurs du marché, ici EnterpriseDB et NTT Open Source Software Center. Ce qui montre une fois de plus que l'Open Source est solidement ancré dans entreprises commerciales.

Bien sûr, Cédric et moi-même, de la partie française de 2ndQuadrant, serons de la partie. Nous interviendrons sur des sujets que nous connaissons bien pour avoir participé à leur développement et pour les déployer et les maintenir en production, repmgr et Londiste.

Et je passe sur d'autres profils, dont les sujets ne serront pas moins intéressants. Bref, si réplication et cluster sont des thèmes que vous voulez conjuguer avec PostgreSQL, c'est l'endroit où passer le début de la semaine prochaine !

Tags

PostgreSQLfr Conferences skytools

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 04 Jul 2011 20:15:00 +0200 http://tapoueh.org/blog/2011/07/04-pret-pour-char11.html Multi-Version support for Extensions http://tapoueh.org/blog/2011/06/29-multi-version-support-for-extensions.html Multi-Version support for Extensions
Wednesday, June 29 2011, 09:50

 

We still have this problem to solve with extensions and their packaging. How to best organize things so that your extension is compatible with before 9.1 and 9.1 and following releases of PostgreSQL?

Well, I had to do it for the ip4r contribution, and I wanted the following to happen:

dpkg-deb: building package `postgresql-8.3-ip4r' ...
dpkg-deb: building package `postgresql-8.4-ip4r' ...
dpkg-deb: building package `postgresql-9.0-ip4r' ...
dpkg-deb: building package `postgresql-9.1-ip4r' ...

And here's a simple enough way to achieve that. First, you have to get your packaging ready the usual way, and to install the build dependencies. Then realizing that /usr/share/postgresql-common/supported-versions from the latest postgresql-common package will only return 8.3 in lenny (yes, I'm doing some backporting here), we have to tweak it.

postgresql-server-dev-8.4
postgresql-server-dev-9.0
postgresql-server-dev-9.1
postgresql-server-dev-all

$ sudo dpkg-divert \
--divert /usr/share/postgresql-common/supported-versions.distrib \
--rename /usr/share/postgresql-common/supported-versions

$ cat /usr/share/postgresql-common/supported-versions
#! /bin/bash

dpkg -l postgresql-server-dev-* \
| awk -F '[ -]' '/^ii/ && ! /server-dev-all/ {print $6}'

Now we are allowed to build our extension for all those versions, so we add 9.1 to the debian/pgversions file. And debuild will do the right thing now, thanks to pg_buildext from postgresql-server-dev-all.

The problem we face is that the built is not an extension as in 9.1, so things like \dx in psql and CREATE EXTENSION will not work out of the box. First, we need a control file. Then we need to remove the transaction control from the install script (here, ip4r.sql), and finally, this script needs to be called ip4r--1.05.sql. Here's how I did it:

$ cat ip4r.control
comment = 'IPv4 and IPv4 range index types'
default_version = '1.05'
relocatable = yes

$ cat debian/postgresql-9.1-ip4r.install
debian/ip4r-9.1/ip4r.so usr/lib/postgresql/9.1/lib
ip4r.control usr/share/postgresql/9.1/extension
debian/ip4r-9.1/ip4r.sql usr/share/postgresql/9.1/extension

$ cat debian/postgresql-9.1-ip4r.links
usr/share/postgresql/9.1/extension/ip4r.sql usr/share/postgresql/9.1/extension/ip4r--1.05.sql

Be careful not to forget to remove any and all BEGIN; and COMMIT; lines from the ip4r.sql file, which meant that I also removed support for Rtree, which is not relevant for modern versions of PostgreSQL saith the script (post 8.2). That means I'm not publishing this very work yet, but I wanted to share the debian/postgresql-9.1-extension.links idea.

Notice that I didn't change anything about the .sql.in make rule, so I didn't have to use the support for module_pathname in the control file.

Now, after the usual debuild step, I can just sudo debi to install all the just build packages and CREATE EXTENSION will run fine. And in 9.0 you get the old way to install it, but it still works:

$ psql -U postgres --cluster 9.0/main -1 \
-f /usr/share/postgresql/9.0/contrib/ip4r.sql
<lots of chatter>

$ psql -U postgres --cluster 9.1/main -c 'create extension ip4r;'
CREATE EXTENSION

That's it :)

Tags

PostgreSQL debian Extensions release ip4r 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 29 Jun 2011 09:50:00 +0200 http://tapoueh.org/blog/2011/06/29-multi-version-support-for-extensions.html Don't be afraid of 'cl http://tapoueh.org/blog/2011/06/blog/2011/06/20-dont-be-afraid-of-cl.html In this blog article, you're shown a quite long function that loop through your buffers to find out if any of them is associated with a file whose full name includes "projects". Well, you should not be afraid of using cl:

(require 'cl)
(loop for b being the buffers
      when (string-match "projects" (or (buffer-file-name b) ""))
      return t)

If you want to collect the list of buffers whose name matches your test, then replace return t by collect b and you're done. Really, this loop thing is worth learning.

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 20 Jun 2011 00:15:00 +0200 http://tapoueh.org/blog/2011/06/blog/2011/06/20-dont-be-afraid-of-cl.html Don't be afraid of 'cl http://tapoueh.org/blog/2011/06/20-dont-be-afraid-of-cl.html Don't be afraid of 'cl
Monday, June 20 2011, 00:15

In this blog article, you're shown a quite long function that loop through your buffers to find out if any of them is associated with a file whose full name includes "projects". Well, you should not be afraid of using cl:

(require 'cl)
(loop for b being the buffers
      when (string-match "projects" (or (buffer-file-name b) ""))
      return t)

If you want to collect the list of buffers whose name matches your test, then replace return t by collect b and you're done. Really, this loop thing is worth learning.

Tags

Emacs

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 20 Jun 2011 00:15:00 +0200 http://tapoueh.org/blog/2011/06/20-dont-be-afraid-of-cl.html Back from Ottawa, preparing for Cambridge http://tapoueh.org/blog/2011/05/30-back-from-ottawa-preparing-for-cambridge.html Back from Ottawa, preparing for Cambridge
Monday, May 30 2011, 11:00

 

While Magnus is all about PG Conf EU already, you have to realize we're just landed back from PG Con in Ottawa. My next stop in the annual conferences is CHAR 11, the Clustering, High Availability and Replication conference in Cambridge, 11-12 July. Yes, on the old continent this time.

This year's pgcon hot topics, for me, have been centered around a better grasp at SSI and DDL Triggers. Having those beasts in PostgreSQL would allow for auditing, finer privileges management and some more automated replication facilities. Imagine that ALTER TABLE is able to fire a trigger, provided by Londiste or Slony, that will do what's needed on the cluster by itself. That would be awesome, wouldn't it?

At CHAR 11 I'll be talking about Skytools 3. You know I've been working on its debian packaging, now is the time to review the documentation and make there something as good looking as the monitoring system are...

Well, expect some news and a nice big picture diagram overview soon, if work schedule leaves me anytime that's what I want to be working on now.

Tags

PostgreSQL debian pgcon Conferences skytools 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 30 May 2011 11:00:00 +0200 http://tapoueh.org/blog/2011/05/30-back-from-ottawa-preparing-for-cambridge.html el-get 2.2 http://tapoueh.org/blog/2011/05/blog/2011/05/26-el-get-22.html We've spotted a little too late for our own taste a discrepancy in the source tree: a work in progress patch landed in git just before to release el-get stable. So we cleaned the tree (thanks again Julien), branched a stable maintenance tree, and released 2.2 from there.

You're back to enjoying el-get :)

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 26 May 2011 12:00:00 +0200 http://tapoueh.org/blog/2011/05/blog/2011/05/26-el-get-22.html el-get 2.2 http://tapoueh.org/blog/2011/05/26-el-get-22.html el-get 2.2
Thursday, May 26 2011, 12:00

We've spotted a little too late for our own taste a discrepancy in the source tree: a work in progress patch landed in git just before to release el-get stable. So we cleaned the tree (thanks again Julien), branched a stable maintenance tree, and released 2.2 from there.

You're back to enjoying el-get :)

Tags

Emacs el-get release

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 26 May 2011 12:00:00 +0200 http://tapoueh.org/blog/2011/05/26-el-get-22.html el-get 2.1 http://tapoueh.org/blog/2011/05/blog/2011/05/26-el-get-21.html Current el-get status is stable, ready for daily use and packed with extra features that make life easier. There are some more things we could do, as always, but they will be about smoothing things further.

Latest released version

el-get version 2.1 is available, with a boatload of features, including autoloads support, byte-compiling in an external clean room Emacs instance, custom support, lazy initialisation support (defering all init functions to eval-after-load), and multi repositories ELPA support.

Version numbering

Version String are now inspired by how Emacs itself numbers its versions. First is the major version number, then a dot, then the minor version number. The minor version number is 0 when still developping the next major version. So 3.0 is a developer release while 3.1 will be the next stable release.

Please note that this versioning policy has been picked while backing 1.2~dev, so 1.0 was a stable release in fact. Ah, history.

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 26 May 2011 10:00:00 +0200 http://tapoueh.org/blog/2011/05/blog/2011/05/26-el-get-21.html el-get 2.1 http://tapoueh.org/blog/2011/05/26-el-get-21.html el-get 2.1
Thursday, May 26 2011, 10:00

Current el-get status is stable, ready for daily use and packed with extra features that make life easier. There are some more things we could do, as always, but they will be about smoothing things further.

Latest released version

el-get version 2.1 is available, with a boatload of features, including autoloads support, byte-compiling in an external clean room Emacs instance, custom support, lazy initialisation support (defering all init functions to eval-after-load), and multi repositories ELPA support.

Version numbering

Version String are now inspired by how Emacs itself numbers its versions. First is the major version number, then a dot, then the minor version number. The minor version number is 0 when still developping the next major version. So 3.0 is a developer release while 3.1 will be the next stable release.

Please note that this versioning policy has been picked while backing 1.2~dev, so 1.0 was a stable release in fact. Ah, history.

Tags

Emacs el-get release

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 26 May 2011 10:00:00 +0200 http://tapoueh.org/blog/2011/05/26-el-get-21.html Preparing for PGCON http://tapoueh.org/blog/2011/05/12-preparing-for-pgcon.html Preparing for PGCON
Thursday, May 12 2011, 10:30

It's this time of the year again, the main international PostgreSQL Conference is next week in Ottawa, Canada. If previous years are any indication, this will be great event where to meet with a lot of the members of your community. The core team will be there, developers will be there, and we will meet with users and their challenging use cases.

This is a very good time to review both what you did in the project those last 12 months, and what you plan to do next year. To help with that, several meeting events are organized. They're like a whole-day round table with a kind of an agenda, with a limited number of invited people in, and very intense on-topic discussions about how to organize ourselves for another great year of innovation in PostgreSQL.

Then we have two days full of talks where I usually learn some new aspect of the project or of the product, and where ideas tend to just pop-up in a continuous race. Being away from home and with people you see only once a year (some of them more than that of course, hi European fellows!) seems to allow for some broader thinking.

The talks I want to go to include Database Scalability Patterns: Sharding for Unlimited Growth by Robert Treat, Maintaining Terabytes by Selena Deckelmann, NTT’s Case Report by Tetsuo Sakata, Hacking the Query Planner by Tom Lane. That's for a first day, right?

Then, on the second day, I notice Range Types by Jeff Davis, SP-GiST - a new indexing infrastructure for PostgreSQL by Oleg and Teodor, The Write Stuff by Greg Smith (a colleague at 2ndQuadrant).

I will miss Serializable Snapshot Isolation in Postgres by Kevin Grittner and Dan Ports, unfortunately, because I'll be talking about Extensions Development at the same time.

Well of course this list is just a first selection, hallway tracks are often what guides me through talks or make me skip some.

See you there!

Tags

PostgreSQL pgcon Extensions

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 12 May 2011 10:30:00 +0200 http://tapoueh.org/blog/2011/05/12-preparing-for-pgcon.html Mailq modeline display http://tapoueh.org/blog/2011/05/blog/2011/05/05-mailq-modeline-display.html If you've not been following along, you might have missed it: it appears to me that even today, in 2011, mail systems work much better when setup the old way. Meaning with a local MTA for outgoing mail. With some niceties, such as sender dependent relayhost maps.

That's why I needed M-x mailq to display the mail queue and have some easy shortcuts in order to operate it (mainly f runs the command mailq-mode-flush, but per site and per id delivery are useful too).

Now, I also happen to setup outgoing mail routes to walk through an SSH tunnel, which thanks to both ~/.ssh/config and cssh (C-= runs the command cssh-term-remote-open, with completion) is a couple of keystrokes away to start. Well it still happens to me to forget about starting it, which causes mails to hold in a queue until I realise it's not delivered, which always take just about too long.

A solution I've been thinking about is to add a little flag in the modeline in my gnus *Group* and *Summary* buffers. The flag would show up as ✔ when no mail is queued and waiting for me to open the tunnel, or ✘ as soon as the queue is not empty. Here's what it looks like here:

Well I'm pretty happy with the setup. The flag is refreshed every minute, and here's as an example how I did setup mailq in my el-get-sources setup:

         (:name mailq
                :after (lambda () (mailq-modeline-display)))

I'm not sure how many of you dear readers are using a local MTA to deliver your mails, but well, the ones who do (or consider doing so) might even find this article useful!

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 05 May 2011 14:10:00 +0200 http://tapoueh.org/blog/2011/05/blog/2011/05/05-mailq-modeline-display.html Mailq modeline display http://tapoueh.org/blog/2011/05/05-mailq-modeline-display.html Mailq modeline display
Thursday, May 05 2011, 14:10

If you've not been following along, you might have missed it: it appears to me that even today, in 2011, mail systems work much better when setup the old way. Meaning with a local MTA for outgoing mail. With some niceties, such as sender dependent relayhost maps.

That's why I needed M-x mailq to display the mail queue and have some easy shortcuts in order to operate it (mainly f runs the command mailq-mode-flush, but per site and per id delivery are useful too).

Now, I also happen to setup outgoing mail routes to walk through an SSH tunnel, which thanks to both ~/.ssh/config and cssh (C-= runs the command cssh-term-remote-open, with completion) is a couple of keystrokes away to start. Well it still happens to me to forget about starting it, which causes mails to hold in a queue until I realise it's not delivered, which always take just about too long.

A solution I've been thinking about is to add a little flag in the modeline in my gnus *Group* and *Summary* buffers. The flag would show up as ✔ when no mail is queued and waiting for me to open the tunnel, or ✘ as soon as the queue is not empty. Here's what it looks like here:

Well I'm pretty happy with the setup. The flag is refreshed every minute, and here's as an example how I did setup mailq in my el-get-sources setup:

         (:name mailq
                :after (lambda () (mailq-modeline-display)))

I'm not sure how many of you dear readers are using a local MTA to deliver your mails, but well, the ones who do (or consider doing so) might even find this article useful!

Tags

Emacs el-get modeline cssh mailq postfix

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 05 May 2011 14:10:00 +0200 http://tapoueh.org/blog/2011/05/05-mailq-modeline-display.html Tables and Views dependencies http://tapoueh.org/blog/2011/05/04-tables-and-views-dependencies.html Tables and Views dependencies
Wednesday, May 04 2011, 11:45

 

Let's say you need to ALTER TABLE foo ALTER COLUMN bar TYPE bigint;, and PostgreSQL is helpfully telling you that no you can't because such and such views depend on the column. The basic way to deal with that is to copy paste from the error message the names of the views involved, then prepare a script wherein you first DROP VIEW ...; then ALTER TABLE and finally CREATE VIEW again, all in the same transaction.

So you have to copy paste also the view definitions. With large view definitions, it quickly gets cumbersome to do so. Well when you're working on operations, you have to bear in mind that cumbersome is a synonym for error prone, in fact — so you want another solution if possible.

Oh, and the other drawback of this solution is that ALTER TABLE will first take a LOCK on the table, locking out any activity. And more than that, the lock acquisition will queue behind current activity on the table, which means waiting for a fairly long time and damaging the service quality on a moderately loaded server.

It's possible to abuse the system catalogs in order to find all views that depend on a given table, too. For that, you have to play with pg_depend and you have to know that internally, a view is in fact a rewrite rule. Then here's how to produce the two scripts that we need:

=# \t
Showing only tuples.

=# \o /tmp/drop.sql
=# select 'DROP VIEW ' || views || ';'
     from (select distinct(r.ev_class::regclass) as views
            from pg_depend d join pg_rewrite r on r.oid = d.objid
           where refclassid = 'pg_class'::regclass
             and refobjid = 'SCHEMA.TABLENAME'::regclass
             and classid = 'pg_rewrite'::regclass
             and pg_get_viewdef(r.ev_class, true) ~ 'COLUMN_NAME') as x;

=# \o /tmp/create.sql
=# select 'CREATE VIEW ' || views || E' AS \n'
       || pg_get_viewdef(views, true) || ';'
     from (select distinct(r.ev_class::regclass) as views
          from pg_depend d join pg_rewrite r on r.oid = d.objid
         where refclassid = 'pg_class'::regclass
           and refobjid = 'SCHEMA.TABLENAME'::regclass
           and classid = 'pg_rewrite'::regclass
           and pg_get_viewdef(r.ev_class, true) ~ 'COLUMN_NAME') as x;

=# \o

Replace SCHEMA.TABLENAME and COLUMN_NAME with your targets here and the first query should give you one row per candidate view. Well if you're not using the \o trick, that is — if you do, check out the generated file instead, with \! cat /tmp/drop.sql for example.

Please note that this catalog query is not accurate, as it will select as a candidate any view that will by chance both depend on the target table and contain the column_name in its text definition. So either filter out the candidates properly (by proper proof reading then another WHERE clause), or just accept that you might DROP then CREATE again more views than need be.

If you need some more details about the \t \o sequence you might be interested in this older article about resetting sequences.

Tags

PostgreSQL catalogs

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 04 May 2011 11:45:00 +0200 http://tapoueh.org/blog/2011/05/04-tables-and-views-dependencies.html Extension module_pathname and .sql.in http://tapoueh.org/blog/2011/05/02-extension-module_pathname-and-sqlin.html Extension module_pathname and .sql.in
Monday, May 02 2011, 17:30

 

While currently too busy at work to deliver much Open Source contributions, let's debunk an old habit of PostgreSQL extension authors. It's all down to copy pasting from contrib, and there's no reason to continue doing $libdir this way ever since 7.4 days.

Let's take an example here, with the prefix extension. This one too will need some love, but is still behind on my spare time todo list, sorry about that. So, in the prefix.sql.in we read

  CREATE OR REPLACE FUNCTION prefix_range_in(cstring)
  RETURNS prefix_range
  AS 'MODULE_PATHNAME'
  LANGUAGE 'C' IMMUTABLE STRICT;

Two things are to change here. First, the PostgreSQL backend will understand just fine if you just say AS '$libdir/prefix'. So you have to know in the sql script the name of the shared object library, but if you do, you can maintain directly a prefix.sql script instead.

The advantage is that you now can avoid a compatibility problem when you want to support PostgreSQL from 8.2 to 9.1 in your extension (older than that and it's no longer supported). You directly ship your script.

For compatibility, you could also use the control file module_pathname property. But for 9.1 you then have to add a implicit Make rule so that the script is derived from your .sql.in. And as you are managing several scripts — so that you can handle versioning and upgrades — it can get hairy (hint, you need to copy prefix.sql as prefix--1.1.1.sql, then change its name at next revision, and think about upgrade scripts too). The module_pathname facility is better to keep for when managing more than a single extension in the same directory, like the SPI contrib is doing.

Sure, maintaining an extension that targets both antique releases of PostgreSQL and CREATE EXTENSION super-powered one(s) (not yet released) is a little more involved than that. We'll get back to that, as some people are still pioneering the movement.

On my side, I'm working with some debian developer on how to best manage the packaging of those extensions, and this work could end up as a specialized policy document and a coordinated team of maintainers for all things PostgreSQL in debian. This will also give some more steam to the PostgreSQL effort for debian packages: the idea is to maintain packages for all supported version (from 8.2 up to soon 9.1), something debian itself can not commit to.

Tags

PostgreSQL debian Extensions release prefix 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 02 May 2011 17:30:00 +0200 http://tapoueh.org/blog/2011/05/02-extension-module_pathname-and-sqlin.html Emacs and PostgreSQL, PL line numbering http://tapoueh.org/blog/2011/04/blog/2011/04/23-emacs-and-postgresql-pl-line-numbering.html  

A while ago I've been fixing and publishing pgsql-linum-format separately. That allows to number PL/whatever code lines when editing from Emacs, and it's something very useful to turn on when debugging.

The carrets on the fringe in the emacs window are the result of (setq-default indicate-buffer-boundaries 'left) and here it's just overloading the image somehow. But the idea is to just M-x linum-mode when you need it, at least that's my usage of it.

You can use el-get to easily get (then update) this little Emacs extension.

]]> dim@tapoueh.org (Dimitri Fontaine) Sat, 23 Apr 2011 10:30:00 +0200 http://tapoueh.org/blog/2011/04/blog/2011/04/23-emacs-and-postgresql-pl-line-numbering.html Emacs and PostgreSQL, PL line numbering http://tapoueh.org/blog/2011/04/23-emacs-and-postgresql-pl-line-numbering.html Emacs and PostgreSQL, PL line numbering
Saturday, April 23 2011, 10:30

 

A while ago I've been fixing and publishing pgsql-linum-format separately. That allows to number PL/whatever code lines when editing from Emacs, and it's something very useful to turn on when debugging.

The carrets on the fringe in the emacs window are the result of (setq-default indicate-buffer-boundaries 'left) and here it's just overloading the image somehow. But the idea is to just M-x linum-mode when you need it, at least that's my usage of it.

You can use el-get to easily get (then update) this little Emacs extension.

Tags

Emacs el-get pgsql-linum-format

]]> dim@tapoueh.org (Dimitri Fontaine) Sat, 23 Apr 2011 10:30:00 +0200 http://tapoueh.org/blog/2011/04/23-emacs-and-postgresql-pl-line-numbering.html Emacs Kicker http://tapoueh.org/blog/2011/04/blog/2011/04/15-emacs-kicker.html Following up on the very popular emacs-starter-kit, I'm now proposing the emacs-kicker. It's about the .emacs file you've seen in older posts here, which I maintain for some colleagues. After all, if they find it useful, some more people might to, so I've decided to publish it.

What you'll find is a very simple 128 lines Emacs user init file, based on el-get for external packages. A not so random selection of those is used, here's the list when you hide some details:

 '(el-get                       ; el-get is self-hosting
   escreen                      ; screen for emacs, C-\ C-h
   php-mode-improved            ; if you're into php...
   psvn                         ; M-x svn-status
   switch-window                ; takes over C-x o
   auto-complete                ; complete as you type with overlays
   emacs-goodies-el             ; the debian addons for emacs
   yasnippet                    ; powerful snippet mode
   zencoding-mode               ; http://www.emacswiki.org/emacs/ZenCoding
   (:name buffer-move           ; move buffers around in windows
   (:name smex                  ; a better (ido like) M-x
   (:name magit                 ; git meet emacs, and a binding
   (:name goto-last-change      ; move pointer back to last change

Another interresting thing to note in this kicker is a choice of some key bindings that are rather unusual (yet) I guess.

(global-set-key (kbd "C-x C-b") 'ido-switch-buffer)
(global-set-key (kbd "C-x C-c") 'ido-switch-buffer)
(global-set-key (kbd "C-x B") 'ibuffer)

Yes, you see that I've rebound C-x C-c to switching buffers. That key is really easy to use and I don't think that M-x kill-emacs deserves it. Keys that are so easy to use should be kept for frequent actions, and quiting emacs is a once-a-day to once-a-month action here. And you can still quit from the window manager button or from the menu or from M-x.

Also Mac users are not left behind, you will see some settings that either are adapted to the system (like choosing another font, keep displaying the menu-bar or not installing the darkish tango-color-mode on this system, where it renders poorly in my opinion), as you can see here:

(if (string-match "apple-darwin" system-configuration)
    (set-face-font 'default "Monaco-13")
  (set-frame-font "Monospace-10"))

(when (string-match "apple-darwin" system-configuration)
  (setq mac-allow-anti-aliasing t)
  (setq mac-command-modifier 'meta)
  (setq mac-option-modifier 'none))

So all in all, I don't expect this emacs-kicker to please everyone, but I expect it to be simple and rich enough (thanks to el-get), and it should be a good kick start that's easy to adapt.

If you want to try it without installing it it's very easy to do so. Just clone the git repository then start an Emacs that will use this. For example that could be, using the excellent Emacs For MacOSX:

 $ /Applications/Emacs.app/Contents/MacOS/Emacs -Q -l init.el

I hope some readers will find it useful! :)

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 15 Apr 2011 21:30:00 +0200 http://tapoueh.org/blog/2011/04/blog/2011/04/15-emacs-kicker.html Emacs Kicker http://tapoueh.org/blog/2011/04/15-emacs-kicker.html Emacs Kicker
Friday, April 15 2011, 21:30

Following up on the very popular emacs-starter-kit, I'm now proposing the emacs-kicker. It's about the .emacs file you've seen in older posts here, which I maintain for some colleagues. After all, if they find it useful, some more people might to, so I've decided to publish it.

What you'll find is a very simple 128 lines Emacs user init file, based on el-get for external packages. A not so random selection of those is used, here's the list when you hide some details:

 '(el-get                       ; el-get is self-hosting
   escreen                      ; screen for emacs, C-\ C-h
   php-mode-improved            ; if you're into php...
   psvn                         ; M-x svn-status
   switch-window                ; takes over C-x o
   auto-complete                ; complete as you type with overlays
   emacs-goodies-el             ; the debian addons for emacs
   yasnippet                    ; powerful snippet mode
   zencoding-mode               ; http://www.emacswiki.org/emacs/ZenCoding
   (:name buffer-move           ; move buffers around in windows
   (:name smex                  ; a better (ido like) M-x
   (:name magit                 ; git meet emacs, and a binding
   (:name goto-last-change      ; move pointer back to last change

Another interresting thing to note in this kicker is a choice of some key bindings that are rather unusual (yet) I guess.

(global-set-key (kbd "C-x C-b") 'ido-switch-buffer)
(global-set-key (kbd "C-x C-c") 'ido-switch-buffer)
(global-set-key (kbd "C-x B") 'ibuffer)

Yes, you see that I've rebound C-x C-c to switching buffers. That key is really easy to use and I don't think that M-x kill-emacs deserves it. Keys that are so easy to use should be kept for frequent actions, and quiting emacs is a once-a-day to once-a-month action here. And you can still quit from the window manager button or from the menu or from M-x.

Also Mac users are not left behind, you will see some settings that either are adapted to the system (like choosing another font, keep displaying the menu-bar or not installing the darkish tango-color-mode on this system, where it renders poorly in my opinion), as you can see here:

(if (string-match "apple-darwin" system-configuration)
    (set-face-font 'default "Monaco-13")
  (set-frame-font "Monospace-10"))

(when (string-match "apple-darwin" system-configuration)
  (setq mac-allow-anti-aliasing t)
  (setq mac-command-modifier 'meta)
  (setq mac-option-modifier 'none))

So all in all, I don't expect this emacs-kicker to please everyone, but I expect it to be simple and rich enough (thanks to el-get), and it should be a good kick start that's easy to adapt.

If you want to try it without installing it it's very easy to do so. Just clone the git repository then start an Emacs that will use this. For example that could be, using the excellent Emacs For MacOSX:

 $ /Applications/Emacs.app/Contents/MacOS/Emacs -Q -l init.el

I hope some readers will find it useful! :)

Tags

Emacs debian el-get switch-window emacs-kicker

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 15 Apr 2011 21:30:00 +0200 http://tapoueh.org/blog/2011/04/15-emacs-kicker.html Some notes about Skytools3 http://tapoueh.org/blog/2011/04/11-some-notes-about-skytools3.html Some notes about Skytools3
Monday, April 11 2011, 11:30

I've been working on skytools3 packaging lately. I've been pushing quite a lot of work into it, in order to have exactly what I needed out of the box, after some 3 years of production and experiences with the products. Plural, yes, because even if pgbouncer and plproxy are siblings to the projets (same developers team, separate life cycle and releases), then skytools still includes several sub-projects.

Here's what the skytools3 packaging is going to look like:

skytools3              Skytool's replication and queuing
python-pgq3            Skytool's PGQ python library
python-skytools3       python scripts framework for skytools
skytools-ticker3       PGQ ticker daemon service
skytools-walmgr3       high-availability archive and restore commands
postgresql-8.4-pgq3    PGQ server-side code (C module for PostgreSQL)
postgresql-9.0-pgq3    PGQ server-side code (C module for PostgreSQL)

This split is needed so that you can install your daemons (we call them consumers) on separate machines than where you run PostgreSQL. But for the walmgr part, it makes no sense to install it if you don't have a local PostgreSQL service, as it's providing archive and restore commands. Then the ticker, you're free to run it on any machine really, so just package it this way (in skytools3 the ticker is written in C and does not depend on the python framework any more).

What you can't see here yet is the new goodies that wraps it as a quality debian package. A new skytools user is created for you when you install the skytools3 package (which contains the services), along with a skeleton file /etc/skytools.ini and a user directory /etc/skytools/. Put in there your services configuration file, and register those service in the /etc/skytools.ini file itself. Then they will get cared about in the init sequence at startup and shutdown of your server.

The services will run under the skytools system user, and will default to put their log into /var/log/skytools/. The pidfile will get into /var/run/skytools/. All integrated, automated.

Next big TODO is about documentation, reviewing it and polishing it, and I think that skytools3 will then get ready for public release. Yes, you read it right, it's happening this very year! I'm very excited about it, and have several architectures that will greatly benefit from the switch to skytools3. More on that later, though! (Yes, my to blog later list is getting quite long now).

Tags

PostgreSQL debian release skytools restore

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 11 Apr 2011 11:30:00 +0200 http://tapoueh.org/blog/2011/04/11-some-notes-about-skytools3.html towards pg_staging 1.0 http://tapoueh.org/blog/2011/03/29-towards-pg_staging-10.html towards pg_staging 1.0
Tuesday, March 29 2011, 15:30

If you don't remember about what pg_staging is all about, it's a central console from where to control all your PostgreSQL databases. Typically you use it to manage your development and pre-production setup, where developers ask you pretty often to install them some newer dump from the production, and you want that operation streamlined and easy.

Usage

The typical session would be something like this:

pg_staging> databases foodb.dev
                    foodb      foodb_20100824 :5432
           foodb_20100209      foodb_20100209 :5432
           foodb_20100824      foodb_20100824 :5432
                pgbouncer           pgbouncer :6432
                 postgres            postgres :5432

pg_staging> dbsizes foodb.dev
foodb.dev
           foodb_20100209: -1
           foodb_20100824: 104 GB
Total                    = 104 GB

pg_staging> restore foodb.dev
...
pg_staging> switch foodb.dev today

The list of supported commands is quite long now, and documented too (it comes with two man pages). The restore one is the most important and will create the database, add it to the pgbouncer setup, fetch the backup named dbname.`date -I`.dump, prepare a filtered object list (more on that), load pre SQL scripts, launch pg_restore, VACUUM ANALYZE the database when configured to do so, load the post SQL scripts then optionaly switch the pgbouncer setup to default to this new database.

Filtering

The newer option is called tablename_nodata_regexp, and here's its documentation in full:

List of table names regexp (comma separated) to restore without content. The pg_restore catalog TABLE DATA sections will get filtered out. The regexp is applied against schemaname.tablename and non-anchored by default.

This comes to supplement the schemas and schemas_nodata options, that allows to only restore objects from a given set of schemas (filtering out triggers that will calls function that are in the excluded schemas, like e.g. Londiste ones) or to restore only the TABLE definitions while skipping the TABLE DATA entries.

Setup

To setup your environment for pg_staging, you need to take some steps. It's not complex but it's fairly involved. The benefit is this amazingly useful central unique console to control as many databases as you need.

You need a pg_staging.ini file where to describe your environment. I typically name the sessions in there by the name of the database to restore followed by a dev or preprod extension.

You need to have all your backups available through HTTP, and as of now, served by the famous apache mod_dir directory listing. It's easy to add support to other methods, but is has not been done yet. You also need to have a cluster wide --globals-only backup available somewhere so that you can easily create the users etc you need from pg_staging.

You also need to run a pgbouncer daemon on each database server, allowing you to bypass editing connection strings when you switch a new database version live.

You also need to install the client script, have a local pgstaging system user and allow it to run the client script as root, so that it's able to control some services and edit pgbouncer.ini for you.

Status

I'm still using it a lot (several times a week) to manage a whole development and pre-production environment set, so the very low code activity of the project is telling that it's pretty stable (last series of commits are all bug fixes and round corners).

Given that, I'm thinking in terms of pg_staging 1.0 soon! Now is a pretty good time to try it and see how it can help you.

Tags

PostgreSQL skytools backup restore pg_staging

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 29 Mar 2011 15:30:00 +0200 http://tapoueh.org/blog/2011/03/29-towards-pg_staging-10.html Extensions in 9.1 http://tapoueh.org/blog/2011/03/01-extensions-in-91.html Extensions in 9.1
Tuesday, March 01 2011, 16:30

If you've not been following closely you might have missed out on extensions integration. Well, Tom spent some time on the patches I've been preparing for the last 4 months. And not only did he commit most of the work but he also enhanced some parts of the code (better factoring) and basically finished it.

At the previous developer meeting his advice was to avoid putting too much into the very first version of the patch for it to stand its chances of being integrated, and while in the review process more than one major PostgreSQL contributor expressed worries about the size of the patch and the number of features proposed. Which is the usual process.

Then what happened is that Tom finally took a similar reasoning as mine while working on the feature. To maximize the benefits, once you have the infrastructure in place, it's not that much more work to provide the really interesting features. What's complex is agreeing on what exactly are their specifications. And in the little time window we got on this commit fest (well, we hijacked about 2 full weeks there), we managed to get there.

So in the end the result is quite amazing, and you can see that on the documentation chapter about it: 35.15. Packaging Related Objects into an Extension.

All the contrib modules that are installing SQL objects into databases for you to use them are now converted to Extensions too, and will get released in 9.1 with an upgrade script that allows you to upgrade from unpackaged. That means that once you've upgraded from a past PostgreSQL release up to 9.1, it will be a command away for you to register extensions as such. I expect third party extension authors (from ip4r to temporal) to release a upgrade-from-unpackaged version of their work too.

Of course, a big use case of the extensions is also in-house PL code, and having version number and multi-stage upgrade scripts there will be fantastic too, I can't wait to work with such a tool set myself. Some later blog post will detail the benefits and usage. I'm already trying to think how much of this version and upgrade facility could be expanded to classic DDL objects…

So expect some more blog posts from me on this subject, I will have to talk about debian packaging an extension (it's getting damn easy with postgresql-server-dev-all — yes it has received some planing ahead), and about how to package your own extension, manage upgrades, turn your current pre-9.1 extension into a full blown extension, and maybe how to stop worrying about extension when you're a DBA.

If you have some features you would want to discuss for next releases, please do contact me!

Meanwhile, I'm very happy that this project of mine finally made it to core, it's been long in the making. Some years to talk about it and then finally 4 months of coding that I'll remember as a marathon. Many Thanks go to all who helped here, from 2ndQuadrant to early reviewers to people I talked to over beers at conferences… lots of people really.

To an extended PostgreSQL (and beyond) :)

Tags

PostgreSQL debian pgcon Conferences Extensions release ip4r 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 01 Mar 2011 16:30:00 +0100 http://tapoueh.org/blog/2011/03/01-extensions-in-91.html desktop-mode and readahead http://tapoueh.org/blog/2011/02/blog/2011/02/23-desktop-mode-and-readahead.html I'm using Desktop Save Mode so that Emacs knows to open again all the buffers I've been using. That goes quite well with how often I start Emacs, that is once a week or once a month. Now, M-x ibuffer last line is as following:

    718 buffers         19838205                  668 files, 15 processes

That means that at startup, Emacs will load that many files. In order not to have to wait until it's done doing so, I've setup things this way:

;; and the session
(setq desktop-restore-eager 20
      desktop-lazy-verbose nil)
(desktop-save-mode 1)
(savehist-mode 1)

Problem is that it's still slow. An idea I had was to use the readahead tool that allows reducing some distributions boot time. Of course this tool is not expecting the same file format as emacs-desktop uses. Still, converting is quite easy is some awk magic. Here's the result:

;;; dim-desktop.el --- Dimitri Fontaine
;;
;; Allows to prepare a readahead file list from desktop-save

(require 'desktop)

(defvar dim-desktop-file-readahead-list
  "~/.emacs.desktop.readahead"
  "*Where to save the emacs desktop `readahead` file list")

(defvar dim-desktop-filelist-command
  "gawk -F '[ \"]' '/desktop-.*-buffer/ {getline; if($4) print $4}' %s"
  "Command to run to prepare the readahead file list")

(defun dim-desktop-get-readahead-file-list (&optional filename dir)
  "get the file list for readahead from dekstop file in DIR, or ~"
  (with-temp-file (or filename dim-desktop-file-readahead-list)
    (insert
     (shell-command-to-string
      (format dim-desktop-filelist-command
              (expand-file-name desktop-base-file-name (or dir "~")))))))

;; This will not work because the hook is run before to add the buffers into
;; the desktop file.
;;
;;(add-hook 'desktop-save-hook 'dim-desktop-get-readahead-file-list)

;; so instead, advise the function
(defadvice desktop-save (after desktop-save-readahead activate)
  "Prepare a readahead(8) file for the desktop file"
  (dim-desktop-get-readahead-file-list))

(provide 'dim-desktop)

The awk construct getline allows to process the next line of the input file, which is very practical here (and in a host of other situations). Now that we have a file containing the list of files Emacs will load, we have to tweak the system to readahead those disk blocks. As I'm currently using KDE again, I've done it thusly:

% cat ~/.kde/Autostart/readahead.emacs.sh
#! /bin/bash

# just readahead the emacs desktop files
# this file listing is maintained directly from Emacs itself
readahead ~/.emacs.desktop.readahead

So, well, it works. The files that Emacs will need are pre-read, so at the time the desktop really gets to them, I see no more disk activity (laptops have a led to see that happening). But the desktop loading time has not changed...

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 23 Feb 2011 16:45:00 +0100 http://tapoueh.org/blog/2011/02/blog/2011/02/23-desktop-mode-and-readahead.html desktop-mode and readahead http://tapoueh.org/blog/2011/02/23-desktop-mode-and-readahead.html desktop-mode and readahead
Wednesday, February 23 2011, 16:45

I'm using Desktop Save Mode so that Emacs knows to open again all the buffers I've been using. That goes quite well with how often I start Emacs, that is once a week or once a month. Now, M-x ibuffer last line is as following:

    718 buffers         19838205                  668 files, 15 processes

That means that at startup, Emacs will load that many files. In order not to have to wait until it's done doing so, I've setup things this way:

;; and the session
(setq desktop-restore-eager 20
      desktop-lazy-verbose nil)
(desktop-save-mode 1)
(savehist-mode 1)

Problem is that it's still slow. An idea I had was to use the readahead tool that allows reducing some distributions boot time. Of course this tool is not expecting the same file format as emacs-desktop uses. Still, converting is quite easy is some awk magic. Here's the result:

;;; dim-desktop.el --- Dimitri Fontaine
;;
;; Allows to prepare a readahead file list from desktop-save

(require 'desktop)

(defvar dim-desktop-file-readahead-list
  "~/.emacs.desktop.readahead"
  "*Where to save the emacs desktop `readahead` file list")

(defvar dim-desktop-filelist-command
  "gawk -F '[ \"]' '/desktop-.*-buffer/ {getline; if($4) print $4}' %s"
  "Command to run to prepare the readahead file list")

(defun dim-desktop-get-readahead-file-list (&optional filename dir)
  "get the file list for readahead from dekstop file in DIR, or ~"
  (with-temp-file (or filename dim-desktop-file-readahead-list)
    (insert
     (shell-command-to-string
      (format dim-desktop-filelist-command
              (expand-file-name desktop-base-file-name (or dir "~")))))))

;; This will not work because the hook is run before to add the buffers into
;; the desktop file.
;;
;;(add-hook 'desktop-save-hook 'dim-desktop-get-readahead-file-list)

;; so instead, advise the function
(defadvice desktop-save (after desktop-save-readahead activate)
  "Prepare a readahead(8) file for the desktop file"
  (dim-desktop-get-readahead-file-list))

(provide 'dim-desktop)

The awk construct getline allows to process the next line of the input file, which is very practical here (and in a host of other situations). Now that we have a file containing the list of files Emacs will load, we have to tweak the system to readahead those disk blocks. As I'm currently using KDE again, I've done it thusly:

% cat ~/.kde/Autostart/readahead.emacs.sh
#! /bin/bash

# just readahead the emacs desktop files
# this file listing is maintained directly from Emacs itself
readahead ~/.emacs.desktop.readahead

So, well, it works. The files that Emacs will need are pre-read, so at the time the desktop really gets to them, I see no more disk activity (laptops have a led to see that happening). But the desktop loading time has not changed...

Tags

Emacs restore

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 23 Feb 2011 16:45:00 +0100 http://tapoueh.org/blog/2011/02/23-desktop-mode-and-readahead.html Back from FOSDEM http://tapoueh.org/blog/2011/02/07-back-from-fosdem.html Back from FOSDEM
Monday, February 07 2011, 11:10

This year we were in the main building of the conference, and apparently the booth went very well, solding lots of PostgreSQL merchandise etc. I had the pleasure to once again meet with the community, but being there only 1 day I didn't spend as much time as I would have liked with some of the people there.

In case you're wondering, my extension's talk went quite well, and several people were kind enough to tell me they appreciated it! There was video recording of it, so we will soon have proofs showing how bad it really was and how polite those people really are :)

I will soon be able to write an article series detailing what's an Extension and how you deal with them, either as a user or an author. Well in fact the goal is for any user to easily become an extension author, as I think lots of people are already maintaining server side code but missing tools to manage it properly. But that will begin once the patch is in, so that I present the real stuff rather than what I proposed to the community… Stay tuned!

Tags

PostgreSQL Conferences Extensions FOSDEM 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 07 Feb 2011 11:10:00 +0100 http://tapoueh.org/blog/2011/02/07-back-from-fosdem.html Going to FOSDEM http://tapoueh.org/blog/2011/02/01-going-to-fosdem.html Going to FOSDEM
Tuesday, February 01 2011, 13:35

A quick blog entry to say that yes:

And I will even do my Extension's talk which had a success at pgday.eu. The talk will be updated to include the last developments of the extension's feature, as some of it changed already in between, and to detail the plan for the ALTER EXTENSION ... UPGRADE feature that I'd like to see included as soon as 9.1, but time is running so fast.

In fact the design for the UPGRADE has been done and reviewed already, but there's yet to reach consensus on how to setup which is the upgrade file to use when upgrading from a given version to another. I've solved it in my patch, of course, by adding properties into the extension's control file. That's the best place to have that setup I think, it allows lots of flexibility, leave the extension's author in charge, and avoids any hard coding of any kind of assumptions about file naming or whatever.

Next days and reviews will tell us more about how the design is received. Meanwhile, we're working on finalizing the main extension's patch, offering pg_dump support.

See you at FOSDEM!

Tags

PostgreSQL Conferences Extensions FOSDEM 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 01 Feb 2011 13:35:00 +0100 http://tapoueh.org/blog/2011/02/01-going-to-fosdem.html Starting afresh with el-get http://tapoueh.org/blog/2011/01/blog/2011/01/11-starting-afresh-with-el-get.html It so happens that a colleague of mine wanted to start using Emacs but couldn't get to it. He insists on having proper color themes in all applications and some sensible defaults full of nifty add-ons everywhere, and didn't want to have to learn that much about Emacs and Emacs Lisp to get started. I'm not even sure that he will Take the Emacs tour.

You would tell me that there's nothing we can do for so unfriendly users. Well, here's what I did:

;; emacs setup

(add-to-list 'load-path "~/.emacs.d/el-get/el-get")
(require 'el-get)
(setq
 el-get-sources
 '(el-get
   php-mode-improved
   psvn
   auto-complete
   switch-window

   (:name buffer-move
          :after (lambda ()
                   (global-set-key (kbd "<C-S-up>")     'buf-move-up)
                   (global-set-key (kbd "<C-S-down>")   'buf-move-down)
                   (global-set-key (kbd "<C-S-left>")   'buf-move-left)
                   (global-set-key (kbd "<C-S-right>")  'buf-move-right)))

   (:name magit
          :after (lambda ()
                   (global-set-key (kbd "C-x C-z") 'magit-status)))

   (:name goto-last-change
          :after (lambda ()
                   ;; azerty keyboard here, don't use C-x C-/
                   (global-set-key (kbd "C-x C-_") 'goto-last-change)))))

(when window-system
   (add-to-list 'el-get-sources  'color-theme-tango))

(el-get 'sync)

;; visual settings
(setq inhibit-splash-screen t)
(menu-bar-mode -1)
(tool-bar-mode -1)
(scroll-bar-mode -1)

(line-number-mode 1)
(column-number-mode 1)

;; Use the clipboard, pretty please, so that copy/paste "works"
(setq x-select-enable-clipboard t)

(set-frame-font "Monospace-10")

(global-hl-line-mode)

;; suivre les changements exterieurs sur les fichiers
(global-auto-revert-mode 1)

;; pour les couleurs dans M-x shell
(autoload 'ansi-color-for-comint-mode-on "ansi-color" nil t)
(add-hook 'shell-mode-hook 'ansi-color-for-comint-mode-on)

;; S-fleches pour changer de fenêtre
(windmove-default-keybindings)
(setq windmove-wrap-around t)

;; find-file-at-point quand ça a du sens
(setq ffap-machine-p-known 'accept) ; no pinging
(setq ffap-url-regexp nil) ; disable URL features in ffap
(setq ffap-ftp-regexp nil) ; disable FTP features in ffap
(define-key global-map (kbd "C-x C-f") 'find-file-at-point)

(require 'ibuffer)
(global-set-key "\C-x\C-b" 'ibuffer)

;; use iswitchb-mode for C-x b
(iswitchb-mode)

;; I can't remember having meant to use C-z as suspend-frame
(global-set-key (kbd "C-z") 'undo)

;; winner-mode pour revenir sur le layout précédent C-c <left>
(winner-mode 1)

;; dired-x pour C-x C-j
(require 'dired-x)

;; full screen
(defun fullscreen ()
  (interactive)
  (set-frame-parameter nil 'fullscreen
                       (if (frame-parameter nil 'fullscreen) nil 'fullboth)))
(global-set-key [f11] 'fullscreen)

With just this simple 87 lines (all included) of setup, my local user is very happy to switch to using our favorite editor. And he's not even afraid (yet) of his ~/.emacs. I say that's a very good sign of where we are with el-get!

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 11 Jan 2011 16:20:00 +0100 http://tapoueh.org/blog/2011/01/blog/2011/01/11-starting-afresh-with-el-get.html Starting afresh with el-get http://tapoueh.org/blog/2011/01/11-starting-afresh-with-el-get.html Starting afresh with el-get
Tuesday, January 11 2011, 16:20

It so happens that a colleague of mine wanted to start using Emacs but couldn't get to it. He insists on having proper color themes in all applications and some sensible defaults full of nifty add-ons everywhere, and didn't want to have to learn that much about Emacs and Emacs Lisp to get started. I'm not even sure that he will Take the Emacs tour.

You would tell me that there's nothing we can do for so unfriendly users. Well, here's what I did:

;; emacs setup

(add-to-list 'load-path "~/.emacs.d/el-get/el-get")
(require 'el-get)
(setq
 el-get-sources
 '(el-get
   php-mode-improved
   psvn
   auto-complete
   switch-window

   (:name buffer-move
          :after (lambda ()
                   (global-set-key (kbd "<C-S-up>")     'buf-move-up)
                   (global-set-key (kbd "<C-S-down>")   'buf-move-down)
                   (global-set-key (kbd "<C-S-left>")   'buf-move-left)
                   (global-set-key (kbd "<C-S-right>")  'buf-move-right)))

   (:name magit
          :after (lambda ()
                   (global-set-key (kbd "C-x C-z") 'magit-status)))

   (:name goto-last-change
          :after (lambda ()
                   ;; azerty keyboard here, don't use C-x C-/
                   (global-set-key (kbd "C-x C-_") 'goto-last-change)))))

(when window-system
   (add-to-list 'el-get-sources  'color-theme-tango))

(el-get 'sync)

;; visual settings
(setq inhibit-splash-screen t)
(menu-bar-mode -1)
(tool-bar-mode -1)
(scroll-bar-mode -1)

(line-number-mode 1)
(column-number-mode 1)

;; Use the clipboard, pretty please, so that copy/paste "works"
(setq x-select-enable-clipboard t)

(set-frame-font "Monospace-10")

(global-hl-line-mode)

;; suivre les changements exterieurs sur les fichiers
(global-auto-revert-mode 1)

;; pour les couleurs dans M-x shell
(autoload 'ansi-color-for-comint-mode-on "ansi-color" nil t)
(add-hook 'shell-mode-hook 'ansi-color-for-comint-mode-on)

;; S-fleches pour changer de fenêtre
(windmove-default-keybindings)
(setq windmove-wrap-around t)

;; find-file-at-point quand ça a du sens
(setq ffap-machine-p-known 'accept) ; no pinging
(setq ffap-url-regexp nil) ; disable URL features in ffap
(setq ffap-ftp-regexp nil) ; disable FTP features in ffap
(define-key global-map (kbd "C-x C-f") 'find-file-at-point)

(require 'ibuffer)
(global-set-key "\C-x\C-b" 'ibuffer)

;; use iswitchb-mode for C-x b
(iswitchb-mode)

;; I can't remember having meant to use C-z as suspend-frame
(global-set-key (kbd "C-z") 'undo)

;; winner-mode pour revenir sur le layout précédent C-c <left>
(winner-mode 1)

;; dired-x pour C-x C-j
(require 'dired-x)

;; full screen
(defun fullscreen ()
  (interactive)
  (set-frame-parameter nil 'fullscreen
                       (if (frame-parameter nil 'fullscreen) nil 'fullboth)))
(global-set-key [f11] 'fullscreen)

With just this simple 87 lines (all included) of setup, my local user is very happy to switch to using our favorite editor. And he's not even afraid (yet) of his ~/.emacs. I say that's a very good sign of where we are with el-get!

Tags

Emacs el-get switch-window

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 11 Jan 2011 16:20:00 +0100 http://tapoueh.org/blog/2011/01/11-starting-afresh-with-el-get.html el-get 1.1, with 174 recipes http://tapoueh.org/blog/2010/12/blog/2010/12/20-el-get-11-with-174-recipes.html Yes, you read it well, el-get currently features 174 recipes, and is now reaching the 1.1 release. The reason for this release is mainly that I have two big chunks of code to review and the current code has been very stable for awhile. It seems better to do a release with the stable code that exists now before to shake it this much. If you're wondering when to jump in the water and switch to using el-get, now is a pretty good time.

New source types

We now have support for the pacman package management for archlinux, and a way to handle a different package name in the recipe and in the distribution. We also have support for mercurial and subversion and darcs.

Also, apt-get will sometime prompt you to validate its choices, that's the infamous Do you want to continue? prompt. We now handle that smoothly.

(el-get 'sync)

In 1.1, that really means synchronous. That means we install one package after the other, and any error will stop it all. Before that, it was an active wait loop over a parallel install: this option is still available through calling (el-get 'wait).

No more failed to install

Exactly. This error you may have encountered sometime is due to trying to install a package over a previous failed install attempt (network outage, disk full, bad work-in-progress recipe, etc). After awhile in the field it was clear that no case where found where you would regret it if el-get just did removed the previous failed installation for you before to go and install again, as aked. So that's now automatic.

Featuring an overhauled :build facility

The build commands can now either be a list, as before, or some that we evaluate for you. That allows for easier to maintain recipes, and here's an exemple of that:

(:name distel
       :type svn
       :url "http://distel.googlecode.com/svn/trunk/"
       :info "doc"
       :build `,(mapcar
                 (lambda (target)
                   (concat "make " target " EMACS=" el-get-emacs))
                 '("clean" "all"))
       :load-path ("elisp")
       :features distel)

As you see that also allows for maintainance of multi-platform build recipes, and multiple emacs versions too. It's still a little too much on the awkward side of things, though, and that's one of the ongoing work that will happen for next version.

Misc improvements

We are now able to byte-compile your packages, and offer some more hooks (el-get-init-hooks has been asked with a nice usage example). There's a new :localname property that allows to pick where to save the local file when using HTTP method for retrieval, and that in turn allows to fix some recipes.

(:name xcscope
       :type http
       :url "http://cscope.cvs.sourceforge.net/viewvc/cscope/cscope/contrib/xcscope/xcscope.el?revision=1.14&content-type=text%2Fplain"
       :localname "xscope.el"
       :features xcscope)

Oh and you even get :before user function support, even if needing it often shows that you're doing it in a strange way. More often than not it's possible to do all you need to in the :after function, but this tool is there so that you spend less time on having a working environment, not more, right? :)

Switch notice

All in all, if you're already using el-get you should consider switching to 1.1 (by issuing M-x el-get-update of course), and if you're hesitating, just join the fun now!

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 20 Dec 2010 16:45:00 +0100 http://tapoueh.org/blog/2010/12/blog/2010/12/20-el-get-11-with-174-recipes.html el-get 1.1, with 174 recipes http://tapoueh.org/blog/2010/12/20-el-get-11-with-174-recipes.html el-get 1.1, with 174 recipes
Monday, December 20 2010, 16:45

Yes, you read it well, el-get currently features 174 recipes, and is now reaching the 1.1 release. The reason for this release is mainly that I have two big chunks of code to review and the current code has been very stable for awhile. It seems better to do a release with the stable code that exists now before to shake it this much. If you're wondering when to jump in the water and switch to using el-get, now is a pretty good time.

New source types

We now have support for the pacman package management for archlinux, and a way to handle a different package name in the recipe and in the distribution. We also have support for mercurial and subversion and darcs.

Also, apt-get will sometime prompt you to validate its choices, that's the infamous Do you want to continue? prompt. We now handle that smoothly.

(el-get 'sync)

In 1.1, that really means synchronous. That means we install one package after the other, and any error will stop it all. Before that, it was an active wait loop over a parallel install: this option is still available through calling (el-get 'wait).

No more failed to install

Exactly. This error you may have encountered sometime is due to trying to install a package over a previous failed install attempt (network outage, disk full, bad work-in-progress recipe, etc). After awhile in the field it was clear that no case where found where you would regret it if el-get just did removed the previous failed installation for you before to go and install again, as aked. So that's now automatic.

Featuring an overhauled :build facility

The build commands can now either be a list, as before, or some that we evaluate for you. That allows for easier to maintain recipes, and here's an exemple of that:

(:name distel
       :type svn
       :url "http://distel.googlecode.com/svn/trunk/"
       :info "doc"
       :build `,(mapcar
                 (lambda (target)
                   (concat "make " target " EMACS=" el-get-emacs))
                 '("clean" "all"))
       :load-path ("elisp")
       :features distel)

As you see that also allows for maintainance of multi-platform build recipes, and multiple emacs versions too. It's still a little too much on the awkward side of things, though, and that's one of the ongoing work that will happen for next version.

Misc improvements

We are now able to byte-compile your packages, and offer some more hooks (el-get-init-hooks has been asked with a nice usage example). There's a new :localname property that allows to pick where to save the local file when using HTTP method for retrieval, and that in turn allows to fix some recipes.

(:name xcscope
       :type http
       :url "http://cscope.cvs.sourceforge.net/viewvc/cscope/cscope/contrib/xcscope/xcscope.el?revision=1.14&content-type=text%2Fplain"
       :localname "xscope.el"
       :features xcscope)

Oh and you even get :before user function support, even if needing it often shows that you're doing it in a strange way. More often than not it's possible to do all you need to in the :after function, but this tool is there so that you spend less time on having a working environment, not more, right? :)

Switch notice

All in all, if you're already using el-get you should consider switching to 1.1 (by issuing M-x el-get-update of course), and if you're hesitating, just join the fun now!

Tags

Emacs debian el-get release

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 20 Dec 2010 16:45:00 +0100 http://tapoueh.org/blog/2010/12/20-el-get-11-with-174-recipes.html Dynamic Triggers in PLpgSQL http://tapoueh.org/blog/2010/11/24-dynamic-triggers-in-plpgsql.html Dynamic Triggers in PLpgSQL
Wednesday, November 24 2010, 16:45

You certainly know that implementing dynamic triggers in PLpgSQL is impossible. But I had a very bad night, being up from as soon as 3:30 am today, so that when a developer asked me about reusing the same trigger function code from more than one table and for a dynamic column name, I didn't remember about it being impossible.

Here's what happens in such cases, after a long time on the problem (yes, overall, that's a slow day). Note that I'm abusing the (record_literal).* notation a lot in there, and even the (record_literal).column_name too.

CREATE OR REPLACE FUNCTION public.update_timestamp()
 RETURNS TRIGGER
 LANGUAGE plpgsql
AS $f$
DECLARE
    ts_column varchar;
    old_timestamp timestamptz;
    attname name;
    n text;
    v text;
BEGIN
    IF TG_NARGS != 1
    THEN
        RAISE EXCEPTION 'Trigger public.update_timestamp() called with % args',
                         TG_NARGS;
    END IF;

    ts_column := TG_ARGV[0];

    EXECUTE 'SELECT n.' || ts_column
         || ' FROM (SELECT ('
         || quote_literal(OLD) || '::' || TG_RELID::regclass
         || ').*) as n'
       INTO old_timestamp;

    -- build NEW record text
    n := '(';
    FOR attname IN
      EXECUTE 'SELECT attname '
           || '  FROM pg_class c left join pg_attribute a on a.attrelid = c.oid'
           || ' WHERE c.oid = $1 and attnum > 0 order by attnum'
       USING TG_RELID
    LOOP
        EXECUTE 'SELECT (' || quote_literal(NEW) || '::' || TG_RELID::regclass || ').' || attname INTO v;

        IF n != '(' THEN n := n || ','; END IF;

        IF attname = ts_column
           AND v::timestamptz IS NOT DISTINCT FROM old_timestamp
        THEN
                n := n || now();
        ELSE
                n := n || COALESCE(v, '');
        END IF;
    END LOOP;
    n := n || ')';

    EXECUTE 'SELECT ($1::' || TG_RELID::regclass || ').*'
      INTO NEW
     USING n;

    RETURN NEW;
END;
$f$;

It's not pretty, and not fast. It's about 2 ms per call on a table with 15 columns, in some preliminary tests. But it sure was a nice challenge!

Tags

plpgsql

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 24 Nov 2010 16:45:00 +0100 http://tapoueh.org/blog/2010/11/24-dynamic-triggers-in-plpgsql.html pg_basebackup http://tapoueh.org/blog/2010/11/07-pg_basebackup.html pg_basebackup
Sunday, November 07 2010, 13:45

Hannu just gave me a good idea in this email on -hackers, proposing that pg_basebackup should get the xlog files again and again in a loop for the whole duration of the base backup. That's now done in the aforementioned tool, whose options got a little more useful now:

Usage: pg_basebackup.py [-v] [-f] [-j jobs] "dsn" dest

Options:
  -h, --help            show this help message and exit
  --version             show version and quit
  -x, --pg_xlog         backup the pg_xlog files
  -v, --verbose         be verbose and about processing progress
  -d, --debug           show debug information, including SQL queries
  -f, --force           remove destination directory if it exists
  -j JOBS, --jobs=JOBS  how many helper jobs to launch
  -D DELAY, --delay=DELAY
                        pg_xlog subprocess loop delay, see -x
  -S, --slave           auxilliary process
  --stdin               get list of files to backup from stdin

Yeah, as implementing the xlog idea required having some kind of parallelism, I built on it and the script now has a --jobs option for you to setup how many processes to launch in parallel, all fetching some base backup files in its own standard (libpq) PostgreSQL connection, in compressed chunks of 8 MB (so that's not 8 MB chunks sent over).

The xlog loop will fetch any WAL file whose ctime changed again, wholesale. It's easier this way, and tools to get optimized behavior already do exist, either walmgr or walreceiver.

The script is still a little python self-contained short file, it just went from about 100 lines of code to about 400 lines. There's no external dependency, all it needs is provided by a standard python installation. The problem with that is that it's using select.poll() that I think is not available on windows. Supporting every system or adding to the dependencies, I've been choosing what's easier for me.

    import select
    p = select.poll()
    p.register(sys.stdin, select.POLLIN)

If you get to try it, please report about it, you should know or easily discover my email!

Tags

PostgreSQL skytools backup

]]> dim@tapoueh.org (Dimitri Fontaine) Sun, 07 Nov 2010 13:45:00 +0100 http://tapoueh.org/blog/2010/11/07-pg_basebackup.html Introducing Extensions http://tapoueh.org/blog/2010/10/21-introducing-extensions.html Introducing Extensions
Thursday, October 21 2010, 13:45

After reading Simon's blog post, I can't help but try to give some details about what it is exactly that I'm working on. As he said, there are several aspects to extensions in PostgreSQL, it all begins here: Chapter 35. Extending SQL.

It's possible, and mostly simple enough, to add your own code or behavior to PostgreSQL, so that it will use your code and your semantics while solving user queries. That's highly useful and it's easy to understand how so when you look at some projects like PostGIS, ip4r (index searches of ip in a range, not limited to CIDR notation), or our own Key Value Store, hstore.

So, what's in an Extension?

An extension in its simple form is a SQL script that you load on your database, but manage separately. Meaning you don't want the script to be part of your backups. Often, that kind of script will create new datatypes and operators, support functions, user functions and index support, and then it would include some C code that ships in a shared library object.

As far as PostgreSQL is concerned, at least in the current version of my patch, the extension is first a meta information file that allows to register it. We currently call that the control file. Then, it's an SQL script that is executed by the server when you create the extension.

If it so happens that the SQL script depends on some shared library objects file, this has to be present at the right place (MODULE_PATHNAME) for the extension to be successfully created, but that's always been the case.

The problem with current releases of PostgreSQL, that the extension patch is solving, is the pg_dump and pg_restore support. We said it, you don't want the SQL script to be part of your dump, because it's not maintained in your database, but in some code repository out there. What you want is to be able to install the extension again at the file system level then pg_restore your database — that depends on it being there.

And that's exactly what the extension patch provides. By now having a SQL object called an extension, and maintained in the new pg_extension catalog, we have an Oid to refer to. Which we do by recording a dependency between any object created by the script and the extension Oid, so that pg_dump can be instructed to skip those.

Examples?

So, let's have a look at what you can do if you play with a patched development server version, or if you play directly from the git repository at http://git.postgresql.org/gitweb?p=postgresql-extension.git;a=shortlog;h=refs/heads/extension

dim ~ createdb exts
dim ~ psql exts
psql (9.1devel)
Type "help" for help.

dim=# \dx+
                                                        List of extensions
        Name        | | |                               Description
--------------------+-+-+-------------------------------------------------------------------------
 adminpack          | | | Administrative functions for PostgreSQL
 auto_username      | | | functions for tracking who changed a table
 autoinc            | | | functions for autoincrementing fields
 btree_gin          | | | GIN support for common types BTree operators
 btree_gist         | | | GiST support for common types BTree operators
 chkpass            | | | Store crypt()ed passwords
 citext             | | | case-insensitive character string type
 cube               | | | data type for representing multidimensional cubes
 dblink             | | | connect to other PostgreSQL databases from within a database
 dict_int           | | | example of an add-on dictionary template for full-text search
 dict_xsyn          | | | example of an add-on dictionary template for full-text search
 earthdistance      | | | calculating great circle distances on the surface of the Earth
 fuzzystrmatch      | | | determine similarities and distance between strings
 hstore             | | | storing sets of key/value pairs
 int_aggregate      | | | integer aggregator and an enumerator (obsolete)
 intarray           | | | one-dimensional arrays of integers: functions, operators, index support
 isn                | | | data types for the international product numbering standards
 lo                 | | | managing Large Objects
 ltree              | | | data type for hierarchical tree-like structure
 moddatetime        | | | functions for tracking last modification time
 pageinspect        | | | inspect the contents of database pages at a low level
 pg_buffercache     | | | examine the shared buffer cache in real time
 pg_freespacemap    | | | examine the free space map (FSM)
 pg_stat_statements | | | tracking execution statistics of all SQL statements executed
 pg_trgm            | | | determine the similarity of text, with indexing support
 pgcrypto           | | | cryptographic functions
 pgrowlocks         | | | show row locking information for a specified table
 pgstattuple        | | | obtain tuple-level statistics
 prefix             | | | Prefix Match Indexing
 refint             | | | functions for implementing referential integrity
 seg                | | | data type for representing line segments, or floating point intervals
 tablefunc          | | | various functions that return tables, including crosstab(text sql)
 test_parser        | | | example of a custom parser for full-text search
 timetravel         | | | functions for implementing time travel
 tsearch2           | | | backwards-compatible text search functionality (pre-8.3)
 unaccent           | | | text search dictionary that removes accents
(36 rows)

Ok I've edited the output in a visible way, to leave the Version and Custom Variable Classes column out. It's taking lots of screen place and it's not that useful here. Maybe the classes one will even get dropped out of the patch before reaching 9.1, we'll see.

Let's pick an extension there and install it in our new database:

exts=# create extension pg_trgm;
NOTICE:  Installing extension 'pg_trgm' from '/Users/dim/pgsql/exts/share/contrib/pg_trgm.sql', with user data
CREATE EXTENSION
exts=# \dx
                                           List of extensions
  Name   |  |  |                       Description
---------+--+--+---------------------------------------------------------
 pg_trgm |  |  | determine the similarity of text, with indexing support
(1 row)

See, that was easy enough. Same thing, the extra columns have been removed. So, what's in this extension, will you ask me, what are those objects that you would normally (that is, before the patch) find in your pg_dump backup script?

exts=# select * from pg_extension_objects('pg_trgm');
    class     | classid | objid |                                                                objdesc
--------------+---------+-------+----------------------------------------------------------------------------------------------------------------------------------------
 pg_extension |    3996 | 18498 | extension pg_trgm
 pg_proc      |    1255 | 18499 | function set_limit(real)
 pg_proc      |    1255 | 18500 | function show_limit()
 pg_proc      |    1255 | 18501 | function show_trgm(text)
 pg_proc      |    1255 | 18502 | function similarity(text,text)
 pg_proc      |    1255 | 18503 | function similarity_op(text,text)
 pg_operator  |    2617 | 18504 | operator %(text,text)
 pg_type      |    1247 | 18505 | type gtrgm
 pg_proc      |    1255 | 18506 | function gtrgm_in(cstring)
 pg_proc      |    1255 | 18507 | function gtrgm_out(gtrgm)
 pg_type      |    1247 | 18508 | type gtrgm[]
 pg_proc      |    1255 | 18509 | function gtrgm_consistent(internal,text,integer,oid,internal)
 pg_proc      |    1255 | 18510 | function gtrgm_compress(internal)
 pg_proc      |    1255 | 18511 | function gtrgm_decompress(internal)
 pg_proc      |    1255 | 18512 | function gtrgm_penalty(internal,internal,internal)
 pg_proc      |    1255 | 18513 | function gtrgm_picksplit(internal,internal)
 pg_proc      |    1255 | 18514 | function gtrgm_union(bytea,internal)
 pg_proc      |    1255 | 18515 | function gtrgm_same(gtrgm,gtrgm,internal)
 pg_opfamily  |    2753 | 18516 | operator family gist_trgm_ops for access method gist
 pg_opclass   |    2616 | 18517 | operator class gist_trgm_ops for access method gist
 pg_amop      |    2602 | 18518 | operator 1 %(text,text) of operator family gist_trgm_ops for access method gist
 pg_amproc    |    2603 | 18519 | function 1 gtrgm_consistent(internal,text,integer,oid,internal) of operator family gist_trgm_ops for access method gist
 pg_amproc    |    2603 | 18520 | function 2 gtrgm_union(bytea,internal) of operator family gist_trgm_ops for access method gist
 pg_amproc    |    2603 | 18521 | function 3 gtrgm_compress(internal) of operator family gist_trgm_ops for access method gist
 pg_amproc    |    2603 | 18522 | function 4 gtrgm_decompress(internal) of operator family gist_trgm_ops for access method gist
 pg_amproc    |    2603 | 18523 | function 5 gtrgm_penalty(internal,internal,internal) of operator family gist_trgm_ops for access method gist
 pg_amproc    |    2603 | 18524 | function 6 gtrgm_picksplit(internal,internal) of operator family gist_trgm_ops for access method gist
 pg_amproc    |    2603 | 18525 | function 7 gtrgm_same(gtrgm,gtrgm,internal) of operator family gist_trgm_ops for access method gist
 pg_proc      |    1255 | 18526 | function gin_extract_trgm(text,internal)
 pg_proc      |    1255 | 18527 | function gin_extract_trgm(text,internal,smallint,internal,internal)
 pg_proc      |    1255 | 18528 | function gin_trgm_consistent(internal,smallint,text,integer,internal,internal)
 pg_opfamily  |    2753 | 18529 | operator family gin_trgm_ops for access method gin
 pg_opclass   |    2616 | 18530 | operator class gin_trgm_ops for access method gin
 pg_amop      |    2602 | 18531 | operator 1 %(text,text) of operator family gin_trgm_ops for access method gin
 pg_amproc    |    2603 | 18532 | function 1 btint4cmp(integer,integer) of operator family gin_trgm_ops for access method gin
 pg_amproc    |    2603 | 18533 | function 2 gin_extract_trgm(text,internal) of operator family gin_trgm_ops for access method gin
 pg_amproc    |    2603 | 18534 | function 3 gin_extract_trgm(text,internal,smallint,internal,internal) of operator family gin_trgm_ops for access method gin
 pg_amproc    |    2603 | 18535 | function 4 gin_trgm_consistent(internal,smallint,text,integer,internal,internal) of operator family gin_trgm_ops for access method gin
(38 rows)

This function main intended users are the extension authors themselves, so that it's easy for them to figure out which system identifier (the objid column) has been attributed to some SQL objects from their install script. With this knowledge, you can prepare some upgrade scripts. But that's for another patch altogether, so we'll get back to the matter in another blog entry.

So we chose trgm as an example, let's follow the documentation and create a test table and a custom index in there, just so that the extension is put to good use. Then let's try to DROP our extension, because we're testing the infrastructure, right?

exts=# create table test(id bigint, name text);
CREATE TABLE
exts=# CREATE INDEX idx_test_name ON test USING gist (name gist_trgm_ops);
CREATE INDEX
exts=# drop extension pg_trgm;
ERROR:  cannot drop extension pg_trgm because other objects depend on it
DETAIL:  index idx_test_name depends on operator class gist_trgm_ops for access method gist
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

Of course PostgreSQL is smart enough here — the extension patch had nothing special to do to achieve that, apart from recording the dependencies. Next, as we didn't drop extension pg_trgm cascade;, it's still in the database. So let's see what a pg_dump will look like. As it's quite a lot of text to paste, let's see the pg_restore catalog instead. And that's a feature that needs to be known some more, too.

dim ~ pg_dump -Fc exts | pg_restore -l | grep -v '^;'
1812; 1262 18497 DATABASE - exts dim
1; 3996 18498 EXTENSION - pg_trgm
1813; 0 0 COMMENT - EXTENSION pg_trgm
6; 2615 2200 SCHEMA - public dim
1814; 0 0 COMMENT - SCHEMA public dim
1815; 0 0 ACL - public dim
320; 2612 11602 PROCEDURAL LANGUAGE - plpgsql dim
1521; 1259 18543 TABLE public test dim
1809; 0 18543 TABLE DATA public test dim
1808; 1259 18549 INDEX public idx_test_name dim

As you see, the only SQL object that got into the backup are an EXTENSION and its COMMENT. Nothing like the types or the functions that the pg_trgm script creates.

What does it means to extension authors?

In order to be an extension, you have to prepare a control file where to give the necessary information to register your script. This file must be named extension.control if the script is named extension.sql, at least at the moment. This file can benefit from some variable expansion too, like does the current extension.sql.in, in that if you provide an extension.control.in file the term VERSION will be expanded to whatever $(VERSION) is set to in your Makefile.

If you never wrote a C coded extension for PostgreSQL, this might look complex and irrelevant. Baseline is that you need a Makefile so that you can benefit easily from the PostgreSQL infrastructure work and have the make install operation place your files at the right place, including the new control file.

That's it for today, folks

A next blog entry will detail what happens with extensions providing user data, and the CREATE EXTENSION name WITH NO DATA; variant. Stay tuned!

Tags

PostgreSQL Extensions release ip4r plpgsql backup restore prefix 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 21 Oct 2010 13:45:00 +0200 http://tapoueh.org/blog/2010/10/21-introducing-extensions.html Extensions: writing a patch for PostgreSQL http://tapoueh.org/blog/2010/10/15-extensions-writing-a-patch-for-postgresql.html Extensions: writing a patch for PostgreSQL
Friday, October 15 2010, 11:30

 

These days, thanks to my community oriented job, I'm working full time on a PostgreSQL patch to terminate basic support for extending SQL. First thing I want to share is that patching the backend code is not as hard as one would think. Second one is that git really is helping.

“Not as hard as one would think, are you kidding me?”, I hear some say. Well, that's true. It's C code in there, but with a very good layer of abstractions so that you're not dealing with subtle problems that much. Of course it happens that you have to, and managing the memory isn't an option. That said, palloc() and the memory contexts implementation makes that as easy as in lots of cases, you don't have to think about it.

PostgreSQL is very well known for its reliability, and that's not something that just happened. All the source code is organized in a way that makes it possible, so your main task is to write code that looks as much as possible like the existing surrounding code. And we all know how to copy paste, right?

So, my current work on the extensions is to make it so that if you install hstore in your database (to pick an example), your backup won't contain any hstore specific objects (types, functions, operators, index support objects, etc) but rather a single line that tells PostgreSQL to install hstore again.

CREATE EXTENSION hstore;

The feature already works in my git branch and I'm extracting infrastructure work in there to ease review. That's when git helps a lot. What I've done is create a new branch from the master one, then cherry pick the patches of interest. Well sometime you have to resort to helper tools. I've been told after the fact that using git cherry-pick -n would have allowed the following to be much simpler:

dim ~/dev/PostgreSQL/postgresql-extension git cherry-pick 3f291b4f82598309368610431cf2a18d7b7a7950
error: could not apply 3f291b4... Implement dependency tracking for CREATE EXTENSION, and DROP EXTENSION ... CASCADE.
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit -c 3f291b4'
dim ~/dev/PostgreSQL/postgresql-extension git status \
| awk '/modified/ && ! /both/ && ! /genfile/ {print $3}
       /deleted/ {print $5}
       /both/    {print $4}' \
| xargs echo git reset -- \
| sh
Unstaged changes after reset:
M       src/backend/catalog/dependency.c
M       src/backend/catalog/heap.c
M       src/backend/catalog/pg_aggregate.c
M       src/backend/catalog/pg_conversion.c
M       src/backend/catalog/pg_namespace.c
M       src/backend/catalog/pg_operator.c
M       src/backend/catalog/pg_proc.c
M       src/backend/catalog/pg_type.c
M       src/backend/commands/extension.c
M       src/backend/commands/foreigncmds.c
M       src/backend/commands/opclasscmds.c
M       src/backend/commands/proclang.c
M       src/backend/commands/tsearchcmds.c
M       src/backend/nodes/copyfuncs.c
M       src/backend/nodes/equalfuncs.c
M       src/backend/parser/gram.y
M       src/include/catalog/dependency.h
M       src/include/commands/extension.h
M       src/include/nodes/parsenodes.h

That's what I did to prepare a side branch containing only changes to a part of my current work. I had to filter the diff so much only because I'm commiting in rather big steps, rather than very little chunks at a time. In this case that means I had a single patch with several units of changes and I wanted to extract only one. Well, it happens that even in such a case, git is helping!

There's more to say about the extension related feature of course, but that'll do it for this article. I'd just end up with the following nice diffstat of 4 days of work:

dim ~/dev/PostgreSQL/postgresql-extension git --no-pager diff master..|wc -l
    3897
dim ~/dev/PostgreSQL/postgresql-extension git --no-pager diff master..|diffstat
 doc/src/sgml/extend.sgml               |   46 ++
 doc/src/sgml/ref/allfiles.sgml         |    2
 doc/src/sgml/ref/create_extension.sgml |   95 ++++
 doc/src/sgml/ref/drop_extension.sgml   |  115 +++++
 doc/src/sgml/reference.sgml            |    2
 src/backend/access/transam/xlog.c      |   95 ----
 src/backend/catalog/Makefile           |    1
 src/backend/catalog/dependency.c       |   25 +
 src/backend/catalog/heap.c             |    9
 src/backend/catalog/objectaddress.c    |   14
 src/backend/catalog/pg_aggregate.c     |    7
 src/backend/catalog/pg_conversion.c    |    7
 src/backend/catalog/pg_namespace.c     |   13
 src/backend/catalog/pg_operator.c      |    7
 src/backend/catalog/pg_proc.c          |    7
 src/backend/catalog/pg_type.c          |    8
 src/backend/commands/Makefile          |    3
 src/backend/commands/comment.c         |    6
 src/backend/commands/extension.c       |  688 +++++++++++++++++++++++++++++++++
 src/backend/commands/foreigncmds.c     |   19
 src/backend/commands/functioncmds.c    |    7
 src/backend/commands/opclasscmds.c     |   13
 src/backend/commands/proclang.c        |    7
 src/backend/commands/tsearchcmds.c     |   25 +
 src/backend/nodes/copyfuncs.c          |   22 +
 src/backend/nodes/equalfuncs.c         |   18
 src/backend/parser/gram.y              |   51 ++
 src/backend/tcop/utility.c             |   27 +
 src/backend/utils/adt/genfile.c        |  193 +++++++++
 src/backend/utils/init/postinit.c      |    3
 src/backend/utils/misc/Makefile        |    2
 src/backend/utils/misc/cfparser.c      |  113 +++++
 src/backend/utils/misc/guc-file.l      |   26 -
 src/backend/utils/misc/guc.c           |  160 ++++++-
 src/bin/pg_dump/common.c               |    6
 src/bin/pg_dump/pg_dump.c              |  520 ++++++++++++++++++++++--
 src/bin/pg_dump/pg_dump.h              |   10
 src/bin/pg_dump/pg_dump_sort.c         |    7
 src/bin/psql/command.c                 |    3
 src/bin/psql/describe.c                |   45 ++
 src/bin/psql/describe.h                |    3
 src/bin/psql/help.c                    |    1
 src/include/catalog/dependency.h       |    1
 src/include/catalog/indexing.h         |    6
 src/include/catalog/pg_extension.h     |   61 ++
 src/include/catalog/pg_proc.h          |   13
 src/include/catalog/toasting.h         |    1
 src/include/commands/extension.h       |   54 ++
 src/include/nodes/nodes.h              |    2
 src/include/nodes/parsenodes.h         |   20
 src/include/parser/kwlist.h            |    1
 src/include/utils/builtins.h           |    4
 src/include/utils/cfparser.h           |   18
 src/include/utils/guc.h                |   11
 src/makefiles/pgxs.mk                  |   21 -
 55 files changed, 2456 insertions(+), 188 deletions(-)

Tags

PostgreSQL Extensions backup

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 15 Oct 2010 11:30:00 +0200 http://tapoueh.org/blog/2010/10/15-extensions-writing-a-patch-for-postgresql.html Date puzzle for starters http://tapoueh.org/blog/2010/10/08-date-puzzle-for-starters.html Date puzzle for starters
Friday, October 08 2010, 10:00

The PostgreSQL IRC channel is a good place to be, for all the very good help you can get there, because people are always wanting to remain helpful, because of the off-topics discussions sometime, or to get to talk with community core members. And to start up your day too.

This morning's question started simple : “how can I check if today is the "first sunday fo the month". or "the second tuesday of the month" etc?”

And the first version of the answer, quite simple it is too:

dim=#   with begin(d) as (select date_trunc('month', 'today'::date)::date)
dim-# select d + 7 - extract(dow from d)::int as sunday from begin;
   sunday
------------
 2010-10-03
(1 row)

So you just have to compare the result of the function with 'today'::date and there you go. The problem is that the question could be read in the other way round, like, what is today in first or second day name of this month format? Once more, RhodiumToad to the rescue:

select to_char(current_date,
               '"' || ((ARRAY['First','Second','Third','Fourth','Fifth'])
                             [(extract(day from current_date)::integer - 1)/7 + 1]
                      )
                   || '" Day');
     to_char
------------------
 Second Friday
(1 row)

That's a straight answer to the question, read that way!

But the part that I found nice to play with was my first reading of the question, as I don't get to lose my ideas that easily, you see… so what about writing a function to return the date of any nth occurrence of a given day of week in a given month, defaulting to this very month?

create or replace function get_nth_dow_of_month
 (
  nth int,
  dow int,
  begin date default current_date
 )
 returns date
 language sql
 strict
 as
$$
with month(d) as (
  select generate_series(date_trunc('month', $3),
                         date_trunc('month', $3) + interval '1 month - 1 day',
                         interval '1 day')::date
),
     repeat as (
  select d, extract(dow from d) as dow, (d - date_trunc('month', $3)::date) / 7 as repeat
    from month
)
select d
  from repeat
 where dow = $2 and repeat = $1;
$$;

dim=# select get_nth_dow_of_month(0, 0);
 get_nth_dow_of_month
----------------------
 2010-10-03
(1 row)

dim=# select get_nth_dow_of_month(1, 4, '2010-09-12');
 get_nth_dow_of_month
----------------------
 2010-09-09
(1 row)

So you see we just got the first Sunday of this month (0, 0) and the second Thursday (1, 4) of the previous one. Any date within a month is a good way to tell which month you want to work in, as the function's written, abusing date_trunc like it does.

Now the way the function is written is unfinished. You want to fix it in one of two ways. Either stop using generate_series to only output one row at a time, or fix the API so that you can ask for more than a nth dow at a time. Of course, that was a starter for me, not a problem I need to solve directly, and that was a good excuse for a blog entry, so I won't fix it. That's left as an exercise to our interested readers!

Tags

PostgreSQL 9.1

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 08 Oct 2010 10:00:00 +0200 http://tapoueh.org/blog/2010/10/08-date-puzzle-for-starters.html Resuming work on Extensions, first little step http://tapoueh.org/blog/2010/10/07-resuming-work-on-extensions-first-little-step.html Resuming work on Extensions, first little step
Thursday, October 07 2010, 17:15

 

Yeah I'm back on working on my part of the extension thing in PostgreSQL.

First step is a little one, but as it has public consequences, I figured I'd talk about it already. I've just refreshed my git repository to follow the new master one, and you can see that here http://git.postgresql.org/gitweb?p=postgresql-extension.git;a=commitdiff;h=9a88e9de246218e93c04b6b97e1ef61d97925430.

It's been easier than I feared, mainly:

$ git --no-pager diff master..extension
$ git --no-pager format-patch master..extension
$ cp 0001-First-stab-at-writing-pg_execute_from_file-function.patch ..
$ git checkout master
$ git pull -f pgmaster
$ git reset --hard pgmaster/master
$ git checkout extension
$ git reset --hard master
$ git am -s ../0001-First-stab-at-writing-pg_execute_from_file-function.edit.patch
$ git status
$ git log --short | head
$ git log -n2 --oneline
$ git push -f

So that's still more steps that one want to call dead simple, but still. The format-patch command is to save my work away (all patches that are in the extension branch but not in the master — well that was only one of them here). Then, as the master repository URL didn't change, I can simply pull the changes in. Of course I had a nice message warning: no common commits.

Once pulled, I trashed my local copy and replaced it with the new official one, that's git reset --hard pgmaster/master, then in the extension branch I could trash it and have it linked to the local master again.

Of course, the git am method wouldn't apply my patch as-is, there was some underlying changes in the source files, the identification tag changed from $PostgreSQL$ to, e.g., src/backend/utils/adt/genfile.c, and I had to cope with that. Maybe there's some tool (git am -3 ?) to do it automatically, I just copy edited the .patch file.

Lastly, it's all about checking the result and publishing the result. This last line is git push -f and is when I just trashed and replaced my postgresql-extension community repository. I don't think anybody was following it, but should it be the case, you will have to reinit your copy.

More blog posts to come about extensions, as I arranged to have some real time to devote on the topic. At least I was able to arrange things so that I can work on the subject for real, and the first thing I did, the very night before it was meant to begin, is catch a tonsillitis. Lost about a week, not the project! Stay tuned!

Tags

PostgreSQL Extensions

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 07 Oct 2010 17:15:00 +0200 http://tapoueh.org/blog/2010/10/07-resuming-work-on-extensions-first-little-step.html el-get reaches 1.0 http://tapoueh.org/blog/2010/10/blog/2010/10/07-el-get-reaches-10.html It's been a week since the last commits in the el-get repository, and those were all about fixing and adding recipes, and about notifications. Nothing like core plumbing you see. Also, 0.9 was released on 2010-08-24 and felt pretty complete already, then received lots of improvements. It's high time to cross the line and call it 1.0!

Now existing users will certainly just be moderatly happy to see the tool reach that version number, depending whether they think more about the bugs they want to see fixed (ftp is supported, only called http) and the new features they want to see in (info documentation) or more about what el-get does for them already today...

For the new users, or the yet-to-be-convinced users, let's take some time and talk about el-get. A FAQ like session might be best.

How is el-get different from ELPA?

ELPA is the Emacs Lisp Package Archive and is also known as package.el, to be included in Emacs 24. This allows emacs list extension authors to package their work. That means they have to follow some guidelines and format their contribution, then propose it for upload.

This requires licence checks (good) and for the new official ELPA mirror it even requires dead-tree papers exchange and contracts and copyright assignments, I believe.

Why have both?

While ELPA is a great thing to have, it's so easy to find some high quality Emacs extension out there that are not part of the offer. Either authors are not interrested into uploading to ELPA, or they don't know how to properly package for it (it's only simple for single file extensions, see).

So el-get is a pragmatic answer here. It's there because it so happens that I don't depend only on emacs extensions that are available with Emacs itself, in my distribution site-lisp and in ELPA. I need some more, and I don't need it to be complex to find it, fetch it, init it and use it.

Of course I could try and package any extension I find I need and submit it to ELPA, but really, to do that nicely I'd need to contact the extension author (upstream) for him to accept my patch, and then consider a fork.

With el-get I propose distributed packaging if you will. Let's have a look at two recipes here. First, the el-get one itself:

(:name el-get
       :type git
       :url "git://github.com/dimitri/el-get.git"
       :features el-get
       :compile "el-get.el")

Then a much more complex one, the bbdb one:

(:name bbdb
       :type git
       :url "git://github.com/barak/BBDB.git"
       :load-path ("./lisp" "./bits")
       :build ("./configure" "make autoloads" "make")
       :build/darwin ("./configure --with-emacs=/Applications/Emacs.app/Contents/MacOS/Emacs" "make autoloads" "make")
       :features bbdb
       :after (lambda () (bbdb-initialize))
       :info "texinfo")

The idea is that it's much simpler to just come up with a recipe like this than to patch existing code and upload it to ELPA. And anybody can share their recipes very easily, with or without proposing them to me, even if I very much like to add some more in the official el-get list.

As a user, you don't even need to twiddle with recipes, mostly, because we already have them for you. What you do instead is list them in el-get-sources.

So, show me how you use it?

Yeah, sure. Here's a sample of my dim-packages.el file, part of my .emacs suite. Yeah a single .emacs does not suit me anymore, it's a complete .emacs.d now, but that's because that's how I like it organised, you know. So, here's the example:

;;; dim-packages.el --- Dimitri Fontaine
;;
;; Set el-get-sources and call el-get to init all those packages we need.
(require 'el-get)
(add-to-list 'el-get-recipe-path "~/dev/emacs/el-get/recipes")

(setq el-get-sources
      '(cssh el-get switch-window vkill google-maps yasnippet verbiste mailq sicp

        (:name magit
               :after (lambda () (global-set-key (kbd "C-x C-z") 'magit-status)))

        (:name asciidoc
               :type elpa
               :after (lambda ()
                        (autoload 'doc-mode "doc-mode" nil t)
                        (add-to-list 'auto-mode-alist '("\\.adoc$" . doc-mode))
                        (add-hook 'doc-mode-hook '(lambda ()
                                                    (turn-on-auto-fill)
                                                    (require 'asciidoc)))))

        (:name goto-last-change
               :after (lambda ()
                        (global-set-key (kbd "C-x C-/") 'goto-last-change)))

        (:name auto-dictionary :type elpa)
        (:name gist            :type elpa)
        (:name lisppaste       :type elpa)))

(el-get) ; that could/should be (el-get 'sync)
(provide 'dim-packages)

Ok that's not all of it, but it should give you a nice idea about what problem I solve with el-get and how. In my emacs startup sequence, somewhere inside my ~/.emacs.d/init.el file, I have a line that says (require 'dim-packages). This will set el-get-sources to the list just above, then call (el-get), the main function.

This main function will check each given package and install it if necessary (including build the package, as in make autoloads; make), then init it. What init means exactly depends on what the recipe says. That can include byte-compiling some files, caring about load-path, load and require commands, caring about Info-directory-list and ginstall-info too, and some more.

So in short, it will make it so that your emacs instance is ready for you to use. And you get the choice to use the given el-get recipes as-is, like I did for cssh, el-get, switch-window and others, up to sicp, or to tweak them partly, like in the magit example where I've added a user init function (the :after property) to bind magit-status to C-x C-z here. You can even embed a full recipe inline in the el-get-sources variable, that's the case for each item that gives its :type property, like asciidoc or gist.

And, as you see, we're using ELPA a lot in this sources, so el-get isn't striving to replace it at all, it's just trying to accomodate to a broader world.

I read that the el-get-install is asynchronous, tell me more.

Yeah, right, the example above says (el-get) at its end, and in the cases when el-get has to install or build sources, this will be done asynchronously. Which means that not only several sources will get processed at once (using your multi cores, yeah) but that it will let emacs start up as if it was ready.

It happens that's usually what I want, because I seldom add sources in my setup, but in theory that can break your emacs. What I do is start it again or fix by hand, what you can do instead is (el-get 'sync) so that emacs is blocked waiting for el-get to properly install and initialize all the sources you've setup. Your choice, just add the 'sync parameter there.

Now, explain me why it is better this way, again, please?

Well, before I wrote el-get, trying out a new extension, setting it up etc was something quite involved, and that I had to redo on several machines. The only way not to redo it was to include the extension's code into my own git repository (my emacs.d is in git, of course).

And putting code I don't maintain into my own git repository is something I frown upon. I have no business pretending I'll maintain the code, and I know I will never think to check the URL where I've found it for updates. That's when I though noting down the URL somewhere.

Also, what about sharing the extension with friends. Uneasy, at best.

Enters el-get and I can just add an entry to el-get-sources, based on a file somewhere in my own el-get-recipe-path. When I'm happy with this file, I can contribute it to el-get proper or just send it over to any interested recipient. Adding it to your sources is easy. Copy the file in your el-get-recipe-path somewhere, add its name to your el-get-sources, then M-x el-get-install it. Done. If you were given the :after function, it's all setup already.

If you contribute the recipe to el-get, then M-x el-get-update RET el-get RET and you get it on this other machine where you also use Emacs. Or you can tell your friend to do the same and benefit from your packaging.

Well, sounds good. What recipes do you have already?

I count 67 of them already. One of them is just a book in info format, with no elisp at all, can you spot it?

ELISP> (directory-files "~/dev/emacs/el-get/recipes/" nil "el$")

("auctex.el" "auto-complete-etags.el" "auto-complete-extension.el"
"auto-complete.el" "auto-install.el" "autopair.el" "bbdb.el"
"blender-python-mode.el" "color-theme-twilight.el" "color-theme.el"
"cssh.el" "django-mode.el" "el-get.el" "emacs-w3m.el" "emacschrome.el"
"emms.el" "ensime.el" "erc-highlight-nicknames.el" "erc-track-score.el"
"escreen.el" "filladapt.el" "flyguess.el" "gist.el" "google-maps.el"
"google-weather.el" "goto-last-change.el" "haskell-mode.el"
"highlight-parentheses.el" "hl-sexp.el" "levenshtein.el" "magit.el"
"mailq.el" "maxframe.el" "multi-term.el" "muse-blog.el" "nognus.el"
"nterm.el" "nxhtml.el" "offlineimap.el" "package.el" "popup-kill-ring.el"
"pos-tip.el" "pov-mode.el" "psvn.el" "pymacs.el" "rainbow-mode.el"
"rcirc-groups.el" "rinari.el" "ropemacs.el" "rt-liberation.el" "scratch.el"
"session.el" "sicp.el" "smex.el" "switch-window.el" "textile-mode.el"
"todochiku.el" "twitter.el" "twittering-mode.el" "undo-tree.el"
"verbiste.el" "vimpulse-surround.el" "vimpulse.el" "vkill.el" "xcscope.el"
"xml-rpc-el.el" "yasnippet.el")

Ok, I want to try it, what's next?

Visit the following URL http://github.com/dimitri/el-get and follow the install instructions. You're given a scratch installer there, that's some elisp code you copy paste into *scratch* then execute there, and you have el-get ready to serve.

An excellent idea I stole at ELPA!

Hey, I already know what el-get is, what's new in 1.0?

The changelog is quite full of good stuff, really:

  • Implement el-get recipes so that el-get-sources can be a simple list of symbols. Now that there's an authoritative git repository, where to share the recipes is easy.
  • Add support for emacswiki directly, save from having to enter the URL
  • Implement package status on-disk saving so that installing over a previously failed install is in theory possible. Currently `el-get' will refrain from removing your package automatically, though.
  • Fix ELPA remove method, adding a "removed" state too.
  • Implement CVS login support.
  • Add lots of recipes
  • Add support for `system-type' specific build commands
  • Byte compile files from the load-path entries or :compile files
  • Implement support for git submodules with the command `git submodule update —init —recursive`
  • Add catch-all post-install and post-update hooks
  • Add desktop notification on install/update.

I'm still using the deprecated emacswiki version, what now?

That version didn't have recipes, and the new version should be perfectly happy with your current el-get-sources, so that I recommend using the scratch installer too. Don't forget to add el-get itself into your el-get-sources list, of course!

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 07 Oct 2010 13:30:00 +0200 http://tapoueh.org/blog/2010/10/blog/2010/10/07-el-get-reaches-10.html el-get reaches 1.0 http://tapoueh.org/blog/2010/10/07-el-get-reaches-10.html el-get reaches 1.0
Thursday, October 07 2010, 13:30

It's been a week since the last commits in the el-get repository, and those were all about fixing and adding recipes, and about notifications. Nothing like core plumbing you see. Also, 0.9 was released on 2010-08-24 and felt pretty complete already, then received lots of improvements. It's high time to cross the line and call it 1.0!

Now existing users will certainly just be moderatly happy to see the tool reach that version number, depending whether they think more about the bugs they want to see fixed (ftp is supported, only called http) and the new features they want to see in (info documentation) or more about what el-get does for them already today...

For the new users, or the yet-to-be-convinced users, let's take some time and talk about el-get. A FAQ like session might be best.

How is el-get different from ELPA?

ELPA is the Emacs Lisp Package Archive and is also known as package.el, to be included in Emacs 24. This allows emacs list extension authors to package their work. That means they have to follow some guidelines and format their contribution, then propose it for upload.

This requires licence checks (good) and for the new official ELPA mirror it even requires dead-tree papers exchange and contracts and copyright assignments, I believe.

Why have both?

While ELPA is a great thing to have, it's so easy to find some high quality Emacs extension out there that are not part of the offer. Either authors are not interrested into uploading to ELPA, or they don't know how to properly package for it (it's only simple for single file extensions, see).

So el-get is a pragmatic answer here. It's there because it so happens that I don't depend only on emacs extensions that are available with Emacs itself, in my distribution site-lisp and in ELPA. I need some more, and I don't need it to be complex to find it, fetch it, init it and use it.

Of course I could try and package any extension I find I need and submit it to ELPA, but really, to do that nicely I'd need to contact the extension author (upstream) for him to accept my patch, and then consider a fork.

With el-get I propose distributed packaging if you will. Let's have a look at two recipes here. First, the el-get one itself:

(:name el-get
       :type git
       :url "git://github.com/dimitri/el-get.git"
       :features el-get
       :compile "el-get.el")

Then a much more complex one, the bbdb one:

(:name bbdb
       :type git
       :url "git://github.com/barak/BBDB.git"
       :load-path ("./lisp" "./bits")
       :build ("./configure" "make autoloads" "make")
       :build/darwin ("./configure --with-emacs=/Applications/Emacs.app/Contents/MacOS/Emacs" "make autoloads" "make")
       :features bbdb
       :after (lambda () (bbdb-initialize))
       :info "texinfo")

The idea is that it's much simpler to just come up with a recipe like this than to patch existing code and upload it to ELPA. And anybody can share their recipes very easily, with or without proposing them to me, even if I very much like to add some more in the official el-get list.

As a user, you don't even need to twiddle with recipes, mostly, because we already have them for you. What you do instead is list them in el-get-sources.

So, show me how you use it?

Yeah, sure. Here's a sample of my dim-packages.el file, part of my .emacs suite. Yeah a single .emacs does not suit me anymore, it's a complete .emacs.d now, but that's because that's how I like it organised, you know. So, here's the example:

;;; dim-packages.el --- Dimitri Fontaine
;;
;; Set el-get-sources and call el-get to init all those packages we need.
(require 'el-get)
(add-to-list 'el-get-recipe-path "~/dev/emacs/el-get/recipes")

(setq el-get-sources
      '(cssh el-get switch-window vkill google-maps yasnippet verbiste mailq sicp

        (:name magit
               :after (lambda () (global-set-key (kbd "C-x C-z") 'magit-status)))

        (:name asciidoc
               :type elpa
               :after (lambda ()
                        (autoload 'doc-mode "doc-mode" nil t)
                        (add-to-list 'auto-mode-alist '("\\.adoc$" . doc-mode))
                        (add-hook 'doc-mode-hook '(lambda ()
                                                    (turn-on-auto-fill)
                                                    (require 'asciidoc)))))

        (:name goto-last-change
               :after (lambda ()
                        (global-set-key (kbd "C-x C-/") 'goto-last-change)))

        (:name auto-dictionary :type elpa)
        (:name gist            :type elpa)
        (:name lisppaste       :type elpa)))

(el-get) ; that could/should be (el-get 'sync)
(provide 'dim-packages)

Ok that's not all of it, but it should give you a nice idea about what problem I solve with el-get and how. In my emacs startup sequence, somewhere inside my ~/.emacs.d/init.el file, I have a line that says (require 'dim-packages). This will set el-get-sources to the list just above, then call (el-get), the main function.

This main function will check each given package and install it if necessary (including build the package, as in make autoloads; make), then init it. What init means exactly depends on what the recipe says. That can include byte-compiling some files, caring about load-path, load and require commands, caring about Info-directory-list and ginstall-info too, and some more.

So in short, it will make it so that your emacs instance is ready for you to use. And you get the choice to use the given el-get recipes as-is, like I did for cssh, el-get, switch-window and others, up to sicp, or to tweak them partly, like in the magit example where I've added a user init function (the :after property) to bind magit-status to C-x C-z here. You can even embed a full recipe inline in the el-get-sources variable, that's the case for each item that gives its :type property, like asciidoc or gist.

And, as you see, we're using ELPA a lot in this sources, so el-get isn't striving to replace it at all, it's just trying to accomodate to a broader world.

I read that the el-get-install is asynchronous, tell me more.

Yeah, right, the example above says (el-get) at its end, and in the cases when el-get has to install or build sources, this will be done asynchronously. Which means that not only several sources will get processed at once (using your multi cores, yeah) but that it will let emacs start up as if it was ready.

It happens that's usually what I want, because I seldom add sources in my setup, but in theory that can break your emacs. What I do is start it again or fix by hand, what you can do instead is (el-get 'sync) so that emacs is blocked waiting for el-get to properly install and initialize all the sources you've setup. Your choice, just add the 'sync parameter there.

Now, explain me why it is better this way, again, please?

Well, before I wrote el-get, trying out a new extension, setting it up etc was something quite involved, and that I had to redo on several machines. The only way not to redo it was to include the extension's code into my own git repository (my emacs.d is in git, of course).

And putting code I don't maintain into my own git repository is something I frown upon. I have no business pretending I'll maintain the code, and I know I will never think to check the URL where I've found it for updates. That's when I though noting down the URL somewhere.

Also, what about sharing the extension with friends. Uneasy, at best.

Enters el-get and I can just add an entry to el-get-sources, based on a file somewhere in my own el-get-recipe-path. When I'm happy with this file, I can contribute it to el-get proper or just send it over to any interested recipient. Adding it to your sources is easy. Copy the file in your el-get-recipe-path somewhere, add its name to your el-get-sources, then M-x el-get-install it. Done. If you were given the :after function, it's all setup already.

If you contribute the recipe to el-get, then M-x el-get-update RET el-get RET and you get it on this other machine where you also use Emacs. Or you can tell your friend to do the same and benefit from your packaging.

Well, sounds good. What recipes do you have already?

I count 67 of them already. One of them is just a book in info format, with no elisp at all, can you spot it?

ELISP> (directory-files "~/dev/emacs/el-get/recipes/" nil "el$")

("auctex.el" "auto-complete-etags.el" "auto-complete-extension.el"
"auto-complete.el" "auto-install.el" "autopair.el" "bbdb.el"
"blender-python-mode.el" "color-theme-twilight.el" "color-theme.el"
"cssh.el" "django-mode.el" "el-get.el" "emacs-w3m.el" "emacschrome.el"
"emms.el" "ensime.el" "erc-highlight-nicknames.el" "erc-track-score.el"
"escreen.el" "filladapt.el" "flyguess.el" "gist.el" "google-maps.el"
"google-weather.el" "goto-last-change.el" "haskell-mode.el"
"highlight-parentheses.el" "hl-sexp.el" "levenshtein.el" "magit.el"
"mailq.el" "maxframe.el" "multi-term.el" "muse-blog.el" "nognus.el"
"nterm.el" "nxhtml.el" "offlineimap.el" "package.el" "popup-kill-ring.el"
"pos-tip.el" "pov-mode.el" "psvn.el" "pymacs.el" "rainbow-mode.el"
"rcirc-groups.el" "rinari.el" "ropemacs.el" "rt-liberation.el" "scratch.el"
"session.el" "sicp.el" "smex.el" "switch-window.el" "textile-mode.el"
"todochiku.el" "twitter.el" "twittering-mode.el" "undo-tree.el"
"verbiste.el" "vimpulse-surround.el" "vimpulse.el" "vkill.el" "xcscope.el"
"xml-rpc-el.el" "yasnippet.el")

Ok, I want to try it, what's next?

Visit the following URL http://github.com/dimitri/el-get and follow the install instructions. You're given a scratch installer there, that's some elisp code you copy paste into *scratch* then execute there, and you have el-get ready to serve.

An excellent idea I stole at ELPA!

Hey, I already know what el-get is, what's new in 1.0?

The changelog is quite full of good stuff, really:

  • Implement el-get recipes so that el-get-sources can be a simple list of symbols. Now that there's an authoritative git repository, where to share the recipes is easy.
  • Add support for emacswiki directly, save from having to enter the URL
  • Implement package status on-disk saving so that installing over a previously failed install is in theory possible. Currently `el-get' will refrain from removing your package automatically, though.
  • Fix ELPA remove method, adding a "removed" state too.
  • Implement CVS login support.
  • Add lots of recipes
  • Add support for `system-type' specific build commands
  • Byte compile files from the load-path entries or :compile files
  • Implement support for git submodules with the command `git submodule update —init —recursive`
  • Add catch-all post-install and post-update hooks
  • Add desktop notification on install/update.

I'm still using the deprecated emacswiki version, what now?

That version didn't have recipes, and the new version should be perfectly happy with your current el-get-sources, so that I recommend using the scratch installer too. Don't forget to add el-get itself into your el-get-sources list, of course!

Tags

Emacs Muse el-get Extensions release switch-window cssh mailq rcirc

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 07 Oct 2010 13:30:00 +0200 http://tapoueh.org/blog/2010/10/07-el-get-reaches-10.html Regexp performances and Finite Automata http://tapoueh.org/blog/2010/09/blog/2010/09/26-regexp-performances-and-finite-automata.html  

The major reason why I dislike perl so much, and ruby too, and the thing I'd want different in the Emacs Lisp API so far is how they set developers mind into using regexp. You know the quote, don't you?

Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.

That said, some situations require the use of regexp — or are so much simpler to solve using them than the maintenance hell you're building here ain't that big a drag. The given expressiveness is hard to match with any other solution, to the point I sometime use them in my code (well I use rx to lower the burden sometime, just see this example).

(rx bol (zero-or-more blank) (one-or-more digit) ":")
"^[[:blank:]]*[[:digit:]]+:"

The thing you might want to know about regexp is that computing them is an heavy task usually involving parsing their representation, compiling it to some executable code, and then executing generated code. It's been showed in the past (as soon as 1968) that a regexp is just another way to write a finite automata, at least as soon as you don't need backtracking. The writing of this article is my reaction to reading Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...), a very interesting article — see the benchmarks in there.

The bulk of it is that we find mainly two categories of regexp engine in the wild, those that are using NFA and DFA intermediate representation techniques, and the others. Our beloved PostgreSQL sure offers the feature, it's the ~ and ~* operators. The implementation here is based on Henry Spencer's work, which the aforementioned article says

became very widely used, eventually serving as the basis for the slow regular expression implementations mentioned earlier: Perl, PCRE, Python, and so on.

Having a look at the actual implementation shows that indeed, current PostgreSQL code for regexp matching uses intermediate representations of them as NFA and DFA. The code is quite complex, even more than I though it would be, and I didn't have the time it would take to check it against the proposed one from the simple and fast article.

postgresql/src/backend/regex
  -rw-r--r--   1 dim  staff   4362 Sep 25 20:59 COPYRIGHT
  -rw-r--r--   1 dim  staff    614 Sep 25 20:59 Makefile
  -rw-r--r--   1 dim  staff  28217 Sep 25 20:59 re_syntax.n
  -rw-r--r--   1 dim  staff  16589 Sep 25 20:59 regc_color.c
  -rw-r--r--   1 dim  staff   3464 Sep 25 20:59 regc_cvec.c
  -rw-r--r--   1 dim  staff  25036 Sep 25 20:59 regc_lex.c
  -rw-r--r--   1 dim  staff  16845 Sep 25 20:59 regc_locale.c
  -rw-r--r--   1 dim  staff  35917 Sep 25 20:59 regc_nfa.c
  -rw-r--r--   1 dim  staff  50714 Sep 25 20:59 regcomp.c
  -rw-r--r--   1 dim  staff  17368 Sep 25 20:59 rege_dfa.c
  -rw-r--r--   1 dim  staff   3627 Sep 25 20:59 regerror.c
  -rw-r--r--   1 dim  staff  27664 Sep 25 20:59 regexec.c
  -rw-r--r--   1 dim  staff   2122 Sep 25 20:59 regfree.c

So all in all, I'll continue avoiding regexp as much as I currently do, and will maintain my tendency to using awk when I need them on files (it allows to refine the searching without resorting to more and more pipes in the command line). And as far as resorting to using regexp in PostgreSQL is concerned, it seems that the code here is already about topnotch. Once more.

]]> dim@tapoueh.org (Dimitri Fontaine) Sun, 26 Sep 2010 21:00:00 +0200 http://tapoueh.org/blog/2010/09/blog/2010/09/26-regexp-performances-and-finite-automata.html Regexp performances and Finite Automata http://tapoueh.org/blog/2010/09/26-regexp-performances-and-finite-automata.html Regexp performances and Finite Automata
Sunday, September 26 2010, 21:00

 

The major reason why I dislike perl so much, and ruby too, and the thing I'd want different in the Emacs Lisp API so far is how they set developers mind into using regexp. You know the quote, don't you?

Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.

That said, some situations require the use of regexp — or are so much simpler to solve using them than the maintenance hell you're building here ain't that big a drag. The given expressiveness is hard to match with any other solution, to the point I sometime use them in my code (well I use rx to lower the burden sometime, just see this example).

(rx bol (zero-or-more blank) (one-or-more digit) ":")
"^[[:blank:]]*[[:digit:]]+:"

The thing you might want to know about regexp is that computing them is an heavy task usually involving parsing their representation, compiling it to some executable code, and then executing generated code. It's been showed in the past (as soon as 1968) that a regexp is just another way to write a finite automata, at least as soon as you don't need backtracking. The writing of this article is my reaction to reading Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...), a very interesting article — see the benchmarks in there.

The bulk of it is that we find mainly two categories of regexp engine in the wild, those that are using NFA and DFA intermediate representation techniques, and the others. Our beloved PostgreSQL sure offers the feature, it's the ~ and ~* operators. The implementation here is based on Henry Spencer's work, which the aforementioned article says

became very widely used, eventually serving as the basis for the slow regular expression implementations mentioned earlier: Perl, PCRE, Python, and so on.

Having a look at the actual implementation shows that indeed, current PostgreSQL code for regexp matching uses intermediate representations of them as NFA and DFA. The code is quite complex, even more than I though it would be, and I didn't have the time it would take to check it against the proposed one from the simple and fast article.

postgresql/src/backend/regex
  -rw-r--r--   1 dim  staff   4362 Sep 25 20:59 COPYRIGHT
  -rw-r--r--   1 dim  staff    614 Sep 25 20:59 Makefile
  -rw-r--r--   1 dim  staff  28217 Sep 25 20:59 re_syntax.n
  -rw-r--r--   1 dim  staff  16589 Sep 25 20:59 regc_color.c
  -rw-r--r--   1 dim  staff   3464 Sep 25 20:59 regc_cvec.c
  -rw-r--r--   1 dim  staff  25036 Sep 25 20:59 regc_lex.c
  -rw-r--r--   1 dim  staff  16845 Sep 25 20:59 regc_locale.c
  -rw-r--r--   1 dim  staff  35917 Sep 25 20:59 regc_nfa.c
  -rw-r--r--   1 dim  staff  50714 Sep 25 20:59 regcomp.c
  -rw-r--r--   1 dim  staff  17368 Sep 25 20:59 rege_dfa.c
  -rw-r--r--   1 dim  staff   3627 Sep 25 20:59 regerror.c
  -rw-r--r--   1 dim  staff  27664 Sep 25 20:59 regexec.c
  -rw-r--r--   1 dim  staff   2122 Sep 25 20:59 regfree.c

So all in all, I'll continue avoiding regexp as much as I currently do, and will maintain my tendency to using awk when I need them on files (it allows to refine the searching without resorting to more and more pipes in the command line). And as far as resorting to using regexp in PostgreSQL is concerned, it seems that the code here is already about topnotch. Once more.

Tags

PostgreSQL Emacs

]]> dim@tapoueh.org (Dimitri Fontaine) Sun, 26 Sep 2010 21:00:00 +0200 http://tapoueh.org/blog/2010/09/26-regexp-performances-and-finite-automata.html Postfix sender_dependent_relayhost_maps http://tapoueh.org/blog/2010/09/23-postfix-sender_dependent_relayhost_maps.html Postfix sender_dependent_relayhost_maps
Thursday, September 23 2010, 14:30

 

The previous article about M-x mailq has raised several mails asking me details about the Postfix setup I'm talking about. The problem we're trying to solve is having a local MTA to send mails, so that any old-style Unix tool just works, instead of only the MUA you've spent time setting up.

Postfix makes it possible to do that quite easily, but it gets a little more involved if you have more than one relayhost that you want to use depending on your current From address. Think personal email against work email, or avoiding your ISP network when sending your private mails, hoping directly on a server you own or trust.

So how do you do just that? Let's see the relevant parts of main.cf.

relayhost = your.default.relay.host.here
relay_domains = domain.org, work-domain.com, other-domain.info
smtp_sender_dependent_authentication = yes
sender_dependent_relayhost_maps = hash:/etc/postfix/relaymap

The relaymap looks like this:

# comments
user@domain.org         mail.domain.org
local@work-domain.com   smtp.work-domain.com
# that requires a local tunnel started with ssh, see ~/.ssh/config
me@other-domain.info    [127.0.0.1]:10025

You need to use postmap on this file before to reload or restart your local instance of Postfix.

Also, you should want to crypt your communication to your preferred relay host, using TLS goes like this:

smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl-passwords
smtp_sasl_mechanism_filter = digest-md5
smtp_sasl_security_options = noanonymous
smtp_sasl_mechanism_filter = login, plain
smtp_sasl_type = cyrus

smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache
smtp_tls_loglevel = 2
smtp_use_tls = yes
smtp_tls_security_level = may

The password file will need to get parsed by postmap too, and would better be set with limited read access, and looks like this:

mail.domain.org        user@domain.org:password
smtp.work-domain.com   local@work-domain.com:h4ckm3
[127.0.0.1]:10025      me@other-domain.info:guess

Hope this help you get started, at least that's a document I would have enjoyed reading when I first started to setup my local relaying MTA.

Oh, and now that you have this, I hope you will enjoy my M-x mailq tool for occasions when you're wondering why you're not receiving an answer back yet, then start the ssh tunnel…

Tags

mailq postfix

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 23 Sep 2010 14:30:00 +0200 http://tapoueh.org/blog/2010/09/23-postfix-sender_dependent_relayhost_maps.html Scratch that itch: M-x mailq http://tapoueh.org/blog/2010/09/23-scratch-that-itch-m-x-mailq.html Scratch that itch: M-x mailq
Thursday, September 23 2010, 09:30

Nowadays, most people would think that email is something simple, you just setup your preferred client (that's called a MUA) with some information such as the smtp host you want it to talk to (that's call a MTA and this one is your relayhost). Then there's all the receiving mails part, and that's smtp again on the server side. Then there's how to get those mail, read them, flag them, manage them, and that's better served by IMAP. Let's talk about sending mails in smtp for this entry.

The traditional way to handle mail sending is to have your own MTA on each system you use — there used to be a sysadmin team caring about all those systems, but we're lost in the personal computer era now — that only means you are the sysadmin. So about any Unix tool that wants to send a mail will do so with the command /usr/bin/sendmail to queue the outgoing message.

My typical workstation setup includes a full-blown MTA (my choice is Postfix) that will choose the next relay host depending on the message From field: I don't want to trust any local default relayhost. Note that the next relay is connected to with authentication and over an encrypted protocol.

We're getting there, really. But I don't know a better way to present a software, little as it be, other than talking about the need that leads to its development.

Some relaying I do atop an ssh tunnel, and it happens that I send mail and have forgotten about setting up the aforementioned tunnel. In this case, the advantage is that it will not block my MUA (gnus, in quite good shape those days, receiving lots of love), as the queueing happens as usual. The drawback is that Postfix will silently queue the mail until it's able to deliver it, which can take days.

Enters M-x mailq! Ok, I could be doing M-! mailq and see Mail queue is empty in the message area, but then as soon as the queue's not empty I need to resort to some shell or terminal in order to flush the queue — that's after setting up the tunnel, as easy as C-= remote in my case, see cssh. Scratching that itch, I now only have to hit f here, to flush the queue. And from the gnus *Group* and *Summary* buffers, it's M-q to see the mail queue.

Thanks to http://forum.ubuntu-fr.org/viewtopic.php?id=218883 here's a visual sample of the mailq mode, where you see the mail queue in colors and the keymap you're offered.

So you could even flush only a given queue id or a given site, or just kill the current id or the current site so that it's a C-y away. I hope it's useful for you too — oh, and it's already in the el-get recipes, of course!

Tags

el-get cssh mailq postfix

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 23 Sep 2010 09:30:00 +0200 http://tapoueh.org/blog/2010/09/23-scratch-that-itch-m-x-mailq.html switch-window reaches 0.8 http://tapoueh.org/blog/2010/09/blog/2010/09/13-switch-window-reaches-08.html I wanted to play with the idea of using the whole keyboard for my switch-window utility, but wondered how to get those keys in the right order and all. Finally found quail-keyboard-layout which seems to exists for such uses, as you can see:

(loop with layout = (split-string quail-keyboard-layout "")
  for row from 1 to 4
  collect (loop for col from 1 to 12
 ("q" "w" "e" "r" "t" "y" "u" "i" "o" "p" "[" "]")
 ("a" "s" "d" "f" "g" "h" "j" "k" "l" ";" "'" "\\")
 ("z" "x" "c" "v" "b" "n" "m" "," "." "/" " " " "))

So now switch-window will use that (but only the first 10 letters) instead of hard-coding numbers from 1 to 9 as labels and direct switches. That makes it more suitable to cssh users too, I guess.

In other news, I think el-get is about ready for its 1.0 release. Please test it and report any problem very soon before the release!

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 13 Sep 2010 17:45:00 +0200 http://tapoueh.org/blog/2010/09/blog/2010/09/13-switch-window-reaches-08.html switch-window reaches 0.8 http://tapoueh.org/blog/2010/09/13-switch-window-reaches-08.html switch-window reaches 0.8
Monday, September 13 2010, 17:45

I wanted to play with the idea of using the whole keyboard for my switch-window utility, but wondered how to get those keys in the right order and all. Finally found quail-keyboard-layout which seems to exists for such uses, as you can see:

(loop with layout = (split-string quail-keyboard-layout "")
  for row from 1 to 4
  collect (loop for col from 1 to 12
 ("q" "w" "e" "r" "t" "y" "u" "i" "o" "p" "[" "]")
 ("a" "s" "d" "f" "g" "h" "j" "k" "l" ";" "'" "\\")
 ("z" "x" "c" "v" "b" "n" "m" "," "." "/" " " " "))

So now switch-window will use that (but only the first 10 letters) instead of hard-coding numbers from 1 to 9 as labels and direct switches. That makes it more suitable to cssh users too, I guess.

In other news, I think el-get is about ready for its 1.0 release. Please test it and report any problem very soon before the release!

Tags

Emacs el-get release switch-window cssh

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 13 Sep 2010 17:45:00 +0200 http://tapoueh.org/blog/2010/09/13-switch-window-reaches-08.html Window Functions example remix http://tapoueh.org/blog/2010/09/12-window-functions-example-remix.html Window Functions example remix
Sunday, September 12 2010, 21:35

 

The drawback of hosting a static only website is, obviously, the lack of comments. What happens actually, though, is that I receive very few comments by direct mail. As I don't get another spam source to cleanup, I'm left unconvinced that's such a drawback. I still miss the low probability of seeing blog readers exchange directly, but I think a tapoueh.org mailing list would be my answer, here...

Anyway, David Fetter took the time to send me a comment by mail with a cleaned up rewrite of the previous entry SQL, here's it for your pleasure!

WITH t AS (
    SELECT
        o, w,
        CASE WHEN
            LAG(w) OVER(w) IS DISTINCT FROM w AND
            ROW_NUMBER() OVER (w) > 1 /* Eliminate first change */
        THEN 1
        END AS change
    FROM (
        VALUES
            (1, 5),
            (2, 10),
            (3, 7),
            (4, 7),
            (5, 7)
    ) AS data(o, w)
    WINDOW w AS (ORDER BY o) /* Factor out WINDOW */
)
SELECT SUM(change) FROM t;

As you can see David chose to filter the first change in the subquery rather than hacking it away with a simple -1 at the outer level. I'm still wondering which way is cleaner (that depends on how you look at the problem), but I think I know which one is simpler! Thanks David for this blog entry!

Tags

PostgreSQL

]]> dim@tapoueh.org (Dimitri Fontaine) Sun, 12 Sep 2010 21:35:00 +0200 http://tapoueh.org/blog/2010/09/12-window-functions-example-remix.html Window Functions example http://tapoueh.org/blog/2010/09/09-window-functions-example.html Window Functions example
Thursday, September 09 2010, 16:35

So, when 8.4 came out there was all those comments about how getting window functions was an awesome addition. Now, it seems that a lot of people seeking for help in #postgresql just don't know what kind of problem this feature helps solving. I've already been using them in some cases here in this blog, for getting some nice overview about Partitioning: relation size per “group”.

Now, another example use case rose on IRC today. I'll quote directly our user here:

hey there, how can i count the number of (value) changes in one column?

example: a table with a column weight. let's say we have 5 rows, having the following values for weight: 5, 10, 7, 7, 7. the number of changes of weight would be 2 here (from 5 to 10 and 10 to 7). any idea how I could do that in SQL using PGSQL 8.4.4? GROUP BY or count(distinct weight) obviously does not work. thx in advance

Now, several of us began talking about window functions and about the fact that you need some other column to identify the ordering of those weights, obviously, because that's the only way to define what a change is in this context. Let's have a first try at it.

=# select o, w,
          case when lag(w) over(order by o) is distinct from w then 1 end as change
     from (values (1, 5), (2, 10), (3, 7), (4, 7), (5, 7)) as data(o, w);
 o | w  | change
---+----+--------
 1 |  5 |      1
 2 | 10 |      1
 3 |  7 |      1
 4 |  7 |
 5 |  7 |
(5 rows)

Not too bad, but of course we are seeing a false change on the first line, as for any window of rows you define the previous one, given by lag() over(), will be NULL. The easiest way to accommodate is the following:

=# select sum(change) -1 as changes
     from (select case when lag(w) over(order by o) is distinct from w
                       then 1
                   end as change
             from (values (1, 5),
                          (2, 10),
                          (3, 7),
                          (4, 7),
                          (5, 7)) as t(o, w)) as x;
 changes
---------
       2
(1 row)

So don't be shy and go read about window functions in SQL expressions and window function processing in the query table expressions. That's a very nice tool to have and my guess is that you will soon enough realize the only reason why you could think you don't have a need for them is that you didn't know it existed, and what you can do with it. Sharpen your saw! :)

Tags

PostgreSQL

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 09 Sep 2010 16:35:00 +0200 http://tapoueh.org/blog/2010/09/09-window-functions-example.html Synchronous Replication http://tapoueh.org/blog/2010/09/06-synchronous-replication.html Synchronous Replication
Monday, September 06 2010, 18:05

Although the new asynchronous replication facility that ships with 9.0 ain't released to the wide public yet, our hackers hero are already working on the synchronous version of it. A part of the facility is rather easy to design, we want something comparable to DRBD flexibility, but specific to our database world. So synchronous would either mean recv, fsync or apply, depending on what you need the standby to have already done when the master acknowledges the COMMIT. Let's call that the service level.

The part of the design that's not so easy is more interesting. Do we need to register standbys and have the service level setup per standby? Can we get some more flexibility and have the service level set on a per-transaction basis? The idea here would be that the application knows which transactions are meant to be extra-safe and which are not, the same way that you can set synchronous_commit to off when dealing with web sessions, for example.

Why choosing? I hear you ask. Well, it's all about having more data safety, and a typical setup would contain an asynchronous reporting server and a local failover synchronous server. Then add a remote one, too. So even if we pick the transaction based facility, we still want to be able to choose at setup time which server to failover to. Than means we don't want that much flexibility now, we want to know where the data is safe, we don't want to have to guess.

Some way to solve that is to be able to setup a slave as being the failover one, or say, the sync one. Now, the detail that ruins it all is that we need a timeout to handle worst cases when a given slave loses its connectivity (or power, say). Now, the slave ain't in sync any more and some people will require that the service is still available (timeout but COMMIT) and some will require that the service is down: don't accept a new transaction if you can't make its data safe to the slave too.

The answer would be to have the master arbitrate between what the transaction wants and what the slave is setup to provide, and what it's able to provide at the time of the transaction. Given a transaction with a service level of apply and a slave setup for being async, the COMMIT does not have to wait, because there's no known slave able to offer the needed level. Or the COMMIT can not happen, for the very same reason.

Then I think it all flows quite naturally from there, and while arbitrating the master could record which slave is currently offering what service level. And offering the information in a system view too, of course.

The big question that's not answered in this proposal is how to setup that being unable to reach the wanted service level is an error or a warning?

That too would need to be for the master to arbitrate based on a per standby and a per transaction setting, and in the general case it could be a quorum setup: each slave is given a weight and each transaction a quorum to reach. The master sums up the weights of the standby that ack the transaction at the needed service level and the COMMIT happens as soon as the quorum is reached, or is canceled as soon as the timeout is reached, whichever comes first.

Such a model allows for very flexible setups, where each standby has a weight and offers a given service level, and each transaction waits until a quorum is reached. Giving the right weights to your standbys (like, powers of two) allow you to set the quorum in a way that only one given standby is able to acknowledge the most important transactions. But that's flexible enough you can change it at any time, it's just a weight that allows a sum to be made, so my guess would be it ends up in the feedback loop between the standby and its master.

The most appealing part of this proposal is that it doesn't look complex to implement, and should allow for highly flexible setups. Of course, the devil is in the details, and we're talking about latencies in the distributed system here. That's also being discussed on the mailing list.

Tags

PostgreSQL release

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 06 Sep 2010 18:05:00 +0200 http://tapoueh.org/blog/2010/09/06-synchronous-replication.html Want to share your recipes? http://tapoueh.org/blog/2010/08/blog/2010/08/31-want-to-share-your-recipes.html Yes, that's another el-get related entry. It seems to take a lot of my attention these days. After having setup the git repository so that you can update el-get from within itself (so that it's self-contained), the next logical step is providing recipes.

By that I mean that el-get-sources entries will certainly look a lot alike between a user and another. Let's take the el-get entry itself:

(:name el-get
       :type git
       :url "git://github.com/dimitri/el-get.git"
       :features "el-get")

I guess all el-get users will have just the same 4 lines in their el-get-sources. So let's call that a recipe, and have el-get look for yours into the el-get-recipe-path directories. A recipe is found looking in those directories in order, and must be named package.el. Now, el-get already contains a handful of them, as you can see:

ELISP> (directory-files "~/dev/emacs/el-get/recipes/" nil "[^.]$")
("auctex.el" "bbdb.el" "cssh.el" "el-get.el" "emms.el" "erc-track-score.el"
 "escreen.el" "google-maps.el" "haskell-mode.el" "hl-sexp.el" "magit.el"
 "muse-blog.el" "nxhtml.el" "psvn.el" "rainbow-mode.el" "rcirc-groups.el"
 "vkill.el" "xcscope.el" "xml-rpc-el.el" "yasnippet.el")

Please note that you can have your own local recipes by adding directories to el-get-recipe-path. So now your minimalistic el-get-sources list will look like '(el-get cssh screen), say. And if you want to override a recipe, for instance to use the default one but still have a personal :after function containing your own setup, then simply have your el-get-source entry a partial entry. Missing :type and el-get will merge your local overrides atop the default one.

Finally, the way to share your recipes is by sending me an email with the file, or to do the same over the github interface, I guess I'll still receive a mail then.

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 31 Aug 2010 14:15:00 +0200 http://tapoueh.org/blog/2010/08/blog/2010/08/31-want-to-share-your-recipes.html Want to share your recipes? http://tapoueh.org/blog/2010/08/31-want-to-share-your-recipes.html Want to share your recipes?
Tuesday, August 31 2010, 14:15

Yes, that's another el-get related entry. It seems to take a lot of my attention these days. After having setup the git repository so that you can update el-get from within itself (so that it's self-contained), the next logical step is providing recipes.

By that I mean that el-get-sources entries will certainly look a lot alike between a user and another. Let's take the el-get entry itself:

(:name el-get
       :type git
       :url "git://github.com/dimitri/el-get.git"
       :features "el-get")

I guess all el-get users will have just the same 4 lines in their el-get-sources. So let's call that a recipe, and have el-get look for yours into the el-get-recipe-path directories. A recipe is found looking in those directories in order, and must be named package.el. Now, el-get already contains a handful of them, as you can see:

ELISP> (directory-files "~/dev/emacs/el-get/recipes/" nil "[^.]$")
("auctex.el" "bbdb.el" "cssh.el" "el-get.el" "emms.el" "erc-track-score.el"
 "escreen.el" "google-maps.el" "haskell-mode.el" "hl-sexp.el" "magit.el"
 "muse-blog.el" "nxhtml.el" "psvn.el" "rainbow-mode.el" "rcirc-groups.el"
 "vkill.el" "xcscope.el" "xml-rpc-el.el" "yasnippet.el")

Please note that you can have your own local recipes by adding directories to el-get-recipe-path. So now your minimalistic el-get-sources list will look like '(el-get cssh screen), say. And if you want to override a recipe, for instance to use the default one but still have a personal :after function containing your own setup, then simply have your el-get-source entry a partial entry. Missing :type and el-get will merge your local overrides atop the default one.

Finally, the way to share your recipes is by sending me an email with the file, or to do the same over the github interface, I guess I'll still receive a mail then.

Tags

Emacs Muse el-get cssh rcirc

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 31 Aug 2010 14:15:00 +0200 http://tapoueh.org/blog/2010/08/31-want-to-share-your-recipes.html Happy Numbers http://tapoueh.org/blog/2010/08/blog/2010/08/30-happy-numbers.html After discovering the excellent Gwene service, which allows you to subscribe to newsgroups to read RSS content (blogs, planets, commits, etc), I came to read this nice article about Happy Numbers. That's a little problem that fits well an interview style question, so I first solved it yesterday evening in Emacs Lisp as that's the language I use the most those days.

A happy number is defined by the following process. Starting with any positive integer, replace the number by the sum of the squares of its digits, and repeat the process until the number equals 1 (where it will stay), or it loops endlessly in a cycle which does not include 1. Those numbers for which this process ends in 1 are happy numbers, while those that do not end in 1 are unhappy numbers (or sad numbers).

Now, what about implementing the same in pure SQL, for more fun? Now that's interesting! After all, we didn't get WITH RECURSIVE for tree traversal only, did we?

Unfortunately, we need a little helper function first, if only to ease the reading of the recursive query. I didn't try to inline it, but here it goes:

create or replace function digits(x bigint)
  returns setof int
  language sql
as $$
  select substring($1::text from i for 1)::int
    from generate_series(1, length($1::text)) as t(i)
$$;

That was easy: it will output one row per digit of the input number — and rather than resorting to powers of ten and divisions and remainders, we do use plain old text representation and substring. Now, to the real problem. If you're read what is an happy number and already did read the fine manual about Recursive Query Evaluation, it should be quite easy to read the following:

with recursive happy(n, seen) as (
    select 7::bigint, '{}'::bigint[]
  union all
    select sum(d*d), h.seen || sum(d*d)
      from (select n, digits(n) as d, seen
              from happy
           ) as h
  group by h.n, h.seen
    having not seen @> array[sum(d*d)]
)
  select * from happy;
  n  |       seen
-----+------------------
   7 | {}
  49 | {49}
  97 | {49,97}
 130 | {49,97,130}
  10 | {49,97,130,10}
   1 | {49,97,130,10,1}
(6 rows)

Time: 1.238 ms

That shows how it works for some happy number, and it's easy to test for a non-happy one, like for example 17. The query won't cycle thanks to the seen array and the having filter, so the only difference between an happy and a sad number will be that in the former case the last line output by the recursive query will have n = 1. Let's expand this knowledge into a proper function (because we want to be able to have the number we test for happiness as an argument):

create or replace function happy(x bigint)
  returns boolean
  language sql
as $$
with recursive happy(n, seen) as (
    select $1, '{}'::bigint[]
  union all
    select sum(d*d), h.seen || sum(d*d)
      from (select n, digits(n) as d, seen
              from happy
           ) as h
  group by h.n, h.seen
    having not seen @> array[sum(d*d)]
)
  select n = 1 as happy
    from happy
order by array_length(seen, 1) desc nulls last
   limit 1
$$;

We need the desc nulls last trick in the order by because the array_length() of any dimension of an empty array is NULL, and we certainly don't want to return all and any number as unhappy on the grounds that the query result contains a line input, {}. Let's now play the same tricks as in the puzzle article:

=# select array_agg(x) as happy
     from generate_series(1, 50) as t(x)
    where happy(x);
              happy
----------------------------------
 {1,7,10,13,19,23,28,31,32,44,49}
(1 row)

Time: 24.527 ms

=# explain analyze select x
                     from generate_series(1, 10000) as t(x)
                    where happy(x);
                      QUERY PLAN
------------------------------------------------------------
 Function Scan on generate_series t
     (cost=0.00..265.00 rows=333 width=4)
     (actual time=2.938..3651.019 rows=1442 loops=1)
   Filter: happy((x)::bigint)
 Total runtime: 3651.534 ms
(3 rows)

Time: 3652.178 ms

(Yes, I tricked the EXPLAIN ANALYZE output so that it fits on the page width here). For what it's worth, finding the first 10000 happy numbers in Emacs Lisp on the same laptop takes 2830 ms, also running a recursive version of the code.

Update, the Emacs Lisp version, inline:

(defun happy? (&optional n seen)
  "return true when n is a happy number"
  (interactive)
  (let* ((number    (or n (read-from-minibuffer
                           "Is this number happy: ")))
         (digits    (mapcar
                     'string-to-int
                     (subseq (split-string number "") 1 -1)))
         (squares   (mapcar (lambda (x) (* x x)) digits))
         (happiness (apply '+ squares)))
    (cond ((eq 1 happiness)      t)
          ((memq happiness seen) nil)
          (t
           (happy? (number-to-string happiness)
                   (push happiness seen))))))

(defun find-happy-numbers (&optional limit)
  "find all happy numbers from 1 to limit"
  (interactive)
  (let ((count (or limit
                   (read-from-minibuffer
                    "List of happy numbers from 1 to: ")))
        happy)
    (dotimes (n (string-to-int count))
      (when (happy? (number-to-string (1+ n)))
        (push (1+ n) happy)))
    (nreverse happy)))
]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 30 Aug 2010 11:00:00 +0200 http://tapoueh.org/blog/2010/08/blog/2010/08/30-happy-numbers.html Happy Numbers http://tapoueh.org/blog/2010/08/30-happy-numbers.html Happy Numbers
Monday, August 30 2010, 11:00

After discovering the excellent Gwene service, which allows you to subscribe to newsgroups to read RSS content (blogs, planets, commits, etc), I came to read this nice article about Happy Numbers. That's a little problem that fits well an interview style question, so I first solved it yesterday evening in Emacs Lisp as that's the language I use the most those days.

A happy number is defined by the following process. Starting with any positive integer, replace the number by the sum of the squares of its digits, and repeat the process until the number equals 1 (where it will stay), or it loops endlessly in a cycle which does not include 1. Those numbers for which this process ends in 1 are happy numbers, while those that do not end in 1 are unhappy numbers (or sad numbers).

Now, what about implementing the same in pure SQL, for more fun? Now that's interesting! After all, we didn't get WITH RECURSIVE for tree traversal only, did we?

Unfortunately, we need a little helper function first, if only to ease the reading of the recursive query. I didn't try to inline it, but here it goes:

create or replace function digits(x bigint)
  returns setof int
  language sql
as $$
  select substring($1::text from i for 1)::int
    from generate_series(1, length($1::text)) as t(i)
$$;

That was easy: it will output one row per digit of the input number — and rather than resorting to powers of ten and divisions and remainders, we do use plain old text representation and substring. Now, to the real problem. If you're read what is an happy number and already did read the fine manual about Recursive Query Evaluation, it should be quite easy to read the following:

with recursive happy(n, seen) as (
    select 7::bigint, '{}'::bigint[]
  union all
    select sum(d*d), h.seen || sum(d*d)
      from (select n, digits(n) as d, seen
              from happy
           ) as h
  group by h.n, h.seen
    having not seen @> array[sum(d*d)]
)
  select * from happy;
  n  |       seen
-----+------------------
   7 | {}
  49 | {49}
  97 | {49,97}
 130 | {49,97,130}
  10 | {49,97,130,10}
   1 | {49,97,130,10,1}
(6 rows)

Time: 1.238 ms

That shows how it works for some happy number, and it's easy to test for a non-happy one, like for example 17. The query won't cycle thanks to the seen array and the having filter, so the only difference between an happy and a sad number will be that in the former case the last line output by the recursive query will have n = 1. Let's expand this knowledge into a proper function (because we want to be able to have the number we test for happiness as an argument):

create or replace function happy(x bigint)
  returns boolean
  language sql
as $$
with recursive happy(n, seen) as (
    select $1, '{}'::bigint[]
  union all
    select sum(d*d), h.seen || sum(d*d)
      from (select n, digits(n) as d, seen
              from happy
           ) as h
  group by h.n, h.seen
    having not seen @> array[sum(d*d)]
)
  select n = 1 as happy
    from happy
order by array_length(seen, 1) desc nulls last
   limit 1
$$;

We need the desc nulls last trick in the order by because the array_length() of any dimension of an empty array is NULL, and we certainly don't want to return all and any number as unhappy on the grounds that the query result contains a line input, {}. Let's now play the same tricks as in the puzzle article:

=# select array_agg(x) as happy
     from generate_series(1, 50) as t(x)
    where happy(x);
              happy
----------------------------------
 {1,7,10,13,19,23,28,31,32,44,49}
(1 row)

Time: 24.527 ms

=# explain analyze select x
                     from generate_series(1, 10000) as t(x)
                    where happy(x);
                      QUERY PLAN
------------------------------------------------------------
 Function Scan on generate_series t
     (cost=0.00..265.00 rows=333 width=4)
     (actual time=2.938..3651.019 rows=1442 loops=1)
   Filter: happy((x)::bigint)
 Total runtime: 3651.534 ms
(3 rows)

Time: 3652.178 ms

(Yes, I tricked the EXPLAIN ANALYZE output so that it fits on the page width here). For what it's worth, finding the first 10000 happy numbers in Emacs Lisp on the same laptop takes 2830 ms, also running a recursive version of the code.

Update, the Emacs Lisp version, inline:

(defun happy? (&optional n seen)
  "return true when n is a happy number"
  (interactive)
  (let* ((number    (or n (read-from-minibuffer
                           "Is this number happy: ")))
         (digits    (mapcar
                     'string-to-int
                     (subseq (split-string number "") 1 -1)))
         (squares   (mapcar (lambda (x) (* x x)) digits))
         (happiness (apply '+ squares)))
    (cond ((eq 1 happiness)      t)
          ((memq happiness seen) nil)
          (t
           (happy? (number-to-string happiness)
                   (push happiness seen))))))

(defun find-happy-numbers (&optional limit)
  "find all happy numbers from 1 to limit"
  (interactive)
  (let ((count (or limit
                   (read-from-minibuffer
                    "List of happy numbers from 1 to: ")))
        happy)
    (dotimes (n (string-to-int count))
      (when (happy? (number-to-string (1+ n)))
        (push (1+ n) happy)))
    (nreverse happy)))

Tags

PostgreSQL Emacs

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 30 Aug 2010 11:00:00 +0200 http://tapoueh.org/blog/2010/08/30-happy-numbers.html welcome el-get scratch installer http://tapoueh.org/blog/2010/08/blog/2010/08/27-welcome-el-get-scratch-installer.html  

A very good remark from some users: installing and managing el-get should be simpler. They wanted both an easy install of the thing, and a way to be able to manage it afterwards (like, update the local copy against the authoritative source). So I decided it was high time for getting the code out of my ~/.emacs.d git repository and up to a public place: http://github.com/dimitri/el-get.

Then, I added some documentation (a README), and then, a *scratch* installer, following great ideas from ELPA. So have at it, it's a copy paste away!

Don't forget to setup your el-get-sources and include there the el-get source for updates, there's nothing magic about it so it's up to you. You may notice that it's not yet possible to init el-get from el-get-sources, though, that's the drawback of the lack of magic. So you will have to still add an explicit (require 'el-get) before to go and define you own el-get-sources then finally (el-get). I don't think that's a problem I need to solve, though.

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 27 Aug 2010 14:15:00 +0200 http://tapoueh.org/blog/2010/08/blog/2010/08/27-welcome-el-get-scratch-installer.html welcome el-get scratch installer http://tapoueh.org/blog/2010/08/27-welcome-el-get-scratch-installer.html welcome el-get scratch installer
Friday, August 27 2010, 14:15

 

A very good remark from some users: installing and managing el-get should be simpler. They wanted both an easy install of the thing, and a way to be able to manage it afterwards (like, update the local copy against the authoritative source). So I decided it was high time for getting the code out of my ~/.emacs.d git repository and up to a public place: http://github.com/dimitri/el-get.

Then, I added some documentation (a README), and then, a *scratch* installer, following great ideas from ELPA. So have at it, it's a copy paste away!

Don't forget to setup your el-get-sources and include there the el-get source for updates, there's nothing magic about it so it's up to you. You may notice that it's not yet possible to init el-get from el-get-sources, though, that's the drawback of the lack of magic. So you will have to still add an explicit (require 'el-get) before to go and define you own el-get-sources then finally (el-get). I don't think that's a problem I need to solve, though.

Tags

Emacs el-get

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 27 Aug 2010 14:15:00 +0200 http://tapoueh.org/blog/2010/08/27-welcome-el-get-scratch-installer.html Playing with bit strings http://tapoueh.org/blog/2010/08/26-playing-with-bit-strings.html Playing with bit strings
Thursday, August 26 2010, 17:45

The idea of the day ain't directly from me, I'm just helping with a very thin subpart of the problem. The problem, I can't say much about, let's just assume you want to reduce the storage of MD5 in your database, so you want to abuse bit strings. A solution to use them works fine, but the datatype is still missing some facilities, for example going from and to hexadecimal representation in text.

create or replace function hex_to_varbit(h text)
 returns varbit
 language sql
as $$
  select ('X' || $1)::varbit;
$$;

create or replace function varbit_to_hex(b varbit)
 returns text
 language sql
as $$
  select array_to_string(array_agg(to_hex((b << (32*o))::bit(32)::bigint)), '')
    from (select b, generate_series(0, n-1) as o
            from (select $1, octet_length($1)/4) as t(b, n)) as x
$$;

To understand the magic in the second function, let's walk through the tests one could do when wanting to grasp how things work in the bitstring world (using also some reading of the fine documentation, too).

=# select ('101011001011100110010110'::varbit << 0)::bit(8);
   bit
----------
 10101100
(1 row)

=# select ('101011001011100110010110'::varbit << 8)::bit(8);
   bit
----------
 10111001
(1 row)

=# select ('101011001011100110010110'::varbit << 16)::bit(8);
   bit
----------
 10010110
(1 row)

=# select * from *TEMP VERSION OF THE FUNCTION FOR TESTING*
 o |                b                 |    x
---+----------------------------------+----------
 0 | 10101100101111010001100011011011 | acbd18db
 1 | 01001100110000101111100001011100 | 4cc2f85c
 2 | 11101101111011110110010101001111 | edef654f
 3 | 11001100110001001010010011011000 | ccc4a4d8
(4 rows)

What do we get from that, will you ask? Let's see a little example:

=# select hex_to_varbit(md5('foo'));
                                                          hex_to_varbit
----------------------------------------------------------------------------------------------------------------------------------
 10101100101111010001100011011011010011001100001011111000010111001110110111101111011001010100111111001100110001001010010011011000
(1 row)

=# select md5('foo'), varbit_to_hex(hex_to_varbit(md5('foo')));
               md5                |          varbit_to_hex
----------------------------------+----------------------------------
 acbd18db4cc2f85cedef654fccc4a4d8 | acbd18db4cc2f85cedef654fccc4a4d8
(1 row)

Storing varbits rather than the text form of the MD5 allows us to go from 6510 MB down to 4976 MB on a sample table containing 100 millions rows. We're targeting more that that, so that's a great win down here!

In case you wonder, querying the main index on varbit rather than the one on text for a single result row, the cost of doing the conversion with varbit_to_hex seems to be around 28 µs. We can afford it.

Hope this helps!

Tags

PostgreSQL

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 26 Aug 2010 17:45:00 +0200 http://tapoueh.org/blog/2010/08/26-playing-with-bit-strings.html el-get news http://tapoueh.org/blog/2010/08/blog/2010/08/26-el-get-news.html I've been receiving some requests for el-get, some of them even included a patch. So now there's support for bzr, CSV and http-tar, augmenting the existing support for git, git-svn, apt-get, fink and ELPA formats.

Also, as the install and even the build are completely asynchronous — there's a pending bugfix for the building, which is now using start-process-shell-command. The advantage of doing so is that you're free to use Emacs as usual while el-get is having your piece of elisp code compiled, which can take time.

The drawback is that it's uneasy to to do the associated setup at the right time without support from el-get, so you have the new option :after which takes a functionp object: please consider using that to give your own special setup for the external emacs bits and pieces you're using.

Let's see some examples of the new features:

  (:name xml-rpc-el
         :type bzr
         :url "lp:xml-rpc-el")

  (:name haskell-mode
         :type http-tar
         :options ("xzf")
         :url "http://projects.haskell.org/haskellmode-emacs/haskell-mode-2.8.0.tar.gz"
         :load "haskell-site-file.el"
         :after (lambda ()
                  (add-hook 'haskell-mode-hook 'turn-on-haskell-doc-mode)
                  (add-hook 'haskell-mode-hook 'turn-on-haskell-indentation)))

  (:name auctex
         :type cvs
         :module "auctex"
         :url ":pserver:anonymous@cvs.sv.gnu.org:/sources/auctex"
         :build ("./autogen.sh" "./configure" "make")
         :load  ("auctex.el" "preview/preview-latex.el")
         :info "doc")

As you can see, there are also the new options :module (only used by CVS so far) and :options (only used by http-tar so far). With this later method, the :options key allows you to have support for virtually any kind of tar compression (.tar.bz2, etc).

The CVS support currently does not include authentication against the anonymous pserver, because the only repository I've been asked support for isn't using that, and the couple of servers that I know of are either wanting no password at the prompt, or a dummy one. That's for another day, if needed at all.

That pushes the little local hack to more than a thousand lines of elisp code, and the next steps include proposing it to ELPA so that getting to use it is easier than ever. You'd just have to choose whether to install ELPA from el-get or the other way around.

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 26 Aug 2010 16:30:00 +0200 http://tapoueh.org/blog/2010/08/blog/2010/08/26-el-get-news.html el-get news http://tapoueh.org/blog/2010/08/26-el-get-news.html el-get news
Thursday, August 26 2010, 16:30

I've been receiving some requests for el-get, some of them even included a patch. So now there's support for bzr, CSV and http-tar, augmenting the existing support for git, git-svn, apt-get, fink and ELPA formats.

Also, as the install and even the build are completely asynchronous — there's a pending bugfix for the building, which is now using start-process-shell-command. The advantage of doing so is that you're free to use Emacs as usual while el-get is having your piece of elisp code compiled, which can take time.

The drawback is that it's uneasy to to do the associated setup at the right time without support from el-get, so you have the new option :after which takes a functionp object: please consider using that to give your own special setup for the external emacs bits and pieces you're using.

Let's see some examples of the new features:

  (:name xml-rpc-el
         :type bzr
         :url "lp:xml-rpc-el")

  (:name haskell-mode
         :type http-tar
         :options ("xzf")
         :url "http://projects.haskell.org/haskellmode-emacs/haskell-mode-2.8.0.tar.gz"
         :load "haskell-site-file.el"
         :after (lambda ()
                  (add-hook 'haskell-mode-hook 'turn-on-haskell-doc-mode)
                  (add-hook 'haskell-mode-hook 'turn-on-haskell-indentation)))

  (:name auctex
         :type cvs
         :module "auctex"
         :url ":pserver:anonymous@cvs.sv.gnu.org:/sources/auctex"
         :build ("./autogen.sh" "./configure" "make")
         :load  ("auctex.el" "preview/preview-latex.el")
         :info "doc")

As you can see, there are also the new options :module (only used by CVS so far) and :options (only used by http-tar so far). With this later method, the :options key allows you to have support for virtually any kind of tar compression (.tar.bz2, etc).

The CVS support currently does not include authentication against the anonymous pserver, because the only repository I've been asked support for isn't using that, and the couple of servers that I know of are either wanting no password at the prompt, or a dummy one. That's for another day, if needed at all.

That pushes the little local hack to more than a thousand lines of elisp code, and the next steps include proposing it to ELPA so that getting to use it is easier than ever. You'd just have to choose whether to install ELPA from el-get or the other way around.

Tags

Emacs el-get

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 26 Aug 2010 16:30:00 +0200 http://tapoueh.org/blog/2010/08/26-el-get-news.html el-get and dim-switch-window status update http://tapoueh.org/blog/2010/08/blog/2010/08/09-el-get-and-dim-switch-window-status-update.html  

Thanks to you readers of Planet Emacsen taking the time to try those pieces of emacs lisp found in my blog, and also the time to comment on them, some bugs have been fixed, and new releases appeared.

el-get had some typo kind of bug in its support for apt-get and fink packages, and I managed to break the elpa and http support when going all asynchronous by forgetting to update the call convention I'm using. Fixing that, I also switched to using url-retrieve so that the http support also is asynchronous. That makes the version 0.5, available on emacswiki el-get page.

Meanwhile dim-switch-window.el got some testers too and got updated with a nice fix, or so I think. If you're using it with a small enough emacs frame, or some very little windows in there, you'd have noticed that the number get so big they don't fit anymore, and all you see while it's waiting for your window number choice is... blank windows. Not very helpful. Thanks to the following piece of code, that's no longer the case as of the current version, available on emacswiki switch-window page.

In short, where I used to blindly apply dim:switch-window-increase on the big numbers to display, the code now checks that there's enough room for it to get there, and adjust the increase level scaling it down if necessary. Very simple, and effective too:

    (with-current-buffer buf
      (text-scale-increase
       (if (> (/ (float (window-body-height win))
                 dim:switch-window-increase)
              1)
           dim:switch-window-increase
         (window-body-height win)))
      (insert "\n\n    " (number-to-string num)))

Centering the text in the window's width is another story entirely, as the text-scale-increase ain't linear on this axis. I'd take any good idea, here's what I'm currently at, but it's not there yet:

    (with-current-buffer buf
      (let* ((w (window-width win))
             (h (window-body-height win))
             (increased-lines (/ (float h) dim:switch-window-increase))
             (scale (if (> increased-lines 1) dim:switch-window-increase h))
             (lines-before (/ increased-lines 2))
             (margin-left (/ w h) ))
        ;; increase to maximum dim:switch-window-increase
        (text-scale-increase scale)
        ;; make it so that the hyuge number appears centered
        (dotimes (i lines-before) (insert "\n"))
        (dotimes (i margin-left)  (insert " "))
        (insert (number-to-string num))))

So, if you're using one or the other (both?) of those utilities, update your local version of them!

Note: I also fixed a but in rcirc-groups this week-end, but I'll talk about it in another entry, if I may.

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 09 Aug 2010 15:35:00 +0200 http://tapoueh.org/blog/2010/08/blog/2010/08/09-el-get-and-dim-switch-window-status-update.html el-get and dim-switch-window status update http://tapoueh.org/blog/2010/08/09-el-get-and-dim-switch-window-status-update.html el-get and dim-switch-window status update
Monday, August 09 2010, 15:35

 

Thanks to you readers of Planet Emacsen taking the time to try those pieces of emacs lisp found in my blog, and also the time to comment on them, some bugs have been fixed, and new releases appeared.

el-get had some typo kind of bug in its support for apt-get and fink packages, and I managed to break the elpa and http support when going all asynchronous by forgetting to update the call convention I'm using. Fixing that, I also switched to using url-retrieve so that the http support also is asynchronous. That makes the version 0.5, available on emacswiki el-get page.

Meanwhile dim-switch-window.el got some testers too and got updated with a nice fix, or so I think. If you're using it with a small enough emacs frame, or some very little windows in there, you'd have noticed that the number get so big they don't fit anymore, and all you see while it's waiting for your window number choice is... blank windows. Not very helpful. Thanks to the following piece of code, that's no longer the case as of the current version, available on emacswiki switch-window page.

In short, where I used to blindly apply dim:switch-window-increase on the big numbers to display, the code now checks that there's enough room for it to get there, and adjust the increase level scaling it down if necessary. Very simple, and effective too:

    (with-current-buffer buf
      (text-scale-increase
       (if (> (/ (float (window-body-height win))
                 dim:switch-window-increase)
              1)
           dim:switch-window-increase
         (window-body-height win)))
      (insert "\n\n    " (number-to-string num)))

Centering the text in the window's width is another story entirely, as the text-scale-increase ain't linear on this axis. I'd take any good idea, here's what I'm currently at, but it's not there yet:

    (with-current-buffer buf
      (let* ((w (window-width win))
             (h (window-body-height win))
             (increased-lines (/ (float h) dim:switch-window-increase))
             (scale (if (> increased-lines 1) dim:switch-window-increase h))
             (lines-before (/ increased-lines 2))
             (margin-left (/ w h) ))
        ;; increase to maximum dim:switch-window-increase
        (text-scale-increase scale)
        ;; make it so that the hyuge number appears centered
        (dotimes (i lines-before) (insert "\n"))
        (dotimes (i margin-left)  (insert " "))
        (insert (number-to-string num))))

So, if you're using one or the other (both?) of those utilities, update your local version of them!

Note: I also fixed a but in rcirc-groups this week-end, but I'll talk about it in another entry, if I may.

Tags

Emacs el-get release switch-window rcirc

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 09 Aug 2010 15:35:00 +0200 http://tapoueh.org/blog/2010/08/09-el-get-and-dim-switch-window-status-update.html Editing constants in constraints http://tapoueh.org/blog/2010/08/09-editing-constants-in-constraints.html Editing constants in constraints
Monday, August 09 2010, 14:45

 

We're using constants in some constraints here, for example in cases where several servers are replicating to the same federating one: each origin server has his own schema, and all is replicated nicely on the central host, thanks to Londiste, as you might have guessed already.

For bare-metal recovery scripts, I'm working on how to change those constants in the constraints, so that pg_dump -s plus some schema tweaking would kick-start a server. Here's a PLpgSQL snippet to do just that:

  FOR rec IN EXECUTE
$s$
SELECT schemaname, tablename, conname, attnames, def
  FROM (
   SELECT n.nspname, c.relname, r.conname,
          (select array_accum(attname)
             from pg_attribute
            where attrelid = c.oid and r.conkey @> array[attnum]) as attnames,
          pg_catalog.pg_get_constraintdef(r.oid, true)
   FROM pg_catalog.pg_constraint r
        JOIN pg_class c on c.oid = r.conrelid
        JOIN pg_namespace n ON n.oid = c.relnamespace
   WHERE r.contype = 'c'
ORDER BY 1, 2, 3
       ) as cons(schemaname, tablename, conname, attnames, def)
WHERE attnames @> array['server']::name[]
$s$
  LOOP
    rec.def := replace(rec.def, 'server = ' || old_id,
                                'server = ' || new_id);

    sql := 'ALTER TABLE ' || rec.schemaname || '.' || rec.tablename
        || ' DROP CONSTRAINT ' || rec.conname;
    RAISE NOTICE '%', sql;
    RETURN NEXT;
    EXECUTE sql;

    sql := 'ALTER TABLE ' || rec.schemaname || '.' || rec.tablename
        || ' ADD ' || rec.def;
    RAISE NOTICE '%', sql;
    RETURN NEXT;
    EXECUTE sql;

  END LOOP;

This relies on the fact that our constraints are on the column server. Why would this be any better than a sed one-liner, would you ask me? I'm fed up with having pseudo-parsing scripts and taking the risk that the simple command will change data I didn't want to edit. I want context aware tools, pretty please, to feel safe.

Otherwise I'd might have gone with pg_dump -s| sed -e 's:\(server =\) 17:\1 18:' but this one-liner already contains too much useless magic for my taste (the space before 17 ain't in the group match to allow for having \1 18 in the right hand side. And this isn't yet parametrized, and there I'll need to talk to the database, as that's were I store the servers name and their id (a bigserial — yes, the constraints are all generated from scripts). I don't want to write an SQL parser and I don't want to play loose, so the PLpgSQL approach is what I'm thinking as the best tool here. Opinionated answers get to my mailbox!

Tags

PostgreSQL plpgsql

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 09 Aug 2010 14:45:00 +0200 http://tapoueh.org/blog/2010/08/09-editing-constants-in-constraints.html debian packaging PostgreSQL extensions http://tapoueh.org/blog/2010/08/06-debian-packaging-postgresql-extensions.html debian packaging PostgreSQL extensions
Friday, August 06 2010, 13:00

 

In trying to help an extension debian packaging effort, I've once again proposed to handle it. That's because I now begin to know how to do it, as you can see in my package overview page at debian QA facility. There's a reason why I proposed myself here, it's that yet another tool of mine is now to be found in debian, and should greatly help extension packaging there. You can already check for the postgresql-server-dev-all package page if you're that impatient!

Back? Ok, so I used to have two main gripes against debian support for PostgreSQL. The first one, which is now feeling alone, is that both project release support policy aren't compatible enough for debian stable to include all currently supported stable PostgreSQL major version. That's very bad that debian stable will only propose one major version, knowing that the support for several of them is in there.

The problem is two fold: first, debian stable has to maintain any distributed package. There's no deprecation policy allowing for droping the ball. So the other side of this coin is that debian developers must take on themselves maintaining included software for as long as stable is not renamed oldstable. And it so happens that there's no debian developer that feels like maintaining end of lined PostgreSQL releases without help from PostgreSQL Core Team. Or, say, without official statement that they would help.

Now, why I don't like this situation is because I'm pretty sure there's very few software development group offering as long and reliable maintenance policy as PostgreSQL is doing, but debian will still happily distribute unknown-maintenance-policy pieces of code in its stable repositories. So the uncertainty excuse is rather poor. And highly frustrating.

Note: you have to admit that the debian stable management model copes very well with all the debian included software. You can't release stable with a new PostgreSQL major version unless each and every package depending on PostgreSQL will actually work with the newer version, and the debian scripts will care for upgrading the cluster. Where it's not working good is when you're using debian for a PostgreSQL server for a proprietary application, which happens quite frequently too.

The consequence of this fact leads to my second main gripe against debian support for PostgreSQL: the extensions. It so happens that the PostgreSQL extensions are developped for supporting several major versions from the same source code. So typically, all you need to do is recompile the extension against the new major version, and there you go.

Now, say debian new stable is coming with 8.4 rather than 8.3 as it used to. You should be able to just build the extensions (like prefix), without changing the source package, nor droping postgresql-8.3-prefix from the distribution on the grounds that 8.3 ain't in debian stable anymore.

I've been ranting a lot about this state of facts, and I finally provided a patch to the postgresql-common debian packaging, which made it into version 110: welcome pg_buildext. An exemple of how to use it can be found in the git branch for prefix, it shows up in debian/pgversions and debian/rules files.

As you can see, the pg_buildext tool allows you to list the PostgreSQL major versions the extension you're packaging supports, and only those that are both in your list and in the current debian supported major version list will get built. pg_buildext will do a VPATH build of your extension, so it's capable of building the same extension for multiple major versions of PostgreSQL. Here's how it looks:

        # build all supported version
        pg_buildext build $(SRCDIR) $(TARGET) "$(CFLAGS)"

        # then install each of them
        for v in `pg_buildext supported-versions $(SRCDIR)`; do \
                dh_install -ppostgresql-$$v-prefix ;\
        done

And the files are to be found in those places:

dim ~/dev/prefix cat debian/postgresql-8.3-prefix.install
debian/prefix-8.3/prefix.so usr/lib/postgresql/8.3/lib
debian/prefix-8.3/prefix.sql usr/share/postgresql/8.3/contrib

dim ~/dev/prefix cat debian/postgresql-8.4-prefix.install
debian/prefix-8.4/prefix.so usr/lib/postgresql/8.4/lib
debian/prefix-8.4/prefix.sql usr/share/postgresql/8.4/contrib

So you still need to maintain debian/pgversions and the postgresql-X.Y-extension.* files, but then a change in debian support for PostgreSQL major versions will be handled automatically (there's a facility to trigger automatic rebuild when necessary).

All this ranting to explain that pretty soon, the extenion's packages that I maintain will no longer have to be patched when dropping a previously supported major version of PostgreSQL. I'm breathing a little better, so thanks a lot Martin!

Tags

PostgreSQL debian Extensions release prefix

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 06 Aug 2010 13:00:00 +0200 http://tapoueh.org/blog/2010/08/06-debian-packaging-postgresql-extensions.html Querying the Catalog to plan an upgrade http://tapoueh.org/blog/2010/08/05-querying-the-catalog-to-plan-an-upgrade.html Querying the Catalog to plan an upgrade
Thursday, August 05 2010, 11:00

 

Some user on IRC was reading the releases notes in order to plan for a minor upgrade of his 8.3.3 installation, and was puzzled about potential needs for rebuilding GIST indexes. That's from the 8.3.5 release notes, and from the 8.3.8 notes you see that you need to consider hash indexes on interval columns too. Now the question is, how to find out if any such beasts are in use in your database?

It happens that PostgreSQL is letting you know those things by querying its system catalogs. That might look hairy at first, but it's very worth getting used to those system tables. You could compare that to introspection and reflexive facilities of some programming languages, except much more useful, because you're reaching all the system at once. But, well, here it goes:

SELECT schemaname, tablename, relname, amname, indexdef
  FROM pg_indexes i
       JOIN pg_class c ON i.indexname = c.relname and c.relkind = 'i'
       JOIN pg_am am ON c.relam = am.oid
 WHERE amname = 'gist';

Now you could replace the WHERE clause with WHERE amname IN ('gist', 'hash') to check both conditions at once. What about pursuing the restriction on the hash indexes rebuild to schedule, as they should only get done to indexes on interval columns. Well let's try it:

SELECT schemaname, tablename, relname as indexname, amname, indclass
  FROM pg_indexes i
       JOIN pg_class c on i.indexname = c.relname and c.relkind = 'i'
       JOIN pg_am am on c.relam = am.oid
       JOIN pg_index x on x.indexrelid = c.oid
 WHERE amname in ('btree', 'gist')
       and schemaname not in ('pg_catalog', 'information_schema');

We're not there yet, because as you notice, the catalogs are somewhat optimized and not always in a normal form. That's good for the system's performance, but it makes querying a bit uneasy. What we want is to get from the indclass column if there's any of them (it's an oidvector) that applies to an interval data type. There's a subtlety here as the index could store interval data even if the column is not of an interval type itself, so we have to find both cases.

Well the subtlety applies after you know what an operator class is: “An operator class defines how a particular data type can be used with an index” is what the CREATE OPERATOR CLASS manual page teaches us. What we need to know here is that an index will talk to an operator class to get to the data type, either the column data type or the index storage one.

SELECT schemaname, tablename, relname as indexname, amname, indclass, opcname, typname
  FROM pg_indexes i
       JOIN pg_class c on i.indexname = c.relname and c.relkind = 'i'
       JOIN pg_am am on c.relam = am.oid
       JOIN pg_index x on x.indexrelid = c.oid
       JOIN pg_opclass o
         on string_to_array(x.indclass::text, ' ')::oid[] @> array[o.oid]::oid[]
       JOIN pg_type t on o.opckeytype = t.oid
WHERE amname = 'hash' and t.typname = 'interval'

UNION ALL

SELECT schemaname, tablename, relname as indexname, amname, indclass, opcname, typname
  FROM pg_indexes i
       JOIN pg_class c on i.indexname = c.relname and c.relkind = 'i'
       JOIN pg_am am on c.relam = am.oid
       JOIN pg_index x on x.indexrelid = c.oid
       JOIN pg_opclass o
         on string_to_array(x.indclass::text, ' ')::oid[] @> array[o.oid]::oid[]
       JOIN pg_type t on o.opcintype = t.oid
WHERE amname = 'hash' and t.typname = 'interval';

Most certainly this query will return no row for you, as hash indexes are not widely used, mainly because they are not crash tolerant. For seeing some results you could remove the amname restriction of course, that would show the query is working, but don't forget to add the restriction back to plan for the upgrade!

But hey, why walking the extra mile here, would you ask me? After all, in the second query we would already have had the information we needed should we added the indexdef column, albeit in a human reader friendly way: the resultset would then contain the CREATE INDEX command you need to issue to build the index from scratch. That would be enough for checking only the catalog, but the extra mile allows you to produce a SQL script to build the indexes that need your attention post upgrade. That last step is left as an exercise for the reader, though.

Tags

PostgreSQL release catalogs

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 05 Aug 2010 11:00:00 +0200 http://tapoueh.org/blog/2010/08/05-querying-the-catalog-to-plan-an-upgrade.html el-get http://tapoueh.org/blog/2010/08/blog/2010/08/04-el-get.html I've been using emacs for a long time, and a long time it took me to consider learning Emacs Lisp. Before that, I didn't trust my level of understanding enough to be comfortable in managing my setup efficiently.

One of the main problems of setting up Emacs is that not only you tend to accumulate so many tricks from EmacsWiki and blog posts that your .emacs has to grow to a full ~/.emacs.d/ directory (starting at ~/.emacs.d/init.el), but also you finally depend on several librairies of code you're not authoring nor maintaining. Let's call them packages.

Some of them will typically be available on ELPA, which allows you to breathe and keep cool. But most of them, let's face it, are not there. Most of the packages I use I tend to get them either from debian (see apt-rdepends for having the complete list of packages that depends on emacs, unfortunately I'm not finding an online version of the tool to link too), or from ELPA, or from their own git repository somewhere. Some of them even I get directly from an obscure website not maintained anymore, but always there when you need them.

Of course, my emacs setup is managed in a private git repository. Some people on #emacs are using git submodules (or was it straight import) for managing external repositories in there, but all I can say is that I frown on this idea. I want an easy canonical list of packages I depend on to run emacs, and I want this documentation to be usable as-is. Enters el-get!

As we're all damn lazy, here's a visual introduction to el-get:

(setq el-get-sources
      '((:name bbdb
               :type git
               :url "git://github.com/barak/BBDB.git"
               :load-path ("./lisp" "./bits")
               :info "texinfo"
               :build ("./configure" "make"))

        (:name magit
               :type git
               :url "http://github.com/philjackson/magit.git"
               :info "."
               :build ("./autogen.sh" "./configure" "make"))

        (:name vkill
               :type http
               :url "http://www.splode.com/~friedman/software/emacs-lisp/src/vkill.el"
               :features vkill)

        (:name yasnippet
               :type git-svn
               :url "http://yasnippet.googlecode.com/svn/trunk/")

        (:name asciidoc         :type elpa)
        (:name dictionary-el    :type apt-get)
        (:name emacs-goodies-el :type apt-get)))

(el-get)

So now you have a pretty good documentation of the packages you want installed, where to get them, and how to install them. For the advanced methods (such as elpa or apt-get), you basically just need the package name. When relying on a bare git repository, you need to give some more information, such as the URL to clone and the build steps if any. Then also what features to require and maybe where to find the texinfo documentation of the package, for automatic inclusion into your local Info menu.

The good news is that not only you now have a solid readable description of all that in a central place, but this very description is all (el-get) needs to do its magic. This command will check that each and every package is installed on your system (in el-get-dir) and if that's not the case, it will actually install it. Then, it will init the packages: that means caring about the load-path, the Info-directory-list (and dir texinfo menu building), the loading of the emacs-lisp files, and finally it will require the features.

Here's a prettyfied ielm session that will serve as a demo:

ELISP> (el-get)
("aspell-en" "aspell-fr" "muse" "dictionary" "htmlize" "bbdb" "google-maps"
"magit" "emms" "nxhtml" "vkill" "xcscope" "yasnippet" "asciidoc"
"auto-dictionary" "css-mode" "gist" "lua-mode" "lisppaste")

All the packages being already installed, it's running fast enough that I won't bother measuring the run time, that seems to be somewhere around one second.

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 04 Aug 2010 22:30:00 +0200 http://tapoueh.org/blog/2010/08/blog/2010/08/04-el-get.html el-get http://tapoueh.org/blog/2010/08/04-el-get.html el-get
Wednesday, August 04 2010, 22:30

I've been using emacs for a long time, and a long time it took me to consider learning Emacs Lisp. Before that, I didn't trust my level of understanding enough to be comfortable in managing my setup efficiently.

One of the main problems of setting up Emacs is that not only you tend to accumulate so many tricks from EmacsWiki and blog posts that your .emacs has to grow to a full ~/.emacs.d/ directory (starting at ~/.emacs.d/init.el), but also you finally depend on several librairies of code you're not authoring nor maintaining. Let's call them packages.

Some of them will typically be available on ELPA, which allows you to breathe and keep cool. But most of them, let's face it, are not there. Most of the packages I use I tend to get them either from debian (see apt-rdepends for having the complete list of packages that depends on emacs, unfortunately I'm not finding an online version of the tool to link too), or from ELPA, or from their own git repository somewhere. Some of them even I get directly from an obscure website not maintained anymore, but always there when you need them.

Of course, my emacs setup is managed in a private git repository. Some people on #emacs are using git submodules (or was it straight import) for managing external repositories in there, but all I can say is that I frown on this idea. I want an easy canonical list of packages I depend on to run emacs, and I want this documentation to be usable as-is. Enters el-get!

As we're all damn lazy, here's a visual introduction to el-get:

(setq el-get-sources
      '((:name bbdb
               :type git
               :url "git://github.com/barak/BBDB.git"
               :load-path ("./lisp" "./bits")
               :info "texinfo"
               :build ("./configure" "make"))

        (:name magit
               :type git
               :url "http://github.com/philjackson/magit.git"
               :info "."
               :build ("./autogen.sh" "./configure" "make"))

        (:name vkill
               :type http
               :url "http://www.splode.com/~friedman/software/emacs-lisp/src/vkill.el"
               :features vkill)

        (:name yasnippet
               :type git-svn
               :url "http://yasnippet.googlecode.com/svn/trunk/")

        (:name asciidoc         :type elpa)
        (:name dictionary-el    :type apt-get)
        (:name emacs-goodies-el :type apt-get)))

(el-get)

So now you have a pretty good documentation of the packages you want installed, where to get them, and how to install them. For the advanced methods (such as elpa or apt-get), you basically just need the package name. When relying on a bare git repository, you need to give some more information, such as the URL to clone and the build steps if any. Then also what features to require and maybe where to find the texinfo documentation of the package, for automatic inclusion into your local Info menu.

The good news is that not only you now have a solid readable description of all that in a central place, but this very description is all (el-get) needs to do its magic. This command will check that each and every package is installed on your system (in el-get-dir) and if that's not the case, it will actually install it. Then, it will init the packages: that means caring about the load-path, the Info-directory-list (and dir texinfo menu building), the loading of the emacs-lisp files, and finally it will require the features.

Here's a prettyfied ielm session that will serve as a demo:

ELISP> (el-get)
("aspell-en" "aspell-fr" "muse" "dictionary" "htmlize" "bbdb" "google-maps"
"magit" "emms" "nxhtml" "vkill" "xcscope" "yasnippet" "asciidoc"
"auto-dictionary" "css-mode" "gist" "lua-mode" "lisppaste")

All the packages being already installed, it's running fast enough that I won't bother measuring the run time, that seems to be somewhere around one second.

Tags

Emacs Muse debian el-get

]]> dim@tapoueh.org (Dimitri Fontaine) Wed, 04 Aug 2010 22:30:00 +0200 http://tapoueh.org/blog/2010/08/04-el-get.html Database Virtual Machines http://tapoueh.org/blog/2010/08/03-database-virtual-machines.html Database Virtual Machines
Tuesday, August 03 2010, 13:30

Today I'm being told once again about SQLite as an embedded database software. That one ain't a database server but a software library that you can use straight into your main program. I'm yet to use it, but it looks like its SQL support is good enough for simple things — and that covers loads of things. I guess read-only cache and configuration storage would be the obvious ones, because it seems that SQLite use cases aren't including mixed concurrency, that is workloads with concurrent readers and writers.

The part that got my full attention is The Virtual Database Engine of SQLite, as this blog title would imply. It seems to be the same idea as what MonetDB calls their MonetDB Assembly Language, and I've been trying to summarize some idea about it in my Next Generation PostgreSQL article.

The main thing is how to further optimize PostgreSQL given what we have. It seems that among the major road blocks in the performance work is how we get the data from disk and to the client. We're still spending so many time in the CPU that the disk bandwidth are not always saturated, and that's a problem. Further thoughts on the full length article, but that's just about a one page section now!

Tags

PostgreSQL

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 03 Aug 2010 13:30:00 +0200 http://tapoueh.org/blog/2010/08/03-database-virtual-machines.html Partitioning: relation size per “group” http://tapoueh.org/blog/2010/07/26-partitioning-relation-size-per-group.html Partitioning: relation size per “group”
Monday, July 26 2010, 17:00

 

This time, we are trying to figure out where is the bulk of the data on disk. The trick is that we're using DDL partitioning, but we want a “nice” view of size per partition set. Meaning that if you have for example a parent table foo with partitions foo_201006 and foo_201007, you would want to see a single category foo containing the accumulated size of all the partitions underneath foo.

Here we go:

select groupe, pg_size_pretty(sum(bytes)::bigint) as size, sum(bytes)
  from (
select relkind as k, nspname, relname, tablename, bytes,
         case when relkind = 'r' and relname ~ '[0-9]{6}$'
              then substring(relname from 1 for length(relname)-7)

              when relkind = 'i' and  tablename ~ '[0-9]{6}$'
              then substring(tablename from 1 for length(tablename)-7)

              else 'core'
          end as groupe
  from (
  select nspname, relname,
         case when relkind = 'i'
              then (select relname
                      from pg_index x
                           join pg_class xc on x.indrelid = xc.oid
                           join pg_namespace xn on xc.relnamespace = xn.oid
                     where x.indexrelid = c.oid
                    )
              else null
           end as tablename,
         pg_size_pretty(pg_relation_size(c.oid)) as relation,
         pg_total_relation_size(c.oid) as bytes,
         relkind
    from pg_class c join pg_namespace n on c.relnamespace = n.oid
   where c.relkind in ('r', 'i')
         and nspname in ('public', 'archive')
         and pg_total_relation_size(c.oid) > 32 * 1024
order by 5 desc
       ) as s
       ) as t
group by 1
order by 3 desc;

Note that by simply removing those last two lines here, you will get a detailed view of the indexes and tables that are taking the most volume on disk at your place.

Now, what about using window functions here so that we get some better detailed view of historic changes on each partition? With some evolution figure in percentage from the previous partition of the same year, accumulated size per partition and per year, yearly sum, you name it. Here's another one you might want to try, ready for some tuning (schema name, table name, etc):

WITH s AS (
  select relname,
         pg_relation_size(c.oid) as size,
         pg_total_relation_size(c.oid) as tsize,
         substring(substring(relname from '[0-9]{6}$') for 4)::bigint as year
    from pg_class c
         join pg_namespace n on n.oid = c.relnamespace
   where c.relkind = 'r'
     -- and n.nspname = 'public'
     -- and c.relname ~ 'stats'
     and substring(substring(relname from '[0-9]{6}$') for 4)::bigint >= 2008
order by relname
),
     sy AS (
  select relname,
         size,
         tsize,
         year,
         (sum(size) over w_year)::bigint as ysize,
         (sum(size) over w_month)::bigint as cumul,
         (lag(size) over (order by relname))::bigint as previous
    from s
  window w_year  as (partition by year),
         w_month as (partition by year order by relname)
),
     syp AS (
  select relname,
         size,
         tsize,
         rank() over (partition by year order by size desc) as rank,
         case when ysize = 0 then ysize
              else round(size / ysize::numeric * 100, 2) end as yp,
         case when previous = 0 then previous
              else round((size / previous::numeric - 1.0) * 100, 2) end as evol,
         cumul,
         year,
         ysize
    from sy
)
  SELECT relname,
         pg_size_pretty(size) as size,
         pg_size_pretty(tsize) as "+indexes",
         evol, yp as "% annuel", rank,
         pg_size_pretty(cumul) as cumul, year,
         pg_size_pretty(ysize) as "yearly sum",
         pg_size_pretty((sum(size) over())::bigint) as total
    FROM syp
ORDER BY relname;

Hope you'll find it useful, I certainly do!

Tags

PostgreSQL

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 26 Jul 2010 17:00:00 +0200 http://tapoueh.org/blog/2010/07/26-partitioning-relation-size-per-group.html dim-switch-window.el: fixes http://tapoueh.org/blog/2010/07/blog/2010/07/26-dim-switch-windowel-fixes.html Thanks to amazing readers of planet emacsen, two annoyances of switch-window.el have already been fixed! The first is that handling of C-g isn't exactly an option after all, and the other is that you want to avoid the buffer creation in the simple cases (1 or 2 windows only), because it's the usual case.

I've received code to handle the second case, that I mostly merged. Thanks a lot guys, the new version is on emacswiki already!

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 26 Jul 2010 11:55:00 +0200 http://tapoueh.org/blog/2010/07/blog/2010/07/26-dim-switch-windowel-fixes.html dim-switch-window.el: fixes http://tapoueh.org/blog/2010/07/26-dim-switch-windowel-fixes.html dim-switch-window.el: fixes
Monday, July 26 2010, 11:55

Thanks to amazing readers of planet emacsen, two annoyances of switch-window.el have already been fixed! The first is that handling of C-g isn't exactly an option after all, and the other is that you want to avoid the buffer creation in the simple cases (1 or 2 windows only), because it's the usual case.

I've received code to handle the second case, that I mostly merged. Thanks a lot guys, the new version is on emacswiki already!

Tags

Emacs switch-window

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 26 Jul 2010 11:55:00 +0200 http://tapoueh.org/blog/2010/07/26-dim-switch-windowel-fixes.html dim-switch-window.el http://tapoueh.org/blog/2010/07/blog/2010/07/25-dim-switch-windowel.html So it's Sunday and I'm thinking I'll get into el-get sometime later. Now is the time to present dim-switch-window.el which implements a visual C-x o. I know of only one way to present a visual effect, and that's with a screenshot:

So as you can see, it's all about showing a big number in each window, tweaking each window's name, and waiting till the user press one of the expected key — or timeout and stay on the same window as before C-x o. When there's only 1 or 2 windows displayed, though, the right thing happen and you see no huge number (in the former case, nothing happens, in the latter, focus moves to the other window).

The code for that can be found on emacswiki under the name switch-window.el. Hope you'll find it useful!

]]> dim@tapoueh.org (Dimitri Fontaine) Sun, 25 Jul 2010 13:25:00 +0200 http://tapoueh.org/blog/2010/07/blog/2010/07/25-dim-switch-windowel.html dim-switch-window.el http://tapoueh.org/blog/2010/07/25-dim-switch-windowel.html dim-switch-window.el
Sunday, July 25 2010, 13:25

So it's Sunday and I'm thinking I'll get into el-get sometime later. Now is the time to present dim-switch-window.el which implements a visual C-x o. I know of only one way to present a visual effect, and that's with a screenshot:

So as you can see, it's all about showing a big number in each window, tweaking each window's name, and waiting till the user press one of the expected key — or timeout and stay on the same window as before C-x o. When there's only 1 or 2 windows displayed, though, the right thing happen and you see no huge number (in the former case, nothing happens, in the latter, focus moves to the other window).

The code for that can be found on emacswiki under the name switch-window.el. Hope you'll find it useful!

Tags

Emacs el-get switch-window

]]> dim@tapoueh.org (Dimitri Fontaine) Sun, 25 Jul 2010 13:25:00 +0200 http://tapoueh.org/blog/2010/07/25-dim-switch-windowel.html ClusterSSH gets dsh support http://tapoueh.org/blog/2010/07/blog/2010/07/23-clusterssh-gets-dsh-support.html If you don't know about ClusterSSH, it's a project that builds on M-x term and ssh to offer a nice and simple way to open remote terminals. It's available in ELPA and developed at github cssh repository.

The default binding is C-= and asks for the name of the server to connect to, in the minibuffer, with completion. The host list used for the completion comes from tramp and is pretty complete, all the more if you've setup ~/.ssh/config with HashKnownHosts no.

So the usual way to use cssh.el would be to just open a single remote connection at a time. But of course you can open as many as you like, and you get them all in a mosaic of term in your emacs frame, with an input window at the bottom to control them all. There were two ways to get there, either opening all remote hosts whose name is matching a given regexp, that would be using C-M-= or getting to IBuffer and marking there the existing remote terms you want to control all at once then use C-=.

Well I've just added another mode of operation by supporting enhanced dsh group files. In such files, you're supposed to have a remote host name per line and that's it. We've added support for line containing @group kind of lines so that you can include another group easily. To use the facility, either open your ~/.dsh/group directory in dired and type C-= when on the right line, or simply use the global C-= you already know and love. Then, type @ and complete to any existing group found in your cssh-dsh-path (it defaults to the right places, so chances are you will never have to edit this one). And that's it, Emacs will open one term per remote host you have in the dsh group you just picked. With a *cssh* controler window, too.

Coming next, how I solved my init.el dependancies burden thanks to el-get!

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 23 Jul 2010 22:20:00 +0200 http://tapoueh.org/blog/2010/07/blog/2010/07/23-clusterssh-gets-dsh-support.html ClusterSSH gets dsh support http://tapoueh.org/blog/2010/07/23-clusterssh-gets-dsh-support.html ClusterSSH gets dsh support
Friday, July 23 2010, 22:20

If you don't know about ClusterSSH, it's a project that builds on M-x term and ssh to offer a nice and simple way to open remote terminals. It's available in ELPA and developed at github cssh repository.

The default binding is C-= and asks for the name of the server to connect to, in the minibuffer, with completion. The host list used for the completion comes from tramp and is pretty complete, all the more if you've setup ~/.ssh/config with HashKnownHosts no.

So the usual way to use cssh.el would be to just open a single remote connection at a time. But of course you can open as many as you like, and you get them all in a mosaic of term in your emacs frame, with an input window at the bottom to control them all. There were two ways to get there, either opening all remote hosts whose name is matching a given regexp, that would be using C-M-= or getting to IBuffer and marking there the existing remote terms you want to control all at once then use C-=.

Well I've just added another mode of operation by supporting enhanced dsh group files. In such files, you're supposed to have a remote host name per line and that's it. We've added support for line containing @group kind of lines so that you can include another group easily. To use the facility, either open your ~/.dsh/group directory in dired and type C-= when on the right line, or simply use the global C-= you already know and love. Then, type @ and complete to any existing group found in your cssh-dsh-path (it defaults to the right places, so chances are you will never have to edit this one). And that's it, Emacs will open one term per remote host you have in the dsh group you just picked. With a *cssh* controler window, too.

Coming next, how I solved my init.el dependancies burden thanks to el-get!

Tags

Emacs el-get cssh

]]> dim@tapoueh.org (Dimitri Fontaine) Fri, 23 Jul 2010 22:20:00 +0200 http://tapoueh.org/blog/2010/07/23-clusterssh-gets-dsh-support.html Emacs and PostgreSQL http://tapoueh.org/blog/2010/07/blog/2010/07/22-emacs-and-postgresql.html Those are my two all times favorite Open Source Software. Or Free Software in the GNU sense of the world, as both the BSD and the GPL are labeled free there. Even if I prefer the The Debian Free Software Guidelines as a global definition and the WTFPL license. But that's a digression.

I think that Emacs and PostgreSQL do share a lot in common. I'd begin with the documentation, which quality is amazing for both projects. Then of course the extensibility with Emacs Lisp on the one hand and catalog-driven operations on the other hand. Whether you're extending Emacs or PostgreSQL you'll find that it's pretty easy to tweak the system while it's running. The other comparison points are less important, like the fact the both the systems get about the same uptime on my laptop (currently 13 days, 23 hours, 57 minutes, 10 seconds).

So of course I'm using Emacs to edit PostgreSQL .sql files, including stored procedures. And it so happens that line numbering in plpgsql is not as straightforward as one would naively think, to the point that we'd like to have better tool support there. So I've extended Emacs linum-mode minor mode to also display the line numbers as computed per PostgreSQL, and here's what it looks like:

Now, here's also the source code, pgsql-linum-format. Hope you'll enjoy!

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 22 Jul 2010 09:30:00 +0200 http://tapoueh.org/blog/2010/07/blog/2010/07/22-emacs-and-postgresql.html Emacs and PostgreSQL http://tapoueh.org/blog/2010/07/22-emacs-and-postgresql.html Emacs and PostgreSQL
Thursday, July 22 2010, 09:30

Those are my two all times favorite Open Source Software. Or Free Software in the GNU sense of the world, as both the BSD and the GPL are labeled free there. Even if I prefer the The Debian Free Software Guidelines as a global definition and the WTFPL license. But that's a digression.

I think that Emacs and PostgreSQL do share a lot in common. I'd begin with the documentation, which quality is amazing for both projects. Then of course the extensibility with Emacs Lisp on the one hand and catalog-driven operations on the other hand. Whether you're extending Emacs or PostgreSQL you'll find that it's pretty easy to tweak the system while it's running. The other comparison points are less important, like the fact the both the systems get about the same uptime on my laptop (currently 13 days, 23 hours, 57 minutes, 10 seconds).

So of course I'm using Emacs to edit PostgreSQL .sql files, including stored procedures. And it so happens that line numbering in plpgsql is not as straightforward as one would naively think, to the point that we'd like to have better tool support there. So I've extended Emacs linum-mode minor mode to also display the line numbers as computed per PostgreSQL, and here's what it looks like:

Now, here's also the source code, pgsql-linum-format. Hope you'll enjoy!

Tags

PostgreSQL Emacs debian plpgsql pgsql-linum-format

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 22 Jul 2010 09:30:00 +0200 http://tapoueh.org/blog/2010/07/22-emacs-and-postgresql.html Background writers http://tapoueh.org/blog/2010/07/19-background-writers.html Background writers
Monday, July 19 2010, 16:30

There's currently a thread on hackers about bg worker: overview and a series of 6 patches. Thanks a lot Markus! This is all about generalizing a concept already in use in the autovacuum process, where you have an independent subsystem that require having an autonomous daemon running and able to start its own workers.

I've been advocating about generalizing this concept for awhile already, in order to have postmaster able to communicate to subsystems when to shut down and start and reload, etc. Some external processes are only external because there's no need to include them by default in to the database engine, not because there's no sense to having them in there.

So even if Markus work is mainly about generalizing autovacuum so that he has a coordinator to ask for helper backends to handle broadcasting of writesets for Postgres-R, it still could be a very good first step towards something more general. What I'd like to see the generalization handle are things like PGQ, or the pgagent scheduler. In some cases, pgbouncer too.

What we're missing there is an API for everybody to be able to extend PostgreSQL with its own background processes and workers. What would such a beast look like? I have some preliminary thoughts about this in my Next Generation PostgreSQL article, but that's still early thoughts. The main idea is to steal as much as sensible from Erlang Generic Supervisor Behaviour, and maybe up to its Generic Finite State Machines behavior. In the Erlang world, a behavior is a generic process.

The FSM approach would allow for any user daemon to provide an initial state and register functions that would do some processing then change the state. My feeling is that if those functions are exposed at the SQL level, then you can talk to the daemon from anywhere (the Erlang ideas include a globally —cluster wide— unique name). Of course the goal would be to provide an easy way for the FSM functions to have a backend connected to the target database handle the work for it, or be able to connect itself. Then we'd need something else here, a way to produce events based on the clock. I guess relying on SIGALRM is a possibility.

I'm not sure about how yet, but I think getting back in consultancy after having opened 2ndQuadrant France has some influence on how I think about all that. My guess is that those blog posts are a first step on a nice journey!

Tags

PostgreSQL

]]> dim@tapoueh.org (Dimitri Fontaine) Mon, 19 Jul 2010 16:30:00 +0200 http://tapoueh.org/blog/2010/07/19-background-writers.html Logs analysis http://tapoueh.org/blog/2010/07/13-logs-analysis.html Logs analysis
Tuesday, July 13 2010, 14:15

Nowadays to analyze logs and provide insights, the more common tool to use is pgfouine, which does an excellent job. But there has been some improvements in logs capabilities that we're not benefiting from yet, and I'm thinking about the CSV log format.

So the idea would be to turn pgfouine into a set of SQL queries against the logs themselves once imported into the database. Wait. What about having our next PostgreSQL version, which is meant (I believe) to include CSV support in SQL/MED, to directly expose its logs as a system view?

A good thing would be to expose that as a ddl-partitioned table following the log rotation scheme as setup in postgresql.conf, or maybe given in some sort of a setup, in order to support logrotate users. At least some facilities to do that would be welcome, and I'm not sure plain SQL/MED is that when it comes to source partitioning.

Then all that remains to be done is a set of SQL queries and some static or dynamic application to derive reports from there.

This is yet again an idea I have in mind but don't have currently time to explore myself, so I talk about it here in the hope that others will share the interest. Of course, now that I work at 2ndQuadrant, you can make it so that we consider the idea in more details, up to implementing and contributing it!

Tags

PostgreSQL

]]> dim@tapoueh.org (Dimitri Fontaine) Tue, 13 Jul 2010 14:15:00 +0200 http://tapoueh.org/blog/2010/07/13-logs-analysis.html Using indexes as column store? http://tapoueh.org/blog/2010/07/08-using-indexes-as-column-store.html Using indexes as column store?
Thursday, July 08 2010, 11:15

 

There's a big trend nowadays about using column storage as opposed to what PostgreSQL is doing, which would be row storage. The difference is that if you have the same column value in a lot of rows, you could get to a point where you have this value only once in the underlying storage file. That means high compression. Then you tweak the executor to be able to load this value only once, not once per row, and you win another huge source of data traffic (often enough, from disk).

Well, it occurs to me that maybe we could have column oriented storage support without adding any new storage facility into PostgreSQL itself, just using in new ways what we already have now. Column oriented storage looks somewhat like an index, where any given value is meant to appear only once. And you have links to know where to find the full row associated in the main storage.

There's a work in progress to allow for PostgreSQL to use indexes on their own, without having to get to the main storage for checking the visibility. That's known as the Visibility Map, which is still only a hint in released versions. The goal is to turn that into a crash-safe trustworthy source in the future, so that we get covering indexes. That means we can use an index and skip getting to the full row in main storage and get the visibility information there.

Now, once we have that, we could consider using the indexes in more queries. It could be a win to get the column values from the index when possible and if you don't output more columns from the heap, return the values from there. Scanning the index only once per value, not once per row.

There's a little more though on the point in the Next Generation PostgreSQL article I've been referencing already, should you be interested.

Tags

PostgreSQL release

]]> dim@tapoueh.org (Dimitri Fontaine) Thu, 08 Jul 2010 11:15:00 +0200 http://tapoueh.org/blog/2010/07/08-using-indexes-as-column-store.html MVCC in the Cloud http://tapoueh.org/blog/2010/07/06-mvcc-in-the-cloud.html MVCC in the Cloud
Tuesday, July 06 2010, 10:50

At CHAR(10) Markus had a talk about Using MVCC for Clustered Database Systems and explained how Postgres-R does it. The scope of his project is to maintain a set of database servers in the same state, eventually.

Now, what does it mean to get "In the Cloud"? Well there are more than one answer I'm sure, mine would insist on including this "Elasticity" bit. What I mean here is that it'd be great to be able to add or lose nodes and stay online. Granted, that what's Postgres-R is providing. Does that make it ready for the "Cloud"? Well it happens so that I don't think so.

Once you have elasticity, you also want scalability. That could mean lots of thing, and Postgres-R already provides a great deal of it, at the connect and reads level: you can do your business unlimited on any node, the others will eventua