Skytools 3.1.3 enters debian

First news is that Skytools 3.1.3 has been entering debian today (I hope that by the time you reach that URL, it's been updated to show information according to the news here, but I might be early). As there's current a debian freeze to release wheezy (and you can help squash some bugs), this version is only getting uploaded to experimental for now. Thanks to the tireless work of Christoph Berg though, this version is already available from apt.postgresql.org.

Upgrading to PGQ 3

The other news is that I've been testing live upgrade scenario where we want to upgrade from PGQ to PGQ3, and it works pretty well, and it's quite simple to achieve too. Here's how.

So the first thing is to shut down the current ticker process. Then we install the new packages, assuming that you did follow the step in the wiki pointed above, please go read apt.postgresql.org again now if needs be.

pgqadm.py ticker.ini -s
sudo apt-get install postgresql-9.1-pgq3 skytools3-ticker skytools3

The ticker is not running anymore, we have the right version of the software installed. Next step is to upgrade the database parts of PGQ:

psql -f /usr/share/skytools3/pgq.upgrade_2.1_to_3.0.sql ...
psql -1 -f /usr/share/postgresql/9.1/contrib/pgq.upgrade.sql ...

Of course replace those ... with options such as your actual connection string. I tend to always add -vON_ERROR_STOP=1 to all these commands, so that I don't depend on having the right .psqlrc on the particular server I'm connected to. Also remember that if you want to do that for more than one database, you need to actually run that pair of commands for each of them.

Now it's time to restart the new ticker. The main changes from the previous one is that it is now a C program called pgqd that knows how to tick for any number of databases, so that you only have to have one instance around per cluster now.

sudo /etc/init.d/skytools3 start
tail -f /var/log/skytools/pgqd.log

Those two commands are taking for granted that you did prepare the pgqd setup the debian and skytools way, by adding your config in /etc/skytools3/pgqd.ini and editing /etc/skytools.ini accordingly, so that it's automatically taken into account at machine boot.

Note that I did actually exercised the procedure above while running a pgbench test replicated with londiste. Of course the replication has been lagging a little while no ticker was running, and then it catched-up as fast as it could, in that case:

INFO {count: 245673, ignored: 0, duration: 422.104366064}

Happy Hacking!

So if you have any batch processing needs, remember to consider what PGQ has to offer. And yes if you're running some cron job to compute things out of the database for you, you are doing some batch processing.

We're now thinking to support the i386 and amd64 architectures for lenny, squeeze, wheezy and sid, and also for maverick and natty, maybe oneiric too while at it.

It's still the very beginning of the effort, and it was triggered by the decision to move sid to 9.1. While it's a good decision in itself, I still hate to have to pick only one PostgreSQL version per debian stable release when we have all the technical support we need to be able to support all stable releases that upstream is willing to maintain. If you've been living under a rock, or if you couldn't care less about debian choices, the problem here for debian is ensuring security (and fixes) updates for PostgreSQL — they promise they will handle the job just fine in the social contract, and don't want to have to it without support from PostgreSQL if a debian stable release contains a deprecated PostgreSQL version.

That opens the door for PostgreSQL community to handle the packaging of its solutions as a service to its debian users. We intend to open with support for 8.4, 9.0 and 9.1, and maybe 8.3 too, as Christoph Berg is doing good progress on this front. See, it's teamwork here!

We still have more work to do, and setting up the build environment so that we are able to provide the packages for so much targets will indeed be interesting. Getting there, a step after another.

Rather than talking about what pgfincore is all about (A set of functions to manage pages in memory from PostgreSQL), I will talk about its packaging and support as a debian package. Here's the first example of a modern multi-version packaging I have to offer. pgfincore packaging supports building for 8.4 and 9.0 and 9.1 out of the box, even if the only binary you'll find in debian sid is the 9.0 one, as you can check on the pgfincore debian source package page.

Also, this is the first package I've done properly using the newer version of debhelper, which make the debian/rules file easier than ever. Let's have a look at it:

SRCDIR = $(CURDIR)
TARGET = $(CURDIR)/debian/pgfincore-%v
PKGVERS = $(shell dpkg-parsechangelog | awk -F '[:-]' '/^Version:/ { print substr($$2, 2) }')
EXCLUDE = --exclude-vcs --exclude=debian

include /usr/share/postgresql-common/pgxs_debian_control.mk

override_dh_auto_clean: debian/control
        pg_buildext clean $(SRCDIR) $(TARGET) "$(CFLAGS)"
        dh_clean

override_dh_auto_build:
        # build all supported version
        pg_buildext build $(SRCDIR) $(TARGET) "$(CFLAGS)"

override_dh_auto_install:
        # then install each of them
        for v in `pg_buildext supported-versions $(SRCDIR)`; do \
                dh_install -ppostgresql-$$v-pgfincore ;\
        done

orig: clean
        cd .. && tar czf pgfincore_$(PKGVERS).orig.tar.gz $(EXCLUDE) pgfincore

%:
        dh $@

The debian/rules file is known to be the corner stone of your debian packaging, and usually is the most complex part of it. It's a Makefile at its heart, and we can see that thanks to the debhelper magic it's not that complex to maintain anymore.

Then, this file is using support from a bunch of helpers command, each of them comes with its own man page and does a little part of the work. The overall idea around debhelper is that what it does covers 90% of the cases around, and it's not aiming for more. You have to override the parts where it defaults to being wrong.

Here for example the build system has to produce files for all three supported versions of PostgreSQL, which means invoking the same build system three time with some changes in the environment (mainly setting the PG_CONFIG variable correctly). But even for that we have a debian facility, that comes in the package postgresql-server-dev-all, called pg_buildext. As long as your extension build system is VPATH friendly, it's all automated.

Please read that last sentence another time. VPATH is the thing that allows Make to find your source tree somewhere in the system, not in the current working directory. That allows you to cleanly build the same sources in different build locations, which is exactly what we need here, and is cleanly supported by PGXS, the PostgreSQL Extension Building Infrastructure.

Which means that the main Makefile of pgfincore had to be simplified, and the code layout too. Some advances Make features such as $(wildcard ...) and all will not work here. See what we got at the end:

ifndef VPATH
SRCDIR = .
else
SRCDIR = $(VPATH)
endif

EXTENSION    = pgfincore
EXTVERSION   = $(shell grep default_version $(SRCDIR)/$(EXTENSION).control | \
               sed -e "s/default_version[[:space:]]*=[[:space:]]*'\([^']*\)'/\1/")

MODULES      = $(EXTENSION)
DATA         = sql/pgfincore.sql sql/uninstall_pgfincore.sql
DOCS         = doc/README.$(EXTENSION).rst

PG_CONFIG    = pg_config

PG91         = $(shell $(PG_CONFIG) --version | grep -qE "8\.|9\.0" && echo no || echo yes)

ifeq ($(PG91),yes)
all: pgfincore--$(EXTVERSION).sql

pgfincore--$(EXTVERSION).sql: sql/pgfincore.sql
        cp $< $@

DATA        = pgfincore--unpackaged--$(EXTVERSION).sql pgfincore--$(EXTVERSION).sql
EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql
endif

PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

deb:
        dh clean
        make -f debian/rules orig
        debuild -us -uc -sa

No more Make magic to find source files. Franckly though, when your sources are 1 c file and 2 sql files, you don't need that much magic anyway. You just want to believe that a single generic Makefile will happily build any project you throw at it, only requiring minor adjustment. Well, the reality is that you might need some more little adjustments if you want to benefit from VPATH building, and having the binaries for 8.4 and 9.0 and 9.1 built seemlessly in a simple loop. Like we have here for pgfincore.

Now the Makefile still contains a little bit of magic, in order to parse the extension version number from its control file and produce a script named accordingly. Then you'll notice a difference between the postgresql-9.1-pgfincore.install file and the postgresql-9.0-pgfincore.install. We're just not shipping the same files:

debian/pgfincore-9.0/pgfincore.so usr/lib/postgresql/9.0/lib
sql/pgfincore.sql usr/share/postgresql/9.0/contrib
sql/uninstall_pgfincore.sql usr/share/postgresql/9.0/contrib

As you can see here:

debian/pgfincore-9.1/pgfincore.so usr/lib/postgresql/9.1/lib
debian/pgfincore-9.1/pgfincore*.sql usr/share/postgresql/9.1/extension
sql/pgfincore--unpackaged--1.0.sql usr/share/postgresql/9.1/extension

So, now that we uncovered all the relevant magic, packaging and building your next extension so that it supports as many PostgreSQL major releases as you need to will be that easy.

For reference, you might need to also tweak /usr/share/postgresql-common/supported-versions so that it allows you to build for all those versions you claim to support in the debian/pgversions file.

$ sudo dpkg-divert \
--divert /usr/share/postgresql-common/supported-versions.distrib \
--rename /usr/share/postgresql-common/supported-versions

$ cat /usr/share/postgresql-common/supported-versions
#! /bin/bash

dpkg -l postgresql-server-dev-* \
| awk -F '[ -]' '/^ii/ && ! /server-dev-all/ {print $6}'

All of this will come pretty handy when we finally sit down and work on a way to provide binary packages for PostgreSQL and its extensions, and all supported versions of those at that. This very project is not dead, it's just sleeping some more.

Well, I had to do it for the ip4r contribution, and I wanted the following to happen:

dpkg-deb: building package `postgresql-8.3-ip4r' ...
dpkg-deb: building package `postgresql-8.4-ip4r' ...
dpkg-deb: building package `postgresql-9.0-ip4r' ...
dpkg-deb: building package `postgresql-9.1-ip4r' ...

And here's a simple enough way to achieve that. First, you have to get your packaging ready the usual way, and to install the build dependencies. Then realizing that /usr/share/postgresql-common/supported-versions from the latest postgresql-common package will only return 8.3 in lenny (yes, I'm doing some backporting here), we have to tweak it.

$ dpkg -l postgresql-server-dev-* | awk '/^ii/ {print $2}'
postgresql-server-dev-8.3
postgresql-server-dev-8.4
postgresql-server-dev-9.0
postgresql-server-dev-9.1
postgresql-server-dev-all

$ sudo dpkg-divert \
--divert /usr/share/postgresql-common/supported-versions.distrib \
--rename /usr/share/postgresql-common/supported-versions

$ cat /usr/share/postgresql-common/supported-versions
#! /bin/bash

dpkg -l postgresql-server-dev-* \
| awk -F '[ -]' '/^ii/ && ! /server-dev-all/ {print $6}'

Now we are allowed to build our extension for all those versions, so we add 9.1 to the debian/pgversions file. And debuild will do the right thing now, thanks to pg_buildext from postgresql-server-dev-all.

The problem we face is that the built is not an extension as in 9.1, so things like \dx in psql and CREATE EXTENSION will not work out of the box. First, we need a control file. Then we need to remove the transaction control from the install script (here, ip4r.sql), and finally, this script needs to be called ip4r--1.05.sql. Here's how I did it:

$ cat ip4r.control
comment = 'IPv4 and IPv4 range index types'
default_version = '1.05'
relocatable = yes

$ cat debian/postgresql-9.1-ip4r.install
debian/ip4r-9.1/ip4r.so usr/lib/postgresql/9.1/lib
ip4r.control usr/share/postgresql/9.1/extension
debian/ip4r-9.1/ip4r.sql usr/share/postgresql/9.1/extension

$ cat debian/postgresql-9.1-ip4r.links
usr/share/postgresql/9.1/extension/ip4r.sql usr/share/postgresql/9.1/extension/ip4r--1.05.sql

Be careful not to forget to remove any and all BEGIN; and COMMIT; lines from the ip4r.sql file, which meant that I also removed support for Rtree, which is not relevant for modern versions of PostgreSQL saith the script (post 8.2). That means I'm not publishing this very work yet, but I wanted to share the debian/postgresql-9.1-extension.links idea.

Notice that I didn't change anything about the .sql.in make rule, so I didn't have to use the support for module_pathname in the control file.

Now, after the usual debuild step, I can just sudo debi to install all the just build packages and CREATE EXTENSION will run fine. And in 9.0 you get the old way to install it, but it still works:

$ psql -U postgres --cluster 9.0/main -1 \
-f /usr/share/postgresql/9.0/contrib/ip4r.sql
<lots of chatter>

$ psql -U postgres --cluster 9.1/main -c 'create extension ip4r;'
CREATE EXTENSION

That's it :)

This year's pgcon hot topics, for me, have been centered around a better grasp at SSI and DDL Triggers. Having those beasts in PostgreSQL would allow for auditing, finer privileges management and some more automated replication facilities. Imagine that ALTER TABLE is able to fire a trigger, provided by Londiste or Slony, that will do what's needed on the cluster by itself. That would be awesome, wouldn't it?

At CHAR 11 I'll be talking about Skytools 3. You know I've been working on its debian packaging, now is the time to review the documentation and make there something as good looking as the monitoring system are...

Well, expect some news and a nice big picture diagram overview soon, if work schedule leaves me anytime that's what I want to be working on now.

Let's take an example here, with the prefix extension. This one too will need some love, but is still behind on my spare time todo list, sorry about that. So, in the prefix.sql.in we read

  CREATE OR REPLACE FUNCTION prefix_range_in(cstring)
  RETURNS prefix_range
  AS 'MODULE_PATHNAME'
  LANGUAGE 'C' IMMUTABLE STRICT;

Two things are to change here. First, the PostgreSQL backend will understand just fine if you just say AS '$libdir/prefix'. So you have to know in the sql script the name of the shared object library, but if you do, you can maintain directly a prefix.sql script instead.

The advantage is that you now can avoid a compatibility problem when you want to support PostgreSQL from 8.2 to 9.1 in your extension (older than that and it's no longer supported). You directly ship your script.

For compatibility, you could also use the control file module_pathname property. But for 9.1 you then have to add a implicit Make rule so that the script is derived from your .sql.in. And as you are managing several scripts — so that you can handle versioning and upgrades — it can get hairy (hint, you need to copy prefix.sql as prefix--1.1.1.sql, then change its name at next revision, and think about upgrade scripts too). The module_pathname facility is better to keep for when managing more than a single extension in the same directory, like the SPI contrib is doing.

Sure, maintaining an extension that targets both antique releases of PostgreSQL and CREATE EXTENSION super-powered one(s) (not yet released) is a little more involved than that. We'll get back to that, as some people are still pioneering the movement.

On my side, I'm working with some debian developer on how to best manage the packaging of those extensions, and this work could end up as a specialized policy document and a coordinated team of maintainers for all things PostgreSQL in debian. This will also give some more steam to the PostgreSQL effort for debian packages: the idea is to maintain packages for all supported version (from 8.2 up to soon 9.1), something debian itself can not commit to.

What you'll find is a very simple 128 lines Emacs user init file, based on el-get for external packages. A not so random selection of those is used, here's the list when you hide some details:

 '(el-get                       ; el-get is self-hosting
   escreen                      ; screen for emacs, C-\ C-h
   php-mode-improved            ; if you're into php...
   psvn                         ; M-x svn-status
   switch-window                ; takes over C-x o
   auto-complete                ; complete as you type with overlays
   emacs-goodies-el             ; the debian addons for emacs
   yasnippet                    ; powerful snippet mode
   zencoding-mode               ; http://www.emacswiki.org/emacs/ZenCoding
   (:name buffer-move           ; move buffers around in windows
   (:name smex                  ; a better (ido like) M-x
   (:name magit                 ; git meet emacs, and a binding
   (:name goto-last-change      ; move pointer back to last change

Another interresting thing to note in this kicker is a choice of some key bindings that are rather unusual (yet) I guess.

(global-set-key (kbd "C-x C-b") 'ido-switch-buffer)
(global-set-key (kbd "C-x C-c") 'ido-switch-buffer)
(global-set-key (kbd "C-x B") 'ibuffer)

Yes, you see that I've rebound C-x C-c to switching buffers. That key is really easy to use and I don't think that M-x kill-emacs deserves it. Keys that are so easy to use should be kept for frequent actions, and quiting emacs is a once-a-day to once-a-month action here. And you can still quit from the window manager button or from the menu or from M-x.

Also Mac users are not left behind, you will see some settings that either are adapted to the system (like choosing another font, keep displaying the menu-bar or not installing the darkish tango-color-mode on this system, where it renders poorly in my opinion), as you can see here:

(if (string-match "apple-darwin" system-configuration)
    (set-face-font 'default "Monaco-13")
  (set-frame-font "Monospace-10"))

(when (string-match "apple-darwin" system-configuration)
  (setq mac-allow-anti-aliasing t)
  (setq mac-command-modifier 'meta)
  (setq mac-option-modifier 'none))

So all in all, I don't expect this emacs-kicker to please everyone, but I expect it to be simple and rich enough (thanks to el-get), and it should be a good kick start that's easy to adapt.

If you want to try it without installing it it's very easy to do so. Just clone the git repository then start an Emacs that will use this. For example that could be, using the excellent Emacs For MacOSX:

 $ /Applications/Emacs.app/Contents/MacOS/Emacs -Q -l init.el

I hope some readers will find it useful! :)

What you'll find is a very simple 128 lines Emacs user init file, based on el-get for external packages. A not so random selection of those is used, here's the list when you hide some details:

 '(el-get                       ; el-get is self-hosting
   escreen                      ; screen for emacs, C-\ C-h
   php-mode-improved            ; if you're into php...
   psvn                         ; M-x svn-status
   switch-window                ; takes over C-x o
   auto-complete                ; complete as you type with overlays
   emacs-goodies-el             ; the debian addons for emacs
   yasnippet                    ; powerful snippet mode
   zencoding-mode               ; http://www.emacswiki.org/emacs/ZenCoding
   (:name buffer-move           ; move buffers around in windows
   (:name smex                  ; a better (ido like) M-x
   (:name magit                 ; git meet emacs, and a binding
   (:name goto-last-change      ; move pointer back to last change

Another interresting thing to note in this kicker is a choice of some key bindings that are rather unusual (yet) I guess.

(global-set-key (kbd "C-x C-b") 'ido-switch-buffer)
(global-set-key (kbd "C-x C-c") 'ido-switch-buffer)
(global-set-key (kbd "C-x B") 'ibuffer)

Yes, you see that I've rebound C-x C-c to switching buffers. That key is really easy to use and I don't think that M-x kill-emacs deserves it. Keys that are so easy to use should be kept for frequent actions, and quiting emacs is a once-a-day to once-a-month action here. And you can still quit from the window manager button or from the menu or from M-x.

Also Mac users are not left behind, you will see some settings that either are adapted to the system (like choosing another font, keep displaying the menu-bar or not installing the darkish tango-color-mode on this system, where it renders poorly in my opinion), as you can see here:

(if (string-match "apple-darwin" system-configuration)
    (set-face-font 'default "Monaco-13")
  (set-frame-font "Monospace-10"))

(when (string-match "apple-darwin" system-configuration)
  (setq mac-allow-anti-aliasing t)
  (setq mac-command-modifier 'meta)
  (setq mac-option-modifier 'none))

So all in all, I don't expect this emacs-kicker to please everyone, but I expect it to be simple and rich enough (thanks to el-get), and it should be a good kick start that's easy to adapt.

If you want to try it without installing it it's very easy to do so. Just clone the git repository then start an Emacs that will use this. For example that could be, using the excellent Emacs For MacOSX:

 $ /Applications/Emacs.app/Contents/MacOS/Emacs -Q -l init.el

I hope some readers will find it useful! :)

Here's what the skytools3 packaging is going to look like:

skytools3              Skytool's replication and queuing
python-pgq3            Skytool's PGQ python library
python-skytools3       python scripts framework for skytools
skytools-ticker3       PGQ ticker daemon service
skytools-walmgr3       high-availability archive and restore commands
postgresql-8.4-pgq3    PGQ server-side code (C module for PostgreSQL)
postgresql-9.0-pgq3    PGQ server-side code (C module for PostgreSQL)

This split is needed so that you can install your daemons (we call them consumers) on separate machines than where you run PostgreSQL. But for the walmgr part, it makes no sense to install it if you don't have a local PostgreSQL service, as it's providing archive and restore commands. Then the ticker, you're free to run it on any machine really, so just package it this way (in skytools3 the ticker is written in C and does not depend on the python framework any more).

What you can't see here yet is the new goodies that wraps it as a quality debian package. A new skytools user is created for you when you install the skytools3 package (which contains the services), along with a skeleton file /etc/skytools.ini and a user directory /etc/skytools/. Put in there your services configuration file, and register those service in the /etc/skytools.ini file itself. Then they will get cared about in the init sequence at startup and shutdown of your server.

The services will run under the skytools system user, and will default to put their log into /var/log/skytools/. The pidfile will get into /var/run/skytools/. All integrated, automated.

Next big TODO is about documentation, reviewing it and polishing it, and I think that skytools3 will then get ready for public release. Yes, you read it right, it's happening this very year! I'm very excited about it, and have several architectures that will greatly benefit from the switch to skytools3. More on that later, though! (Yes, my to blog later list is getting quite long now).

At the previous developer meeting his advice was to avoid putting too much into the very first version of the patch for it to stand its chances of being integrated, and while in the review process more than one major PostgreSQL contributor expressed worries about the size of the patch and the number of features proposed. Which is the usual process.

Then what happened is that Tom finally took a similar reasoning as mine while working on the feature. To maximize the benefits, once you have the infrastructure in place, it's not that much more work to provide the really interesting features. What's complex is agreeing on what exactly are their specifications. And in the little time window we got on this commit fest (well, we hijacked about 2 full weeks there), we managed to get there.

So in the end the result is quite amazing, and you can see that on the documentation chapter about it: 35.15. Packaging Related Objects into an Extension.

All the contrib modules that are installing SQL objects into databases for you to use them are now converted to Extensions too, and will get released in 9.1 with an upgrade script that allows you to upgrade from unpackaged. That means that once you've upgraded from a past PostgreSQL release up to 9.1, it will be a command away for you to register extensions as such. I expect third party extension authors (from ip4r to temporal) to release a upgrade-from-unpackaged version of their work too.

Of course, a big use case of the extensions is also in-house PL code, and having version number and multi-stage upgrade scripts there will be fantastic too, I can't wait to work with such a tool set myself. Some later blog post will detail the benefits and usage. I'm already trying to think how much of this version and upgrade facility could be expanded to classic DDL objects…

So expect some more blog posts from me on this subject, I will have to talk about debian packaging an extension (it's getting damn easy with postgresql-server-dev-all — yes it has received some planing ahead), and about how to package your own extension, manage upgrades, turn your current pre-9.1 extension into a full blown extension, and maybe how to stop worrying about extension when you're a DBA.

If you have some features you would want to discuss for next releases, please do contact me!

Meanwhile, I'm very happy that this project of mine finally made it to core, it's been long in the making. Some years to talk about it and then finally 4 months of coding that I'll remember as a marathon. Many Thanks go to all who helped here, from 2ndQuadrant to early reviewers to people I talked to over beers at conferences… lots of people really.

To an extended PostgreSQL (and beyond) :)

New source types

We now have support for the pacman package management for archlinux, and a way to handle a different package name in the recipe and in the distribution. We also have support for mercurial and subversion and darcs.

Also, apt-get will sometime prompt you to validate its choices, that's the infamous Do you want to continue? prompt. We now handle that smoothly.

(el-get 'sync)

In 1.1, that really means synchronous. That means we install one package after the other, and any error will stop it all. Before that, it was an active wait loop over a parallel install: this option is still available through calling (el-get 'wait).

No more failed to install

Exactly. This error you may have encountered sometime is due to trying to install a package over a previous failed install attempt (network outage, disk full, bad work-in-progress recipe, etc). After awhile in the field it was clear that no case where found where you would regret it if el-get just did removed the previous failed installation for you before to go and install again, as aked. So that's now automatic.

Featuring an overhauled :build facility

The build commands can now either be a list, as before, or some that we evaluate for you. That allows for easier to maintain recipes, and here's an exemple of that:

(:name distel
       :type svn
       :url "http://distel.googlecode.com/svn/trunk/"
       :info "doc"
       :build `,(mapcar
                 (lambda (target)
                   (concat "make " target " EMACS=" el-get-emacs))
                 '("clean" "all"))
       :load-path ("elisp")
       :features distel)

As you see that also allows for maintainance of multi-platform build recipes, and multiple emacs versions too. It's still a little too much on the awkward side of things, though, and that's one of the ongoing work that will happen for next version.

Misc improvements

We are now able to byte-compile your packages, and offer some more hooks (el-get-init-hooks has been asked with a nice usage example). There's a new :localname property that allows to pick where to save the local file when using HTTP method for retrieval, and that in turn allows to fix some recipes.

(:name xcscope
       :type http
       :url "http://cscope.cvs.sourceforge.net/viewvc/cscope/cscope/contrib/xcscope/xcscope.el?revision=1.14&content-type=text%2Fplain"
       :localname "xscope.el"
       :features xcscope)

Oh and you even get :before user function support, even if needing it often shows that you're doing it in a strange way. More often than not it's possible to do all you need to in the :after function, but this tool is there so that you spend less time on having a working environment, not more, right? :)

Switch notice

All in all, if you're already using el-get you should consider switching to 1.1 (by issuing M-x el-get-update of course), and if you're hesitating, just join the fun now!

New source types

We now have support for the pacman package management for archlinux, and a way to handle a different package name in the recipe and in the distribution. We also have support for mercurial and subversion and darcs.

Also, apt-get will sometime prompt you to validate its choices, that's the infamous Do you want to continue? prompt. We now handle that smoothly.

(el-get 'sync)

In 1.1, that really means synchronous. That means we install one package after the other, and any error will stop it all. Before that, it was an active wait loop over a parallel install: this option is still available through calling (el-get 'wait).

No more failed to install

Exactly. This error you may have encountered sometime is due to trying to install a package over a previous failed install attempt (network outage, disk full, bad work-in-progress recipe, etc). After awhile in the field it was clear that no case where found where you would regret it if el-get just did removed the previous failed installation for you before to go and install again, as aked. So that's now automatic.

Featuring an overhauled :build facility

The build commands can now either be a list, as before, or some that we evaluate for you. That allows for easier to maintain recipes, and here's an exemple of that:

(:name distel
       :type svn
       :url "http://distel.googlecode.com/svn/trunk/"
       :info "doc"
       :build `,(mapcar
                 (lambda (target)
                   (concat "make " target " EMACS=" el-get-emacs))
                 '("clean" "all"))
       :load-path ("elisp")
       :features distel)

As you see that also allows for maintainance of multi-platform build recipes, and multiple emacs versions too. It's still a little too much on the awkward side of things, though, and that's one of the ongoing work that will happen for next version.

Misc improvements

We are now able to byte-compile your packages, and offer some more hooks (el-get-init-hooks has been asked with a nice usage example). There's a new :localname property that allows to pick where to save the local file when using HTTP method for retrieval, and that in turn allows to fix some recipes.

(:name xcscope
       :type http
       :url "http://cscope.cvs.sourceforge.net/viewvc/cscope/cscope/contrib/xcscope/xcscope.el?revision=1.14&content-type=text%2Fplain"
       :localname "xscope.el"
       :features xcscope)

Oh and you even get :before user function support, even if needing it often shows that you're doing it in a strange way. More often than not it's possible to do all you need to in the :after function, but this tool is there so that you spend less time on having a working environment, not more, right? :)

Switch notice

All in all, if you're already using el-get you should consider switching to 1.1 (by issuing M-x el-get-update of course), and if you're hesitating, just join the fun now!

Back? Ok, so I used to have two main gripes against debian support for PostgreSQL. The first one, which is now feeling alone, is that both project release support policy aren't compatible enough for debian stable to include all currently supported stable PostgreSQL major version. That's very bad that debian stable will only propose one major version, knowing that the support for several of them is in there.

The problem is two fold: first, debian stable has to maintain any distributed package. There's no deprecation policy allowing for droping the ball. So the other side of this coin is that debian developers must take on themselves maintaining included software for as long as stable is not renamed oldstable. And it so happens that there's no debian developer that feels like maintaining end of lined PostgreSQL releases without help from PostgreSQL Core Team. Or, say, without official statement that they would help.

Now, why I don't like this situation is because I'm pretty sure there's very few software development group offering as long and reliable maintenance policy as PostgreSQL is doing, but debian will still happily distribute unknown-maintenance-policy pieces of code in its stable repositories. So the uncertainty excuse is rather poor. And highly frustrating.

Note: you have to admit that the debian stable management model copes very well with all the debian included software. You can't release stable with a new PostgreSQL major version unless each and every package depending on PostgreSQL will actually work with the newer version, and the debian scripts will care for upgrading the cluster. Where it's not working good is when you're using debian for a PostgreSQL server for a proprietary application, which happens quite frequently too.

The consequence of this fact leads to my second main gripe against debian support for PostgreSQL: the extensions. It so happens that the PostgreSQL extensions are developped for supporting several major versions from the same source code. So typically, all you need to do is recompile the extension against the new major version, and there you go.

Now, say debian new stable is coming with 8.4 rather than 8.3 as it used to. You should be able to just build the extensions (like prefix), without changing the source package, nor droping postgresql-8.3-prefix from the distribution on the grounds that 8.3 ain't in debian stable anymore.

I've been ranting a lot about this state of facts, and I finally provided a patch to the postgresql-common debian packaging, which made it into version 110: welcome pg_buildext. An exemple of how to use it can be found in the git branch for prefix, it shows up in debian/pgversions and debian/rules files.

As you can see, the pg_buildext tool allows you to list the PostgreSQL major versions the extension you're packaging supports, and only those that are both in your list and in the current debian supported major version list will get built. pg_buildext will do a VPATH build of your extension, so it's capable of building the same extension for multiple major versions of PostgreSQL. Here's how it looks:

        # build all supported version
        pg_buildext build $(SRCDIR) $(TARGET) "$(CFLAGS)"

        # then install each of them
        for v in `pg_buildext supported-versions $(SRCDIR)`; do \
                dh_install -ppostgresql-$$v-prefix ;\
        done

And the files are to be found in those places:

dim ~/dev/prefix cat debian/postgresql-8.3-prefix.install
debian/prefix-8.3/prefix.so usr/lib/postgresql/8.3/lib
debian/prefix-8.3/prefix.sql usr/share/postgresql/8.3/contrib

dim ~/dev/prefix cat debian/postgresql-8.4-prefix.install
debian/prefix-8.4/prefix.so usr/lib/postgresql/8.4/lib
debian/prefix-8.4/prefix.sql usr/share/postgresql/8.4/contrib

So you still need to maintain debian/pgversions and the postgresql-X.Y-extension.* files, but then a change in debian support for PostgreSQL major versions will be handled automatically (there's a facility to trigger automatic rebuild when necessary).

All this ranting to explain that pretty soon, the extenion's packages that I maintain will no longer have to be patched when dropping a previously supported major version of PostgreSQL. I'm breathing a little better, so thanks a lot Martin!

One of the main problems of setting up Emacs is that not only you tend to accumulate so many tricks from EmacsWiki and blog posts that your .emacs has to grow to a full ~/.emacs.d/ directory (starting at ~/.emacs.d/init.el), but also you finally depend on several librairies of code you're not authoring nor maintaining. Let's call them packages.

Some of them will typically be available on ELPA, which allows you to breathe and keep cool. But most of them, let's face it, are not there. Most of the packages I use I tend to get them either from debian (see apt-rdepends for having the complete list of packages that depends on emacs, unfortunately I'm not finding an online version of the tool to link too), or from ELPA, or from their own git repository somewhere. Some of them even I get directly from an obscure website not maintained anymore, but always there when you need them.

Of course, my emacs setup is managed in a private git repository. Some people on #emacs are using git submodules (or was it straight import) for managing external repositories in there, but all I can say is that I frown on this idea. I want an easy canonical list of packages I depend on to run emacs, and I want this documentation to be usable as-is. Enters el-get!

As we're all damn lazy, here's a visual introduction to el-get:

(setq el-get-sources
      '((:name bbdb
               :type git
               :url "git://github.com/barak/BBDB.git"
               :load-path ("./lisp" "./bits")
               :info "texinfo"
               :build ("./configure" "make"))

        (:name magit
               :type git
               :url "http://github.com/philjackson/magit.git"
               :info "."
               :build ("./autogen.sh" "./configure" "make"))

        (:name vkill
               :type http
               :url "http://www.splode.com/~friedman/software/emacs-lisp/src/vkill.el"
               :features vkill)

        (:name yasnippet
               :type git-svn
               :url "http://yasnippet.googlecode.com/svn/trunk/")

        (:name asciidoc         :type elpa)
        (:name dictionary-el    :type apt-get)
        (:name emacs-goodies-el :type apt-get)))

(el-get)

So now you have a pretty good documentation of the packages you want installed, where to get them, and how to install them. For the advanced methods (such as elpa or apt-get), you basically just need the package name. When relying on a bare git repository, you need to give some more information, such as the URL to clone and the build steps if any. Then also what features to require and maybe where to find the texinfo documentation of the package, for automatic inclusion into your local Info menu.

The good news is that not only you now have a solid readable description of all that in a central place, but this very description is all (el-get) needs to do its magic. This command will check that each and every package is installed on your system (in el-get-dir) and if that's not the case, it will actually install it. Then, it will init the packages: that means caring about the load-path, the Info-directory-list (and dir texinfo menu building), the loading of the emacs-lisp files, and finally it will require the features.

Here's a prettyfied ielm session that will serve as a demo:

ELISP> (el-get)
("aspell-en" "aspell-fr" "muse" "dictionary" "htmlize" "bbdb" "google-maps"
"magit" "emms" "nxhtml" "vkill" "xcscope" "yasnippet" "asciidoc"
"auto-dictionary" "css-mode" "gist" "lua-mode" "lisppaste")

All the packages being already installed, it's running fast enough that I won't bother measuring the run time, that seems to be somewhere around one second.

One of the main problems of setting up Emacs is that not only you tend to accumulate so many tricks from EmacsWiki and blog posts that your .emacs has to grow to a full ~/.emacs.d/ directory (starting at ~/.emacs.d/init.el), but also you finally depend on several librairies of code you're not authoring nor maintaining. Let's call them packages.

Some of them will typically be available on ELPA, which allows you to breathe and keep cool. But most of them, let's face it, are not there. Most of the packages I use I tend to get them either from debian (see apt-rdepends for having the complete list of packages that depends on emacs, unfortunately I'm not finding an online version of the tool to link too), or from ELPA, or from their own git repository somewhere. Some of them even I get directly from an obscure website not maintained anymore, but always there when you need them.

Of course, my emacs setup is managed in a private git repository. Some people on #emacs are using git submodules (or was it straight import) for managing external repositories in there, but all I can say is that I frown on this idea. I want an easy canonical list of packages I depend on to run emacs, and I want this documentation to be usable as-is. Enters el-get!

As we're all damn lazy, here's a visual introduction to el-get:

(setq el-get-sources
      '((:name bbdb
               :type git
               :url "git://github.com/barak/BBDB.git"
               :load-path ("./lisp" "./bits")
               :info "texinfo"
               :build ("./configure" "make"))

        (:name magit
               :type git
               :url "http://github.com/philjackson/magit.git"
               :info "."
               :build ("./autogen.sh" "./configure" "make"))

        (:name vkill
               :type http
               :url "http://www.splode.com/~friedman/software/emacs-lisp/src/vkill.el"
               :features vkill)

        (:name yasnippet
               :type git-svn
               :url "http://yasnippet.googlecode.com/svn/trunk/")

        (:name asciidoc         :type elpa)
        (:name dictionary-el    :type apt-get)
        (:name emacs-goodies-el :type apt-get)))

(el-get)

So now you have a pretty good documentation of the packages you want installed, where to get them, and how to install them. For the advanced methods (such as elpa or apt-get), you basically just need the package name. When relying on a bare git repository, you need to give some more information, such as the URL to clone and the build steps if any. Then also what features to require and maybe where to find the texinfo documentation of the package, for automatic inclusion into your local Info menu.

The good news is that not only you now have a solid readable description of all that in a central place, but this very description is all (el-get) needs to do its magic. This command will check that each and every package is installed on your system (in el-get-dir) and if that's not the case, it will actually install it. Then, it will init the packages: that means caring about the load-path, the Info-directory-list (and dir texinfo menu building), the loading of the emacs-lisp files, and finally it will require the features.

Here's a prettyfied ielm session that will serve as a demo:

ELISP> (el-get)
("aspell-en" "aspell-fr" "muse" "dictionary" "htmlize" "bbdb" "google-maps"
"magit" "emms" "nxhtml" "vkill" "xcscope" "yasnippet" "asciidoc"
"auto-dictionary" "css-mode" "gist" "lua-mode" "lisppaste")

All the packages being already installed, it's running fast enough that I won't bother measuring the run time, that seems to be somewhere around one second.

I think that Emacs and PostgreSQL do share a lot in common. I'd begin with the documentation, which quality is amazing for both projects. Then of course the extensibility with Emacs Lisp on the one hand and catalog-driven operations on the other hand. Whether you're extending Emacs or PostgreSQL you'll find that it's pretty easy to tweak the system while it's running. The other comparison points are less important, like the fact the both the systems get about the same uptime on my laptop (currently 13 days, 23 hours, 57 minutes, 10 seconds).

So of course I'm using Emacs to edit PostgreSQL .sql files, including stored procedures. And it so happens that line numbering in plpgsql is not as straightforward as one would naively think, to the point that we'd like to have better tool support there. So I've extended Emacs linum-mode minor mode to also display the line numbers as computed per PostgreSQL, and here's what it looks like:

Now, here's also the source code, pgsql-linum-format. Hope you'll enjoy!

I think that Emacs and PostgreSQL do share a lot in common. I'd begin with the documentation, which quality is amazing for both projects. Then of course the extensibility with Emacs Lisp on the one hand and catalog-driven operations on the other hand. Whether you're extending Emacs or PostgreSQL you'll find that it's pretty easy to tweak the system while it's running. The other comparison points are less important, like the fact the both the systems get about the same uptime on my laptop (currently 13 days, 23 hours, 57 minutes, 10 seconds).

So of course I'm using Emacs to edit PostgreSQL .sql files, including stored procedures. And it so happens that line numbering in plpgsql is not as straightforward as one would naively think, to the point that we'd like to have better tool support there. So I've extended Emacs linum-mode minor mode to also display the line numbers as computed per PostgreSQL, and here's what it looks like:

Now, here's also the source code, pgsql-linum-format. Hope you'll enjoy!

The tool provides a lot of commands to either dump or restore a database. It comes with documentation covering about it all, except for the londiste support part, which will be there in time for 1.0.0 release. The Todo list is getting smaller and smaller, the version you'll soon find in debian sid is already called 0.9.

So, how do you go about using this software, and what service it implements?

it's all about deriving a staging environment from your backups

To validate backups, you want to restore them and check the database you get from them. And your developers will want to sometime refresh the database they're working with. And you could have both an integration environment and a pre-live one: On the former, you develop new code atop a stable set of data; while on the latter you test stable enough code (ready to go live) on a set of data as near as live data as possible.

And you want to be flexible about it, so that there's not a fulltime job to handle retoring databases each and every days, for project A integration or project B pre-live testing, or project C accounting snapshot. Or you name it.

And of course you want to have a single point of control of all your databases. Let's call it the controler.

setting up pg_staging

The pg_staging setup consists of one pg_staging.ini file wherein you describe your different target databases (those dev and prelive ones), and of course where to get the production backups from. Currently you have to serve the backups file in a format suitable for pg_restore (that means you use either pg_dump -Ft or pg_dump -Fc) on an apache folder. The produced HTML will get parsed.

So you setup the DEFAULT section with common settings, then one section per target: the databases you want to restore. Tell pg_staging where they are (host), etc, and it'll be able to drive them.

In order to being able to host more than a single restored dump on a staging server, for the same database, we use pgbouncer:

pg_staging> pgbouncer some_db.dev
              some_db      some_db_20091029 :5432
     some_db_20090717      some_db_20090717 :5432
     some_db_20091029      some_db_20091029 :5432

So as explained into the pg_staging(1) man page, you have to open non-interactive SSH connection from the controler to the hosts where the databases will get restored. Then you have to do a minimal setup pgbouncer on the hosts with a trust connection. It'll get used from pg_staging for adding newly restored database and have them accessible. Then you can also switch the new database to being the virtual some_db so that you avoid editing any connection string on your softwares.

Also, install the pgstaging-client package on every host you target. The client is a simple shell script that must run as root (sudo is used) in order to replace your pgbouncer setup or manage your londiste services.

See man 5 pg_staging for available options, including schemas to filter out either completely or just skipping data restoring in those.

pg_staging usage

Now you're all setup, you can begin to enjoy using pgstaging. Enter the console and see what you have in there.

$ pg_staging
Welcome to pg_staging 0.9.
pg_staging> databases
...
pg_staging> restore some_db.dev
...
pg_staging> pgbouncer some_db.dev
...
pg_staging> dbsizes --all some_db.dev
...
pg_staging> psql some_db.dev
some_db_20091125=#

And as you can see in man pg_staging there are a lot of commands already. You can for example obtain a new pg_restore catalog from a dump file, with some schemas commented out. It will even comment out triggers that are using a function which is defined in a filtered out schema, for example a PGQ trigger. And much much more.

pg_staging will even allow you to dump your production databases, but consider installing a separate instance of it on the machine serving the backups to your local network thanks to an apache directory listing!

Roadmap to 1.0.0

What's remain to be done is testing and having PITR based restoring to work, and adding some documentation (tutorial, which this blog post about is; and londiste support). At this point, unless some reader here asks for a new feature (set), I'll consider pg_staging ready for 1.0.0. After all, we're using it about daily here :)

Consider commenting, you should be able to easily spot my private mail address...

In order to make it clear that I intend to maintain this version, the number has 3 digits rather than 2... which is also what PostgreSQL users will expect.

The only last minute change is that you can now use the first version of the two following rather than the second one:

-  create index idx_prefix on prefixes using gist(prefix gist_prefix_range_ops);
+  create index idx_prefix on prefixes using gist(prefix);

For you information, I'm thinking about leaving pgfoundry as far as the source code management goes, because I'd like to be done with CVS. I'd still use the release file hosting though at least for now. It's a burden but it's easier for the users to find them, when they are not using plain apt-get install. That move would lead to host prefix and pgloader and the backports over there at github, where my next pet project, pg_staging, will be hosted too.

The way to see this pgfoundry leaving is that if everybody does the same, then migrating the facility to some better or more recent hosting software will be easier. Maybe some other parts of the system are harder than the sources to migrate, though. If that's the case I'll consider moving them out too, maybe getting listed on the PostgreSQL Software Catalogue will prove enough as far as web presence goes?

Time to do some bug fixing already, as hstore-new packaging is using a bash'ism I shouldn't rely on (or so the debian buildfarm is telling me) and for preprepare I was waiting for inclusion before to go improving the GUC management, stealing some code from Selena's pgGearman :)

As some of you wonder about prefix 1.0 scheduling, it should soon get there now it's been in testing long enough and no bug has been reported. Of course releasing 1.0 in august isn't good timing, so maybe I should just wait some more weeks.

Anyway, the plan for prefix is to now wait something like 2 weeks, then, baring showstopper bugs, release the 1.0 final version. If you have a use for it, now is the good time for testing it!

About upgrading a current prefix installation, the advice is to save data as text instead of prefix_range, remove prefix support, install new version, change again the columns data type:

BEGIN;
  ALTER TABLE foo
     ALTER COLUMN prefix
             TYPE text USING text(prefix);

  DROP TYPE prefix_range CASCADE;
  \i prefix.sql

  ALTER TABLE foo
     ALTER COLUMN prefix
             TYPE prefix_range USING prefix_range(prefix);

  CREATE INDEX idx_foo_prefix ON foo
         USING gist(prefix gist_prefix_range_ops);
COMMIT;

Note: I just added the gist_prefix_range_ops as default for type prefix_range so it'll be optional to specify this in final 1.0. I got so used to typing it I didn't realize we don't have to :)

  SELECT foo, bar
    FROM prefixes
   WHERE prefix @> '012345678'
ORDER BY length(prefix) DESC
   LIMIT 1;

And here's a simple stupid benchmark of the new function, here in prefix-1.0~rc2.tar.gz. And it'll soon reach debian, if my QA dept agrees (my sponsor is a QA dept all by himself!).

First some preparation:

dim=#   create table prefixes (
dim(#          prefix    prefix_range primary key,
dim(#          name      text not null,
dim(#          shortname text,
dim(#          status    char default 'S',
dim(#
dim(#          check( status in ('S', 'R') )
dim(#   );
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "prefixes_pkey" for
 table "prefixes"
CREATE TABLE
Time: 74,357 ms
dim=#   \copy prefixes from 'prefixes.fr.csv' with delimiter ; csv quote '"'
Time: 200,982 ms
dim=# select count(*) from prefixes ;
 count
-------
 11966
(1 row)
Time: 3,047 ms

And now for the micro-benchmark:

dim=# \o /dev/null
dim=# select length(prefix) from prefixes;
Time: 16,040 ms
dim=# select length(prefix::text) from prefixes;
Time: 23,364 ms
dim=# \o

Hope you enjoy!

If you have any use for it (as some VoIP companies have already), please consider testing it, in order for me to release a shiny 1.0 next week! :)

Recent changes include getting rid of those square brackets output when it's not neccesary, fixing btree operators, adding support for more operators in the GiST support code (now supported: @>, <@, =, &&). Enjoy!

The problem is that Emacs Muse is broken on emacs-snapshot, partly because of Htmlize which is unable to find the face fonts (I got (error "Invalid face")), partly because of my configuration itself:

hunk ./dim-muse.el 22
-      '(("pgsql.tapoueh.org" $
-        (,@(muse-project-alist-dirs "~/dev/muse/site") $
+      '(("pgsql.tapoueh.org" ("~/dev/muse/site"
+        ;;(,@(muse-project-alist-dirs "~/dev/muse/site") $

The solution was to switch to using Emacs 22 on sid for pgsql.tapoueh.org editing, while using EmacsCVS for other activities.

And I'm using the patched Htmlize on both the versions, by the way.

The problem is that Emacs Muse is broken on emacs-snapshot, partly because of Htmlize which is unable to find the face fonts (I got (error "Invalid face")), partly because of my configuration itself:

hunk ./dim-muse.el 22
-      '(("pgsql.tapoueh.org" $
-        (,@(muse-project-alist-dirs "~/dev/muse/site") $
+      '(("pgsql.tapoueh.org" ("~/dev/muse/site"
+        ;;(,@(muse-project-alist-dirs "~/dev/muse/site") $

The solution was to switch to using Emacs 22 on sid for pgsql.tapoueh.org editing, while using EmacsCVS for other activities.

And I'm using the patched Htmlize on both the versions, by the way.