My First Slashdot Effect

January, 22 2015

Thanks to the Postgres Weekly issue #89 and a post to Hacker News front page (see Pgloader: A High-speed PostgreSQL Swiss Army Knife, Written in Lisp it well seems that I just had my first Slashdot effect...


New release: pgloader 3.2

January, 16 2015

PostgreSQL comes with an awesome bulk copy protocol and tooling best known as the COPY and \copy commands. Being a transactional system, PostgreSQL COPY implementation will ROLLBACK any work done if a single error is found in the data set you're importing. That's the reason why pgloader got started: it provides with error handling for the COPY protocol.


Earlier this year we did compare compare Aggregating NBA data, PostgreSQL vs MongoDB then talked about PostgreSQL, Aggregates and histograms where we even produced a nice Histogram chart directly within the awesome psql console. Today, let's get that same idea to the next level, with pgcharts:


pgloader loads data into PostgreSQL. The new version is stable enough nowadays that it's soon to be released, the last piece of the 3.1.0 puzzle being full debian packaging of the tool.


A long time ago we talked about how to Import fixed width data with pgloader, following up on other stories still online at Postgres OnLine Journal and on David Fetter's blog. Back then, I showed that using pgloader made it easier to import the data, but also showed quite poor performances characteristics due to using the debug mode in the timings. Let's update that article with current pgloader wonders!


In our previous article about Loading Geolocation Data, we did load some data into PostgreSQL and saw the quite noticable impact of a user transformation. As it happens, the function that did the integer to IP representation was so naive as to scratch the micro optimisation itch of some Common Lisp hackers: thanks a lot guys, in particular stassats who came up with the solution we're seeing now.


Loading Geolocation Data

October, 01 2013

As I've been mentionning in the past already, I'm currently rewriting pgloader from scratch in Common Lisp. In terms of technical debt that's akin to declaring bankrupcy, which is both sad news and good news as there's suddenly new hope of doing it right this time.


This blog of mine is written in the very good Emacs Muse format, that I find much more friendly to writing articles than both org-mode and markdown-mode that I both use in a regular basis too. The main think that I like in Muse that those two others lack is the support for displaying images inline.


Last week came with two bank holidays in a row, and I took the opportunity to design a command language for pgloader. While doing that, I unexpectedly stumbled accross a very nice AHAH! moment, and I now want to share it with you, dear reader.


Emacs Conference

March, 04 2013

The Emacs Conference is happening, it's real, and it will take place at the end of this month in London. Check it out, and register at Emacs Conference Event Brite. It's free and there's still some availability.


Playing with pgloader

February, 12 2013

While making progress with both Event Triggers and Extension Templates, I needed to make a little break. My current keeping sane mental exercise seems to mainly involve using Common Lisp, a programming language that ships with about all the building blocks you need.


pgloader: what's next?

January, 28 2013

pgloader is a tool to help loading data into PostgreSQL, adding some error management to the COPY command. COPY is the fast way of loading data into PostgreSQL and is transaction safe. That means that if a single error appears within your bulk of data, you will have loaded none of it. pgloader will submit the data again in smaller chunks until it's able to isolate the bad from the good, and then the good is loaded in.


Lost in scope

January, 09 2013

Thanks to Mickael on twitter I got to read an article about loosing scope with some common programming languages. As the blog article Lost in scope references functional programming languages and plays with both Javascript and Erlang, I though I had to try it out with Common Lisp too.


CL Happy Numbers

November, 20 2012

A while ago I stumbled upon Happy Numbers as explained in programming praxis, and offered an implementation of them in SQL and in Emacs Lisp. Yeah, I know. Why not, though?


Concurrent Hello

November, 04 2012

Thanks to Mickael on twitter I ran into that article about implementing a very basic Hello World! program as a way to get into a new concurrent language or facility. The original article, titled Concurrent Hello World in Go, Erlang and C++ is all about getting to know The Go Programming Language better.


Fast and stupid?

August, 22 2012

I stumbled onto an interesting article about performance when using python, called Python performance the easy(ish) way, where the author tries to get the bet available performances out of the dumbiest possible python code, trying to solve a very simple and stupid problem.


Peter Norvig published a while ago a very nice article titled Solving Every Sudoku Puzzle wherein he presents a programmatic approach to solving that puzzle game.