pgloader loads data into PostgreSQL. The new version is stable enough nowadays that it's soon to be released, the last piece of the 3.1.0 puzzle being full debian packaging of the tool.

 

A long time ago we talked about how to Import fixed width data with pgloader, following up on other stories still online at Postgres OnLine Journal and on David Fetter's blog. Back then, I showed that using pgloader made it easier to import the data, but also showed quite poor performances characteristics due to using the debug mode in the timings. Let's update that article with current pgloader wonders!

 

As presented at the PostgreSQL Conference Europe the new version of pgloader is now able to fully migrate a MySQL database, including discovering the schema, casting data types, transforming data and default values. Sakila is the traditional MySQL example database, in this article we're going to fully migrate it over to PostgreSQL.

 

Back From Dublin

November, 05 2013

Last week I had the pleasure to present two talks at the awesome PostgreSQL Conference Europe. The first one was actually a tutorial about Writing & using Postgres Extensions where we spent 3 hours on what are PostgreSQL Extensions, what you can expect from them, and how to develop a new one. Then I also had the opportunity to present the new version of pgloader in a talk about Migrating from MySQL to PostgreSQL.

 

In our previous article about Loading Geolocation Data, we did load some data into PostgreSQL and saw the quite noticable impact of a user transformation. As it happens, the function that did the integer to IP representation was so naive as to scratch the micro optimisation itch of some Common Lisp hackers: thanks a lot guys, in particular stassats who came up with the solution we're seeing now.

 

Loading Geolocation Data

October, 01 2013

As I've been mentionning in the past already, I'm currently rewriting pgloader from scratch in Common Lisp. In terms of technical debt that's akin to declaring bankrupcy, which is both sad news and good news as there's suddenly new hope of doing it right this time.

 

About the only time when I will accept to work with MySQL is when you need help to migrate away from it because you decided to move to PostgreSQL instead. And it's already been too much of a pain really, so after all this time I began consolidating what I know about that topic and am writing a software to help me here. Consider it the MySQL Migration Toolkit.

 

Playing with pgloader

February, 12 2013

While making progress with both Event Triggers and Extension Templates, I needed to make a little break. My current keeping sane mental exercise seems to mainly involve using Common Lisp, a programming language that ships with about all the building blocks you need.

 

pgloader: what's next?

January, 28 2013

pgloader is a tool to help loading data into PostgreSQL, adding some error management to the COPY command. COPY is the fast way of loading data into PostgreSQL and is transaction safe. That means that if a single error appears within your bulk of data, you will have loaded none of it. pgloader will submit the data again in smaller chunks until it's able to isolate the bad from the good, and then the good is loaded in.

 

Another day, another migration from MySQL to PostgreSQL... or at least that's how it feels sometimes. This time again I've been using some quite old scripts to help me do the migration.

 

pgloader constant cols

August, 12 2011

The previous articles in the pgloader series detailed How To Use PgLoader then How to Setup pgloader, then what to expect from a parallel pgloader setup, and then pgloader reformating. Another need you might encounter when you get to use pgloader is adding constant values into a table's column.

 

pgloader reformating

August, 05 2011

Back to our series about pgloader. The previous articles detailed How To Use PgLoader then How to Setup pgloader, then what to expect from a parallel pgloader setup. This article will detail how to reformat input columns so that what PostgreSQL sees is not what's in the data file, but the result of a transformation from this data into something acceptable as an input for the target data type.

 

Parallel pgloader

August, 01 2011

This article continues the series that began with How To Use PgLoader then detailed How to Setup pgloader. We have some more fine points to talk about here, today's article is about loading your data in parallel with pgloader.

 

pgloader en parallèle

August, 01 2011

Dans la série des articles sur pgloader, nous détaillons cette fois la configuration du paraléllisme de chargement des données. Une fois de plus, je vous réfère à l'article de référence en anglais : Parallel pgloader.

 

Configurer pgloader

July, 29 2011

Je viens de publier un billet en anglais intitulé How to Setup pgloader, qui complète l'écriture en cours d'un tutoriel pgloader plus complet. Une fois de plus, je n'ai pas pris le temps de traduire cet article en français avant de savoir si cela vous intéresse, ô lecteurs. Si c'est le cas il suffit de me l'indiquer par mail (ou courriel, après tout) pour que j'ajoute cela dans ma TODO liste.

 

How to Setup pgloader

July, 29 2011

In a previous article we detailed how to use pgloader, let's now see how to write the pgloader.conf that instructs pgloader about what to do.

 

C'est une question qui revient régulièrement, et à laquelle je pensais avoir apporté une réponse satisfaisante avec les exemples pgloader. Ce document ressemble un peu à un tutoriel, en anglais, et je l'ai détaillé dans l'article how to use pgloader sur ce même site, en anglais. Si la demande est suffisante, je le traduirai en français.

 

How To Use PgLoader

July, 22 2011

This question about pgloader usage coms in quite frequently, and I think the examples README goes a long way in answering it. It's not exactly a tutorial but is almost there. Let me paste it here for reference:

 

So, following previous blog entries about importing fixed width data, from Postgres Online Journal and David (perl) Fetter, I couldn't resist following the meme and showing how to achieve the same thing with pgloader.

 

Yes. This pgloader project is still maintained and somewhat active. Development happens when I receive a complaint, either about a bug in existing code or a feature in yet-to-write code. If you have a bug to report, just send me an email!

 

prefix 1.0.0

October, 06 2009

So there it is, at long last, the final 1.0.0 release of prefix! It's on its way into the debian repository (targetting sid, in testing in 10 days) and available on pgfoundry to.