PostgreSQL on Dimitri Fontaine

Consolidating databases with Postgres logical replication

Tue, 06 Oct 2026 13:45:23 +0200

This is part 2 of a series about Postgres logical replication use-cases, and about how the feature set has evolved over the past ten years and ten releases, one architecture at a time. Part 1 built a hub-and-workers system for write scalability and has the table of what each release added, which this post assumes. Part 3 covers a zero-downtime major upgrade.

Ten years of Postgres logical replication

Tue, 22 Sep 2026 17:26:01 +0200

A long time ago I ran a write-heavy system on a hub and a handful of workers. Each worker took a share of the application traffic and wrote events locally. The hub owned the reference data (customers, plans, prices), pushed it down to the workers, and pulled every worker’s events back up to compute the invoices. The plumbing was Londiste and PgQ: triggers on every table, a queue per node, a ticker, and a Python daemon per hop. It worked, and it was a lot of moving parts to explain to anyone new.

Postgres 10 shipped logical replication in 2017, and 19 is the tenth release that has it. Every release since Postgres 10 has taken a piece of that plumbing and made it a line of SQL.

This is the first article in a series about Postgres logical replication use-cases, and about how the feature set has evolved over the past ten years and ten releases. The question is the application developer’s one, not the DBA’s: which architectures can I deploy with Postgres core alone today, what does each release change about that, and where do I still need something else? I built three architectures for real, across three posts:

Hub and workers, spreading the write load across servers — this post.
Consolidation: many databases, different applications and schemas, into one, then re-exported as a change stream for a CDC consumer.
Zero-downtime major upgrade, with a way back.

A fourth post, covering what is left out of this series in less detail — geo-replication, BDR-style multi-active setups, plain CDC and triggers — is also planned.

Plan Advice in PostgreSQL 19

Tue, 15 Sep 2026 18:27:53 +0200

There is a conversation that happens in every PostgreSQL shop eventually. A query that has been fine for a year gets slow overnight. Nothing was deployed. The data grew a little, ANALYZE ran, and the planner — entirely reasonably, on the numbers it had — picked a different plan. The old plan was better. You would like it back.

PostgreSQL 19 ships two new modules for exactly this: pg_plan_advice, which can read a plan back out as a string and enforce it later, and pg_stash_advice, which keeps those strings keyed by query id and applies them automatically.

Getting Ready for PostgreSQL 19

Thu, 03 Sep 2026 15:09:30 +0200

PostgreSQL 19 Beta 3 shipped on August 13, 2026, and the release notes have been filled in as of 2026-07-18 — still marked subject to change, and the GA date isn’t announced yet, but following the project’s usual September/October cadence general availability should land within the next few weeks. That makes now the right time to read through what’s changing, the same way I did for PostgreSQL 11 through 18 a few weeks ago.

This is not a changelog dump. It’s the subset of PG 19 I think is worth knowing about before you upgrade: a handful of compatibility breaks that will bite people who don’t read release notes, and the SQL-level additions I found genuinely useful once I started poking at them. Every query below ran against a real PostgreSQL 19 Beta 3 instance — no hand-waving about syntax that might work.

Update, 21 September 2026. Five features below have been withdrawn from PostgreSQL 19 since this went out: MERGE PARTITIONS and SQL/PGQ (already marked in their own sections), FOR PORTION OF on 15 September, the pg_get_*_ddl() functions on 12 September, and — from the watch list at the end — online data checksums on 16 September. Each section keeps what I ran on Beta 3 and now says what happened to it. The pg_get_*_ddl() functions and online checksums came out of the 19 branch only and are still in the development branch for 20; the other three came out of both. Everything else in this article is still in the release branch.

Every query in this article ran against the Lab, the same free dataset bundle used in the rest of this blog (F1 data, geopolitical data, music data, and more), pinned to a real PostgreSQL 19 Beta 3 instance. PG 19 support isn’t the Lab’s default yet, but there is a prebuilt beta image on the registry, for both linux/amd64 and linux/arm64 — so POSTGRES_VERSION=19beta3 PG_MAJOR=19 docker compose up pulls it rather than building anything, and every query below ran on it. One caveat added after publication: the image behind that tag has since been rebuilt, and the rebuild crashes the server on every FOR PORTION OF statement I tried; the captures in this article are from the original build. (PG_MAJOR only matters if Compose ends up building; it costs nothing to set.) Plain docker compose up still pulls the PG 16 image, and stays the default until 19 reaches general availability.

Introducing YeSQL: Practical PostgreSQL, One Concept at a Time

Tue, 18 Aug 2026 10:00:00 +0200

I’m happy to share something new: YeSQL is live, a free set of 24 short PostgreSQL lessons — one concept, one runnable query, real data, no signup. It’s free to use today, and it’s also a prototype for something I’ve wanted to build for a while: making every query in The Art of PostgreSQL runnable, right on the page.

Introducing sqlfmt: an SQL gofmt-style formatter

Tue, 11 Aug 2026 14:30:37 +0200

Formatting SQL tends to bring some of the same questions again and again: should we uppercase clause keywords? should we put the separating comma at the start of a line to ease refactoring? how to align the SQL clauses with one-another?

Over the years I have grown my own SQL style and didn’t find tooling that would implement it. Also, I’ve been asked here and there if there is a tool that would replicate The Art of PostgreSQL SQL indentation style… and now there is finally a good answer to that question!

sqlfmt is a gofmt-style formatter that implements my own favorite SQL indentation style. One opinionated style, no configuration knobs. Run it, commit the result, move on.

SQL Improvements in PostgreSQL 11–18: A Personal Selection

Tue, 28 Jul 2026 16:39:25 +0200

Seven major versions of PostgreSQL shipped between 2018 and 2025, one per year without exception, and each with a changelog of 150 to 200 user-visible changes. Each release covers a broad canvas — performance, replication, administration, and security — but every one of them also advanced the SQL layer, filling gaps in the standard, adding missing functionality, or cleaning up long-standing rough edges. Working through the new edition of The Art of PostgreSQL forced me to catalogue them all; this is my selection of the features I kept reaching for while rewriting the examples. I hope it’s useful beyond that context — organized by theme, with the version each feature landed in.

To get a sense of where the community actually puts its effort, Noriyoshi Shinoda of Hewlett-Packard Enterprise Japan has been publishing a meticulous “PostgreSQL New Features with Examples” series for every major release since version 9.4 — each edition catalogues every user-visible change with a working code example. Counting his categories across PG 11–18 gives a clear picture of contributor priorities:

Release	SQL	Performance	Admin & Ops	Replication	Security	Other
PG 11	5	6	5	3	4	3
PG 12	4	6	7	3	3	5
PG 13	7	8	10	6	5	6
PG 14	4	7	6	5	3	5
PG 15	2	5	6	5	3	8
PG 16	4	6	8	5	7	8
PG 17	5	7	8	8	4	6
PG 18	4	3	4	3	2	5
Sum	35	48	54	38	31	46

New features by category per release (Shinoda count)

SQL is the smallest category. Admin & Ops leads across the whole period, followed by Performance — which reflects who funds the project: Amazon (Aurora), Google (AlloyDB), and Microsoft (Azure) all have strong commercial interests in PostgreSQL being easy to operate, scale, and replicate at cloud scale. Replication & HA shows the steepest growth trajectory, climbing from 3 features in PG 11 to 8 in PG 17, tracking the same commercial priorities. The framing of this article is deliberately SQL-centric: execution strategy, query syntax, data types, schema modeling, and DML. Administration, replication, and authentication improvements are real too; a few that struck me as particularly worth knowing about are collected at the end.

pgcopydb v0.18

Wed, 01 Jul 2026 16:05:56 +0200

Hot off the press: pgcopydb v0.18 is out!

It’s the biggest release the project has had — 88 commits since v0.17, which shipped in August 2024. I took a break from my Open Source responsibilities for a while, because I was lacking employer support to make it happen.

What is pgcopydb

pgcopydb copies a PostgreSQL database to another PostgreSQL server, as fast as possible when physical file copy isn’t available. It parallelises the COPY across all tables simultaneously, builds indexes in parallel after data is loaded, and supports Change Data Capture via logical replication for minimal-downtime migrations. It is designed to be restartable: state is tracked in a local SQLite catalog so an interrupted run can resume where it left off.

Headline Features of pgcopydb v0.18

v0.18 brings compatibility with PostgreSQL 16, 17, and 18; a pgoutput-default CDC engine with significant reliability and performance improvements; regular-expression-based filtering; Citus-to-Citus migration support; and 24 bug fixes.

Postgres HA: roles are dynamic

Tue, 14 Dec 2021 22:36:36 +0100

High-Availability comes with some impact on your architecture choices, in particular when applied to RDBMS such as Postgres. One such impact is the idea of a failover. When implementing database HA, it is usually expected that both the service and the data are maintained available in the face of operational faults. The most common way to implement resilience includes automated (or manual) failover, where a new primary is elected among a list of standby nodes.

In other words, as soon as Postgres High-Availability is implemented, the roles of your Postgres nodes are dynamic. The fact that a given node is a primary or a standby at any given point in time ceases to be relevant to understanding your architecture. In fact, the only thing that’s now given about the role of a node is that it will change. Otherwise you don’t have failover capability, and then, you probably don’t have HA in the first place, right?

In this article we are going to try and understand what having dynamic roles for Postgres nodes in a HA system means.

An introduction to the pg_auto_failover project

Wed, 10 Nov 2021 17:11:29 +0100

We just released pg_auto_failover version 1.6.3 on GitHub, and the binary packages should be already available at the usual PGDG and CitusData places, both for debian based distributions and RPM based distributions too.

This article is an introduction to the pg_auto_failover project: we answer the Five W questions, starting with why does the project exist in the first place?

TL;DR pg_auto_failover is an awesome project. It fills the gap between “Postgres is awesome, makes developping my application so much easier, it solves so many problems for me!” and the next step “so, how do I run Postgres in Production?”. If you’re not sure how to bridge that gap yourself, how to deploy your first production system with automated failover, then pg_auto_failover is for you. It is simple to use, user friendly, and well documented. Star it on the pg_auto_failover GitHub repository and get started today. Consider contributing to the project, it is fully Open Source, and you are welcome to join us.

Buckle up, our guide tour is starting now!

PostgreSQL as a Microservice

Tue, 08 Jun 2021 13:40:00 +0200

The MACI French podcast honoured me with an invitation to a guest appearance on their weekly schedule. As you can imagine, we talked about many things related to PostgreSQL… and also reacted to some newsworthy articles carefully curated by the MACI team. One of the topics we discussed in the podcast started with looking at PostgreSQL through the angle of it being one of the microservices that your application would be composed of.

2020: Online Conferences

Fri, 27 Nov 2020 13:10:00 +0100

Among a lot of other changes, the year 2020 brings Online Conferences to us. In the Postgres community too we now record our talks at home and send a video file to be playedto a virtual audience, and sometimes shared later in a platform online.

So this year I did participate in Postgres Vision 2020 where I did deliver a talk about The Art of PostgreSQL. This a talk all about the book that I have written and self-publish at The Art of PostgreSQL: learn how to turn thousands of lines of code into simple SQL queries.

List PostgreSQL tables using extensions

Tue, 12 Nov 2019 19:30:00 +0200

Postgres has extensions, and that’s awesome! Of course as the author of CREATE EXTENSION I’m a little biased… just remember that the ability to extend Postgres is way more than just this command. The whole database system has been design from the ground up to allow for extensibility. Parts of the design is to be found in the way you can register new objects at runtime: functions of course, and also data types, operators, index support structures such as operator classes and families, even index access methods!

Today’s article shows a query that you can use to list those tables in your schemas that are using a data type which is provided by an extension.

Postgres Connection Strings and psql

Wed, 04 Sep 2019 11:38:02 +0200

PostgreSQL connection strings embedded in your application can take two different forms: the key-value notation or the postgresql:// URI scheme. When it comes to using psql though, another form of connection string is introduced, with command line options -h -p -U and environment variable support.

In this short article you will learn that you can use either of the three different forms in psql and thus easily copy & paste you application connection string right at the console to test it!

The Art Of PostgreSQL

Mon, 26 Aug 2019 10:15:00 +0100

I did it again! Today I am releasing the new edition of my book, with a new title: “The Art of PostgreSQL”. I’m very happy (and quite excited) to declare my book as Generally Available!

The Art of PostgreSQL is the new edition of my previous release, Mastering PostgreSQL in Application Development. It contains mostly fixes to the old content, a new title, and a new book design (PDF and paperback). Content wise, The Art of PostgreSQL also comes with a new whole chapter about PostgreSQL Extensions.

The new chapter covers extensions such as hstore, pg_trgm, intarray, earthdistance, ip4r, and hll or HyperLogLog, one of the all times favorite extensions of Craig Kerstiens… who made himself available to answer my questions and share his view of PostgreSQL Extensions in an interview!

Preventing SQL Injections

Sat, 10 Nov 2018 15:40:01 +0100

An SQL Injection is a security breach, one made famous by the Exploits of a Mom xkcd comic episode in which we read about little Bobby Tables:

PostgreSQL implements a protocol level facility to send the static SQL query text separately from its dynamic arguments. An SQL injection happens when the database server is mistakenly led to consider a dynamic argument of a query as part of the query text. Sending those parts as separate entities over the protocol means that SQL injection is no longer possible.

Geolocation with PostgreSQL

Fri, 24 Aug 2018 12:11:33 +0200

We have loaded Open Street Map points of interests in the article The Most Popular Pub Names — which compares PostgreSQL with MongoDB for simple geographical queries, and is part of our PostgreSQL Extensions article series. In today’s article, look at how to geolocalize an IP address and locate the nearest pub, all within a single SQL query!

For that, we are going to use the awesome ip4r extension from RhodiumToad.

PostgreSQL Concurrency: an Article Series

Tue, 14 Aug 2018 11:49:02 +0300

PostgreSQL is a relational database management system. It’s even the world’s most advanced open source one of them. As such, as its core, Postgres solves concurrent access to a set of data and maintains consistency while allowing concurrent operations.

In the PostgreSQL Concurrency series of articles here we did see several aspects of how to handle concurrent use cases of your application design with PostgreSQL. The main thing to remember is that a Database Management System first task is to handle concurrency access to the data for you.

Scheduled Data Processing: How to use cron?

Wed, 01 Aug 2018 11:24:25 +0200

A previous article in the PostgreSQL Concurrency series covered how to manage concurrent retweets in an efficient way: in Computing and Caching, we learnt how to maintain a cache right in your PostgreSQL database, using MATERIALIZED VIEWS. We also had a look at how to take care of Batch Updates and Concurrency.

While in the first case we are providing a solution to a technical problem where we want to solve performance issues while keeping the same semantics, in the second case we are actually implementing a part of the application’s Business Logic as a scheduled job.

Today’s article shows a modern technique to handle the scheduling of those business oriented activities that are not tied to any user activity. When thinking about it this way, you certainly don’t want to implement the backbone of your business logic in a shell script that’s directly maintained in the production environment, do you?

Batch Updates and Concurrency

Mon, 23 Jul 2018 22:45:43 +0200

This article fits in the PostgreSQL Concurrency series, where we installed a tweeter like application schema and had all the characters from Shakespeare’s A Midsummer Night’s Dream tweet their own lines in our database in PostgreSQL Concurrency: Data Modification Language.

A previous article in the series covered how to manage concurrent retweets in an efficient way: Computing and Caching, where we learn how to maintain a cache right in your PostgreSQL database, thanks for materialized views. We even went as far as maintaining an external cache in another application layer using PostgreSQL LISTEN and NOTIFY features and a Golang application.

Today’s article is going to address concurrency in the context of updating data in a batch. This activity is quite common, as soon as your system is connected to other systems either internally or with external providers. While it’s pretty easy to ingest new data, and easy enough to update data from an external source when nothing happens in your database, doing the operation safely with concurrent activity is more complex. Once more though, PostgreSQL comes with all the tooling you need to handle that situation.

PostgreSQL LISTEN/NOTIFY

Thu, 19 Jul 2018 12:58:21 +0200

Today’s article shows how to maintain an external cache in another application layer. In this article we are going to maintain an in-memory cache in a Golang service, using PostgreSQL LISTEN and NOTIFY features.

PostgreSQL Event Based Processing

Mon, 16 Jul 2018 09:27:54 +0200

In the previous article of the series Modeling for Concurrency, we saw how to model your application for highly concurrent activity. It was a follow-up to the article entitled PostgreSQL Concurrency: Isolation and Locking, which was a primer on PostgreSQL isolation and locking properties and behaviors.

Today’s article takes us a step further and builds on what we did in the previous articles in our series. After having had all the characters from Shakespeare’s A Midsummer Night’s Dream tweet their own lines in our database in PostgreSQL Concurrency: Data Modification Language, and having had them like and retweet a lot in PostgreSQL Concurrency: Isolation and Locking, we saw how to manage concurrent retweets in an efficient way in Computing and Caching.

What we did implement in the previous article is a cache system, all with its necessary cache invalidation policy. Sometimes though, the processing of an event needs to happen within the same transaction where the event is registered in your system. PostgreSQL makes it possible to maintain a summary table transactionally thanks to its trigger support. Today, we’re going to dive in how to maintain a summary table with triggers, and its impact on concurrency.

Computing and Caching

Fri, 13 Jul 2018 13:10:21 +0200

Let’s continue to dive in PostgreSQL Concurrency. In the previous article of the series, Modeling for Concurrency, we saw how to model your application for highly concurrent activity. It was a follow-up to the article entitled PostgreSQL Concurrency: Isolation and Locking, which was a primer on PostgreSQL isolation and locking properties and behaviors.

Today’s article takes us a step further and builds on what we did in the previous articles in our series. After having had all the characters from Shakespeare’s A Midsummer Night’s Dream tweet their own lines in our database in PostgreSQL Concurrency: Data Modification Language, and having had them like a retweet a lot in PostgreSQL Concurrency: Isolation and Locking, it’s time to think about how to display our counters in an efficient way.

In this article, we’re going to think about when we should compute results and when we should cache them for instant retrieval, all within the SQL tooling. The SQL tooling for handling cache is a MATERIALIZED VIEW, and it comes with cache invalidation routines, of course.

Modeling for Concurrency

Tue, 10 Jul 2018 10:26:47 +0200

Let’s continue to dive in PostgreSQL Concurrency. Last week’s article PostgreSQL Concurrency: Isolation and Locking was a primer on PostgreSQL isolation and locking properties and behaviors.

Today’s article takes us a step further and builds on what we did last week, in particular the database modeling for a tweet like application. After having had all the characters from Shakespeare’s A Midsummer Night’s Dream tweet their own lines in our database in PostgreSQL Concurrency: Data Modification Language, it’s time for them to do some actions on the tweets: likes and retweet.

Of course, we’re going to put concurrency to the test, so we’re going to have to handle very very popular tweets from the play!

PostgreSQL Concurrency: Isolation and Locking

Tue, 03 Jul 2018 13:30:13 +0200

This article is a primer on PostgreSQL Isolation and Locking properties and behaviors. You might be interested into the previous article in the series: PostgreSQL Concurrency: Data Modification Language.

PostgreSQL Concurrency: Data Modification Language

Mon, 25 Jun 2018 09:58:53 +0200

Postgres exposes its concurrency APIs in the SQL language, in particular in the DML parts of it: you can read the Data Manipulation Language chapter of the PostgreSQL docs for all the details.

PostgreSQL Data Types

Thu, 24 May 2018 14:47:05 +0200

Today it’s time to conclude our series of PostgreSQL Data Types articles with a recap. The series cover lots of core PostgreSQL data types and shows how to benefit from the PostgreSQL concept of a data type: more than input validation, a PostgreSQL data type also implements expected behaviors and processing functions.

This allows an application developer to rely on PostgreSQL for more complex queries, having the processing happen where the data is, for instance when implementing advanced JOIN operations, then retrieving only the data set that is interesting for the application.

PostgreSQL Data Types: Point

Mon, 07 May 2018 10:46:17 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL Point type.

In order to put the Point datatype in a context where it makes sense, we’re going to download a complete geolocation data set and normalize it, thus making good use of both the normalization good practice and those other PostgreSQL data types we’ve been learning about in the previous articles of this series.

Buckle-up, this is a long article with a lot of SQL inside.

PostgreSQL Data Types: ENUM

Wed, 02 May 2018 11:00:26 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL ENUM type.

This data type has been added to PostgreSQL in order to make it easier to support migrations from MySQL. Proper relational design would use a reference table and a foreign key instead.

PostgreSQL Data Types: JSON

Mon, 30 Apr 2018 09:49:33 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL JSON type.

PostgreSQL has built-in support for JSON with a great range of processing functions and operators, and complete indexing support. The documentation covers all the details in the chapters entitled JSON Types and JSON Functions and Operators.

PostgreSQL Data Types: XML

Mon, 23 Apr 2018 18:18:48 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL XML type.

The SQL standard includes a SQL/XML which introduces the predefined data type XML together with constructors, several routines, functions, and XML-to-SQL data type mappings to support manipulation and storage of XML in a SQL database, as per the Wikipedia page.

PostgreSQL Data Types: Arrays

Fri, 20 Apr 2018 14:47:25 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL array data types.

Arrays can be used to denormalize data and avoid lookup tables. A good rule of thumb for using them that way is that you mostly use the array as a whole, even if you might at times search for elements in the array. Heavier processing is going to be more complex than a lookup table.

PostgreSQL Data Types: Ranges

Wed, 18 Apr 2018 13:41:12 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL ranges data type.

Range types are a unique feature of PostgreSQL, managing two dimensions of data in a single column, and allowing advanced processing. The main example is the daterange data type, which stores as a single value a lower and an upper bound of the range as a single value. This allows PostgreSQL to implement a concurrent safe check against overlapping ranges, as we’re going to see in this article.

PostgreSQL Data Types: Network Addresses

Mon, 16 Apr 2018 12:32:53 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce network address types.

PostgreSQL includes support for both cidr, inet, and macaddr data types. Again, those types are bundled with indexing support and advanced functions and operator support.

PostgreSQL Data Types: Date and Time Processing

Fri, 13 Apr 2018 13:35:47 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce date and time based processing functions.

Once the application’s data, or rather the user data is properly stored as timestamp with time zone, PostgreSQL allows implementing all the processing you need to. In this article we dive into a set of examples to help you get started with time based processing in your database. Can we boost your reporting skills?

PostgreSQL Data Types: Text Processing

Wed, 11 Apr 2018 23:15:42 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce some of the PostgreSQL text processing functions.

There’s a very rich set of PostgreSQL functions to process text — you can find them all in the string functions and operators documentation chapter — with functions such as overlay(), substring(), position() or trim(). Or aggregates such as string_agg(). There are also regular expression functions, including the very powerful regexp_split_to_table(). In this article we see practical example putting them in practice.

PostgreSQL Data Types: Text Encoding

Mon, 09 Apr 2018 13:33:01 +0200

Continuing our series of PostgreSQL Data Types today we’re going to introduce the PostgreSQL text data type. The first notion to understand when processing text in any program is of course the notion of encoding.

So when addressing the text datatype we must mention encoding settings, and possibly also issues. An encoding is a particular representation of characters in bits and bytes. In the ASCII encoding the letter A is encoded as the 7-bits byte 1000001, or 65 in decimal, or 41 in hexadecimal. All those numbers are going to be written the same way on-disk, and the letter A too.

PostgreSQL Data Types: an intro

Fri, 06 Apr 2018 11:32:43 +0200

Today, we’re going to begin a dive into the PostgreSQL Data Types. As my colleague Will Leinweber said recently in his talk Constraints: a Developer’s Secret Weapon that he gave at pgDay Paris: database constraints in Postgres are the last line of defense.

The most important of those constraints is the data type, or the attribute domain in normalization slang. By declaring an attribute to be of a certain data type, then PostgreSQL ensures that this property is always true, and then implements advanced processing features for each data type, so that you may push the computation to the data, when needed.

This article is the first of a series that will go through many of the PostgreSQL data types, and we open the journey with boolean.

Object Relational Database Management System

Thu, 22 Mar 2018 17:40:39 +0100

PostgreSQL is the world’s most advanced open source database, and per the PostgreSQL Wikipedia page it is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards compliance.

In this article, we try to understand why would PostgreSQL be named an object-relational thing. What is Object Oriented Programming and how does that apply to a database system?

Database Normalization and Primary Keys

Fri, 09 Mar 2018 18:41:33 +0100

In our previous article we saw three classic Database Modelization Anti-Patterns. The article also contains a reference to a Primary Key section of my book The Art of PostgresQL, so it’s only fair that I would now publish said Primary Key section!

So in this article, we dive into Primary Keys as being a cornerstone of database normalization. It’s so important to get Primary Keys right that you would think everybody knows how to do it, and yet, most of the primary key constraints I’ve seen used in database design are actually not primary keys at all.

Database Modelization Anti-Patterns

Thu, 08 Mar 2018 18:00:43 +0100

Next week we see two awesome PostgreSQL conferences in Europe, back to back, with a day in between just so that people may attend both! In chronological order we have first Nordic pgDay in Oslo where I will have the pleasure to talk about Data Modeling, Normalization and Denormalization. Then we have pgday.paris with an awesome schedule and a strong focus on the needs of application developers!

Sustainable Open Source Development

Tue, 20 Feb 2018 21:08:38 +0100

Current trend in software deployments is to rely on open source software for entire production stacks. You can find open source software in the core technical stacks of every startup out there, I’m told. If you’re using Cloud based offerings, most of Cloud providers are running Free/Libre Open Source Software as their foundation.

This article is a deep dive into the economic models behind successful open source projects and communities, and how as a professional, enterprise grade user, you depend on the long-term sustainability of all the open source projects you’re using. And because you depend on the projects you’re using to be successful, how to contribute and guarantee their success.

Find the number of the longest continuously rising days for a stock

Tue, 06 Feb 2018 23:24:17 +0100

Today I want to react to an article that claims that Relational Algebra Is the Root of SQL Problems in which the author hand-waves the following position:

SQL becomes more a hindrance to data manipulation than an efficient tool. SQL’s greatest problem isn’t in the implementation level, but at its theory foundation. The problem can’t be solved by application optimization. Relational algebra isn’t sophisticated enough for handling the complicated data manipulation scenarios.

Then they go on to several arguments from authority to “prove” their point. My reading of the article is that SQL is very hard when you didn’t care to learn it, as most technologies are.

In this article, we’re going to look at the simple examples provided where apparently SQL makes it so much harder to find a solution compared to writing some Java or C++ code. Contrary to the original article, we go as far as to actually writing both the SQL solution and a complete Python solution, so that we can compare.

Exporting a Hierarchy in JSON: with recursive queries

Wed, 31 Jan 2018 18:00:01 +0100

In another article here, entitled on JSON and SQL, we saw in great details how to import a data set only available as a giant JSON file. Then we normalized the data set, so as to be able to write SQL and process our data. This approach is sometimes very useful and was a good way to learn some of the JSON functions provided by PostgreSQL.

In this article, we’re going to use SQL to export the data from our relational model into a JSON document. The trick that makes it complex in this example is that we have a recursive data model, with a notion of a parent row that exists in the same table as the current one. That’s a nice excuse to learn more about the SQL construct WITH RECURSIVE.

A Year in Review: Most Read Articles in 2017

Sun, 28 Jan 2018 22:46:43 +0100

It seems to be usual nowadays to review the previous year, and readers apparently like Top-N Lists — that’s you now, so let’s hope that my understanding works with you too.

Of course 2018 will see its own amount of new and original content added to this blog, with a continuous focus towards how to make the best out of the SQL powerful programming language, and its advanced concurrency semantics.

Working at Citus Data

Thu, 18 Jan 2018 12:05:13 +0100

You might have read it in the news already in Citus’ blog post by Sumedh Pathak: PostgreSQL Expert Dimitri Fontaine joins Citus Data. I am very happy to join a talented team here at Citus, and excited to work on an Open Source solution for distributed SQL on-top of PostgreSQL! In this article I’m going to cover my first technical contributions to Citus database, as it happens that a few patches of mine made it to the main source tree already.

TL;DR It’s good to be working on PostgreSQL related source code again, and to have the opportunity to solve PostgreSQL related problems at scale!

Migrating to PostgreSQL, the White Paper

Thu, 11 Jan 2018 10:01:45 +0100

After having been involved in many migration projects over the last 10 years, I decided to publish the following White Paper in order to share my learnings.

The paper is titled Migrating to PostgreSQL, Tools and Methodology and details the Continuous Migration approach. It describes how to migrate from another relational database server technology to PostgreSQL. The reasons to do so are many, and first among them is often the licensing model.

Submit a Talk to pgDay Paris Today!

Tue, 26 Dec 2017 16:06:34 +0100

pgDay Paris 4th edition happens March 15, 2018! And the Call for Papers is still open, as it closes with the year: the submission deadline is December 31st, 2017.

As I’ve been organizing the first edition of pgDay Paris back in 2015, this very conference is special to me! It’s been an amazing experience launching a new PostgreSQL conference…

Hats off to Magnus without whom the first edition would never have happened, and to Vik of course, who took over the organisation of the conference!

Setting up psql, the PostgreSQL CLI

Fri, 22 Dec 2017 15:23:43 +0100

PostgreSQL ships with an interactive console with the command line tool named psql. It can be used both for scripting and interactive usage and is moreover quite a powerful tool. Interactive features includes autocompletion, readline support (history searches, modern keyboard movements, etc), input and output redirection, formatted output, and more.

Mastering PostgreSQL: a reader's interview

Wed, 13 Dec 2017 17:05:56 +0100

Florent Fourcot has read Mastering PostgreSQL in Application Development and has seen tremendous inprovements in his production setup from reading the first chapters and applying the book advices to his use case.

Here’s an interview run with Florent where he explains the context in which such improvements has been made!

Mastering PostgreSQL: more about the docker image

Tue, 12 Dec 2017 23:27:55 +0100

The Enterprise Edition of Mastering PostgreSQL in Application Development ships with a docker image that hosts both a PostgreSQL server instance with a pre-loaded database, the one that’s used throughout the book examples, and also with a Jupyter Network notebook that hosts SQL queries thanks to the sql_magic plugin.

Queen, Princesses, and Workers

Mon, 11 Dec 2017 10:19:24 +0100

The PostgreSQL community made the explicit choice some times ago that they would not use the infamous master and slave terminology. Instead, the documentation introduces the concepts of High Availability, Load Balancing, and Replication with the terms Primary and Standby, and the even more generic term Replica is used in contexts when only the data flow is considered, rather than the particular role of a node.

Scaling Python released

Tue, 05 Dec 2017 18:10:00 +0100

Today I am very pleased to announce the release of the book Scaling Python from my good friend Julien Danjou!

As Julien says, Python applications can handle millions of requests. Well, we know here that it’s easier on them when they are using PostgreSQL of course!

Simple Data Modeling with a Test Data Set

Mon, 27 Nov 2017 16:23:44 +0100

In How to Write SQL we saw how to write SQL queries as separate .sql files, and we learnt about using query parameters with the psql syntax for that (:variable, :'variable', and :"identifier").

For writing our database model, the same tooling is all we need. An important aspect of using psql is its capacity to provide immediate feedback, and we can also have that with modeling too.

The Mode Ordered-Set Aggregate Function

Mon, 13 Nov 2017 18:15:51 +0100

In our article Exploring a Data Set in SQL we discovered a data set related to music: the Chinook sample database.

Our discovery led us to find albums containing tracks of multiple genres, and for the analytics we were then pursuing, we wanted to clean the data set and assign a single genre per album. We did that in SQL of course, and didn’t actually edit the data.

Finding the most frequent input value in a group is a job for the mode() WITHIN GROUP (ORDER BY sort_expression) Ordered-Set Aggregate Function, as documented in the PostgreSQL page about Aggregate Functions.

What's in a name: “Mastering”

Thu, 09 Nov 2017 09:34:36 +0100

Now that my book Mastering PostgreSQL in Application Development is released (and selling well, thanks guys!), I’ve had some questions about the title.

The idea is that to become good at anything, we need to practice. We practice a lot, and it’s even better when we are actively trying to learn, following what’s named deliberate practice.

Mastering PostgreSQL in Application Development launches!

Mon, 06 Nov 2017 10:48:43 +0100

Today is the day my book Mastering PostgreSQL in Application Development launches! I’m all excited that everybody interested is now able to actually read my book!

Mastering PostgreSQL in Application Development targets application developers who want to learn SQL properly, and actually master this programming language. Most developers don’t think of SQL as a programming language, mainly because they don’t have full control of the execution plan of their queries.

Buy it online!

When using SQL, you don’t get to write how to fetch the data you are interested in, your job is to describe the data set you need in the simplest possible way. Sometimes this is pretty easy: I want all the information we have in the table drivers about the driver whom driverid is 1 translates easily to

Set Returning Functions and PostgreSQL 10

Fri, 13 Oct 2017 13:25:21 +0200

PostgreSQL 10 is now available for everyone to use, and hinted by David Fetter I had to review my previous article on Json and SQL to adapt to Set Returning Functions changes.

A Set Returning Function is a PostgreSQL Stored Procedure that can be used as a relation: from a single call it returns an entire result set, much like a subquery or a table.

It used to be possible to use SRF in the SELECT clause, with dubious (but useful at times) semantics, and also in scalar contexts. The semantics have been fixed and are now much clearer, and the uses in scalar contexts are forbidden — they were a hack and never made sense anyway.

on Json and SQL

Mon, 18 Sep 2017 10:49:03 +0200

PostgreSQL has had proper json support for a while now. The unique extensibility approach of the PostgreSQL system allows it to enable native SQL friendly JSON processing.

In this article we’ll play with the Magic: the Gathering card data in JSON format data set, provided with a CC0 licence, and process the information provided. We also see how to normalize a JSON document into a proper database model that benefits from some PostgreSQL advanced features, and how to then inject the JSON documents into the normalized database schema. Finally, we compare some non-trivial processing done against both versions of the database schema.

Mastering PostgreSQL in Application Development

Mon, 11 Sep 2017 15:22:23 +0200

Mastering PostgreSQL in Application Development is the full title of the book I am currently writing. Running the PostgreSQL is YeSQL series of blog posts has shown me developers need a PostgreSQL book for developers. A book with the same properties as the YeSQL series articles in this blog:

we use real world data sets to put every query and SQL technique we learn in the context of a user story or business case,

Regular Expressions and Grouping Sets

Mon, 14 Aug 2017 16:37:53 +0200

There’s a very rich set of PostgreSQL functions to process text, you can find them all at the String Functions and Operators documentation chapter, with functions such as overlay, substring, position or trim. Or aggregates such as string_agg. And then regular expression functions, including the very powerful regexp_split_to_table.

SQL Regression Tests

Tue, 08 Aug 2017 17:55:51 +0200

In a previous article here we saw How to Write SQL in your application code. The main idea in that article is to maintain your queries in separate SQL files, where it’s easier to maintain them. In particular if you want to be able to test them again in production, and when you have to work and rewrite queries.

from MySQL to PostgreSQL

Thu, 06 Jul 2017 17:30:11 +0200

Today pgloader v3.4.1 is released and available! This new release comes with 110 commits as show in github compare view.

This release of pgloader is following the tradition of simplifying things for users, or if you allow me to quote Alan Kay, I believe that if simple things should be simple, complex things should be possible.

Playing with Unicode

Mon, 03 Jul 2017 14:32:29 +0200

The reason why I like Unicode a lot is because it allows me to code in text based environments and still have nice output. Today, we’re going to play with Regional Indicator Symbol, which is implemented as a Unicode combinaison of letters from 🇦 to 🇿. For instance, if you display 🇫 then 🇷 concatenated together, you get 🇫🇷. Let’s try that from our PostgreSQL prompt!

PostgreSQL and the calendar

Fri, 30 Jun 2017 14:35:59 +0200

The modern calendar is a trap for the young engineer’s mind. We deal with the calendar on a daily basis and until exposed to its insanity it’s rather common to think that calendar based computations are easy. That’s until you’ve tried to do it once. A very good read about how the current calendar came to be the way it is now is Erik’s Naggum The Long, Painful History of Time.

SQL and Business Logic

Mon, 19 Jun 2017 13:30:19 +0200

Business logic is supposed to be the part of the application where you deal with customer or user facing decisions and computations. It is often argued that this part should be well separated from the rest of the technical infrastructure of your code. Of course, SQL and relational database design is meant to support your business cases (or user stories), so then we can ask ourselves if SQL should be part of your business logic implementation. Or actually, how much of your business logic should be SQL?

Exploring a Data Set in SQL

Tue, 13 Jun 2017 13:47:08 +0200

Sometimes you need to dive in an existing data set that you know very little about. Let’s say we’ve been lucky to have had a high level description of the business case covered by a database, and then access to it. Our next step is figuring out data organisation, content and quality. Our tool box: the world’s most advanced open source database, PostgreSQL, and its Structured Query Language, SQL.

How to Write SQL

Thu, 08 Jun 2017 13:23:26 +0200

Kris Jenkins cooked up a very nice way to embed SQL in your code: YeSQL for Clojure. The main idea is that you should be writing your SQL queries in .sql files in your code repository and maintain them there.

The idea is very good and it is now possible to find alternative implementations of the Clojure yesql library in other languages. Today, we are going to have a look at one of them for the python programming language: anosql.

Find The Missing Integer

Tue, 30 May 2017 19:56:54 +0200

A recent interview question that I had to review was spelled like this:

Find missing int element into array 1..100

Of course at first read I got it wrong, you have only one integer to look for into the array. So while the obvious idea was to apply classic sorting techniques and minimize array traversal to handle complexity (time and space), it turns out there’s a much simpler way to do it if you remember your math lessons from younger. But is it that much simpler?

Meetup PostgreSQL à Paris

Thu, 02 Oct 2014 11:49:00 +0200

Mercredi 8 octobre se tiendra le prochain Meetup PostgreSQL à Paris dans les locaux de Mozilla Europe, dont la capacité est de 90 personnes ! Venez nombreux !

PostgreSQL, Aggregates and Histograms

Fri, 21 Feb 2014 13:25:00 +0100

In our previous article Aggregating NBA data, PostgreSQL vs MongoDB we spent time comparing the pretty new MongoDB Aggregation Framework with the decades old SQL aggregates. Today, let’s showcase more of those SQL aggregates, producing a nice histogram right from our SQL console.

Aggregating NBA data, PostgreSQL vs MongoDB

Mon, 17 Feb 2014 23:40:00 +0100

When reading the article Crunching 30 Years of NBA Data with MongoDB Aggregation I coulnd’t help but think that we’ve been enjoying aggregates in SQL for 3 or 4 decades already. When using PostgreSQL it’s even easy to actually add your own aggregates given the SQL command create aggregate.

Denormalizing Tags

Thu, 24 Oct 2013 13:40:00 +0200

In our Tour of Extensions today’s article is about advanced tag indexing. We have a great data collection to play with and our goal today is to be able to quickly find data matching a complex set of tags. So, let’s find out those lastfm tracks that are tagged as blues and rhythm and blues, for instance.

We’re going to use the Last.fm dataset from the Million Song Dataset project here. The article where they present their similarity searches is providing the whole dataset they have as a SQLite database and they propose to read their python script demo_tags_db.py to make sense of it.

An Interview about MariaDB and PostgreSQL

Wed, 16 Oct 2013 21:07:00 +0200

At the Open World Forum two weeks ago I had the pleasure to meet with Colin Charles. We had a nice talk about the current state of both MariaDB and PostgreSQL, and even were both interviewed by the Open World Forum Team. The interview is now available online. Dear French readers, it’s in English.

Here’s the video:

Executive Summary: MariaDB is a drop-in fully Open Source replacement for MySQL and sees quite some progress and innovation being made, and PostgreSQL is YeSQL!

PostgreSQL Autonomous Transaction

Mon, 14 Oct 2013 11:25:00 +0200

PostgreSQL is an all round impressive Relational DataBase Management System which implements the SQL standard (see the very useful reference page Comparison of different SQL implementations for details). PostgreSQL also provides with unique solutions in the database market and has been leading innovation for some years now. Still, there’s no support for Autonomous Transactions within the server itself. Let’s have a look at how to easily implement them with PL/Proxy.

PostgreSQL data recovery

Tue, 17 Sep 2013 10:39:00 +0200

The following story is only interesting to read if you like it when bad things happen, or if you don’t have a trustworthy backup policy in place. By trustworthy I mean that each backup you take must be tested with a test recovery job. Only tested backups will prove useful when you need them. So go read our Backup and Restore documentation chapter then learn how to setup Barman for handling physical backups and Point In Time Recovery. Get back when you have proper backups, including recovery testing in place. We are waiting for you. Back? Ok, let’s see how bad you can end up without backups, and how to still recover. With luck.

Using trigrams against typos

Fri, 06 Sep 2013 16:15:00 +0200

In our ongoing Tour of Extensions we played with earth distance in How far is the nearest pub? then with hstore in a series about trigger, first to generalize Trigger Parameters then to enable us to Auditing Changes with Hstore. Today we are going to work with pg_trgm which is the trigrams PostgreSQL extension: its usage got seriously enhanced in recent PostgreSQL releases and it’s now a poor’s man Full Text Search engine.

Auditing Changes with Hstore

Tue, 27 Aug 2013 17:35:00 +0200

In a previous article about Trigger Parameters we have been using the extension hstore in order to compute some extra field in our records, where the fields used both for the computation and for storing the results were passed in as dynamic parameters. Today we’re going to see another trigger use case for hstore: we are going to record changes made to our tuples.

Trigger Parameters

Fri, 23 Aug 2013 12:08:00 +0200

Sometimes you want to compute values automatically at INSERT time, like for example a duration column out of a start and an end column, both timestamptz. It’s easy enough to do with a BEFORE TRIGGER on your table. What’s more complex is to come up with a parametrized spelling of the trigger, where you can attach the same stored procedure to any table even when the column names are different from one another.

Understanding Window Functions

Tue, 20 Aug 2013 12:04:00 +0200

There was SQL before window functions and SQL after window functions: that’s how powerful this tool is. Being that of a deal breaker unfortunately means that it can be quite hard to grasp the feature. This article aims at making it crystal clear so that you can begin using it today and are able to reason about it and recognize cases where you want to be using window functions.

*We see a part of the data as if through a little window*

How far is the nearest pub?

Mon, 05 Aug 2013 08:11:00 +0200

In our recent article about The Most Popular Pub Names we did have a look at how to find the pubs nearby, but didn’t compute the distance in between that pub and us. That’s because how to compute a distance given a position on the earth expressed as longitude and latitude is not that easy. Today, we are going to solve that problem nonetheless, thanks to PostgreSQL Extensions.

The Most Popular Pub Names

Fri, 02 Aug 2013 10:19:00 +0200

In his article titled The Most Popular Pub Names Ross Lawley did show us how to perform some quite interesting geographic queries against MongoDB, using some nice Open Data found at the Open Street Map project.

Archiving data as fast as possible

Fri, 05 Jul 2013 15:30:00 +0200

In a recent article here we’ve been talking about how do do Batch Updates in a very efficient way, using the Writable CTE features available in PostgreSQL 9.1. I sometime read how Common Table Expressions changed the life of fellow DBAs and developers, and would say that Writable CTE are at least the same boost again.

Simple Case for Pivoting in SQL

Thu, 04 Jul 2013 15:55:00 +0200

In a recent article Craig Kerstiens from Heroku did demo the really useful crosstab extension. That function allows you to pivot a table so that you can see the data from different categories in separate columns in the same row rather than in separate rows. The article from Craig is Pivoting in Postgres.

*Pivoting a matrix, also known as a matrix transposition*

Let’s do the same setup as he did, with a table containing some randomly generated data about hypothetical visits to a web page, say, by date then by operating system.

Nearest Big City

Thu, 02 May 2013 11:34:00 +0200

In this article, we want to find the town with the greatest number of inhabitants near a given location.

A very localized example

We first need to find and import some data, and I found at the following place a CSV listing of french cities with coordinates and population and some numbers of interest for the exercise here.

To import the data set, we first need a table, then a COPY command:

Bulk Replication

Mon, 18 Mar 2013 14:54:00 +0100

In the previous article here we talked about how to properly update more than one row at a time, under the title Batch Update. We did consider performances, including network round trips, and did look at the behavior of our results when used concurrently.

A case where we want to apply the previous article approach is when replicating data with a trigger based solution, such as SkyTools and londiste. Well, maybe not in all cases, we need to have a amount of UPDATE trafic worthy of setting up the solution. As soon as we know we’re getting to replay important enough batches of events, though, certainly using the batch update tricks makes sense.

Batch Update

Fri, 15 Mar 2013 10:47:00 +0100

Performance consulting involves some tricks that you have to teach over and over again. One of them is that SQL tends to be so much better at dealing with plenty of rows in a single statement when compared to running as many statements, each one against a single row.

HyperLogLog Unions

Tue, 26 Feb 2013 12:44:00 +0100

In the article from yesterday we talked about PostgreSQL HyperLogLog with some details. The real magic of that extension has been skimmed over though, and needs another very small article all by itself, in case you missed it.

*Which Set Operation do you want for counting unique values?*

The first query here has the default level of magic in it, really. What happens is that each time we do an update of the HyperLogLog hash value, we update some data which are allowing us to compute its cardinality.

PostgreSQL HyperLogLog

Mon, 25 Feb 2013 10:23:00 +0100

If you’ve been following along at home the newer statistics developments, you might have heard about this new State of The Art Cardinality Estimation Algorithm called HyperLogLog. This technique is now available for PostgreSQL in the extension postgresql-hll available at https://github.com/aggregateknowledge/postgresql-hll and soon to be in debian.

Extensions Templates

Tue, 08 Jan 2013 17:53:00 +0100

In a recent article titled Inline Extensions we detailed the problem of how to distribute an extension’s package to a remote server without having access to its file system at all. The solution to that problem is non trivial, let’s say. But thanks to the awesome PostgreSQL Community we finaly have some practical ideas on how to address the problem as discussed on pgsql-hackers, our development mailing list.

*PostgreSQL is first an Awesome Community*

The solution we talked about is to use templates, and so I’ve been working on a patch to bring templates for extensions to PostgreSQL. As we’re talking about 3 new system catalogs, that’s a big patch in term of lines of code. In term of features though, it’s quite an easy one.

Inline Extensions

Thu, 13 Dec 2012 11:34:00 +0100

We’ve been having the CREATE EXTENSION feature in PostgreSQL for a couple of releases now, so let’s talk about how to go from here. The first goal of the extension facility has been to allow for a clean dump and restore process of contrib modules. As such it’s been tailored to the needs of deploying files on the file system because there’s no escaping from that when you have to ship binary and executable files, those infamous .so, .dll or .dylib things.

Prefixes and Ranges

Tue, 16 Oct 2012 10:47:00 +0200

It’s been a long time since I last had some time to spend on the prefix PostgreSQL extension and its prefix_range data type. With PostgreSQL 9.2 out, some users wanted me to update the extension for that release, and hinted me that it was high time that I fix that old bug for which I already had a patch.

`prefix_range` release 1.2.0

I’m sorry it took that long. It’s now done, you can have prefix 1.2.0 from https://github.com/dimitri/prefix or if you want a tagged tarball then you can use this link: https://github.com/dimitri/prefix/tarball/v1.2.0.

Reset Counter

Fri, 05 Oct 2012 09:44:00 +0200

I’ve been given a nice puzzle that I think is a good blog article opportunity, as it involves some thinking and window functions.

PostgreSQL 9.3

Sat, 15 Sep 2012 18:43:00 +0200

PostgreSQL 9.2 is released! It’s an awesome new release that I urge you to consider trying and adopting, an upgrade from even 9.1 should be very well worth it, as your hardware could suddenly be able to process a much higher load. Indeed, better performances mean more work done on the same budget, that’s the name of the game!

As a PostgreSQL contributor though, the release of 9.2 mainly means to me that it’s time to fully concentrate on preparing 9.3. The developement season of which as already begun, by the way, so some amount of work has already been done here.

Clean PGQ Subconsumers

Thu, 26 Apr 2012 15:05:00 +0200

Now that you’re all using the wonders of Cooperative Consumers to help you efficiently and reliably implement your business constraints and offload them from the main user transactions, you’re reaching a point where you have to clean up your development environment (because that’s what happens to development environments, right?), and you want a way to start again from a clean empty place.

Here we go. It used to be much more simple than that, so if you’re still using PGQ from Skytools2, just jump to the next step.

PGQ Coop Consumers

Mon, 12 Mar 2012 14:43:00 +0100

While working a new PostgreSQL architecture for an high scale project that used to be in the top 10 of internet popular web sites (in terms of visitors), I needed to be able to off load some processing from the main path: that’s called a batch job. This needs to be transactional: don’t run the job if we did rollback; the transaction, process all events that were part of the same transaction in the same transaction, etc.

Extension White Listing

Thu, 08 Mar 2012 14:25:00 +0100

PostgreSQL 9.1 includes proper extension support, as you might well know if you ever read this very blog here. Some hosting facilities are playing with PostgreSQL at big scale (hello Heroku!) and still meet with small caveats making their life uneasy.

To be specific, only superusers are allowed to install C coded stored procedures, and that impacts a lot of very useful PostgreSQL extension: all those shiped in the contrib package are coded in C. Now, Heroku is not giving away superuser access to their hosted customers in order to limit the number of ways they can shoot themselves in the foot. And given PostgreSQL security model, being granted database owner is mostly good enough for day to day operation.

pgbouncer munin plugin

Wed, 16 Nov 2011 14:00:00 +0100

It seems that if you search for a munin plugin for pgbouncer it’s easy enough to reach an old page of mine with an old version of my plugin, and a broken link. Let’s remedy that by publishing here the newer version of the plugin. To be honest, I though it already made its way into the official munin 1.4 set of plugins, but I’ve not been following closely enough.

As the plugin is 300 lines of python code, it’s not a good idea to just inline it here, so please grab it at pgbouncer_.

Extensions en simple SQL

Mon, 31 Oct 2011 14:22:00 +0100

La conférence européenne à Amsterdam était un très bon évènement de la communauté, avec une organisation impeccable dans un hôtel accueillant. J’ai eu le plaisir d’y parler des extensions et de leur usage dans le cadre du développement applicatif « interne », sous le titre Extensions are good for business logic.

L’idée de ma présentation, que la plupart d’entre vous a loupé je suppose (en tout cas je n’avais qu’une petite poignée de français dans la salle, et j’espère avoir des lecteurs qui n’étaient pas à Amsterdam), l’idée est d’utiliser les mécanismes offerts par les extensions afin de maintenir le code PL que vous utilisez en production.

Scaling Stored Procedures

Thu, 06 Oct 2011 18:23:00 +0200

In the news recently stored procedures where used as an excuse for moving away logic from the database layer to application layer, and to migrate away from a powerful technology to a simpler one, now that there’s no logic anymore in the database.

It’s not the way I would typically approach scaling problems, and apparently I’m not alone on the Stored Procedures camp. Did you read this nice blog post Mythbusters: Stored Procedures Edition already? Well it happens in another land that where my comfort zone is, but still has some interesting things to say.

Skytools3: walmgr

Wed, 21 Sep 2011 17:21:00 +0200

Let’s begin the Skytools 3 documentation effort, which is long overdue. The code is waiting for you over at github, and is stable and working. Why is it still in release candidate status, I hear you asking? Well because it’s missing updated documentation.

WalMgr is the Skytools component that manages WAL shipping for you, and archiving too. It knows how to prepare your master and standby setup, how to take a base backup and push it to the standby’s system, how to archive (at the satndby) master’s WAL files as they are produced and have the standby restore from this archive.

PostgreSQL 9.1

Wed, 14 Sep 2011 10:00:00 +0200

PostgreSQL 9.1 est dans les bacs ! Vous n’avez pas encore cette nouvelle version en production ? Pas encore évalué pourquoi vous devriez envisager de migrer à cette version ? Il existe beaucoup de bonnes raisons de passer à cette version, et peu de pièges.

Nous commençons à lire des articles qui reprennent la nouvelle dans la presse française, et j’ai le plaisir de mentionner celui de programmez.com qui annonce « un système d’extensions inégalé ». En tant que développeur des Extensions dans PostgreSQL, je ne peux qu’être non seulement d’accord avec eux, mais aussi flatté :)

Éviter les injections SQL

Wed, 07 Sep 2011 11:36:00 +0200

Nous avons parlé la dernière fois les règles d' échappement de chaînes avec PostgreSQL, et mentionné qu’utiliser ces techniques afin de protéger les données insérées dans les requêtes SQL n’était pas une bonne idée dans la mesure où PostgreSQL offre une fonctionnalité bien plus adaptée.

Nous faisons face ici à un problème de sécurité très bien décrit dans le billet humoristique de Little Boby Tables, dont je vous recommande la lecture. L’idée est simple, la mise en place de contre mesure fourmille de pièges subtils, à moins d’utiliser la solution décrite ci-après.

Skytools, version 3

Fri, 26 Aug 2011 21:30:00 +0200

You can find skytools3 in debian experimental already, it’s in release candidate status. What’s missing is the documentation, so here’s an idea: I’m going to make a blog post series about skytools next features, how to use them, what they are good for, etc. This first article of the series will just list what are those new features.

Here are the slides from the CHAR(11) talk I made last month, about that very subject:

Échappement de chaînes

Thu, 18 Aug 2011 19:01:00 +0200

Parmis les nouveautés de la prochaine version de PostgreSQL, la fameuse 9.1, il faut signaler le changement de valeur par défaut de la variable standard_conforming_strings, qui passe à vraie.

En effet, l’utilisation d’échappements avec le caractère « anti-slash » n’est pas conforme au standard SQL. Le paramètre standard_conforming_strings permet de contrôler le comportement de PostgreSQL lorsqu’il lit une chaîne de caractère dans une requête SQL.

Voyons quelques exemples :

dimitri=# set standard_conforming_strings to true;
SET
dimitri=# select 'hop''';
 ?column? 
----------
 hop'
(1 ligne)

dimitri=# select 'hop\'';
dimitri'# ';
 ?column? 
----------
 hop\';
 
(1 ligne)

dimitri=# select E'hop\'';
 ?column? 
----------
 hop'
(1 ligne)

dimitri=# set standard_conforming_strings to false;
SET
dimitri=# select E'hop\'';
 ?column? 
----------
 hop'
(1 ligne)

dimitri=# select 'hop\'';
ATTENTION: utilisation non standard de \' dans une cha&#xEE;ne litt&#xE9;rale
LIGNE 1 : select 'hop\'';
 ^
ASTUCE : Utilisez '' pour &#xE9;crire des guillemets dans une cha&#xEE;ne ou utilisez
la syntaxe de cha&#xEE;ne d'&#xE9;chappement (E'...').
 ?column? 
----------
 hop'
(1 ligne)

Il existe un moyen de forcer PostgreSQL à accepter l’utilisation d’échappements avec « anti-slash » indépendamment de la valeur de standard_conforming_strings, c’est la notation préfixée avec E. Il est recommandé de toujours l’utiliser dès lors que la chaîne de caractère contient des « anti-slash » utilisés comme échappement (du caractère simple guillemet en général).

See Tsung in action

Tue, 02 Aug 2011 10:30:00 +0200

Tsung is an open-source multi-protocol distributed load testing tool and a mature project. It’s been available for about 10 years and is built with the Erlang system. It supports several protocols, including the PostgreSQL one.

When you want to benchmark your own application, to know how many more clients it can handle or how much gain you will see with some new shiny hardware, Tsung is the tool to use. It will allow you to record a number of sessions then replay them at high scale.
pgfouine supports Tsung and is able to turn your PostgreSQL logs into Tsung sessions, too.

Next month partitions

Wed, 27 Jul 2011 22:35:00 +0200

When you do partition your tables monthly, then comes the question of when to create next partitions. I tend to create them just the week before next month and I have some nice nagios scripts to alert me in case I’ve forgotten to do so. How to check that by hand in the end of a month?

Preparing for PGCON

Thu, 12 May 2011 10:30:00 +0200

It’s this time of the year again, the main international PostgreSQL Conference is next week in Ottawa, Canada. If previous years are any indication, this will be great event where to meet with a lot of the members of your community. The core team will be there, developers will be there, and we will meet with users and their challenging use cases.

This is a very good time to review both what you did in the project those last 12 months, and what you plan to do next year. To help with that, several meeting events are organized. They’re like a whole-day round table with a kind of an agenda, with a limited number of invited people in, and very intense on-topic discussions about how to organize ourselves for another great year of innovation in PostgreSQL.

Tables and Views dependencies

Wed, 04 May 2011 11:45:00 +0200

Let’s say you need to ALTER TABLE foo ALTER COLUMN bar TYPE bigint;, and PostgreSQL is helpfully telling you that no you can’t because such and such views depend on the column. The basic way to deal with that is to copy paste from the error message the names of the views involved, then prepare a script wherein you first DROP VIEW then ALTER TABLE and finally CREATE VIEW again, all in the same transaction.

Dynamic Triggers in PLpgSQL

Wed, 24 Nov 2010 16:45:00 +0100

You certainly know that implementing dynamic triggers in PLpgSQL is impossible. But I had a very bad night, being up from as soon as 3:30 am today, so that when a developer asked me about reusing the same trigger function code from more than one table and for a dynamic column name, I didn’t remember about it being impossible.

Here’s what happens in such cases, after a long time on the problem (yes, overall, that’s a slow day). Note that I’m abusing the (record_literal).* notation a lot in there, and even the (record_literal).column_name too.

pg_basebackup

Sun, 07 Nov 2010 13:45:00 +0100

Hannu just gave me a good idea in this email on -hackers, proposing that pg_basebackup should get the xlog files again and again in a loop for the whole duration of the base backup. That’s now done in the aforementioned tool, whose options got a little more useful now:

Usage: pg_basebackup.py [-v] [-f] [-j jobs] "dsn" dest

Options:
 -h, --help show this help message and exit
 --version show version and quit
 -x, --pg_xlog backup the pg_xlog files
 -v, --verbose be verbose and about processing progress
 -d, --debug show debug information, including SQL queries
 -f, --force remove destination directory if it exists
 -j JOBS, --jobs=JOBS how many helper jobs to launch
 -D DELAY, --delay=DELAY
 pg_xlog subprocess loop delay, see -x
 -S, --slave auxilliary process
 --stdin get list of files to backup from stdin

Yeah, as implementing the xlog idea required having some kind of parallelism, I built on it and the script now has a --jobs option for you to setup how many processes to launch in parallel, all fetching some base backup files in its own standard ( libpq) PostgreSQL connection, in compressed chunks of 8 MB (so that’s not 8 MB chunks sent over).

Extensions: writing a patch for PostgreSQL

Fri, 15 Oct 2010 11:30:00 +0200

These days, thanks to my community oriented job, I’m working full time on a PostgreSQL patch to terminate basic support for extending SQL. First thing I want to share is that patching the backend code is not as hard as one would think. Second one is that git really is helping.

“Not as hard as one would think, are you kidding me?”, I hear some say. Well, that’s true. It’s C code in there, but with a very good layer of abstractions so that you’re not dealing with subtle problems that much. Of course it happens that you have to, and managing the memory isn’t an option. That said, palloc() and the memory contexts implementation makes that as easy as in lots of cases, you don’t have to think about it.

Date puzzle for starters

Fri, 08 Oct 2010 10:00:00 +0200

The PostgreSQL IRC channel is a good place to be, for all the very good help you can get there, because people are always wanting to remain helpful, because of the off-topics discussions sometime, or to get to talk with community core members. And to start up your day too.

This morning’s question started simple : “how can I check if today is the “first sunday fo the month”. or “the second tuesday of the month” etc?”

Resuming work on Extensions, first little step

Thu, 07 Oct 2010 17:15:00 +0200

Yeah I’m back on working on my part of the extension thing in PostgreSQL.

First step is a little one, but as it has public consequences, I figured I’d talk about it already. I’ve just refreshed my git repository to follow the new master one, and you can see that here http://git.postgresql.org/gitweb?p=postgresql-extension.git;a=commitdiff;h=9a88e9de246218e93c04b6b97e1ef61d97925430.

It’s been easier than I feared, mainly:

$ git --no-pager diff master..extension
$ git --no-pager format-patch master..extension
$ cp 0001-First-stab-at-writing-pg_execute_from_file-function.patch ..
$ git checkout master
$ git pull -f pgmaster
$ git reset --hard pgmaster/master
$ git checkout extension
$ git reset --hard master
$ git am -s ../0001-First-stab-at-writing-pg_execute_from_file-function.edit.patch 
$ git status
$ git log --short | head
$ git log -n2 --oneline
$ git push -f

So that’s still more steps that one want to call dead simple, but still. The format-patch command is to save my work away (all patches that are in the extension branch but not in the master — well that was only one of them here). Then, as the master repository URL didn’t change, I can simply pull the changes in. Of course I had a nice message warning: no common commits.

Window Functions example remix

Sun, 12 Sep 2010 21:35:00 +0200

The drawback of hosting a static only website is, obviously, the lack of comments. What happens actually, though, is that I receive very few comments by direct mail. As I don’t get another spam source to cleanup, I’m left unconvinced that’s such a drawback. I still miss the low probability of seeing blog readers exchange directly, but I think a tapoueh.org mailing list would be my answer, here…

Anyway, David Fetter took the time to send me a comment by mail with a cleaned up rewrite of the previous entry SQL, here’s it for your pleasure!

Window Functions example

Thu, 09 Sep 2010 16:35:00 +0200

So, when 8.4 came out there was all those comments about how getting window functions was an awesome addition. Now, it seems that a lot of people seeking for help in #postgresql just don’t know what kind of problem this feature helps solving. I’ve already been using them in some cases here in this blog, for getting some nice overview about Partitioning: relation size per “group”.

*That's another way to count change*

Now, another example use case rose on IRC today. I’ll quote directly our user here:

Synchronous Replication

Mon, 06 Sep 2010 18:05:00 +0200

Although the new asynchronous replication facility that ships with 9.0 ain’t released to the wide public yet, our hackers hero are already working on the synchronous version of it. A part of the facility is rather easy to design, we want something comparable to DRBD flexibility, but specific to our database world. So synchronous would either mean recv, fsync or apply, depending on what you need the standby to have already done when the master acknowledges the COMMIT. Let’s call that the service level.

Happy Numbers

Mon, 30 Aug 2010 11:00:00 +0200

After discovering the excellent Gwene service, which allows you to subscribe to newsgroups to read RSS content ( blogs, planets, commits, etc), I came to read this nice article about Happy Numbers. That’s a little problem that fits well an interview style question, so I first solved it yesterday evening in Emacs Lisp as that’s the language I use the most those days.

A happy number is defined by the following process. Starting with any positive integer, replace the number by the sum of the squares of its digits, and repeat the process until the number equals 1 (where it will stay), or it loops endlessly in a cycle which does not include 1. Those numbers for which this process ends in 1 are happy numbers, while those that do not end in 1 are unhappy numbers (or sad numbers).

Playing with bit strings

Thu, 26 Aug 2010 17:45:00 +0200

The idea of the day ain’t directly from me, I’m just helping with a very thin subpart of the problem. The problem, I can’t say much about, let’s just assume you want to reduce the storage of MD5 in your database, so you want to abuse bit strings. A solution to use them works fine, but the datatype is still missing some facilities, for example going from and to hexadecimal representation in text.

Editing constants in constraints

Mon, 09 Aug 2010 14:45:00 +0200

We’re using constants in some constraints here, for example in cases where several servers are replicating to the same federating one: each origin server has his own schema, and all is replicated nicely on the central host, thanks to Londiste, as you might have guessed already.

Querying the Catalog to plan an upgrade

Thu, 05 Aug 2010 11:00:00 +0200

Some user on IRC was reading the releases notes in order to plan for a minor upgrade of his 8.3.3 installation, and was puzzled about potential needs for rebuilding GIST indexes. That’s from the 8.3.5 release notes, and from the 8.3.8 notes you see that you need to consider hash indexes on interval columns too. Now the question is, how to find out if any such beasts are in use in your database?

Database Virtual Machines

Tue, 03 Aug 2010 13:30:00 +0200

Today I’m being told once again about SQLite as an embedded database software. That one ain’t a database server but a software library that you can use straight into your main program. I’m yet to use it, but it looks like its SQL support is good enough for simple things — and that covers loads of things. I guess read-only cache and configuration storage would be the obvious ones, because it seems that SQLite use cases aren’t including mixed concurrency, that is workloads with concurrent readers and writers.

Partitioning: relation size per “group”

Mon, 26 Jul 2010 17:00:00 +0200

This time, we are trying to figure out where is the bulk of the data on disk. The trick is that we’re using DDL partitioning, but we want a “nice” view of size per partition set. Meaning that if you have for example a parent table foo with partitions foo_201006 and foo_201007, you would want to see a single category foo containing the accumulated size of all the partitions underneath foo.

Background writers

Mon, 19 Jul 2010 16:30:00 +0200

There’s currently a thread on hackers about bg worker: overview and a series of 6 patches. Thanks a lot Markus! This is all about generalizing a concept already in use in the autovacuum process, where you have an independent subsystem that require having an autonomous daemon running and able to start its own workers.

I’ve been advocating about generalizing this concept for awhile already, in order to have postmaster able to communicate to subsystems when to shut down and start and reload, etc. Some external processes are only external because there’s no need to include them by default in to the database engine, not because there’s no sense to having them in there.

Logs analysis

Tue, 13 Jul 2010 14:15:00 +0200

Nowadays to analyze logs and provide insights, the more common tool to use is pgfouine, which does an excellent job. But there has been some improvements in logs capabilities that we’re not benefiting from yet, and I’m thinking about the CSV log format.

So the idea would be to turn pgfouine into a set of SQL queries against the logs themselves once imported into the database. Wait. What about having our next PostgreSQL version, which is meant (I believe) to include CSV support in SQL/MED, to directly expose its logs as a system view?

Using indexes as column store?

Thu, 08 Jul 2010 11:15:00 +0200

There’s a big trend nowadays about using column storage as opposed to what PostgreSQL is doing, which would be row storage. The difference is that if you have the same column value in a lot of rows, you could get to a point where you have this value only once in the underlying storage file. That means high compression. Then you tweak the executor to be able to load this value only once, not once per row, and you win another huge source of data traffic (often enough, from disk).

Back from PgCon2010

Thu, 27 May 2010 14:26:00 +0200

This year’s edition has been the best pgcon ever for me. Granted, it’s only my third time, but still :) As Josh said the “Hall Track” in particular was very good, and the Dev Meeting has been very effective!

Extensions

This time I prepared some slides to present the extension design and I tried hard to make it so that we get to agree on a plan, even recognizing it’s not solving all of our problems from the get go. I had been talking about the concept and design with lots of people already, and continued to do so while in Ottawa on Monday evening and through all Tuesday. So Wednesday, I felt prepared. It proved to be a good thing, as I edited the slides with ideas from several people I had the chance to expose my ideas to! Thanks Greg Stark and Heikki Linnakangas for the part we talked about at the meeting, and a lot more people for the things we’ll have to solve later (Hi Stefan!).

Finding orphaned sequences

Wed, 17 Mar 2010 13:35:00 +0100

This time we’re having a database where sequences were used, but not systematically as a default value of a given column. It’s mainly an historic bad idea, but you know the usual excuse with bad ideas and bad code: the first 6 months it’s experimental, after that it’s historic.

*Not talking about genome orphaned sequences here, though*

Still, here’s a query for 8.4 that will allow you to list those sequences you have that are not used as a default value in any of your tables:

Getting out of SQL_ASCII, part 2

Tue, 23 Feb 2010 17:30:00 +0100

So, if you followed the previous blog entry, now you have a new database containing all the static tables encoded in UTF-8 rather than SQL_ASCII. Because if it was not yet the case, you now severely distrust this non-encoding.

Now is the time to have a look at properly encoding the live data, those stored in tables that continue to receive write traffic. The idea is to use the UPDATE facilities of PostgreSQL to tweak the data, and too fix the applications so as not to continue inserting badly encoded strings in there.

Getting out of SQL_ASCII, part 1

Thu, 18 Feb 2010 11:37:00 +0100

It happens that you have to manage databases designed by your predecessor, and it even happens that the team used to not have a DBA. Those histerical raisins can lead to having a SQL_ASCII database. The horror!

What SQL_ASCII means, if you’re not already familiar with the consequences of such a choice, is that all the text and varchar data that you put in the database is accepted as-is. No checks. At all. It’s pretty nice when you’re lazy enough to not dealing with strange errors in your application, but if you think that t’s a smart move, please go read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky now. I said now, I’m waiting for you to get back here. Yes, I’ll wait.

Resetting sequences. All of them, please!

Tue, 16 Feb 2010 16:23:00 +0100

So, after restoring a production dump with intermediate filtering, none of our sequences were set to the right value. I could have tried to review the process of filtering the dump here, but it’s a one-shot action and you know what that sometimes mean. With some pressure you don’t script enough of it and you just crawl more and more.

Still, I think how I solved it is worthy of a blog entry. Not that it’s about a super unusual clever trick, quite the contrary, because questions involving this trick are often encountered on the support IRC.

PgCon 2009

Wed, 27 May 2009 14:30:00 +0200

I can’t really compare PgCon 2009 with previous years versions, last time I enjoyed the event it was in 2006, in Toronto. But still I found the experience to be a great one, and I hope I’ll be there next year too!

I’ve met a lot of known people in the community, some of them I already had the chance to run into at Toronto or Prato, but this was the first time I got to talk to many of them about interresting projects and ideas. That only was awesome already, and we also had a lot of talks to listen to: as others have said, it was really hard to get to choose to go to only one place out of three.

Skytools 3.0 reaches alpha1

Tue, 14 Apr 2009 00:00:00 +0200

It’s time for Skytools news again! First, we did improve documentation of current stable branch with hosting high level presentations and tutorials on the PostgreSQL wiki. Do check out the Londiste Tutorial, it seems that’s what people hesitating to try out londiste were missing the most.

The other things people miss out a lot in current stable Skytools (version 2.1.9 currently) are cascading replication (which allows for switchover and failover) and DDL support. The new incarnation of skytools, version 3.0 reaches alpha1 today. It comes with full support for cascading and DDL, so you might want to give it a try.

Importing XML content from file

Thu, 05 Feb 2009 00:00:00 +0100

The problem was raised this week on IRC and this time again I felt it would be a good occasion for a blog entry: how to load an XML file content into a single field?

The usual tool used to import files is COPY, but it’ll want each line of the file to host a text representation of a database tuple, so it doesn’t apply to the case at hand. RhodiumToad was online and offered the following code to solve the problem:

Skytools ticker daemon and londiste

Tue, 03 Feb 2009 00:00:00 +0100

One of the difficulties in getting to understand and configure londiste reside in the relation between the ticker and the replication. This question was raised once more on IRC yesterday, so I made a new FAQ entry about it: How do this ticker thing relates to londiste?

Comparing Londiste and Slony

Sat, 31 Jan 2009 00:00:00 +0100

In the page about Skytools I’ve encouraged people to ask some more questions in order for me to be able to try and answer them. That just happened, as usual on the #postgresql IRC, and the question is What does londiste lack that slony has?

Controling HOT usage in 8.3

Wed, 28 Jan 2009 00:00:00 +0100

As it happens, I’ve got some environments where I want to make sure HOT ( aka Heap Only Tuples) is in use. Because we’re doing so much updates a second that I want to get sure it’s not killing my database server. I not only wrote some checking view to see about it, but also made a quick article about it in the French PostgreSQL website. Handling around in #postgresql means that I’m now bound to write about it in English too!

Londiste Trick

Wed, 21 Jan 2009 00:00:00 +0100

So, you’re using londiste and the ticker has not been running all night long, due to some restart glitch in your procedures, and the on call admin didn’t notice the restart failure. If you blindly restart the replication daemon, it will load in memory all those events produced during the night, at once, because you now have only one tick where to put them all.

The following query allows you to count how many events that represents, with the magic tick numbers coming from pgq.subscription in columns sub_last_tick and sub_next_tick.

new site, using new software

Sat, 06 Dec 2008 00:00:00 +0100

Oh and check out the skytools page too. Emacs Muse is so great a project that instead of just working on how to publish a website with this tool, I found myself editing a rather large document about londite.py.