home of ogirginc

How to fix a mismatch between psql and server?

Thu, 22 Feb 2024 00:00:00 GMT

You have installed a new Postgresql with Homebrew, but whenever you type psql, you get a warning like below:

> psql
psql (14.11 (Homebrew), server 16.2 (Homebrew))
WARNING: psql major version 14, server major version 16.
         Some psql features might not work.
Type "help" for help.

This happens because when you install a new version of Postgresql, Homebrew still uses the old version even after you have to stop the old version of Postgresql through the brew services stop command. We can confirm it like this:

> brew link postgresql@16
Linking /opt/homebrew/Cellar/postgresql@16/16.2...
Error: Could not symlink bin/clusterdb
Target /opt/homebrew/bin/clusterdb
is a symlink belonging to postgresql@14. You can unlink it:
  brew unlink postgresql@14

To force the link and overwrite all conflicting files:
  brew link --overwrite postgresql@16

To list all files that would be deleted:
  brew link --overwrite postgresql@16 --dry-run

To fix this, you need to explicitly tell Homebrew to stop using the old version and start using the new version.

First, use unlink to stop using the old version:

 > brew unlink postgresql@14
Unlinking /opt/homebrew/Cellar/postgresql@14/14.11... 334 symlinks removed.

Second, use link to start using the new version:

> brew link postgresql@16
Linking /opt/homebrew/Cellar/postgresql@16/16.2... 811 symlinks created.

If you need to have this software first in your PATH instead consider running:
  echo 'export PATH="/opt/homebrew/opt/postgresql@16/bin:$PATH"' >> ~/.zshrc

After these two, you can check to see if it’s fixed or not:

> psql --version
psql (PostgreSQL) 16.2 (Homebrew)

Fixed! 🎉

How to install multiple Postgresql versions on macOS?

Tue, 05 Jan 2021 00:00:00 GMT

So, you have decided to have multiple Postgresql versions on your Mac. Lucky for you, I have tried all options I was able to find and decided Peter Eisentraut’s Postgresql Common package, which is a Postgresql database cluster manager for Postgresql.

What is a database cluster?

A database cluster is a bit of an ambiguous term. Its definition varies depending on the context. In this article, I am not going to explain what it means in production. On local machines, it is a collection of databases that are managed by a single database server. Clusters allow us to install multiple versions of Postgresql on our macOS. To manage these clusters, we will be using a package called Postgresql Common.

Postgresql Common with Homebrew

First, we will install the Postgresql Common formulae:

brew install petere/postgresql/postgresql-common

With these formulae installed, we can have multiple versions of PostgreSQL in parallel. We should be able to install any version of Postgresql starting with version 8.3. If you do want to check the available versions, use the Homebrew’s search:

brew search petere/postgresql

Or you check the Github repo to see the available version on your browser: https://github.com/petere/homebrew-postgresql

If I wanted to install the version 12, this is how I would do it:

brew install petere/postgresql/postgresql@12

This command will fetch the latest minor version of the provided major version. For example, if there are 4 minor versions available, we would have version 12.4 installed automatically. Depending on your internet speed, it should not take more than a couple of minutes to download. Same goes for installing it with the make command, which depends on how fast the computer is.

Creating a cluster for the first time

To create a cluster, we will be using the pg_createcluster wrapper scripts. These scripts are just wrappers around pg_ctl and initdb to make managing easier. Now, let's create our first cluster. We will name this one as the main but you can choose whatever name you like.

pg_createcluster 12 main

We have created a new cluster named main which is version 12.4. If you like, we can check the status of the cluster with the pg_lsclusters wrapper script.

pg_lsclusters

This should list all the clusters, with some additional information, we have created in our machine.

Ver Cluster Port Status Owner    Data directory                        Log file
12  main    5432 down   ogirginc /usr/local/var/lib/postgresql/12/main /usr/local/var/log/postgresql/postgresql-12-main.log

Let's start our newly created cluster, we will be using pg_ctlcluster.

pg_ctlcluster 12 main start

Just to be sure, check the Status with pg_lsclusters. You should see the word down being replaced with online. Also, the output colour change might change depending on your theme. In my case, it was red turned to green.

Ver Cluster Port Status Owner    Data directory                        Log file
12  main    5432 online ogirginc /usr/local/var/lib/postgresql/12/main /usr/local/var/log/postgresql/postgresql-12-main.log

After starting your cluster, rest is the same with any Postgresql installation. Create a database with created command and list those with psql -c '\l' command to check if all is good. You should do something similar to this:

List of databases
   Name    |  Owner   | Encoding |   Collate   | Ctype |   Access privileges
-----------+----------+----------+-------------+-------+-----------------------
 ogirginc  | ogirginc | UTF8     | en_US.UTF-8 | UTF-8 |
 postgres  | ogirginc | UTF8     | en_US.UTF-8 | UTF-8 |
 template0 | ogirginc | UTF8     | en_US.UTF-8 | UTF-8 | =c/ogirginc          +
           |          |          |             |       | ogirginc=CTc/ogirginc
 template1 | ogirginc | UTF8     | en_US.UTF-8 | UTF-8 | =c/ogirginc          +
           |          |          |             |       | ogirginc=CTc/ogirginc
(4 rows)

That’s it! Now just repeat the process for each version you want to install, and it should work with a problem. If you do encounter any problems, check the Troubleshooting section below. If the problem still continues, feel free to open an issue and I would try my best to help!

Troubleshooting

Authentication failed for user

Why?

The default Postgres configuration sets the client authentication method to peer or md5, which can prevent connection to the database.

Solution

Run psql -c 'show hba_file;' to get pg_hba.conf’s path. The output should look like this:

hba_file
-----------------------------------------------
 /usr/local/etc/postgresql/12/main/pg_hba.conf
(1 row)

Open the pg_hba.conf file with your editor and go to the end of it, where we can see the preferred authentication methods. If you don’t have any sensitive data and your local database is not accessible through the internet, I would suggest changing the METHOD to trust for ease of use. This is how I use it:

# Database administrative login by Unix domain socket
local   all             ogirginc                                trust

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            md5
host    replication     all             ::1/128                 md5

No such file or directory

Why?

When creating a cluster, pg_ctl fails to find conf.d because it is missing. It’s a known bug. Check #45 and #49 for additional details.

Example output of pg_ctlcluster 12 main start:

Error: /usr/local/opt/postgresql@12/bin/pg_ctl /usr/local/opt/postgresql@12/bin/pg_ctl start -D /usr/local/var/lib/postgresql/12/main -l /usr/local/var/log/postgresql/postgresql-12-main.log -s -o  -c config_file="/usr/local/etc/postgresql/12/main/postgresql.conf" -c external_pid_file="/usr/local/var/run/postgresql/12-main.pid"  exited with status 1:
2020-07-13 10:32:51.376 GMT [9841] LOG:  could not open configuration directory "/usr/local/etc/postgresql/12/main/conf.d": No such file or directory
2020-07-13 10:32:51.376 GMT [9841] FATAL:  configuration file "/usr/local/etc/postgresql/12/main/postgresql.conf" contains errors
pg_ctl: could not start server
Examine the log output.

Solution

Create the conf.d with the mkdir command.

mkdir /usr/local/etc/postgresql/12/main/conf.d

Perl required

Why?

You get the error below because the required Perl files could not be found. Check #44 for additional details.

configure: error: header file <perl.h> is required for Perl

Solution

Try installing Xcode, which should install all necessary Perl files and specify the Xcode that you wish to use for command line developer tools like this:

sudo xcode-select --switch /Applications/Xcode.app/

LANG error

Why?

For some unknown reasons, the operating system’s locales are messed up.

Solution

First export the desired language and then create the cluster. An example for en_US would be like this:

export LANG=en_US.UTF-8
pg_createcluster 12 main

However, I would highly recommend fixing the locales as it might cause additional non-database related problems.

Different versions for `psql` and `server`

Why?

You have installed multiple Postgresql versions with Homebrew and psql automatically picks the latest version.

Solution

When trying to use psql with an earlier version of a Postgresql, a warning will be presented.

psql (13.0 (Homebrew petere/postgresql), server 12.4 (Homebrew petere/postgresql))

First of all, this is just a warning. Secondly, most, if not all things, should work in psql. However, tools like pg_dump might fail. To fix when you get this warning, export the desired Postgresql version to the path.

export PATH="/usr/local/opt/postgresql@12/bin:$PATH"

When retrying, there should not be any warning about version mismatch.

❯ psql
psql (12.4 (Homebrew petere/postgresql))
Type "help" for help.
ogirginc=*#*

Create a database with a different port

Why?

When you create a new cluster, Postgresql assigns the next available port number to this new cluster, which results in not being able to connect to the default 5432 port. An example of an error, while trying to run psql for the newly created second cluster:

psql: error: could not connect to server: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

Solution

To fix this connection error, we have to specifically tell psql which port it should use to connect to the database by exporting the PGPORT environment variable. Check with the pg_lsclusters wrapper script to which port do you need to connect the database.

Ver Cluster Port Status Owner    Data directory                        Log file
12  main    5433 online ogirginc /usr/local/var/lib/postgresql/12/main /usr/local/var/log/postgresql/postgresql-12-main.log
13  main    5432 down   ogirginc /usr/local/var/lib/postgresql/13/main /usr/local/var/log/postgresql/postgresql-13-main.log

To use version 12, I need to set PGPORT to 5433:

export PGPORT=5433

With this explicit port setting, we can use psql or any other Postgresql utilities without any problems.

psql (12.4 (Homebrew petere/postgresql))
Type "help" for help.
ogirginc=*#*

You call this simple?

Why?

I am fully aware it is not simple nor easy, but it gets the job done.

Solution

Try your luck with Postgres.app. Maybe, it will work for you. :)

Using PRIMARY KEY in PostgreSQL's GROUP BY

Sun, 23 Feb 2025 00:00:00 GMT

Have you noticed that in some cases, PostgreSQL does not ask you to include the selected column in the GROUP BY clause?

Until recently, I haven't spent time on understanding why this happens. Turns out, there is a simple explanation. Compared to the standard SQL, PostgreSQL actually has a special handling for the GROUP BY clause. So, what happens here is, when a PRIMARY KEY is included in a GROUP BY clause, it actually changes how the clause works, which allows selecting non-aggregated columns without including them in the GROUP BY clause.

Let's dive into this a bit more.

The Standard SQL Expectation

In standard SQL, the rule is clear! If you're using a GROUP BY clause, every column in your SELECT statement must either:

Be included in the GROUP BY clause, or
Be used within an aggregate function (like SUM, COUNT, MAX, etc.)

And you know, this does make sense – if you're grouping rows together, the database needs to know which value to display for a column that might have different values within the group.

However, things are a bit different in the land of PostgreSQL.

PostgreSQL's Special Handling

Check this simplified example of what I was working on recently:

SELECT
  users.id,
  users.full_name,
  users.time_zone,
  message_checkins.created_at::date AS checkin_date,
  COUNT(*) as check_count
FROM
  message_checkins
JOIN
  users ON message_checkins.user_id = users.id
GROUP BY
  users.id, users.full_name, checkin_date

Notice that I am selecting users.time_zone, but it's not in the GROUP BY clause. Turns out, in many database systems, this would cause an error. However, in PostgreSQL, this works perfectly fine.

OK, that's great, but why??

Functional Dependencies to the Rescue

PostgreSQL implements a concept called "functional dependency", which is not a very self-explanatory name if you ask me. If you want to get even more confused, you can read the official documentation on this:

When GROUP BY is present, or any aggregate functions are present, it is not valid for the SELECT list expressions to refer to ungrouped columns except within aggregate functions or when the ungrouped column is functionally dependent on the grouped columns, since there would otherwise be more than one possible value to return for an ungrouped column. A functional dependency exists if the grouped columns (or a subset thereof) are the primary key of the table containing the ungrouped column.

In my own simplified terms; if you have a PRIMARY KEY in your GROUP BY clause, you can select any column from the same table without including it in the GROUP BY clause.

To get back to my example above:

users.id is the primary key for the users table.
For any specific users.id, there is exactly one corresponding users.time_zone value.
Therefore, users.time_zone is functionally dependent on users.id.

Since we're grouping by users.id, PostgreSQL knows that all rows with the same users.id will also have the same users.time_zone, so it doesn't require you to include it in the GROUP BY.

Database Portability

Ha. Ha. When is the last time this was a real issue for you? Be honest.

Performance Implications

Sure, taking advantage of PostgreSQL's functional dependency understanding can make queries more concise, especially for complex queries with many columns. However, it's worth noting that:

The query optimizer needs to verify these functional dependencies.
For very complex queries, explicitly listing all columns might help the query planner.

To conclude, unless you're working with very large datasets or very complex queries, you shouldn't notice a significant difference in performance – probably.

The 'postmaster.pid' already exists

Wed, 14 Aug 2024 00:00:00 GMT

If you've encountered the following error message while working with PostgreSQL on an Apple M series machine using Homebrew, you're not alone:

2024-08-13 08:53:06.060 +03 [64564] FATAL:  lock file "postmaster.pid" already exists
2024-08-13 08:53:06.060 +03 [64564] HINT:  Is another postmaster (PID 773) running in data directory "/opt/homebrew/var/postgresql@16"?

This error typically indicates that there is a lock file named postmaster.pid already present in the data directory, potentially because another instance of PostgreSQL is running or because a previous instance was not shut down properly.

Quick Fix

To resolve this issue, you can remove the existing lock file. Use the rm command to delete the postmaster.pid file.

rm /opt/homebrew/var/postgresql@16/postmaster.pid

After removing the lock file, you should be able to start PostgreSQL without encountering the "postmaster.pid" error.

Why Does This Happen?

The postmaster.pid file is created by PostgreSQL to ensure that only one instance of the server is running at a time in a given data directory. If PostgreSQL crashes or is not shut down properly, this file may not be removed, leading to the error when you try to start PostgreSQL again.

Make sure that no other instance of PostgreSQL is running before doing removing the file, as it could lead to data corruption.

Enable relativenumber for neo-tree

Fri, 15 Nov 2024 00:00:00 GMT

If you are a lazyvim user and want to enable relative line numbers, first create a file for neo-tree.nvim if you don’t have already, under the plugins folder:

return {
  "nvim-neo-tree/neo-tree.nvim",
}

Next, we are going to update our config by addin an option with event handler to display relative line numbers when the neo-tree buffer becomes active:

return {
  "nvim-neo-tree/neo-tree.nvim",
  opts = {
    event_handlers = {
      {
        event = "neo_tree_buffer_enter",
        handler = function()
          vim.opt_local.number = true
          vim.opt_local.relativenumber = true
        end,
      },
    },
  },
}

Now, when you open the neo-tree, you should see the line numbers.

Fix SSL error using Heroku’s Redis for Flipper

Thu, 04 Jul 2024 00:00:00 GMT

Hello Redis::CannotConnectError 👋, my old friend. I did not miss you at all.

Years later, I once again, got the SSL connection error while I was adding the Flipper gem to our project, which is hosted on Heroku and uses Heroku’s Redis.

For a quick solution, check this:

# frozen_string_literal: true
require 'flipper/adapters/redis'

Flipper.configure do |config|
  config.default do
    redis = Redis.new(
      url: ENV.fetch('REDIS_URL', nil),
      ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
    )

    adapter = Flipper::Adapters::Redis.new(redis)
    Flipper.new(adapter)
  end
end

If you want to dig deep and understand why it is safe to use “verify none”, you can check the How to solve the SSL error for Redis 6 on Heroku? article that I wrote years ago.

Show SQL on Ruby on Rails console

Tue, 22 Oct 2024 00:00:00 GMT

Time to time, I like to test my ActiveRecord queries on my apps’ consoles. Play it, tweak it. Quite straightforward to do it on a Rails console.

However, there is a slight problem with this method; you cannot see the generated SQL nor how many milliseconds were spent on this query. To solve it, you need to update the ActiveRecord’s logger to log the output on standard output:

ActiveRecord::Base.logger = Logger.new(STDOUT)

This is really all you need. Now, when you run an ActiveRecord query, you will also get the SQL with load time.

If you really want to get this every time, which I don’t suggest, update your irbrc file like below:

if defined?(Rails::Console)
  ActiveRecord::Base.logger = Logger.new(STDOUT)
end

FYI, the reason why I don’t suggest is, I don’t always need to see the SQL query while playing around with models. If that’s not the case for you, definitely go for it.

How to solve the SSL error for Redis 6 on Heroku?

Thu, 03 Feb 2022 00:00:00 GMT

If you try to connect a Ruby on Rails app with a Heroku Redis add-on (excluding the Hobby Dev plan), there is a very high chance for you to get the error below:

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error:
certificate verify failed (self signed certificate in certificate chain)

Reason

From version 6 and above, Redis requires using TLS to connect. However, Heroku does not use SSL internally. They terminate SSL at the router level and forward requests from there to your application via HTTP, which is safe as all these do happen behind Heroku’s firewall. Also, let’s face it, Heroku’s security measures –probably– are better than yours.

Solution

To fix this, you will need to use OpenSSL::SSL::VERIFY_NONE for your Redis client.

Redis.new(
  url: 'url',
  driver: :ruby,
  ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
)

If you do use Sidekiq, configuration should be done through the Sidekiq initializers:

*# config/initializers/sidekiq.rb*

Sidekiq.configure_server do |config|
  config.redis = { ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end

Sidekiq.configure_client do |config|
  config.redis = { ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end

Troubleshooting

No error with Redis 6!?

Why?

Assuming you have double-checked the Redis version, the plan is probably on the Hobby Dev version, which does support both HTTP and HTTPS connections.

Solution

For the Hobby Dev plan, you should see two environment variables set under the app’s “Config Vars” section. If you do plan to keep the add-on as Hobby Dev, no change is needed.

If you do plan to upgrade the add-on to Premium 0 or above, you need to use VERIFY_NONE as above.

home of ogirginc

How to fix a mismatch between psql and server?

How to install multiple Postgresql versions on macOS?

What is a database cluster?

Postgresql Common with Homebrew

Creating a cluster for the first time

Troubleshooting

Authentication failed for user

Why?

Solution

No such file or directory

Why?

Solution

Perl required

Why?

Solution

LANG error

Why?

Solution

Different versions for psql and server

Why?

Solution

Create a database with a different port

Why?

Solution

You call this simple?

Why?

Solution

Using PRIMARY KEY in PostgreSQL's GROUP BY

The Standard SQL Expectation

PostgreSQL's Special Handling

Functional Dependencies to the Rescue

Database Portability

Performance Implications

The 'postmaster.pid' already exists

Quick Fix

Why Does This Happen?

Enable relativenumber for neo-tree

Fix SSL error using Heroku’s Redis for Flipper

Links

Show SQL on Ruby on Rails console

How to solve the SSL error for Redis 6 on Heroku?

Reason

Solution

Troubleshooting

No error with Redis 6!?

Why?

Solution

Links

Different versions for `psql` and `server`