Upgrading PostgreSQL – Patroni Cluster and Switching from Python 2 to Python 3

Python 2 has officially completed its life as of Jan 1st, 2020 and the Python 2.x branch will not be maintained anymore. You might be seeing repeated notices of deprecation in log files and terminals like:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.

Python 2 had a wonderful life, coming from a humble background and becoming one of the most popular languages and the most popular scripting language . It’s no wonder Patroni-like tools are developed in Python.

Thanks to the growing popularity of Patroni, there are a large number of Patroni cluster setups across different companies. Now “ How to migrate from Python 2 to Python 3 with the least possible downtime ” becomes a hanging question for Operations teams and DBAs. Additionally, chances are that the cluster will be running an older version of Patroni. If so, the upgrade plan can take up both these challenges at the same time:

1. Upgrade Patroni Software to Latest release
2. Switch to Python 3 from Python 2

This blog post is to demonstrate how easy it is, and Patroni’s High Availablity features will be leveraged when we upgrade individual nodes. There are a few key points to note.

• We should not be disturbing the default Python 2 installation (its binaries and modules) because there could be other programs using them.
• Python 2 and Python 3 installations will go to different directories so they can coexist in almost all Linux distros.
• We should avoid multiple Patroni installations (different versions) in a node because they will cause confusion in the long run. So we must remove existing ones.
• Patroni in each node of the cluster can be using different versions of Python because Patroni is not interfering with PostgreSQL replication.
• Since Python precompiles scripts it loads to memory for execution, we can replace the Python script while the corresponding program is running in a loop.

Verification of the Version

We shall verify the version of Python and Patroni we use before proceeding. For example:

$ ps -eaf | grep patroni
postgres 17418 17066 0 10:19 pts/0 00:00:01 /bin/python /usr/bin/patroni /etc/patroni/patroni.yml
postgres 17494 17470 0 10:23 pts/1 00:00:00 grep --color=auto patron

As we see in this node, Patroni invoked Python from /bin/python. Commonly, this will be a symbolic link to the actual Python binary of a specific version.

$ ls -l /bin/python
lrwxrwxrwx. 1 root root 7 Dec 23 10:39 /bin/python -> python2

We won’t be replacing this because it may break other Python-based programs in the system. It’s worth noting the exact Python version.

$ /bin/python --version
Python 2.7.5

Upgrade

Since Python 3 binaries go to a different location in the system, there is nothing stopping us from installing it. It is not going to affect Patroni anyway. I would suggest using the Python from the repository of the Linux distribution.

On CentOS/RHEL, you can use the yum package manager for installing Python binaries and development packages and C compiler as follows:

sudo yum install -y python3-pip pyOpenSSL python3-devel gcc

The above-mentioned step may change depending on the Linux distribution. For example, on Ubuntu, you may install

sudo apt install python3-pip

and for the remaining installation, we can use pip for Python 3 (pip3) and execute the following commands one by one:

sudo pip3 install --upgrade setuptools
sudo pip3 install psycopg2-binary
sudo pip3 install wheel

These steps may remain the same for all environments.

IMPORTANT:In this blog post, steps are demonstrated with a fresh, minimal installation of Python 3. Mixing pip installation with the package manager of Linux distribution can be dangerous. Please follow either of the ways to install python packages/modules. It is recommended to avoid installing additional Python packages in a Patroni node as the root user.

Now we are entering the tricky part: uninstalling and installing Patroni while it is running. This is possible because Python does a precompilation of Python code and loads it to memory for execution. So replacing the Python source is almost equivalent to replacing a source file of a compiled program.

The Patroni in the system might have installed in different ways. RPM/DEB packages, Patroni Source code from Git repo of the project, or pip installation. Use the corresponding uninstall method for the same. In case your installation is using RPM/DEB packages, the package managers like YUM or APT provides their own removal options

Note: Percona Distribution for PostgreSQL provides RPM and DEB packages for installation of Patroni.

Even if the existing installation of Patroni is installed using easy_install or source install, pip can help you to remove it.

$ sudo pip uninstall patroni
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Uninstalling patroni-1.5.3:
  Would remove:
    /usr/bin/patroni
    /usr/bin/patroni_aws
    /usr/bin/patroni_wale_restore
    /usr/bin/patronictl
    /usr/lib/python2.7/site-packages/patroni-1.5.3-py2.7.egg
Proceed (y/n)? y
  Successfully uninstalled patroni-1.5.3

As we can see in this case, we uninstalled an older version 1.5.3. It’s good practice to make sure that Patroni no longer exists in the previous location:

head /usr/bin/patroni
head: cannot open '/usr/bin/patroni' for reading: No such file or directory

Now we can install the new version of Patroni with Python 3. As of writing this blog, the latest release is 1.6.3.

$ sudo pip3 install patroni[etcd]                                                                                                                                                         
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting patroni[etcd]
  Using cached https://files.pythonhosted.org/packages/8f/35/37eeb1b7d54a7d47205050bb6415a74fc7db7a3128e584fc54ff0f5b4cc2/patroni-1.6.3-py3-none-any.whl
...
Installing collected packages: patroni
Successfully installed patroni-1.6.3

The new location of Patroni can be different, but make sure to verify the Python version:

$ which patroni
/usr/local/bin/patroni
[postgres@pg1 ~]$ head /usr/local/bin/patroni
#!/usr/bin/python3

During this entire operation of uninstalling and reinstallation of the new version of Patroni, old Patroni will be running. No shutdown is required until this stage. But now the time has come and it’s a matter of new Patroni taking over the PostgreSQL.

Make sure that new Patroni is in the path and service files are updated accordingly.

$which patroni
/usr/local/bin/patroni

Now we should be stopping the Patroni service and starting it with new Patroni. Most administrators prefer to manage the Patroni service via Systemd.  In case your environment is also managed by systemd, the following steps can be used. As the first step, verify the systemd service configuration (unit file). If Patroni is located in a different directory, please do the required changes to the path.

You can open the systemd service in your favorite text editor:

sudo vi /etc/systemd/system/patroni.service

However, the best way to edit the service configuration file is to use systemctl’s edit option like

sudo systemctl edit --full patroni.service

which will open the service configuration for you for editing.

There will be a line mentioning the command to be executed to start the service (ExecStart). Edit this as per the Patroni location.

ExecStart=/usr/local/bin/patroni /etc/patroni/patroni.yml

Once editing is completed, reload the configuration:

sudo systemctl daemon-reload

Now we shall restart the Patroni deamon. This will cause the associated PostgreSQL instance to go down and start up again.

sudo systemctl restart patroni

Finally, we shall verify that the node joined back to the cluster with new Patroni.

-bash-4.2$ patronictl -c /etc/patroni/patroni.yml list
+----------+--------+------+--------+---------+----+-----------+
| Cluster  | Member | Host |  Role  |  State  | TL | Lag in MB |
+----------+--------+------+--------+---------+----+-----------+
| pgcluser |   pg0  |  pg0 | Leader | running |  2 |           |
| pgcluser |   pg1  |  pg1 |        | running |  2 |         0 |
| pgcluser |   pg2  |  pg2 |        | running |  2 |         0 |
+----------+--------+------+--------+---------+----+-----------+

We may explicitly verify that the Patroni is started with Python 3 and we have the latest version:

$ ps -eaf | grep patroni
postgres 16899 1 0 11:24 ? 00:00:02 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni/patroni.yml
postgres 16951 16462 0 11:42 pts/0 00:00:00 grep --color=auto patroni
 
$ /usr/local/bin/patroni --version
patroni 1.6.3

This blog post demonstrates how the upgrade of Python and Patroni can be achieved with the least disruption to service. Only Patroni in one node of the cluster needs to be restarted at a time. We shall repeat the above-mentioned steps in other nodes, also one by one, to upgrade the entire cluster.