Trying Ansible alternatives in python

My VPS (Virtual Private Server) is getting old and the company that runs it announced that next month, the promotion I used will expire and it will now cost more than three times to run it. Also, I would like to update other machines I have home, as they use old Ubuntu.

This has led me to seek some kind of deployment automation, so I can specify my infrastructure as a code and ideally never ever again spend much time with it.

Why not Ansible

I don't like YAML based configuration languages that overgrow into scripting languages . It's a classic example of the Greenspun's tenth rule :

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

It is a nightmare to debug it, there is no support in IDE and so on.

The feeling of mess every time I try to use Ansible has led me to explore alternatives , preferably in python, which would define "recipes" or whatever as simple python code.

As it turned out, there aren't so many.

Fabric

• https://github.com/fabric/fabric

Almost every article on the topic of " python ansible alternative " mentions fabric . It looks great, but like something slightly different. It is a system for running commands on remote hosts. Which is definitely part of what Ansible does, but I would also like some kind of abstraction layer.

>>> def disk_free(c):
...     uname = c.run('uname -s', hide=True)
...     if 'Linux' in uname.stdout:
...         command = "df -h / | tail -n1 | awk '{print $5}'"
...         return c.run(command, hide=True).stdout.strip()
...     err = "No idea how to get disk space on {}!".format(uname)
...     raise Exit(err)

I mean, I don't want to manually run apt install nginx and then parse the output and try to decide whether the command successfully run (I did it before with paramiko and it is not fun, trust me).

I want something slightly more advanced, which does the parsing of standard utilities for me, and ideally with support of multiple OS, so when I decide to use CentOS instead of the Ubuntu server I am using now, it can cope with the different utilities for me.

Fabtools

• https://github.com/fabtools/fabtools

Fabtools looks almost exactly like what I want:

from fabric.api import *
from fabtools import require
import fabtools

@task
def setup():
    # Require some Debian/Ubuntu packages
    require.deb.packages([
        'imagemagick',
        'libxml2-dev',
    ])

    # Require a Python package
    with fabtools.python.virtualenv('/home/myuser/env'):
        require.python.package('pyramid')

    # Require an email server
    require.postfix.server('example.com')

    # Require a PostgreSQL server
    require.postgres.server()
    require.postgres.user('myuser', 's3cr3tp4ssw0rd')
    require.postgres.database('myappsdb', 'myuser')

    # Require a supervisor process for our app
    require.supervisor.process('myapp',
        command='/home/myuser/env/bin/gunicorn_paster /home/myuser/env/myapp/production.ini',
        directory='/home/myuser/env/myapp',
        user='myuser'
        )

    # Require an nginx server proxying to our app
    require.nginx.proxied_site('example.com',
        docroot='/home/myuser/env/myapp/myapp/public',
        proxy_url='http://127.0.0.1:8888'
        )

    # Setup a daily cron task
    fabtools.cron.add_daily('maintenance', 'myuser', 'my_script.py')

It has just one downside; it also looks dead.

Last commit is 9 months ago, there is 78 unresolved issues, 28 waiting pull requests and list of supported operating systems is ancient:

• Debian family:
  • Debian 6 ( squeeze ), 7 ( wheezy ), 8 ( jessie )
  • Ubuntu 10.04 ( lucid ), 12.04 ( precise ), 14.04 ( trusty )

That 14 in the Ubuntu 14.04 (trusty) is the year of release: 2014.

The documentation also suck, and there is weird confusion about forks.

Fabrix

• https://github.com/makhomed/fabrix

Then there is Fabrix , which looks like something between Fabric and Fabtools :

from fabrix.api import is_file_not_exists, yum_install
from fabrix.api import edit_file, edit_ini_section, replace_line

def install_php():

    if is_file_not_exists("/etc/yum.repos.d/epel.repo"):
        yum_install("https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm")

    if is_file_not_exists("/etc/yum.repos.d/remi-php70.repo"):
        yum_install("https://rpms.remirepo.net/enterprise/remi-release-7.rpm")

    edit_file("/etc/yum.repos.d/remi-php70.repo",
        edit_ini_section("[remi-php70]",
            replace_line("enabled=0", "enabled=1")
        )
    )

    yum_install("""
            php-cli
            php-common
            php-fpm
            php-gd
            php-mbstring
            php-mysql
            php-pdo
            php-pear
            php-pecl-imagick
            php-process
            php-xml
            php-opcache
            php-mcrypt
            php-soap
    """)

It's kinda funny to me, because I've created something very similar with paramiko some time ago.

It also looks dead, only 204 commits and one contributor, last commit 15 months ago. I don't want to build my system on something that is already dead.

pyinfra

• https://github.com/Fizzadar/pyinfra

pyinfra looks promising and very not dead: 2233 commits, 14 contributors, last commit yesterday . That's what I am talking about!

from pyinfra.operations import apt

apt.packages(
    {'Install iftop'},
    'iftop',
    sudo=True,
    update=True,
)

Example works exactly as I wanted; declarative language, important parameters as, you know, parameters and not strings. Only weird thing is to specify description as set , but whatever, I can see the line of reasoning here.

Documentation is also promising:

Trying pyinfra

As the pyinfra is the only thing that looks like it is not dead and it can do what I want, the decision is not that hard. So, lets try it:

$ pip install --user pyinfra

Now lets try hello world. I struggle with the port for a moment, because I use nonstandard port for tunneling through hotel and airport wifi, quick peek to help show that I should use --port parameter:

$ pyinfra kitakitsune.org --port 443 exec -- echo "hello world"
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [kitakitsune.org] Connected

--> Proposed changes:
    Ungrouped:
    [kitakitsune.org]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell (u'echo hello world',)
[kitakitsune.org] hello world
    [kitakitsune.org] Success

--> Results:
    Ungrouped:
    [kitakitsune.org]   Successful: 1   Errors: 0   Commands: 1/1

Looks good. Let's try to create a more complicated deployment for a virtual Ubuntu server, that I've created some time ago in VirtualBox. I slightly struggle with the inventory.py file, but then I find in the documentation correct parameters:

my_hosts = [
    ('192.168.0.106', {"ssh_port": "4433", "ssh_user": "b"}),
]

I quickly check whether it works with following deployment.py file:

from pyinfra.modules import server

server.shell('echo "hello world"')

Which I run using following command:

pyinfra -v inventory.py deployment.py

--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.0.106] Connected

--> Preparing operations...
    Loading: deployment.py
    [192.168.0.106] Ready: deployment.py

--> Proposed changes:
    Groups: my_hosts / inventory
    [192.168.0.106]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell ('echo "hello world"',)
[192.168.0.106] >>> sh -c 'echo "hello world"'
[192.168.0.106] hello world
    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Sudo

Ok, so it works. Nice. Now lets try it with --sudo :

--> Starting operation: Server/Shell ('echo "hello world"',)
[192.168.0.106] >>> sudo -S -H -n sh -c 'echo "hello world"'
[192.168.0.106] sudo: a password is required
    [192.168.0.106] Error
--> pyinfra error: No hosts remaining!

I've googled and looked into the documentation and also source code, but it looks like there is no way how to specify sudo password, which I find extremely weird. Someone even created an issue a day before:

• https://github.com/Fizzadar/pyinfra/issues/305

It looks like the sudo with password is simply not supported. So I've disabled sudo password in /etc/sudoers by changing the line

%sudo    ALL=(ALL:ALL) ALL

to:

%sudo      ALL=(ALL:ALL) NOPASSWD:ALL

After this, the code now works:

pyinfra -v inventory.py deployment.py --sudo

--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.0.106] Connected

--> Preparing operations...
    Loading: deployment.py
    [192.168.0.106] Ready: deployment.py

--> Proposed changes:
    Groups: my_hosts / inventory
    [192.168.0.106]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell ('echo "hello world"',)
[192.168.0.106] >>> sudo -S -H -n sh -c 'echo "hello world"'
[192.168.0.106] hello world
    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Setup nginx

Let's try something more complicated - to set up a nginx server and upload a correct config file for it:

from pyinfra.modules import apt


SUDO=True

apt.packages('nginx', update=True,present=True)

And it worked, with one exception, which is parsing of the output (yes, that's why parsing sucks):

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/util.py", line 471, in read_buffer
    _print(line)
  File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/util.py", line 455, in _print
    line = print_func(line)
  File "/home/bystrousak/.local/lib/python2.7/site-packages/pyinfra/api/connectors/util.py", line 61, in <lambda>
    print_func=lambda line: '{0}{1}'.format(print_prefix, line),
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2192' in position 74: ordinal not in range(128)
2020-06-11T23:55:28Z <Greenlet at 0x7fde3f4dd6b0: read_buffer('stdout', <paramiko.ChannelFile from <paramiko.Channel 2 (op, <Queue at 0x7fde39f60f30 queue=deque([('stdout', u, print_func=<function <lambda> at 0x7fde39f5e950>, print_output=True)> failed with UnicodeEncodeError

    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Which again is weird, but I am probably using Czech localization, so I am not that surprised.

Running it again shows that the operation was executed successfully:

--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.0.106] Connected

--> Preparing operations...
    Loading: deployment.py
    Loaded fact deb_packages
    [192.168.0.106] Ready: deployment.py

--> Proposed changes:
    Groups: my_hosts / inventory
    [192.168.0.106]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Apt/Packages ('nginx', u'update=True', u'present=True')
[192.168.0.106] >>> sudo -S -H -n sh -c 'apt-get update'
[192.168.0.106] Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
[192.168.0.106] Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
[192.168.0.106] Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
[192.168.0.106] Hit:4 http://archive.ubuntu.com/ubuntu bionic-security InRelease
[192.168.0.106] Reading package lists...
    [192.168.0.106] Success

--> Results:
    Groups: my_hosts / inventory
    [192.168.0.106]   Successful: 1   Errors: 0   Commands: 1/1

Config file

So, how do I upload a config file for the nginx?

from pyinfra.modules import files

files.put(
    'configs/nginx.conf',
    '/etc/nginx/nginx.conf',
    user='root',
    group='root',
    mode='644',
)

You can also download files, sync whole directories and so on. Quite nice.

Start nginx

from pyinfra.modules import server

init.systemd('nginx', running=True, restarted=True, enabled=True)

Beautiful.

Conclusion

I really like pyinfra . It has its quirks, but they are just little annoyances, unlike big annoyances I find in other products, like Ansible. But unlike Ansible, it is understandable, easy to use, and it uses Python, which I know and like, and my IDE can give me autocomplete and debugger, unlike other, YAML based domain specific languages.

Here is a whole config for the nginx deployment:

from pyinfra.modules import apt
from pyinfra.modules import init
from pyinfra.modules import files


SUDO=True


apt.packages(
    'nginx',
    update=True,
    present=True,
)

files.put(
    'configs/nginx.conf',
    '/etc/nginx/nginx.conf',
    user='root',
    group='root',
    mode='644',
)

init.systemd(
    'nginx',
    running=True,
    restarted=True,
    enabled=True,
)

Relevant discussion

• /r/devops (20 comments)

• /r/python (3 comments)

Updated @2020/06/12

Changed heated tone about the Ansible, as this was not the point. Added callout about the VS Code Ansible addon.