Using Vault to Store the Master Key for Data at Rest Encryption on Percona Server for MongoDB

Since the release of Percona Server MongoDB 3.6.13 (PSMDB), you have been able to use Vault to store the encryption keys for data at rest encryption. Here’s how to set it up.

First, you need to have a Vault server up and running. My colleague, Jericho, has an article on setting up Vault for Percona Server titled Using the keyring_vault Plugin with Percona Server for MySQL 5.7 . In this post, I will provide the same instructions for installing and setting up Hashicorp Vault for testing (Thank you Jericho!).

Vault Installation(run as root or use sudo):

1.Download, extract and install Vault:

# wget https://releases.hashicorp.com/vault/1.3.2/vault_1.3.2_linux_amd64.zip
# unzip vault_1.3.2_linux_amd64.zip 
# mv vault /usr/sbin

Make sure to place it somewhere on your PATH

2.Place initial vault configuration in /etc/vault

# mkdir /etc/vault
# cd /etc/vault
# cat vault.hcl
listener "tcp" {
 address = "192.168.0.114:8200"
 tls_cert_file="/etc/vault/vault.crt"
 tls_key_file="/etc/vault/vault.key"
}
storage "file" {
  path = "/var/lib/vault"
}
disable_mlock=true

You can use a port different than 8200. disable - mlock = true is needed if you want to start the Vault server as a non-root user.

3.Generate SSL certificates. To be able to create SSL certificates without going through prompts, we will place those entries in a configuration file:

# cd /etc/vault
# cat ssl.conf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
 
[req_distinguished_name]
C = US
ST = NC
L =  R
O = Percona
CN = *
 
[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names
 
[alt_names]
IP = 192.168.0.114

4.Run the commands below to generate the certificates and store them in /etc/vault.

# cd /etc/vault
# openssl req -config ssl.conf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout vault.key -out vault.crt
# cat vault.key vault.crt > vault.pem

5.Ensure that the keys are only accessible by the owner.

# chmod 400 vault.key
# chmod 400 vault.pem
# chmod 400 vault.crt

6.Set environment variables needed to access the vault.

# export VAULT_CACERT=/etc/vault/vault.crt
# export VAULT_ADDR='https://192.168.0.114:8200'

7.Start Vault in the background.

# vault server -config=/etc/vault/vault.hcl >> /var/log/vault.log 2>&1 &
[1] 7336

8.Initialize Vault and store the unseal keys and the initial root token generated in this step:

# vault operator init
Unseal Key 1: Gdu0HtSSctKUvn0ssi+GWhKEDZBWMAmulKfiAroCt+iw
Unseal Key 2: UEf61dpqPvjD+ftFaCJTy+KzKlSTIsT75qwe4gVLR9w4
Unseal Key 3: am/vq6P/SNs7aIKiLr7gEkeV8Pn/ilkV5r+HbwTpn7Vk
Unseal Key 4: aZYbiu2C3zrDklyWxf0JaqAouGRM95g0a9vz1JIk6jHD
Unseal Key 5: IApVQeN6kxCU1dr/H/Dc+1Afn6CJ5VVhGdAPNC1tGgVL
 
Initial Root Token: s.AqVdpn2C4NVJgtkZkRrKeFtm
 
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
 
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
 
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

9.Unseal the Vault by supplying the three unseal keys generated above. You need to unseal Vault every time the Vault server is started.

# vault operator unseal Gdu0HtSSctKUvn0ssi+GWhKEDZBWMAmulKfiAroCt+iw
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       6be37706-61cb-b372-cc88-b0e89e651aff
Version            1.3.2
HA Enabled         false
 
# vault operator unseal UEf61dpqPvjD+ftFaCJTy+KzKlSTIsT75qwe4gVLR9w4
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       6be37706-61cb-b372-cc88-b0e89e651aff
Version            1.3.2
HA Enabled         false
 
# vault operator unseal am/vq6P/SNs7aIKiLr7gEkeV8Pn/ilkV5r+HbwTpn7Vk
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.3.2
Cluster Name    vault-cluster-752be70f
Cluster ID      32ebb89d-2583-23b6-513d-c44220e12ac0
HA Enabled      false

10.Now it’s time to log in to Vault and use the initial root token.

# vault login s.AqVdpn2C4NVJgtkZkRrKeFtm
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
 
Key                  Value
---                  -----
token                s.AqVdpn2C4NVJgtkZkRrKeFtm
token_accessor       EZo7M0JbZjicINWMLee573qL
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

11.At this point, on Vault, you will need to enable the KV Secrets Engine – Version 2, the only engine supported by PSMDB. To do so, you will need to run the following:

# vault secrets enable -path secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

12.Next, you will need to create a Vault Policy to access this datastore. As per https://www.vaultproject.io/docs/secrets/kv/kv-v2/ , reading and writing versions need to be prefixed with the path data/.

# cat mongodb.hcl 
path "secret/data/dc/*" {
  capabilities = ["create","read","update","delete"]
}

13.To upload the policy to the Vault server, run:

# vault policy write mongodb-policy mongodb.hcl
Success! Uploaded policy: mongodb-policy

14.Finally, you will need to create an access token to be used by MongoDB. Remember to create a token per MongoDB instance.

# vault token create -policy=mongodb-policy
Key                  Value
---                  -----
token                s.cFy5NxA72Wk7VhVH45VJ4Rib
token_accessor       yoVBWgUVtDpzIomIRGTvBRd6
token_duration       768h
token_renewable      true
token_policies       ["default" "mongodb-policy"]
identity_policies    []
policies             ["default" "mongodb-policy"]

You will need to use the token above to connect MongoDB to Vault.

15.Done.

Incorporating Vault in MongoDB

In MongoDB, you’ll need to create a token based on the generated token and CA file generated in the Vault. Let’s place them in /etc/mongodb:

1.Create a mongodb config directory that will store both token and CA file.

# mkdir -p /etc/mongodb
# chown mongod:mongod /etc/mongodb

2.Place the token value in the token file and copy the contents of vault.crt from the Vault server.

# cd /etc/mongodb
# cat token 
s.cFy5NxA72Wk7VhVH45VJ4Rib
# cat vault.crt 
-----BEGIN CERTIFICATE-----
MIIDbDCCAlSgAwIBAgIJAOsptGVHJfKYMA0GCSqGSIb3DQEBCwUAMEQxCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJOQzEKMAgGA1UEBwwBUjEQMA4GA1UECgwHUGVyY29u
YTEKMAgGA1UEAwwBKjAeFw0yMDAzMzAxODE4NDlaFw0zMDAzMjgxODE4NDlaMEQx
CzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEKMAgGA1UEBwwBUjEQMA4GA1UECgwH
UGVyY29uYTEKMAgGA1UEAwwBKjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAKSRMI4/rbiWitDZtnDzlCVi0BDalmn4ieG93mt1Vn0IfN4NOmQeZ5fVRBve
Y26ugAT7Nq92puvV6vLk3ky8K8dkNPLe4wPEMPWnrHJIS4zqcmJWbw618yymVufm
h59cI/jyYCPMyLXkcZLuOnSZszoKnN/jEZq6WKNxgFuc6evzfxN4AbM7ffRk2mj5
FfMbtpToBgJS/U9UMS7EvzXYslUyq4bgBx3ygcmUyas7Ej7aie2fb0tzW/dMJ6/F
M2kYnHTM+pxr+Eq5QndBcjgptv/iqdnmIbNHogY+4TOCQwyke83SphgIwtI9MCLN
xUDN5nZEeRY8RINk3+irL/Kq0ocCAwEAAaNhMF8wHQYDVR0OBBYEFFS+R00NplQg
c6XFyYNLmrMJFa4lMB8GA1UdIwQYMBaAFFS+R00NplQgc6XFyYNLmrMJFa4lMAwG
A1UdEwQFMAMBAf8wDwYDVR0RBAgwBocEwKgAcjANBgkqhkiG9w0BAQsFAAOCAQEA
h9dKQoTILcv43mVbowCENm0ujPCW74CM/VXc+k8s3A5+h2tHtrO6NYATQBA07L+k
dwuethgqaqzbaEJqF766UiKv00hCkY8fMSyptLRtYG3WtxDWN/kGGvSjkzbYYqDP
RK1X5eI7PT7rtyu+FMWEiFLgQE6e5pz8MOITvdtQviJ35jUD1FfCpzTWppqeCc0g
9dVhzej1KDYTpoTcUkGLYDF1AMwEdXXqI77zdbhzQfmSlt1hAJGBKJFvAGTM2XHb
vd0QhgGbfzryv1b+eDTL7wWsYckNEcADpizHinxBpKC9/cnJZZtt6D35NvVAMkjy
+8wuOopSb6PCjrxVAz75FA==
-----END CERTIFICATE-----

3.Ensure the files are readable by mongod only, as MongoDB will complain and fail during startup with lax permissions.

# chown mongod:mongod /etc/mongodb/token
# chown mongod:mongod /etc/mongodb/vault.crt
# chmod 400 /etc/mongodb/token
# chmod 400 /etc/mongodb/vault.crt

4.Place in /etc/mongod.conf the Vault configuration under the security section. As a sample naming convention, given that the hostname of this server is psmongodb1, the secret path will be secret/data/dc/psmongodb1 as well:

# cat /etc/mongod.conf
# mongod.conf, Percona Server for MongoDB
# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/
 
*** redacted ***
processManagement:
  fork: true
  pidFilePath: /var/run/mongod.pid
 
# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1
 
#security:
security:
  enableEncryption: true
  vault:
    serverName: 192.168.0.114
    port: 8200
    secret: secret/data/dc/psmongodb1
    tokenFile: /etc/mongodb/token
    serverCAFile: /etc/mongodb/vault.crt
 
*** redacted ***

5.Then start MongoDB:

# systemctl start mongod

You can only enable encryption on an empty database. It’s not possible to re-encrypt an existing, previously unencrypted database. One way to do it is to create a backup, stop mongod, delete data files, enable encryption, start mongod, and restore the database from backup. If you have a replica set, you can apply changes in a rolling manner (one member at the time). Data will be synced automatically from other nodes. Each node has to be encrypted separately.

6.Check the logs if using the key was successful:

2020-03-30T19:05:48.929+0000 I ACCESS   [main] Initialized External Auth Session
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] MongoDB starting : pid=8026 port=27017 dbpath=/var/lib/mongo 64-bit host=psmongodb1.example.com
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] db version v3.6.17-4.0
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] git version: 96e9c7218eebc2995dd847d6185f20f102a86055
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] modules: none
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] build environment:
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-03-30T19:05:48.933+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongod.pid" }, security: { enableEncryption: true, vault: { disableTLSForTesting: false, port: 8200, secret: "secret/data/dc/psmongodb1", serverCAFile: "/etc/mongodb/vault.crt", serverName: "192.168.0.114", tokenFile: "/etc/mongodb/token" } }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongo/mongod.log" } }
2020-03-30T19:05:49.024+0000 I STORAGE  [initandlisten] Master key is absent in the Vault. Generating and writing one.
2020-03-30T19:05:49.076+0000 I STORAGE  [initandlisten] Initializing KeyDB with wiredtiger_open config: create,config_base=false,extensions=[local=(entry=percona_encryption_extension_init,early_load=true,config=(cipher=AES256-CBC,rotation=false))],encryption=(name=percona,keyid=""),log=(enabled,file_max=5MB),transaction_sync=(enabled=true,method=fsync),
2020-03-30T19:05:49.556+0000 I STORAGE  [initandlisten] Encryption keys DB is initialized successfully
2020-03-30T19:05:49.556+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),encryption=(name=percona,keyid="/default"),extensions=[local=(entry=percona_encryption_extension_init,early_load=true,config=(cipher=AES256-CBC)),local=(entry=percona_encryption_extension_init,early_load=true,config=(cipher=AES256-CBC)),],

As per the logs, it was able to generate a master key and use the Percona encryption. At this point, data at rest encryption has been configured on this server.

7.Done.

Take note that if the Vault server is down, you will not be able to start up your server. Given this new dependency on obtaining the master key from Vault, be sure to design your architecture for fault tolerance.

Encryption Key Rotation

Each encryption key can be used only for a limited time. Rotating master key re-encrypts the keystore using a new master key. The newly generated master key is then stored in the Vault. The entire dataset isn’t re-encrypted.

1.Place in /etc/mongod.conf an extra line in the security section:

# cat /etc/mongod.conf
security:
  enableEncryption: true
  vault:
    serverName: 192.168.0.114
    port: 8200
    secret: secret/data/dc/psmongodb1
    tokenFile: /etc/mongodb/token
    serverCAFile: /etc/mongodb/vault.crt
    rotateMasterKey: true

2.Then restart MongoDB:

# systemctl restart mongod

You’ll get an error message (but that’s expected, as mongod process didn’t start successfully). You can check logs if the encryption key was successfully rotated:

# tail /var/log/mongo/mongod.log
2020-03-30T19:05:48.929+0000 I STORAGE [initandlisten] Encryption keys DB is initialized successfully
2020-03-30T19:05:48.933+0000 I STORAGE [initandlisten] exception in initAndListend std::exception: master key rotation finished successfully, terminating
2020-03-30T19:05:48.933+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2020-03-30T19:05:48.933+0000 I -       [initandlisten] Stopping further Flow Control ticket acquisitions.
2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] now exiting
2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] shutting down with code:100

3.Remove the extra line in /etc/mongod.conf configuration rotateMasterKey : true

4.Done. You can start mongod again, your key was rotated.

Please be advised that this is a simple guideline on how to set up a basic Vault server and should not be used as a template for production usage. We recommend using Percona Consulting Services to assist you in that matter.