Vault - Installation and Configuration

As a part of Devops , while implementing pipelines and integrating with different tools we have to provide credentials many times. This is also the case when running automation. We used to provide credentials manually when asked as a part of automation. 

Managing these credentials is always hard. There are many types of credentials available to secure. Secrets ( credentials ) include passwords, keys, apis , tokens , service accounts and many other things that help in accessing from user access to machine access.

While running Continuous integration using Jenkins or team city, we will be integrating with multiple tools and perform many things like talking to the sonarqube server for code analysis, connecting to artifactory to upload artifacts, talk with blazemeter to do a performance test, talk to tomcat server to deploy application etc. For all these integrations, we will be defining the credentials in Jenkins and will be using them. 

Defining them in a CI tool can lead to other problems like misuse or it is job of the admin to change in jenkins whenever the secret for a tool changes externally.

Having a single tool for managing all these secrets and the tool providing a secure way to access these secrets is always good. HashiCorp Vault is one such tool. 

Vault is a tool for managing and securely accessing secrets. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log. We can configure our CI tool to talk to the Vault and request for the credential when there is a need to access a tool or login to a remote machine.

What good with vault is that it has methods for establishing both user and machine identity. Secrets can be consumed programmatically. Identities are established by token a long lived one or short lived. It has the capability to generate secrets dynamically on a number of backends. A well refined API interface is available to talk to the vault over web. 

In this article we will see how we can configure and run the Vault and create our first secrets.

1.Download the latest vault Zip file based on your operating system and unzip it to a location
[root@ip-172-31-10-86 opt]# wget https://releases.hashicorp.com/vault/1.0.3/vault_1.0.3_linux_amd64.zip
--2019-02-26 05:56:44-- https://releases.hashicorp.com/vault/1.0.3/vault_1.0.3_linux_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.201.183, 2a04:4e42:2f::439

Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.201.183|:443... connected.

HTTP request sent, awaiting response... 200 OK
Length: 35654431 (34M) [application/zip]
Saving to: 'vault_1.0.3_linux_amd64.zip'

2019-02-26 05:56:47 (12.1 MB/s) - 'vault_1.0.3_linux_amd64.zip' saved [35654431/35654431]

[root@ip-172-31-10-86 opt]# unzip vault_1.0.3_linux_amd64.zip
Archive: vault_1.0.3_linux_amd64.zip
inflating: vault

2. Starting the Vault server
Vault is a client/server application. The vault server is the only part that will talk to the backend data stores to write or read secrets. So if we are trying to insert a secret using CLI, it first goes to the server over a TLS connection and then to the back end store which will store the secret. To start the vault server run,

jagadishsoftware$Tue Mar 05@ ./vault server -dev

==> Vault server configuration:

            Api Address: http://127.0.0.1:8200

                    Cgo: disabled

        Cluster Address: https://127.0.0.1:8201

             Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")

              Log Level: info

                  Mlock: supported: false, enabled: false

                Storage: inmem

                Version: Vault v1.0.3

            Version Sha: 85909e3373aa743c34a6a0ab59131f61fd9e8e43

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory

and starts unsealed with a single unseal key. The root token is already

authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

   $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to

seal/unseal the Vault or re-authenticate.

Unseal Key: k/cK6gZZEMzsLDlwCjcz2e2VTPm9ouTRllTcqZhsaRk=

Root Token: s.KFCOxksMZPDFKkKb5AM1MORT

Development mode should NOT be used in production installations!

The vault server when started runs in foreground. We need to take another terminal to work with vault. The most important things in this are the Vault address where the vault is running and the vault root token.

We can see we are running the vault in a development mode. Since we did not pass any host details, the vault considers to run on the local machine and runs on the ip address 127.0.0.1 and port 8200.

We can also pass this information when starting the server as “vault server -dev -dev-listen-address=:”. Once we run this , the server starts on the specified ip address and port.

We need to set an environment variable to the vault to run successfully.

export VAULT_ADDR='http://127.0.0.1:8200'

When ever we run the vault command, the vault command tries to find this env variable to find where the vault server is running. Once it gets the details of the vault server it then connects to it and perform the necessary actions.

3. Copy the vault to the /usr/sbin location for accessing the vault from anywhere in the machine.

jagadishsoftware$Tue Mar 05@ cp ./vault /usr/sbin

4. Confirm the vault is running fine by running the “vault status”.

jagadishsoftware$Tue Mar 05@ ./vault status

Key                  Value

---                   -----

Seal Type         shamir

Initialized         true

Sealed             false

Total Shares     1

Threshold         1

Version            1.0.3

Cluster Name   vault-cluster-857cc8f9

Cluster ID        6dca8955-e042-6519-612e-5144518a3218

HA Enabled      false

5. Login to the vault using the “vault login” command as below,

jagadishsoftware$Tue Mar 05@ ./vault login s.KFCOxksMZPDFKkKb5AM1MORT

Success! You are now authenticated. The token information displayed below

is already stored in the token helper. You do NOT need to run "vault login"

again. Future Vault requests will automatically use this token.

Key                            Value

---                              -----

token                          s.KFCOxksMZPDFKkKb5AM1MORT

token_accessor           0M4ARsmnRi4ZMFxwwfRkjpLd

token_duration           ∞

token_renewable        false

token_policies            ["root"]

identity_policies         []

policies                     ["root"]

The vault login requires the root token that is generated while starting the server. The command can be “vault login ” to login to the vault as root user. This login user can do anything as this was a root user.

6. Create a secret using the Vault CLI as below,

jagadishsoftware$Tue Mar 05@ ./vault kv put secret/hello foo=world

Key                   Value

---                    -----

created_time     2019-03-05T05:04:23.158321Z

deletion_time    n/a

destroyed         false

version             

This writes the pair foo=world to the path secret/hello.now it is important that the path is prefixed with secret/.The secret/ prefix is where arbitrary secrets can be read and written.

Secrets written to Vault are encrypted and then written to backend storage. For our dev server, backend storage is in-memory, but in production this would more likely be on disk or in Consul. Vault encrypts the value before it is ever handed to the storage driver. The backend storage mechanism never sees the unencrypted value and doesn't have the means necessary to decrypt it without Vault.

vault kv put secret/hello foo=world

When we run the above command, foo=world pair is written to the path secret/hello. The path prefix tells vault about the secret engine to which it should route the traffic

When a request comes to the vault, it checks the initial path and based on that it sends the requests to the secrets engine. By default vault enables a secret engine called “kv” at the path secret. The kv secret engine reads and writes raw data to the back end store.

7. Retrieve a Secret

jagadishsoftware$Tue Mar 05@ ./vault kv get secret/hello

====== Metadata ======

Key                  Value

---                   -----

created_time    2019-03-05T05:04:23.158321Z

deletion_time    n/a

destroyed         false

version            1

=== Data ===

Key    Value

---    -----

foo    world

8. Delete a Secret

vault kv delete secret/hello