Monday, February 22, 2016

Ansible vault

While working with automation it is necessary to have a secure system to store various details like Password, variable, SSH keys etc. Ansible does provide a facility called vault which helps sys admin to store sensitive data and use the vault while running playbooks on remote machines. In this article we will how we can use Ansible vault in securing things. We will be seeing on how

1) To encrypt data using Vault
2) Using Ansible valut while running Playbooks

While using Ansible as a configuration management system or a orchestration engine, it is necessary to pass certain data like passwords, keys etc to run the playbooks. These details can be common most times and used multiple times. An automated system that prompts the operator for passwords all the time is not very efficient. To maximize the power of Ansible, secret data has to be written to a file that Ansible can read and utilize the data from within. Though they are stored on the vault we can have these hacked.

For these ansible provides a facility to protect your data at rest. That facility is Vault, which allows for encrypting text files so that they are stored "at rest" in encrypted format. Without the key or a significant amount of computing power, the data is indecipherable.ansible-vault command is provided by Ansible in securing things

Encrypt Data
1) Create a Sample file to encrypt

[root@vx111a vault]# ansible-vault create sample_passwd.yml
Vault password:
Confirm Vault password:

We created a file sample_passwd.yml file which will ask for vault password. If we check the file type and contents,

[root@vx111a vault]# file sample_passwd.yml
sample_passwd.yml: ASCII text

[root@vx111a vault]# cat sample_passwd.yml
$ANSIBLE_VAULT;1.1;AES256
31643134323761626464336334333461636135656435333161636538326132356166303536353838
3661376665336163613139613836313765633836333838320a626262323565653837363735336163
36396639336239383566306439396262383965623338613664383434663765366639636534393634
6239376138303763610a363165323630326231626334633931323732316564316135643033383730
62666630333233366234633366623331326266633932363166656130373164333335

We see that the file is a ASCII file with encrypted contents.

2) Edit the Encrypted file to change the conents of the file
 In Oder to edit the encypted file, we need touse the edit command with the valut.
[root@vx111a vault]# ansible-vault edit sample_passwd.yml
Vault password:

When you try to edit the conents, it will ask for the vault password.

3) Decypt the file contents
[root@vx111a vault]# ansible-vault decrypt sample_passwd.yml
Vault password:
Decryption successful

Once decrypted we can see the conents of the file as,
[root@vx111a vault]# cat sample_passwd.yml
password: vagrant

We can then use the encrypt command to encypt the contents again as,
[root@vx111a vault]# ansible-vault encrypt sample_passwd.yml
Vault password:
Confirm Vault password:
Encryption successful

Ansible does also provide the rekey facility to change the valut password using
[root@vx111a vault]# ansible-vault rekey sample_passwd.yml
Vault password:
New Vault password:
Confirm New Vault password:
Rekey successful

Using vault with Playbooks
Until now we have seen how we can use the Ansible vault in encrypting data. Its no use when the data is encrypted and not being used.  Now we will see how we pass the encrypted data to the Ansible playbooks while running on remote machine.

1) Create a sample yml file with 2 variables as
[root@vx111a vault]# cat main.yml
version: 8.0.32
http_port: 8084

Now encrypt the file using the Ansible-vault,

[root@vx111a vault]# ansible-vault decrypt main.yml
Vault password:
Decryption successful

Check the encryption
[root@vx111a vault]# cat main.yml
$ANSIBLE_VAULT;1.1;AES256
34333132653430616138313235366435613232653662653865663264346632616664666665356437
6133633063396434326538373531326231623536373465360a326334303832366530373535356334
32376162336164643561343462643063326366653039303433666439633064383364633064303939
3139663363366336610a336566353664613536643933633166356536336634363734626664363261
34373865613831646339333866613564373937326262643432353866316339306263346430643434
3030633864663837613934663166616630623966653533383733

Now once the file is encrypted write a playbook as,
[root@vx111a vault]# cat sample-playbook.yml
---
- hosts: cent
  vars_files:
     - main.yml
 
  tasks:
    - name: run echo Command
      local_action: shell echo {{ http_port }}
      register: local_process

    - debug: msg="{{ local_process.stdout }}"


Now we have written a playbook which includes the main.yml file (which is encrypted using Ansible-vault) containing the variables. When we run the playbook as

[root@vx111a vault]# ansible-playbook sample-playbook.yml
ERROR: A vault password must be specified to decrypt /work/vault/main.yml

It clearly says that we need to pass the vault password in running the playbook. The correct way to run that playbook is

[root@vx111a vault]# ansible-playbook sample-playbook.yml --ask-vault-pass
Vault password:

PLAY [cent] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [172.16.202.96]

TASK: [run echo Command] ******************************************************
changed: [172.16.202.96 -> 127.0.0.1]

TASK: [debug msg="{{ local_process.stdout }}"] ********************************
ok: [172.16.202.96] => {
    "msg": "8084"
}

PLAY RECAP ********************************************************************
172.16.202.96              : ok=3    changed=1    unreachable=0    failed=0  

We can see that Ansible asks for the vault password in order to run the playbook. Once the password is provided, it run the playbook and also the Portt 8084 is replaced with values from main.yml file.

This is how we can use the Ansible valut with playbooks.

Password File
Though we provide the vault password every time while running the playbooks, there are some times with cases where we need to run the playbook with out any manual intervention. Ansible does provide an option to create a password file with the vault password and pass the file to Ansible command line as argument.

1) use the same above mail.yml file and encrypt that
[root@vx111a vault]# ansible-vault encrypt main.yml
Vault password:
Confirm Vault password:
Encryption successful

Now create a password file with password as “redhat”. Save in the home location and provide it with correct permissions.
[root@vx111a vault]# echo "redhat" >> ~/.vault_password
[root@vx111a vault]# chmod -R 600 ~/.vault_password

Now run the playbook by passing the password file as argument as,

[root@vx111a vault]# ansible-playbook sample-playbook.yml --vault-password-file ~/.vault_password

PLAY [cent] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [172.16.202.96]

TASK: [run echo Command] ******************************************************
changed: [172.16.202.96 -> 127.0.0.1]

TASK: [debug msg="{{ local_process.stdout }}"] ********************************
ok: [172.16.202.96] => {
    "msg": "8084"
}

PLAY RECAP ********************************************************************
172.16.202.96              : ok=3    changed=1    unreachable=0    failed=0  

This is how we can use the Ansible vault and secure data.
Read More

Ansible Monitor-Alert

At an infra and application level we need to have several monitoring metrics configured right from CPU, memory to application leavel such as Heap and mumber of Database connections. Also, from an infrastructure automation point of view, you need to make sure you start a strong feedback loop by integrating automation with your monitoring and alerting system or systems through constant reporting.

In this article we will how we can configure monitor and alert when a playbook ran. Lets write a sample playbook as,

---
- hosts: cent

  tasks:
    - name: run echo Command
      command: /bin/echo Hello Sample PlayBook

    - name: get the IP address
      shell: hostname
      register: host_name

    - name: Send Mail Alert
      local_action: mail
                    host="127.0.0.1"
                    subject="[Ansible] Testing Mail"
                    body="Hello the Play book ran successfully on {{ host_name.stdout }}"
                    to="jagadesh.manchala@gmail.com"
                    from="root"     

In the above playbook we have multiple tasks as to run a command on remote machine and also get the hostname of the remote machine.

The last one is the important one as we are sending the mail. We have used the local_action element which will make to exeute the mail command on the local machine. We can also configure our playbook in such as to execute the mail command based on the response of the commands above. I just got the hostname and added that to the mail Subject. Now once you run the playbook we can see,


[root@vx111a mail]# ansible-playbook sample-playbook.yml

PLAY [cent] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [172.16.202.96]

TASK: [run echo Command] ******************************************************
changed: [172.16.202.96]

TASK: [get the IP address] ****************************************************
changed: [172.16.202.96]

TASK: [Send Mail Alert] *******************************************************
ok: [172.16.202.96 -> 127.0.0.1]

PLAY RECAP ********************************************************************
172.16.202.96              : ok=4    changed=2    unreachable=0    failed=0  

Now if we check our mail account we can see some thing like this,
We received a mail once the playbook is ran. We can add various logic to extract details from remote machine and add them to our alerting mechanism. In the next article we will see how we can use other monitoring and alerting methods provided by Ansible.
Read More

Ansible delegation

Ansible by default run the task all at once on the remote configure machine. What If a task is based on status of a certain command on another machine?. Say if you are patching a package on a machine and you need to continue until a certain file is available on another machine. This is done in Ansible using the delegation option.

By using the Ansible delegate option we can configure to run a task on a different host than the one that is being configured using the delegate_to key. The module will still run once for every machine, but instead of running on the target machine, it will run on the delegated host. The facts available will be the ones applicable to the current host. Lets write a basic playbook using Ansible delegate

[root@vx111a delegate]# cat sample-playbook.yml
---
- hosts: cent
 
  tasks:
    - name: Install zenity
      action: yum name=zenity state=installed
    - name: configure zenity
      action: template src=hosts dest=/tmp/zenity.conf
    - name: Tell Master
      action: shell echo "{{ansible_fqdn}} done" >> /tmp/list
      delegate_to: 172.16.202.97

We have defined multiple tasks to install zenity, configure zenity and the last task is to echo command on the delegate machineof different IP address.  Ansible-playbook will delegate the last task to the IP address 172.16.202.97. Once we run the playbook we can see


[root@vx111a delegate]# ansible-playbook sample-playbook.yml

PLAY [cent] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [172.16.202.96]

TASK: [Install zenity] ********************************************************
changed: [172.16.202.96]

TASK: [configure zenity] ******************************************************
changed: [172.16.202.96]

TASK: [Tell Master] ***********************************************************
changed: [172.16.202.96 -> 172.16.202.97]

PLAY RECAP ********************************************************************
172.16.202.96              : ok=4    changed=3    unreachable=0    failed=0  

Now check the remote machine 172.16.202.97 for the file created.

root@ubuntu:/home/vagrant# cat /tmp/list
dev.foohost.vm done


More to Come. Happy learning
Read More

Ansible Tags

Tags are another feature in Ansible that is used to run only a subset of tasks/roles. When we have tasks in a playbook and we executed the playbook then all tasks are executed. Byt by using the tags we can make the playbook to execute only a subset of tasks by defining them with tag attribute. In this article we will see how we can create tags and use them to execute a subset of tasks in playbook.

1) Lets create a basic sample playbook as
[root@vx111a tags]# cat sample-playbook.yml
---
- hosts: cent
 
  tasks:
    - name: run echo Command
      command: /bin/echo Hello Sample PlayBook
      tags:
        - sample

    - name: Create Sub Directories
      file:
         dest: "/tmp/html"
         state: directory
         mode: 755
      tags:
        - create

If you check the above example, I have defined 2 tasks defining them with 2 different tags. Now when we execute the playbook all the tasks are executed but if we want only a specific tag to be execute we can run the playbook as,

[root@vx111a tags]# ansible-playbook sample-playbook.yml --tags sample

PLAY [cent] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [172.16.202.96]

TASK: [run echo Command] ******************************************************
changed: [172.16.202.96]

PLAY RECAP ********************************************************************
172.16.202.96              : ok=2    changed=1    unreachable=0    failed=0  
In the above example we ran only the sample tag by passing the argument –tags sample to the command line. In the below example I ran only the create tag.

[root@vx111a tags]# ansible-playbook sample-playbook.yml --tags create

PLAY [cent] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [172.16.202.96]

TASK: [Create Sub Directories] ************************************************
changed: [172.16.202.96]

PLAY RECAP ********************************************************************
172.16.202.96              : ok=2    changed=1    unreachable=0    failed=0  


We can add the register attribute so that we can see what exactly changing. By default ansible runs as if ‘–tags all’ had been specified.This about the tags in Ansible. More to come. Happy learning
Read More

Ansible Roles- An advanced Example

In the previous article we have seen how we can use the Ansible roles in running the playbook on remote machine. In this article we will see a more advanced example of using Ansible roles. In this article we will see how we can configure a bacic Tomcat server on the remote machine using Roles.

1) Lets Create a Directory Structure as,

[root@vx111a roles]# tree tomcat/
tomcat /
── files
── handlers
── meta
── tasks
── templates
└── vars

2) Lets start creating the variable files first tomcat/vars
[root@vx111a vault]# cat main.yml
version: 8.0.32
http_port: 8084

3) Now lets create the tasks file. We will write 3 tasks files
Configure_java.yml – configure Java for Tomcat Server
Configure_tomcat.yml – Configure Tomcat
Update_path.yml – set JAVA_HOME and add java location to PATH in .bashrc file
Main.yml – this contains all the above files using include

Configure_java.yml
---
 - name: Install Java
   copy: src=jdk-8u65-linux-x64.tar.gz dest=/tmp group=root owner=root mode=0755   

 - name: Unpack JAVA
   unarchive: src=/tmp/jdk-8u65-linux-x64.tar.gz dest=/usr/local/ copy=no

This file is the one that we use for configuring java before tomcat is configured. The Java version 8u65 is downloaded and copied to the same /files location. We will copy this using the COPY module to the remote /tmp location. In the second task we will unarchive the copied java to /usr/local. At this point we have not set any JAVA_HOME or added any thing to the PATH variable.

configure_tomcat.yml
---
- name: Remove Older Tomcat Directories
  file: path=/usr/local/tomcat state=absent
     
- stat: path=/usr/local/
  register: tc

- name: Download Tomcat
  get_url: url=http://a.mbbsindia.com/tomcat/tomcat-8/v8.0.32/bin/apache-tomcat-{{ version }}.tar.gz dest=/tmp mode=0755

- name: Unpack Tomcat
  unarchive: src=/tmp/apache-tomcat-{{ version }}.tar.gz dest=/usr/local/ copy=no
  when: tc.isdir is undefined

- name: Change Tomcat Directory Name
  command: mv /usr/local/apache-tomcat-{{ version }} /usr/local/tomcat

- name: Upload server.xml with modified Changes
  template: src=server.xml dest=/usr/local/tomcat/conf
  notify: restart tomcat
          
- name: Upload tomcat-users.xml with users added
  template: src=tomcat-users.xml  dest=/usr/local/tomcat/conf
  notify: restart tomcat

- name: Install Tomcat init.d Script
  copy: src=tomcat-initscript.sh dest=/etc/init.d/tomcat mode=0755  

- name: Start Tomcat
  service: name=tomcat state=started enabled=yes

- name: Wait for Tomcat to Start on Port 8084
  wait_for: host=172.16.202.96 port={{ http_port }}

This file is used to configure Tomcat once the java installation is done. The file contains the following tasks.
1) Check for tomcat folder and remove them.
2) make sure the location /usr/local is available
3) Download tomcat from Internet
4) Un-archive the tomcat and move to /usr/local/apache-tomcat-version
5) Rename the download apache-tomcat-version to tomcat
6) Upload the server.xml file with port changes that exist in the files location to the remote machine tomcat location.
7) Upload the tomcat-users.xml file with users added
8) Upload the tomcat-initscript.sh to the remote machine /etc/init.d location. This file is used to start and stop tomcat as a service.
9) Wait for the Tomcat to start on Port 8084

The version details are extracted from the variable file that we already configured.

Update_path.yml
---
- name: Set Java HOME
  lineinfile: dest=~/.bashrc regexp='^JAVA_HOME'>
                 line="export JAVA_HOME=/usr/local/jdk1.8.0_65/"

- name: Set JAVA PATH
  lineinfile: dest=~/.bashrc regexp='^JAVA_HOME'>
                 line="export PATH=$M2:$PATH:/usr/local/jdk1.8.0_65/bin"

- name: Source Bashrc
  action: shell source ~/.bashrc

This file is used to update the .bashrc file with the necessary Environment variables. The lineinfile module is used to update the .bashrc with necessary variable information. At last we will be using the source command to update the .bashrc file

Main.yml
---
- include: update_path.yml
- include: configure_java.yml
- include: configure_tomcat.yml

This file is the important file for the tomcat role to run. Ansible-playbook uses the main.yml file for running the tasks defined. We included all the other three files in sequence for running the tasks.

All these files are created under /tomcat/tasks location in the tomcat folder.

4) Now lets create the handlers file for restarting the tomcat service. The main.yml is created under the /tomcat /handlers folder as

Main.yml
---
- name: Restart Tomcat
  Service: name=tomcat state=restarted

6. The next step is creating the template files. As we already know that we used server.xml and tomcat-user.xml for modifying the port and adding users to the tomcat. These 2 files are copied under the /tomcat/templates location.
7. The next step is to copy the files into the /tomcat/files location which needs to be copied to the remote location. The files are the java.tar.gz file and the tomcat-initscript.sh location.

8. The file step is to create the actual playbook file for executing the tomcat role. The main.yml looks as
---
- hosts: cent
  roles:
     - tomcat

Once all the steps are done we can see the directory structure as,
[root@vx111a work]# tree tags/
tags/
── ansible.cfg
── hosts
── tomcat
│  
── files
│   │  
── jdk-8u65-linux-x64.tar.gz
│   │   └── tomcat-initscript.sh
│  
── handlers
│   │   └── main.yml
│  
── meta
│  
── tasks
│   │  
── configure_java.yml
│   │  
── configure_tomcat.yml
│   │  
── main.yml
│   │   └── update_path.yml
│  
── templates
│   │  
── server.xml
│   │   └── tomcat-users.xml
│   └── vars
│       └── main.yml
└──
main.yml

You can check the directory structure as above. Let’s run the role which will run the installation, configuration of Java and tomcat.


By this I hope you got a understaning of Roles in Ansbile.I will add more examples using roles. More to come. Happy learnig J
Read More

Ansible Roles

Unilt now we have seen how we can write playbooks and using them with Ansible-playbook command to run them on the remote machine. Ansible playbook is very usefull when we have a limited number of machines and a limited number of changes that needs to be done the remote machine.

Consider a case where we want to configure a system with multiple packages, edit the configuration files using different variables, add files to be copied to remote machines , and add multiple handlers. For this case it is very hard to add all this in one playbook.

Ansible roles are a further level of abstraction that can be usefull for organizing playbook.s
As you add more and more functionality and flexibility to your playbooks, they can become unwieldy and difficult to maintain as a single file. Roles allow you to create very minimal playbooks that then look to a directory structure to determine the actual configuration steps they need to perform.

Organizing things into roles also allows you to reuse common configuration steps between different types of servers. This is already possible by "including" other files within a playbook, but with roles, these types of links between files are automatic based on a specific directory hierarchy.

In this article we will see how we can write a simple role in for installing Apache on the remote machine. Create a directory structure as below.

[root@vx111a roles]# tree apache/
apache/
── files
── handlers
── meta
── tasks
── templates
└── vars

The directories are explained below

Directory
Description
tasks
The tasks folder should contain a main.yml file, which should include a list of the tasks for this role. Any task includes that are contained in these roles will look for their files in this folder also. This allows you to split a large number of tasks into separate files, and use other features of task includes.
files
The files folder is the default location for files in the roles that are used by the copy or the script module.
templates
The templates directory is the location where the template module will automatically look for the jinja2 templates included in the roles.
handlers
The handler’s folder should contain a main.yml file, which specifies the handlers for the roles, and any includes in that folder will also look for the files in the same location.
vars
The vars folder should contain a main.yml file, which contains the variables for this role.
meta
The meta folder should contain a main.yml file. This file can contain settings for the role, and a list of its dependencies. This feature is available only in Ansible 1.3 and above.

Now lets create a necessary files for running the playbook. For the article purpose we will keep this simple.

1) Create a main.yml file in tasks directory which contains the necessary instructions for executing this.

[root@vx111a roles]# cat apache/tasks/main.yml
---
   - name: Install Apache
     yum: pkg=httpd state=latest
     notify:
     - restart apache

2) Now since we have defined the notify we need to define the handler which is created in the handlers folder as

[root@vx111a roles]# cat apache/handlers/main.yml
---
 - name: restart apache
   service: name=httpd state=restarted

Now when you come out the apache folder and use the tree command we can see the directory structure as,

[root@vx111a roles]# tree apache/
apache/
── files
── handlers
│   └── main.yml
── meta
── tasks
│   └── main.yml
── templates
└── vars

We have the main.yml file in both tasks and handler only as we have not defined any variables (var folder) , files to be copied to remote location ( file folder ) and any templates to be editited ( templates folder).
Now let’s create our main playbook file to running the apache role on the remote machine as,
[root@vx111a roles]# cat play.yml
---
 - hosts: cent
   roles:
     - apache

We just defined the role name in the playbook using the roles element.This will make the Ansible playbook command to check for the apache role (directory structure) for running the apache role playbook on the remote machine. The play.yml is located beside the directory apache (not inside the directory). Once we execute the playbook we can see,
[root@vx111a roles]# ansible-playbook play.yml
PLAY [cent] ******************************************************************
GATHERING FACTS ***************************************************************
ok: [172.16.202.96]

TASK: [apache | Install Apache] ***********************************************
changed: [172.16.202.96]

NOTIFIED: [apache | restart apache] *******************************************
changed: [172.16.202.96]

PLAY RECAP ********************************************************************
172.16.202.96              : ok=3    changed=2    unreachable=0    failed=0  


Thus the basics of Roles in Anisble. In the next article we will see a more advanced example of using  roles in Ansible.
Read More