Category: Ansible

Using Ansible to provision VMs on AWS

Posted on | by ccadmin | Leave a Comment on Using Ansible to provision VMs on AWS

Using Ansible to provision VMs on AWS

I have been asked on several occasions to show how to use Ansible to provision VMs on Amazon Web Services (AWS). This is “commoditization virtualization” on demand just by running a single playbook, which is pretty cool.

Why automation in the first place?

If your reading this article and have any experience configuring A Unix/Linux/Windows server, whether it is a mail server, web server, whatever, you know how time consuming it is to:

  • Partition the disk
  • Create user accounts
  • Install software packages and updates
  • Configure the server application
  • etc…

You had to wait for the packages to install, configure and test the application to make sure it runs and that can take a few hours.

Now, that the servers are virtual and live in a cloud machine somewhere and you now have to configure more than a dozen of them… that’s a lot of time and you have better things to do. Tools like Ansible are the answer to configuring multiple machines.

Spining up VMs using AWS web console

AWS allows you to log into a web console, choose your VM image and bring them up one at a time. It will give you ssh credentials to allow you to login to the VMs you just made and from there, use Ansible to manage and configure them. Would it be nice if you could manage the provisioning of VM instances from Ansible? Yes you can…but there is a bit of work to make it work.

I will describe the way you can do it with several step script and (in another article) programmatically.

How do I know how many VMs I have in inventory?

The challenge of dynamic inventory is the program/playbook does not know what is in the inventory ahead of time. However, if we apply the cattle not pets approach and let Ansible take care of itempotents of the VMs (it won’t clobber the VMs or exceed the constraints of number of VMs that exist) then this can make our lives easier.

Without knowing the inventory, and checking it ahead of time, you are running blind.

Programmatically is the best way to manage and track dynamic inventory and use Ansibles modules to provision VMs in AWS.

Using a playbook to provision VMS.

In my opinion this is a clunky way to use Ansible.

The problem with this way is there is no clean way to see what is the current inventory that is on AWS, you have to run a separate program before running the playbook so you can see what is currently in inventory.

When this is done you end up writing three or four separate scripts to manage this process. In the long run, this becomes difficult to maintain since you have to look at other scripts to understand what is going on.
Writing maintainable code is a key principle.

Build the playbook to provision AWS cloud services.

Playbook is set to local host.

AWS keys are needed for AWS account access.

BOTO Python API libraries are installed.

Just in case you are not aware, BOTO is the API AWS uses for programmatically managing AWS services.

AWS cloud account

Log into AWS Management Console

Under user account, select “security credentials”

In the left hand column, select user

Select the security tab.

Look for security access key.

This is what you will need for boto/ansible.

Running Vagrant

I have created a Vagrant file with an Ansible playbook for managing AWS through a Linux VM created and managed by Vagrant.

First install VirtualBox then install Vagrant.

Download from Github the Vagrant Ansible AWS files.

Change into the vagrant file directory and type:

vagrant up

It will take a while for all the dependencies to be downloaded.

Once vagrant is fully up, type:

vagrant ssh

to access the shell of the vm.

Preparing for instances.

Change directory to the ansible playbook directory and modify the following files:

aws-vars.yml

and add your AWS keys.

Provisioning AWS instances.

from the shell, type:

ansible-playbook AWS-provision.yml

to start provisioning instances in AWS.

You can watch from the AWS console the instances being provisioned.

Terminating instances

from the shell, type:

ansible-playbook AWS-terminate.yml

to terminate the ec2 instances that were provisioned in your account by the provisioning playbook.

You can watch the instances terminated from the AWS console.

This is only the beginning…

With these examples, we just created self contained machines just by running an Ansible playbook. However, we can setup a virtual container network that allows you to place in a private network such items as private networks where you have access to file servers database servers an “internal” and “external” network with a “firewall”. and more complex designs.

I may cover these examples in future articles.

In the meantime, have a great day.

Do you Get Git? Git for source control and DevOps

Posted on | by ccadmin | Leave a Comment on Do you Get Git? Git for source control and DevOps

Source control is one of the core strategies for DevOps since having developers and operations team members have access to each others processes require a copy of the code and configurations available to everyone involved.
Whether you write code or just have files you continually have to revise, we need to store our files and keep copies safe.
Git provides a solution for source control and collaboration. In fact one of the largest collaborative open source platforms GitHub is built on GIT. They were recently acquired by Microsoft for over 7.5 Billion dollars.

The Problem that GIT addresses.

When I was starting out and didn’t know any better (some people say I still don’t), I would make a copy and store it in a directory. I also may store a new file with a updated name like Original.txt. I would then have a copy named Originalv2.txt Originalv3.txt as I kept making changes and updates to my file.

However, this over time became quite messy to look at and confusing after a short time.

Further, if I wanted to share this group of files with someone, I would have to tar or zip the file up and send it.
On top of this, if I’m working with someone to make changes, they would send me a file and hopefully, I add it to the right place in the project…this gets very inefficient.
This is where source control comes in.
There are many different solutions such as subversion, mercurial, etc, we’ll focus on Git.

What can Git do?

Linus Torvalds and the developers of the Linux kernel needed to replace their proprietary source control software (BitKeeper) and he made a request to the community of developers to create a source control software that had the following requirements:

Open source and free
Distributed not centralized
Able to work offline

Git is Open Source and free. Enough is said, If you need to make changes to the code to the program, you can.  Further, there is no fee to pay for the software.  Support is available from a large and active community.

Distributed not centralized. Everyone has a copy of the files. If you need to see the code that is written or a configuration file, it’s right there.

Able to work offline.  Changes can be made locally and on a central repository. You can work offline with Git. Make changes to your files and commit them both offline and on the network. Many source control systems don’t allow for offline support.

You need source control as you write code. You may make a mistake and need to roll back. Your system may be running abnormally and your files may end up corrupted… but fear not! You made a backup of that program file…or did you… Maybe its an older backup that you need to recover your code from…how far back…

This is where Git is very useful to have. You can make a local revision backup of your program.

Rather than explaining how it works, go and download a copy of Git.

Installing Git

Get git here

or use Chocolaty

$ choco install git

Git supports a variety of ways to setup a repository.
Local, HTTP/HTTPS (smart with authentication and dumb),SSH/SCP
In this article, we’ll cover local.

Setup local repository for your own files

$ git init

then

$ git add file1 file2 file3 …

or

$ git *

then

$ git commit -m ‘initial commit’

There’s more

You can upload your files to a remote repository, clone a remote repository and have this used in a DevOps environment.  It works very well with Continuous Integration/Continuous Deployment where you can setup a tool like Jenkins to start processing the code out of your repository as soon as there is a change.

For more info on Git, download the book ProGit free in e-book format.

Suggestions and ideas for using Git.
Lastly, what good is knowing about a tool without knowing how you can use it?
Many people have found other ways to use Git besides for source control for coding.
Besides using Git for making copies of code on the repository, you can use it for making
backups. People have written articles, books and distributed it using Git.
In systems administration, you can keep critical systems files using Git
I use it for my Vagrantfiles and Ansible playbooks for my systems configurations.

Let me know your comments and suggestions you may have.

Until then, have a good day.

Windows package management with Chocolatey

Posted on | by ccadmin | Leave a Comment on Windows package management with Chocolatey

What does DevOps have to do with Windows package management?

Good question, since maintaining consistency with software tools is a key principle in Development Operations. You don’t always have a choice on what environment or platform you can run and you are stuck with maintaining a Windows environment. Windows has been a pain for myself and many other people taking care of Window machines because the business needs them. Besides the most common problems in maintaining a Windows machine, installation and management of software is a big pain. It’s very likely that you have to support Windows in a desktop environment and server environment. I always thought, being an old school Unix Sys Admin, “Why can’t we have an automated package manager like apt-get, pkgadd, which are available in Unix/Linux in Windows?”
The answer I and many others found is in Chocolatey.

What is Chocolatey?

Chocolatey is a nuget based Windows package manager. Nuget is a .net framework library manager developed by Microsoft. The mechanisms used in Nuget is used for installing Windows packages.
Chocolatey does the following:
1 – Install software
2 – Upgrades software
3 – Remove software
4 – Installs dependencies
5 – Support unattended installs
6 – Support private repositories
7 – Works with automated provisioning systems (Ansible, Chef, Puppet, etc)
8 – Allows creation of private custom packages

Installing Chocolatey.

First, Chocolatey requires the minimum of Windows 7 sp 1 with Powershell Version 3 or later. Windows 10 on up meets the minimum requirement..

Install via powershell
As administrator, run the following on powershell window:
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString(‘https://chocolatey.org/install.ps1’))

https://chocolatey.org/api/v2/package/chocolatey

After you install chocolatey, you can add any package you want.

$ choco install firefox

It will install the latest version of firefox.

To update firefox,

$ choco upgrade firefox

It will update firefox to the latest version.

To list all of the packages you have installed on your machine.

$ choco list –local-only

Chocolatey v0.10.11
7zip.portable 18.5

…

yed 3.18.1.1
84 packages installed.

This creates a list of all applications managed by chocolatey.

There are other things you can do with chocolatey such as creating your own packages, running your on repository and use tools like Ansible, Puppet or Chef to deploy these packages to your Windows machines. It’s beyond the scope of this article. I may cover this in a future article, just be aware it’s available.

Until then, have a good day.

Roy

The virtues of Ansible – Running Ansible on Windows

Posted on | by ccadmin | Leave a Comment on The virtues of Ansible – Running Ansible on Windows

Devops and Ansible

The mantra of DevOps (Development Operations) is Infrastructure is Code.
For those who are not familiar with DevOps, it is a culture and philosophy of computer and network infrastructure management geared around development and delivery of software and services. DevOps uses tools geared towards automation of infrastructure, delivery of software and services.
Ansible is one of those indispensable tools used in DevOps. With Ansible, you can manage your whole network infrastructures , deploy and manage virtual machines, install and configure VMs, provision and update software and services, etc. Written in python, and the configuration files are “programmed” in YAML (Yet Another Markup Language). It is a easy to learn tool for mass provisioning of Linux, Windows, Mac and virtual infrastructure using scriptable command line.
Windows operating system is ubiquitous and is probably one of the easiest, lowest cost hardware/software combinations to get for your testing and development and production environment.

The Windows Dilemma

However if you like me, are running Windows, there is one problem, Ansible does not run on Windows. The developers of Ansible have no intention of porting it to Windows (at this time). Even though, there “is” a way to use Linux like libraries on Windows (using things like CygWin) , however, it’s not very stable. You need a stable platform to do mission critical work. So, how do I run Ansible in Windows?

Introducing Vagrant

The answer to this dilemma is to use Virtualization. “Wait a minute…”, you might say, “Isn’t this a clunky and bulky way to run a tool by getting a clunkier interface to run it on?” This is where a virtualization environment for developers like Vagrant comes into play. Vagrant, from command line allows you to do some cool things:

1 – Use your favorite Windows based editor to edit your playbooks in which you can make changes to your files THEN run them without repackaging the image (as in the case of Docker).
2 – With command line within PowerShell, QUICKLY spin up your Linux VM to run Ansible.
3 – From the same command line, enter the Linux VM command line and run Ansible commands and run your playbooks.
4 – If you want to run the Vagrant VM for a dedicated task involving Ansible, with one command, you can do just that.

For those not familiar with Vagrant, this is a piece of cake.

How to set up Ansible in Vagrant

For the purpose of demonstrating how to do this in Windows, I will show you how to do this in a Windows 7 SP1 machine. I will also assume you are running the latest version of VirtualBox hypervisor. Vagrant will also work with VMWare and HyperV and other versions of Windows (7/8/10), but this will be left (for now) as an exercise you can do on your own.

Also, my preferred way to add software to Windows machines is to use a package manager like Chocolatey. Using automation with package management is the DevOps way. Follow this link for instructions installing Chocolatey.

Steps for setting up Vagrant

1 – Download and install VirtualBox

a – use chocolatey
C:\> choco install virtualbox

b – manually install
Download from here: https://www.virtualbox.org/wiki/Downloads

2 – Download and install vagrant

a – use chocolatey
C:\> choco install vagrant

b – manually install
Download from here: https://www.vagrantup.com/downloads.html

3 – Download this vagrant file from the git repository and unpack it. This is a Vagrantfile template. You can use git to get the file.

4 – Change directory to vagrant directory and run “vagrant up”

5 – Access VM shell, run “vagrant ssh” It will take you into a Linux shell.

6 – To exit out of the VM shell, type ‘exit’ and hit enter. You will be back to the powershell command prompt.

7 – To shut down vagrant and the VM, run “vagrant halt” and it will shut down the VM.

I would recommend reading the README.md file in the directory for additional details.

How do these files work?

This is not a full tutorial on how to work with Vagrant, just enough to get you running with Ansible on Windows. We may cover the particulars of Ansible and Vagrant another time.

What is a Vagrant file?

The Vagrantfile tells vagrant how to provision the VM. The VM will be installed with Python and Ansible. It’s a file written in the Ruby programming language. It’s pretty straightforward how you can configure it to work with your development environment. Anyhow, from here, you can experiment with Ansible on the VM itself.

Once you access Ansible through Vagrant, you can start using Ad Hoc commands and writing playbooks for Ansible and executing them right there. Do some searching on the internet on what you can do with Ansible. It’s a very powerful tool.

That was quite a bit of info, how do I wrap my head around this?

That is the unfortunate nature of any profession, especially the technology profession.  You need to know many different tools and concepts in order to use them, and many of the tools by themselves don’t make much sense until you find a context to use these tools.  Each one of the tools and technologies I used in this intro have their own use in the DevOps area.  You have to dive right in and play with them in order to find out how they can help you to use them in your own particular situation.   I may cover each of these tools  individually in a future article.

Until then, have a good day.

Roy