At TaskRabbit we use Ansible to configure and manage our servers. Ansible is a great tool which allows you write easy-to-use playbooks to configure your servers, deploy your applications, and more.

The problem

Normally, you run ansible commands from your laptop as you need them. This is great when provisioning or deploying, but annoying that it would be hard to automate. Ansible has a product called Ansible Tower which allows you to run those same commands via a web-UI, schedule them, and respond to web hooks. Tower is a nifty piece of software that does a lot of things right, however we were having trouble keeping out inventories (lists of servers) up-to-date between the lists in our ansible git repository and the the Tower server itself.

The main issue is a change in philosophy. Ansible (the CLI tool) expects that your inventories live local to the project in MAKEFILE-like files located in sensible places like ./inventories/production and ./inventories/staging. Ansible Tower expects that your inventory is dynamic, and always obtainable from a remote source like Amazon EC2’s API, or from a VMware Cluster. While we do use these services to host our servers, not all servers that are present should be ansible’d, and more importantly, not all variables that ansible needs will be obtainable from those sources.

In the Ansible project repo, you can keep both the groups and lists of servers, along with variables like this:

1########### 2## HOSTS ## 3########### 4 5mysql-master.myapp.com 6mysql-slave1.myapp.com 7mysql-slave2.myapp.com 8 9redis.myapp.com 10 11web1.myapp.com 12web2.myapp.com 13web3.myapp.com 14 15resque1.myapp.com 16resque2.myapp.com 17 18############ 19## GROUPS ## 20############ 21 22[production] 23mysql-master.myapp.com 24mysql-slave1.myapp.com 25mysql-slave2.myapp.com 26redis.myapp.com 27web1.myapp.com 28web2.myapp.com 29web3.myapp.com 30resque1.myapp.com 31resque2.myapp.com 32 33[production:vars] 34host_memory=8GB 35host_disk=20GB 36ansible_ssh_user=root 37 38## DB ## 39 40[mysql] 41mysql-master.myapp.com 42mysql-slave1.myapp.com 43mysql-slave2.myapp.com 44 45[mysql:master] 46mysql-master.myapp.com 47 48[mysql:vars] 49host_memory=32GB 50host_disk=5120GB 51 52[redis] 53redis.myapp.com 54 55[app] 56web1.myapp.com 57web2.myapp.com 58web3.myapp.com 59resque1.myapp.com 60resque2.myapp.com 61 62[app:unicorn] 63web1.myapp.com 64web2.myapp.com 65web3.myapp.com 66 67[app:resque] 68resque1.myapp.com 69resque2.myapp.com 70

This type of layout allows you to define things in a simple way: — hosts belong to groups — groups can have variables — you can override default variables with later group definitions down the file.

To demonstrate this, you can see how all servers start with 8GB of RAM, but the mysql group later overrides this to 32GB. You also get the added bonus of having your entire infrastructure defined in one place.

Our workflow appends this file when we add and remove servers. This means that with a simple git pull you can be sure that any ansible command you run will be run on the correct collection of servers. We wanted Tower to source the same file developers would be using locally, and not reading in (potentially divergent information) via APIs.

Ansible Tower has a feature called "Dynamic Inventory" which allows you to define your inventory via some other method, as long as it presents a standardized JSON output. Tower can reference these things as what they call an "Inventory Script". Using these tools, the question became: "How can we source a file as if it were a changing API?"

The answer had a few parts (in ruby):

1. Find the inventory file

Tower does not keep the git repo of your ansible project(s) in a single place. It versions them and moves them around as you update it. To that end, finding the most current version of your ./inventories/produciton file is non trivial:

1class InventoryFinder 2 3 def find(inventory_file) 4 # On Production server 5 if File.exists? '/var/lib/awx/projects/' 6 folder = Dir.glob('/var/lib/awx/projects/*').max { |a,b| File.ctime(a) <=> File.ctime(b) } 7 return folder + '/inventories/' + inventory_file 8 # Assume we are within the proper project 9 else 10 return File.dirname(__FILE__) + '/../inventories/' + inventory_file 11 end 12 end 13 14end

2. Parse the Inventory

You can define groups and variables in a few legal ways within an inventory file. You can do the [group:vars] method in the example above, or you can do it in-line as you define the server for the first time. Keeping all this in mind, here’s our parser:

1class InventoryParser 2 3 def initialize(inventory_path) 4 @inventory_path = inventory_path 5 @data = { 6 "_meta" => { 7 "hostvars" => {} 8 } 9 } 10 end 11 12 def inventory_path 13 @inventory_path 14 end 15 16 def data 17 @data 18 end 19 20 def ignored_variables 21 [ 22 'ansible_ssh_user' 23 ] 24 end 25 26 def file_lines 27 File.read( inventory_path ).split("\n") 28 end 29 30 def parse 31 current_section = nil 32 33 file_lines.each do |line| 34 parts = line.split(' ') 35 next if parts.length == 0 36 next if parts.first[0] == "#" 37 next if parts.first[0] == "/" 38 if parts.first[0] == '[' 39 current_section = parts.first.gsub('[','').gsub(']','') 40 if data[current_section].nil? && !current_section.include?(':vars') 41 data[current_section] = [] 42 end 43 next 44 end 45 46 # varaible block 47 if !current_section.nil? && current_section.include?(':vars') 48 parts = line.split('=') 49 key = parts[0] 50 value = parts[1] 51 col = current_section.split(':') 52 col.pop 53 group = col.join(':') 54 fill_hosts_with_group_var(group, key, value) 55 # host block (could still have in-line variables) 56 else 57 hostname = parts.shift 58 ensure_host_variables(hostname) 59 d = {} 60 61 while parts.length > 0 62 part = parts.shift 63 words = part.split('=') 64 d[words.first] = words.last unless ignored_variables.include? words.first 65 end 66 67 data[current_section].push(hostname) if current_section 68 d.each do |k,v| 69 data["_meta"]["hostvars"][hostname][k] = v 70 end 71 72 end 73 end 74 75 return data 76 end 77 78 def ensure_host_variables(hostname) 79 if data["_meta"]["hostvars"][hostname].nil? 80 data["_meta"]["hostvars"][hostname] = {} 81 end 82 end 83 84 def fill_hosts_with_group_var(group, key, value) 85 return if ignored_variables.include? key 86 87 if value.include?("'") || value.include?('"') 88 value = eval(value) 89 end 90 91 data[group].each do |hostname| 92 ensure_host_variables(hostname) 93 data["_meta"]["hostvars"][hostname][key] = value 94 end 95 end 96 97end

You will also note that we choose to ignore certain variables, via ignored_variables, that we want defined somewhere else within ansible tower (for example SSH options).

As a note, one feature of ansible’s inventory DSL that is not supported here is the notion of children

3. Running it

Once those classes are defined, you can create a single file (per environment) like so:

1#!/usr/bin/env ruby 2 3require 'json' 4 5class InventoryFinder 6 #... 7end 8 9class InventoryParser 10 #... 11end 12 13path = InventoryFinder.new.find('production') 14data = InventoryParser.new(path).parse 15 16puts JSON.pretty_generate( data )

You can load this code into the dynamic inventory and it will be ready to run!

4. Keeping it in sync

The final step is to ensure that any time a job is run from Tower, both the project repository and inventory are always updated. There are a few hooks you need to enable to do so:

First, on the setting for the project, you can enable a git pull before each project run. Be sure to enable Update on Launch under SCM options.

Then, the same option, Update on Launch can be enabled under the inventory source. When you define your inventory, you need to source is a "custom script", and from there, you can choose the inventory reader defined above.

With this place, we are able to have our cake and eat it too: — one file which contains all of our configuration — allow developers to keep an up-to-date inventory source locally within the git ansible project — Ansible Tower can source that file, and ensure that it is up-to-date before we run any job

Hi, I'm Evan

I write about Technology, Software, and Startups. I use my Product Management, Software Engineering, and Leadership skills to build teams that create world-class digital products.

Get in touch