Provisioning – Using Chef with Vagrant

Introduction

This is the third in a series of four posts. In the first post, the devops tool Vagrant was introduced. Vagrant makes it easy to standardise the environment in which software is developed and deployed. By default, Vagrant uses VirtualBox as its standardisation provider. However, as was shown in the second post, it’s easy to use alternative providers such as Amazon AWS, for Cloud virtualisation, for instance.

Up to now the focus has been mainly on handling the life cycle of these standardised environments, using commands such as vagrant init, vagrant up, vagrant suspend,  vagrant halt  and vagrant destroy.     This post shifts the goalposts and focuses on the provisioning of such environments once they have been set up. In other words, how to configure them with the required software stack.

Naturally, there are many ways of going about this. The simplest way is probably to gain access to an instance via vagrant ssh, thereafter installing and configuring the software stack with a succession of regular shell commands. When working from an Ubuntu image towards an Apache installation, the commands might be something along the lines of

This approach might suffice for setting up one or two occasional instances. However, it doesn’t scale and is error-prone.

A better approach is to create a shell script that contains the various installation and configuration steps. The script can be kept in the external Vagrant-initialised directory, and shared with the standardised environment as described in posts 1 and 2. As the environment is brought up for the first time, the script is run, provisioning the system.

This approach is immediately more amenable to automation and its results should be deterministic. Furthermore, since the provisioning instructions are stored as text, they can easily be included in a version control system such as git. This introduces an important mind-shift: treating infrastructure as code. This means that changes in infrastructure provisioning can be held to the same level of scrutiny as changes to software source code. This also implies pull requests and code reviews; being able to easily track changes over time; and, if necessary, being able to roll back to some earlier state.

However, it’s possible to do even better. Wouldn’t it be great to have a central repository of scripts, both official and community-maintained? Or to have scripts that run equally well on different standardised environments? For example, on Ubuntu, Centos, ArchLinux, OS X or Windows Server. Or perhaps to have a setup in which a master instance is able to instruct many thousands of client instances on how they are to be provisioned, or how their provisioning is to change. This is what configuration management tools such as Ansible, Chef, Puppet, Salt and others offer.

Chef

In this post, we’ll use one of these tools, Chef, to set up an Apache web-server. As alluded to above, Chef is ordinarily used in a client-server configuration. However, it is also possible to use a stand-alone version of Chef called Chef Solo. We’ll couch Chef Solo within Vagrant.

The word “Chef” certainly reminds one of food and its preparation. It hardly comes as a surprise that many of the tools and resources that accompany Chef extend this metaphor. So, for example, Chef scripts are called recipes, while a set of related recipes are collected in a cookbook. There is an entire supermarket of publicly available cookbooks (see here). Presently the supermarket contains over 2000 cookbooks, covering a very broad range of provisioning recipes. We’re especially interested in one of these cookbooks: “apache2“.

In what follows, we’ll first make exclusive use of cookbooks written by others. While doing so we’ll accomplish most of what we’d like. However, to round out the exercise, we’ll create a cookbook of our own. In general, apart from introducing new functionality, this allows for more powerful (and nuanced) ways of interacting with other cookbooks. In our case, we’ll use it for a small but important adjustment.

Other cookbooks only

In the same way as apt is a package manager for Debian and Ubuntu, and gem is a package manager for Ruby, a tool called Berkshelf is a package manager for Chef. Currently Berkshelf is bundled with the Chef Development Kit. The “Chef-DK” is available for download for various operating systems. In the case of Mac, the step-by-step installer comes in the form of a dmg file.

Once Chef-DK has been installed on the host system, it’s necessary to update the PATH environment variable so that  /opt/chefdk/bin is included in the first position. Depending on the shell being used, this may be done in  ~/.bashrc  or ~/.zshrc, for example, followed by a  source ~/.bashrc or a source ~/.zshrc, as appropriate.

At this point, the Berkshelf plugin for Vagrant can be installed by executing the following command:

This need be done only once.

Then, as in part 1 and part 2, we can create a new directory on the host machine and Vagrant-initialise it:

As before, this will create a configuration file called Vagrantfile within the  chef_solo_experiment  directory, and Ubuntu 14.04 LTS will be used as the base image.

In the spirit of the “infrastructure as code” theme, the necessary cookbooks and their source should be specified in a dedicated  Berksfile file in the project’s root directory. In our case, we specify that the source of the cookbooks is the URL of the official supermarket, and that we are interested in the cookbook called “apache2″:

Although only the “apache2″ cookbook is listed here, Berkshelf will automatically determine and resolve the full dependency tree.

At this point we require some small modifications to the Vagrantfile:

Here line 3 enables Berkshelf integration, line 4 sets Chef Solo as the Vagrant provisioner, and line 5 adds the default Chef recipe from the “apache2″ cookbook.

Before using this configuration to launch an instance, though, it’s worth remembering that we’re setting up a web-server. Therefore it would be nice to be able to point a browser at it. By adding the following line above line 3 in the snippet above, the Guest OS will be given the IP address 192.168.99.99. However, the address will only be accessible from either the Guest or the Host.

Finally, to launch the instance, we use the regular command vagrant up.

It’s clear from the output that scrolls by that Chef Solo is being installed on the Guest, together with the required cookbook and its dependencies. In fact, the dependency graph is neatly laid out in a newly generated file, Berksfile.lock:

Pointing a browser at the designated address, 192.16.99.99, reveals the following:

Despite the permissions warning, it’s clear that we’ve succeeded in installing Apache. However, we’d like to be able to see the proper “index.html” file. We do this next by creating a very simple cookbook of our own that conveys an additional instruction to the “apache2″ cookbook.

Own cookbook

Another very useful feature of Berkshelf is that it is able to generate the necessary minimal structure for new cookbooks. Execute the following command from within the parent of chef_solo_experiment. It will create the structure required for a cookbook called “mycookbook”.

The command’s output indicates the file structure that it has produced:

As requested, run bundle install to fetch any new gems. (It’s possible that you might need to install bundler first with sudo gem install bundler.)

Probably the most striking of the generated files is Vagrantfile, a file normally generated by the vagrant init command. In this case, the Vagrantfile has already been customised for use with Chef Solo and Berkshelf. Modify this  Vagrantfile to use the same base image and network setup as before:

Let us now create an extremely simple default recipe for the “mycookbook” cookbook. Open the file recipes/default.rb and add the following line:

The default recipe for “mycookbook” therefore does nothing besides import the “apache2″ cookbook.

Open the metadata file so that the pertinent details of the new cookbook can be specified:

In the same file, also specify the cookbook’s direct dependencies. In this case the only direct dependency is “apache2″. Therefore add:

Finally, run vagrant up.

As before, apache2 is installed, and pointing a browser at 192.168.99.99 returns the following message:

This is exactly as far as we progressed in the previous section.

We will now very slightly extend the cookbook’s default recipe. Add the following to the recipes/default.rb Ruby file:

This snippet enables the “000-default” site, in effect performing the following in an abstract, cross-platform manner:

To re-provision the running instance, run vagrant reload followed by vagrant provision. Refresh the browser:

The designated “index.html” shows as expected.

As a bonus, notice that the “mycookbook” directory was initialised as a git repository by Berkshelf. Therefore the updated infrastructure is ready to be committed and subsequently presented as a pull request.

More

Note, we have hardly scratched the surface of Chef in this post. It is truly a huge topic.

• For more information on Chef in general, see here. This page also lists various books about Chef.
• For more information on Chef recipes, see here.
• For more information on the Ruby-based domain-specific language used in Chef recipes, see here.
• Fore more information on Berkshelf, see here.
• For more information on Chef “resources”, see here.

References

• A slightly dated but still relevant comparison of Ansible, Chef, Puppet and Salt.