This post covers day two of CfgMgmtCamp in Gent, Belgium. If you haven't read the first it is here.

Implementing Infrastructure as Code

Kief Morris (@kief) - Cloud Practises Lead at Thought Works and  author of Infrastructure as Code

Morris had always followed the trends in technology. He stayed up-to-date about DevOps, Infrastructure as Code, containers and micro-services. Morris was not a pioneer, he waited like many others for the Infrastructure as Code book. It never showed and after a while he decided to write it himself.

Speed vs Risk

DevOps is about speed and quality, speed helps quality through interation.

There is a notion in technology: you can either move fast and break thing or you can move slow and get things right. Moving fast is the only way to keep things right. Continuous changes to production, in a predicable, reliable manner is the only way to patch a bug.

Morris brought up the issue of configuration drift and server sprawl. Many companies have a massive sprawl of servers and environments. Some are bare-metal, other virtualised and now everything is off to cloud services. This leads to configuration drift between the platforms and environments. Configuration drift mentioned a lot during this years CfgMgmtCamp. To combat drift developers have been moving towards continuous synchronization or immutable infrastructure.

Automation Fear Cycle

Morris highlighted the "Automation Fear Cycle". 

  • Make a change outside of your configuration tool
  • Your servers are inconsistent
  • You're afraid of running your configuration tool it will break something

Small, fast changes to your configuration, rather than large batch changes.


Continuous synchronisation if configurations can lead to "Automated destruction". 

You design a system for automatic configuration change, pushing out every change. You get a systematic method of ruining everything. But with better testing this can catch this.

Dynamic Infrastructure

Organise your infrastructure to be dynamic. Tools like terraform and cloudform allow you to design your infrastructure as code:

  • One definition, many environments
    • You should structure your code to reduce risks by making small, frequent changes. Having one configuration to handle many environments can cause issues.
  • One definition, one environment
    • This can lead to configuration drift. A change made in testing but not QA can be forgot and not copied over.
  • One template to rule them all
    • Differences between the environments are variables which change between environments.

Now that we have our configuration, how can we make sure it is correct? Morris suggests splitting the infrastructure into smaller pieces for testing.

  • Create tests for Roles.
  • Create tests for Servers.
  • Create tests for full infrastructure.

Infrastructure has to be dynamic and allow change. All infrastructure needs to cope with growth, evolving requirements and expanding teams. Today's infrastructure should be designed to:

  • Enable frequent changes
  • Keep units loosely coupled
  • Align with organisational structure

    we can avoid the end goal designs and evolve to meet our current needs.

What do we measure to improve for teams/product development?

  • Measure and optimise the elapsed time from identifying a need to satisfying it. This is your customers and users care about.
  • Rebuild (recover) - 0 to complete infrastructure.
  • Creating new environments.
  • Updating existing environments.
  • Introducing a new test stacks.

Mgmt - Next Generation Configuration Management

James Shubin (@purpleidea) - RedHat

See my previous post. The link above is to James Shubin's Debconf talk in February 2016

My Journey Into technology Through InSpec

Annie Hedgie (@anniehedgie) - Cloud Automation Engineer at 10th Magnitude

How can you help get someone into tech:

Hedgie told the tale of her Journey into technology. Hedgie was a Casting Director for local acts in Texas. One of her biggest titles was There will be Blood, unfortunately she received no credit for her work.. This was one of the things that pushed over the line, the film industry is a long tail industry. Few people are making all the money and the vast majority are working for the glory of it.

Hedgie started making reclaimed wood decorations for a Christmas market. A stall behind her was selling crockpot recipes, simple and easy to produce. They were likely downloaded from the internet and then bound in a notebook. She kept finding herself entering saturated or unvalued markets.

As her children were getting ready to go back to school, Hedgie knew she had to get back into full time employment. She started studying for her GMAT exam and she learnt about the Agile Mindset from Linda Rising.

During an interview, Hedgie found herself sitting across from an sexist, idiot. Somehow, during their conversation Hedgie had mentioned her children. From that point onwards the interviewer kept remarking "Oh, it will be difficult since you have kids." "Don't you feel bad not seeing your kids." The interview ended in shambles. Hedgie decided to start putting together a website to sell her Christmas decorations. On April 21st she push her first commit to GitHub.

Hedgie's husband (a department head in a POS company), invited the creators of InSpec for diner. Hedgie's husband was with bringing velocity to the company. Although he had his security compliance team members blocking him for over year. Hedgie's husband asked her to try InSpec for 3 weeks to see if it was as easy and simple as the creators had made out it was. InSpec is an automated security compliance check framework. If Hedgie a non-programmer could use it, the compliance team could to. Thus began her journey to becoming a Cloud Automation Engineer.

Mentoring and teaching

How can you help get someone into tech:

  • How can you convince someone to give you 3 weeks?
    • How can you give someone 3 weeks?
  • What problem can they solve for you?
  • How can you lend your privilege
    • For someone to learn and to attempt new things?
    • To create an opportunity for someone less privileged than yourself?
  • What is your apprentice excited about working on?
    • How can you allow them to work on it?
  • How can you create a sense of urgency for them?

Hedgie continued to take risks and took opportunities given to her by others. There are many out there lending their privilege for those getting into technology. So find the person your team is missing and go teach them.

TDD for Ops

Thomas Krag (@ildiroen) - Infrastructure Engineer at Ecletic IQ 

Kerim Satirli (ksatirli)- Infrastructure Engineer at OlinData

Unfortunately there are not slides for this talk.

Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is improved to pass the new tests, only. - Wikipedia

TDD is a practice older than the phrase. Developers were practising TDD during NASA's Mercury project in the 1960s. TDD and may other Extreme Programming practices, were in swing long before the 1990s. As the industry has adopted Infrastructure as Code (IaC), we need to adopt TDD. We need testing to be sure we are not doing harm with our changes. As developers do with their code, we will do with ours. 

A way to do this is with RSpec, a technology based on Behaviour Driven Development (BDD, a form of TDD and ATDD). There are alternatives: ServerSpec, TestInfra and Goss.

Krag and Satirli provided a link to a getting started guide for testing IaC.

Unfortunately, myself and another attendee had a small argument with Krag and Satirli. They described TDD as writing all the tests before coding for software development. This is not only wrong, but a dangerous myth about TDD. When they came to describe, TDD for infrastructure they described the correct process. This confused some of the audience of why the process is different. The conversation became weird and awkward about fine detail. It didn't help the discussion and left the session a little bit sour.

Foreman and Ansible Fact Caching

Toshaan Bharvani (@toshywoshy) - IT Consultant at VanTosh

Bharvani gave an annual report on the progress of The Foreman's Ansible integration. Brarvani gave a brief overview of what The Foreman was and its history for those unfamiliar with it. The Foreman is a server life-cyclce management tool, originally a dashboard for puppet. Now The Foreman expects to work with all popular configuration and provisioning tools. Through its plugin frame work, you can extended its capabilities.

There are two plugins required to make The Foreman and Ansible work together:

  • Plugin-ansible
  • Proxy-plugin-ansible

To update the facts used by The Foreman, you need to add The Foreman callback script to Ansible. ~/.ansible/plugins/callback/

The talk ended with a discussing Ansible and The Foreman with Bharvani and the room.

Dropping OS images on bare metal

Lukas Zapletal (@lzap) - Software Engineer at RedHat

Zapletal gave a entertaining talk about bare-metal provisioning in the data center.

Today's data centers are almost completely virtualised. Setting up your full virtualised and managed data centre is the chicken and egg problem.

How do you set up your first machine reliably to set up the rest. An easy answer: You can drop images directly onto bare metal. Tools required for provisioning need to be there to setup everything else. We can setup the provisioning host and deploy images to setup all the hosts based on those images.

Image-base Provisioning

Creating Images

We could create the images by hand (be careful with the size!) and use virt-sysprep to un-configure/depersonalise the installation. The other option could be to use cloud-init, which is locked down but can be preseeded via USB, floppy or the network. For cloud-init the infrastructure to download the image and/or the preseed does need to be in place.


virt-builder can be used to create virtual machine images quickly. It is a fast tool which allows a lot of customization.

Foreman Discovery

Foreman Discovery enables Metal-as-a-Service functionality to The Foreman.

This plugin enables Foreman to do automatic bare-metal discovery of unknown
nodes on the provisioning network. New nodes self-register into Foreman and
upload facts collected by Facter (serial id, network interfaces, memory,
disks).  The registered nodes show up on Discovered Hosts page and
provisioning can be initiated either manually (via UI/CLI or API) or
automatically via predefined Discovery Rules.