Configuration Management Camp is a 2 day conference about Open Source Configuration management software. Its a well attended event and only a short train ride from Brussels, in the beautiful town of Gent.

I discussed in my previous post, this is the second time I have been fortunate enough to attend both FOSDEM and CfgMgmtCamp. This post will cover the talks at which I was present, and my thoughts on them.

Unfortunately, unlike FOSDEM CfgMgmtCamp only record and made available the main track talks which you can find here.

Configuration Management is a Solved Problem?

John Vincent (@lusis) - MailChimp, Operations Staff Engineer

Vincent has worked as a Sysadmin for 10 years and a retired core organiser for DevopsDays. Back in 2011 during DevOps Days Mountain View, Vincent declaring configuration management a solved problem, the core of the problem at least. Clay Shafer (Co-founder of Puppet Labs) commented at the time that it was not.

It has been 6 years since that talk, Vincent asked what has changed? Configuration management in the terms of: packages, templates, users, files and daemons is pretty much solved. Many tools are available to manage these resources.

Managing orchestration, secret management, application life-cycle and binary distribution hasn't been solved. Vincent believes these are not, they are different problems and should be handled by other tools.

Vincent's vision for the next generation configuration management tool will actively enforce configurations (the model of mgmt). Catalogues will be truly compiled, a single binary which will only configure that server for that catalogue. Configuration management services will become distributed and have multiple masters instead of a single point of failure.

Vincent has an interesting view of configuration management. I agree that the scope of CM should be narrowed to his core list. We need to develop tools more dedicated to being competent managing core resources instead of orchestrating VMs and containers. Other tools managing more complex challenges, like binary distribution, container management and application life-cycles.

Resilient System Require Resilient People

Hannah Foxwell - Product Manager at Server Density

Foxwell presented an excellent talk about resilience - the ability to recover quickly - and the importance of caring for those behind the systems.

"People are not immutable, they are not HA"

"People are more difficult that machines, it is hard to fix a person."

Foxwell discussed how implementing change is hard for organizations and hard for people to deal with. It takes a toll on people, they need time but they can deal with it better if they are taught resilience. Resilience has to and can only be learnt, it is not a trait but a still.

Change is always happening, you cannot "just" upgrade a human. Foxwell argued, people within teams and organizations are treated like software and this is harmful to them. You can't expect them to behave like machines. There is no AB switches in a person's mind, change will happen but it is a slow process.

On Foxwell's first day of work, she was shipped off for a day to a country hotel as part of a graduate training program (corporate brain washing). Although she doesn't remember much of it, she did remember that an employee gave presentation on the "The well of dispair". That they were going to experience a slump of motivation after graduation and that it was okay.

The emotional cycle of change

How to Build a Resilient Team

The culture of a team or organisation can help and benefit our colleges. Even improve their lives, lots of people are anxious due top changes happening to and around them. Fostering an open and supportive team culture can provide the comfort and safety your colleges need.

To make a team resilient you need to:

How to become resilient yourself

I'm with Foxwell in saying that we need to take good care of the people running our systems. This site actually got its name from "half systems, half people", an idea I had about where your effort in a project should go. As a employee of Server Density she is also a champion for #HumanOps which focuses on the human aspect of running infrastructure.

Intro to Foreman

Daniel Garcia (@dLobatog) - RedHat

Garcia provided a quick tour of The Foreman's features, use cases and an overview of how it manages server life cycles.

Unfortunately this talk was not record and I could not find the slides for it. I do have an audio recording if you wish to hear it please just get in contact with me (@neuralsandwich on twitter).

The typical Foreman life-cycle is as follows:

Provisioning --> Configuring --> Monitoring

The Foreman controls the parameters provided to configuration management systems to configure a server. The Foreman currently supports Puppet, Ansible and Chef, through its plugin infrastructure.

The Foreman can setup bare-metal servers though PXE booting and can optionally load The Foreman Discovery. A tool for hardware discovery, it will probe the server for its capabilities and report there back to The Foreman. Virtual machines can be deployed by image or cloud-init.

Provisioning is managed by controlling: DNS, DHCP, ActiveDirectory/IPA realms and PuppetCA autosign. The Foreman will assign hosts their IP address, which allows it to continue to configuration and general management of the server.

The Truth and Nothing but the Truth

Why type systems to configuration management

Henri Linberg (@hel) - Puppet

Linberg discussed truth, its subjectiveness and what that means for configuration management.

He starts with an example, a poster: Dr. Batty's Asthmas Cigarettes: Not suitable for children under the age of 6.

There a many questions that arise from this poster: how can smoking be good for asthma? Is this a scam? Why not for children under 6? What is the truth here?

It turns out these cigarettes did no have nicotine but atropine which, in small doses opens up the air passages in the lungs. It is hard to know the truth when you just look at something.

Linberg proceeded to describe the different types of truth (divine, coherence of belief, logical truth and scientifically observable) and now they derive they compare. Paradoxes are a type of truth but untruth, Linberg describes how we might make sense of them by redefining their meanings but to do so we need higher level of languages than sentences.

So what does all of this have to do with configuration management? CM is meant to setup a machine and prove that it is as described but that description is based on a language. Which ever language the configuration was described in and the ideas that language was designed on. Linberg used this rabbit whole of thinking while designing the Puppet 4 type system. Attempting to create strong but flexible fundamentals which help safely describe the truth of your configuration and not some misunderstanding of it.

(Ab)using Docker for automated testing of Ansible roles on multiple distress with Travis-CI

Bert Van Vreckem - Lecturer ICT Gent

Van Vreckem presented a short but interesting use of Travis CI to test Ansible Playbook configurations for CentOS. By using Travis which only has Ubuntu and macOS available and installing docker, he is able to test CentOS using docker images.

This is a nice little workaround for Travis CI's lack of platform support.

Building robust and scalable environments with Katello

Rich Jerrido - Technical Product Marketing Manager at RedHat

Jerrido introduced Katello a The Foreman project which manages content for server deployment. Katello uses Pulp to manage software repositories and the synchronisation of external sources. Katello also supports the distribution of containers and files.

Katello is a cool tool if you need to manage data/content deployment to servers.

Docker is the New Tarbell

Walter Heck (@walterheck) - Technical directory at OlinData

Heck discussed if the technology we have developed is progress? Are we moving forwards and allowing ourselves to do more or are we just repeating the past in a different way.

He explored several examples

Exhibit 1: Configuration Management

Configuration management replaces bash scripts for server setup. We no longer need have to write scripts. But we still do, just in a different language. There have been many problems solved in the mean time but we still have to write code to manage the configurations of servers.

Exhibit 2: Docker is a new Tarball

Instead of given code to users, we provide docker images. Or docker images based on public docker images with no docker file. It is still up to others to figure out what to do with it or how to use it. This isn't simpler, is it better?

Really this should not be about the technology but the communication around that technology. Communication on the use of technology is key.

Exhibit 3: Amazon replaces VMWare

No more long term hardware lock-ins with long contracts. Now we have cloud based vendor lock-in. If you implement your services in the amazon way, you end up locked into the amazon cloud. There is some benefit, you can move faster but you haven't really innovated.


Heck describes the goal as reducing complex it but questioned if complex system could be described or created in a non-complex manner. He questioned who was asking for all of the current innovations? Many of these new technologies bring their own issues to the table when adopting but they do save us some time.

The main take away from the talk was that IT serves as a business function. It has to do what is best for the business. There are times that the business doesn't know what is best and needs guidance.


Day one of CfgMgmtCamp was interesting. It lacked some of the technical content of` day two but I enjoyed the main tracks focus on the concepts, state of the field and the human aspect of managing infrastructure. It is important that we all remember what is important in the field rather than focusing on minutiae and technical details which are ultimately irrelevant.