Idiot's Guide To The Supervisor In Habitat 0.22

by Paul Adams

Habitat 0.22 is out and with it comes changes to the default behaviour of the Supervisor. Pay attention, folks!

Habitat 0.22.x is now in the wild and, with it, comes an all-new studio that automatically loads a supervisor in the background. Sweet. But, also: here be dragons.

Studio And Supervisor: A Reminder


Honestly, the Habitat docs describe it well...

The Habitat Studio is a clean, self-contained, minimal environment in which you can develop, build, and package software that is free from any upstream operating system distribution. All tools and dependencies included in the Studio are installed through Habitat packages, thus preventing any unwanted dependencies from being used by your package.

When I am working on a plan, I am not typically developing it inside the studio. That is to say, I am not entering the studio and firing up emacs to hack at my or any hook scripts. Primarily, I view it as the place I go to cleanly build and run my plan, while still under development, without outside influence from that pesky host operating system.


More from the docs...

The supervisor is a process manager that has two primary responsibilities. First, it starts and monitors the child app service defined in the package. Second, it receives and acts upon configuration changes from other supervisors to which it is connected. A service will be reconfigured through hooks if its configuration has changed.

Notice that schtick about receives and acts upon configuration changes from other supervisors? Supervisors can form "rings" in which they gossip to each other, sharing pertinent information like "hey, here's a config change that's just been applied". This enables services deployed as groups to have their configs all updated together.

Another important usage of supervisor (rings) is to provide support for different deployment topologies. For example, when your service supports the concept of a "master" and "worker" nodes. I'll get back to this another time. For now, it is enough to know these topologies exist.

Another important thing to note: it is not a one-to-one relationship between supervisors and services. One supervisor can nurture many services. And here is where my idiot's guide starts.

So What's Up In 0.22?

So if it is not clear by now, the idiot in question is me.

When you enter a Habitat Studio in 0.22 you're going to be welcomed by this cheery little message:

** The Habitat Supervisor has been started in the background.
** Use 'hab sup start' and 'hab sup stop' to start and stop services.
** Use the 'slog' command to stream the supervisor log.
** Adding 'NO_BG_SUP=1' to your .studiorc file will disable the background supervisor.

Having a supervisor launched immediately is a nice idea. You can get to the business of loading, running and stopping as many services as you'd like immediately. Which is exactly what I was doing quite happily. Until I continued my work on CrateDB.

I'm not going to talk too much about what my CrateDB plan does. I'm saving that until after I have showed it off at ChefConf.

The Automatic Supervisor Trap

Well... here's how you would normally start a service using Habitat: hab start <origin>/<name> or, in my case (because I am using a topology): hab start <origin>/<name> --topology leader. These commands instruct the supervisor to go and find the named package and run it. Simple as that.

Or Is It?

When using the leader-follower topology you need to provide the address of a peer supervisor to connect to: hab start <origin>/<name> --topology leader --peer some.supervisor.ip.address

Notice the idiot mistake? It's kinda subtle. I hope you read those docs I linked, otherwise you ain't got a chance of noticing.

The mistake comes in two related parts:

  • hab start does not simply load your service, it starts a supervisor, too;
  • --topology is an option to the service, --peer is an option to your new supervisor.

So that's the problem: 0.22 gives me a shiny studio with an automatically-launched supervisor for convenience. But that supervisor is not part of any ring. When you launch a service using hab start... --peer in studio in 0.22, you do not actually launch a new supervisor, a ring is never formed and so your topology election never completes!

The Remedy

This is easy enough. In fact the solution has been staring you in the face the whole time. Remember this?

** Adding 'NO_BG_SUP=1' to your .studiorc file will disable the background supervisor.

A simple echo "NO_BG_SUP=1" >> .studiorc inside the directory where you start your studio will do the trick. Sadly, this reverts your studio to pre-0.22 behaviour, but makes it much easier for you to test topologies in the studio.