Habitat 0.22 is out and with it comes changes to the default behaviour of the Supervisor. Pay attention, folks!
Habitat 0.22.x is now in the wild and, with it, comes an all-new studio that automatically loads a supervisor in the background. Sweet. But, also: here be dragons.
Honestly, the Habitat docs describe it well...
The Habitat Studio is a clean, self-contained, minimal environment in which you can develop, build, and package software that is free from any upstream operating system distribution. All tools and dependencies included in the Studio are installed through Habitat packages, thus preventing any unwanted dependencies from being used by your package.
When I am working on a plan, I am not typically developing it inside
the studio. That is to say, I am not entering the studio and firing up
emacs to hack at my plan.sh or any hook scripts. Primarily, I view
it as the place I go to cleanly build and run my plan, while still
under development, without outside influence from that pesky host
More from the docs...
The supervisor is a process manager that has two primary responsibilities. First, it starts and monitors the child app service defined in the package. Second, it receives and acts upon configuration changes from other supervisors to which it is connected. A service will be reconfigured through hooks if its configuration has changed.
Notice that schtick about
receives and acts upon configuration
changes from other supervisors? Supervisors can form "rings" in which
they gossip to each other, sharing pertinent information like "hey,
here's a config change that's just been applied". This enables
services deployed as groups to have their configs all updated
Another important usage of supervisor (rings) is to provide support for different deployment topologies. For example, when your service supports the concept of a "master" and "worker" nodes. I'll get back to this another time. For now, it is enough to know these topologies exist.
Another important thing to note: it is not a one-to-one relationship between supervisors and services. One supervisor can nurture many services. And here is where my idiot's guide starts.
So if it is not clear by now, the idiot in question is me.
When you enter a Habitat Studio in 0.22 you're going to be welcomed by this cheery little message:
** The Habitat Supervisor has been started in the background. ** Use 'hab sup start' and 'hab sup stop' to start and stop services. ** Use the 'slog' command to stream the supervisor log. ** Adding 'NO_BG_SUP=1' to your .studiorc file will disable the background supervisor.
Having a supervisor launched immediately is a nice idea. You can get to the business of loading, running and stopping as many services as you'd like immediately. Which is exactly what I was doing quite happily. Until I continued my work on CrateDB.
I'm not going to talk too much about what my CrateDB plan does. I'm saving that until after I have showed it off at ChefConf.
Well... here's how you would normally start a service using Habitat:
hab start <origin>/<name> or, in my case (because I am using a
hab start <origin>/<name> --topology leader. These
commands instruct the supervisor to go and find the named package and
run it. Simple as that.
Or Is It?
When using the leader-follower topology you need to provide the
address of a peer supervisor to connect to:
hab start <origin>/<name>
--topology leader --peer some.supervisor.ip.address
Notice the idiot mistake? It's kinda subtle. I hope you read those docs I linked, otherwise you ain't got a chance of noticing.
The mistake comes in two related parts:
hab startdoes not simply load your service, it starts a supervisor, too;
--topologyis an option to the service,
--peeris an option to your new supervisor.
So that's the problem: 0.22 gives me a shiny studio with an
automatically-launched supervisor for convenience. But that supervisor
is not part of any ring. When you launch a service using
start... --peer in studio in 0.22, you do not actually launch a new
supervisor, a ring is never formed and so your topology election never
This is easy enough. In fact the solution has been staring you in the face the whole time. Remember this?
** Adding 'NO_BG_SUP=1' to your .studiorc file will disable the background supervisor.
echo "NO_BG_SUP=1" >> .studiorc inside the directory where
you start your studio will do the trick. Sadly, this reverts your
studio to pre-0.22 behaviour, but makes it much easier for you to test
topologies in the studio.