There's no place like ~

Dotfiles are not programs

This solves a certain set of problems, namely the job of getting your configuration files synchronized between machines (usually with the help of source control like GitHub). However, the configurations themselves are sometimes only valid for a given version of a program, with some specific set of features, which is outside of the purview of a dotfile management tool. And rightly so, the job of these tools is to help you keep your configuration safe and reproducible should you have to move to a new machine or clone your configuration.

However, this does not solve the underlying problem, one of consistent user environments, only provides us with ways to make it a bit less painful. Given the right set of constructs, one can cobble together something resembling a consistent user environment through scripting and other hacks (as I have with chezmoi in combination with brew and miniforge) but it’s definitely a bit fragile and requires a certain discipline to ensure that you only edit your configuration files through the tooling and only install new packages through your home-baked bootstrapping scripts.

A $HOME of cards

Buried inside of my dotfiles repo is a collection of one-off scripts, templated configuration files, and data files that manage to squeeze out something nominally tolerable. The only way I make sure that I don’t break things is through a pre-commit Git hook that re-generates all my configuration files and runs some scripts to install things before allowing the commit to happen. Unfortunately, even though Homebrew does a decent job of working on both macOS and Linux, I still need to install some packages from the native package manager (usually pacman but sometimes apt), which requires a song-and-dance of installing things on macOS through a Brewfile that’s templated. The templating allows me to skip installing packages on Linux that either do not work correctly or are installable through the system package manager.

My ideal setup

Ideally, my compute environment would be:

• Portable
• Reproducible
• Type-checked

Portable - in that I can take my configuration files from a macOS system to a Linux system, or perhaps even a Windows system, and it “just works”. I have all the tools I want available at the expected versions and I can upgrade all of them just by changing the version numbers. In addition, one requirement that my current setup does not achieve, is that it must also not require system-level access to achieve its final state.

Reproducible - in that, given a system in any state, I can re-create the end state exactly the same way every time. If I were to delete all of my home directory and then re-run this process, it would be back as though I had not removed my home directory (arguably minus my personal data files, though theoretically those could be a part of this.)

Type-checked - any configuration I create has the ability to ensure I haven’t done something silly. For example, in my current setup I can guarantee you that an errant quote or other minor change in my shell scripts could go undetected until it does something incorrect. This incurs extra mental overhead to ensure I don’t break things in addition to making sure they work on multiple systems.

Cue Nix

Nix (and it’s related project, NixOS), for those unaware, is a project that is working to provide a system for fully-reproducible builds. This is something that I came to love having worked at Amazon / AWS, and something I often miss in my every day life. Often, people will use containers (or jails, VMs, etc.) to achieve this but it doesn’t integrate well with day-to-day life. These are okay for reproducible builds - in that we can easily generate some sort of binary by installing things, injecting source code and then building it; however, attempting to use this process as part of a user’s workflow is either slow, requires a ton of tooling, or is cumbersome. Nix takes the approach of building software in isolated environments from the ground-up (i.e from the C library all the way up), and allowing for composable environments, without using containers or virtual machines.

Nix, by itself, targets independent environments – such that you define an environment complete with packages, hooks and other things – and then you switch from your shell “into” that environment. Additionally, this description of the environment is in fact a complete program on its own! It takes inputs (sets of packages, configurations, etc.) and then transforms those inputs into an output (the environment you are going to get) using the Nix programming language. You can think of it sort of like a container, but at the user-space level; no system modification occurs, root access is not required, no need to mount directories into a container, and so on. Nix checks the boxes of being reproducible and type-checked.

Additionally, there are tools that sit on top of Nix, among them is an awesome project called home-manager. As one might infer from the name, the goal of home-manager is to extend the reproducibility and type-safety into the way that users manage their home directories on UNIX-like systems in a programmatic, reproducible, way. In theory, this is like the holy grail of home management. No longer do you need to ask yourself “is froopy-tool v0.4.5 installed on this machine?” because the answer will always be “yes” if you describe it in your environment using home-manager.

It was amazing!

Once I discovered home-manager it seemed too good to be true – I could define my home directory, complete with the exact set of packages I wanted to install, and I could do this across both Linux and macOS systems. A simple example is configuring Git – I always forget exactly which sections there are in a git config file, but home-manager has a Nix program to define the configuration, so I could easily have the following snippet:

  config = {
    home.packages = with pkgs; [
      git
    ]

    programs.git = {
      enable = true;
      aliases = {
        undo = "reset HEAD~1 --mixed";
      };
      extraConfig = {
        color = {
          ui = "auto";
        };
        pull = {
          rebase = true;
        };
      };
    };
  };

And, when running home-manager switch, it would build git (or fetch it from a cache of pre-built Nix package), and then use a custom Nix program included with home-manager to programmatically generate ~/.gitconfig as well as install git into my environment. As a bonus it works on anything that supports Nix!

The bad…

But Nix can only ensure that its model works in an environment where it owns everything. In order to produce consistent output, it needs guaranteed well-known inputs (as we say: garbage in, garbage out). One of the biggest challenges that we, as software engineers, face is the level of abstraction required to smooth out the differences between systems. Given the rate of changes to software it’s definitely an uphill battle sometimes; one that requires constant attention, updates and management to keep in check.

Unfortunately, at the end of the day, the differences are enough that things don’t work as well as I had originally hoped. It’s not that the majority of things didn’t work; in fact, it was the opposite – most things worked as I had expected. The problem (at least for me) is that it’s that the times when it doesn’t work, you don’t have much recourse rather than to dig into the underlying problem. In my case, it was something as trivial-seeming as installing an R package that failed and the amount of work needed to figure out why went beyond the amount of energy I wanted to invest into the problem. I think the one downfall of Nix (and I am not alone in this opinion as far as I can tell) is that using it well requires a steep learning curve to use it correctly, beyond just poking buttons and hoping it works.

For me, I need something that is transparent enough that I can fix it quickly and get on with things. So, as a result I ended up abandoning Nix (for now) and settling on hammering together a combination of chezmoi, Homebrew and shell scripts to at least meet somewhere in the middle.

The ugly

So, with all of that behind us, here is what I settled on:

However, because I am lazy / forgetful / absent-minded I can’t really rely on myself to remember to do these things all the time. In particular because the act of editing my dotfiles really isn’t just vi ~/.zshrc any longer - chezmoi requires that you edit the dotfiles stored in its repository and then apply the changes. Editing the dotfiles can either involve editing the files directly inside the chezmoi repository (usually located in ~/.local/share/chezmoi), or invoking the more convenient chezmoi edit ~/.zshrc which will figure out which template file to edit and open it in $EDITOR for you. So, I did what every lazy person does and I wrote a ZSH function, edit_dotfiles to automate the things I need to do so I only need to remember one step.

… something Rube Goldberg would be proud of

edit_dotfiles takes a list of files, invokes chezmoi edit on each of them in turn, and then adds them to git with a simple commit message. Running commit invokes a pre-commit hook that does two things: generates my chezmoi data file (with my per-machine / per-environment config data such as e-mail address) and then runs chezmoi apply. chezmoi apply in turn uses the config data and the templates to render the actual dotfiles on my system and then invokes any run_ scripts in the repository. One of these will run homebrew bundle which will install any homebrew packages that are missing. And, because these happen in a pre-commit hook, if they fail then the changes won’t get committed (and therefore not pushed)

So the flow looks something like this:

Making it work

function edit_dotfiles() {
  if (( $# == 0 ))
  then echo usage: edit_dotfiles path ...; fi
  for i; do
    chezmoi edit $i
    echo "Committing changes to $i..."
    ( cd $HOME/.local/share/chezmoi && git add . && git commit -m "Update to $i")
    ( cd $HOME/.local/share/chezmoi && git push origin main)
  done
  echo "Reloading ZSH config..."
  source ~/.zshrc
}