About one year ago, I started running NixOS as my daily operating system, and after this one year and much reflection, I’ve concluded that it is time to switch away form NixOS as my primary system again. This blog post will serve as a snapshot of what it was like for myself to be using NixOS, both a reflection of what I found appealing about NixOS and my own learning along the way, as well as the frustrations that ultimately pushed me away.
What NixOS does well
Closeness of package installation and configuration — system replicability
I have two primary machines that I daily drive, and I want the experience between these two machines to be as identical as possible, meaning that I want the same set of programs installed with as-close-to-identical theming and configurations as reasonably possible. This means keeping both the master package list synced, along with a handful configurations files that is relevant to the usage experience. The problem, however, comes when there are configurations that are intrinsically different the 2 machines: slightly different hardware capabilities requiring different driver packages; different screen real estate requiring slightly different configuration fields; different storage sizes means that backup handling is different and such. When you want something to be “nearly the same”, it gets stuck in the awkward middle ground of not being able to stuff everything in a git repository and call it a day, but at the same time much too cumbersome to maintain two separate instances of the configurations files because too many entries needs to be kept in sync.
The Nix way of configuration systems is glorious: the requirement of a package is almost immediately followed by the configuration snippet, and the configuration being part of a programming logic means that logic can be built into that extract configuration: the same package can be installed with slightly different configuration flag based on the differences between the multiple systems that you are maintaining, all the while retaining identical configurations everywhere.
The fact that nix generates the actual configuration files that are placed in the /etc or other directory during build
time also means that it solves one of my biggest grips managing configuration files on Linux: configuration files can
grow into monstrous multi-thousand line entities, while only three lines are relevant to the configuration that I want
to do. The nix configuration system means that the “configuration” that I interact with (i.e. the .nix file in
question) needs only to keep those relevant lines massively reducing the amount of stuff I need to keep in my
configuration repository.
Another problem that the nix method of configuration solves is that cross-package configurations can now be done in whatever method makes the most sense for system maintainer: for example, instead of all “network configurations” sitting in a gigantic file, with comment snippets that point to the configuration of other programs that may or may not be out of date, nix then allows the network configurations to sit directly behind each package install, so that it is much easier to keep track of why configurations are enabled, and also cleanly disables them when a package becomes obsolete.
The tools provided in nix is the closest thing I got to system replicability (not reproducibility, that is a
distinction that I will discuss in a little later). The fact that I can run nix as a local package manager also means
I can replicate all my terminal experience to the many computing clusters that I need to interact with was the biggest
push for me to start investigating how to use nix in the first place.
Exotic package dependencies and explicitly adding dependencies
Despite the best efforts of distribution maintainers, there will be odd cases where the official package manager does not properly list “all” dependencies. Maybe there is some odd format conversion routine of package “A” that requires extra python package “B” and is not listed because it is not considered a “core” feature of the primary package (One example that I ran into was the conversion o DXF files to regular SVG files using inkscape). So you go ahead and explicitly install this package in the command line package “B”. Great, now there is this dangling package “B” that is labeled as being explicitly requested user, but to the package manager, this package is completely unrelated to package “A”; the problem here goes both ways: if you no longer use package “A” and uninstall it, you now have this unnecessary package “B” floating around in your system; or after sometime when performing system maintenance, you see that package “B” is not required by anything else, so you uninstalled it because you think you don’t need it, thereby unwittingly breaking that fix for package “A”.
The nix way of adding packages to the system means that there such dependencies can actually be properly tracked: for example, when you install a packaged with python as a dependency, you can also specify exactly which addition python packages you want to go along with the python environment in question.
Temporary development installs
C and C++ does not have package managers. So when you need to build a C/C++ project, you might be tempted to install the build dependencies with your system package manager. Great! Now you have a whole host of low-level packages that you don’t know if you can or cannot install after you are done with the project, or after you moved the project else where. This environment isolation problem for C/C++ project is probably why all newer languages try to have this function built in.
The shell function of nix essentially the functionality of project-level package management to anything
you might be using, regardless what language you might be using (even Fortran if you still need to work with that for
whatever reason).
This also extends to the use of temporary packages for interactive sessions. If there is a transient issue that you need
to solve just this once, you can quickly jump into a shell with the temporary diagnostic tools, and cleanly jump out,
where it would look like the package never existed in the first place (the actual files are, of course, cleaned up after
the next nix-garbage-collect call).
Explicitly defining what is exposed to the user — functionality hiding
For certain packages, you might just want some of the functionality to be exposed to the user: An example is that you
might want to have the icat module around from the kitty terminal for the fullest implementation of the
image display protocol, but still want to use your favorite terminal emulator instead of kitty, and/or you
don’t want an extra application entry taking up space in your application manager. In nix, there are ways to make sure
packages are installed, but only exposed to the user through other functionalities.
I add this mainly as a nicety, not strictly as a “must have” for system management. I like having my environment clean, but not to the extent that all unused entries should be purged at all times.
In this one year experiment of daily driving NixOS, it made me think more carefully about what package-manager systems should try and do, what makes for a “good” system maintenance experience, and where other package managers (system or otherwise) are failing to provide tools that is useful. And just for gaining this experience, I would always be happy that I took this route of trying to daily drive NixOS.
And now for the parts of NixOS that ultimately pushed me away from NixOS in search of another solution.
What makes Nix infuriating/frustrating
Doing everything “the Nix way”
While I do think nix configurations is amazing, at the end of the day it is just another layer of abstraction on top of the existing abstractions found in Linux configuration system.
At some points, it felt redundant: Is having a line vim.opt.relativenumber = true in a lua file really that much
different than adding, neovim={enable=True; relativenumber=True} in a nix file? Especially when neovim’s lua file
will eventually contain additional items anyway?
At other points, it’s frustrating when the abstractions are leaky. If the option flag that you are looking for is not implemented in nix, you either have to implement the functionality in nix yourself (after a year I still don’t properly understand how the think in terms of functional programming languages, more on that later), or resort of having the “file contents” be injected into configuration files, at which point the question becomes why don’t I just managed configurations directly with the configuration files instead of having nix as an “extra layer”?
The worst problem is when nix abstraction bleeds into everything in a way that is incompatible with any other systems.
One of the more egregious examples I ended up having was the configuration files I had for zsh, where I realized that
the generation of the zsh is so fragmented that I would be impossible to move away from nix without major refactoring
of how machine-specific configurations are handled. Packages not being able to be ported between Nix systems and other
FHS-based systems was already becoming problematic, and it hit me that if my configurations were going this
route as well, how much am I really giving up for the promise of reproducibility?
No proper method of temporarily pinning a package
While there is a promise of reproducibility and rollback on the system level, what I found in practice is that getting
the nix to actually install packages that was not on the HEAD of the nixpkgs channels was incredibly involved. Here
is an example that actually happened to myself: there was a regression in a GUI application (KiCad in this case),
where I need to roll back to version 8.0.X while the leading branch is at 8.0.Y. To do so in nix, I need to either:
- Split my repository reference
nixpkgsintonixpkgs-unstable(rolling release) andnixpkgs-24.11(latest stable), and point just one specific package to usernixpkgs-24.11, this is, of course, working on the assumption that version that I wanted was still being used in the stable branch, if not, I would need to point to an even oldernixpkgsversion, where there isn’t an easy way to look up which version of in whichnixpkgsbranch, and pray that that branch is still kept in the official nix cache system. If it is not in the cache system, I’m looking at building the “entire” package compile chain (yes, including the all the compilers of low-level languages and all the high-level language interpreter of that specific branch) on my personal machine. - Maintain a temporary branch of the
<package>.nixfile to always use a custom version. (Which is something that I don’t want to do)
When I was using ArchLinux, rolling back to a version is effectively: going to the Arch time machine to find the
snapshot of the package at a certain date, download the package and install it with pacman -U. Sure, now there is no
guarantee that there is package will 100% function as with the NixOS forcing you to replicate the entire environment at
some snapshot, but in most cases, I’m not trying to roll back a package to a completely different version, this is just
a sub-sub version change, where I can most likely track the total number of changes that are done, do I really want to
have a complete separate compile environment in my machine just to roll back?
At least for my use cases, the promise of “NixOS being declarative” feel just a bit broken. In language-specific package
manager (think pip and npm related tool chains), pinning a package version is as
straightforwards as changing a few lines in the configuration to something like package1==x.y.z. There are no such
equivalences in nix, which, at least to myself, defeats quite a bit of the purpose of wanting to claim your entire
system is declarative: You can only declare what is allowed by nixpkgs. I would much rather nix have some equivalent
of pip/npm’s system, where you can explicitly pin package version, and then complain loudly if something is
obviously in conflict.
The fragility of the monolithic nixpkgs repository
Despite the mantra of NixOS being of package reproducibility and dependency isolation, I have found that, in practice,
the rolling release of the NixOS package repository to be surprisingly fragile and prone to breaking, especially when
compared to other rolling release versions like Archlinux. During my one-year span of using NixOS, multiple times the
“stable” version of the Linux kernel and “stable” version of the nvidia drivers where simply not compatible, large GUI
packages having esoteric build fails after a repository update (FreeCad and KiCad are particularly prone to this for
some reason); and to top this all off, the update to the rolling release branch simply is not as immediate as other
distributions. While I found this odd that there are so many hiccups in the package distribution process for a Linux
distribution that has been around nearly as long as Arch, I still understand that nix
expects users to be more hands on with package management. All of this would have been tolerable if it was not for the
final issue that I don’t see an easy way of fixing.
The lack of documentation
This was probably the final straw that pushed to start looking for nix alternatives after the whole fiasco with package
pinning and breaking repositories. Nix, being a functional language, is already hard enough to generalize as
is, and when trying to look up how to do something, there isn’t centralized wiki that I can reference (I mean,
there is, but that hardly contains enough information, and the fact there are two wiki’s
also highlights the problem), and I have to resort to scouring Reddit or Discourse for
solution patches that I don’t fully understand, have difficultly generalizing to a more complete solution, and am not
certain will still be a good solution 6 months down the line. Maybe the documentation is fine, and I just too
procedural-oriented to actually understand now to navigate nix documentations, but having to learn all the oddities of a
functional language, the oddities of interacting with nixpkgs, and having no solid ground to work off in terms of
references is just too much, especially in a pinch when I need to change something “right now”.
What do I actually want in from Nix-like systems
After some long reflections just after the frustrating experience with nix, I wanted to rethink why it was that I was
attracted to nix in the first place. Is it actual reproducibility and declarative-ness? To be honest, not really. I
don’t actually care what my system looks like 100 day ago, and I certainly am not going to dedicate hard drive space for
a snapshot of my machine state to trace these steps. If this is not the actual target that I want out of my system
management experience, I don’t think NixOS is actually the distribution for me.
What I really want out of nix, when tracing the line-of-thought in my initial praise of nix, can effectively be
boiled down to just 2 points:
- Closeness of installation and configurations.
- Ease of temporary installs and cleanup
What I actually want is replicability: the ability to ensure the same experience across multiple machines; while reproducibility should always guarantee replicability, just wanting replicability should not require the amounts of stringent-ness and tie-in as NixOS demands.
Is there a way to achieve what I want without having to completely dive tied to the nix ecosystem, outside of
hand-rolling everything myself?
The alternate solution — decman
Turns out, I’m not the only one what has the same frustrations with NixOS, and a solution for Archlinux has already
been developed: decman!
Breaking it down, decman takes the listed packages (both standard and AUR) and uses this to generate a list of
install/remove commands after comparing this with the current system state. It has very base-line modules to also handle
the generation of files to fixed paths, as well as listing commands to run after installation/upgrades (such as enabling
services), this effectively covers all the basis on my requirements! But what is my operations is not covered by
existing decman functionalities? Because decman is written in Python, it is very easy to patch in
functions. And being python, this is not just limited to the simple stuff like adding machine/user-specific
configurations using if statements on environment. Need something in incrementally update a cfg file? Python has a
module for that! Want to use symbolic links instead of explicitly writing file contents for certain
configuration files? Python has a module for that! Want to download something from a fixed URL to use for
your configuration? Python has a module for that! Want to perform the same operation over a list of
files/modules? You can write looping in whatever method you have in mind!
It felt very freeing to be able to patch whatever function one has in mind directly on-the-fly with whatever modules
Python has to offer instead of having to wade through the fragmented information of how nix/nixpkgs functions on the
internet. The decman package serves as a nice base to start off “80% of the way there”, and you the then free to build
up extra extensions that best suit what you want to do with your system. This ethos I feel is much more fitting of how I
want to manage my personal systems. Nix in comparison feels much more robust (less likely to generate something that can
catastrophically break your system), but at the same time much more rigid (not being allowed to do something outside
predefined limits) to the point of feeling suffocating. A similar statement seems to be from the developer of decman.
What about remote package installation?
One of the big benefits that pushed me to working with nix was the ability to install arbitrary packages on any machine,
regardless of if I have root access or not. If I am no longer using nix, what am I going to do about that? Another
Linux distributions nicely slides into the picture: Gentoo! Their “prefix” project for allowing the
Gentoo build system portage to be portable on any POSIX-like machine is very simple to set up! You
can find the instructions here, and it is basically [3 lines of commands][prefix-boostrap] (minus the many, many hours
you will need to wait for the package compilation)
Actually, that was a lie, because the CERN AFS file system is an incredibly annoying file system to work with. In this particular case, AFS not allowing pipe files to be created, breaking many of the installation scripts handled by the Gentoo package manager. Luckily, Gentoo has the foresight of isolated all building actions into a dedicated
$EPREFIX/tmpdirectory, so as long as you bind that path to a directory that is not under the AFS directory (like the system/tmpdirectory), most install scripts would work. Another outlier is during the installation of thesgmlpackage used for documentation generation, as when it invokes theflockcommand, AFS will effectively freeze up. To get around this, you actually need to installsgmlon another machine, then copy the contents in the$EPREFIX/usr/sharedirectory under AFS. To reiterate, this is not a problem with Gentoo, but a rant of how AFS is a terrible file system to work with from the user side.
Having everything compile from source also has other benefits compared with running nix. Even though
nix-portable tries to be as “native” as possible, nix-portable having to go through proot
still introduces oddities that is unavoidable: users ID needs to be masked, so all other users showing up as nobody
when inspecting system resource usages in htop is one example. With everything being compiled from source, packages
installed with portage is about as native as one can get, with basically mirrored functionality from what I have on my
local installations. This is not to say portage was all smooth sailing: btop simply refuses to compile is oddity
that I have not solved yet, and getting GUI programs such as kitty to compile required additional flags to be toggled
in the prefix system (something that was not immediately obvious to me when I first ran the command). The replicability
aspect right now doesn’t expand beyond keeping a list of packages in plain text, but with proper
configuration file management, it is already providing 95% functionality of what I want out of nix running
on remote machines.
Will you never be using nix?
I think nix still has its place. decman is effectively a wrapper for vanilla pacman, and is not designed to work on
per-directory or per-project bases, only for system level package management. So when I am working on a project that
uses a tool chain that does not have a good in-built package management system (think C/C++), then nix is the perfect
solution for this.
Conclusion
So TL;DR: Got skill-issued by a functional language, so I’m giving up on NixOS for now. The tools that I will now be using be using from now one is:
- Archlinux with
decmanfor all my personal machines. - [Gentoo][`portage`]gentoo-portage for machines where I don’t have root access.
- Nix as the package manager for individual project.
Are these tools perfect? Far from it, and there is much configuration to be done (THE RICING NEVER STOPS). But the biggest upside for using this is I can now say that: