Reproducible Development Environment

Experiments with NixOS, OrgStack and Cursor

I have been hacking on a FUSE filesystem recently and I realised that Linux (as opposed to macOS, which is my primary OS/development machine) is a much better development environment for such a project because

1. Availability of tools like strace

2. More documentation on the internet vs macOS

3. libfuse is more stable in failure modes vs OSXFuse (which means when I did something bad Linux crashes more gracefully than macOS)

Tweet: https://twitter.com/divyenduz/status/1712769548389065079

Demo: https://www.loom.com/share/aef864889f3840c7af85551bd479fc83

To set up a Linux dev environment still using macOS as the main operating system, I was using a hosted linux server (SSDNodes.com) but the latency was too much, so I thought about doing a local Linux VM (NixOS) and SSH into it.

Note: this is not a tutorial, I have skimmed fast through the details of how I made things work. If something is unclear, please feel free to reach out.

Goal

Since I am new to NixOS, the goal is simple, make this FUSE project run smoothly. I don’t have to be a Nix expert today but it should also not come in my way once my happy development path is set up. I should be able to pick my NixOS battles.

Eventually, I would like to move all my development to this setup, so switching to a new laptop can be “ideally“ one command.

Note that this is an experimental setup, I haven’t used it enough yet to comment on, if it is good or not.

Note my understanding of Nix is limited, this is almost the first time I “dove deep” into it. If I am doing something incorrectly, please let me know.

How it Works

For getting NixOS VM, I used OrbStack, this was the easiest part of the process. Going from nothing to a running NixOS VM took less than 5 minutes.

Cursor + Remote-SSH

Cursor (VSCode’s fork) remote SSH was the first hurdle, two problems

1. SSH credentials didn’t work, orb does some magic so, just ssh orb should work™ and it does in Teminal but Cursor failed to resolve the orb URL. So, I had to use the raw connection details from OrbStack.

   

2. remote-ssh didn’t work, in hindsight I know that is because NixOS + VSCode need some work to play together. Mainly, it doesn’t work because the node binary bundled with VSCode/Cursor doesn’t work in NixOS (different architectures). There are three ways to fix this

   1. Copy a working node in the ~/.cursor-server/bin/<version> folder. We will get later to how can you install such a node version in NixOS.

   2. Use nixos-cursor-server, I made this as a fork of nixos-vscode-server, and to my surprise it worked! All I had to do was global rename
      /s/vscode-server/cursor-server/g
      in text and file names and rebuild flake.nix file with
      nix flake update

   3. Use nix-ld, it is a way to “Run unpatched dynamic binaries on NixOS”

This was enough (I tried them all, settled with b.) to make remote-SSH work, however the in-editor terminal is still not working, it fails with:

Interestingly, it works in VSCode, just fails in terminal. I have a hunch about why this might be happening but in the interest of setting up the development environment further, I skipped this for now. I can always SSH with iterm and use tmux (which is also my normal development flow)

libfuse (and dynamic library loading)

FUSE bindings require libfuse to be installed on the machine, installing dependencies is easy enough in NixOS, for global dependencies (and CLIs), edit the configuration nix file

sudo vi /etc/nixos/configuration.nix

and run

sudo nixos-rebuild switch

Here is how the relevant part of my configuration.nix looks like:

My expectation was that, just doing this will both install fuse and make it available for bindings. It did install fuse but the binding command failed with

As the error message suggests, it was unable to find libfuse. Linux has a way to find packages (they are expected to be in well known places) and there are hooks to load them from non-standard paths.

On NixOS, things are installed at /nix/<*> path instead of /usr/lib or /lib for example. I eventually fixed this properly (while building Prisma, further in the blog post) but at this moment, I did what the error message told me to do i.e. find libfuse and put that in PKG_CONFIG_PATH

Then, put the first path (I am not sure why there are so many) in an env var like

PKG_CONFIG_PATH=<path-to-folder-containing-fuse.pc>:$PKG_CONFIG_PATH

I later realised that this is the “wrong” way to do it, and the proper (to the best of my knowledge, please correct me if I am wrong) way to do it is via nix-shell (idk, maybe flake is even better but I haven’t used them yet).

Anyways, setting this env var made “npm install” work which means node-gyp was able to find libfuse and everyone was happy.

Prisma (more dynamic linking)

Prisma’s architecture involves binaries that it downloads in the background.

Prisma’s architecture

The problem is that they don’t have precompiled binaries for NixOS. Probably not for the library or something (I am not sure if this is a bug or not) but this is what happens if I try to run

npx prisma version

I know that you can compile + provide custom binary paths to Prisma CLI and that’s the path I took. That is cloning prisma-engines repo and running

cargo build --release

The problem, cargo expects some libs to be installed at available at standard paths, like openssl etc

The same solutions that were needed for libfuse + bindings were needed here, however, I read a bit more and realized a better way to do dynamic linking using nix-shell

nix-shell provides a temporary shell unique to the project (shell.nix file lives in the project). Using this file, you can easily populate env vars and otherwise the build environment for a project to work. Here is how my shell.nix file looks for prisma-engines repository

(I think it is missing the “correct” way to link openssl, because I patched it the same way I did libfuse, using OPENSSL_DIR env var). For some binary dependencies, it is using nix-ld.

Proper way might be this: NixOS + Libraries

Once cargo build --release worked, I had to make the correct env vars, to do that, I used shell.nix in the main project directory now with following contents

Note: I had to rename the libquery_engine.so binary to libquery_engine.node, to make it work (I am not sure yet where this requirement comes from but only .node works)

Summary

This blog post and my setup is WIP (as you can see) but I am surprised with the amount of good documentation that NixOS has on various topics and I was able to get the thing to work. Overall, I hope to make this VM my primary development environment eventually and open source my configuration.