iPXE Bootable NixOS Images with GitLab CI and Chainloading

NixOS, especially config.system.build.* can feel pretty esoteric some times. I’ve had my share of tussles with the infrastructure, but I have enough of an understanding to provide working examples and advice here. This guide is intended to help the reader set up loading custom NixOS images through CI artifacts and iPXE with the minimum of local infrastructure. It requires good knowledge of NixOS, a GitLab account, a DHCP server, and a server you can use for TFTP.

What is iPXE

iPXE, also referred to as netboot in the NixOS codebase, is a network-integrated bootloader. It is built on top of regular PXE, a common standard supported by most (wired) NICs. Most modern computers should have settings in the BIOS and boot menu options to boot from PXE. PXE boot will standardly load a binary from a TFTP server provided by DHCP. My DHCP server runs pfsense, where it was a matter of turning on tftpd and using the network boot subsection in the DHCP configuration to set the ‘next server’ (the IP of the TFTP service) to the same host, and the BIOS file name to undionly.kpxe. Undionly.kpxe is an iPXE binary meant to be loaded from an already existing PXE environment, and this is exactly how I use it. You can download a copy, or compile your own (which I’ll explain later).

It has a network stack built in, and is designed to make booting (mostly Linux and BSD) images from FTP, HTTP/S, SANs, and such. I’ve used it together with https://netboot.xyz to provide easy access to different live and rescue boot options from my home network without bothering with a rescue CD. Netboot.xyz works because of a feature iPXE has called chainloading, where it can load scripts dynamically from other sources.

Chainloading allows iPXE to pull in code from a different source, and boot according to that source rather than relying on user input for selecting a kernel image, initrd, or kernel arguments/run-time options. It does this by downloading text script from user-provided (or pre-programmed) location, written with a pre-defined set of keywords like any other scripting language.

My Chainloading Setup

If you just load undionly, you will keep connecting to DHCP and downloading the same file. You can escape to a command line and do things manually, but you probably don’t want that. Instead, you need to embed a script into a custom version of undionly. To do this, you need just two files.

with import <nixpkgs> {};
stdenv.mkDerivation rec {
  name = "ipxe-menu";
  rev = "0b3000bbece3f96d1c4e00aaf93968f6905105bb";

  makeFlags = [ "bin/undionly.kpxe" "EMBED=${menuscript}" ];

  src = fetchgit {
	inherit rev;
    url = "https://git.ipxe.org/ipxe.git";
	sha256 = "15jld9fkcxc8jvqpcv9yx3dkzw7g42867xb0dcwc9prdmkziz464";
  };
  
  # clone gives us a directory with the first seven digits of the rev at the end of it
  # sourceRoot = "ipxe-0b3000b/src";
  setSourceRoot = "sourceRoot=$(ls|grep ipxe-)/src";

	prePatch = ''
		substituteInPlace config/general.h --replace "#undef	DOWNLOAD_PROTO_HTTPS" "#define	DOWNLOAD_PROTO_HTTPS"
	'';

  menuscript = ./intogit.ipxe;

  installPhase = ''
    mkdir -p $out/
    cp bin/undionly.kpxe $out/
  '';

  buildInputs = [
    binutils gcc gnumake perl lzma
  ];
}

This first file is a build script for my custom version of undionly. It will take intogit.ipxe, a file relative to this file’s location, and compile it into a copy of iPXE that also has https enabled. DO NOT chainload with iPXE from the internet, you are putting yourself at a huge risk of a MITM attack.

#!ipxe

dhcp

chain --autofree https://gitlab.com/6AA4FD/netboot/-/raw/master/menu.ipxe

This is my first iPXE script. iPXE takes a pretty long time to compile, so I don’t want to recompile it every time I need to change something. This file will likely never have to change, it will just sit in a TFTP directory and point my iPXE clients to a different file. In this case, the next file is another ipxe script, but one that is not compiled (a .kpxe file) but interpreted (a .ipxe file).

This menu in turn has an option for loading https://netboot.xyz, but if you want to test something right away, try setting the thing after chain –autofree to https://netboot.xyz and try booting a live “cd”.

My menu script is a little bit more complicated, but I’ll get to that later. First I have to explain how you can make a NixOS image for use with iPXE.

NixOS Images

NixOS is an operating system that builds most of its environment from code managed in a monorepo. It includes a programming language and interpreter for this, and users make changes to their system by changing a configuration file, and rebuilding it with nixos-rebuild. In a similar fashion, we can use a configuration file to build a .iso file for making a regular live/installer usb, and even for making the files suitable for ipxe boot. So what is necessary for iPXE boot?

iPXE boot generally requires at minimum a kernel, and normally an initial ramdisk as well StackOverflow. NixOS has mainline code for generating these things automatically, kept in nixpkgs/nixos/release.nix that you can call easily to test. It may take a little while because of mkSquashFs, but you can run nix-build -A netboot '<nixpkgs/nixos/release.nix> and have an example of this to look at yourself. It produces a bzImage, an initrd, a .ipxe script, and a lib directory and System.map we don’t need.

If you like this image as-is, you’re in luck. Just upload it to file hosting, and run chain --autofree https://path.to/netboot.ipxe and it will parse the relative paths in netboot.ipxe and load everything right. Awesome!

However, I don’t have a lot of interest in an image like this. I want something I can preconfigure and use without any stateful shenanigans. So here’s how I produce that. You can check my GitLab for the normal file I use that also holds a preconfigured iso image, but here are the relevant parts.

{ system ? "x86_64-linux", nixpkgs ? import <nixpkgs> { inherit system; }, nixos ? import <nixpkgs/nixos> { inherit system; } }:

let

	nixosWNetBoot = import <nixpkgs/nixos> { configuration = { imports = [ <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix> ]; }; };

	mkNetboot = nixpkgs.pkgs.symlinkJoin {
		name = "netboot";
		paths = with nixosWNetBoot.config.system.build; [
			netbootRamdisk
			kernel
			netbootIpxeScript
		];
		preferLocalBuild=true;
	};

in {
	pix.ipxe = mkNetboot;
}

Just save this as default.nix, add your configuration options to configuration under nixosWNetBoot, and run nix-build -A pix.ipxe and you will be ready to go.

Serving it With GitLab

You may not want to pay for your own hosting, or you may not want to manually build and upload these files to it. This is where GitLab CI comes in. GitLab provides free CI minutes for every user, and you can configure them with a .gitlab-ci.yml file in the root of every repository you own. Here is mine, adapted to a minimal example.

image: nixos/nix:latest

pix-ipxe:
  script:
    - nix-build -A pix.ipxe
    - mkdir -p result
    - cp -rLf result results-pix-ipxe
  artifacts:
    paths:
      - results-pix-ipxe/
  only:
    - tags
    - triggers
    - schedules

You can go without everything under only: if you just want to build every commit, but I git push frequently enough that I don’t want to accidentally waste a bunch of CI time and electricity. This will provide your netboot.ipxe file under https://gitlab.com/<yourusername>/<yourproject>/-/jobs/artifacts/master/raw/results-pix-ipxe/netboot.ipxe?job=pix-ipxe and then you can just point chain --autofree and it will work as far as I can tell.