Running NixOS on the Banana Pi R4: Building the boot sequence

Before starting

Thanks for your comprehension, and have a great time reading this.

How did we get there

It has been a while since I started waiting for the Banana Pi R4. I thought of everything I could do with my own Linux gateway instead of being forced to get along my ISP’s. As soon as the full bundle with the Wi-Fi 7 card was available and the first reviews appeared, my command was already confirmed. I’ve been waiting it for nearly a year, and initially though that I would just run OpenWrt on it. I mean, It is for sure the easiest solution (even if it would without doubt need some tweaks). However, after a quick thought, I felt like this router would fit even better in my NixOS environment. I mean, I have a centralized NixOS configuration for all my machines that can be easily built from any of them and be deployed on another with a single command. It would make even more sense too cross-compile my router configuration from my PC with its Ryzen 7 5800X and deploy it on the BPI-R4.

However, there are three caveats with this choice:

But I just felt that I would still be more comfortable with another machine in my NixOS environment than with one outside of it that I’ll have to manage its own way. This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.

For your information, this is what I had before starting this project:

For a NixOS SD image generation on ARM machines, some people prefer to build it with their Nix configuration already applied to it, and then just flash into their board. In my case, I prefer the philosophy of having an image that just serve as an installation media, and then manually do all the installation process as I would do on a PC.

Today’s post is the first part about building our boot components and getting access to the U-Boot console.

Let’s get started !

The ARMv8 boot sequence

Before focusing on the Linux kernel or NixOS, we’ll have to build the initial boot sequence for the R4. It would be too easy to just install our bootloader and having an UEFI running it directly right ? The boot phase in ARM-based board is often a bit trickier as it is on x86 hosts. On our case, we need a firmware setting up the ARM secure environment (which is ARM TrustedFirmware-A) and a bootloader (U-Boot).

On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that is run to init our board) is standardized with the following elements:

It’s in fact a bit more complicated than that, but we’ll keep this simple representation.

ARM provides a standard and open implementation for BL2, BL31 and BL32 called TrustedFirmware-A. Its documentation is available here. Finally, our BL33 will be U-Boot, a bootloader widely used for ARM SoCs. Once the board reach U-Boot, it will be able to boot NixOS.

Luckily, nixpkgs provides two functions to build these two elements as Nix derivations: buildUBoot and buildArmTrustedFirmware.

Cross compilation on NixOS (for Nix-beginners)

As I plan to build everything from a x86_64 host, we have to set up a cross-compilation system to build our boot sequence for ARMv8A. Once again, nixpkgs make it fairly easy with its embedded cross-compilation system.

For example, if we want to build any aarch64 package from any architecture (like hello), we can just run:

nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform' -A hello

It is possible, because as any package in nixpkgs, hello is declared through a Nix recipe (a callPackage derivation) that is called by the callPackage function. This function setup a bunch of things and among them cross-compilation by looking at the crossSystem parameter provided to nixpkgs.

To cross-compile every program we need, we just have to create the following default.nix file:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
{
  myProgram = pkgs.callPackage ./myProgram { };
}

Here, we just need to have a myProgram directory with a default.nix file containing a callPackage derivation for our program, and then call nix-build -A myProgram.

Building U-Boot

At the time I’m writing this post, most of current Linux images for the BPI R4 uses frank-w’s U-Boot fork. This is because the BPI-R4 is not supported by mainline U-boot. The modifications frank-w did to bring this support are just few commits. We can easily export them as patch and apply them on top of mainline U-Boot.

Let’s create the following default.nix file:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
{
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
}

In a u-boot directory, we will create a default.nix file and a patches directory that will contain all the patches that I grabbed from frank-w’s U-Boot. Within the u-boot/default.nix file, we will just have to write the following callPackage derivation:

{ buildUBoot, ... }:
buildUBoot {
  defconfig = "mt7988a_bpir4_sd_defconfig";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraPatches = [
    ./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch
    ./patches/0002-Fix-PCIE-on-BPIR4.patch
    ./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch
    ./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch
    ./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch
    ./patches/0006-defconfig-r4-update-with-pcie-options.patch
    ./patches/0007-defconfig-r4-add-pstore.patch
    ./patches/0008-defconfig-r4-update-emmc-defconfig.patch
    ./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch
    ./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch
    ./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch
  ];

  extraConfig = ''
    CONFIG_FIT=n
    CONFIG_USE_DEFAULT_ENV_FILE=n
  '';

  filesToInstall = [ "u-boot.bin" ];
}

The buildUboot function will actually grab the source from the U-Boot repo and apply each patch we give it through the extraPatches argument. It will then copy the defconfig file from U-Boot config folder, and apply the extraConfig content on top. It will eventually compile it and grab the files provided through filesToInstall from the build artifacts and put it into the Nix store.

ARM TrustedFirmware-A

Like U-Boot, the mainline version of TrustedFirmware does not currently support the BPI-R4. For now, Linux distributions are using this fork. I don’t really know for sure who is behind this account, and if I should really trust it. Nonetheless, there is a bit too many commits for me to manage manually, so we’ll here just grab this fork’s sources. As I told earlier, we’ll use the nixpkgs’s buildArmTrustedFirmware function. Before that, and as we can see on the frank-w’s U-Boot build scripts, we’ve to pass the U-Boot binary as input. It’ll be included within the final fip.bin file as BL33.

To be honest there, I don’t really know if TrustedFirmware is mandatory to start our board. Maybe U-Boot could’ve been enough, at the expense of not setting up the ARM secure environment. Maybe U-Boot could itself set up this secure environment. But for now, I decided to proceed like the OpenWrt and Debian images. To confirm, I’ll surely dig in the TrustedFirmware and ARM documentations later. I’ll eventually do a follow-up post later if I achieve to figure out how everything is working.

So, let’s create our callPackage derivation in trusted-firmware/default.nix:

{
  buildArmTrustedFirmware,
  fetchFromGitHub,
  dtc,
  ubootBpiR4,
  ubootTools,
  openssl,
  ...
}:
(buildArmTrustedFirmware rec {
  platform = "mt7988";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraMakeFlags = [
    "USE_MKIMAGE=1"
    "BOOT_DEVICE=sdmmc"
    "DRAM_USE_COMB=1"
    "BL33=${ubootBpiR4}/u-boot.bin"
    "all"
    "fip" 
  ];

  filesToInstall = [
    "build/${platform}/release/bl2.img"
    "build/${platform}/release/fip.bin"
  ];
}).overrideAttrs (old: {
  src = fetchFromGitHub {
    owner = "mtk-openwrt";
    repo = "arm-trusted-firmware";
    rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca";
    hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw=";
  };
  version = "2.10.0-mtk";
  nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ];
}

This is very similar what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation, and then grab the two output files that we need. However, in our situation, there are some extra steps. Both buildUBoot and buildArmTrustedFirmware assumes that you’re building the mainline U-Boot and TF-A. It was the case with our U-Boot build to which, but not for our TF-A. So we need to override the derivation produced by buildArmTrustedFirmware to set the src argument ourselves and pass supplementary build dependencies in nativeBuildInputs.

As we can see, this callPackage derivation needs to access our previous ubootBpiR4 derivation. All the other derivations arguments are provided by nixpkgs itself through the callPackage function, but not ubootBpiR4 that we declared ourselves. Given that, we can just easily pass it as is in the callPackage argument. This makes our default.nix file looks like the following:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
}

Now, we can run nix-build -A armTrustedFirmwareBpiR4 to build everything we need !

Create our boot image

The final step is just to create an empty image and flash what we have built so far at the matching address ranges. This will be done with a simple bash script derivation in the image/default.nix file.

{
  runCommand,
  armTrustedFirmwareBpiR4,
  gptfdisk,
  ...
}:
runCommand "bpi-r4-image" {
  nativeBuildInputs = [
    gptfdisk
  ];
} ''
  IMAGE=$out/nixos-r4-image.img

  mkdir $out

  dd if=/dev/zero of=$IMAGE bs=1M count=4000
  sgdisk -o $IMAGE
  sgdisk -a 1 -n 1:34:8191     -A 1:set:2   -t 1:8300 -c 1:"bl2"        $IMAGE
  sgdisk -a 1 -n 2:8192:9215   -A 2:set:63	-t 2:8300 -c 2:"u-boot-env"	$IMAGE
  sgdisk -a 1 -n 3:9216:13311  -A 3:set:63	-t 3:8300 -c 3:"factory"    $IMAGE
  sgdisk -a 1 -n 4:13312:17407 -A 4:set:63	-t 4:8300 -c 4:"fip"        $IMAGE

  dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync
  dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync
''

The nixpkgs runCommand function creates a derivation that runs a bash script into nixpkgs’s standard environment. We can add dependencies with the second runCommand argument. In this case, we’ll add the gptfdisk package as a dependency to use sgdisk within our script. Finally, the third argument is the script in itself. For the script in itself, we’ll create the derivation output directory and generate a zero-filled 4 GB image with dd. Then, we want to generate our partition table with sgdisk with the following partition map:

Block size of 512 bytes

Blocks 34 to 8191: BL2
Blocks 8192 to 9215: Space for U-Boot environment variables
Blocks 9216 to 13311: Factory
Blocks 13312 to 17407: FIP

Finally, we flash our TF-A build outputs into the matching partitions with dd.

We add this callPackage derivation into our default.nix:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
  image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; };
}

And we can build our boot sequence components with nix-shell -A image to get our nixos-r4-image.img.

Flashing and running

The only thing left to do is flashing our image into a SD card.

dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress

After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot. We get the following logs with minicom:

F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 380E 5800
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0600 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [3000]
MK: 0000 0000 [0000]
T0: 0000 0221 [0101]
Jump to BL

NOTICE:  BL2: v2.10.0   (release):
NOTICE:  BL2: Built : 00:00:00, Jan  1 1980
NOTICE:  WDT: Cold boot
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7988
NOTICE:  EMI: Using DDR unknown settings
NOTICE:  EMI: Detected DRAM size: 4096 MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  BL2: Booting BL31
NOTICE:  BL31: v2.10.0  (release):
NOTICE:  BL31: Built : 00:00:00, Jan  1 1980


U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000)

CPU:   MediaTek MT7988
Model: mt7988-rfb
DRAM:  4 GiB
Core:  49 devices, 19 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from nowhere... OK
In:    serial@11000000
Out:   serial@11000000
Err:   serial@11000000
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
eth0: ethernet@15100000
BPI-R4>

And here we have our initial access to the U-Boot console !

Sources

I successfully got to this point after myself reading some documentation and a bunch of blog posts. Those where the resources I used: