First, getting macfuse setup on my system via Nix isn’t something directly supported. I had to go into the Nix derivation for e2fsprogs and remove the guards against running on darwin. Additionally I found e2fsprogs doesn’t even compile. setxattr and getxattr can’t be assigned to because the function signatures don’t match. So I make them match with best guess by adding some parameters and declaring a couple of others as unsigned (link will come to the commit by the time I publish). I believe this is because osxfuse has some differences in the API, or perhaps there are version incompatibilities. It builds after that. I struggle with nix-flakes for a bit but eventually I notice that fuse-ext2 is indeed on my PATH - but I’m not certain which of my incantations brought it about. I’m still very new to nix-flakes and much of it was forced upon me from other things I desired. I basically jumped in the cockpit seat of the nix-flakes plane and grabbed the stick, hoping for the best. I haven’t collided with the planet Earth yet but there’s still time for that.

I try to mount my disk with invocations such as these:

sudo fuse-ext2 /dev/disk4s1 /Volumes/4s1
sudo fuse-ext2 /dev/disk4s2 /Volumes/4s2

Because I don’t know which one is which and there are only two partitions for disk4.

I don’t have the exact message or dialog screenshots unfortunately, but this is where I find out that I indeed need to disable SIP (System Integrity Protection). This involves rebooting the system into recovery mode, firing up the terminal there, and running csrutil disable. uptime reports I’m at 70 days - a lot of days earned via hard won knowledge on battery draining activities on my laptop (such as bluetooth). My pride suffers a blow, but I sally forth. Then I spend about 7 reboots or so trying to get my laptop to actually boot into recovery mode.

macOS has changed their reboot key bindings more than they’ve changed their power adapter connections, and that’s saying a lot. Naturally I searched for how to do this, and was happy with the holding of Cmd+R during the boot-up. It doesn’t work, which in my experience means the timing is wrong, so I try it several more time. The technical timing is important so I try it more than once, which is how we wind up with me going to 7. The 7th attempt was me finding out those instructions were for Intel Macs and I’m on Apple Silicon. Right - glad we have that distinguishment. My mind immediately demands “Why? Why make these different?” but answers I can come up with are depressing in a very literal way.

I do the csrutil disable which prompts me but works anyways. Knowing I have done the nasty successfully and my work here is done, I reboot to go back to my normal runtime. I attempt to mount again with:

sudo fuse-ext2 /dev/disk4s2 /Volumes/4s2

But again am prompted - my Mac tells me it has blocked the kernel extension, and I must reboot to bless it. Great. Couldn’t we have just done that earlier? Or maybe just let me queue up here at least so I don’t need to go into a mode where I can’t call upon my lifelines for help. Alas, more reboots ahoy. At least this time they give me really vague instructions. This probably would’ve confused me into doing more reboots and searching, but I’ve already been here. Now that I’ve blessed this kernel extension from Benjamin (who is that?), I go back to try it again. Still blocked. The System Settings section tells me there’s a kernel extension that needs to be blessed - it doesn’t give me indication that I’ve blessed anything already. Ugh. Why is this such a miserable experience? Again I reboot into recovery mode, bring up the Security Utility, and bless the kernel extension. This time it goes away. I was operating in the suspicion that there might be more than one kernel extension involved, and all of them must be blessed individually.

Now I try again. I have some vague memory that mount processes don’t typically create the actual mount point directory in Linux, so I assume it’s the same here.

sudo mkdir /Volumes/4s2
sudo fuse-ext2 /dev/disk4s2 /Volumes/4s2

Nothing happens. The directory is empty and I have exit code 253 from fuse-ext2 (having your prompt show the last exit code is so incredibly useful). What’s 253 mean? I check the man page - nothing. From searching on other people trying things out, I add a -o debug to the invocation. No output, same code. I add -v to the front of the from/to arguments. No change in output/code. I worry this is resulting from my hack earlier to e2fsprogs. But I would expect to see something here. There is mention of a /var/log/fuse-ext2_util.log in some tickets I see, but I don’t have that.

I try:

sudo mount -t fuse-ext2 /dev/disk4s2 /Volumes/4s2

mount: exec /Library/Filesystems/fuse-ext2.fs/Contents/Resources/mount_fuse-ext2 for /Volumes/4s2: No such file or directory
mount: /Volumes/4s2 failed with 72

The No such file or directory portion can’t be correct on the latter path - /Volumes/4s2 exists. I check the /Library/Filesystems/... path and it is indeed non-existent. Curious, looking in /Library/Filesystems directly shows me something of relevance:

ls /Library/Filesystems

NetFSPlugins
macfuse.fs

So I try this modified mount:

sudo mount -t macfuse.fs /dev/disk4s2 /Volumes/4s2

macFUSE mount version 4.5.0

This program is not meant to be called directly. The macFUSE library calls it.

Available mount options:
    -o allow_other         allow access to others besides the user who mounted
                           the file system
    -o allow_recursion     allow a mount point that itself resides on a macFUSE
                           volume (by default, such mounting is disallowed)
    -o allow_root          allow access to root (can't be used with allow_other)
    -o auto_xattr          handle extended attributes entirely through ._ files
    -o blocksize=<size>    specify block size in bytes of "storage"
    -o daemon_timeout=<s>  timeout in seconds for kernel calls to daemon
    -o debug               turn on debug information printing
    -o default_permissions let the kernel handle permission checks locally
    -o defer_permissions   defer permission checks to file operations themselves
    -o direct_io           use alternative (direct) path for kernel-user I/O
    -o extended_security   turn on macOS extended security (ACLs)
    -o fsid=<fsid>         set the second 32-bit component of the fsid
    -o fsname=<name>       set the file system's name
    -o fssubtype=<num>     set the file system's fssubtype identifier
    -o fstypename=<name>   set the file system's type name
    -o iosize=<size>       specify maximum I/O size in bytes
    -o jail_symlinks       contain symbolic links within the mount
    -o local               mark the volume as "local" (default is "nonlocal")
    -o negative_vncache    enable vnode name caching of non-existent objects
    -o sparse              enable support for sparse files
    -o volname=<name>      set the file system's volume name

Available negative mount options:
    -o noalerts            disable all graphical alerts (if any) in macFUSE Core
    -o noappledouble       ignore Apple Double (._) and .DS_Store files entirely
    -o noapplexattr        ignore all "com.apple.*" extended attributes
    -o nobrowse            mark the volume as non-browsable by the Finder
    -o nolocalcaches       meta option equivalent to noreadahead,noubc,novncache
    -o noreadahead         disable I/O read-ahead behavior for this file system
    -o nosynconclose       disable sync-on-close behavior (enabled by default)
    -o nosyncwrites        disable synchronous-writes behavior (dangerous)
    -o noubc               disable the unified buffer cache for this file system
    -o novncache           disable the vnode name cache for this file system
mount: /Volumes/4s2 failed with 64

So mount is looking less promising. This appears to be mount just spewing the usage information of macfuse because it is giving macfuse an erroneous invocation. Is this intended to ever work? I found this invocation from other lost souls who were experiencing no luck here as well, so it might not mean much.

Following some other tickets, I want to just confirm that everything I have is in order. Oftentimes errors come from the user, not the software. I find sudo diskutil list gives some good answers.

<snip>
/dev/disk6 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *127.9 GB   disk6
   1:                 DOS_FAT_32 FIRMWARE                31.5 MB    disk6s1
   2:                      Linux                         127.8 GB   disk6s2

I have been operating with the notion that I was using disk4, but between reboots and swapping the physical card out to stow my laptop between attempts, it must have moved. Sigh. Let’s do this again. At least I know with confidence that I want to be mounting the “2” partition.

sudo fuse-ext2 /dev/disk6s2 /Volumes/4s2

I didn’t rename the directory to a matching 6s2 because I just want to try something. Results are promising and with a delay:

Mounting /dev/disk6s2 Read-Only.
Use 'force' or 'rw+' options to enable Read-Write mode

And then:

ls /Volumes/4s2

Wow something worked! I’m going to treat myself.

Now let’s clean up the directory name, and make it read+write.

sudo umount /Volumes/4s2
sudo mv /Volumes/{4,6}s2
sudo fuse-ext2 -o rw+ /dev/disk6s2 /Volumes/4s2

Actually just trying to scroll through some shell history wound up making the machine unusable, and it eventually necessitated a reboot. I’ve lost any home-made directories under /Volume and so I can just make them the way I want them. I am a little worried that maybe my hacks I made to g2fsprogs have introduced some critical instability to my kernel, but I fail to see how the xattrs stuff would have anything to do with my shell history. Perhaps it’s due to some zsh magic?

The commands again, and note that now we’re back to disk4:

sudo mkdir /Volumes/picard
sudo fuse-ext2 -o rw+ /dev/disk4s2 /Volumes/picard

And that worked! Now to make edits to the card.

I did forget about networking, which isn’t covered in the boot section linked above. I had worked on this earlier but the 169.254.0.0/16 address space is way too large to nmap over. Instead I read in the boot section there’s an “OTG” (On The Go) USB setting that is enabled by default in config.txt (at least in my case, being it’s a Pi 4B). I don’t want to just plug it in and find out, and I actually did try raspberrypi.local earlier, I thought, and that didn’t work. I found a ping in my history to it in fact, so I know this was attempted and failed. There may be something else involved. This answer specifically talks about macOS and OTG. It does mention to use Bonjour to find the Pi, which apparently it advertises on.

The command link invocation is provided in this answer:

dns-sd -Gv4v6 raspberrypi.local

That doesn’t work. I do recall some chatter in the OTG+MacBook answer that talked about some problems with OTG and certain cables. It links to an article about an issue with USB-C on the Pi that is probably causing me problems. Essentially it means that the Pi isn’t powered properly if the USB-C cable is “e-marked” (meaning it can identify/query additional features).

So how do I know if the Pi got sufficient power or not? I can check the SD card to see if the configuration files were removed on-boot.

ls /Volumes/FIRMWARE | grep ssh
ls /Volumes/FIRMWARE | grep userconf

ssh
userconf

Sure enough, they are both present. So the Pi didn’t boot properly. I did have a solid red light, with an occasionally blinking green light. I don’t know what that means yet, but perhaps I should learn. I have read it is programmable and may have changed over time, so I don’t want to go too far down that path. In any case, we didn’t make it. For what it’s worth, the Pi was connected for a solid 5-10 minutes.

I only glanced at the USB-C diagrams in the article linked prior. I also skimmed through parts of the article itself. I would need a specifically bad USB-C cable - it would need to transmit data but not be “e-marked”. I suspect that would be hard to find.

I did learn somewhat recently that USB cables (even USB-C) sometimes are hard-wired to actually be a true “charging cable” in that all they do is provide power, and all of the data pins are grounded or shared. That’s not part of the spec, and it explains why a lot of people get cables that “work” but sometimes don’t (via reviews or one’s own personal experience). It’s a real shame we don’t have standard, human-readable markings for these differences. This video was really enlightening on the topic, and also shows off some fascinating circuitry in the thunderbolt cables which share the USB-C form factor.

So I need to plug this in via a battery or wall outlet, and I need to run an Ethernet cable between the Pi and my laptop. I do have the equipment on hand for all of this.

I have blinking Ethernet lights. The Pi has a solid red light, and the green light is doing a dot-dot kind of pattern, repeatedly.

I’ve done a little looking into for the light patterns. The Pi 4B uses a new set of patterns. However my 2 pattern (two of anything but nothing else) doesn’t show up on the list. Checking my command history from earlier, I can see that I’m using a NixOS image. Perhaps that is related. I got that from this instruction available on the official NixOS documentation. It does assume a 4B model, but might be demanding more RAM than what I gave the Pi. My purchase order for the Pi shows it as 2 GB, but the documentation says it should be 4GB. I don’t know how hard that limitation is. Also, and perhaps most critically, this cannot work because it doesn’t use the Pi’s normal configuration. Okay now I think we’re onto something. I did want to do this via NixOS but I think I might be best off saving that for another day. This post has gotten enormous from all of the research and attempts already. I’ve also noticed the documentation tails behind the current NixOS image version. Ugh. I just need to do this myself - later.

I came across Headless Raspberry Pi on NixOS Discourse and user Sweenu states they have a working version of NixOS being deployed to a Raspberry Pi. The source is all posted to https://github.com/sweenu/nixfiles. From some brief research, it does use a tool called digga that states that it is no longer recommended for use, which is expanded upon in issue#503 for the repository. In essence, digga seems to have some customization / scaling difficulties because it focused on ergonomics first. There isn’t any real alternatives recommended that are purpose built for what digga does (which, to be honest, I’m not entirely sure what it does). I’d prefer to keep my research to shiny new things to a minimum, so I hope I can adlib what should be already working for someone else, but to fit my needs. Primarily I want image generation that gives me a working NixOS based Raspberry Pi image, but also have it configured in such a way that I don’t have to re-flash the SD card every time I want to make a change. In other words, I want to be able to build an image but also be able to manage the host configuration via a standard Nix configuration.

In the Discourse, Sweenu states the Pi can be updated thusly:

nix flake update
deploy ".#grunfeld"

Where grunfeld is the name of the Raspberry Pi.

In the repository’s README, it states the image can be created and copied like so:

nixos-generate --flake '.#grunfeld' --format sd-aarch64 --system aarch64-linux
unzstd -d {the output path from the command above} -o nixos-sd-image.img
sudo dd if=nixos-sd-image.img of=/dev/sda bs=64K status=progress

This is all very promising. I’ve done some copying/adlibing of the repository in my proton-nix repository, which I plan on using to declare my local network configuration. I do have some security concerns about open sourcing it, especially because both public and private keys seem to be present in the repository (even if the private keys are encrypted at rest). I would like a way to share this, without the private bits hanging out for all to see. This has been something I’ve looked into before with Nix and nix-flakes in particular because flakes really wants everything to be self contained, lest it be marked as “impure”. From reading Ian’s blog I mentioned earlier, it does seem like one can reference a file using a git repository link. If flakes is happy with this, then I think I could be very happy with it as well.

But first, we need a detour. I don’t have a means of actually using NixOS because I’m on macOS. As I understand, this does require work with a container or VM.

On this current MacBook, I’ve successfully avoided running Docker. I’ve come to liken Docker to bloatware (whether that’s fair or not). On my home network, I’ve had a degree of success with Podman. However there’s hitches with it, as there always is. I also have a very small amount of space available on my MacBook, and containerd containers aren’t known for their small size - unless they use Alpine, I guess. Either way it will need an entire VM to work - because macOS doesn’t have cgroups et. al. There appears to be some promising work done to make containers work on macOS but it’s way too immature (as of [2023-12-24 Sun]) for the kind of work we’re doing here.

My detour needs an additional detour: I have to clean up some space. I presently have < 30GB free and I would love to be running with closer to 100GB to feel like I can quickly iterate on things without constantly needing to do a nix gc or docker system prune --force, and also thereby increasing my build iteration times significantly (those caches are present for good reason).

Begin space cleaning montage!

Alright I’ve deleted a bunch of things I had no business keeping, and things I probably won’t remember that I deleted later. ncdu was instrumental there!

In a vacuum I’d prefer podman to avoid my perceived bloatware of Docker proper. I don’t recall any current perils, and there does appear to be some support for running podman on aarch64-darwin (my specific setup), so let’s give it a shot.

I did find nixpkgs#169118 which outlines some issues with podman on macOS, but they seem to be resolved and nobody feels confident enough to declare it done. This commit that refers to the ticket just installs qemu and podman together (but with no direct relationship - I don’t know how that’s supposed to work). qemu is a significant part of working with the VM, I’ve gathered.

I found Just, Nix Shell And Podman Are A Killer Combo and learned that Just is a thing. The post has some great incantations in it that I was happy to leverage into this with-podman.sh script:

#!/usr/bin/env bash
set -euo pipefail
container_name="nix-run"
script="$@"

podman machine init --cpus 12 --memory 8192 --disk-size 50 \
      --volume $HOME:$HOME || true
podman machine start || true
podman container ls -a | grep $container_name > /dev/null || \
        podman create -t --name $container_name -w /workdir \
            -v $PWD:/workdir nixos/nix
container_id=$(podman start $container_name)
echo "$container_id"
podman exec $container_id $script
podman stop $container_name || true
podman machine stop

Which basically sets up the Podman VM, starts the container nixos/nix container (named nix-run), executes the script (whatever I pass in for args), and then cleans it all up. It’s very expensive but it’s also something I can use thoughtlessly and I really like that.

Here’s an example run:

@ ./with-podman.sh ls -al
Error: podman-machine-default: VM already exists
Error: cannot start VM podman-machine-default: VM already running or starting
a34354199be6cca047309f9d29ecefeeb1c4521d776536cc6305bad8380874b6
nix-run
total 12
drwxr-xr-x 11 root nobody  352 Dec 24 13:17 .
dr-xr-xr-x  1 root root     77 Dec 24 13:26 ..
drwxr-xr-x 10 root nobody  320 Dec 24 13:23 .git
-rw-r--r--  1 root nobody  283 Dec 22 00:50 README.org
-rw-r--r--  1 root nobody 3809 Dec 24 00:29 flake.nix
drwxr-xr-x  3 root nobody   96 Dec 24 00:30 hosts
drwxr-xr-x  4 root nobody  128 Dec 24 00:40 modules
drwxr-xr-x  3 root nobody   96 Dec 24 00:42 pkgs
drwxr-xr-x  3 root nobody   96 Dec 24 00:52 profiles
drwxr-xr-x  4 root nobody  128 Dec 24 00:41 shell
-rwxr-xr-x  1 root nobody  529 Dec 24 13:26 with-podman.sh
nix-run
Waiting for VM to exit...
Machine "podman-machine-default" stopped successfully

~/dev/proton-nix logan@scandium 0 [22:27:20] 75s
$

Note it took 75 seconds to run on my system just to do ls but it works without me doing anything fancy, I’m running NixOS, and the repository I’m working in is properly volume mounted. I’m not really worried about speed here, especially because these invocations I’m about to do will generate disk images - an already lengthy affair I expect to take on the order of minutes. Perhaps I will find short cuts to speed up iteration as I experience pain with building the image improperly, but I want to reserve my grease until the wheels begin to squeak.

A couple of asides:

1. I looked at Just since this is my first exposure to it. From some quick reading around, it seems to be a marginally better make. make is portable, standard, and yes it has aged but I believe it has aged moderately well. I would not drop portability and familiarity (due to it being fairly standard) just for a marginal improvement. Plus, if the file starts to get too squirrelly, I’d rather just break out the complex bits into standalone scripts. If I really could wave a wand and have anything I wanted in terms of love to make, it would be to give me something like rake -T where I can simply print all of the tasks available to me. Reading the code is not documentation, even if it is a Makefile.
2. I tend to go down the rabbit hole on a lot of these ventures, but I’m not posting all of it as I go. Indeed, I just closed a tab pointing to r/programminganimememes, which was serving as nothing other than pure distraction. I’d followed link to it from the same article linked above.

Now that I have a real Nix system available to me, I need to try to run my paired down repository from sweenu and see if I can build an image from it.

The Pi seems to “boot”, but results in the double-green flashes I had before I went down the NixOS image approach. I happen to have a second Pi with me, and have tried it there, as well as a second card with the same image. I don’t feel like this has ruled out hardware problems.

My card is from SanDisk, and is the SanDisk Ultra 128GB microSDXC UHS-I Card (SDSQUNC-128G-GN6MA) variant. https://elinux.org/RPi_SD_cards indicates that this should work. Many of the recent SanDisk cards are reported as working. The Amazon reviews say that they specifically work for Raspberry Pis as well. Even though I got them from the same batch (as well as the Pis), I suspect this is not a hardware issue, but I haven’t ruled it out.

I need a display to see if there’s some other critical information before I can continue.

I did install f3 utilities to test it (with f3read and f3write) but I haven’t taken them for a spin yet.

My travels are now complete, and I’m able to inspect things with my home network. A curious thing that came up was the displays ports. The new (circa 2023) Raspberry Pi’s come with micro-HDMI connections for display. I did some searching around and found that HDMI actually requires royalties are paid (perhaps the Pis are exempt?) at the cost of $0.15 USD or $0.05 (or lower) if the HDMI logo is displayed per Wikipedia’s HDMI article. The Pi has the logo printed on PCB. The capabilities between HDMI. are on par with each other, from what various posts I could find. Apologists have reported that HDMI is more universal, and thus more in line with the goals of the Pi to make cheaply accessible computers. I still had to buy an adapter to connect it to any of my displays made in the last 10 years. Granted, the adapter wasn’t terribly expensive. It did halt my work though. Now I have one, and a display to connect it to. The results were surprising!

I can see the Pi has actually booted up, and it works! It successfully boots, and the flashing green light must just be “activity” that is on some regular interval. The Bluetooth or WiFi radio, perhaps? Either way, I have a working Pi! Let’s see if I can log in.

$ ssh iron.proton
The authenticity of host 'iron.proton (192.168.254.102)' can't be established.
ED25519 key fingerprint is SHA256:E3b/T0lQiqItcIKbh2n6Wb/sOEV9nFbrVojJXlKmHrQ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'iron.proton' (ED25519) to the list of known hosts.

[logan@iron:~]$

Wow! I’d forgotten I’d added one of my authorized keys to the list. I have to admit, I’m pretty giddy. I do have a password to update. A quick passwd invocation shows the initialPassword field is good, but you won’t see anything interesting here:

[logan@iron:~]$ passwd
Changing password for logan.
Current password:
New password:
Retype new password:
passwd: password updated successfully

And then to see if I can use sudo:

[logan@iron:~]$ sudo ls

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

For security reasons, the password you type will not be visible.

[sudo] password for logan:

[logan@iron:~]$

So this ticks all of my boxes for getting a Pi configured on the baseline, but lets go through the list to be sure:

• [X] The Pi has the desired hostname.
• [X] The Pi has my user.
• [X] My user can be logged into via SSH.
• [X] My user is a sudoer.

That’s everything!