Categories
Docker Software

How to build x86 (and others!) Docker images on an M1 Mac

TLDR;

Want a set of commands you can copy/paste? Jump to the TLDR; at the bottom.

Background

I jumped on the Apple Silicon band wagon as soon as I heard how awesome they were and I was not disappointed. My Apple Silicon MacBook Air is now my daily driver that I use for work as a software engineer and for personal projects.

I extensively use Docker in the projects I work on, so this led to a unique problem. When I build a Docker image on my Mac – it builds an ARM version (specifically arm64). This means this image can’t run on any other device like a raspberry pi (linux/arm/v7) or a typical server (linux/amd64) as the application binaries inside are not compatible.

Fortunately, Docker has supported cross CPU architecture builds for a while now through an experimental feature called buildx. It’s a CLI plugin that integrates the Moby BuildKit toolkit. This allows you to build a Docker Image for a variety of different CPU architectures and it uses QEMU under the hood to do the emulation.

How to build a multi-architecture Docker Image on Apple Silicon

This guide assumes you have an Apple Silicon equipped Mac running macOS Big Sur. It was written with an Apple M1 equipped MacBook Air so results may vary across devices.

Step 1: enable experimental Docker Desktop features

The Docker buildx feature is currently “experimental” so we need to enable Docker Desktop’s experimental feature support.

To do so, open up Docker Desktop then navigate to Preferences.

Once you’re there, select “Experimental Features” and toggle the slider to on. Click on “Apply & Restart” to save the changes and restart the Docker daemon.

If you can’t see an “Experimental Features” option, you may have to sign up for the Docker developer program at this link. I suspect it’s a new thing which is why only recently created accounts seem to need to sign up.

Once you’ve enabled experimental features, you can close the Docker Desktop preferences. In your terminal, open the folder that contains the Dockerfile you wish to build for multiple architectures. Run the docker buildx ls command to list the current builder instances. You should see something similar to below.

$ docker buildx ls              
NAME/NODE DRIVER/ENDPOINT STATUS  PLATFORMS
default * docker                  
  default default         running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6

Next create a new builder instance with docker buildx create --use so we can perform multiple builds in parallel. Without this step, you’ll have to use the default Docker one which only supports a single platform per build. You’ll see it created if you run docker buildx ls again.

$ docker buildx lsNAME/NODE          DRIVER/ENDPOINT             STATUS  PLATFORMS
reverent_banach *  docker-container                    
  reverent_banach0 unix:///var/run/docker.sock running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6
default            docker                              
  default          default                     running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6

Now you can use buildx like below to start a multi-architecture build. You’ll have to push it straight to a registry (either the public or a private one) with --push if you want Docker to automatically manage the multi-architecture manifest for you. Don’t forget to tag it (the example is using the open source MemberMatters software I wrote) and add a list of all the platforms that you wish to build for. You can see the compatible platforms from the previous docker buildx ls command. The command below will build an image for both Apple Silicon Macs (linux/arm64), and standard x86 platforms (linux/amd64).

$ docker buildx build --platform linux/amd64,linux/arm64 --push -t membermatters/membermatters .

The first time you run a build, you’ll have to wait for the Moby BuildKit image to download so you’ll see something like this.

$ docker buildx build --platform linux/amd64,linux/arm64 --push -t membermatters/membermatters .
[+] Building 16.5s (7/43)                                                                                                                                 
 => [internal] booting buildkit                                                                                                                     10.4s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                   8.1s
 => => creating container buildx_buildkit_admiring_shirley0 

Once the build is finished, it will be automatically uploaded to your configured registry. Docker will also automatically manage the manifest list for you. This allows Docker to combine the separate builds for each architecture into a single “manifest”. This means users can do a normal docker pull <image> and the Docker client will automatically work out the correct image for their CPU architecture – pretty neat!

TLDR; Version

  1. Open the Docker Desktop dashboard then open up Preferences (cog icon). Go to “Experimental Features” then turn it on and apply it.
  2. Next create a new builder instance with docker buildx create --use. This lets you specify multiple docker platforms at once.
  3. To build your Dockerfile for typical x86 systems and Apple Silicon Macs, run docker buildx build --platform linux/amd64,linux/arm64 --push -t <tag_to_push> .
  4. Done. Please note that you have to push directly to a repository if you want Docker Desktop to automatically manage the manifest list for you (this is probably something you want). Read the paragraph above to find out why. 😉

Categories
3D Printing Linux Raspberry Pi Software

How to get great webcam quality with Octoprint

I have OctoPrint set up to help me manage my 3D printer, record timelapses and remotely monitor it. OctoPrint is a great tool and something I strongly recommend to everyone. I’ve also setup a webcam so that OctoPrint can stream the footage in real time, and also create awesome timelapses. I have a Logitech C920 – one of the highest regarded webcams available, and yet I was still getting poor results. Read on to find out how I fixed this.

The Logitech C920 is a great webcam and can be had for as little as $100. However, in the case of mounting it up close on a 3D printer with a quickly moving subject, the auto focus and exposure really struggles. I’ve got a fixed mount, and a consistent lighting set up in my printer’s enclosure so there’s no reason for the focus or exposure to be adjusted once it’s correct.

Luckily, it’s easy enough to manually configure these settings. These commands were tested on a Raspberry Pi 3+ Model B with a Logitech C920 webcam. Results may vary on other setups.

First of all, you’ll want to turn off auto exposure by running the following command on your OctoPrint Pi:

v4l2-ctl -c exposure_auto=1

Once you have turned off auto exposure, it’s time to play around with the exposure value. A good starting value is probably around 600, but it varies with your specific set up. You should play around with this number by adding or subtracting 100 at a time until you get a good quality image. You can open up the OctoPrint webcam stream while you’re doing this. Run the command below to set your exposure to a value of 600:

v4l2-ctl -c exposure_absolute=600

Now your exposure is dialed in, it’s time to move onto the focus. This was the biggest problem for me as the webcam was hunting for focus all the time resulting in a near constant blurry image. Turn off auto focus by running the command below:

v4l2-ctl -c focus_auto=0
A picture of my 3D printer (CR-10 v2) webcam setup. The printer control box and OctoPrint Pi are off to the left out of frame.

With auto focus off, we can now play around with the focus settings. My webcam is mounted above my 3D printer towards the front, and is about 50cm away from the build plate (pictured above). Start with a focus value of 1 and then work your way up from there by adding 1 at a time. If you’re not seeing much difference between the values, try jumping up 5 at a time then fine tuning it when it’s almost there. Set the focus value to 1 with this command:

v4l2-ctl -c focus_absolute=1

Putting it all together

Now that we’ve got all the settings dialed in, it’s time to make them stick. By default, these settings won’t persist if you reboot your OctoPrint Pi. First you should combine all the commands together like the code snippet below, be sure to replace with your dialed in values:

v4l2-ctl -c exposure_auto=1 && v4l2-ctl -c exposure_absolute=600 && v4l2-ctl -c focus_auto=0 && v4l2-ctl -c focus_absolute=2

You’ll need to open a config file called /home/pi/mjpg-streamer/start.sh. Note, some users have reported that /etc/rc.local is now the correct place (you’ll need to use sudo when editing this). You may have to try both. Use your favourite text editor. If you’re new to editing files on the command line, you should look up how to use nano and come back here once you’re familiar. Create a new line at the very end of the file and the code snippet from above there. Save and exit. Now whenever OctoPrint starts up the mjpg-streamer service your custom camera settings will take affect.

Following these steps helped me dramatically boost the quality of the live stream, and timelapse footage of my prints. It went from over exposed and fuzzy to nice and clear with a good exposure.

Note: I’ve been using the excellent “octolapse” plugin recently which has similar functionality (and a GUI!) built right in. So I’d recommend to give that a go if you don’t mind the extra plugin and complexity.

Categories
armbian Raspberry Pi Software

The fastest way to clone an SD card on macOS

If you have a raspberry pi or other single board computer and would like to make a backup of it or even clone it to another SD card it can take a long time. Your first thought is to probably use the built in “Disk Utility”. Unfortunately this has issues reading linux partitions (well in my experience) and is often slow. This simple command line trick will have you copying or cloning a full disk image of your SD card in record time!

WARNING: Be very careful when running any command with sudo dd in it. If you type any of the parameters incorrectly you may accidently erase or overwrite important data.

Requirements:

  • macOS running a recent version (this guide was tested on macOS Catalina).
  • basic knowledge of command line operations.
  • Make sure you’ve got homebrew installed. You can visit this link to find out how to download and install homebrew if you haven’t already got it.
  • After you’ve installed homebrew, you’ll need to install a package called core-utils. Do so by running brew install coreutils in your terminal. It should take a few minutes to run.

Identify your sd card:

You’ll need to find out which disk your SD card represents. You can run diskutil list and should see an output like below:

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.0 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD — Data     396.0 GB   disk1s1
   2:                APFS Volume Preboot                 81.9 MB    disk1s2
   3:                APFS Volume Recovery                528.5 MB   disk1s3
   4:                APFS Volume VM                      4.3 GB     disk1s4
   5:                APFS Volume Macintosh HD            11.0 GB    disk1s5

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *31.9 GB    disk4
   1:             Windows_FAT_32 boot                    268.4 MB   disk4s1
   2:                      Linux                         31.6 GB    disk4s2

From that output we can see that our SD card must be /dev/disk4 as our card is 32GB in size and has a fat32 and linux partition (standard for most raspberry pi images). You should add an r in front of disk4 so it looks like this /dev/rdisk4. The r means when we’re copying, it will use the “raw” disk. For an operation like this, it is much more efficient.

Copy the SD card as a disk image (dmg)

Now you should run the following command, replacing 4 with whatever number you identified as your sd card:

sudo gdd if=/dev/rdisk4 of=sd_backup.dmg status=progress bs=16M

Tip: you can experiment with different numbers for the block size by replacing bs=16M with larger or smaller numbers to see if it makes a difference to the speed.

You should see some progress feedback telling you the transfer speed. If you’d like to experiment with different block sizes, just type ctrl + c to cancel the command, then you can run it again.

Once the command has finished running, you’ll end up with a file in your home directory called sd_backup.dmg. If you’d like to backup multiple SD cards (or keep multiple backups!) simply replace sd_backup.dmg with a different file name. This will contain a complete disk image of your SD card. If you’d like to restore it, or clone it to another SD card, read on.

Copy the disk image (dmg) to your SD card

You’ll first need to unmount your SD card. Do not click the eject button in finder, but run this command, replacing 4 with whatever number you identified as your sd card sudo diskutil unmountDisk /dev/disk4.

Then to copy the image, run the following command:

sudo gdd of=/dev/rdisk4 if=sd_backup.dmg status=progress bs=16M

Tip: you can experiment with different numbers for the block size by replacing bs=16M with larger or smaller numbers to see if it makes a difference to the speed.

You should see some progress feedback telling you the transfer speed. If you’d like to experiment with different block sizes, just type ctrl + c to cancel the command, then you can run it again.

Once the command has finished running, your SD card should be an exact copy of the disk image you specified.

Categories
Home Automation Software

Control your Avocent PDU from python

I got my hands on an Avocent PDU (Model: PM3012V). This thing is pretty cool, it has 20 outlets on it and each one can be remotely switched on or off via it’s control interface. Just plug it into a spare network port and you’d think you’ve got a 20 channel home automation relay bank, well not quite. There is not proper API for this thing meaning you can’t setup your voice assistant (google home etc) or automation software to easily control it. That’s where I come in!

I spent an evening with burp suite, firefox and the awful Avocent web interface. I went through every single network request to and from the PDU the whole way from logging in to commanding an outlet to switch on or off. I replicated these requests in python and culled all the unnecessary ones. End result is the  avocentpdu  module. (super original name right?)

You can check out more information and the documentation on my GitHub repository right here. I’ll post an update once I’ve finished writing my custom home assistant component.