“exec format error” or How Macs, M1s, Docker images, and AWS ECS/EKS conspired to waste a weekend.

This is more a note to myself and team so that we can avoid hitting this problem again in the future. One thing that we noticed during the most recent build process was that we were beginning to encounter “exec format errors” when attempting to build locally on some machines (M1s of course) and deploy those images using CDK/Docker.

exec /usr/local/bin/docker-entrypoint.sh: exec format error

It was clear that we were generating Docker images that were incompatible with ECS, but it wasn’t entirely clear why. Somehow whatever was causing our incompatible images was hiding in plain sight — but how.

To “cut to the chase”, the issue is entirely related to what Docker does on an M1 mac. When it builds an image, under the covers it is building the image to be compatible with ARM. While that’s obvious, if you’re not typically building multi-platform images — you will probably overlook it as a possible culprit. We immediately began assuming that the issue was cloud-side with ECS and that maybe we were getting an incompatible Graviton image somehow. Wasted days chasing that down.

In our typical Github process, we would have never seen this as the Github build containers are AMD64 based, but we were experimenting ‘locally’ on Intel and M1 Macs to get the devops pipeline correct before moving it to Github.

When building on a M1 mac (or any other ‘off architecture’ build platform), you need to specifically tell docker that the image that it is building is one that you want to use on an AMD64.

The Docker documentation describes how this can be accomplished https://docs.docker.com/engine/reference/builder/#from using the — platform argument with the FROM command.

So instead of putting

FROM alpine:latest

You have to specify

FROM --platform=linux/amd64 alpine:latest 

This will tell Docker to specifically build for linux/amd64 when it is generating the image and now your image that is being built on MacOS will be compatible and deployable to ECS/EKS.

Or course, the ‘correct’ answer is to just let your devops toolchain running in github/etc. perform the builds since those are linux/amd64 based containers, but who knows — you may actually need to build for an ARM or other architecture one day and now you know how to do it.