When one of my clients reported that their build times had suddenly jumped from two minutes to more than twenty, I knew something unusual was going on.
They had recently switched to multi-architecture Docker builds on CircleCI, and the slowdown hit everyone — developers, pipelines, and release schedules.
At first, it wasn’t clear what caused it or when it started. But after several rounds of debugging, research, and testing, the root cause – and the fix – became clear.
Here’s the story of what happened and what I learned along the way.
Understanding Multi-Architecture Container Images
Before diving into the solution, it helps to clarify what multi-architecture (multi-arch) container images are.
A multi-arch container image is designed to run on multiple CPU architectures, typically amd64 and arm64.
There are good reasons to use them. In this client’s case:
- They wanted to deploy on AWS Graviton (ARM) instances for better performance and lower cost.
- Several developers were working on Apple Silicon (ARM) Macs.
A typical multi-arch build looks like this
docker buildx build --platform linux/amd64,linux/arm64 \ -t {AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/{REPO}:1.4.0
This uses Docker Buildx, which relies on BuildKit– unlike the older docker build, it supports cross-architecture builds.
Why The Build Times Increased So Drastically
When you build an arm64 image on an amd64 machine, BuildKit spins up a QEMU-emulated ARM runtime to execute ARM binaries during the build process.
This emulation introduces a heavy performance overhead. It might be tolerable for small Python or Go images, but when the build includes large apt install sequences, native C/C++ compilation, or Ruby workloads, the slowdown becomes severe. In my case, it was a combination of heavy apt installs and a Ruby workload, which pushed the arm64 build time off a cliff.
In this project, QEMU caused the build to run about ten times slower.
An amd64 build that normally took two minutes stretched to more than twenty minutes for the arm64 version.
How Multi-Arch Builds Actually Work
A common misconception is that a multi-arch image is a single image that automatically supports multiple architectures.
In reality, Buildx creates separate images for each architecture and then generates a manifest.
The manifest is essentially a pointer that tells Docker which image to pull based on the system’s architecture.
When a user pulls a multi-arch image, Docker first fetches the manifest, checks the local architecture, and then downloads the matching image.
This process is transparent to the user, which makes deployment easier but hides how the images are actually structured.
Inspecting the build logs made the issue clear. The amd64 build completed in about two minutes, while the arm64 build took around twenty minutes or more due to QEMU emulation.
The Investigation and the Real Fix
While reviewing the CircleCI workflow, I noticed that all builds were running on amd64 nodes.
My first instinct was to increase the node size, but resource usage was already low. The slowdown came entirely from emulation, not lack of CPU or memory.
The real solution was to split the builds: run the amd64 build on an amd64 node and the arm64 build on an arm64 node, then merge them with a manifest. Running these jobs in parallel removes the need for emulation and dramatically reduces build time.
At first glance, this might seem more expensive since it requires two nodes. However, because the builds finish much faster, the total CI duration is shorter and the cost is effectively lower, especially when using CircleCI’s own compute resources.
Implementing the Solution
Build the images separately:
# Runs on an amd64 node
docker buildx build \
--platform linux/amd64 \
-t ${ECR}/${REPO}:${TAG}-amd64 \
--push .
# Runs on an arm64 node
docker buildx build \
--platform linux/arm64 \
-t ${ECR}/${REPO}:${TAG}-arm64 \
--push .
Create and push the manifest:
docker manifest create ${ECR}/${REPO}:${TAG} \
--amend ${ECR}/${REPO}:${TAG}-amd64 \
--amend ${ECR}/${REPO}:${TAG}-arm64
docker manifest push ${ECR}/${REPO}:${TAG}
Once the manifest is pushed, you can use it in your Helm charts, ECS task definitions, or any other deployment configuration.
Conclusion
Although this example comes from my work with CircleCI, the same principle applies to any CI/CD platform.
Avoiding emulation by building each architecture on its native hardware yields significant time savings and improves reliability.
In this case, simply splitting the build process reduced build times from over twenty minutes back down to two.
If your multi-arch builds are running slower than expected, check where they are executing. In my experience, running each build on its matching architecture was all it took to make them ten times faster.
Lessons Learned
- QEMU emulation can dramatically slow down multi-architecture Docker builds when running arm64 builds on amd64 machines.
- Buildx does not produce a single “universal” image, but multiple architecture-specific images linked through a manifest.
- Splitting amd64 and arm64 builds to run on their respective native nodes eliminates emulation overhead and improves build times by up to 10x.
- Running parallel builds on native architectures can actually reduce total CI time and cost, even when using two separate nodes.
- Always inspect your build logs and node architecture before scaling resources—sometimes the problem is not hardware, but how it’s being used.
