> In Linux, Docker uses kernel features. It does not use any hypervisor.
> In Windows, Docker desktop uses Hyper-V.
> In Mac, Docker desktop uses Apple Hypervisor.framework.
Cool, thanks for sharing. It's these kind of experiences that I was hoping to gather from making this post.
Did you notice at the end that you did NOT end up choosing ARM? You ended up going with x86_64 because that's what made more sense for your backend. That's part of my point - developers should choose their backend architecture based on the performance and pricing of their backend, not their development laptop. And if that decision is "we should keep using x86", then there will be a big performance hit in development.
Back in the UNIX glory days, I was responsible for keeping a software stack running across Windows NT (later 2000), Aix, HP-UX, Solaris, each with its own CPU architecture.
Thanks, I didn't know about c6g, m6g, r6g. I've updated the post to remove this mention - I only counted a1 instances.
That still leaves storage optimized and GPU optimized instances missing. I'm guessing storage should be easy enough to add, but what about GPU?
From my novice experience with GPUs, they need finicky drivers that must be ported by the GPU manufacturers, so I figure it might take a while to get competitive ARM GPU instances.
> 1. If emulating aarch64 (arm64) on x86_64 is 6x slower (on your system, btw, it's not an universal constant), it doesn't mean emulating x86_64 on aarch64 will be 6x slower. It'd probably be worse, or at least that's my gut feeling.
Yup, performance benchmarks are inherently flawed and nobody knows anything right now without the hardware. However if ARM -> x86 emulation is anything like x86 -> ARM emulation, I would expect a really big performance loss.
> 2. Generic container images like the Ubuntu mentioned usually have aarch64 (arm64) support, so running the x86_64 image makes no sense for the presented use-case.
Ah actually I address this in the article, and even run an arm64 image. The short version is, it would be a lot of work to convert your whole backend infrastructure to ARM just because you got a new laptop.
> 3. You won't be able to use most software because they don't release ARM binaries ... and the example uses `wget` && `tar xf`, with no binary signature check. As someone who has been porting stuff from x86_64 to aarch64 for a couple of years, I admit I've seen this pattern frequently. The most obvious solution is to build from sources, which would have been better off on x86_64 too, instead of fetching a prebuilt (and unverified) binary from the internet. Maybe there are some CPU flags the compiler could notice and apply optimizations which are not included in the prebuilt binary.
Yes, if only everything were built from source! I'm not saying there's no solution, just that the solution would be a lot of work. If the library is obscure enough and the errors are strange enough, it might be so much work as to be impossible to the busy web developer.
My goal was to write a kind of hand-wavy article to get people talking about this problem.
I agree on the performance loss. Just for kicks, I ran the same commands on some real aarch64 (32 cores, 3.0GHz, ARMv8.? - can't remember and already logged off the machine, but I can double check tomorrow).
Without further context, numbers:
someuser@some-aarch64-machine:~$ docker run arm64v8/ubuntu bash -c 'dd if=/dev/urandom bs=4k count=10k | gzip > /dev/null'
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 2.18298 s, 19.2 MB/s
someuser@some-aarch64-machine:~$ docker run amd64/ubuntu bash -c 'dd if=/dev/urandom bs=4k count=10k | gzip > /dev/null'
warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 6.72324 s, 6.2 MB/s
A 3x slowdown is not as bad as 6x, but it's still quite a bit. I also saw a slowdown of ~4x when I tried this experiment on a native Linux x86_64 running ARM - perhaps the Mac -> Linux virtualization slowed it down further.
5x may have been a bit alarmist, but regardless we should brace ourselves for a big performance hit on x86_64 virtualization.
I'm surprised it's only a 3x slowdown.
But the single-thread performance of native execution (without emulation) is worse on aarch64, which was expected.
Imo, a better benchmark would take into account the multithread performance with/without emulation.
I'm glad somebody said something! Yes the gzip perf test is pretty silly, but illustrates a significant difference. /dev/urandom throughput on this setup was about 100 MB / s so it wasn't a bottleneck for this test - the bottlneck was gzip.
Feel free to come up with a performance test yourself! I personally want to know what an HTTP test would look like. You can run an ARM image by running:
docker run -it arm64v8/ubuntu
Unfortunately, Rosetta 2 is not going to help here. Rosetta 2 translates x86 -> ARM, but only on Mac binaries. It does not translate Linux binaries, and cannot reach inside a Docker image.
Was your emulation done with qemu user space emulator[1] (the syscall translation layer) or qemu system emulator[2] (the VM)? If it was qemu-system you might have better numbers with qemu-user-static, which does binary translation similar to Rosetta 2 rather than a being a full system emulator with all its overhead.
You can probably use qemu-user-static to translate x86-64-only binaries in a Linux container on an ARM machine, too, but I have never tried.
That's strange, in my experience it shouldn't have 6x slowdown. Probably might be due to several factors, but here's your test, running on my system without Docker:
Ryzen 3900X (host machine)
$ dd if=/dev/urandom bs=4k count=10k | gzip >/dev/null
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 1.02284 s, 41.0 MB/s
qemu-aarch64-static
$ dd if=/dev/urandom bs=4k count=10k | proot -R /tmp/aarch64-alpine -q qemu-aarch64-static sh -c 'gzip >/dev/null'
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 3.33964 s, 12.6 MB/s
> Emulators can run a different architecture between the host and the guest, but simulate the guest operating system at about 5x-10x slowdown.
I think this is a misleading statement because it implies that there is a constant performance overhead associated with CPU emulation. In reality, the performance relies heavily on the workload, more so with JIT-ed emulators.
Regarding this specific benchmark, I think there are two main factors contributing to the poor performance. The first factor is that the benchmark completes in a short period of time. With JITs, performance tends to improve for long running processes because JITs can cache translation results allowing you to amortize the translation overhead. Another factor is that your benchmark is especially heavy on I/O, meaning that it spends a lot of time translating syscalls instead of running native instructions.
I'd also like to add that CPU emulators sans syscall translation should work for any binaries, even those targeted for Linux. It would require a copy of the Linux kernel, but Docker won't work without it anyways.
So I'm not familiar with how Darwin does things, but on most FOSS unixes it's easy to use qemu to run one arch on another, either full system or just user mode emulation (which when wired up correctly lets you seamlessly execute ex. ARM binaries on an x86 system). I would expect it to be easy enough to either set up user mode translation, or just swap Docker's backing hypervisor with an x86 VM. Or, worst case, just run qemu-system-x86_64 on your ARM Mac, run Linux inside that VM, and run Docker on that Linux; SSH in and it should be mostly transparent.
One benchmark would be to track down a python/JS/etc based "hello world" demo container. Base one version on Intel and the other on ARM, and measure each versions container build-time and request latency after it is set-up.
If changing the base image is all that's needed and both Dockerfiles otherwise assume ubuntu, this should not take too long.
Author here. That's a major point of the article - "are we screwed?"
I'm not an expert on virtualization but I wanted to see some discussion on this topic, because it feels like we might be screwed and nobody is talking about.
Anyway I was happy to see Docker worked, at least on a basic level.
Cool. Thanks for writing it. It summarizes and collects a lot of issues we were all grumbling about here and there. The main hurdles for Docker are organization, not technical. However, the other issues you bring up are going to be more technical (same as you, though, not a hypervisor expert and/or we're going to be at the mercy of big vendors like Apple, Oracle, and Microsoft. Those are much harder problems to overcome.
The speaker mentions that Redis, MongoDB and background jobs were replaced by Erlang.
What does he mean by that exactly? Erlang provides some persistent state storage? Or is he just saying he used Erlang database drivers to access Redis / MongoDB?
You can use ETS and Mnesia, and overall it is built to be distributed so you can pass messages between processes/nodes without needing something like RabbitMQ.
Most of the time Erlang and OTP provide what you need already without having to reach for an external tool. (Obviously depending on your use case)
Furthermore, job processing is much easier with Erlang because the processes and the scheduling mechanism - there could be no job, only processes doing their things.
It sounds like you're skeptical about companies that affect people's habits. So avoid these terms:
- finance
- advertising
- entertainment
I'd focus on companies that sell to other businesses. Changing the habits of businesses is usually considered a good thing.
And sell something less abstract. If you sell laptops, routers or office chairs, nobody's going to think you're a bad person. Just maybe a bit boring.
Software-only products are all about information. It's about gathering information about people (gray area) or understanding lots of information about people (gray area). So work for a company that does something in the physical world.
> skeptical about companies that affect people's habits
I would rephrase as companies whose express intent is to manipulate people's habits for their own gain. Entertainment is fine as long as it's done in relative good-faith (not gambling or manipulative microtransactions). Though the game dev sphere isn't a great job market right now for unrelated reasons.
> It's about gathering information about people (gray area) or understanding lots of information about people (gray area).
That's a very narrow lens. Software can also be about gathering and understanding information about the non-human world, for scientific reasons or otherwise. It can be about empowering people to manage and utilize their own information. It can be about empowering people to create new information. The key question is whether or not the intent is to make someone's life worse in order to make money.
> I would rephrase as companies whose express intent is to manipulate people's habits for their own gain. Entertainment is fine as long as it's done in relative good-faith (not gambling or manipulative microtransactions).
Wondering - is Netflix a good company? On one hand, they're a straightforward entertainment company, but on the other hand they optimize content and suggestions on maximizing viewing hours. Their goal is to change habits to watch more and more Netflix.
How about Apple? They sell hardware and software that people like and they try to respect user privacy. But they design their software to aggressively lock users into their own platform and prevent them from trying competitors.
> It can be about empowering people to manage and utilize their own information. It can be about empowering people to create new information.
This is almost exactly Google's mission statement, but most people here would put Google in the gray area.
My point is, almost no company's goal is to make peoples' lives worse. But all companies are fundamentally trying to manipulate people to buy more of their product. The question is if most customers feel good about the company after the transaction, if they feel like they were manipulated "too much".
Netflix is a good example, because it can be contrasted with Hulu. Netflix uses aggressive dark patterns like auto-playing trailers to push content on you; people have repeatedly asked for the option to turn these off and Netflix refuses, because it gets people to watch more. They've made the express choice to rob their users of agency for the sake of increasing engagement. Hulu, on the other hand... doesn't do that. I'm not saying they're saints, but their interface respects user-agency.
