The holy grail of high-availability isn't upgrading stateless software. It's upgrading stateful ones.
Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.
In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like
handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State);
handle_call(Request, From, State) when is_record(State, state2) -> ...
(you'll want to do something similar on handle_cast and handle_info if you use those). You have to do a little work, but I don't see how you avoid that?
Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency.
> Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency.
If we're talking about a gen_server, the state is per process, and once the process has switched to the new code, it won't go back, so there's no problem with old code and new state. In non gen_server code, you do need to be careful about when you hit a boundary that gets you into new code; you'd typically want it to be your process's main loop, since that usually tail recurses and doesn't leave a stack in the old code. It is difficult to reason about a situation where you call into new code, and that returns to old code; it's much better to avoid it.
The concurrent case is OK too, each process manages its own state, and upgrades it when it switches to new code. Are you thinking about changes to messages that are being passed and/or global state? In that case, like with any distributed system, you need to load in stages: first load code that can handle old and new messages, then trigger sending new messages (code load or config setting), then load code that only handles new messages.
You touched on most points, primarily avoiding the situation of reasoning about concurrent old and new code. I didn't know a gen_server manages code loading like this, thank you for that.
An upgrade doesn't involve only the in-memory state per process though. It also involves state outside the process, like state on disk. Even if each process upgrades it's own state (I'm assuming the gen_server isn't limited to in-memory state; I don't know), an old process accessing from disk a data structure that differs from the one used by the new process isn't safe. You can't just upgrade old processes in stages.
An upgrade can also involve multiple processes. It's hard to upgrade all of them at once. As you mentioned, in the hardest case of all, a distributed system, loading in stages may be the only option, provided the system was explicitly designed such that old and new processes can coexist without safety issues.
Synchronize to a checkpoint. Serialize all state. Run data structure upgrading code. Deserialize upgraded state.
Of course possible to do in-place, but I'd imagine serialization makes testing it easier. Not to mention sending error reports if something goes wrong. Having all that state in a bug report could help a bit!
Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.