> I've never heard of this happening in any Linux distribution. Can you be more specific?

It was a couple of years ago. I know somewhere I have notes on it, but I can't find them just this minute. I remember that it had something to do with one of the components in my apache-mpm-worker--php5--libapache2-mod-fcgid--apache2-suexec-custom--libapache-mod-security stack. It was something like, mpm-worker no longer supported libapache2-mod-fcgid or some such thing. At the time, I was really good about doing regular server updates, so when it happened, I spent some time researching it but eventually found the package had been removed during the update and was no longer supported, with no workaround aside from finding a new way to build an apache server.

Either it got fixed or I'm running a slightly different stack now than I was at the time. Sorry I can't be more helpful.

> This is because of a shortcut that somebody took...

> You might want to look into "DevOps"...

That somebody was me, and I'm aware of devops. Pretty big fan of it actually. Unfortunately, I'm just a small MSP, the owner and the senior tech and the sole software developer and the sysadmin, and I don't charge enough. The servers exist as an add-on service for my clients, especially ones that have special needs that other hosting companies can't easily meet. They work, the stacks I built a few years ago are robust and efficient (one of my customers was featured on a popular national radio show with a reputation for killing websites, but their site stayed up and responsive the whole time). Reproducible deployments, centralized management, further improvements to automated security, etc. are all on my to-do list -- along with like a couple dozen other things.

I was supposed to rebuild all of the servers during last December, typically a slow period, but instead it was unseasonably, hair-on-fire, brain-meltingly busy, and it still hasn't let up.

> It was something like, mpm-worker no longer supported libapache2-mod-fcgid or some such thing.

No stable release distribution ever does this within a release, unless there is a security issue that cannot be fixed any other way. In this case would you prefer to have remained vulnerable?

Of course, mistakes can happen, but they would be fixed in a further regression update, or you could have even looked at fixing the bug yourself.

Based on the package names you're likely talking about Debian or Ubuntu. In both cases, you could have just downgraded the packages for a quick (albeit temporary) end to your emergency.

> I've never heard of this happening in any Linux distribution. Can you be more specific?

Not the OP but I run a personal server for mail etc., and I do remember one LTS → LTS upgrade of Ubuntu that removed the DKIM milter that my postfix config depended on.

So it happens.

> You might want to look into "DevOps". The idea is that you script your deployments together with automated tests for them

For a personal server the things that in my experience break are related to configuration files.

This can be subtle things, like some new option enabled by default that conflicts in some edge case, it can be upgrade scripts that butcher existing customized configuration files, or it can be a total restructure of how a package has structured its config.

All of this has happened to me, and I don’t think having scripts able to redeploy my server would have been of any help resolving these issues.

> ...and I do remember one LTS → LTS upgrade of Ubuntu...

That's different. Supported features do change between distribution releases. But you don't ever need to do an emergency security update between releases. You have years to plan for a release upgrade. The grandparent was referring to unexpected emeregency breakages during updates within a release, which is an entirely separate thing.

For people with >2 servers, that might be the right solution, but for the others, scripting deployment might be just more overhead.

I can see that, especially if someone is administrating their own small site, and hasn't had experience in larger shops. But at the minimum (and since it is cheap enough) you should have at least 3 - 5 servers (esp. if you are making money off them) -- dev, test, staging, production, and failover. Just breaking the mirror to the failover box, and upgrading the primary would allow for an easy and quick backout procedure.

Oh, and there is another upgrade trick (I'll have to see if I still have my old writeup on it, and post it as a Gist). You can query the package manager, and make a few lists. First, list the packages (and versions) that are installed. Secondly, get a list of any file owned by a package that has changed since package installation (compare the file MD5 sum to the package's record of same). This should be a small list of files (mostly configuration related) that can be backed up. This gives you a good way to roll back changes if needed, and keep a system documented.

Finally, with Yum, you can roll back updates. Take a look at "yum history", and "yum history undo". This has saved me a couple times.

I must acknowledge, that I don't have so much administration knowledge as you have. I come more from the development corner. And I really "hate" that stuff. I also have to say, that what you write is probable right. But I think, there are some people like me, that have one or two servers for some projects running (and I know even people, that have websites or even servers running with even less knowledge) who just want that the stuff works and don't have time or ambition to optimize things.

Do you know, if this rollback stuff is also available for Debian based systems? Sounds pretty good. I regret, I don't know yum, it is for RPM-based systems, I know, but the last systems I used where all Ubuntu and Debian.

I haven't done a lot with Debian based systems -- I've traditionally been a Slackware / DIY / Redhat person. I just looked through the documentation and a bit of the source code for apt-get, and nothing popped out at me for a rollback feature. If I come across anything I'll update this comment for you.

But I know what you mean about hating the sysadmin side of things. Of course there are people that really love it too -- there's sort of a mindset that you have to get into, just like with development. Maybe this could be an idea for a new service -- matching up programmers with side projects, with sysadmins that are looking for side projects to help manage.

Yea, that would be really a great idea. I have a friend, whom I ask for help some time, but he also has only limited time.

And I lack the time to really dive into automation stuff and that sort. So many interesting tools are available, but you always have to take time and learn it first.

> matching up programmers with side projects, with sysadmins that are looking for side projects to help manage.

That sounds like a fantastic idea. Even in my case, I do "OK" as a sysadmin most of the time, but would love some occasional guidance or assistance.

> Do you know, if this rollback stuff is also available for Debian based systems?

Over half the servers I admin are running some flavor of Debian, and have for at least five years or so. To the best of my knowledge there's no rollback method for Debian updates, short of relying on filesystem tricks like using ZFS, discussed upthread.

Thank you for your information! That would be really a fine feature. I love Debian for its stability (as it is said ... I just lack time, to compare myself) but the administration tools are still a black box for me and I have the feeling, that they could need a brush-up.