The downside of preventive maintenance
Phil Pilgrim (PhiPi)
Posts: 23,514
You may think I should've titled this, "The downside of uptime." 
But here's an article that dovetails nicely with my approach to system upgrades, although it's not necessarily about that:
I've found that my own systems -- especially Windows -- are more reliable if I don't mess with them and that "upgrades" are what cause my biggest problems and greatest instability. Consequently, I don't do system upgrades anymore, and -- obviously -- have automatic upgrades disabled. I even regret upgrading WinXP SP1 to SP3 because that computer runs much more slowly now. My Linux box uses a version of Ubuntu that is no longer maintained, but what's there Just Works.
I think the recent forum upgrade also illustrates this point. Some things that worked well before are still broken since the upgrade.
Does this mean that I advocate no change whatsoever? No, of course not. But change, when it happens, should come in big chunks, not incremental pieces. Eventually, I will build a Linux box to replace the old Ubuntu system, probably with the latest version of Linux Mint. And, who knows? A Windows 8 system may also be on the horizon!
-Phil
But here's an article that dovetails nicely with my approach to system upgrades, although it's not necessarily about that:I've found that my own systems -- especially Windows -- are more reliable if I don't mess with them and that "upgrades" are what cause my biggest problems and greatest instability. Consequently, I don't do system upgrades anymore, and -- obviously -- have automatic upgrades disabled. I even regret upgrading WinXP SP1 to SP3 because that computer runs much more slowly now. My Linux box uses a version of Ubuntu that is no longer maintained, but what's there Just Works.
I think the recent forum upgrade also illustrates this point. Some things that worked well before are still broken since the upgrade.
Does this mean that I advocate no change whatsoever? No, of course not. But change, when it happens, should come in big chunks, not incremental pieces. Eventually, I will build a Linux box to replace the old Ubuntu system, probably with the latest version of Linux Mint. And, who knows? A Windows 8 system may also be on the horizon!
-Phil
Comments
When the client side software update is triggered by some OS update, or library update, maybe server side update, or outside data or data transfer being on a different version, it all kind of breaks down. (which is a lot of the time) Amazon went though a similar thing during their growth period. Bezos decided that NOTHING would happen at Amazon that didn't happen through standard, defined, interfaces. He basically said to anyone who didn't document and use and publish their systems interfaces, "you are fired". That's what it took, because the amount of work was too high to make sense otherwise.
They went through hell, but now they've got something few of their competitors have. That thing is well compartmentalized systems, that do not require interlinked updates and all that ugly testing / debugging that comes along for the ride. Where systems there do not need to change, they probably just sit there now, like Phil describes, operating over standard protocols, on whatever OS made sense at the time. Amazon can now make very significant changes and add new product functions / services to the list with a impact foot print most companies would dream of.
We will see more of this as companies try to scale and IT departments deal with increasingly lean budgets and thicker requirements. It isn't possible today to meet thick requirements without customization, and the moment one does that, the update / test treadmill starts to grind... That costs a lot of $$$ I remember the first sendmail server we built on BSD. Actually, it was IRIX, running on a nice Indy sitting on a shelf. The second one was BSD. Anyway, once configured to do it's thing as a front end to the very painful exchange, we quite literally forgot about it, along with a few other basic IT tools. Some had uptimes in years, and it was always sad to turn them off, because we mostly did to replace them entirely.
Integration with Office, shared calendars, OS updates, mail server updates, client updates, etc... basically rendered mail too expensive to host in house, so I've put it all on a server somewhere, keeping that stuff to a minimum. Enjoyed running mine, but interlinked requirements forced an amount of IT work that wasn't practical given our size. I do have one mail server on a "slurp feed", where it gets CC'ed on all the mail. Log onto that one and fetch anything they've sent for years. It's a closed can, Linux, web browser interface, no updates, just storage and annual hardware check. (it's for legal)
'
Just say NO! to automatic up-dates.
'
I think we all felt the brunt of SP-3...I was so unhappy with an older machine I had back in those days,I reformatted the hard drive and reloaded the original XP-OS from the recovery disk "pre-service pack"
Some issues really need to be addressed, and just ignoring them isn't a wise policy. This is mostly in the area of security holes. Feature upgrades beyond that are (or should be) optional. Not that security patches aren't optional, because they are. If you want to run your own system that way, that's one thing. But I wouldn't want third-party systems to have gaping holes that allow personal information to be compromised.
Now, it that little bit of smug hubris isn't leaving me open to the ravages of bad karma, I don't know what is!
-Phil
MSE is pretty good stuff. The other companies can't make the kinds of connections Microsoft can. On this one issue, I strongly agree with their integrated approach. Very high value add today, particularly given where Windows is at on security overall. They will embed it deeply in Windows 8, and the others will cry foul, and it's going to be a crappy fight. In the end though, that's where security / detection type software needs to be.
On the other hand, my Ubuntu Linux V10.4 with updates still runs nicely on a dual boot with my XP SP3 that is running at a snails pace and a mother board that has only 512Mbyte of Dram (share with video).
Linux has long supported legacy hardware whereas commercial systems seem to be pushing new hardware. And my impression of Linux updates is that they are better prepared and more focused on the purposes of security or better operation. I have no problem with continued automatic update of Ubuntu Linux.
There are core differences that create some of the problems. Ext3, the Linux file system does not require defragmentation as it managed within the file system, Linux doesn't have a cumbersome registry that seems to grow clutter and need clean up, and there is no need for the voodoo of 3rd party anti-virus support. In other words, having built Linux on the Unix model - it seems to just run fast and cleaner of its own accord. And enhancements really have delivered. Ext3 seems to be better than NTFS for global searches of files and general operation. I suspect that MS has been stuck with a less than ideal file system due to license barriers.
The real comparison should be Apple versus Linux as both are Unix based and have greater similarities. But I suspect you will always find trends of commercial bloat regardless. And a commercial OS will always attempt to force users to subscribe to 'improvements' as that is where the cash flow it.
In other words, there really are two-worlds - commercial OSes and public-domain OSes. I don't think you can lump them all together as far as systems management. Nonetheless, if an installation of Linux is quite stable and there are sufficient barriers to exploits from the outside world, there is really no reason to bother with any automatic updates. Many Linux office servers operate in this mode for a decade or more without failure - and even then the failure is usually hardware dying of old age.
I have to work info-sec in a university and I loathe the vendor supplied 'magic-box' most of all. Oh yes, that's great. The 5 year old XP workstation that runs the whole electrical and HVAC plant that they won't pay to properly wall off and the vendor will approve patches every 420 days or so. But hey, we have a firewall so it's all good right? or " Hey, your vuln scanner running a scan on my box keeps knocking it over!!!! Stop that", ummm you know we have people being laptops on campus from home right? That might be a problem if your mission-critical server can be knocked dead by ratteling the door.
I do get it, being knocked out because MS changed a regkey, or the type of a return value is just as bad, but the idea that stuff on the inside can be soft and gooey because you built a wall around it really drives me nuts. There are some nice stories out there that show that uptime is not the only goal. You need to evaluate the risk of both the patch and the vuln in the environment.
The fact that near brain-dead people are still employed everywhere and they don't surf the web on VM's
Then after I upgrade my operating system and find it is no longer working properly, I have these people telling me I'm an idiot for having upgraded it at all:
What I have had trouble with is systems that have performed so flawlessly and for so long that when they do break, or a change is needed, I can barely find the source code much less remember how the hell they work. When I was doing a service call every month or two I would remember the back door password, the file schema, and what directories everything was in. Now I have to go hunting, sometimes in old disk drive images, for documentation I wrote in 1988.
Just say no, to preventative maintenance!
That's such a relief to know! That means I can put off replacing the ball joints on my '82 Benz diesel (258K miles), even though they squeak and groan. Thank you, sir! You've saved me several hundred dollars in unnecessary expense!
-Phil
Your article was universally true.
Bill M.
Yep.
I agree with the risk / reward evaluation. Depending on the user base, assets under threat and overall requirements, policy can vary considerably, up to and including interior policy. Our current clientele and user base doesn't warrant that, and I'm thankful for it.