Fathers Day - How to pay for a weekend off

[websnail]

Despite the thread title, Fathers Day itself this year (the UK one) was actually really good fun. Being allowed a lie in, enjoying some amazing family time, etc... but in truth I get to enjoy that a lot anyway... What clinched it was the way I set out on Friday evening to go camping with the fateful decision to leave all bar the phone at home.

What could possibly go wrong?

Oh, the list turns out to be quite long

1. Web server has a coniption fit at noon and goes offline until a kind customer pings me via facebook to let me know. Smartphone and some forethought (ok luck!) saves the day and allows me to restart the errant service.
   .
   
2. Unbeknownst to me, workshop server decides the web server is getting all the attention and the main hard drive with all our files, server data, etc... eats itself. Computer techs will recognise the "Click of death" reference... Yup, that hard drive ain't gonna hunt no more.
   
   Of course I didn't find this out until Monday morning.
   .
   
3. Monday morning rolls in, and I discover the disaster/chaos/sobbing-inducing-mess [delete per preference] and check disaster recovery plans. Luckily backup server and systems are in full working order but I was understandably wary of shutting down, moving or even touching it in case I was cursed. Probably just as well. So ordered in replacement hard drives, learned about virtual disk backups and discovered there is an eye to the storm where things are calm. I set up camp there, although $deity only knows how!!
   .
   
4. Backup set for the Monday evening so I'm feeling confident except that the PC I had setup as the client, had Windoze Updates set to run at 3am which rebooted the machine and killed the backup roughly 2 hours from completion (7 hours in!).
   
   What calm?
   .
   
5. I recovered from that, resolved a few additional lessons and found a better backup solution which worked! Yay. So we got our backup system setup and moved into the workshop yesterday...
   
   ... only to find a bloomin' ants nest! ready to swarm... RIGHT where the server used to live. And no, they didn't kill the drive but I suspect they would have anyway.
   .
   
6. So, deal with those and start things up to discover slow internet connection. Spend hours debugging and troubleshooting the server move, IP address, DNS and DHCP changes only to finally narrow down the problems to the 3G/Mobile/Cellular ISP we use for our broadband. So, regionally that has gone down too!
   .
7. Luckily! We have a backup system in place that's more expensive and (for now!!) is working so we're still alive and kicking but all in all I have to question my sanity, or indeed that of anyone wishing to associate themselves with my good self for this week as it seems I'm cursed and a fridge could yet drop on you... (You're looking up nervously right now aren't you! ).

BUT, there is a ray of sunshine in all of this...

By some miracle we had the back-up server so no data was lost, we've learned a ton about virtual disk backups and found solutions so we won't (hopefully) be returning without a solution in the future...

... and (the best bit), I managed to land a i1-Pro profiling kit via eBay after quite a while considering what to get so we could provide better profiles for our inks and a new project I'm toe-dipping into.

Can't complain overall and having a son who I wouldn't trade for anything (Not even sleep!)... but sheesh, what a saga..

Laugh, while you can... that fridge is on its way

 

[stratman]

What a nightmare.

Are you using RAID? Backups are critical but a mirrored RAID may have kept things going until your return to discover the issue and make repairs.

My brother just had his own nightmare at work when his server took a dump. Over 10 years of good service. Unfortunately he suffered a failure on one RAID disk that was unknown until the mirrored disk crapped out. His new server has built in battery backup (besides an external UPS) and remote notification of an issue, which I think would include an alert to your smart phone, so you could know before a customer calls.

 

[websnail]

What a nightmare.

I had more eloquent turns of phrase to describe events but I'll go with your SFW terminology

Are you using RAID? Backups are critical but a mirrored RAID may have kept tings going until your return to discover the issue and make repairs.

Regrettably my servers don't have that option due to the VMWare compatility limitations but it's something I'm looking into now as apparently HP E200(?) RAID arrays are an option if I can find a couple at a reasonable price.

For now I've opted for redundancy in terms of a system I discovered thanks to this mess although cost there is still high so free, unscheduled is the way forward for now.

My brother just had his own nightmare at work when his server took a dump. Over 10 years of good service. Unfortunately he suffered a failure on one RAID disk that was unknown until the mirrored disk crapped out. His new server has built in battery backup (besides an external UPS) and remote notification of an issue, which I think would include an alert to your smart phone, so you could know before a customer calls.

Ouchy Ouch, Mc Ouch with sprigs of holly for good measure!

Yeah I had something similar in the last school I worked tech support for. I was putting in RAID but needed to power down the server first, so I did and discovered the Primary data drive (similar age to your brothers) wouldn't fire up anymore. It had been running on fumes and decided enough was enough when the power shutdown. Backups were non-existant despite the previous tech monkey being paid to change tapes over every week..

So, I've learned that RAID is all well and good but backups, different physical locations, solid systems and TESTING of all the above are essential. As I said, we lucked out thanks to the duplication of servers so nothing was lost completely, just two days of my life along with the associated gremlins... Fun, fun, fun!

 

[stratman]

So, I've learned that RAID is all well and good but backups, different physical locations, solid systems and TESTING of all the above are essential.

You are preaching to the choir, webbie!

Could not agree more. Glad you are back up and stronger than ever.

 

[websnail]

You are preaching to the choir, webbie!

Aye but do you have to try singing the soprano bits?

 

[stratman]

 

[The Hat]

 

[websnail]

Now there's a mental image that requires brain bleach!