As regular readers know, I’m an advocate – often an annoyingly persistent advocate – for backing up.
As I write this, it’s been kind of a rough week here at Ask Leo! world headquarters – not for me, but for some of the organizations and people that I support in my spare time.
While the issues at hand are getting resolved – and I’ll walk through some of them as a bit of an example – it all served to remind me just how fragile our digital lives can seem at times.
And that beyond backing up, one of the most important things we can control is exactly how we react when things go wrong – because sooner or later things will go wrong.
Become a Patron of Ask Leo! and go ad-free!
Five days without mail
A couple of groups that I support receive all of their email through a spam-filtering service1. That means that when you send someone in that group email, it first goes to this service where it’s analyzed. Only the non-spam (or “ham” as it’s sometimes referred to) is then delivered to the group’s actual mail server and into the recipient’s email account.
On Thursday a system administrator made a very reasonable and innocuous change to the DNS for that group’s domain. DNS is the mapping of a name – like “askleo.com” – to the actual IP address of the server that domain resides on. It also includes the information on where the email destined for accounts on that domain should be delivered. The change, as I said, was for all intents and purposes, completely justified and benign. It’s a change I could see myself making.
But of course, it wasn’t benign. It had the unanticipated side-effect of confusing the heck out of the spam filter service.
Unfortunately, we didn’t realize that this change had been made until four days after it happened. Like I said, it was expected to be a completely benign change. On Thursday I got the call: “we stopped getting email!”. For the next four days we worked on the problem without really knowing the root cause.
Only in a conversation with the system administrator on Monday did we both happen to mention Thursday – as the date of the change, and the date of the email stoppage. Not being a big believer in coincidence, it was clear we’d found our smoking gun.
But wait! There’s more!
You’d think that would be the end of it, but nope.
We got the email for that group up and running, and by Tuesday all was well.
On Wednesday I got a message from a different group that I’m also helping out (Corgi-related, this time :-) ) that mail sent to their discussion alias was suddenly bouncing.
Took a bit of investigating, but this, too, turned out to be related to the same anti-spam service. The chronology:
- Over the weekend, before we understood the root cause, the service “rebuilt” the anti-spam account for the problematic domain in the hopes that whatever was broken would get fixed.2
- An unanticipated side-effect was that two other domains associated with my billing account were “disassociated” from the domain we were working on. They became orphans. The discussion list that started bouncing was on one of those domains.
- On Wednesday, several days later, an audit process at the anti-spam service noted the orphan entries … and deleted them. Email to those domains started bouncing immediately.
Once again a phone call to the anti-spam service (they’re on speed dial by this time) and some investigative work on their part, and all is rebuilt and repaired – presumably including my bill.
Meanwhile in another corner of the world
As the events above were wrapping up I got an email from a friend for whom I am also the webmaster and occasional tech support. One day her laptop – the center of her business – wouldn’t turn on.
Now, we’re several miles and hours apart, so actually physically helping with that repair wasn’t in the cards. She did find someone local. They got the machine working, but, against all expectations, seemed unable to repair, reconstruct or otherwise reconfigure her email (Outlook) to its pre-failure condition.
She texted and emailed in a panic. Understandable, I think, because as I said – her business lives not just on her laptop, but in Outlook. Not getting that back the way it was could have some dire consequences.
Sometimes the experience that I bring is nothing more than thinking ‘oh, this looks interesting, let’s poke it.’
I was able to gain access to her Carbonite online backup and confirm that her PST files – the repository of all her Outlook information – were present and backed up right up to the day of or before the failure. One way or another we should be able restore her world.
I expect to run a remote session to her machine later using TeamViewer to see what that’ll take. (This’ll also be my first direct experience with Carbonite. I’m hoping to have good things to report.3)
And then 100 yards to the east
My neighbor had asked for some assistance with a printer. It was a wireless printer of some sort, but would only print from one of his two computers; interestingly enough the desktop and not the wireless laptop.
Printer setup can be amazingly complex and frustrating, and honestly is best dealt with in person. I let him know that I’d stop by some time. Problem was that I let him know that two weeks ago before all of the above (and a volunteer event) happened.
Finally as things calmed down I was able to have a look.
The upshot was that the printer was both wired and wireless. It was connected via USB to the computer that worked. It turned out that it wasn’t connected to his wireless network at all.
Sometimes the experience that I bring is nothing more than thinking “oh, this looks interesting, let’s poke it”. Years of experience poking things often leads me to solutions that others might not discover. The setup menu for the printer was anything but simple or user friendly. But I poked at it enough in places that looked like they might relate, and sure enough after a few minutes we’d connected the printer to the network, the laptop to the printer, and were printing a test page.
Bullet, meet foot
On top of everything else, for a portion of the time that all the problems above were occurring, mail that I was sending out was silently going nowhere.
I’d made a change that had the unanticipated side-effect of confusing Gmail (the mail service I use) as to exactly how mail should get sent from me. The confusion resulted in the kind of failure that takes 24 hours to report, and then keeps trying for 5 days until giving up completely.
I thought everyone was just avoiding my messages. In reality my messages had gone exactly nowhere – and they wouldn’t, ever.
I’d cleanly and silently shot myself in the foot.
Fortunately I had an insight as to why I might be getting ignored by the world, and discovered and fixed the problem about 24 hours after I’d created it.
Our fragile world
A common thread to this week is the unanticipated side effect of seemingly minor and inconsequential actions.
Yes, our world is fragile.
Our world is also complex.
While it might seem to some, that people such as me can navigate it with ease, the reality is that there’s a lot of tap dancing and hand-wringing going on in the background that you often don’t see. Yes, I lost sleep this week – though not because I didn’t think things wouldn’t get fixed, but rather because the service that I was providing was letting people down.
By far, the single most important thing you can do when things go wrong is simply not to panic. Of course, prepare beforehand, like my friend’s backup, but more importantly simply refuse to panic. I know that’s difficult – particularly when important data might be at stake, but honestly – panicking almost always makes things worse – often much worse.
Panicking never helps4.
And if there’s one thing you can count on in this fragile world, something will go wrong, go wrong, gowrong, gowr..asd1234123asd23e
6 comments on “How Fragile Things Can Seem”
Regarding Carbonite taking days to restore a large file… I found that it took a while to restore files when my computer died and I got a new computer. I’ve found that since then, Carbonite no longer throttle the bandwidth, so it wouldn’t take as long – but if there are specific files you want restored, then you have to specify them as wanted first. (I assume that’s still the case, although I haven’t tried restoring in a long time!)
Their help (which I won’t link to, as it’ll probably cause this to be marked as spam!) mentions a possible restore speed of up to 100GB per day, assuming a *consistent* 10Mbps, with a decent speed of computer, and very little (or no) other internet use.
ps your “very important” note in the leave-a-reply section says “because of a rise *an* comment…” instead of “because of a rise *in* comment…”
I’m not seeing anywhere near that kind of restore speed. NOTHING even close. And ironically from what I can tell it’s restoring the file I prioritized AFTER it restores something else. Oh, and it still says “preparing”, not “restoring” – though disk space is definitely disappearing and it appears temporary files are being created that correspond to the desired files. So I guess throw “horrible restore user interface and experience” into my knee-jerk conclusions so far. Not a good first impression.
Leo, job security is a great thing to have these days!
I have owned and used computers for over 25 years. They have become much easier to use and more reliable. But more and more people are relying on online services for important parts of their lives. These services can change in an instant without reason or notice.
Years ago my XP operating system crashed and I was not able to recover the email in the Microsoft Outlook program database. After that, I developed a policy of never letting anyone other than myself control my data or anything else on my computer. This includes software that needs to phone home for registration keys. I lost my investment in a paint program that was bought out by Corel.
The cloud is convenient but security breaches and questionable EULA conditions at Dropbox show the dark side of using the cloud.
I have not been totally successful implementing my policy. My critical CAD program needs to phone home to install itself. I have trusted GoDaddy with my email ever since the XP Outlook crash. Then when trying to network my Vista and Win 7 computers, I just gave up, got lazy and started using DropBox.
GoDaddy is now pushing email users to the Microsoft Cloud and I am looking for a desktop email client but none seem to be supported these days. I have found AeroFS to replace DropBox. While they do not store my files for me they do control synchronization, which having tried to set up my own system on my home network, is not a chore to be taken lightly as one can loose all data in a heartbeat. AeroFS also seems to have used the same stupid original boilerplate EULA conditions as Dropbox ….
I will always have a compromised system until I fully migrate to Linux, learn to use open source Blender for my CAD needs and become brave enough to manage my own synchronization.
Leo offers the best advice to always back up you system. I would also offer to always have a backup computer mirroring your main computer. Computers are dirt cheap commodities these days. I am still running 8 year old Lenovo ThinkPads without any problem and have plenty of backup.
Two thoughts: look into Thunderbird for a desktop email client, and Bittorrent Sync for a dropbox like syncing utility that does not have it’s own cloud storage (I provide my own).
IT support is still as much art as science. Experience tells me, “this doesn’t seem right,” and when I dig into it, I solve the problem at hand. Sometimes I rely on Google, sometimes I don’t — but all the problems get solved.
Had a good experience with Carbonite tech support while replacing a client’s home office computer.