Wednesday, March 26, 2014


I read two articles today:

One: We’re F****D, It’s Over: Coming Back from the Brink
Two: How I Fired Myself.

These paragraphs stood out:

I remember Hotmail’s CEO Sabeer calling us all in a room. “Hey, Rex (the COO), how long will it take to restore the email from the tape backups?” And I’ll never forget his answer: “Um, those got really expensive, so we stopped doing them about a month ago.” Gulp. Long … painful … silence.

I found myself on the phone to Rackspace, leaning on a desk for support, listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago. Ah.

God, that's painful to read. It gave me chills... Note to self: Backup all the thing. Never cancel backup services. And test backups to make sure they work!