One: We’re F****D, It’s Over: Coming Back from the Brink
Two: How I Fired Myself.
These paragraphs stood out:
I remember Hotmail’s CEO Sabeer calling us all in a room. “Hey, Rex (the COO), how long will it take to restore the email from the tape backups?” And I’ll never forget his answer: “Um, those got really expensive, so we stopped doing them about a month ago.” Gulp. Long … painful … silence.
I found myself on the phone to Rackspace, leaning on a desk for support, listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago. Ah.
God, that's painful to read. It gave me chills... Note to self: Backup all the thing. Never cancel backup services. And test backups to make sure they work!