A Day in the Life of a Sys Admin: Horror Stories and Saving the Day – Cloud Blog – VMware

The world of the sys admin is full of constant challenges and new things to learn. From systems engineers and senior technicians to security strategists and consultants, you’ve told us your horror stories from years of looking for lost data, plugging devices back in, and unwittingly being in charge of the heating and cooling systems. Here are our favorite stories.

1. When IT work causes physical exhaustion

In the good old days, we had a physical server. The server’s on/off button was very sensitive…one day, a colleague had to hold the button for 30 minutes as they’d pressed the wrong server button, and we had to get everyone off the system before he could release it.

2. Who’s in charge of temperature control, anyway?

The fan went out on the NAS system we used for backup. We only ran weekly backups and as luck would have it, the server clapped out that week. When we went to start the restore, we noticed the NAS was down and the inside of the unit was cooked and useless. Some lessons are best learned the hard way…

3. Is floor cleaning part of the job description?

We installed a new AC unit, then had rain leaking from the roof directly into the server room.  I have pictures with plastic sheets and buckets diverting the leak. At least we got a good cleaning under the raised floor.

4. Just hope your boss doesn’t find the problem before you do

We inadvertently mixed test users with real ones during an emergency upgrade of one of the Exchange servers 10 years ago. I got a call in the middle of the night that a boss of the boss of my boss couldn’t send an email.

5. The simplest tasks sometimes hold the most peril

There was the time I added that ESXi host without changing the time. The first machine moved to it was a domain controller, which then got its time set to six months in the future. Luckily, I saw it not too long later, but not before a good part of the company was down for 20 minutes.

6. Just one key stroke can cause a world of havoc

I accidentally deleted an organizational unit (OU) with all the call center PCs in it, before Active Directory had the recycle bin option, and then had to rejoin them all before the next shift started.

7. Do we really need backups?

I was on a call, responding to an incident, and asked if they had backup in place. The answer was yes, and it turned out the backup had been successful for the last 31 days. So we looked at the logs. They said backup had been completed within five seconds. The problem? Someone had excluded the main data from the backup job.

8. When hardware isn’t dependable

I once had to pull out an ESX host (a 4U HP DL580) to replace a bad memory DIMM.  I slid out the server and the disk array mounted above it wasn’t actually mounted and fell!  Thankfully, the ESX host was at the bottom of the rack and the array didn’t fall far. I managed to lift it back onto its rack arms, while the array stayed online. The array in question served our primary file server!

A special thanks to the Sys Admins in the Champions community for their contributions to this article. Champions is a vibrant community where customers engage with peers, speak to experts and access exclusive content. Along the way they earn points and redeem rewards, grow their professional network and share feedback about their VMware experiences.

Tell us your stories on Twitter!