Wikipedia defines a face-palm as the physical gesture of placing one’s hand flat across one’s face or lowering one’s face into one’s hand or hands. The gesture is seen in many cultures as a display of frustration, disappointment, embarrassment, shock, surprise, or sarcasm. In no culture is the face-palm more significant than in IT. Back in August, we published an article The 31 Worst Face-Palm Moments in IT in which we shared some of the things we had seen that were just stunningly worthy of the classic face-palm move. In today’s post, we’re delighted to present the sequel, 18 More Face-Palm Moments in IT. Get those hands ready…
1. Here’s one that Exchange admins will love. I had a customer whose email address policy was firstname.lastname@example.org, %email@example.com. IT would create a new user, and only populate the first and last name. After the account provisioning was complete and the mailbox created, they would go back and manually edit the email address for each user to include the middle initial… rather than just adding the letter into the AD account when they created it.
2. Another customer, operating in strange and remote parts of the world, still wanted to provide full domain services to all their users. With a domain controller in each location connected back to HQ by satellite, they would start them up each day when their region was provided their few hours of power, let them run until the lights went out, and then repeat the process, again and again. They could never figure out why AD replication was reporting that it could never complete.
3. I recently saw a user email a 20 MB file to a distribution list of 7,000 members.
4. Then there was the customer who demanded that we disable Exchange ActiveSync access to their systems, because it did not meet their security policy. We spent a week explaining how secure EAS was, how EAS policies could implement every restriction that they said they needed… all to no avail. They demanded we turned it off. So we did. Then the CEO couldn’t get his email on his iPhone. So you know what we did next, right?
5. Another customer I worked with thought they had all the datacenter redundancy they could possibly need. Two datacenters, redundant Internet connections into each… what could go possibly go wrong? Well, both datacenters were within a mile of one another, and the “redundant” Internet connections’ last mile, of course, terminated in the same POP.
6. This one happened to me, and it was my own fault. I knew the UPS was going bad and needed replacing. I ordered the replacement, racked it, and connected it. I was planning to connect all the servers and gear on the coming weekend. Of course, the thunderstorm that took out the power decided to come on Thursday.
7. I once consulted with a company where all the IT team had to use Macs so that they could support the executives. All of the IT team were Windows users and all of the core systems were Windows-based. Parallels, VDI, and creative solutions for RDP were the rule of the day in IT, as every one of them had a Mac, but every one of them used it as a very expensive remote terminal or virtual machine host.
8. I once saw a deployment engineer from a company show up at corporate headquarters to do a multi-day install of a core system. The problem was, he was dispatched to the billing address of the customer, not the datacenter address, which was on the other side of the country.
9. I myself travelled to a remote datacenter to deploy servers, only to get there and find out that someone ordered the wrong power cables for the servers. Fortunately for me, this customer was in Florida, so I got to spend a day at Disney waiting for the right parts to arrive.
10. At a company I worked with a few years ago, a web developer turned on anonymous FTP because he wanted to transfer a large amount of data from his workstation to the production webserver. Unfortunately for him, FTP was open to the outside world for what was intended to be anonymous download only. The server did eventually run out of space from all the illegal content that was uploaded to it.
11. Rather than handling it with policy or software, a company decided to epoxy all the USB ports closed so that no one could plug a portable drive in. Fast forward two years. When they went to turn all the leased equipment back in, they found out that the vendor considered the hardware destroyed, and billed them for the full residual on all of them.
12. An Internet registrar once had to reset all user passwords for their domains, and reset the initial password to the name of the domain. They then sent an email to all their customers, informing them each individually of the change and their initial password. Of course, if you own example.com and are told your password is “example” how hard would it be for you to guess someone else’s password?
13. If you’ve ever worked with plumbing, you know that pipes can burst, connectors can leak, and water and electricity don’t mix. A frozen pipe burst in the office above ours one time, and the low point in their floor was right above our computer room. By the time building security realized there was a problem, there was already two inches of standing water in our computer room. Building engineering cut power to the floor, but of course, all our UPS systems kept everything going and they didn’t know about the EPO switch. By the time they called us, all the servers on the bottom shelves of the racks were standing in water. Amazingly though, they never lost power or shorted out, and eventually we were able to gracefully shut them down until the flood waters receded.
14. Back in the early days of Active Directory, I was hired to do a domain migration for a company of 10K users. They wanted to stand up an entirely new forest, get all of the DCs built and ready to go, and then do a “big bang” style migration over the course of a weekend. Basically they needed to do this as quickly and completely as possible, sparing no expense, do whatever you need to in order to get it done. No one wanted to explain the urgency to me, but eventually it came out that the new CEO didn’t like the name of the domain, and ordered it changed immediately.
15. Speaking of excess, another customer of mine’s arm was twisted by a subsidiary. The parent wanted to spin off the subsidiary in advance of selling them off. The sub was willing to commit their staff to the project, but only if the parent bought them all new hardware so they could migrate to a pristine environment at no cost to themselves. It was that, or they would just “clone the forest” and cut the network connectivity. This was definitely an example of the child telling the parent what to do.
16. Title this one “How Not to Secure Your Lab.” I worked with another guy one time who built up a lab environment in AWS. Treating it like a lab, he set a pretty lame password, something like p@ssw0rd, on the admin accounts. Being on the Internet, this proved to be a very bad idea, as he lost access to everything in just about an hour! Open RDP + lame password = not yours anymore.
17. A past employer near the Florida coast had a Unix admin whose skills were considered so mission critical, they paid him an outrageous fortune, and flew him and his family inland at company expense every time a hurricane threatened, just in case. He absolutely abused his position and power for over a year, getting the best office, newest hardware, special treatment, etc. until he just quit without notice one day. The college intern we had, that was making ¼ of what this guy was making, took over the duties without skipping a beat.
18. And the best of the worst… I’m currently consulting on a project where the status summary call from 900 to 1000, the team standup from 1100 to 1130, the leadership brief from 1300 to 1400, the engineering work session from 1400 to 1600, and the end of day wrap call from 1700 to 1730… All Daily. So in an eight-hour day, we literally spend more than half our time in meetings. And they wonder why the project is behind schedule.
So with another 18 cringe worthy tales from IT, tell us your best of your worst. Share some of your own horror stories that just made you need to conceal the larger part of your visage with a closed hand, shake your head gently from side to side, and count to 100.