Custard10
From DotelTech
- Client: Accounting Office
- Problem: ME
- My Job: Administrator Consultant
Here is the story of me being an ITARD. (similar to custard, only being an IT professional)
Now I must say this was not COMPLETELY my fault. However I should have payed more attention, as an IT staff we must assume things are not as they seam.
I'm called out on site about an UPS that is beeping and needs replacement batteries. Simple enough you may think. Thats certainly what I thought. I had been given the model number of the UPS, and brought out the batteries with me. The manager that had made the call had left for the day, so the secretary pointed me to the UPS I was called to work on.
Attached to the UPS was the Fileserver, Email Server, Firewall, and DSL modem, the UPS was one designed to have it's batteries hot swapped. One of the partners comes up behind me and asks me how long the systems will be down. At this point I sealed my fate and uttered the following phrase.
"If I do my job correctly the server will not go down"
Yes reading that phrase NOW I realize that there is no way that this can NOT go wrong now. I sealed my fate and punched an express ticket for doom and catastrophe. But after all it's just changing a battery pack on something designed to be changed in less then 5 minutes. Bah nothing can go that wrong. I evidently missed the chapter on foreshadowing in my IT manual.
Now let me rewind 2 years when the last lazy/incompetent/retarded/cheap IT tech replaced the batteries. I of course have to imagine because he/she/they are no where to be found to throttle. They receive the battery pack, only to find it's not the correct model, The connectors don't fit, and are the wrong style. Now instead of waiting to receive the CORRECT battery, a stoke of "genius" hits the tech. Long story short it is made to work.
Back to me now. Unknowing the mess that lays behind this battery cover, I forge on. I've replaced dozens of these. Nothing special, nothing different. I remove both screws holding the battery cover in place, setting the cover to the side I pull on the battery sliding it forward............(insert the music of doom)
I get the UPS out about 2 inches and suddenly the entire UPS explodes in a shower of sparks and smoke. All the servers go silent, and the lights in the building go out. Keep in mind one of the companies partners is standing no more then 3 feet behind me. It's dark, I'm surrounded by smoke and now drawing a crowd of employees.
I pull the UPS, and reset the breakers, I still dunno why not only did the zone breaker pop, but the master as well. UPS is dead, So I wire past it to get server backup. Lucky for me, No damage to servers or attached equipment.
Upon examining the UPS I found the previous tech had attached giant washers to the battery terminals in order to WEDGE the contacts together. This causes a metal ridge that would scrape against the 110volt leads if the battery was ever removed, which they did. Placing a nice chunk of metal between two high voltage feeds is not considered a good idea.
We ate the cost for a replacement UPS, the client was a very good sport and understood how it happened. But to top it all off, the secretary also pointed me to the WRONG UPS. The UPS that needed new batteries was actually for the phone system. Took me 2 years before I was able to change batteries without de-energizing the entire system. Scary Scary Scary.