What’s the ‘freakiest’ event or situation you’ve experienced in production?
My first job at a university working in the data centre. Came across one of our water cooled super computers spewing water all over the cluster. It’s amazing how frequently water sneaks into data centers.
Having to literally build my own network to work off of because the organization’s network is locked down to un-usability for anything. I also now hate T-mobile’s CGNAT with a passion that I never thought I could feel over a networking architecture decision.
Once one of the backbone Zyxels stopped working, so we went to the floor on which it was located. We look in the rack, as expected no blinking lights.
We didn’t have a spare and I’m into electronics, so in order to keep things going, I decided to take a look at it. Open it up… jeus fucking christ 😲… a swarm of roaches rushes in all directions! I just rushed out and told the cleaning ladies to get a vacuum cleaner in there ASAP. They did and got most of them, but eventually we did have a minor roach infestation in the office and had to disinsect a few times.
After the storm was over, I took a look at it… apparently, they laid eggs underneath the PSU and when the eggs started hatching, they short circuited it. The PSU was fried. Luckily, the output voltage of the PSU was marked on the PCB - 12V. So, I just modded an ATX to power the switch. Still works like that to this day, lol 😂.