When we develop web sites, we stumble across problems, right? And sometimes, just sometimes, we do learn from them and know what to do next time. At other times, though, the problem is something we never would have expected.
I thought I’d share two of these situations where the problem was far from what everyone thought at first.
The guy responsible for the servers did some intense research (another problem factor was that the servers were stationed abroad). The conclusion he came to was that the heat and hot exhaust air coming from other servers in the rack. This issue was addressed, and as far as I know the problem was gone.
This has never been scientifically verified, though no matter what was the actual cause, it was from any of the the usual suspects.
The architect in my current project told me about when he was working at the major airport in Stockholm, Arlanda. One of their systems went down for a while, at the exact same time each morning. It automatically restarted, but that wasn’t sufficient, so someone had to manually restart it again.
They started thinking that it had to be an outside factor causing this problem every day, so they went to the server room at the time of the usual server outage. Lo and behold, when the clock strikes, a cleaning woman walks in, pulls out the cord for the server, plugs her vacuum cleaner in and starts vacuuming. Once done, she put the server cord back in the outlet again and left.
Sometimes, things are more FUBAR than you can even begin to fathom.