When things stop working, the solution isn’t always obvious
When we develop web sites, we stumble across problems, right? And sometimes, just sometimes, we do learn from them and know what to do next time. At other times, though, the problem is something we never would have expected.
I thought I’d share two of these situations where the problem was far from what everyone thought at first.
Incomplete files
About a year ago, working on a web site where we Gzipped the JavaScript files in production, one day I got the message that the interactivity was all messed up on the live web site. I started sifting through the code, and after some initial tests, the problem was that the whole content of the JavaScript files wasn’t delivered (for all web browsers, not just the notorious IE Gzip bug). For some reason, they were only parsed about halfway through, and then stopped.
The guy responsible for the servers did some intense research (another problem factor was that the servers were stationed abroad). The conclusion he came to was that the heat and hot exhaust air coming from other servers in the rack. This issue was addressed, and as far as I know the problem was gone.
This has never been scientifically verified, though no matter what was the actual cause, it was from any of the the usual suspects.
Cleaning
The architect in my current project told me about when he was working at the major airport in Stockholm, Arlanda. One of their systems went down for a while, at the exact same time each morning. It automatically restarted, but that wasn’t sufficient, so someone had to manually restart it again.
They started thinking that it had to be an outside factor causing this problem every day, so they went to the server room at the time of the usual server outage. Lo and behold, when the clock strikes, a cleaning woman walks in, pulls out the cord for the server, plugs her vacuum cleaner in and starts vacuuming. Once done, she put the server cord back in the outlet again and left. 🙂
The moral
Sometimes, things are more FUBAR than you can even begin to fathom. 🙂
Hahaha priceless
Loved the stories!
haha… love the second one mate.
The cursed bed, where every patient carried in in the night, found dead in the morning – I have heard exactly the same story, but it was not a server room, but an intensive care unit in the USA…
In software engineering one of the first stories told involves a true story of an american guy who brought a new car. After a month he began complaining that every time he brought vanilla ice cream his car wouldnt start…
Eventually they sent out an engineer who stayed for a week.
The family ritual was that every day after dinner the family voted on the flavour of today's ice cream for dessert. The dad would then get in the car and get the ice cream and drive straight home. But every time it was vanilla the car would not start.
No it wasn't the flavour.
The engineer eventually realised the vanilla ice cream was near the front door. It took about 1 minute longer to buy other flavours. The car actually had a fault which meant that in a certain timeframe the car wouldn't start.
So I guess the lesson is that the cause of problems may not be directly apparent, as you say. Ice cream doesn't stall new cars 🙂
I actually heard that same story (the cleaner with the vacuum cleaner) but it was in ICU at a hospital … but instead of servers going down, people who were on life support machines were mysteriously dying! 🙁
Hehe top notch, specially the cleaning lady.
Thanks, guys!
Ceriak, Dave,
I have heard variations on this theme before too. Not unlikely that it does happen in some places, though. In this case I can only retell the story, and I do hope it's true, because it's cute and funny. 🙂
Steven,
Yes, I've heard that story before, and it is indeed a great one! Sometimes unusual measure helps us finding the true problems.
its very often hard to track down errors because our human mind initially sees the obvious cause as the reason… actually getting into the proper open mind to accept that which is not obvious, and sometimes illogical or even preposterous, is a true skill of the problem solver.
We had the same power outage every morning problem at work and it took us a while finding the why as well…
Another company I worked at had another issue. Terrible scratches appeared mysteriously on our monitors… The cleaning people scoured them with scotch brite. Ugh!
Goulven,
Scotch brite?! Oh, man, that's just terrible!