An October ghost story from the enterprise IT trenches
Editor’s Note: This story is true. It has been fictionalized to make it more entertaining. We did not bother to change the names of the guilty – be they servers or people. Some of the technical details might be wrong. Quite probably, nobody cares. After all, it has been over 15 years.
Author’s Note: To Andras, Ted, Praashant, George, and all the other folks from the IMF: Know that your contributions to this tale are acknowledged; I didn’t solve this mystery alone. You were a good team, I miss you guys, and hope you’re all doing well.
I don’t particularly believe in ghosts. But I do believe in things that go bump in the night, especially if they’re plugged into an electrical outlet and somebody else is paying the bill.
Most IT hauntings are mundane. Odd noises are cooling fans, flickering lights are power supplies, and mysterious messages come from monitoring software whose emails everyone has filtered out of sight.
This one happened at the International Monetary Fund, involved a SharePoint farm that had been misbehaving for weeks, and took a peculiar turn late on a Friday afternoon. But I’m getting ahead of myself. Every murder mystery needs its list of suspects, and ours began with the usual collection of people who were quite certain they hadn’t done anything wrong.
The scene of the crime
I was working under contract at the International Monetary Fund in Washington, DC. This was perhaps fifteen or so years ago, and I hadn’t been with the IMF for very long. I joined the Web Content Group, or WCG, which primarily looked after the SharePoint applications, sites, and other things people interacted with using their web browsers.
Next door, organizationally speaking, was the Web Engineering Group — WEG. They dealt with the infrastructure, web servers, and other plumbing required to keep our work from becoming an elaborate collection of attractive error messages. WCG looked after the application; WEG looked after what kept it alive. We cooperated, but a failure could wander back and forth across that boundary for weeks.
I had already developed a good relationship with a couple of the WEG guys, particularly Andras, who was Austrian, and a Sikh colleague whose name I’ve unfortunately forgotten. They shared an office, and Andras had opinions about ours.
My officemate, Praashant, seemed to spend a remarkable amount of his workday on the phone to India. He also had a slow cooker at his desk, in which he sometimes cooked fish. Now, I’m not opposed to fish. I’m quite fond of it, actually. But when it has been simmering in an office for several hours, it acquires a certain diplomatic authority over the entire corridor. Andras would arrive to complain about the smell, and sometimes remained long enough for us to talk about actual work. So there was a social benefit to the fish.
Production was a real enterprise SharePoint farm: at least three web frontends — WFEs, for those of you who enjoy alphabet soup — plus application-tier machines and load-balanced SQL infrastructure for several thousand employees.
A WFE receives the user’s HTTP request and serves SharePoint pages. With several of them, you don’t tell users to remember whether they visited `wfe01` or `wfe02` last Tuesday. You give the farm a shared name, and the infrastructure decides which machine answers. That’s the theory, anyway. It is also why building a system with redundancy doesn’t make your problems go away. It gives your problems friends, and sometimes they agree to take turns.
The first sighting
The haunting began, as many hauntings do, with somebody reporting something that nobody else had witnessed. A user complained that the site was producing an HTTP 500 error, or something in that family. Not every time. Not even reliably enough for us to reproduce it when we wanted to.
The topology was new to me. Others knew the server names better than I did, but nobody could explain the symptom, and I knew how to test a theory.
For anyone who doesn’t spend their days deciphering web-server tantrums, an HTTP 500 means the server encountered an internal error while attempting to handle your request. Contrast that with a 404, in which the server says it can’t find the requested resource, or a connection failure, where you might never have reached an HTTP server at all.
Of course, these distinctions aren’t always tidy. GitHub may return a 404 for a private repository if you’ve switched accounts and no longer have permission to see it. That’s not a connection failure, but it sure looks like your repository vanished. Your Mileage May Vary from website to website, I suppose. A status code isn’t necessarily the whole story. A 500 is useful in a peculiar way. It is bad news, certainly, but at least something answered. That distinction will become rather important later.
At first, our tests seemed to suggest that the site worked. We’d go to one of the known servers by its machine name, and the page would appear. Someone else would try it and have the same result. If we couldn’t reproduce the error, what exactly were we supposed to fix?
This was the point where the problem began to gather a small collection of shrugs. You know how this goes. WCG could not demonstrate that its application was broken on demand. WEG could not point to a persistently failed server. People would suggest things, test them once or twice, and move on to more immediate work. And the user complaint would come back.
The situation had apparently been drifting along for weeks, which I wasn’t willing to accept. If a user can observe a failure, it exists, even if it refuses to appear during your fifteen-second demonstration.
I was confident I could solve it. Call that arrogance if you like, but I had spent years taking poorly behaved systems apart, and this looked less like an impossible problem than one nobody had investigated properly. Once a problem like that gets its teeth into me, I’m inclined to bite back.
There was a repeatable class of symptom, a defined system, and only so many places where the request could travel. That ought to be enough to start asking better questions. As it turned out, we had been asking the wrong ones.
Suspect number one: IIS itself
My first theory involved Windows Server and IIS — Internet Information Services, the web server underneath SharePoint. Those of you who have managed Windows web applications may remember the notorious loopback-check behavior. In some configurations, a server trying to access one of its own sites using a particular host name or alias can encounter an authentication problem, even though access through another name works. SharePoint adds enough authentication, host-header, and internal service traffic to make that a reasonable place to look.
The classic Windows loopback-check symptom is more often an authentication failure, such as a 401, than a generic 500. But the report was imprecise and I wasn’t ready to discard the idea without testing it. Besides, I could *invalidate* that theory.
Consider the difference between a machine resolving the farm’s shared hostname to itself and resolving that same name to one of its neighbors. If self-reference was the trigger, preventing it ought to change the behavior.
So, with WEG’s cooperation, we used the Windows `hosts` file to force deterministic name resolution for the servers’ own requests. Rather than allowing a WFE to resolve certain farm names back to itself, the mappings directed it toward another member of the farm. The object was to see whether forcing traffic away from local self-reference would make the problem disappear.
This wasn’t intended as a production high-availability design. The `hosts` file is a local override; it doesn’t consult the load balancer or adapt when the address you hard-coded stops working. For diagnosis, its stubbornness can be useful.
We controlled the destination and repeated the tests. The error remained, which was enough reason not to continue treating loopback as the leading explanation.
I don’t remember that we established every last detail of why the first idea failed, and there was no reason to keep defending it. An attractive explanation is still wrong if changing the condition it depends upon doesn’t remove the symptom. So I crossed off the loopback theory and looked for another suspect.
Names, addresses, and the trick they play on you
Here’s a short detour about names, because the entire mystery hinges on this. Imagine a farm with three known frontends. We’ll call them `WFE-A`, `WFE-B`, and `WFE-C`. Those are illustrative names, not the IMF’s actual hostnames — I’m not going to pretend I remember machine inventories from a decade and a half ago.
Each computer has an individual name by which an administrator can reach it directly. But the website also has a shared DNS name that ordinary users type into their browsers. That name may resolve to a virtual service address on a load balancer, or DNS may distribute multiple addresses through round-robin responses. Sometimes the architecture combines both approaches, because apparently one layer of indirection isn’t enough excitement for the average systems engineer. DNS tells a client where to attempt a connection. It is not, by itself, a guarantee that the application behind that address is healthy.
A hardware load balancer such as an F5 BIG-IP receives traffic for a shared service and chooses a backend pool member according to its configuration. It can perform health checks and stop selecting servers that fail those checks. The operative words here are can and according to its configuration.
A health check that asks whether a machine responds to a network probe is not the same as a health check that verifies SharePoint is returning useful pages. An application might respond with HTTP 500 all day while a shallow check concludes that the port is open and therefore everything is splendid. Now, the thing that took us a while to appreciate was that the *names* were telling us different stories. When we addressed individual WFEs by their machine names, they worked. When we approached the same application through the shared service name, we could encounter the error. That changes the list of possible causes. A shared name puts us on a different traffic path. If every named destination behaves and the shared path sometimes fails, the distinction is no longer a trivial detail of naming; it becomes a map of where to investigate.
Of course, that seems obvious *after* you understand the topology. I certainly didn’t arrive at the IMF with the entire network diagram engraved on my skull. If you know how to do that, please tell me. I could have saved a fortune on notebooks.
And there was another wrinkle worth recording: all our client-side tests had to be performed from our own workstations. We weren’t sitting at the reporting user’s desk, and we couldn’t simply borrow their machine for an afternoon.
In principle, our workstations should have behaved like any other IMF user’s computer. In practice, IT people sometimes do terrible things to their `hosts` files to keep working around an application that’s misbehaving. We were quite capable of having local overrides that made one engineer’s test path different from another’s. A result obtained through a locally overridden name doesn’t tell you what an ordinary DNS lookup would have done. So we kept the direct-server and shared-service tests distinct, paying attention not only to whether the page loaded but *how we got there*.
Suspect number two: the bad frontend
With the loopback idea failing to account for the problem, I moved on to the next plausible theory. One of our known WFEs might have been misconfigured. This was a perfectly respectable hypothesis. We had multiple machines that were supposed to behave alike. Perhaps one had different IIS settings, a missing component, an authentication problem, or some local bit of SharePoint configuration that hadn’t been replicated correctly. If the shared service sometimes chose the defective machine, we’d see intermittent errors. If it chose a healthy machine, we’d see success.
That would explain both why the user sometimes saw the error and why our direct visits to particular servers seemed normal — except that direct visits to *every* known machine were continuing to succeed. So we needed an experiment that could prove or disprove the idea rather than merely make it sound plausible.
Ideally we would have examined the load balancer’s pool and logs, identified which destination received a failed request, and followed the connection until something confessed. With the right access and people available, that might have been the cleanest route.
But we didn’t have all of that at our fingertips, particularly as the investigation ran into Friday afternoon. You don’t always get the ideal tools when the problem decides to present itself. Sometimes you have to work with the controls you *do* have. I proposed a process of elimination. We would remove one known WFE from service, test the application through the shared DNS name, and test the surviving machines directly by their individual hostnames. Then we’d remove another and repeat. Each change was coordinated with Web Engineering, because casually shutting down production web servers without talking to the people responsible for them is an excellent way to become a former contractor.
The goal was to control one variable at a time: which known frontends remained available. Changing the number of live servers can also affect load-balancer behavior, caching, and persistent sessions. Real networks aren’t laboratory glassware. But we could at least avoid changing several settings at once and then announcing that whichever problem stopped happening must have been cured by our favorite adjustment.
At each stage we compared the same categories of result. What happens through the shared service name? What happens when we request the site from each surviving server directly? Does the intermittent failure persist, disappear, or become more frequent? Do those observations agree with the hypothesis that one of our named WFEs is the culprit? This last question matters more than it may appear.
A troubleshooting exercise isn’t a scavenger hunt for settings you can fiddle with. You should know what each experiment is expected to tell you, including what conclusion becomes untenable if the result goes the wrong way. Otherwise, you can spend an entire career rebooting things and claiming victory whenever the problem gets bored and leaves for an hour.
Closing the circle
We started shutting down the known frontends, one at a time, and repeated the same tests after each change. The individual machine-name requests continued to work, but the shared service name remained our troublemaker. I expected that sooner or later we’d take the defective WFE out of the rotation and the error would disappear. Or, if the bad machine were the last one standing, the failures would become consistent. Either outcome would have helped us put a name to the problem; neither happened when expected.
After each shutdown I reviewed what we’d seen and what the evidence actually supported. We couldn’t call a machine guilty merely because an error occurred while it was online, any more than you can arrest everyone at a bus stop because somebody stole a wallet there. We needed the behavior to change when a particular server disappeared. We were narrowing the population of known frontends, but our results were stubbornly refusing to follow any member of that population.
Eventually we got down to the last known WFE. We checked it directly from our workstations and the site behaved. We used the shared service name, which ought to have had just that one known web server available to it, and about half of our requests still failed. The other half worked. Now, if your model of the farm says there’s only one remaining web frontend, what do you do with a result like that? You don’t get to wave it away simply because you’ve reached the end of your list. A one-server system ought not behave like two different respondents are taking turns, at least not without some other machinery in the path that you haven’t accounted for.
It was beginning to resemble one of those mysteries where the murderer escapes a locked room without using the door. We hadn’t found a guilty WFE. The direct requests gave the last suspect an excellent alibi, and taking the others offline hadn’t eliminated the intermittent failures. Perhaps the load balancer was doing something unfortunate; perhaps the DNS path wasn’t quite what we believed it was. There were other possibilities, of course, but the evidence was telling us that our inventory of candidates, or our understanding of the shared traffic path, was incomplete.
It’s sensible to check your work, particularly when administrators may have been fiddling with their own `hosts` files. But repeatedly asking the same question doesn’t improve the answer. You need a new experiment that distinguishes between the theories still standing. We’d removed the known servers, one by one. The next useful question was not which of those servers was broken, but whether something else could still answer the shared name.
What if the server we were looking for wasn’t on our list at all?
That would be a very different suspect. It would also suggest a rather direct test, although not one that anybody particularly wanted to propose on a production SharePoint farm late on a Friday afternoon.
Shut it down!
It was late on Friday afternoon. The IMF used a nine-and-a-half-day work schedule, so many employees weren’t even in the office that day. After four o’clock, the building was getting fairly quiet.
I retreated to Andras’ office and stared at our results. There were usually a few Heinekens in his fridge — my gift to him — and for a moment I was tempted to declare the matter somebody else’s problem. But a direct server name worked; the shared name failed half the time. We were running out of suspects, not evidence. I suggested to WEG that we shut down the final known frontend.
Understand what I was proposing. We weren’t shutting down SQL or the application tier; I wanted the final known web frontend removed from the path that should have served the site. We had a restoration plan. This wasn’t some grand philosophical decision to burn the farm down and see which crops grew back. It was a specific test.
If there really were no web frontends left serving that shared address, the site should stop responding as a web application. We might get a connection failure, an unavailable-service response from the network tier, or some other infrastructure-level indication that nothing useful remained behind the name. Exactly what the client would display would depend on the components in the path. But we should not continue receiving the same SharePoint-site HTTP 500 as if another web server were still working. If we did, then the hypothesis of an unaccounted-for respondent would become very difficult to ignore.
So we took that last known WFE offline. We went back to our workstations. We requested the site by its shared name. And it answered.
HTTP 500.
Again.
HTTP 500.
Again.
HTTP 500.
Every time.
There’s a particular satisfaction in discovering that something is much more broken than you suspected, not because you want it broken, but because the evidence has finally stopped contradicting itself.
Before, the shared name gave us good pages some of the time and an error the rest. With one known healthy server and an unknown other destination, that could make sense. Now the last healthy known server was gone, and the shared name produced nothing but errors. We hadn’t repaired anything. We’d made the visible result worse, and in doing so performed the most useful experiment of the week. The named machines weren’t the failure. The common service path was sending at least some traffic someplace outside our inventory of known frontends. Once we removed the last known healthy respondent, the abnormal path was all that remained.
Something was still out there. Maybe the call was coming from inside the house.
Now, if you are an engineer reading this, you are entitled to raise a hand and say, “Wait. Could the load balancer itself have generated that 500? Could it have been a proxy, an alternate virtual server, a stale connection, or some other device?” Yes. Those are fair questions. An HTTP status code alone doesn’t identify the physical host that generated it. With the right access, we might have interrogated the F5, inspected pool membership and health monitors, or traced a connection. We didn’t have all the answers yet, and I’m not going to pretend that we did.
But the remaining response proved that the system we had been describing to ourselves was not the complete system. Whether the extra respondent was a server, a proxy, or a configuration nobody remembered, there was another piece to find.
It was time to go looking.
Grab the Flashlights
There were only a few of us left from WCG and WEG. The offices were largely empty. Somebody produced flashlights, and we began working our way around the rooms, checking workspaces and looking for machines that had no business serving a production website.
There are more elegant ways to locate an unknown host: switch-port inventories, network tracing, management consoles. Those tools exist for good reasons.
But an organization may own a tool without giving you, at four-something on Friday afternoon, access to it, the person who administers it, or anybody who remembers how to use it. Sometimes the most sophisticated instrument you have available is a flashlight and a willingness to look under furniture. We checked the rooms, looking for powered-on computers nobody had mentioned.
Eventually, we found it.
Under a contractor’s desk sat a computer acting as a web server, quietly going about its business. It had been left there, connected and operational, while everybody had been examining the official SharePoint farm and wondering which of its known members had developed an intermittent personality disorder. I don’t know that we ever established, beyond doubt, who had configured it or what the original purpose had been. Perhaps it was intended for development, an experiment, or some temporary work that had somehow found its way into production traffic. Perhaps somebody thought it was a clever workaround. Fifteen years later, I see no reason to turn the story into an accusation against someone who wasn’t there to explain themselves. Besides, the machine didn’t need a sinister origin to be a serious problem.
It only needed to exist somewhere that mattered, without anyone knowing it was there.
Think about it. We had an enterprise SharePoint farm, redundant infrastructure, and people whose job was to maintain it. And somehow, beneath somebody’s desk, there was a box with enough influence over production service delivery to make users think the whole farm was defective. We’d been auditing the orchestra while an uninvited musician played a wrong note from the coat closet.
Web Engineering unplugged the machine. We tested the shared site name again. This time, it stopped answering. At last, a failure that made sense.
That was the decisive confirmation. Removing the unexpected machine changed the result in exactly the way our revised model predicted. Once we brought the proper web frontends and traffic distribution back into their normal operating configuration, the website worked. The haunting, at least for the site users, was over.
The ghost wasn’t the only problem
We had found the unwanted respondent, but there was another lesson buried in the incident. Something about the traffic distribution mechanism hadn’t been doing its job. At a minimum, the system was willing to keep directing client requests into a path that returned errors, rather than consistently selecting healthy web frontends. The F5 configuration, or the relationship between DNS and the load-balancing tier, deserved a serious inspection.
A load balancer is not clairvoyant. It cannot tell that a site is healthy just because the operator once gave a backend server a nice reassuring name. It needs an appropriate health monitor — ideally a request that demonstrates the actual application can serve what users need, rather than merely confirming that an IP address responds or TCP port 80 or 443 accepts connections. If the only question you ask the backend is, “Are you alive?”, it is perfectly capable of replying, “Yes, and I am broken.”
Of course, there’s a difference between showing that failed service was still being reached and proving exactly which F5 setting was responsible. I do not have the old configuration dump, and I wouldn’t claim we identified a specific monitor, pool, persistence rule, or DNS record from the evidence of that Friday alone.
But failing paths weren’t being excluded the way they should have been. That needed attention later, or the next genuinely broken frontend could have produced the same pattern. It’s worth making another distinction here. DNS round robin and hardware load balancing aren’t synonyms. DNS hands out addresses; a load balancer can make decisions about backend services after the client reaches it. Depending on how they’re composed, either one can obscure where traffic really goes. And the `hosts` file can bypass ordinary resolution on a particular machine entirely.
That is why a working direct hostname tells you only that that direct path worked. It doesn’t certify the shared service, and it doesn’t tell you where some other client’s DNS lookup or persistent connection will lead. I wish we’d known all of the topology when the complaint first came in. We didn’t. But we gradually learned enough by changing one condition at a time and refusing to merge unlike test results into a comforting statement that “the site works.”
The site worked by individual server name. The site failed intermittently by shared service name. And when every known frontend was gone, it still answered. Those are three different facts. Keeping them separate solved the case.
What I took away from it
I have spent a great deal of my career dealing with software and hardware that work perfectly well when considered individually, then become complete bastards when assembled into a system. The temptation is always to hunt for the broken part. Sometimes that’s exactly what you should do. Replace the failing disk, repair the corrupted configuration, restart the stuck service, and get on with your life.
But systems have properties that none of their parts possess in isolation. DNS introduces names that don’t necessarily point where you expect. Load balancers introduce decisions the client never sees. Authentication introduces distinctions between what a local server can do and what a remote client can do. And organizations introduce inventories, contracts, access restrictions, and team boundaries that influence how you debug it.
That’s why I don’t think of this as a story about a clever trick for finding a bad SharePoint server. None of the known servers proved defective. What we found were the limits of our assumptions. The first theory was wrong; testing it taught us something.
The second theory was incomplete; we kept eliminating suspects until its predictions stopped making sense. Then, rather than running in circles for another few weeks, we changed the question from “Which known server is broken?” to “What else could still be answering?” That last pivot is the one I want other engineers to remember.
There’s a difference between testing things until the error goes away and conducting an experiment that teaches you something regardless of the outcome. If you’re merely trying random changes, success may be as mysterious as failure. If you know what each result means, you can make progress even when the immediate symptom gets worse. In fact, on that Friday we made the site fail every single time before we repaired it. And that was the moment when the investigation finally became easy.
The inventory is not the network, and the approved architecture is not necessarily the running architecture. The documentation is a story somebody told about the infrastructure at some point in the past, and like most stories it may have an unreliable narrator.
TL;DR The documentation lies.
Computers don’t care whether they appear on your diagram, or whether the person who configured them has gone home, changed jobs, or forgotten they exist. If they’re powered up, connected, and receiving requests, they’ll continue doing whatever someone configured them to do. Usually without complaint. Sometimes with an HTTP 500.
There’s an old rule in troubleshooting that I think ought to be written in blood on the wall of every server room: If you eliminate every possible cause and the problem still happens, your list of possible causes is wrong. So, next time you’re faced with a web application that works when you visit each server directly but fails when you use its shared name, resist the temptation to announce that the whole farm is haunted. Test each path. Change one thing. Measure again. Write down what you actually observed, not what you expected to observe. And when the evidence refuses to identify any of your suspects, consider whether somebody neglected to tell you about another one.
Then get a flashlight.
I wrapped up that Friday with a burger and a double bourbon on the rocks at The Bottom Line on I Street. In that basement, under its hanging lamps, I’d overheard White House staffers reveal their deepest, darkest secrets. The interesting stories seldom made the official reports. Mine wouldn’t either, at least not then. Afterward, I limped my way home on the Orange Line to New Carrollton, where my wife picked me up. One more case closed, one more strange story under my belt.
Like Kolchak the Night Stalker, I knew it wouldn’t be my last.
