The Dashboard Said Connected. The Link Was Dead.

Earlier this week, on a site we look after, a WAN link turned out to be configured with one wrong digit in its gateway address. One character. Off-subnet, unreachable, dead.

The dashboard reported that connection as Connected. Latency read one millisecond — which, if you sit with it for a second, is the number you get when nothing is being measured at all. A real link on that circuit reads about twelve.

The site kept working, because the backup link silently carried everything. Nothing alarmed. Nobody called. And every inbound sales call to that business rode a double-NATed satellite connection, which does not present as an outage. It presents as the phones are weird sometimes. A lost job looks like a customer who changed their mind.

Nothing was broken

This is the part worth sitting with, because the obvious lesson is the wrong one. None of the tools were broken. Every one of them reported exactly what it was designed to report. The monitoring platform checked whether the device answered; it did. The dashboard displayed the interface status field; it said connected. The RMM agent was healthy, because the machine it ran on was fine.

Good products. Working correctly. Producing an answer that was wrong.

That is not a failure of any one vendor, and we would not want it read as one. It is arithmetic. A vendor builds one integration that has to serve every customer they have, so it reads the handful of endpoints the average customer needs and stops there. That is the only rational place to stop when you are building one thing for everybody. The trouble is that the state which actually breaks things — the settings that cost you jobs — tends to live a tier deeper than average.

A dashboard is not a measurement

The habit we would encourage in any business, whether or not we are the ones doing the checking, is to stop treating a green status as evidence.

A status field is a claim the device is making about itself. Sometimes that claim is checked against reality and sometimes it is copied forward from configuration. Connected can mean "traffic is flowing on this circuit" or it can mean "an interface is administratively up and nobody has asked it anything hard." Those look identical on a screen.

The questions that actually protect revenue are narrower and more annoying to ask. Does the login page of the platform this business runs on actually render — the real page, with real content — or did a server merely answer? Is the primary link carrying traffic, or is the backup quietly carrying all of it? Do the sites that were built identically still match each other?

Those are answerable. They are just not the questions an off-the-shelf dashboard was built to answer.

What this means if you run a small business

You do not need a bigger platform. That is usually the first suggestion, and for most businesses this size it is the wrong one — an enterprise log and correlation platform is built and priced for organizations with a full-time security operations team, and it would have ingested the same wrong answer anyway. The status field said Connected; that is the value that gets indexed, faithfully, at considerable expense. Somebody still has to know that one millisecond is impossible on that circuit and write the rule that says so.

What you need is for somebody to be asking the narrow questions on your behalf, and to be honest with you about the difference between what was confirmed and what was assumed.

That is most of what we do. Not more dashboards — fewer, better questions, asked against the equipment itself, with the answers written down as either confirmed on the device or assumed. Documentation that quietly presents a guess as a fact is the same failure as a dashboard reading connected, and it is why so much inherited documentation is worse than none at all.

If you are not sure whether anyone is asking those questions about your network, that uncertainty is itself the answer, and it is worth a conversation.