Monitoring systems is not true observability. In monitoring you’re thinking and trying to enumerate all known failure states. Everything is dependant on your prior knowledge and intuition of what can go wrong or is necessary to debug.
If your system becomes more complex, there will be more and more unknown failure states that are not anticipated. Without true observability, you will miss this and have no real insights into these novel states.
It is a mixture of a reactive and proactive way. You’re proactively thinking of all possible failure status whilst being reactive to unknown ones.
Dangers of not having true observability
When you cannot recreate the novel system state or have an unknown state you might need to ship code to gather this information. Not only is this very slow, it might take a while before you capture meaningful data.
The more complex a system grows, the harder it is to sift through the noise to get an actual signal, also known as the signal-to-noise ratio.
…
Reference
Majors, Charity, et al. Observability Engineering: Achieving Production Excellence. First edition, O’Reilly, 2022.