I often thought about how a perfect monitoring solution should like; then the following requirements come up:
1) consistent: When the monitoring indicates a problem, there really is a problem with the application.
2) reliable: When there is a problem that hinders the users from properly working on the application, the monitoring will indicate this, before the support line is overwhelmed by calls reporting that your application has a problem.
3) informative: When a monitoring indicates a problem, there should a recommendation how this problem can be fixed with minimal impact. This recommendation can be either documented offline (operation guidelines) or in the monitoring itself.
4) proactive: a monitoring resource should detect problems before they happen. Sounds strange, but many problems (excessive use of memory) can be reported before they actually have a big impact.
The requirement 2) is a hard one; essentially it requires that all problem situations are pre-known and the presence of such a problem situation can be indicated by the monitoring resource. In fact most situations are not known to anybody, until they happen and are fully analyzed and understood.
1) can fulfilled very easily: never report a problem. From a purely academical view this requirement is fulfilled then, but actually it is not really usable. A more usable approach is to report only when it's 99,99% clear that no user can work anymore (e.g. on a CQ authoring system when a vital OSGI service isn't available any more). But the more subtile problems cannot be caught that way.
3) requires a certain amount of experience with the application and the will to write good documentation, that is kept up to date. And 4) requires knowledge about typical problems and the signs of problems.
All these requirements often need an extension to the application to provide an interface to the monitoring, where the monitoring can fetch the data from and decide what to do with it. But monitoring is the poor child of application development; it is neither a functional requirement nor a non-functional requirement with the importance of usability, performance or availability (aah, by the way: how should we measure that? No-one cares, as long you guarantee 99,9% ...), but only a requirement of the operations team - no-one spends time or money on it, until operations ask for it. "Ups, we already spent our budget on other things". Only a few operations teams have the management standing, that they can deny such applications to run on their machines then, in most cases they are just overruled by management. Then, the operations team usually comes up with some thoughts and tries to fill the gap themselves, but in most times that does not work very well. Especially requirement 2) is then often violated and 3) and 4) are not implemented at all. But operations can show some green bulbs in the monitoring system to the management.
So, as a last requirement (which should be the very first requirement, though):
5) there should be a proper monitoring at all. A monitoring that only watches for a running process cannot detect that this process already internally has deadlocked and isn't working anymore.
Finally, when you need to implement a complex application, make sure that some of its internal state can be exposed to an external monitoring solution, which helps to operate your application. Treat this as a normal "must-have" feature with specification, implementation and test. You will make your operations team really happy.
If you do not do that, your monitoring system are actually the users, when they complain about non-working functionality, which must be fixed by your operations team then. And that brings costs (service calls) and negative management attention. Nothing one wants to have.
