The
monitoring considered in the OP is for server farms and networks where the main challenges are rates of false alarms and rates of missed detections.
The challenge is to find means of monitoring that will permit selecting the rate of false alarms are willing to tolerate and, then, for that rate, get the lowest rate possible for missed detections.
Thus, would like to use the Neyman-Pearson result. Usually, however, for this context, do not have enough data for that. E.g., typically are quite short on data on the anomalies are trying to detect, and more short as the systems become more reliable.
From the above, we see that necessarily and inescapably such monitoring is some continually applied statistical hypothesis tests.
Apparently in practice, false alarm rate is not known and not reasonable to select or even to adjust.
Then we see that we need tests are both multi-dimensional and distribution-free.
A special case of high interest is zero-day problems, that is, detecting problems never seen before. So, this is behavioral monitoring -- any behavior sufficiently unusual is regarded as an anomaly, that is, evidence of something wrong.
From all I can see, so far the monitoring community has yet to take these points to heart.
The OP's remarks on thresholds are on target: Thresholds have been the old, lame, weak workhorse of monitoring far too long.
If anyone is actually seriously interested in this subject, let me know. Some years ago I concluded that no one was interested!
The challenge is to find means of monitoring that will permit selecting the rate of false alarms are willing to tolerate and, then, for that rate, get the lowest rate possible for missed detections.
Thus, would like to use the Neyman-Pearson result. Usually, however, for this context, do not have enough data for that. E.g., typically are quite short on data on the anomalies are trying to detect, and more short as the systems become more reliable.
From the above, we see that necessarily and inescapably such monitoring is some continually applied statistical hypothesis tests.
Apparently in practice, false alarm rate is not known and not reasonable to select or even to adjust.
Then we see that we need tests are both multi-dimensional and distribution-free.
A special case of high interest is zero-day problems, that is, detecting problems never seen before. So, this is behavioral monitoring -- any behavior sufficiently unusual is regarded as an anomaly, that is, evidence of something wrong.
From all I can see, so far the monitoring community has yet to take these points to heart.
The OP's remarks on thresholds are on target: Thresholds have been the old, lame, weak workhorse of monitoring far too long.
If anyone is actually seriously interested in this subject, let me know. Some years ago I concluded that no one was interested!