Best Practices for Creating Alert Detectors

Many time series-centric systems make it easy for you to alert on well-behaved raw metrics, using simple thresholds and with a large time tolerances. However, real-world data and real-world scenarios are rarely this well-behaved, simple or forgiving. In many cases, cloud operators not only need to know as soon as possible when there is an actual issue (preferably within seconds), but also alerting must cover cases where data is aperiodic, ephemeral, and delayed. Alerting also needs to account for composite or otherwise processed signals, cyclical or seasonal patterns, outliers, scale, and the potential for “flapping” behavior.

SignalFx provides a number of capabilities that allow you to alert on these real-world scenarios accurately and in a timely manner.