Notes on Effective Monitoring and Alerting by Slawek Ligus --- ===== intro ===== Monitoring allow you to identify issues in your systems while or before they happen. The term itself has been used to broadly, but usually when people talk about this, they are talking about the way you get to know what is happening inside a system. This can be done in two ways, in a proactive way or reactive way. The first involves you watching some timeseries or dashboard, and do stuff with that information in hand. The latter involves you getting an alarm and checking the system. Monitoring and alarming are not trivial issues, there are some things you have to keep in mind to make a good system, for example, Usually to set off an alarm, you must set a condition or threshold, a metric (like cpu, mem, or whatever) must get to, in order to fire the alarm. The thing is that setting a right number for this threshold is tricky, too loose and you will get flooded with alerts, too tight and you may lose important info. For us to get a good number you need to have a baseline, this is how the system behaves in normal conditions, as the system grows this also may change so you need to keep updating not just the threshold, but the alerting configuration themselves. Extra Benefits of Alerting and Monitoring Systems ------------------------------------------------- Once you have like a good alerting and monitoring system in place, you can start doing interesting stuff, for example, when you have enough data points, and have been monitoring for a while, you can start forecasting issues using statistical models and so on. You can also have automation, have Autonomic Computing, which is how our body works, meaning, the system sees an issue, and it tries to regulate it or fix it by itself, you can setup self-healing self-management solutions. A Monitoring System ------------------- The systems usually have the same three functional parts: 1. Data Collection ofc we need data to start doing anything, so we need to collect data from logs, servers, databases, network equipment, this means that you may need to instrument the components to send data, or have something (like prometheus) scraping data from them. 2. Data Aggregation and Storage The data inputs are grouped by their properties and stored in their respective metrics. The timeseries generated by this, are now crossed by the thresholds/alarms we have and fire any if the condition is met. 3. Presentation The operator can visualize this timeseries and watch an overview of the current state of the system A good system has a single point of reference for all teams, where the platform engineers, developers, data engineers can come together to interact with the info freely. How to Alert ------------ The initial setup attempts to define abnormal conditions by setting up threshold for exceptional metric values. Usually from the baseline values, this values need to be tweaked and adjusted during the lifetime of the system. Common Challenges ----------------- - Baselining Baselines are volatile, the system is likely to change and usually people forget about changing this. - Coverage обычно when the alerts are setup is when the system is starting to get worked on, then this is often neglected and the system does not have full coverage, just for a few parts. - Manageability Large systems have tons of metrics, they become expensive to manage. - Accuracy Sometimes faults will remain undetected, some other times alerts will go off even if there is no issue at all. These are two related issues which need to be worked on. ========== monitoring ========== Monitoring goes beyond getting alerts, evidence based decision making, and automation, it has a key role in the absorption of knowledge. You cannot manage what you do not measure. We went a bit about the bases a monitoring system has, lets see what composes monitoring as a whole. Building Blocks --------------- We need to gather the data before doing anything with it. We have different types of data collection agents: - white box - log parsers get specific info from log entries - log scanners count occurrences of strings in log files. Count the number of ERROR in logs - interface readers Read system interfaces. Getting the CPU utilization. - black box - probes ping a system from the outside to see if it is alive - sniffers monitor network interfaces ### Monitoring Overhead vs The Observer Effect When you instrument a service for it to send info or smth, you are adding a bit of overhead, this is normal and should not be mistaken with the Observer Effect, which refers to when the behaviour of an entity changes when it is being observed. As for the instrumentation, you should keep it as simple as possible, just gather info and push it. So the overhead is not big. ### There is another way to get info from a system without using agents, this is helpful in systems with lots of restrictions. For example you could periodically exec diagnostics commands via ssh, though these types of data collection have a few disadvantages, since they introduce more overhead. Coverage -------- To have a full picture with your monitoring you should have these 3 metrics: - resource availability - network: all data is delivered thru the network which in fact will fact some latency. - computational resources: cpu, mem - software performance This is the solution stack, starting form the OS, going thru middleware and ending in the app itself, we need info from all layers. - user behaviour (where applicable) Metrics ------- These are the inputs you will plot into a time series, each metric consists of a recorder measurement value with a timestamp and some properties describing it. Time series are specially helpful since they illustrate the historical data. You can also layer in a single plot different time series and find correlations. Summary statistics --- * n - count of inputs per interval * sum - sum of values from all inputs * avg - mean for all inputs * p0-p100 - percentiles (0-100) of the input values * deviation - standard deviation from the avg of the dist of the collected inputs. Frequency Distributions and Percentiles - Frequency distributions is basically putting values in bins, and creating a histogram for them. Usually histograms will follow the normal distribution. Nevertheless system data is usually not that regular and typically displays a long right tail. The dist is skewed to the right. Summary statistics plotted as data points are convenient for observing change, but they may not tell you the complete story From the logs you can extract raw inputs and show them in a histogram to show their relative frequency. Percentiles make distributions easy to interpret; a percentile is a number where certain amount of scores are at or below that percentile. If I say p50, that means that half of the values are at or below that number. An example would be, the response time at p98 is 3 seconds, this means that 98% of the requests are are at or below 3 seconds. *Rate of change* is the point at which a plot goes up or down. *Time granularity" is how short is the period of time we use for the data points. If we use a small one is called fine granularity which is not always convenient since we may be taking a lot of noise for us to calc anything. Understanding Metrics --------------------- Each metrics can be seen in 3 general ways: - Amount: discrete or continuous values, like disk space, packet size. - Time Delay: time required for an action to complete, like response time - Amount per Time: amounts flowing thru the system for periods of time. Like bits per seconds, for example when you download something. These 3 can easily be visualized using the example of an HTTP request. - Amount would be the size of the http request, in bytes. - Time Delay would be the response time of such request - Amount per Time: Number of requests per seconds. Meaning how many requests went thru per a fix amount of time. The first two are *positively correlated* the larger the request, the longer the response time. The latter is negatively correlated with the first two, the lower the requests per seconds, the larger the request and the delay. Common time series patterns --------------------------- Check /img/ts-patterns.png - spiky: usually noticed on metrics relating to bandwidth usage and computational resources that are prone to sudden changes due to how they get utilized - steady: data point has low variability at no-zero level. Usually seen when measuring availability and underutilized resources. If no trends are found you can get thresholds out of this. - counter: variable increases at a constant time interval, do not decrease but it may get reset. For example, number of seconds since last update. - bursty: comes from components which have a sustained period of activity and then drop to a baseline. - cyclic: sinusoidal pattern, most of data points which come from human activity will have this. - binary: metrics recording 1 and 0 - sparse: metrics which do not receive inputs at every interval; for example when recording errors. - stairy: metrics that rise and fall but then stay at the same level for a period of time. Usually seen on disk measurement, free memory, number of processes running. Drawing Conclusions from Time Series ------------------------------------ Interpretation of anomalies --- In order to figure out what is going on, you first need to identify the type of recorded quantity (flow, stock, availability, throughput) and the summary statistic used (n, sum, avg, percentiles). Now you can see them thru two lenses, *bulk* and *distribution*. The first will tell you how the system is operating, like how much memory it is using, how many requests has it got, and so on, and it will tell you what is happening right now. The latter will explore a time window, and then analyse how the system behaved in that time window. Usually you can do this with percentiles or avgs, for example, from 7 to 9pm, what was the p95 latency, meaning, what was the latency for the worst 5% of requests. i We get an idea on how well the system is doing. The interpretation varies a bit from the type of recorded quantity, so remember: - bulk: n, sum (what happened in total/what is happening) - distribution: p50, p99 (how the system has been behaving, and sometimes avg depending on the metric) - flow - bulk: input levels, processing rate - dist: bottlenecks, early indications of resource saturation - stock (anything that fills up or drains) - bulk: available space, state since last reset, messages in queue - dist: how erratic the disk filling up is, like maybe smth is writing violently to it. (also read more about Memory jitter) - availability this takes binary data points and avg it between the number of data points it took. - bulk: total downtime minutes, total checks - dist: level of availability - throughput: - bulk: requests finished - dist: intensity of utilization, wobble score (variacion), a high values means that at one point it can process just say 2 requests, but at the next one it process 100. Which is an indication of burstiness. Frequently Encountered Anomalies -------------------------------- Here are diff types of patterns you will find in time series - Flatting Effect /img/timeseries-pat/flattening.png When the line in the plot reaches an *artificiallly steady point*. Usually it means a saturation of resources or discontinuation of the flow. Like the line wont go above 200ms because at 200ms we drop the connection. That kind of stuff. - Warm-Up Effect /img/timeseries-pat/warmup.png When adding a new server to the fleet, it will process data at an slower pace while it warms up. - Regular Anomalies Spikes happening at equally spaced periods of time, like at every hour you see an spike. Could be external or internal, check logs at such hours to find out what is happening. - Spikes During Troughs (Valles) The main take away for this is that p99 is subject to your sample size, so if you are getting a bunch of requests, but then you hit a troughs, p99 will get request it wouldn't have when it was getting a bunch of requests. So always try to use a big enough sample size. Determining Causality --------------------- Three logical steps 1. Find correlation First go find the time series relevant to the system that is failing and try to see if there is some relation between them. But remember correlation does not imply causation, that is why we need to go to the next step. 2. Establish direction Try to find which one failed first, time granularity comes handy here, but we need to find if the problem comes down or upstream. 3. Rule out confounding factors Sometimes many components may display abnormal behavior, but it does not necessarily means they are contributing to the issue at hand. Finding Trends -------------- Storing CPU and mem usage for more than 4 weeks may be wasteful, what it is more useful to store for long is usage patterns. ======== alerting ======== Alerting is a battle of sensitivity (when is something abnormal) vs specificity (when is it safe to assume that no problem exists) The goal of an alert should be to tell an operator there is a noticeable performance degradation, which can be seen in three ways - decreased quality of input - increased response time - loss of availability prereq ------ What do you need to setup and alert system? First you need a monitoring platform that has an easy way to deploy data gathering agents, and that is able to plot different timseries into a single chart. While also having good alarm configurations. Then you also need to have an audit trail of what is happening in your system at every time, like when someone deploys something Finally, you must have some place to create the tickets and track them. Understand failure and its impact --------------------------------- First and foremost you need to setup a clear range of significance for your team to setup your priorities right. Setting up Significance level --- ofc not all failures require an operator attention, the failures that do must be of high enough significance that it negatively affect the system's operation. If the system cannot recover by itself. We can break down severity in two main things: - recoverability: can the system fix itself without you having to do anything? - impact: negative effect on system operation. - downtime - partial loss of availability - increased response times - decrease in content quality - suboptimal utilization of resources With this in mind, we can define the levels of severity 1. Critical severely impair the system's operation. If you do not stop this, it translates to unrecoverable losses in productivity or revenue. A deployment is 0/3 on its pods. 2. Urgent partial loss of availability is observed, a significant part of the system becomes unresponsive. Require fast intervention to stop them from scaling to Critical ones. The DB connections are saturated, a percentage of new connections is getting dropped, the queries that do go thru are being able to be fulfilled correctly, but there is a considerable percentage that is getting dropped. 3. Intervention Necessary It wont fix itself, and you need to step in to prevent this to scale into critical. The system is at a high risk of failing if this is not fixed. You check the messages in a queue and see that they are piling up, the consumer is working properly but the load may be too big for it. 4. Recoverable Negative event but non critical to your system, a pod restarted it may develop into a bigger problem if not recovered within an expected timeframe. 5. Inactionable (maybe record this in elastic) The speed of recovery is too high, the issue gets solved before the operator has time to go and fix it. 6. Automation Tasks Fault does not impact system operation, but may be a contributing factor for other bigger issue. 7. Optimization Tasks No effect on system's functionality but making the system run inefficiently 8. Nonissues Anomalies with no perceivable impact. Out of this ONLY THE FIRST FOUR require an alert. For these 4 the alert configuration most be consider two contradictory goals, speed and accuracy. - Critical: we do not care too much about the accuracy on these ones, since a failure is a big failure, it is justified to have some false positives. This is a trade we do for speed. - Urgent: You can wait for more data points before raising the alarm. The alert is issue quickly but with higher confidence than a critical one. - Intervention Necessary: alert most be issued only when we are almost certain that the issue presented is happening. - Recoverable: fire the alert if the recovery is way too long, you are sure it wont be able to do it by itself. Identifying Causes --- There are four big groups of failure types - Resource unavailability of saturation Most common, you are out of memory, network blips, so on and so forth. These gradually build up to saturation, they may be prevented if detected early. A well design system will have mechanisms to gracefully degrade admission under excess load until it comes back to normal. - Software problems The app is failing due to some bug or smth, these can be fixed fast through rollout of prev version or a new version with patch - Misconfigurations You should have your configs in version control so you can rollback. This type of errors are prone to human error - HW Defects not really applicable to us, but yeah the ram stick of a server fails or whatever, the PDU gets fried, so on and so forth Anatomy of an Alarm ------------------- An alarm can be defined as a shell encapsulating a condition and an action that will be fired when that condition is met. All alarms describe the relation between some inputs, which can be split in 3 categories. - metrics monitor - date/time evaluation - other alarms Metrics monitors --- They are made of 4 parts: - name and dimensions of a metric - threshold value - minimal duration of breach - time necessary to clear Usually 1 data point does not mean that the whole system has gone awry. You need to specify how many data points must be outside the threshold in order to trigger the alarm. Same goes the other way around, when the system is failing 1 data point between the threshold does not mean that everything is back to normal, it should be during a sustained period of time. There are four types of thresholds - Upper limit - Lower limit - Outside Range: capped both under and above - Data Points not recorded Time evaluation --- This is another input for your alarm. This defines a function with a temporal condition that returns true/false at a specific point in time. You can use these for... - suppressing a group of alarms for some time (like maintenance windows) - you do not want to trigger alarms over the weekends But these are for suppression mostly Other Alarms as Input -- An alarm is a boolean at the end of the day, so you can have some alarms trigger other alarms Suppression ----------- This means to prevent an alarm from being triggered even if the threshold is breached This can be manual or automatic. Manual should be enabled only for a period of time, so that it automatically starts again, without the human having to remember it must turn the alarm back on. You can also use this to not flood the user with alarms. Say you have a web app, and a database that live on a k8s cluster. - alarm1 = web app - alarm2 = db - alarm3 = k8s if one day k8s dies, then you will have these 3 alerts triggered. That may overwhelm the operator. So a better thing to do is adding some suppression to them. - alarm1 = web app AND not alarm 3 - alarm2 = db AND not alarm 3 - alarm3 = k8s Now if k8s dies, operator can work with that only, and then when k8s is back online, if the web app and db still are failing, the operator would get an alert for each of them. Much more manageable. Aggregation ----------- Group related alarms in order to de-duplicate the amount of notifications. Three forms of aggregation. - Any (input1 or input2 or input3) - All (input1 and input2 and input3) - By Count ( (input1 + input2) >= 2 ) Say you want to alarm as soon as the cpu utilization exceeds 60% on at least 30% of the hosts. - cpu_alarm = sum(cpu_monitor() for cpu_monitor in hosts) > len(hosts) * .3 - cpu_monitor returns whether the cpu exceeds 60% or not Data Pipeline example --------------------- /img/timeseries-pat/pipeline.png The monitor checks that we are processing at least 1 item, if 0 then the monitor is triggered. This example shows a pipeline with the aggregation alarms, if one point of the pipeline fails the alert will be triggered. It also has a suppression for maintenance. As soon as ANY of the components fail, this alert is sent as a whole. /img/timeseries-pat/pipeline-soa.png Now imagine that each components is managed by a diff team, then does not make sense to group them. The alarm config is different now, for the first component in the pipeline if the monitors fires, and we are not in maintenance then we must fire, but for the rest, we also need to check the previous monitor because if it is failing then we should not trigger the alert since the previous team needs to fix its issue first. Setting up Alarms ----------------- This is a four step process: 1. Identifying impact There are 3 main failure groups, decreased quality of output, increased response time, loss of availability. You need to see how the issue is represented in the timeseries. Availability is expressed as the percentage of time the system responds as expected. It is wise to setup an external probe for this. Also note that requests with extreme long response time have the same effect as loss of availability. 2. Establish Severity We went over this in the Significance level, so maybe re-read that part, but it was a combination between system downtime and how fast would it recover by itself. 3. Choosing the right time series Now we must select one metric, that is best capable of our monitoring objectives. When you know which, you must establish the summary statistic to use. - n, sum: good for measure the rate of inflow and outflow. (traffic levels, items processed) - avg, mean (p50): - avg, when components and processes have a fair degree of recoverability - median, when looking for most typical input in the population - high and low percentiles: monitor failures that require immediate intervention - extreme percentiles in input can reveal bottlenecks early these are preferred over avg for detecting faults it is not a good idea to setup monitors around p0 and p100, even p99 may be too much if the sample size is small Configuring Monitors --- A monitor is attached to the