JENNIFER Events is a flexible notification system that allows the operation teams to receive notification about incidents that take place in a system. JENNIFER Event Rules allows you to configure Events settings and thresholds.

In this article, we will take a look at the JENNIFER Events and the EVENT Rules.
 

Event Screen Explained

The EVENT Rule settings and configuration screen can be accessed from the "Management Menu => Event Rule". This screen is divided into 3 Tabs as follow: 

• ERROR EVENT: List all the Errors types that come out of the box with JENNIFER. 
• Metrics EVENT: Allows you to create custom metrics EVENT rule.
• Compare EVENT: Create an EVENT by comparing metrics data. 

All of JENNIFER Events share the following common characteristics: 

• Event Level: JENNIFER categorized events into 3 levels, Normal, Warning and Fatal, you can use these levels to categories events based on severity. 
• White Noice reduction: All of the EVENT Rules allows for white noise reduction settings. For instance, if a slow transaction detected 6 times within one minute, typically this means we should get 6 notifications regarding the transaction. However, sometimes this might be overwhelming or too noisy. Using the Measurement time and Reference count you can specify how often you want to get notified about a certain event.  Using the same example above, you might want to get notified only if 2 occurrences of the slow transaction detected within 1 minute. In that case, you can set the measurement time to 60 sec, and reference count to 2.
• Script AutoRun: This setting allows you to use the EVENT as a trigger to execute an arbitrary shell script.  You can enter the path to shell script, and when the specified EVENT occur, Jennifer will automatically attempt to execute the script.
• On/Off Toggle: You can turn on/off any event rule
• Configurable: You can apply different settings of the same Rule to different targets (agents).  For example, we have a system that consists of multiple agents, one of them get restarts often. In this situation, we can exclude the agent start/agent stop event from that agent and only apply it to the rest of the agent.

ERROR EVENT

The Error Events as stated before come predefined with JENNIFER. Using the ERROR EVENT is straightforward. You can enable/disable events as desired as well as modifying the EVENT level, measurement time. 
 

Metrics EVENT

The Metrics EVENT tab allows you to set your own rule based on metric data. Unlike the ERROR EVENT, the metrics event does not come with any predefined rule. Instead, you will need to define your own rules.
To understand the Metrics event better let us look at the following example.  We will use the metrics event to generate an EVENT when the heap memory usage rate is between 50% and 70%. We will start by adding a new Rule in the Metrics EVENT tab, we will set the Metrics to "Heap Memory Usage Rate" and the EVENT level to "Warning".

Next, we will define the Rule itself, for that we will use the Rule text field. The Rule text field allows you to set the desired threshold using the predefined keyword "value" along with conditional operations. The value will always reference the selected metrics, in this case, the percentage of the heap memory usage rate. So we can write our rule like so "value >= 50 && value <= 70" to get the desired result.

Finally, we can enter a custom message to be shown when the Rule is triggered, however, this is optional so we will just leave it as it is. 
The final configurations should look like the following: 

Whenever the condition in our rule is satisfied, JENNIFER will trigger the corresponding EVENT. We can view the EVENT from the EVENTs list or the notification list. 
Note that unlike ERROR EVENT, the metrics events do not have a link to view the coarsening transaction in X-View.

Compare EVENT

The final tab is the Compare EVENT tab. As the name implies, this setting allows you to generate an EVENT based on comparing metrics data value. For example, we can create a rule to generate an EVENT if the TPS is increased by 15% compared to the same day in the last week. Rule settings can look like the following: 

Note the Filtering condition checkbox, this checkbox acts as a triggering threshold of the Rule. For instance, you don't want the rule to be triggered for low values of the selected metrics. If we apply this filtering condition, then the Rule can be explained as "Generate event if the TPS increased by 15% of the same day in the last week, given that the minimum TPS value is 200". 

Receiving EVENT notifications

JENNIFER supports several channels to receive EVENT notification, includes:

• Email: Send Notification to a specific email.
• Slack: Send Notification to slack user/channel. This is available through the Slack Adapter 
• Telegram: Send Notification to Telegram. This is available through the Telegram Adapter.
• Other: As stated earlier in the article, the EVENT system is flexible, and users can create their own adapters to connect the EVENT system with any desired system they might use.

To be able to send notifications using any of the above channels. you will need to configure EVENT Integration.
The EVENT integration will list all available channel to send notifications, depending on the available adapters/configurations. 

The above example showing Slack and Email channel as they are both available and configured.
To be able to send the EVENT via slack, we simply Enable it by toggling the "External Link" button to "On". Then from the tab bar, we select the EVENT level. In this case, "Fatal", which mean JENNIFER will send a notification to the specified slack user/channel when Fatal events occur.