Incidents are used to alarm the user when configured network events occur, usually for traffic based rules, but also for system-specific events. These notifications can be viewed in the web GUI and may also be delivered on various notification channels. Repeating incidents are counted as such and the time of the first and last occurrence of an incident is remembered. This feature can be disabled for some incidents. What makes an incident unique depends on the type of incident.
The incident feature allows to define rules which are checked on the configured trigger point, like when a connection ends, a SIP call ends, or for checks on ongoing traffic. When such a trigger hits, configurable traffic attributes will be checked and if all attributes of a rule match, an incident is created.
Occurred incidents can be seen in the web interface, additionally reporting via the following notification channels is possible:
- Apache Kafka
- SNMP trap
The first occurrence of a medium or high severity incident will also trigger a status notification which is visible at the top right of the web GUI.
Up to 1000 incidents will be remembered by the system and if this limit is exceeded the oldest incidents will be discarded.
Configuration of incident rules
Incident rules can be defined in the "Configuration of incident rules" tab in the menu "Generic -> Incidents". All changes to the rule configuration will only take effect after saving the current configuration by clicking on the save button at the bottom of the page.
The page shows a table containing the existing rules and their configuration.
Each existing rule can be modified by clicking on the pencil symbol, or deleted by clicking on the "minus" symbol.
New rules can be added by clicking on the "Add rule" button. A dialog appears allowing for configuration of the rule. The same dialog is used when modifying an existing rule.
Add/modify a rule
A rule is defined by the following settings:
- Rule name: This is an arbitrary text describing the purpose of the rule. This text is shown in the incident list and email/Kafka/SNMP trap/syslog ouptut.
- Severity: three different severity values "low", "medium", and "high" can be used to group more important and less important incidents. Reporting channels can be configured to only report incidents of a minimum severity level. A rule can also be disabled by choosing the severity level "disabled". It will not be evaluated and can be enabled later at will.
- Trigger: The trigger defines when a rule is evaluated. For each available trigger, a description is shown next to it giving more details about the trigger. Some triggers are evaluated at a very specific time, like when a VoIP call ends, or are evaluated regularly like for throughput triggers of IP traffic which can be configured to be checked periodically. See list below for a detailed description of the available triggers.
- Attributes: Attributes are used to make actual comparison of expected values vs. actual values.
- Each trigger has a different set of attributes which can be checked for, and some triggers don't need to have an attribute at all. See list below for a detailed description of the available attributes
- Up to four attributes can be added by clicking on the "Add attribute" button.
- Multiple attributes must all match at the same time to let the rule create an incident.
- Each attribute can be compared to a specific value, so that the actual value is lower, equal, or greater than a defined value.
- Some attributes have an additional parameter, like a time span which defines how the attribute value is calculated.
- Virtual link group: The rule can be limited to a selected virtual link group or to be applied for any group. Some triggers cannot be limited to a virtual link group so the configuration will be hidden.
- IP filter: Depending on the selected trigger, the rule can be limited to a specific IP address. In firmware version >= 4.0, the IP filter can also be an IP subnet in the format IP/mask-length (Example: 10.0.0.0/8)
- IP group: Depending on the selected trigger, the rule can be applied to an IP group instead of an individual IP address.
- Virtual link group, IP and IP filter can also be used inversely by using the != comparator
- Report channel: Incidents are always visible in the web interface, but can also be reported via multiple channels which can be configured separately in the tab "Configuration of notification channels". Up to ten channels can be selected so that the incident for this rule is reported on each channel. Also, no channel can be configured so the incident is only accessible on the web interface.
- Aggregation of recurring Incidents: Incidents are aggregated by default. This means the table only shows the number of incidents of the type and the timestamps of the first and the last incident. This can be disabled for most of the incidents, so that you are able to see every indent of the incident-type.
- Time Profiles: With version 4.0 the new setting time profiles was introduced. You are able to set a profile which defines the active time of an incident rule.
- Traffic capturing [since version >= 4.0]: If supported by the trigger, the rule can be configured to capture the network traffic triggering the rule, including some extra time before and after the incident.
- Possible options:
- Disabled: capturing is disabled for this rule
- Live traffic: capturing happens only for live network traffic
- Replay traffic: capturing happens only for replayed network traffic (from PCAP files)
- Always: capturing happens in all traffic processing types.
- Extra capture time: configure the number of seconds before the start of the incident and after the end of the incident.
- If a time span parameter is used for attributes, the capture time includes this time duration as well.
- The traffic is automatically filtered to only contain the traffic that actually triggered the rule, i.e., an IP address or an IP group for IP rules.
- Possible options:
|Trigger name||Description||Attributes||Attribute usage|
|ARP: MAC change for an IP
|This trigger is checked on an ARP response and MAC address changed for a requested IP.||time_since_last_mac||optional|
|DNS: Server is not responding
|This trigger is checked when a DNS server is not responding for some time. A server is considered unresponsive when more than 3 requests to the DNS server went unanswered for a period of more than 5 seconds. Such a server must have answered at least two requests previously.||time_since_first_unanswered_request||optional|
|DNS: Server response error
|This trigger is checked when a DNS server responds a configured error of type format error, server failure or non-existing domain||error_type||mandatory|
|Global: Connection start
|This trigger is checked continuously at connection start. It can be used to report new connections with a certain layer 4 protocol and a given port range.||l4_protocol, port_range, since_start_time||mandatory|
|Global: GPS synchronization status change
|This trigger is checked when the GPS clock synchronization status changes.||gps_sync_status||optional|
|Global: Number of connections
|This trigger is checked continuously whether the amount of newly created connections exceeds a threshold. The update interval is defined by the timespan parameter of the attributes.||new_connections||mandatory|
|Global: Regular expressions
|This trigger allows to configure a list of regular expressions and is checked for each packet whose L7 data matches one of the regular expressions in the list. Since there are no attributes associated with this trigger, this effectively means that any packet which matches one of the regular expressions will result in an incident. The incident also contains information about which connection this packet belongs to as well as which of the regular expressions matches the packet.||no attributes are available for this trigger|
|Global: Ring buffer
|This trigger is checked continuously to report changes in the ring buffer.||used_size, bytes_captured, bytes_dropped||mandatory|
|Global: Speed change of an interface
|This trigger is checked when the speed of an interfaces changes.||interface_speed||optional|
|Global: Speed mismatch for an interface pair
|This trigger is checked when the status or speed of an interfaces changes and mismatches the speed of corresponding interface of a link.||link_speed_difference||optional|
|Global: Status change of an interface
|This trigger is checked when the status of an interfaces changes.||interface_status||optional|
|This trigger is checked continuously for the total traffic of the device. The update interval is defined by the timespan parameter of the attributes.||throughput, throughput_increase, packet_rate, packet_rate_increase||mandatory|
|IP: Connection end
|This trigger checks the attributes whenever an IP flow ended.||total_packets, total_bytes, tcp_handshake_time, percent_retransmissions, zero_window_packets, duration||mandatory|
|IP: Connection start
|This trigger checks the attributes whenever an IP flow starts.||new_connections, geolocation||mandatory|
|IP: Local IP with multiple MAC addresses
|This trigger is checked on each new flow of a local IP address and more than one MAC address uses this IP.||mac_count||optional|
|IP: New local IP address
|This trigger is checked once for each new IP belonging to a private network address range.||since_start_time||optional|
|IP: New local L7 protocol
|This trigger is checked once for each new l7 protocol used by a local IP.||since_start_time||optional|
|IP: TCP handshake
|This trigger is checked after successful TCP handshake.||handshake_time, server_handshake_time, client_handshake_time||mandatory|
|IP: Traffic on IP addresses
|This trigger is checked continuously for each active IP or IP group. The update interval is defined by the timespan parameter of the attributes.||throughput, throughput_increase, packet_rate, packet_rate_increase, total_packets, total_bytes, retransmission_ratio, zero_window_packets, tcp_syn_packets, tcp_fin_packets, tcp_rst_packets||mandatory|
|LACP: Status change of a channel
|This trigger is checked when the status of a LACP port channel changes.||channel_status||optional|
|MAC: New L7 protocol
|This trigger is checked when a unicast MAC address uses a l7 protocol for the first time.||since_start_time||optional|
|MAC: New MAC address
|This trigger is checked once when a new unicast MAC address appears for the first time.||since_start_time||optional|
|MAC: Traffic on MAC addresses
|This trigger is checked continuously for each active MAC address. The update interval is defined by the timespan parameter of the attributes.||broadcast_packet_rate||mandatory|
|PPPoE: PPPoE Discovery traffic
|This trigger is checked continuously for PPPoE discovery traffic. The update interval is defined by the timespan parameter of the attributes.||pppoe_discovery_packets||mandatory|
|Profinet: Traffic of Profinet devices
|This trigger is checked continuously for Profinet traffic. For max jitter, the update interval is defined by the timespan parameter of the attributes. All other attributes will be checked upon event.||alarms_low, alarms_high, errors, frames_lost, frames_repeated, frames_wrong_sequence, max_jitter||mandatory|
|PTP: Timestamp packet
|This trigger is checked when a PTP packet containing a valid timestamp is seen.||time_offset||mandatory|
|QOS: Traffic on QoS classes
|This trigger is checked continuously for each active QoS class. The update interval is defined by the timespan parameter of the attributes.||throughput||mandatory|
|RTP: Traffic for RTP connections
|This trigger is checked continuously for traffic of each RTP connection. The update interval is defined by the timespan parameter of the attributes.||jitter, percent_loss||mandatory|
|SIP: Call end
|This trigger is checked when a SIP call ended.||duration, status, mos, percent_loss, jitter, total_packets, total_bytes, total_caller_packets, total_callee_packets, total_caller_bytes, total_callee_bytes||mandatory|
|SMB: SMB1 negotiation
|This trigger is executed at the beginning of each SMB connection and checks whether insecure SMB1 has been negotiated.||none|
|This trigger is checked during handshake of each SSL connection.||certificate_expires||mandatory|
|SSL: SSL/TLS version negotiation
|This trigger is checked during version negotiation at handshake of each SSL connection.||used_tls_version||mandatory|
Special trigger properties
Some triggers are checked continuously every configured time span period, so the incidents are generated differently than for fixed event specific triggers like a call end.
- Repeating incidents: The following triggers will be evaluated every configured time span and will be re-issued whenever the configured attributes match.
- Start/stop incidents: The following triggers are reported once the configured attributes match and for a second time when the attributes no longer match.
So for repeating incidents you will get repeated incidents for the same attribute every time span. For example, if an IP address has traffic of 100 Mbit/s for 2 minutes and a rule checks for more than 50 Mbit/s over 30 seconds, the rule will hit 4 times. There will be one incident which will contain the exact number of repetitions for reference.
For start/stop incidents, you will only see two rule hits and the incident description will state the start and stop time.
- alarms_low, alarms_high: Whether an low/high alarm occurred for a Profinet device.
- broadcast_packet_rate: The attribute is the number of packets per second on average over the configured timespan for MAC broadcast packets.
- bytes_captured: The amount of bytes captured by a ring buffer in the given timespan.
- bytes_dropped: The amount of bytes dropped by a ring buffer in the given timespan.
- certificate_expires: This is the number of days until the certificate expires. If the certificate is already expired, the value is <= 0.
- channel_status: 0 means that the LACP port channel is not synchronized, 1 means that the LACP port channel is synchronized.
- IP: Connection end: The time between first and last packet of the flow.
- SIP: Call end: The call duration.
- Profinet: Traffic of Profinet devices: Whether errors occured.
- error_type: equal or not equal to:
- Format Error: DNS responds a format error.
- Non-existent Domain: DNS could not find queried domain name.
- Server Failure: DNS responds server failure.
- frames_lost, frames_repeated, frames_wrong_sequence: Whether Profinet frames have been seen with problems in sequence. For loss the count of lost frames is calculated.
- geolocation: checks if a country is part of the connection
- Direction: The direction of traffic
- from: Traffic is coming from the specified country
- to: Traffic is going to the specified country
- any: The specified country is on either side of the connection, or on neither side if the inequality is selected.
- Direction: The direction of traffic
- gps_sync_status: 0 means that the GPS clock in not synchronized, 1 means that the GPS clock is synchronized.
- handshake_time: The TCP handshake time between the first SYN packet and the ACK packet for the SYN/ACK packet of the server.
- client_handshake_time: The TCP handshake time between the SYN/ACK packet of the server and the ACK packet of the client.
- server_handshake_time: The TCP handshake time between the first SYN packet of the client and the SYN/ACK packet of the server.
- interface_speed: The current speed of the interface in Mbit/s.
- interface_status: 0 means interface is down, 1 means interface is up.
- SIP: Call end: The average jitter of the call, using the maximum value of both call sides.
- RTP: Traffic for RTP connections: The average jitter of the RTP connection for the given timespan, using the maximum value of both directions.
- l4_protocol: The layer 4 protocol. Can be TCP, UDP or other.
- link_speed_difference: This is the absolute difference between the speeds of both interface of a link in Mbit/s.
- mac_count: The number of different MAC addresses for the corresponding IP address.
- max_jitter: The max jitter value for a Profinet device in % in a given timespan.
- mos: The average MOS quality value of the call, using the minimum of both call sides.
- new_connections: The amount of newly created connections (TCP and UDP) for the given timespan.
- packet_rate: The packet rate in packets per second on average during the configured timespan.
- packet_rate_increase: The packet rate increase in % during the configured timespan compared to the average packet rate of the given baseline timespan. The noise can be configured to allow deviations that should not lead to trigger the incident.
- SIP: Call end: The percentage of RTP packet loss for the call, accounting packets from both directions.
- RTP: Traffic for RTP connections: The percentage of RTP packet loss for the given timespan, accounting packets from both directions of the RTP connection.
- percent_transmissions: The amount of TCP retransmission as a percentage of the total bytes.
- port_range: The TCP or UDP port. Can be also a range, e.g. 80,443,8443-8445
- pppoe_discovery_packets: The number of PPPoE discovery packets seen during the configured timespan.
- retransmission_ratio: The TCP retransmission ratio seen in the configured timespan.
- MAC: New L7 protocol: This is the number of seconds after packet processing start when a new Layer-7 protocol for the MAC address appeared.
- MAC: New MAC address: This is the number of seconds after packet processing start when the MAC address appeared. This is useful to only report new MAC address after some learning time.
- IP: New local IP address: This is the number of seconds after packet processing start when the IP address appeared. This is useful to only report new IP address after some learning time.
- IP: New local L7 protocol: This is the number of seconds after packet processing start when the Layer-7 protocol for the IP address appeared.
- Global: Connection start: This is the number of seconds after packet processing start when the connection hast been started. This is useful to only report new connections after some learning time.
- status: The call status code (a three digit number, like 200 for Success)
- tcp_handshake_time: The TCP handshake time.
- tcp_fin_packets: The number of TCP FIN packets (RX + TX) seen in the configured timespan.
- tcp_rst_packets: The number of TCP RST packets (RX + TX) seen in the configured timespan.
- tcp_syn_packets: The number of TCP SYN packets (RX + TX) seen in the configured timespan.
- throughput: The throughput bandwidth in bit/s on average during the configured timespan.
- throughput_increase: The throughput bandwidth increase in % during the configured timespan compared to the average throughput of the given baseline timespan. The noise can be configured to allow deviations that should not lead to trigger the incident.
- time_offset: The time offset between the local time and the timestamp seen in the PTP packet.
- time_since_first_unanswered_request: This is the time span between when the trigger is checked and the first DNS request that has not been answered by the DNS server.
- time_since_last_mac: This is the number of seconds between changed MAC addresses. If, for examples, dynamic IP assignment is used, changing MAC addresses is normal so the test can be limited to only a certain amount of time.
- IP: Connection end: The total number of bytes seen for both directions of the flow.
- IP: Traffic on IP addresses, QOS: Traffic on QoS classes, SIP: Call end: The number of bytes seen in the configured timespan.
- total_callee_bytes: The number of bytes seen for the callee of the call.
- total_caller_bytes: The number of bytes seen for the caller of the call.
- IP: Connection end: The total number of packets seen for both directions of the flow.
- IP: Traffic on IP addresses, QOS: Traffic on QoS classes, SIP: Call end: The number of packets seen in the configured timespan.
- total_callee_packets: The number of packets seen for the callee of the call.
- total_caller_packets: The number of packets seen for the caller of the call.
- type: The type of PPPoE discovery packet (PADI, PADO, PADR, PADS, PADT or any).
- used_size: The percentage of the ring buffer that needs to be filled.
- used_tls_version: The TLS version (SSl 3.0, TLS 1.0, TLS 1.1, TLS 1.2 or TLS 1.3)
- zero_window_packets: The number of packets with a TCP window of 0 for both directions of the flow.
- zero_window_packets: The number of zero window packets seen in the configured timespan.
Since firmware version 4.0, it is possible to automatically capture traffic for occurred incidents. These global settings control where capture files are stored and the capturing itself can be enabled for each rule separately.
The incident capture feature requires an active packet ring buffer since the packets are extracted from the buffer at the end of the incident period.
- Capture cooldown period: For each rule a cooldown period prevents multiple captures from happening in fast succession. By default, new captures happen firstly after 5 seconds, but any other value of at least 1 second can be configured. The cooldown is applied to each rule separately, but for each individual rule it does not matter if the same or a different entity triggers an incident. The incident is still reported within the cooldown period, but no additional capture is started.
- Storage device: Select the storage device where the captures should be stored on.
- Storage directory: Enter the directory where capture files should be stored or leave empty to use the top level directory.
- Select packet ring buffer to capture from: if multiple ring buffer cluster are in use, you can select which ring buffer to use for extraction.
- Capture profile: A capture profile can be configured to apply packet truncation rules to the capture file. If unset, the complete packets are captured (if the ring buffer uses separate truncation rules, truncated packets might still be within the capture file).
Incident rules can be active configured to be active in configured time spans (time profiles).
Every time profile allows the user to define one or more time spans per day of the week in which a rule should be active. After saving the user is able select the time profile when editing the rule.
Notes: Overlapping time spans will be merged. The earliest a time span is allowed to start is 0 and the latest end is 24, minutes are not allowed. The rule is not active for a day if there is no time span defined for the day.
Incidents can be reported on different channels. The configuration allows to add new channels so they can be selected in the rule configuration described above.
Each channel can be of type:
email Incidents will be sent to the email address configured in the Global settings.
kafka The incidents are sent to a topic on the configured Apache Kafka server. Firmware >= 4.0. The message is the same as for syslog.
- Bootstrap Server: hostname/ip:port of a Kafka Broker or multiple Brokers separated by comma
- Protocol: Plaintext (no authentication, no encryption), SASL Paintext (Plain authentication, no encryption), SASL SSL (Plain authentication, TLS/SSL encryption)
- Ignore TLS Certificate: Ignore the TLS Certificate of the Brokers (only SASL SSL)
- Username: Broker Login (only SASL)
- Password: Broker Login (only SASL)
- Topic: The name of the topic into which the Incidents are sent.
snmp_trap Incidents will be sent to the configured SNMP trap receiver (firmware >= 4.0). A MIB file with the Allegro Network Multimeter SNMP trap definitions is available for download in the channel configuration dialog.
- Version: Supported SNMP version of the trap receiver (SNMP v2c or SNMP v3)
- Trap receiver (manager) hostname/IP: The trap receiver hostname or IP address
- Community name: SNMP community name expected by the trap receiver
- Authentication protocol: Protocol for authenticated messages (MD5, SHA, SHA-224, SHA-256, SHA-384, SHA-512)
- Authentication password: Pass phrase for authenticated messages
- Privacy protocol: Protocol for message encryption (AES, DES)
- Privacy password: Pass phrase for message encryption
- Security name: Security name for authenticated SNMP messages
- Security level: Security level for SNMP messages (noAuthNoPriv, authNoPriv, authPriv)
- Engine ID: Authoritative (security) engineID (hexadecimal number)
syslog Incidents will be sent to the configured syslog server via TCP on any TCP or UDP port.
Each channel also uses a minimum severity setting, so only incidents are reported which are of at least that severity.
Each channel can be configured to only handle incidents from live traffic or from replayed traffic.
Some incidents cannot be configured via rules and you can choose to get those incidents also via email by enabling the settings at the lower part of the settings page.
Interface burst incident
Burst incidents with milli-second resolution can be generated when the interface throughput exceeds a configurable threshold. The incident contains a graph of traffic for that interface with some data points before and after the threshold has been exceeded depending on the measurement interval. A PCAP link for capturing from the packet ring buffer is shown. For further investigation of that incident, the button "Use as global time range" can be used to set the global range to the start and end of the incident graph (at least 5 seconds) so that all modules of the Allegro Network Multimeter show that time span. The incident generation can be configured as follows:
- Report "throughput threshold exceeded" with severity: report an incident with the selected severity level if the throughput of any network interface exceeded.
- Throughput threshold (Mbit/s): The threshold is configured in Mbit/s.
- How long throughput must be above threshold to generate incident (in milliseconds): The throughput must exceed the threshold for this duration in order to generate the incident. If set to zero (default) the incident is generated immediately after the threshold has been exceeded.
- Throughput cool-down period between two incidents in milliseconds: Defines the time after an incident where no new incident is generated even if the threshold is exceeded. If this period is passed, throughput incidents could be generated again.
This page shows up to the last 1000 incidents occurred on the system. The table can be filtered for specific severity levels, as well as for specific trigger sources by selecting the trigger from the drop down menu.
The list can also be filtered for the subject of the incident.
Individual incidents can be viewed in detail by clicking on the subject. The details page shows detailed information including links to the relevant measurement page.
Incidents can be deleted individually by clicking on the delete button next to the incident, or all incidents can be deleted by clicking on the button on the top right of the page.
Statistics about incident rules
This page shows graphs about how often each rule has been hit both in absolute numbers as well as relatively to how often the rule has been checked.
Incident list per measurement modules
Since incidents are triggered by different measurement modules (as indicate by the prefix of the trigger name, like the mac or ip module), the list of incidents from that specific module can also be seen in the corresponding tab of the measurement module for quicker access. This per-module view only lists those incidents coming from that module, all other potential incidents are hidden and must be accessed in their corresponding module page, or in the global view in the "Generic -> Incident" menu.
Some technical limitations apply:
- continuously checked triggers like "IP traffic" are only evaluated if there was at least one packet in the corresponding time interval. Therefore, rules check for zero packet count or throughput will never match.