REST API description: Difference between revisions

From Allegro Network Multimeter Manual
Jump to navigation Jump to search
Access restrictions were established for this page. If you see this message, you have no access to this page.
No edit summary
No edit summary
 
(100 intermediate revisions by 9 users not shown)
Line 1: Line 1:
* General
This page describes how to access and use the REST API. It allows to post-process data with 3rd party systems. The Allegro Network Multimeter web interface is itself based on this REST API and all displayed statistics can be extracted from the Allegro Network Multimeter with this API.
* Examples


– SSL certificate errors
== General API Setup ==


– Statistics about MAC or IP addresses
=== REST API Interface ===


– Pretty displaying JSON output with jq
All Allegro Network Multimeter statistics are derived from HTTPS requests and provided as JSON objects.
The requests are stateless, i.e. there are no prerequisites and there is no fixed sequence of requests necessary.
Example requests related to a specific module and statistics can be seen in the web interface by opening the browser development console (Ctrl+Shift+I for Chrome and Firefox, F12 for Edge).
 
Here an example of the structured JSON data for the IP overview. This data has been extracted with the Google Chrome developer console while accessing the IP statistics page.
 
[[File:Rest api chrome console.png|800px]]
 
=== Credentials ===
 
The credentials are the same as for the web interface. Users with the admin role are allowed to access all APIs per default. With version 4.1 the access can be configured/restricted in the role settings. A non-admin user has read access to most of the statistics. If you have enabled the pcap role, the capture URL is also possible for the API.
 
Starting with verison 4.2., it is possible to use [[User Profile Settings|API tokens]] instead of username/password. Permissions for API tokens can be configured when creating the token.
 
Allegro Packets recommends to set up a separate non-admin user with or without the pcap role for the REST API of only statistics shall be gathered. This will prevent to accidentally shut down or change any configuration by calling the REST API.


– Traffic counters
== Useful shell commands and their parameters  ==


– Time interval selection
=== Curl ===


– Extract received bytes of the last second of a certain IP
Most examples are written for curl [https://en.wikipedia.org/wiki/CURL]. Curl is available as for many operating systems like Linux or Windows. Curl needs a few parameters for the access of the Allegro Network Multimeter:


– Extract received and transmitted bytes of the last second of a certain IP
The parameter '''-u''' allows you to set a user name and password for the request.


– Extract received and transmitted bytes in a time interval (18/06/11 9:00 - 10:00) of a certain IP
The parameter '''-k''' will allow self-signed certificates. This is not necessary if a proper trusted certificate is installed on the Allegro Network Multimeter.


– List queries
The parameter '''-s''' or '''--silent''' mutes any debugging output.


– Show IP address with the highest amount of traffic
The URL of the API call is the first argument. It is recommended to enclose the API call with the character ' to avoid replacing the argument ( unless you need to replace parts of it )


– Show all peers of a certain IP address
<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/...'</pre>


– Capture a certain IP
Please note that you might need to use <code>curl.exe</code> in windows.


– Capture two IP addresses with ports on a certain layer 4 protocol
==== Using API token ====
When using an API token, the authentication is added using parameter '''-H''' to add a special HTTP request header (with '''API_TOKEN''' as a placeholder for the real token):
curl --silent -k -H "Authorization: Bearer API_TOKEN" '<nowiki>https://allegro-mm-XXX/...'</nowiki>
This can be further simplified by writing the header into a separate file, for example '''api-token-header.txt'''. This also has the benefit that the credentials are not visible to every user on the system in the process list.
Authorization: Bearer API_TOKEN
Now this file can be referenced by curl:
curl --silent -k -H @api-token-header.txt '<nowiki>https://allegro-mm-XXX/...'</nowiki>


– PCAP parameters
=== PowerShell ===


The Integrated Windows PowerShell can be used to access the REST API. This guide requires at least PowerShell v6.


=='''General'''==
The command to call a REST API is '''Invoke-RestMethod''' [https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod].
Invoke-RestMethod on the PowerShell needs a few parameters for the access of the Allegro Network Multimeter:


All statistics of the Allegro Network Multimeter are requested as HTTP requests and provided as a JSON object.
To set the user name for basic authorization, use the '''-Headers''' parameter:
The requests are stateless, i.e. there are no prerequisites and there is no fixed sequence of requests necessary.
 
Example requests related to a certain module and statistics can be seen quite easily in the web interface by opening
<pre>-Headers @{Authorization = ("Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f 'USER', 'PASSWORD'))))}</pre>
the browser development console (Ctrl+Shift+I for Chrome and Firefox, F12 for Edge).
 
The credentials are the same as of the web interface. A non-admin user has read access to most of the statistics.
When sending POST requests (or other requests with a body), the request content type must be specified (not necessary for requests with empty request body (GET, DELETE)):
 
<pre>-ContentType 'application/json'</pre>
 
To disable the certificate check, use:
 
<pre>-SkipCertificateCheck</pre>
 
The URL must be passed with the parameter '''-Uri''', so the full command is:
 
<pre>Invoke-RestMethod -Uri 'https://allegro-mm-XXXX/...' -Headers @{Authorization = ("Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f 'USER', 'PASSWORD'))))} -Method 'Get' -SkipCertificateCheck</pre>


==== Using API token ====
When using an API token, the Authorization header changes (with '''API_TOKEN''' as a placeholder for the real token):
Invoke-RestMethod -Uri '<nowiki>https://allegro-mm-XXXX/...'</nowiki> -Headers @{Authorization = ("Bearer API_TOKEN")} -Method 'Get' -SkipCertificateCheck


=='''Examples'''==
=== jq ===


In this section some examples are given using PowerShell and Linux curl and jq commands.
jq ([https://stedolan.github.io/jq/]) is a powerful tool to extract parameters from a json document. If called without parameters, jq formats the JSON output into a readable format with indenting and new lines. It also allows to select specific values and do basic operations like addition with this values.
Please read the jq documentation for more information.


== API hierarchy ==


'''SSL certificate errors'''
=== Query available URIs with OPTIONS ===


If the default self-signed SSL certificate is used, curl and PowerShell are displaying error messages. You can either
The Allegro Network Multimeter REST API has the fixed URI <code>API/stats</code> for statistics. To see all possible subtrees of a specific request, use the '''OPTIONS''' request instead of '''GET'''. It can be set with the parameter <code>-X OPTIONS</code> for curl.
install another SSL certificate or let the tools ignore the errors.
For curl it is
{| class="wikitable"
|-
| curl -k ...
|}


For PowerShell in recent versions (v6 and later) it is
<pre>
{| class="wikitable"
$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats' -X OPTIONS
|-
{
| Invoke-RestMethod -SkipCertificateCheck ...
  "subResources": [
|}
    "modules",
    "reports",
    "incidentReporting",
    "time",
    "ringBufferReplay",
    "pcap",
    "reset",
    "interfacesError",
    "interfaces",
    "load",
    "processing"
  ]
}
</pre>


This allows you to query for example for all modules that are available for a specific release of the Allegro Network Multimeter:


'''Statistics about MAC or IP addresses'''
<pre>
{| class="wikitable"
$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules' -X OPTIONS
|-
{
| curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/mac/macs/ff:ff:ff:ff:ff:ff'
  "subResources": [
curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3'
    "pppoe",
|}
    "lldp",
    "groups",
    "mpls",
    "opc_ua",
    "quality",
    "ipsec",
    "profinet",
    "multicast",
    "burst_analysis",
    "global_incidents",
    "ptp",
    "ntp",
    "icmp",
    "stp",
    "sip",
    "smb",
    "dpa",
    "l4_ports",
    "netbios",
    "crt",
    "dpi",
    "http",
    "ssl",
    "dns",
    "dhcp",
    "location",
    "qos",
    "ip",
    "mac_protocols",
    "vlan",
    "arp",
    "packet_size",
    "mac",
    "capture"
  ]
}
</pre>


=== URI content parameters ===


'''Pretty displaying JSON output with jq'''
Some modules allow to use a parameter as part of the URI like the IP or Mac address. The path <code>API/stats/modules/ip/ips</code> allows you to use an IP address as next uri element
{| class="wikitable"
|-
|| curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3' | jq
|}


<pre>
$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips' -X OPTIONS
{
  "subResources": [
    "protocol",
    ":ip"
  ]
}
</pre>


'''Traffic counters'''
The path of an IP address shows all further available elements:


Traffic counters are represented as an JSON array with at least 4 lines. The structure is as follows:
<pre>
* line 1: received packets
$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.0.54.254' -X OPTIONS
* line 2: received bytes
{
* line 3: transmitted packets
  "subResources": [
* line 4: transmitted bytes
    "sip_request_responses",
* other lines are module specific
    "peers_ports",
    "sip_responses",
    "sip_requests",
    "qos",
    "ports",
    "connections",
    "protocols",
    "macs",
    "peers",
    "tcpStats"
  ]
}
</pre>


Following counters are supported:
== JSON output traffic counters ==
* interval: Values of the selected time interval. If no interval is specified, this is similar to lastSecond.
* allTime: Values since start of the Allegro Network Multimeter.
* lastSecond: Values of the last second.
* intervalPerSecond: Average per second value of the selected time interval. If no interval is specified, this is similar to lastSecond.


All counters are aggregated counters, either for the selected time interval or since the processing start of the Allegro Network Multimeter. Many traffic counters have 4 separate values. These traffic counters are represented as a JSON array with at least 4 lines. The structure is as follows:
* line 1: '''received packets''', extraction example: jq .lastSecond[0]
* line 2: '''received bytes''', extraction example: jq .lastSecond[1]
* line 3: '''transmitted packets''', extraction example: jq .lastSecond[2]
* line 4: '''transmitted bytes''', extraction example: jq .lastSecond[3]
* additional lines are module specific


'''Time interval selection'''
The following counters exist for many REST APIs like IP, MAC, l4 protocol, l7 protocol and many more:
* '''interval''': Values of the selected time interval. If no interval is specified, this is similar to lastSecond.
* '''allTime''': Values since start of the Allegro Network Multimeter.
* '''lastSecond''': Values of the last second.
* '''intervalPerSecond''': Average per second value of the selected time interval. If no interval is specified, this is similar to lastSecond.


Requests can be given a time interval. Following GET parameters are necessary:
Please note that all counters are byte counters, not bit counters. You need to multiply the counters by 8 to get the bitrate.
* starttime, endtime: Start and end time of the interval. Format: seconds since 1970/01/01 UTC (Unix time, epoch).
* timespan: Duration of the interval selection in seconds


This example extracts the received bytes of the last second of a specific IP.


'''Extract received bytes of the last second of a certain IP'''
<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3' | jq .lastSecond[1]</pre>


{| class="wikitable"
This example extracts received and transmitted bytes of the last second of a specific IP.
|-
|| curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3' | jq .lastSecond[1]
|}


<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3' | jq '.lastSecond[1] + .lastSecond[3]'</pre>


'''Extract received and transmitted bytes of the last second of a certain IP'''
== API parameters ==
{| class="wikitable"
|-
|| curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3' | jq '.lastSecond[1] + .lastSecond[3]'
|}


The Allegro Network Multimeter REST API has a number of query parameters that can be added to modify the request. By default, the API will display the real time counters since the last restart of the processing unit.


'''Extract received and transmitted bytes in a time interval (18/06/11 9:00 - 10:00) of a certain IP'''
This example extracts the amount of received and transmitted bytes for an IP address since the processing start of the Allegro Network Multimeter. It queries via the REST API the JSON and then adds the values second value  ( row 1, rx bytes ) and 4th value ( row 3, tx bytes ) of the interval counters together.


{| class="wikitable"
<pre>
|-
$ curl --silent -k -u USER:PASSWORD "https://allegro-mm/API/stats/modules/ip/ips/10.54.0.254" | jq '.interval[1] + .interval[3]'
|| curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3?timespan=3600&starttime=1528700400&endtime=1528704000' | jq '.interval[1] + .interval[3]'
</pre>
|}


=== Time interval selection ===


'''List queries'''  
Requests can be given a time interval. If present, the '''interval''' counters are adjusted to this interval.  The following GET parameters are necessary:
* '''starttime''', '''endtime''': Start and end time of the interval. Format: seconds since 1970/01/01 UTC (Unix time, epoch). You can use <code>date +%s</code> on your machine to adjust to the best interval. Please consult the man page of date for more parameters.
* '''skiphistorydata''': shall the JSON include the history data without datasets, this reduces the amount of transferred bytes if datasets are required to render a graph, can be false/true default: false
* '''timespan''': required resolution for the graph dataset


List queries are requested with pagination parameters to reduce the size of the resulting JSON object and to in-
This example extracts the amount of received and transmitted bytes for an IP address for the last 24 hours.
crease performance. In the resulting JSON object, the list elements are stored in the displayedItems array.
<pre>
Following list parameters are possible:
$ curl --silent -k -u USER:PASSWORD "https://allegro-mm/API/stats/modules/ip/ips/10.54.0.254?starttime=$(date --date="1 day ago" +%s)&endtime=$(date +%s)&skiphistorydata=true" | jq '.interval[1] + .interval[3]'
</pre>


* sort: Sorting criteria for the list. Following criterias are supported for most lists:
=== List queries ===


– bytes: Received and transmitted bytes (either in selected time interval or since start of the Multimeter)
List queries are requested with pagination parameters to reduce the size of the resulting JSON objects and to increase performance. In the resulting JSON object, the list elements are stored in the displayedItems array.
– rxbytes: Received bytes
The following list parameters are possible:
– txbytes: Transmitted bytes
– bps: Received and transmitted bytes per second (either average per second value of the selected time interval or last second, if no interval is specified)
– rxbps: Received bytes per second
– txbps: Transmitted bytes per second


* reverse: Sort ascending (= false) or descending (= true)
* sort: Sorting criteria for the list. Following criteria's are supported for most lists:
** bytes: Received and transmitted bytes (either in selected time interval or since start of the Allegro Network Multimeter).
** rxbytes: Received bytes.
** txbytes: Transmitted bytes.
** bps: Received and transmitted bytes per second (either average per second value of the selected time interval or last second, if no interval is specified).
** rxbps: Received bytes per second.
** txbps: Transmitted bytes per second.
* reverse: Sort ascending (= false) or descending (= true).
* page: Requested page.
* page: Requested page.
* count: Amount of entries in the list. Maximum is 100.
* count: Amount of entries in the list. Maximum is 100.
* values: Amount of maximal values in history object
* values: Amount of maximal values in history object(s).
 
This example shows IP address with the highest amount of traffic
 
<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips_paged?sort=bps&reverse=true&page=0&count=1' | jq .displayedItems[0].ip</pre>
 
This example shows up to 9999 peers of a specific IP address:
 
<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3/peers?sort=bytes&reverse=true&page=0&count=9999&timespan=60&values=100' | jq '.displayedItems[].ip'</pre>
 
=== Pcap extraction ===
 
The Allegro Network Multimeter allows to extract the raw packets with the REST API with the special capture URI <code>/API/data/modules/capture</code>
 
<pre>curl -k -u USER:PASSWORD 'https://allegro-mm/API/data/modules/capture?expression=ip==10.1.2.3' > path_to/capture.pcap</pre>
 
The available parameters are:
 
* '''startTime''': The start time of the capture. The first packet with exactly this or a later time will start the capture. The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch). If the start time is in the past, make sure you set fromCaptureBuffer parameter accordingly. If not specified, the current time will be used.
* '''endTime''': The end time of the capture. The first packet with exactly this or a later time will stop the capture. The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch). If not specified,  unlimited will be used. The end time can also be set to 'now', in this case the timespan parameter will be taken and the corresponding start time will be calculated.
* '''timespan''': The time span in seconds. Will be used if no startTime but endTime with 'now' is set.
* '''expression''': The filter expression. There are no whitespaces allowed. You may use ‘%20’ instead. See [[Capture module]] for available expressions.
* '''snapPacketLength''': The maximum size of a packet applied on Layer 2 without frame check sequence. If a packet is larger than this value, it is truncated. Use 65535 for unlimited size.
* '''fromCaptureBuffer''': Whether to extract data from the packet ring buffer (= true) or just live traffic (= false).
* '''captureBufferSlotId''': In case a cluster packet ring buffer is used, the id of the ring buffer must be given. The id of the first ring buffer is 0. If this parameter is omitted, 0 will be taken as default value.
* '''captureToMedia''': Whether to store a pcap on an external storage device (= true) or download to your computer (= false).
* '''mm-id''': If you are extracting a Pcap from a parallel Pcap analysis job or a multi device connected Allegro Network Multimeter, you have to specify the device and the slot where to get the data from. The syntax is: <code>mm-id=<device name>:<slot id></code>. If the capture shall be performed on the local device, the device name can be omited (e.g. mm-id=:1 for the first replay slot on the local device).
 
Example to capture everything from now on:
 
<pre>curl -k -u USER:PASSWORD 'https://allegro-mm/API/data/modules/capture' > path_to/capture.pcap</pre>
 
Example to capture a specific IP of the last hour
 
<pre>curl -k -u USER:PASSWORD "https://allegro-mm/API/data/modules/capture?expression=ip==10.1.2.3&starttime=$(($(date --date="1 hour ago" +%s%N)/1000))&endtime=$(($(date +%s%N)/1000))&fromCaptureBuffer=true" > path_to/capture.pcap</pre>
 
Example to capture a specific IP of a given time span of the first parallel Pcap analysis slot
 
<pre>curl -k -u USER:PASSWORD "https://allegro-mm/API/data/modules/capture?expression=ip==10.1.2.3&starttime=$(($(date --date="2020-07-15 08:55:00" +%s%N)/1000))&endtime=$(($(date --date="2020-07-15 09:55:00" +%s%N)/1000))&fromCaptureBuffer=true&mm-id=:1" > path_to/capture.pcap</pre>
 
=== Pcap upload and analysis ===
 
Pcap upload and analysis can also be done via API calls.
 
The PCAP upload is split into 3 commands. First, the replay is being initialized. Then the analyzing is started. In the last step the PCAP is streamed to the Allegro Network Multimeter. Depending on the size of the PCAP the third step could take some time, but the Allegro Network Multimeter already allows access to the statistics of the already analyzed packets via web/API.
 
The example assumes that PCAP parallel analysis is enabled in the global settings, so the PCAP analysis will be done in slot 1 and a dedicated ring buffer is available.
 
<pre>
curl -k -u USER:PASSWORD "https://allegro-mm/API/system/replay/upload" --header "Content-Type: application/json" -d '{"fileName": "abc.pcapng", "fileSize": '$(stat -c %s abc.pcapng)', "replaySlotID": 1, "forceSlotID": true, "useReplayBuffer": true}'
curl -k -u USER:PASSWORD "https://allegro-mm/API/data/pcap?mm-id=:1" --header "Content-Type: application/json" -d '{"command":"start"}'
curl -F 'name=file' -F 'filename=abc.pcapng' -F 'file=@path_to/abc.pcapng' -k -u USER:PASSWORD "https://allegro-mm/API/system/analyze-pcap?replaySlotID=1"
</pre>
 
=== Virtual Link Groups ===
 
The Allegro Network Multimeter REST API allows to access all link groups by the parameter '''group'''. The group index starts at zero, which is the default value. If a virtual link group is enabled.
 
This example extracts the traffic of the IP 10.54.0.254 from the second virtual link group ( index 1 ).
 
<pre>
$ curl --silent -k -u USER:PASSWORD "https://allegro-mm/API/stats/modules/ip/ips/10.54.0.254?group=1" | jq '.interval[1] + .interval[3]'
</pre>
 
=== Parallel pcap analysis ===
 
The Allegro Network Multimeter can process in parallel offline traffic like a pcap file or a ring buffer. In case a parallel PCAP analysis is running, the API call must be given either the additional header field <code>"X-AllegroPackets-Multimeter-ID: :1"</code> or the parameter mm-id with the PCAP instance ID.
 
This allows to extract information of automated pcap uploads.
 
=== Multi-device analysis ===
 
If the Allegro Network Multimeter is configured as a gateway for multiple Allegro devices by the [[Multi-device settings]], you need to add either the additional header field <code>"X-AllegroPackets-Multimeter-ID: hostname"</code> or the parameter mm-id where the hostname must be the same as configured in the multi-device settings.
 
== REST API Examples ==
 
==== MAC statistics ====
 
Extract the packets per second statistic of the MAC broadcast address
 
<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/mac/macs/ff:ff:ff:ff:ff:ff'</pre>
 
==== IP statistics ====
 
<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3'</pre>
 
==== Pretty displaying JSON output with jq ====
 
<pre>curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3' | jq</pre>
 
==== Capture a specific IP ====
 
<pre>curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/data/modules/capture?expression=ip==10.1.2.3' > path_to/capture.pcap</pre>
 
==== Capture two IP addresses with ports on a specific Layer 4 protocol ====
 
<pre>curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/data/modules/capture?expression=IP==10.1.2.3:62887 and IP==10.1.2.100:548 and l4Protocol==TCP' > path_to/capture.pcap</pre>
 
==== Output IP Table as CSV file ====
<pre>curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips_paged?csv=true' > path_to/file.csv</pre>
 
==== Output Connection Table as CSV file ====
<pre>curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/globalConnections/csv?csv=true' > path_to/file.csv</pre>
 
==== Multi-device Capture Python Script Example ====
 
<pre>
#! /usr/bin/python3
 
""" This example script starts a parallel download-capture for each 'multi-device' of a given allegro packets multimeter. """
 
import requests
import threading
import datetime
import shutil
 
 
def start_capture_download(host: str, device: dict, duration: int):
    start = datetime.datetime.now()
    end = start + datetime.timedelta(seconds=duration)
    file = device["host"] + start.strftime("%m-%d-%Y_%H-%M-%S") + ".pcap"
    params = {
        "mm-id": device["id"],
        "endTime": int(end.timestamp() * 1000000),
    }
 
    with session.get(host + "/API/data/modules/capture", params=params, stream=True) as resp:
        with open(file, "wb") as fh:
            shutil.copyfileobj(resp.raw, fh)
 
 
host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
# session.verify = False  # disable ssl verification
 
with session.get(host + "/API/system/multidevice/devices") as resp:
    devices = resp.json()
 
active_devices = []
for device in devices:
    if device["active"]:
        active_devices.append(device)
 
threads = []
for device in active_devices:
    t = threading.Thread(target=start_capture_download, args=(host, device, 30))
    t.start()
    threads.append(t)
 
for t in threads:
    t.join()
</pre>
 
==== Python Script Example - Top IPs ====
 
<pre>
#! /usr/bin/python3
 
import requests
 
requests.packages.urllib3.disable_warnings()
 
host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification
 
params = {
    "sort": "bytes",
    "reverse": True,
    "page": 0,
    "count": 10,
    "timespan": 60,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}
 
with session.get(host + "/API/stats/modules/ip/ips_paged", params=params) as resp:
    ip_list = resp.json()
 
    for ip_entry in ip_list["displayedItems"]:
        bytes_rx = ip_entry["interval"][1] #meaning of index defined in history.rows
        bytes_tx = ip_entry["interval"][3]
        print(ip_entry["ip"] + ": " + str(bytes_rx + bytes_tx) + "B")
</pre>
 
==== Python Script Example - Top IPs pagination ====
 
<pre>
#! /usr/bin/python3
 
import requests
 
requests.packages.urllib3.disable_warnings()
 
host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification
 
params = {
    "sort": "bytes",
    "reverse": True,
    "page": 0,
    "count": 10,
    "timespan": 60,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}
 
with session.get(host + "/API/stats/modules/ip/ips_paged", params=params) as resp:
    ip_list = resp.json()
    number_of_items = ip_list["numberOfItems"]
    number_of_pages = ip_list["numberOfPages"]
    items_per_page = ip_list["itemsPerPage"]
 
for page in range(0, number_of_pages):
    params["page"] = page
    with session.get(host + "/API/stats/modules/ip/ips_paged", params=params) as resp:
        ip_list = resp.json()
 
        for ip_entry in ip_list["displayedItems"]:
            bytes_rx = ip_entry["interval"][1] #meaning of index defined in history.rows
            bytes_tx = ip_entry["interval"][3]
            print(ip_entry["ip"] + ": " + str(bytes_rx + bytes_tx) + "B")
</pre>
 
==== Python Script Example - Top IPs CSV download ====
 
<pre>
#! /usr/bin/python3
 
import requests
import shutil
 
requests.packages.urllib3.disable_warnings()
 
host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification
 
params = {
    "csv": True,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}
 
headers = {
  "Accept-Encoding": "" # for compression use "gzip"
}
 
output = "ip_list_out.csv"
 
with session.get(host + "/API/stats/modules/ip/ips_paged", params=params, headers=headers, stream=True) as resp:
    with open(output, "wb") as fh:
        shutil.copyfileobj(resp.raw, fh)
</pre>
 
==== Python Script Example - Retrieval of global connections and PCAP download of a certain connection ====
 
<pre>
#! /usr/bin/python3
 
import requests
import shutil
import time
import datetime
 
requests.packages.urllib3.disable_warnings()
 
host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification
 
params = {
    "sort": "bytes",
    "reverse": True,
    "mode": "rtpStats",
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}
 
# get all RTP connections, sorted by bytes
with session.get(host + "/API/stats/modules/ip/globalConnections", params=params) as resp:
    asyncID = resp.json()["asyncID"]
    asyncUUID = resp.json()["asyncUUID"]
    #print(resp.json())
 
finished = False
success = False
 
params = {
    "uuid": asyncUUID,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}
 
while not finished:
    with session.get(host + "/API/async/{}".format(asyncID), params=params) as resp:
        if (resp.status_code == 202):
            # request still pending
            time.sleep(1)
            continue;
        else:
            finished = True
            r = resp.json()
            if "errorCode" in r and r["errorCode"] == 0:
                asyncResult = r["asyncResult"]
                success = True
 
# get start and end time of second connection
if success and len(asyncResult["displayedItems"]) > 1:
    rtpConnection = asyncResult["displayedItems"][1]
    print("{}:{} <-> {}:{}".format(rtpConnection["clientIp"],
        rtpConnection["clientPort"],
        rtpConnection["serverIp"],
        rtpConnection["serverPort"]))
    print(rtpConnection["l4ProtocolShortName"] + ", " + rtpConnection["dpiProtocol"])
    start = datetime.datetime.fromtimestamp(rtpConnection["connectionStart"] / 1000)
    end = datetime.datetime.fromtimestamp(rtpConnection["lastActivity"] / 1000)
    print("start: " + start.strftime("%m-%d-%Y %H-%M-%S"))
    print("end: " + end.strftime("%m-%d-%Y %H-%M-%S"))
 
    # download PCAP of connection
    params = {
        "expression": "IP == {}:{} and IP == {}:{}".format(rtpConnection["clientIp"],
            rtpConnection["clientPort"],
            rtpConnection["serverIp"],
            rtpConnection["serverPort"]),
        "fromCaptureBuffer": True,
        "captureBufferSlotId": 0,
        "startTime": rtpConnection["connectionStart"] * 1000,
        "endTime": rtpConnection["lastActivity"] * 1000,
        "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
    }
 
    headers = {
      "Accept-Encoding": "" # for compression use "gzip"
    }
 
    output = "rtp_connection.pcapng"
 
    with session.get(host + "/API/data/modules/capture", params=params, headers=headers, stream=True) as resp:
        with open(output, "wb") as fh:
            shutil.copyfileobj(resp.raw, fh)
 
</pre>
 
 
==== Python Script Example - POST command to pause ring buffer ====
 
This script sends a HTTP POST command to suspend the ring buffer #0 (first ring buffer).
<pre>
#! /usr/bin/python3


import requests
import json


'''Show IP address with the highest amount of traffic'''
requests.packages.urllib3.disable_warnings()


{| class="wikitable"
host = "https://allegro-mm-xxxx"
|-
session = requests.Session()
|| curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips_paged?sort=bps&reverse=true&page=0&count=1' | jq .displayedItems[0].ip
session.auth = ("user", "password")
|}
session.verify = False  # disable ssl verification


PowerShell command:
data = {
    "command": "changeConfig",
    "config": { "isSuspended": True }
}


{| class="wikitable"
with session.post(f"{host}/API/data/modules/capture/buffer/0", data=json.dumps(data)) as resp:
|-
    print(resp.text)
| ((Invoke-RestMethod -Uri 'https://allegro-mm-XXXX/API/stats/modules/ip/ips_paged?sort=bytes&reverse=true&page=0&count=10&timespan=60&values=50' -Headers @{Authorization = ("Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f 'USER', 'PASSWORD'))))} -ContentType'application/json; charset=utf-8' -Method 'Get' -SkipCertificateCheck). displayedItems[0]).ip
</pre>
|}


==== Python Script Example - Server ports used ====


This script generates a CSV file with all IPs and their server ports used.
The output format is as follows:


'''Show all peers of a certain IP address'''
ip address,ip role,server port


{| class="wikitable"
The ip role is either asClient or asServer which indicates whether the IP address runs the server port (asServer) or contacts the server port (asClient).
|-
|| curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3/peers?sort=bytes&reverse=true&page=0&count=9999&timespan=60&values=100' | jq '.displayedItems[].ip'
|}


<pre>
#! /usr/bin/python3


'''Capture a certain IP'''
import requests
import csv
import sys


{| class="wikitable"
requests.packages.urllib3.disable_warnings()
|-
|curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/data/modules/capture?expression=ip==10.1.2.3' > path_to/capture.pcap
|}


# set host
host = "https://allegro-mm-xxxx"


'''Capture two IP addresses with ports on a certain layer 4 protocol'''
session = requests.Session()


# set credential
session.auth = ("admin", "allegro")


{| class="wikitable"
session.verify = False  # disable ssl verification
|-
|curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/data/modules/capture?expression=IP==10.1.2.3:62887 and IP==10.1.2.100:548 and l4Protocol==TCP' > path_to/capture.pcap
|}


csvoutput = csv.writer(sys.stdout)


'''PCAP parameters'''
# loop over list of IPs
with session.get(host + "/API/stats/modules/ip/ips") as resp:
    ip_list = resp.json()


Following parameters are possible:
    for ip in ip_list:
        # for each IP get peers_ports as CSV
        with session.get(host + f"/API/stats/modules/ip/ips/{ip}/peers_ports?csv=true", stream = True) as resp:
            if resp.status_code != 200:
                # IP not existing (anymore?)
                continue


* startTime: The start time of the capture. The first packet with exactly this or a later time will start the capture. The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch).
            ports = {}
If the start time is in the past, make sure you set fromCaptureBuffer parameter accordingly.


* endTime: The end time of the capture. The first packet with exactly this or a later time will stop the capture.
            # loop over all ports and accumulate
            for nr, line in enumerate(resp.iter_lines()):
                if nr == 0:
                    # skip header line
                    continue


The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch).
                line = line.decode()
                if line:
                    for row in csv.reader([line], delimiter=',', quotechar='"'):
                        if len(row) < 7:
                            continue
                        port=row[1]
                        if row[6] == "client":
                            ports.setdefault("asServer",{})[port] = 1
                        else:
                            ports.setdefault("asClient",{})[port] = 1


* expression: The filter expression. There are no whitespaces allowed. You may use ‘%20’ instead.
            # now loop over accumulated results and create CSV lines
* snapPacketLength: The max size of a packet applied on layer 2 without frame check sequence. If a packet is larger than this value, it is truncated. Use 65535 for unlimited size.
            for connection_type, type_ports in ports.items():
* fromCaptureBuffer: Whether to extract data from the packet ring buffer (= true) or just live traffic (= false).
                res = list( map( lambda e: (ip, connection_type, e), type_ports.keys() ) )
* captureToMedia: Whether to store PCAP on external storage device (= true) or download your computer (= false).
                csvoutput.writerows(res)
</pre>

Latest revision as of 06:55, 29 July 2024

This page describes how to access and use the REST API. It allows to post-process data with 3rd party systems. The Allegro Network Multimeter web interface is itself based on this REST API and all displayed statistics can be extracted from the Allegro Network Multimeter with this API.

General API Setup

REST API Interface

All Allegro Network Multimeter statistics are derived from HTTPS requests and provided as JSON objects. The requests are stateless, i.e. there are no prerequisites and there is no fixed sequence of requests necessary. Example requests related to a specific module and statistics can be seen in the web interface by opening the browser development console (Ctrl+Shift+I for Chrome and Firefox, F12 for Edge).

Here an example of the structured JSON data for the IP overview. This data has been extracted with the Google Chrome developer console while accessing the IP statistics page.

Rest api chrome console.png

Credentials

The credentials are the same as for the web interface. Users with the admin role are allowed to access all APIs per default. With version 4.1 the access can be configured/restricted in the role settings. A non-admin user has read access to most of the statistics. If you have enabled the pcap role, the capture URL is also possible for the API.

Starting with verison 4.2., it is possible to use API tokens instead of username/password. Permissions for API tokens can be configured when creating the token.

Allegro Packets recommends to set up a separate non-admin user with or without the pcap role for the REST API of only statistics shall be gathered. This will prevent to accidentally shut down or change any configuration by calling the REST API.

Useful shell commands and their parameters

Curl

Most examples are written for curl [1]. Curl is available as for many operating systems like Linux or Windows. Curl needs a few parameters for the access of the Allegro Network Multimeter:

The parameter -u allows you to set a user name and password for the request.

The parameter -k will allow self-signed certificates. This is not necessary if a proper trusted certificate is installed on the Allegro Network Multimeter.

The parameter -s or --silent mutes any debugging output.

The URL of the API call is the first argument. It is recommended to enclose the API call with the character ' to avoid replacing the argument ( unless you need to replace parts of it )

curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/...'

Please note that you might need to use curl.exe in windows.

Using API token

When using an API token, the authentication is added using parameter -H to add a special HTTP request header (with API_TOKEN as a placeholder for the real token):

curl --silent -k -H "Authorization: Bearer API_TOKEN" 'https://allegro-mm-XXX/...'

This can be further simplified by writing the header into a separate file, for example api-token-header.txt. This also has the benefit that the credentials are not visible to every user on the system in the process list.

Authorization: Bearer API_TOKEN

Now this file can be referenced by curl:

curl --silent -k -H @api-token-header.txt 'https://allegro-mm-XXX/...'

PowerShell

The Integrated Windows PowerShell can be used to access the REST API. This guide requires at least PowerShell v6.

The command to call a REST API is Invoke-RestMethod [2]. Invoke-RestMethod on the PowerShell needs a few parameters for the access of the Allegro Network Multimeter:

To set the user name for basic authorization, use the -Headers parameter:

-Headers @{Authorization = ("Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f 'USER', 'PASSWORD'))))}

When sending POST requests (or other requests with a body), the request content type must be specified (not necessary for requests with empty request body (GET, DELETE)):

-ContentType 'application/json'

To disable the certificate check, use:

-SkipCertificateCheck

The URL must be passed with the parameter -Uri, so the full command is:

Invoke-RestMethod -Uri 'https://allegro-mm-XXXX/...' -Headers @{Authorization = ("Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f 'USER', 'PASSWORD'))))} -Method 'Get' -SkipCertificateCheck

Using API token

When using an API token, the Authorization header changes (with API_TOKEN as a placeholder for the real token):

Invoke-RestMethod -Uri 'https://allegro-mm-XXXX/...' -Headers @{Authorization = ("Bearer API_TOKEN")} -Method 'Get' -SkipCertificateCheck

jq

jq ([3]) is a powerful tool to extract parameters from a json document. If called without parameters, jq formats the JSON output into a readable format with indenting and new lines. It also allows to select specific values and do basic operations like addition with this values. Please read the jq documentation for more information.

API hierarchy

Query available URIs with OPTIONS

The Allegro Network Multimeter REST API has the fixed URI API/stats for statistics. To see all possible subtrees of a specific request, use the OPTIONS request instead of GET. It can be set with the parameter -X OPTIONS for curl.

$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats' -X OPTIONS
{
  "subResources": [
    "modules",
    "reports",
    "incidentReporting",
    "time",
    "ringBufferReplay",
    "pcap",
    "reset",
    "interfacesError",
    "interfaces",
    "load",
    "processing"
  ]
}

This allows you to query for example for all modules that are available for a specific release of the Allegro Network Multimeter:

$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules' -X OPTIONS
{
  "subResources": [
    "pppoe",
    "lldp",
    "groups",
    "mpls",
    "opc_ua",
    "quality",
    "ipsec",
    "profinet",
    "multicast",
    "burst_analysis",
    "global_incidents",
    "ptp",
    "ntp",
    "icmp",
    "stp",
    "sip",
    "smb",
    "dpa",
    "l4_ports",
    "netbios",
    "crt",
    "dpi",
    "http",
    "ssl",
    "dns",
    "dhcp",
    "location",
    "qos",
    "ip",
    "mac_protocols",
    "vlan",
    "arp",
    "packet_size",
    "mac",
    "capture"
  ]
}

URI content parameters

Some modules allow to use a parameter as part of the URI like the IP or Mac address. The path API/stats/modules/ip/ips allows you to use an IP address as next uri element

$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips' -X OPTIONS
{
  "subResources": [
    "protocol",
    ":ip"
  ]
}

The path of an IP address shows all further available elements:

$ curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.0.54.254' -X OPTIONS
{
  "subResources": [
    "sip_request_responses",
    "peers_ports",
    "sip_responses",
    "sip_requests",
    "qos",
    "ports",
    "connections",
    "protocols",
    "macs",
    "peers",
    "tcpStats"
  ]
}

JSON output traffic counters

All counters are aggregated counters, either for the selected time interval or since the processing start of the Allegro Network Multimeter. Many traffic counters have 4 separate values. These traffic counters are represented as a JSON array with at least 4 lines. The structure is as follows:

  • line 1: received packets, extraction example: jq .lastSecond[0]
  • line 2: received bytes, extraction example: jq .lastSecond[1]
  • line 3: transmitted packets, extraction example: jq .lastSecond[2]
  • line 4: transmitted bytes, extraction example: jq .lastSecond[3]
  • additional lines are module specific

The following counters exist for many REST APIs like IP, MAC, l4 protocol, l7 protocol and many more:

  • interval: Values of the selected time interval. If no interval is specified, this is similar to lastSecond.
  • allTime: Values since start of the Allegro Network Multimeter.
  • lastSecond: Values of the last second.
  • intervalPerSecond: Average per second value of the selected time interval. If no interval is specified, this is similar to lastSecond.

Please note that all counters are byte counters, not bit counters. You need to multiply the counters by 8 to get the bitrate.

This example extracts the received bytes of the last second of a specific IP.

curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3' | jq .lastSecond[1]

This example extracts received and transmitted bytes of the last second of a specific IP.

curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3' | jq '.lastSecond[1] + .lastSecond[3]'

API parameters

The Allegro Network Multimeter REST API has a number of query parameters that can be added to modify the request. By default, the API will display the real time counters since the last restart of the processing unit.

This example extracts the amount of received and transmitted bytes for an IP address since the processing start of the Allegro Network Multimeter. It queries via the REST API the JSON and then adds the values second value ( row 1, rx bytes ) and 4th value ( row 3, tx bytes ) of the interval counters together.

$ curl --silent -k -u USER:PASSWORD "https://allegro-mm/API/stats/modules/ip/ips/10.54.0.254" | jq '.interval[1] + .interval[3]'

Time interval selection

Requests can be given a time interval. If present, the interval counters are adjusted to this interval. The following GET parameters are necessary:

  • starttime, endtime: Start and end time of the interval. Format: seconds since 1970/01/01 UTC (Unix time, epoch). You can use date +%s on your machine to adjust to the best interval. Please consult the man page of date for more parameters.
  • skiphistorydata: shall the JSON include the history data without datasets, this reduces the amount of transferred bytes if datasets are required to render a graph, can be false/true default: false
  • timespan: required resolution for the graph dataset

This example extracts the amount of received and transmitted bytes for an IP address for the last 24 hours.

$ curl --silent -k -u USER:PASSWORD "https://allegro-mm/API/stats/modules/ip/ips/10.54.0.254?starttime=$(date --date="1 day ago" +%s)&endtime=$(date +%s)&skiphistorydata=true" | jq '.interval[1] + .interval[3]'

List queries

List queries are requested with pagination parameters to reduce the size of the resulting JSON objects and to increase performance. In the resulting JSON object, the list elements are stored in the displayedItems array. The following list parameters are possible:

  • sort: Sorting criteria for the list. Following criteria's are supported for most lists:
    • bytes: Received and transmitted bytes (either in selected time interval or since start of the Allegro Network Multimeter).
    • rxbytes: Received bytes.
    • txbytes: Transmitted bytes.
    • bps: Received and transmitted bytes per second (either average per second value of the selected time interval or last second, if no interval is specified).
    • rxbps: Received bytes per second.
    • txbps: Transmitted bytes per second.
  • reverse: Sort ascending (= false) or descending (= true).
  • page: Requested page.
  • count: Amount of entries in the list. Maximum is 100.
  • values: Amount of maximal values in history object(s).

This example shows IP address with the highest amount of traffic

curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips_paged?sort=bps&reverse=true&page=0&count=1' | jq .displayedItems[0].ip

This example shows up to 9999 peers of a specific IP address:

curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3/peers?sort=bytes&reverse=true&page=0&count=9999&timespan=60&values=100' | jq '.displayedItems[].ip'

Pcap extraction

The Allegro Network Multimeter allows to extract the raw packets with the REST API with the special capture URI /API/data/modules/capture

curl -k -u USER:PASSWORD 'https://allegro-mm/API/data/modules/capture?expression=ip==10.1.2.3' > path_to/capture.pcap

The available parameters are:

  • startTime: The start time of the capture. The first packet with exactly this or a later time will start the capture. The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch). If the start time is in the past, make sure you set fromCaptureBuffer parameter accordingly. If not specified, the current time will be used.
  • endTime: The end time of the capture. The first packet with exactly this or a later time will stop the capture. The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch). If not specified, unlimited will be used. The end time can also be set to 'now', in this case the timespan parameter will be taken and the corresponding start time will be calculated.
  • timespan: The time span in seconds. Will be used if no startTime but endTime with 'now' is set.
  • expression: The filter expression. There are no whitespaces allowed. You may use ‘%20’ instead. See Capture module for available expressions.
  • snapPacketLength: The maximum size of a packet applied on Layer 2 without frame check sequence. If a packet is larger than this value, it is truncated. Use 65535 for unlimited size.
  • fromCaptureBuffer: Whether to extract data from the packet ring buffer (= true) or just live traffic (= false).
  • captureBufferSlotId: In case a cluster packet ring buffer is used, the id of the ring buffer must be given. The id of the first ring buffer is 0. If this parameter is omitted, 0 will be taken as default value.
  • captureToMedia: Whether to store a pcap on an external storage device (= true) or download to your computer (= false).
  • mm-id: If you are extracting a Pcap from a parallel Pcap analysis job or a multi device connected Allegro Network Multimeter, you have to specify the device and the slot where to get the data from. The syntax is: mm-id=<device name>:<slot id>. If the capture shall be performed on the local device, the device name can be omited (e.g. mm-id=:1 for the first replay slot on the local device).

Example to capture everything from now on:

curl -k -u USER:PASSWORD 'https://allegro-mm/API/data/modules/capture' > path_to/capture.pcap

Example to capture a specific IP of the last hour

curl -k -u USER:PASSWORD "https://allegro-mm/API/data/modules/capture?expression=ip==10.1.2.3&starttime=$(($(date --date="1 hour ago" +%s%N)/1000))&endtime=$(($(date +%s%N)/1000))&fromCaptureBuffer=true" > path_to/capture.pcap

Example to capture a specific IP of a given time span of the first parallel Pcap analysis slot

curl -k -u USER:PASSWORD "https://allegro-mm/API/data/modules/capture?expression=ip==10.1.2.3&starttime=$(($(date --date="2020-07-15 08:55:00" +%s%N)/1000))&endtime=$(($(date --date="2020-07-15 09:55:00" +%s%N)/1000))&fromCaptureBuffer=true&mm-id=:1" > path_to/capture.pcap

Pcap upload and analysis

Pcap upload and analysis can also be done via API calls.

The PCAP upload is split into 3 commands. First, the replay is being initialized. Then the analyzing is started. In the last step the PCAP is streamed to the Allegro Network Multimeter. Depending on the size of the PCAP the third step could take some time, but the Allegro Network Multimeter already allows access to the statistics of the already analyzed packets via web/API.

The example assumes that PCAP parallel analysis is enabled in the global settings, so the PCAP analysis will be done in slot 1 and a dedicated ring buffer is available.

curl -k -u USER:PASSWORD "https://allegro-mm/API/system/replay/upload" --header "Content-Type: application/json" -d '{"fileName": "abc.pcapng", "fileSize": '$(stat -c %s abc.pcapng)', "replaySlotID": 1, "forceSlotID": true, "useReplayBuffer": true}'
curl -k -u USER:PASSWORD "https://allegro-mm/API/data/pcap?mm-id=:1" --header "Content-Type: application/json" -d '{"command":"start"}'
curl -F 'name=file' -F 'filename=abc.pcapng' -F 'file=@path_to/abc.pcapng' -k -u USER:PASSWORD "https://allegro-mm/API/system/analyze-pcap?replaySlotID=1"

Virtual Link Groups

The Allegro Network Multimeter REST API allows to access all link groups by the parameter group. The group index starts at zero, which is the default value. If a virtual link group is enabled.

This example extracts the traffic of the IP 10.54.0.254 from the second virtual link group ( index 1 ).

$ curl --silent -k -u USER:PASSWORD "https://allegro-mm/API/stats/modules/ip/ips/10.54.0.254?group=1" | jq '.interval[1] + .interval[3]'

Parallel pcap analysis

The Allegro Network Multimeter can process in parallel offline traffic like a pcap file or a ring buffer. In case a parallel PCAP analysis is running, the API call must be given either the additional header field "X-AllegroPackets-Multimeter-ID: :1" or the parameter mm-id with the PCAP instance ID.

This allows to extract information of automated pcap uploads.

Multi-device analysis

If the Allegro Network Multimeter is configured as a gateway for multiple Allegro devices by the Multi-device settings, you need to add either the additional header field "X-AllegroPackets-Multimeter-ID: hostname" or the parameter mm-id where the hostname must be the same as configured in the multi-device settings.

REST API Examples

MAC statistics

Extract the packets per second statistic of the MAC broadcast address

curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/mac/macs/ff:ff:ff:ff:ff:ff'

IP statistics

curl --silent -k -u USER:PASSWORD 'https://allegro-mm/API/stats/modules/ip/ips/10.1.2.3'

Pretty displaying JSON output with jq

curl --silent -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips/10.1.2.3' | jq

Capture a specific IP

curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/data/modules/capture?expression=ip==10.1.2.3' > path_to/capture.pcap

Capture two IP addresses with ports on a specific Layer 4 protocol

curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/data/modules/capture?expression=IP==10.1.2.3:62887 and IP==10.1.2.100:548 and l4Protocol==TCP' > path_to/capture.pcap

Output IP Table as CSV file

curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/ips_paged?csv=true' > path_to/file.csv

Output Connection Table as CSV file

curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/stats/modules/ip/globalConnections/csv?csv=true' > path_to/file.csv

Multi-device Capture Python Script Example

#! /usr/bin/python3

""" This example script starts a parallel download-capture for each 'multi-device' of a given allegro packets multimeter. """

import requests
import threading
import datetime
import shutil


def start_capture_download(host: str, device: dict, duration: int):
    start = datetime.datetime.now()
    end = start + datetime.timedelta(seconds=duration)
    file = device["host"] + start.strftime("%m-%d-%Y_%H-%M-%S") + ".pcap"
    params = {
        "mm-id": device["id"],
        "endTime": int(end.timestamp() * 1000000),
    }

    with session.get(host + "/API/data/modules/capture", params=params, stream=True) as resp:
        with open(file, "wb") as fh:
            shutil.copyfileobj(resp.raw, fh)


host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
# session.verify = False  # disable ssl verification

with session.get(host + "/API/system/multidevice/devices") as resp:
    devices = resp.json()

active_devices = []
for device in devices:
    if device["active"]:
        active_devices.append(device)

threads = []
for device in active_devices:
    t = threading.Thread(target=start_capture_download, args=(host, device, 30))
    t.start()
    threads.append(t)

for t in threads:
    t.join()

Python Script Example - Top IPs

#! /usr/bin/python3

import requests

requests.packages.urllib3.disable_warnings()

host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification

params = {
    "sort": "bytes",
    "reverse": True,
    "page": 0,
    "count": 10,
    "timespan": 60,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}

with session.get(host + "/API/stats/modules/ip/ips_paged", params=params) as resp:
    ip_list = resp.json()

    for ip_entry in ip_list["displayedItems"]:
        bytes_rx = ip_entry["interval"][1] #meaning of index defined in history.rows
        bytes_tx = ip_entry["interval"][3]
        print(ip_entry["ip"] + ": " + str(bytes_rx + bytes_tx) + "B")

Python Script Example - Top IPs pagination

#! /usr/bin/python3

import requests

requests.packages.urllib3.disable_warnings()

host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification

params = {
    "sort": "bytes",
    "reverse": True,
    "page": 0,
    "count": 10,
    "timespan": 60,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}

with session.get(host + "/API/stats/modules/ip/ips_paged", params=params) as resp:
    ip_list = resp.json()
    number_of_items = ip_list["numberOfItems"]
    number_of_pages = ip_list["numberOfPages"]
    items_per_page = ip_list["itemsPerPage"]

for page in range(0, number_of_pages):
    params["page"] = page
    with session.get(host + "/API/stats/modules/ip/ips_paged", params=params) as resp:
        ip_list = resp.json()

        for ip_entry in ip_list["displayedItems"]:
            bytes_rx = ip_entry["interval"][1] #meaning of index defined in history.rows
            bytes_tx = ip_entry["interval"][3]
            print(ip_entry["ip"] + ": " + str(bytes_rx + bytes_tx) + "B")

Python Script Example - Top IPs CSV download

#! /usr/bin/python3

import requests
import shutil

requests.packages.urllib3.disable_warnings()

host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification

params = {
    "csv": True,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}

headers = {
  "Accept-Encoding": "" # for compression use "gzip"
}

output = "ip_list_out.csv"

with session.get(host + "/API/stats/modules/ip/ips_paged", params=params, headers=headers, stream=True) as resp:
    with open(output, "wb") as fh:
        shutil.copyfileobj(resp.raw, fh)

Python Script Example - Retrieval of global connections and PCAP download of a certain connection

#! /usr/bin/python3

import requests
import shutil
import time
import datetime

requests.packages.urllib3.disable_warnings()

host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification

params = {
    "sort": "bytes",
    "reverse": True,
    "mode": "rtpStats",
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}

# get all RTP connections, sorted by bytes
with session.get(host + "/API/stats/modules/ip/globalConnections", params=params) as resp:
    asyncID = resp.json()["asyncID"]
    asyncUUID = resp.json()["asyncUUID"]
    #print(resp.json())

finished = False
success = False

params = {
    "uuid": asyncUUID,
    "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
}

while not finished:
    with session.get(host + "/API/async/{}".format(asyncID), params=params) as resp:
        if (resp.status_code == 202):
            # request still pending
            time.sleep(1)
            continue;
        else:
            finished = True
            r = resp.json()
            if "errorCode" in r and r["errorCode"] == 0:
                asyncResult = r["asyncResult"]
                success = True

# get start and end time of second connection
if success and len(asyncResult["displayedItems"]) > 1:
    rtpConnection = asyncResult["displayedItems"][1]
    print("{}:{} <-> {}:{}".format(rtpConnection["clientIp"],
        rtpConnection["clientPort"],
        rtpConnection["serverIp"],
        rtpConnection["serverPort"]))
    print(rtpConnection["l4ProtocolShortName"] + ", " + rtpConnection["dpiProtocol"])
    start = datetime.datetime.fromtimestamp(rtpConnection["connectionStart"] / 1000)
    end = datetime.datetime.fromtimestamp(rtpConnection["lastActivity"] / 1000)
    print("start: " + start.strftime("%m-%d-%Y %H-%M-%S"))
    print("end: " + end.strftime("%m-%d-%Y %H-%M-%S"))

    # download PCAP of connection
    params = {
        "expression": "IP == {}:{} and IP == {}:{}".format(rtpConnection["clientIp"],
            rtpConnection["clientPort"],
            rtpConnection["serverIp"],
            rtpConnection["serverPort"]),
        "fromCaptureBuffer": True,
        "captureBufferSlotId": 0,
        "startTime": rtpConnection["connectionStart"] * 1000,
        "endTime": rtpConnection["lastActivity"] * 1000,
        "mm-id": "local:1" #0 live traffic, 1 1st PCAP analysis
    }

    headers = {
      "Accept-Encoding": "" # for compression use "gzip"
    }

    output = "rtp_connection.pcapng"

    with session.get(host + "/API/data/modules/capture", params=params, headers=headers, stream=True) as resp:
        with open(output, "wb") as fh:
            shutil.copyfileobj(resp.raw, fh)


Python Script Example - POST command to pause ring buffer

This script sends a HTTP POST command to suspend the ring buffer #0 (first ring buffer).

#! /usr/bin/python3

import requests
import json

requests.packages.urllib3.disable_warnings()

host = "https://allegro-mm-xxxx"
session = requests.Session()
session.auth = ("user", "password")
session.verify = False  # disable ssl verification

data = {
    "command": "changeConfig",
    "config": { "isSuspended": True }
}

with session.post(f"{host}/API/data/modules/capture/buffer/0", data=json.dumps(data)) as resp:
    print(resp.text)

Python Script Example - Server ports used

This script generates a CSV file with all IPs and their server ports used. The output format is as follows:

ip address,ip role,server port

The ip role is either asClient or asServer which indicates whether the IP address runs the server port (asServer) or contacts the server port (asClient).

#! /usr/bin/python3

import requests
import csv
import sys

requests.packages.urllib3.disable_warnings()

# set host
host = "https://allegro-mm-xxxx"

session = requests.Session()

# set credential
session.auth = ("admin", "allegro")

session.verify = False  # disable ssl verification

csvoutput = csv.writer(sys.stdout)

# loop over list of IPs
with session.get(host + "/API/stats/modules/ip/ips") as resp:
    ip_list = resp.json()

    for ip in ip_list:
        # for each IP get peers_ports as CSV
        with session.get(host + f"/API/stats/modules/ip/ips/{ip}/peers_ports?csv=true", stream = True) as resp:
            if resp.status_code != 200:
                # IP not existing (anymore?)
                continue

            ports = {}

            # loop over all ports and accumulate
            for nr, line in enumerate(resp.iter_lines()):
                if nr == 0:
                    # skip header line
                    continue

                line = line.decode()
                if line:
                    for row in csv.reader([line], delimiter=',', quotechar='"'):
                        if len(row) < 7:
                            continue
                        port=row[1]
                        if row[6] == "client":
                            ports.setdefault("asServer",{})[port] = 1
                        else:
                            ports.setdefault("asClient",{})[port] = 1

            # now loop over accumulated results and create CSV lines
            for connection_type, type_ports in ports.items():
                res = list( map( lambda e: (ip, connection_type, e), type_ports.keys() ) )
                csvoutput.writerows(res)