Capture module: Difference between revisions

From Allegro Network Multimeter Manual
Jump to navigation Jump to search
Access restrictions were established for this page. If you see this message, you have no access to this page.
(added Planned captures)
Line 103: Line 103:
|}
|}


* '''l4Protocol''': A layer 4 protocol such as TCP or UDP. Protocols can also be OR combined as a comma seperated list.  
* '''l4Protocol''': A layer 4 protocol such as TCP or UDP or any IP protocol number. Protocols can also be OR combined as a comma seperated list.  
:Example:
:Example:
:{| class="wikitable sortable"   
:{| class="wikitable sortable"   

Revision as of 13:23, 7 July 2021

Capture module

The Network Multimeter allows direct capturing of network traffic as a HTTP download to your computer. No packet data is stored on the device itself. Traffic can be directly filtered for specific packets, only the relevant packets will be captured. In addition, it is also possible to capture network traffic to an attached storage device, see the settings section below for details. Capturing network traffic is usually started by clicking on a PCAP button in a certain module. These buttons allow capturing specific traffic, for example for an certain IP address or a network protocol. The capture module allows to configure filter for traffic that has not even started right now, for example for an IP address that is not in use at the moment but later might be used. The capture module page displays all currently running captures and allows starting new captures with specific filters.

Generic modules.png

Current captures

The first part of the page displays all downloads running for the current user session, and all downloads running for other user sessions (like when a download has been started outside the browser by directly using command line tools such as wget or curl). The list contains the client IP and port of the user running the download. The next three counters describe the number of packets captured for the corresponding filter, the number of packets dropped by the capturing module, and the number of ignored packets. Packet drops happen when more packets are captured than can be transferred via HTTP to the client. Ignored packets do not match the given capture filter. The following columns list the applied filter criteria. The last column contains a button to stop the corresponding download. Downloads can also be stopped by clicking the same capture button that started the capture in the corresponding module. If multiple devices have been configured, the list also contains all captures from all multi-devices which can be stopped individually.

Recently captured

This list shows the most recently performed captures for the current user. The most recent capture is displayed on the top. Next to each capture there is a button to permanently save this capture as a favorite as well as a button to simply start this capture again. The Use as expert filter button will copy the capture filter into the expert filter input field and allows for customizing the capture. The button Delete list of recent captures will delete all entries from this list.

Favorites

This list shows favorite capture expressions. A capture can be marked as a favorite either in the capture dialog by clicking on the star button in the top right corner or by marking it as a favorite in the Recently captured list. A description can be given and will be displayed in this list. For each favorite capture a PCAP button is available to simply start this capture again. The Remove favorites button allows for cleaning the list. The Use as expert filter button will copy the capture filter into the expert filter input field and allows for customizing the capture.

Planned captures

On this tab captures to the storage device can be planned ahead of time and these captures can even be set to repeat automatically. A click on the Add planned capture button opens a dialog where the planned capture can be configured. This includes settings like the start date and time of the capture, the duration of the capture, if a capture should repeat and the filter expression used during the capture. It is important to know that planned captures are always stored on the active storage device and thus will not function if no storage device is active.

Capture profiles

Capture profile configuration
Edit capture profile

Capture profiles can be used in the capture dialog for custom packet truncation rules. Such profiles defines custom rules for packet snapshot length similar to ring buffer filters. This allows to use a different snapshot length for specific layer 7 protocols, if for example traffic of one protocol shall be captured completely while for another protocol only the IP header shall be captured. Such a capture profile can be selected in the capture dialog in the "Truncate packet length" select box.

Up to 10 different profiles can be configured by clicking at the "Add profile" button. The add/edit dialog allows to enter a profile name and up to 10 rules for snapshot length. Similar to ring buffer filters, the first rule that matches for the selected type is used to decide for the snapshot length of the actual packet.

Simple capture

The second section of the capture page allow to select some fields to filter network traffic for. By default, only the IP field is visible, the other fields can be enabled by clicking on the corresponding toggle switch. Each line allows to enter a filter criterion for the corresponding network traffic element. To start the capture with the entered filter criteria just click at the Start capture button. For reference, the expert filter expression is shown at the end of the section so it can be used to copy and paste the string into the expert filter section.

Using expert filters to start captures

The third part of the page allows for starting a download for any criterion combination using complex filter expressions. A capture filter is defined in a C-style syntax and supports combination of AND/OR operators, precedence order with parentheses and equal/not equal comparisons. If the filter exp Session can be evaluated to true, the packet is captured. If a value contains a space, the whole value must be quoted with “”. Following operators are supported:

  • and, && : AND operator. The filter expression will match if all operands could be evaluated to true.
  • or, ||: OR operator. The filter expression will match if any operand can be evaluated to true.

Following comparison operators are supported:

  • ==: Will evaluate expression to true if left and right operand are equal.
  • !=: Will evaluate expression to true if left and right operand are not equal.

Following operands are supported:

  • ip: An IP address. The packet is captured if either source or destination IP address of the packet match. A netmask and a port can also be specified. For IPv6 addresses with a specific port, the address must be written in brackets.
Example:

ip == 10.0.0.1

ip == ff02::1:3

ip == 10.0/16

ip == 10.0.0.1:1234

ip == [2a02:810a:1340:1292:1c6b:e58d:6ebc:6cd2]:123

  • mac: A MAC address. The packet is captured if either source or destination MAC address of the packet match.
Example:
mac == 12:34: :56:78:90:ab
  • port: A TCP or UDP port. The packet is captured if either source or destination port match.
Example:
port == 80
  • portrange: A TCP or UDP port range. The range can be a single number or a comma separated list of values or value ranges.
Example:
portrange == 80,100-120,-10,65000-
  • serverport: A TCP or UDP port of a server. The packet is captured if the given port is a port of the server and not of a client.
Example:
serverport =: = 80
  • macProtocol: A MAC protocol such as IPv4 or IPv6. For all seen MAC protocols, please consult the MAC Protocol Statistics module.
Example:
macProtocol == IPv4

macProtocol == "Non IP"

  • l4Protocol: A layer 4 protocol such as TCP or UDP or any IP protocol number. Protocols can also be OR combined as a comma seperated list.
Example:
l4Protocol == ICMP,ICMPv6
  • l7Protocol or dpiProtocol: A layer 7 protocol. Protocols can also be OR combined as a comma seperated list.For all seen protocols please consult the Layer 7 protocols module.
  • countryCode: A country code such as US. For all seen country codes please consult the Geolocation module.
  • arpip: An IP address within an ARP request or response.
  • vlan: A VLAN tag of an outer or inner VLAN. May be a number or none or any.
  • outervlan: A VLAN tag of an outer VLAN. May be a number or none or any.
  • innervlan: A VLAN tag of an inner VLAN. May be a number or none or any.
  • multicastGroup: A multicast IP address or any. The filter will match all IGMP or MLD negotiation packets related to that multicast IP address.
  • rtpPayloadType: The RTP payload type such as PCMU or MP2T. This filter will match all RTP packets with the given payload type.
  • interface: The physical interface. This can be a single number or a range. For interface ids please consult the Interface stats page.
Example:
interface == 1,3-5
  • link: The link pair of two interfaces as stated in Interface stats. A single link number can be given.
  • ptpMsgType: A specific PTP message type number or any for the whole PTP traffic.
  • profinetFrameId: A specific Profinet frame ID.
  • profinetCmOpnum: A specific operation number for Profinet CM (Context Manager) requests or responses. Can also be any for every operation number. Following values are used:
  1. connect
  2. release
  3. read
  4. write
  5. control
  6. read implicit
  • mpls: A label of an outer or inner MPLS. May be a number or none or any.
  • outerMpls: A label of an outer MPLS. May be a number or none or any.
  • innerMpls: A label of an inner MPLS. May be a number or none or any.
  • qosIpDscp: The DSCP value in the IP header. May be a number.
  • qosMplsTc: The traffic class value in the outermost MPLS label stack entry.
  • qosVlanPcp: The priority code point value in the outermost VLAN tag.
  • group: The name of a configured group or ‘default’. If the name contains whitespaces, the name must be enclosed in quotes.
  • badCRC: The value of this operand will be 1 for packets with a CRC error and will be 0 for good packets. Capturing packets with bad CRC is currently only supported on 1Gb interfaces.
  • icmpType: The value of a certain ICMP type (e.g. Echo request 8, Echo reply 0).
  • tcpFlags: A single TCP flag or a list of TCP flags joined by the ‘+’ sign. If a list of flags is given, all flags must be present in the packet. Supported TCP flags are syn, ack, fin, rst, psh and urg.
  • callId: The string value of a SIP call ID or similar identifier (e.g. P-Palladion-ID)
  • ipFragment: If set to 1 all IPv4 fragments will be captured (i.e. packets having the 'More fragments' flag and 'Fragment offset' set). If set to 0 all packets without IPv4 fragmentation will be captured.
  • regexp: The packet payload matches the quoted regular expression (RegEx) to the other side of the == operator or does not match the regular expression to the other side of the != operator. In case of IP packets the matching will be performed on the L7 payload of the packet. In case of non-IP packets the matching will be performed on the whole packet except the Ethernet header. Regular expressions largely support the pattern syntax used by the PCRE library with the exception of certain constructs. An invalid pattern will produce a descriptive error message and prevent the capture from being started.

For a specific precedence you may use parentheses (/).

Examples:

  • The expression
ip == 10.0.0.1:1234 and ip == 10.1.0.1:9876
will match a connection from 10.0.0.1 to 10.1.0.1 or vice versa with the ports 1234 and 9876 involved.
  • The expression
ip == 10.0.0.1 and ip != 10.0.0.2
will match packets having 10.0.0.1 either as source or destination. If a communication peer of 10.0.0.1 is 10.0.0.2 the packets will not be captured.
  • The expression
l4Protocol == ICMP,ICMPv6
will match packets with ICMP or ICMPv6 layer 4 protocols.
  • The expression
portrange == 80,443
will match packets to or from port 80 or 443.
  • The expression
regexp == "allegr[o,a]|HTTP"
will case sensitive match packets that contain the string(s) 'allegro' and/or 'allegra' and/or 'HTTP' anywhere in the payload.
NOTE: The use of regexp is CASE sensitive. You must use the (?i) modifier to enable case insensitive filtering.
  • The expression
regexp == "(?i)allegro|http"
will case insensitive match packets that contain the string(s) 'allegro' and/or 'http' anywhere in the payload.
NOTE: The use of regexp is CASE sensitive. You must use the (?i) modifier to enable case insensitive filtering.

Of course the Allegro Network Multimeter regular expression (RegEx) capture filter, can also be used in combination with our other capture expressions.

  • The expression
regexp == “allegro|analyzer” and l7protocol == "dns"
Will case sensitive match and capture only DNS packets containing the string(s) “allegro” and/or “analyzer.
  • The expression
regexp == “allegro|analyzer” and l7protocol != "dns"
Will case sensitive match and capture all (except DNS) packets containing the string(s) “allegro” and/or “analyzer.

Whenever you are unsure about the outcome of RegEx based packet capturing, you can pre-test the outcome of your expressions on https://pythex.org/. While pre-testing on https://pythex.org/, avoid using the “IGNORECASE” button. Instead use the (?i) modifier for constructing case insensitive expressions, as mentioned above. PCRE/Python based expression examples and explanations you'll find on https://www.programiz.com/python-programming/regex


All captures can be limited to any amount of time or bytes, for example to capture only one minute or one megabyte of traffic.

Below the list of filter criteria, you will find the button to actually start (or stop) the capture. In case the filter expression is invalid, the button is disabled.

Layer 7 protocol capture

Layer 7 protocol detection engine may need several packets to recognize the currently used protocol. For these captures all not yet recognized packets will be skipped. As soon as the protocol recognition is finished, all packets matching the protocol filter will be captured.

Configuration settings

By clicking on the gear button on the top right of the Capture web page, you can access the configuration section.

  • Split PCAP file after this size
This option can be used to limit the size of the PCAP file when storing to an attached device. Once the captured traffic would exceed this threshold, a new PCAP file with the current time stamp is created and the traffic is written to the new file. If the time stamp is still the same, an index is attached to the filename.
When set to 0, no splitting will be done.
  • Split PCAP file after this duration
This option can be used to limit the duration of the PCAP file when storing to an attached device. The duration starts counting with the start of the capture. Once the captured traffic would exceed the duration, a new PCAP file with the current time stamp is created and the traffic is written to the new file.
When set to 0, no splitting will be done.
Both split parameters can be combined. The PCAP file will be split as soon as one threshold has been reached.
  • The maximum number of concurrent packet ring buffers
This option is used to specify how many cluster packet ring buffers can run in parallel.
Be aware that each cluster will have it's own queue in memory and therefore the memory required is the number of cluster packet ring buffers multiplied by the setting of The size in MB for the queue of the packet ring buffer.
A reboot of the device or a restart of the processing is needed for a change to this option to take effect.
  • The size in MB for the queue of the packet ring buffer
This option allows to configure the size of the queue that holds processed packets before they are written to the packet ring buffer. Increasing the size of this queue may help if the disk used for the packet ring buffer cannot keep up with bursts of traffic so that packet drops occur in the packet ring buffer.
Be aware that memory allocated to this queue is not available for storing statistics and metadata so that choosing a large value for this queue reduces the overall data storage time.
Most users will not need to change this value from the default value.
A reboot of the device or a restart of the processing is needed for a change to this option to take effect.
  • The maximum size in MB for the packet reorder buffer when capturing from the packet ring buffer
This setting allows to choose the maximum size that the packet reorder buffer may grow to. For performance reasons the packet ring buffer does not ensure a total order of packets when storing them on disk. The packet reorder buffer is used to restore the correct order of packets in a capture when capturing from the packet ring buffer. A larger packet reorder buffer makes it more likely that the packet order can be restored for all packets. The actual amount of memory used for the packet reorder buffer depends on this setting but also on the amount of free memory in the system so that the effectively used amount of memory may be less than this setting indicates.

Capture settings dialog

Choose capture settings.png

This dialog appears after a capture button has been clicked. Following settings are possible:

  • Start time and end time
By clicking on the input field or on the calendar icon you can choose the start and end time of the capture. The input field is also editable with keyboard and allows entering a time on a second basis.
If the start time is in the past, the complete capture is performed on the stored data of the capture ring buffer. When the capture reaches the newest packets it still continues to read from the capture ring buffer. The dialog will limit the start time input to the earliest data of the capture ring buffer. Be aware, that a possible capture ring buffer filter was applied on the past data and is also applied on future data in this mode.
The start time may also be in the future. The capture is scheduled and starts as soon as a packet is received with a time later than the start time.
If the whole time input field is marked and deleted, the start or end time will reset back to the default value. The default value for start time is now, the capture will start with pushing the Start capturing button. The default value of the end time is unlimited, the capture will not stop unless stopped manually by clicking on the stop button.
Eight buttons offer quick selection of often used time settings.
  • Packet ring buffer
If multiple packet ring buffer clusters are active this dropdown menu allows to choose from which cluster the packets should be captured.
  • Capture type
This drop down menu allows to choose the method how packets are captured. The last successful setting is persistently stored per user. Following methods are available:
  • HTTP download
This is the default method. The capture will start a HTTP download of a PCAP file directly in the browser.
Available HTTP download options:
  • Set file name: allow to configure a custom file name for the capture file
  • Download as zip archive: Download the capture file as a compressed zip archive [New in version 3.0]
  • Disk
This method is only visible if a storage device is active and has some amount of free storage space. The capture will create a PCAP file on the storage device.
If PCAP export via SFTP is enabled, an additional checkbox is displayed to store the capture file in the export directory, slated for upload according the SFTP export settings.
  • Interface
This mode will transmit the captured packets on a physical network interface. It is not available when the system is analyzing a PCAP file or is analyzing the packet ring buffer.
  • ERSPAN
This mode will transmit the captured packets encapsulated in a GRE + ERSPAN header on the management interface to a given target IP address. On the target system the traffic can be selectively captured using the filter ip proto 0x2f when using an application like Wireshark or tcpdump.
  • File name
If browser download or disk storage has been chosen, the file name of the created capture file can be set in this field. The default value shows the filename with date wildcards and the capture filter. The date wildcards are then replaced by the start time of the capture.
Date format specifier can be used as supported by strftime() function call. Some common specifier:
  • %Y for year
  • %m for month
  • %d for day
  • %H for hours
  • %M for minutes
  • %S for seconds
  • Storage directory
If disk storage has been chosen, the target directory on the storage device can be set in this field. Sub-directories on the storage device can be created and inspected on the Storage page.
  • Zip options [New in version 3.0]
If browser download has been chosen and the zip download option is selected, the file size can be configured after which the pcap files within the archive is spit into additional files
The number is entered in megabytes. 0 means no splitting.
  • Interface to transmit on
This dropdown menu is only shown when Capture type is Interface. Here the physical interface on which to transmit captured packets can be selected.
  • ERSPAN target address
This section is only shown when Capture type is ERSPAN. Here the target IP address or hostname for the ERSPAN encapsulated packets must be specified.
  • ERSPAN session ID
This section is only shown when Capture type is ERSPAN. The ERSPAN session ID can be used to differentiate between multiple capture session on the ERSPAN target.
  • Transmit speed
This dropdown menu is only shown when the Capture type is either Interface or ERSPAN and the start time is in the past so that packets are captured from the packet ring buffer. Here the limiting mode can be chosen which controls how fast captured packets are transmitted. Following modes are available:
  • none
No limit will be applied and the packets are transmitted as fast as the network interface and the packet ring buffer allow.
  • limit to bandwidth
A bandwidth limit will be applied so that the given bandwidth in Mbps is not exceeded. The bandwidth can be given as a decimal so that e.g. 500kbps can be configured with a value of 0.5.
  • realtime factor
Packets will be transmitted based on their recorded timing information. This means that with a real time factor of 1.0 packets will be transmitted approximately with the same timing as they were originally received. Using for example a real time factor of 2.0 would transmit the packets with twice the speed than they were received.
  • Transmit bandwidth in Mbps
This is only shown when limit to bandwidth has been selected in the Transmit speed dropdown menu. The meaning of this value is explained in the Transmit speed section.
  • Transmit realtime factor
This is only shown when realtime factor has been selected in the Transmit speed dropdown menu. The meaning of this value is explained in the :Transmit speed section.
  • Truncate packet length:
This dropdown menu is only shown when the Capture type is either HTTP or disk. You can truncate captured Packets with this setting. All packets will be captured, but truncated to the given length if they are longer than this setting. The length setting is applied on layer 2 without frame check sequence.
Possible values are:
  • Full length
  • 64 Bytes
  • 1500 Bytes
  • Custom length with an input field for inserting any length between 64 and 15378 Bytes
  • Capture profile: select a configured capture profile which defines rules about how many bytes are saved depending on the configured rules.
  • PCAP compatibility:
This section is only shown when the Capture type is either HTTP or disk.
  • Omit interface ID
Enabling this option will generate a PCAP file that only contains a single interface and treats all packets as if they arrived on that interface. This may improve compatibility with third party software that cannot handle PCAPs with multiple interfaces IDs.
  • PCAP comment:
Allows to add a user defined comment to the generated PCAP file.
  • Add device information to comment
Enabling this option will insert additional device information such as name, serial, memory size or capture filter into he PCAP comment.

After pushing the Start capture button, the capture starts.

Webshark

The Allegro Nework Multimeter has a preview mode to see the first Megabyte of captured packets directly in the browser. By clicking the Webshark preview button in the capture dialog, the first Megabyte of the requested packets will be extracted. If this is extraction is finished, a modal dialog will open showing the captured packets similar to Wireshark. The capture can be moved from the modal dialog to a separate window by pressing the button in the upper right corner next to the close button.

Capture URL

It is possible to use an external tool like curl for creating and storing a PCAP.

curl -k -u USER:PASSWORD 'https://allegro-mm-XXXX/API/data/modules/capture?startTime=1517306266000000&endTime=1517309267000000&expression=l7Protocol==HTTP&snapPacketLength=65535&fromCaptureBuffer=true' > path_to/capture.pcap

The user name, password and hostname similar to the access of the web interface have to be used. Following parameters are possible:

  • startTime: The start time of the capture. The first packet with exactly this or a later time will start the capture. The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch). If the start time is in the past, make sure you set fromCaptureBuffer parameter accordingly.
  • endTime: The end time of the capture. The first packet with exactly this or a later time will stop the capture. The time format must be microseconds after January, 1st 1970 UTC (Unix time, epoch).
  • expression: The filter expression. There are no whitespaces allowed. You may use ‘%20’ instead.
  • snapPacketLength: The max size of a packet applied on layer 2 without frame check sequence. If a packet is larger than this value, it is truncated. Use 65535 for unlimited size.
  • fromCaptureBuffer: Whether to extract data from the packet ring buffer or just live traffic.
  • captureToMedia: Whether to store PCAP on external storage device or download with your browser on your computer.
  • useSingleInterface: Whether to store only a single interface in the PCAP and treat all packets as if they arrived on that interface. This may improve compatibility with third party software that cannot handle PCAPs with multiple interfaces IDs.