Structuring Large Capture Files
Wireshark filter and other Allegro Network Multimeter tools
This article explains strategies to cope with the challenges and reduce the effort of packet analysis - be it for troubleshooting or for evaluating network quality.
The first part discusses techniques that explain how to structure searches using Wireshark tools. The techniques of filters, colour marking and protocol hierarchies are covered here. The second part describes how the Allegro Network Multimeter can be used to significantly speed up the work with the pcap analyzer.
Flagship Wireshark
The open source program Wireshark is an unrivalled pcap analyzer. The project, which emerged from the predecessor 'Ethereal', has existed since 2006 and has pushed all commercial comparison products off the market since its release. Wireshark is a packet-oriented analyzer for pinpointing problems and displays data logs graphically.
Most busy networks will have many parallel network connections requiring accurate packet analysis. For example, a visit to just one website can generate connections to many other hosts.
The Use of Filters
Pcap file analysis can prove a challenge due to the sheer quantity of data to be handled. Filters can be used to selectively hide connections that are not of interest. The goal is to end up with a relatively manageable set of packets as a starting point for detailed analysis.
In addition to external tools and techniques to filter traffic such as the application of BPF syntax, Wireshark has a number of on-board means to reduce the abundance of information in order to get closer to the relevant information. Wireshark distinguishes two types of filters. The capture filters define which packets are logged. The display filters define which of the captured packets are included in the current analysis. Unfortunately, the two filters use different syntax.
Simple Display Filters in Wireshark
The most common content reduction technique used within Wireshark is the use of display filters.
The simplest use of display filters is to reduce traffic to a single application, a specific protocol, or the exact specification of a data field. To do this, you can either use the menu (Analysis->Display Filter) to select, for example, the HTTP protocol, which will limit the view to all HTTP entries, or insert the desired filter value directly into the filter toolbar. If you want to see only all SIP connections, you just have to type 'sip' in the input field and confirm with 'Enter'
A filter is in use can be seen in the input field of the filter toolbar (highlighted in green). The status bar at the bottom right also shows that a filter is set or that only a certain percentage of the packets are displayed at the time.
Wireshark provides an auto-complete function when entering the display filters directly, so that all available filters with the same letter sequence are suggested when entering a filter.
Comparison Operators
In addition to using simple filters, conditions can also be linked. Wireshark's filter syntax provides for parentheses, logical operators such as 'and' 'or', and comparison operators such as == or !=.
For example, if you want to show 'any TCP traffic from IP address 10.17.2.5 to port 80', the translation to Wireshark's filter syntax is ip.src == 10.17.2.5 and tcp.dstport == 80.
In this example, the conditions are linked with 'and'. Condition 1 states that the source IP address of the packets must be 10.17.2.5 and condition 2 specifies that the protocol must be TCP and the destination port must be 80.
Any number of conditions can be linked to further limit the selection of traffic displayed.
Expression Builder from Wireshark
As a skilled Wireshark user, expressions can be applied freely from memory. Initially, it is easier to use Wireshark's Expression Builder dialogue box to add an expression to the display filter. This dialogue box opens when the term 'Expression' is right-clicked in the filter toolbar. Here, predefined operators can be selected and linked. To check if the selected filter is correct, the filter toolbar turns green. If the filter is invalid, the area is highlighted in red.
If required, Wireshark filters can be saved.
Capture filter
In addition to the display filters described above, which reduce the packets displayed, filters can be applied the moment that traffic recording begins; these are called capture filters, ensuring that network data is limited to the desired selection.
Wireshark capture filters use the same syntax as tcpdump, the libpcap filters. That is, a syntax of byte offsets, hex values, and masks associated with true values to filter the data. Capture filters are not trivial in their application because they are more cryptic than display filters.
The Wireshark manual contains much more information about the filters integrated in Wireshark.
In this article, only the most important filters that Wireshark provides as an on-board tool are addressed. Applying filters more esoteric than the simplest display filters requires in-depth knowledge of Wireshark's filter syntax in order to consistently use filters to address one's research question.
Colour Marking with Wireshark
In addition to the filter functions, Wireshark has a customizable colour coding system. By default for example, all UDP packets are marked in blue, standard TCP transfer in purple and HTTP in green. These colour codes help an administrator to identify the packet types at a glance. User-defined colour rules can be assigned to their own profile and saved, complete the system. Both text and background colours can be customized.
Individual streams can be easily tracked via automatic colour coding. However, in order to see at a glance which connections are active or to which connections individual packets belong, instead of determining this via IP addresses/port numbers, you can regulate this via colour codes. To enable this, after right-clicking the packet, you can click 'Colour connection' in the context menu and select the type of connection (Ethernet, IPv4, IPv6, TCP, UDP, etc.).
Colouring connections based on the transport protocol, individual communication streams between the same IP pair can be distinguished. This enables a more granular analysis than just looking only at the IP address.
Colour coding of individual connections is especially useful in situations where multiple hosts are communicating at the same time, or where there are multiple communication connections between the same hosts that need to be distinguished.
Colour coding is also a viable way to take notes if you want to take a closer look at any potentially suspicious traffic.
Colour coding provides an easy-to-use technique for initial orientation in large pcap data files.
Use of Protocol Hierarchies
The third technique to be described here for better orientation in large pcap files is the use of protocol hierarchies. Compared with colour coding, this requires advanced Wireshark knowledge. The use of protocol hierarchies provides useful clues, for example, when tracking down suspicious applications or protocols.
To display the logs used in a record file, the menu item Statistics -> Log hierarchy must be selected. The hierarchy provides a tree-like view of the logs, including the statistical values for each log. It is called hierarchy since the data is arranged based on the communication layers, as most packets contain multiple encapsulated protocols. Therefore, an HTTP packet is listed under a TCP packet, both under IP packets, and so on.
In addition to the protocol information, an administrator can view how high the protocol's share of the total traffic is, the exact number of packets or the bandwidth of a protocol. If unexpectedly high values are detected in the data entries, this traffic should be examined more closely. For this purpose, the entry is marked with the right mouse button for further analysis. A context menu opens to filter or colour packets directly. Due to the hierarchical structure of the list, it is not possible to sort or reorder the logs.
However, a deductive strategy can be used by filtering out logs that are not of interest directly from the log hierarchy view. A separate capture file can be saved from the filtered results.
Overall, the protocol hierarchy technique is a scaleable way to get an overall view of a pcap file. The protocol hierarchy is often the starting point for further analyses, because here indications of unexpected traffic or errors come to light, such as unexpected protocols or unexpected data ratios of individual protocols. It is often not the presence of a protocol that generates attention, but its relative proportion.
To the 2nd part of the blog article
In the second part of the blog article, read how to use the Allegro Network Multimeter as a pre-filter to filter and structure large capture files.
