Network Working Group L. Degioanni
Internet-Draft F. Risso
Expires: August 30, 2004 Politecnico di Torino
March 2004
PCAP New Generation Dump File Format
pcap
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://
www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 30, 2004.
Copyright Notice
Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract
This document describes a format to dump captured packets on a file.
This format is extensible and it is currently proposed for
implementation in the libpcap/WinPcap packet capture library.
Degioanni & Risso Expires August 30, 2004 [Page 1]
Internet-Draft PCAP New Generation Dump File Format March 2004
Table of Contents
1. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. General File Structure . . . . . . . . . . . . . . . . . . . . 4
2.1 General Block Structure . . . . . . . . . . . . . . . . . . . 4
2.2 Block Types . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Block Hierarchy and Precedence . . . . . . . . . . . . . . . . 5
2.4 Data format . . . . . . . . . . . . . . . . . . . . . . . . . 6
3. Block Definition . . . . . . . . . . . . . . . . . . . . . . . 8
3.1 Section Header Block (mandatory) . . . . . . . . . . . . . . . 8
3.2 Interface Description Block (mandatory) . . . . . . . . . . . 9
3.3 Packet Block (optional) . . . . . . . . . . . . . . . . . . . 13
3.4 Simple Packet Block (optional) . . . . . . . . . . . . . . . . 15
3.5 Name Resolution Block (optional) . . . . . . . . . . . . . . . 16
3.6 Interface Statistics Block (optional) . . . . . . . . . . . . 18
4. Options . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5. Experimental Blocks (deserved to a further investigation) . . 23
5.1 Other Packet Blocks (experimental) . . . . . . . . . . . . . . 23
5.2 Compression Block (experimental) . . . . . . . . . . . . . . . 23
5.3 Encryption Block (experimental) . . . . . . . . . . . . . . . 23
5.4 Fixed Length Block (experimental) . . . . . . . . . . . . . . 24
5.5 Directory Block (experimental) . . . . . . . . . . . . . . . . 25
5.6 Traffic Statistics and Monitoring Blocks (experimental) . . . 25
5.7 Event/Security Block (experimental) . . . . . . . . . . . . . 25
6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 27
7. Most important open issues . . . . . . . . . . . . . . . . . . 28
Intellectual Property and Copyright Statements . . . . . . . . 29
Degioanni & Risso Expires August 30, 2004 [Page 2]
Internet-Draft PCAP New Generation Dump File Format March 2004
1. Objectives
The problem of exchanging packet traces becomes more and more
critical every day; unfortunately, no standard solutions exist for
this task right now. One of the most accepted packet interchange
formats is the one defined by libpcap, which is rather old and does
not fit for some of the nowadays applications especially in terms of
extensibility.
This document proposes a new format for dumping packet traces. The
following goals are being pursued:
o Extensibility: aside of some common functionalities, third parties
should be able to enrich the information embedded in the file with
proprietary extensions, which will be ignored by tools that are
not able to understand them.
o Portability: a capture trace must contain all the information
needed to read data independently from network, hardware and
operating system of the machine that made the capture.
o Merge/Append data: it should be possible to add data at the end of
a given file, and the resulting file must still be readable.
Degioanni & Risso Expires August 30, 2004 [Page 3]
Internet-Draft PCAP New Generation Dump File Format March 2004
2. General File Structure
2.1 General Block Structure
A capture file is organized in blocks, that are appended one to
another to form the file. All the blocks share a common format, which
is shown in Figure 1.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Block Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Block Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ Block Body /
/ /* variable length, aligned to 32 bits */ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Block Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1: Basic block structure.
The fields have the following meaning:
o Block Type (32 bits): unique value that identifies the block.
Values whose Most Significant Bit (MSB) is equal to 1 are reserved
for local use. They allow to save private data to the file and to
extend the file format.
o Block Total Length: total size of this block, in bytes. For
instance, a block that does not have a body has a length of 12
bytes.
o Block Body: content of the block.
o Block Total Length: total size of this block, in bytes. This field
is duplicated for permitting backward file navigation.
This structure, shared among all blocks, makes easy to process a file
and to skip unneeded or unknown blocks. Blocks can be nested one
inside the others (NOTE: needed?). Some of the blocks are mandatory,
i.e. a dump file is not valid if they are not present, other are
optional.
The structure of the blocks allows to define other blocks if needed.
A parser that does non understand them can simply ignore their
content.
Degioanni & Risso Expires August 30, 2004 [Page 4]
Internet-Draft PCAP New Generation Dump File Format March 2004
2.2 Block Types
The currently defined blocks are the following:
1. Section Header Block: it defines the most important
characteristics of the capture file.
2. Interface Description Block: it defines the most important
characteristics of the interface(s) used for capturing traffic.
3. Packet Block: it contains a single captured packet, or a portion
of it.
4. Simple Packet Block: it contains a single captured packet, or a
portion of it, with only a minimal set of information about it.
5. Name Resolution Block: it defines the mapping from numeric
addresses present in the packet dump and the canonical name
counterpart.
6. Capture Statistics Block: it defines how to store some
statistical data (e.g. packet dropped, etc) which can be useful
to undestand the conditions in which the capture has been made.
7. Compression Marker Block: TODO
8. Encryption Marker Block: TODO
9. Fixed Length Marker Block: TODO
The following blocks instead are considered interesting but the
authors believe that they deserve more in-depth discussion before
being defined:
1. Further Packet Blocks
2. Directory Block
3. Traffic Statistics and Monitoring Blocks
4. Alert and Security Blocks
TODO Currently standardized Block Type codes are specified in
Appendix 1.
2.3 Block Hierarchy and Precedence
The file must begin with a Section Header Block. However, more than
Degioanni & Risso Expires August 30, 2004 [Page 5]
Internet-Draft PCAP New Generation Dump File Format March 2004
one Section Header Block can be present on the dump, each one
covering the data following it till the next one (or the end of
file). A Section includes the data delimited by two Section Header
Blocks (or by a Section Header Block and the end of the file),
including the first Section Header Block.
In case an application cannot read a Section because of different
version number, it must skip everything until the next Section Header
Block. Note that, in order to properly skip the blocks until the next
section, all blocks must have the fields Type and Length at the
beginning. This is a mandatory requirement that must be maintained in
future versions of the block format.
Figure 2 shows two valid files: the first has a typical
configuration, with a single Section Header that covers the whole
file. The second one contains three headers, and is normally the
result of file concatenation. An application that understands only
version 1.0 of the file format skips the intermediate section and
restart processing the packets after the third Section Header.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SHB v1.0 | Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Typical configuration with a single Section Header Block
|-- 1st Section --|-- 2nd Section --|-- 3rd Section --|
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SHB v1.0 | Data | SHB V1.1 | Data | SHB V1.0 | Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Configuration with three different Section Header Blocks
Figure 2: File structure example: the Section Header Block.
NOTE: TO BE COMPLETED with some examples of other blocks
2.4 Data format
Data contained in each section will always be saved according to the
characteristics (little endian / big endian) of the dumping machine.
This refers to all fields that are saved as numbers and that span
over two or more bytes.
The approach of having each section saved in the native format of the
generating host is more efficient because it avoids translation of
data when reading / writing on the host itself, which is the most
common case when generating/processing capture dumps.
Degioanni & Risso Expires August 30, 2004 [Page 6]
Internet-Draft PCAP New Generation Dump File Format March 2004
TODO Probably we have to specify something more here. Is what we're
saying enough to avoid any kind of ambiguity?.
Degioanni & Risso Expires August 30, 2004 [Page 7]
Internet-Draft PCAP New Generation Dump File Format March 2004
3. Block Definition
This section details the format of the body of the blocks currently
defined.
3.1 Section Header Block (mandatory)
The Section Header Block is mandatory. It identifies the beginning of
a section of the capture dump file. Its format is shown in Figure 3.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Magic |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Major | Minor |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ Options (variable) /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: Section Header Block format.
The meaning of the fields is:
o Magic: magic number, whose value is the hexadecimal number
0x1A2B3C4D. This number can be used to distinguish section that
have been saved on little-endian machines from the one saved on
big-endian machines.
o Major: number of the current mayor version of the format. Current
value is 1.
o Minor: number of the current minor version of the format. Current
value is 0.
o Options: optionally, a list of options (formatted according to the
rules defined in Section 4) can be present.
Aside form the options defined in Section 4, the following options
are valid within this block:
+----------------+----------------+----------------+----------------+
| Name | Code | Length | Description |
+----------------+----------------+----------------+----------------+
| Hardware | 2 | variable | An ascii |
| | | | string |
Degioanni & Risso Expires August 30, 2004 [Page 8]
Internet-Draft PCAP New Generation Dump File Format March 2004
| | | | containing the |
| | | | description of |
| | | | the hardware |
| | | | used to create |
| | | | this section. |
| | | | |
| Operating | 3 | variable | An ascii |
| System | | | string |
| | | | containing the |
| | | | name of the |
| | | | operating |
| | | | system used to |
| | | | create this |
| | | | section. |
| | | | |
| User | 3 | variable | An ascii |
| Application | | | string |
| | | | containing the |
| | | | name of the |
| | | | application |
| | | | used to create |
| | | | this section. |
+----------------+----------------+----------------+----------------+
Table 1
The Section Header Block does not contain data but it rather
identifies a list of blocks (interfaces, packets) that are logically
correlated. This block does not contain any reference to the size of
the section it is currently delimiting, therefore the reader cannot
skip a whole section at once. In case a section must be skipped, the
user has to repeatedly skip all the blocks contained within it; this
makes the parsing of the file slower but it permits to append several
capture dumps at the same file.
3.2 Interface Description Block (mandatory)
The Interface Description Block is mandatory. This block is needed to
specify the characteristics of the network interface on which the
capture has been made. In order to properly associate the captured
data to the corresponding interface, the Interface Description Block
must be defined before any other block that uses it; therefore, this
block is usually placed immediately after the Section Header Block.
An Interface Description Block is valid only inside the section which
it belongs to. The structure of a Interface Description Block is
shown in Figure 4.
Degioanni & Risso Expires August 30, 2004 [Page 9]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Interface ID | LinkType |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SnapLen |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ Options (variable) /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: Interface Description Block format.
The meaning of the fields is:
o Interface ID: a progressive number that identifies uniquely any
interface inside current section. Two Interface Description Blocks
can have the same Interface ID only if they are in different
sections of the file. The Interface ID is referenced by the packet
blocks.
o LinkType: a value that defines the link layer type of this
interface.
o SnapLen: maximum number of bytes dumped from each packet. The
portion of each packet that exceeds this value will not be stored
in the file.
o Options: optionally, a list of options (formatted according to the
rules defined in Section 4) can be present.
In addition to the options defined in Section 4, the following
options are valid within this block:
+----------------+----------------+----------------+----------------+
| Name | Code | Length | Description |
+----------------+----------------+----------------+----------------+
| if_name | 2 | Variable | Name of the |
| | | | device used to |
| | | | capture data. |
| | | | |
| if_IPv4addr | 3 | 8 | Interface |
| | | | network |
| | | | address and |
| | | | netmask. |
| | | | |
| if_IPv6addr | 4 | 17 | Interface |
Degioanni & Risso Expires August 30, 2004 [Page 10]
Internet-Draft PCAP New Generation Dump File Format March 2004
| | | | network |
| | | | address and |
| | | | prefix length |
| | | | (stored in the |
| | | | last byte). |
| | | | |
| if_MACaddr | 5 | 6 | Interface |
| | | | Hardware MAC |
| | | | address (48 |
| | | | bits). |
| | | | |
| if_EUIaddr | 6 | 8 | Interface |
| | | | Hardware EUI |
| | | | address (64 |
| | | | bits), if |
| | | | available. |
| | | | |
| if_speed | 7 | 8 | Interface |
| | | | speed (in |
| | | | bps). |
| | | | |
| if_tsaccur | 8 | 1 | Precision of |
| | | | timestamps. If |
| | | | the Most |
| | | | Significant |
| | | | Bit is equal |
| | | | to zero, the |
| | | | remaining bits |
| | | | indicates the |
| | | | accuracy as as |
| | | | a negative |
| | | | power of 10 |
| | | | (e.g. 6 means |
| | | | microsecond |
| | | | accuracy). If |
| | | | the Most |
| | | | Significant |
| | | | Bit is equal |
| | | | to zero, the |
| | | | remaining bits |
| | | | indicates the |
| | | | accuracy as as |
| | | | negative power |
| | | | of 2 (e.g. 10 |
| | | | means 1/1024 |
| | | | of second). If |
| | | | this option is |
| | | | not present, a |
Degioanni & Risso Expires August 30, 2004 [Page 11]
Internet-Draft PCAP New Generation Dump File Format March 2004
| | | | precision of |
| | | | 10^-6 is |
| | | | assumed. |
| | | | |
| if_tzone | 9 | 4 | Time zone for |
| | | | GMT support |
| | | | (TODO: specify |
| | | | better). |
| | | | |
| if_flags | 10 | 4 | Interface |
| | | | flags. (TODO: |
| | | | specify |
| | | | better. |
| | | | Possible |
| | | | flags: |
| | | | promiscuous, |
| | | | inbound/outbou |
| | | | nd, traffic |
| | | | filtered |
| | | | during |
| | | | capture). |
| | | | |
| if_filter | 11 | variable | The filter |
| | | | (e.g. "capture |
| | | | only TCP |
| | | | traffic") used |
| | | | to capture |
| | | | traffic. The |
| | | | first byte of |
| | | | the Option |
| | | | Data keeps a |
| | | | code of the |
| | | | filter used |
| | | | (e.g. if this |
| | | | is a libpcap |
| | | | string, or BPF |
| | | | bytecode, and |
| | | | more). More |
| | | | details about |
| | | | this format |
| | | | will be |
| | | | presented in |
| | | | Appendix XXX |
| | | | (TODO). |
| | | | |
| if_opersystem | 12 | variable | An ascii |
| | | | string |
| | | | containing the |
Degioanni & Risso Expires August 30, 2004 [Page 12]
Internet-Draft PCAP New Generation Dump File Format March 2004
| | | | name of the |
| | | | operating |
| | | | system of the |
| | | | machine that |
| | | | hosts this |
| | | | interface. |
| | | | This can be |
| | | | different from |
| | | | the same |
| | | | information |
| | | | that can be |
| | | | contained by |
| | | | the Section |
| | | | Header Block |
| | | | (Section 3.1) |
| | | | because the |
| | | | capture can |
| | | | have been done |
| | | | on a remote |
| | | | machine. |
+----------------+----------------+----------------+----------------+
Table 2
3.3 Packet Block (optional)
A Packet Block is the standard container for storing the packets
coming from the network. The Packet Block is optional because packets
can be stored either by means of this block or the Simple Packet
Block, which can be used to speed up dump generation. The format of a
packet block is shown in Figure 5.
Degioanni & Risso Expires August 30, 2004 [Page 13]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Interface ID | Drops Count |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp (High) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp (Low) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Captured Len |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Packet Len |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Packet Data |
| |
| /* variable length, byte-aligned */ |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ Options (variable) /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5: Packet Block format.
The Packet Block has the following fields:
o Interface ID: Specifies the interface this packet comes from, and
corresponds to the ID of one of the Interface Description Blocks
present in this section of the file (see Figure 4).
o Drops Count: a local drop counter. It specified the number of
packets lost (by the interface and the operating system) between
this packet and the preceding one. The value xFFFF (in
hexadecimal) is reserved for those systems in which this
information is not available.
o Timestamp (High): the most significative part of the timestamp. in
standard Unix format, i.e. from 1/1/1970.
o Timestamp (Low): the less significative part of the timestamp. The
way to interpret this field is specified by the 'ts_accur' option
(see Figure 4) of the Interface Description block referenced by
this packet. If the Interface Description block does not contain a
'ts_accur' option, then this field is expressed in microseconds.
o Captured Len: number of bytes captured from the packet (i.e. the
Degioanni & Risso Expires August 30, 2004 [Page 14]
Internet-Draft PCAP New Generation Dump File Format March 2004
length of the Packet Data field). It will be the minimum value
among the actual Packet Length and the snapshot length (defined in
Figure 4).
o Packet Len: actual length of the packet when it was transmitted on
the network. Can be different from Captured Len if the user wants
only a snapshot of the packet.
o Packet Data: the data coming from the network, including
link-layer headers. The length of this field is Captured Len. The
format of the link-layer headers depends on the LinkType field
specified in the Interface Description Block (see Section 3.2) and
it is specified in Appendix XXX (TODO).
o Options: optionally, a list of options (formatted according to the
rules defined in Section 4) can be present.
3.4 Simple Packet Block (optional)
The Simple Packet Block is a lightweight container for storing the
packets coming from the network. Its presence is optional.
A Simple Packet Block is similar to a Packet Block (see Section 3.3),
but it is smaller, simpler to process and contains only a minimal set
of information. This block is preferred to the standard Packet Block
when performance or space occupation are critical factors, such as in
sustained traffic dump applications. A capture file can contain both
Packet Blocks and Simple Packet Blocks: for example, a capture tool
could switch from Packet Blocks to Simple Packet Blocks when the
hardware resources become critical.
The Simple Packet Block does not contain the Interface ID field.
Therefore, it must be assumed that all the Simple Packet Blocks have
been captured on the interface previously specified in the Interface
Description Block.
Figure 6 shows the format of the Simple Packet Block.
Degioanni & Risso Expires August 30, 2004 [Page 15]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Packet Len |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Packet Data |
| |
| /* variable length, byte-aligned */ |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6: Simple Packet Block format.
The Packet Block has the following fields:
o Packet Len: actual length of the packet when it was transmitted on
the network. Can be different from captured len if the packet has
been truncated.
o Packet data: the data coming from the network, including
link-layers headers. The length of this field can be derived from
the field Block Total Length, present in the Block Header.
The Simple Packet Block does not contain the timestamp because this
is one of the most costly operations on PCs. Additionally, there are
applications that do not require it; e.g. an Intrusion Detection
System is interested in packets, not in their timestamp.
The Simple Packet Block is very efficient in term of disk space: a
snapshot of length 100 bytes requires only 16 bytes of overhead,
which corresponds to an efficiency of more than 86%.
3.5 Name Resolution Block (optional)
The Name Resolution Block is used to support the correlation of
numeric addresses (present in the captured packets) and their
corresponding canonical names and it is optional. Having the literal
names saved in the file, this prevents the need of a name resolution
in a delayed time, when the association between names and addresses
can be different from the one in use at capture time. Moreover, The
Name Resolution Block avoids the need of issuing a lot of DNS
requests every time the trace capture is opened, and allows to have
name resolution also when reading the capture with a machine not
connected to the network.
The format of the Name Resolution Block is shown in Figure 7.
Degioanni & Risso Expires August 30, 2004 [Page 16]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Record Type | Record Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Record Value |
| /* variable length, byte-aligned */ |
| + + + + + + + + + + + + + + + + + + + + + + + + +
| | | | |
+-+-+-+-+-+-+-+-+ + + + + + + + + + + + + + + + + + + + + + + + +
. . . other records . . .
| Record Type == end_of_recs | Record Length == 00 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ Options (variable) /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7: Name Resolution Block format.
A Name Resolution Block is a zero-terminated list of records (in the
TLV format), each of which contains an association between a network
address and a name. There are three possible types of records:
+----------------+----------------+----------------+----------------+
| Name | Code | Length | Description |
+----------------+----------------+----------------+----------------+
| end_of_recs | 0 | 0 | End of records |
| | | | |
| ip4_rec | 1 | Variable | Specifies an |
| | | | IPv4 address |
| | | | (contained in |
| | | | the first 4 |
| | | | bytes), |
| | | | followed by |
| | | | one or more |
| | | | zero-terminate |
| | | | d strings |
| | | | containing the |
| | | | DNS entries |
| | | | for that |
| | | | address. |
| | | | |
| ip6_rec | 1 | Variable | Specifies an |
| | | | IPv6 address |
| | | | (contained in |
| | | | the first 16 |
| | | | bytes), |
Degioanni & Risso Expires August 30, 2004 [Page 17]
Internet-Draft PCAP New Generation Dump File Format March 2004
| | | | followed by |
| | | | one or more |
| | | | zero-terminate |
| | | | d strings |
| | | | containing the |
| | | | DNS entries |
| | | | for that |
| | | | address. |
+----------------+----------------+----------------+----------------+
Table 3
After the list or Name Resolution Records, optionally, a list of
options (formatted according to the rules defined in Section 4) can
be present.
A Name Resolution Block is normally placed at the beginning of the
file, but no assumptions can be taken about its position. Name
Resolution Blocks can be added in a second time by tools that process
the file, like network analyzers.
In addiction to the options defined in Section 4, the following
options are valid within this block:
+----------------+----------------+----------------+----------------+
| Name | Code | Length | Description |
+----------------+----------------+----------------+----------------+
| ns_dnsname | 2 | Variable | An ascii |
| | | | string |
| | | | containing the |
| | | | name of the |
| | | | machine (DNS |
| | | | server) used |
| | | | to perform the |
| | | | name |
| | | | resolution. |
+----------------+----------------+----------------+----------------+
3.6 Interface Statistics Block (optional)
The Interface Statistics Block contains the capture statistics for a
given interface and it is optional. The statistics are referred to
the interface defined in the current Section identified by the
Interface ID field.
The format of the Interface Statistics Block is shown in Figure 8.
Degioanni & Risso Expires August 30, 2004 [Page 18]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IfRecv |
| (high + low) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IfDrop |
| (high + low) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FilterAccept |
| (high + low) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OSDrop |
| (high + low) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UsrDelivered |
| (high + low) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Interface ID | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ Options (variable) /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8: Interface Statistics Block format.
The fields have the following meaning:
o IfRecv: number of packets received from the interface during the
capture. This number is reported as a 64 bits value, in which the
most significat bits are located in the first four bytes of the
field.
o IfDrop: number of packets dropped by the interface during the
capture due to lack of resources.
o FilterAccept: number of packets accepeted by filter during current
capture.
o OSDrop: number of packets dropped by the operating system during
the capture.
o UsrDelivered: number of packets delivered to the user.
UsrDelivered can be different from the value 'FilterAccept -
OSDropped' because some packets could still lay in the OS buffers
when the capture ended.
Degioanni & Risso Expires August 30, 2004 [Page 19]
Internet-Draft PCAP New Generation Dump File Format March 2004
o Interface ID: reference to an Interface Description Block.
o Reserved: Reserved to future use.
o Options: optionally, a list of options (formatted according to the
rules defined in Section 4) can be present.
In addiction to the options defined in Section 4, the following
options are valid within this block:
+----------------+----------------+----------------+----------------+
| Name | Code | Length | Description |
+----------------+----------------+----------------+----------------+
| isb_starttime | 2 | 8 | Time in which |
| | | | the capture |
| | | | started; time |
| | | | will be stored |
| | | | in two blocks |
| | | | of four bytes |
| | | | each, |
| | | | containing the |
| | | | timestamp in |
| | | | seconds and |
| | | | nanoseconds. |
| | | | |
| isb_endtime | 3 | 8 | Time in which |
| | | | the capture |
| | | | started; time |
| | | | will be stored |
| | | | in two blocks |
| | | | of four bytes |
| | | | each, |
| | | | containing the |
| | | | timestamp in |
| | | | seconds and |
| | | | nanoseconds. |
+----------------+----------------+----------------+----------------+
Degioanni & Risso Expires August 30, 2004 [Page 20]
Internet-Draft PCAP New Generation Dump File Format March 2004
4. Options
Almost all blocks have the possibility to embed optional fields.
Optional fields can be used to insert some information that may be
useful when reading data, but that it is not really needed for packet
processing. Therefore, each tool can be either read the content of
the optional fields (if any), or skip them at once.
Skipping all the optional fields at once is straightforward because
most of the blocks have a fixed length, therefore the field Block
Length (present in the General Block Structure, see Section 2.1) can
be used to skip everything till the next block.
Options are a list of Type - Length - Value fields, each one
containing a single value:
o Option Type (2 bytes): it contains the code that specifies the
type of the current TLV record. Option types whose Most
Significant Bit is equal to one are reserved for local use;
therefore, there is no guarantee that the code used is unique
among all capture files (generated by other applications). In case
of vendor-specific extensions that have to be identified uniquely,
vendors must request an Option Code whose MSB is equal to zero.
o Option Length (2 bytes): it contains the length of the following
'Option Value' field.
o Option Value (variable length): it contains the value of the given
option. The length of this field as been specified by the Option
Length field.
Options may be repeated several times (e.g. an interface that has
several IP addresses associated to it). The option list is terminated
by a special code which is the 'End of Option'.
The format of the optional fields is shown in Figure 9.
Degioanni & Risso Expires August 30, 2004 [Page 21]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Option Code | Option Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Option Value |
| /* variable length, byte-aligned */ |
| + + + + + + + + + + + + + + + + + + + + + + + + +
| / / / |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ . . . other options . . . /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Option Code == opt_endofopt | Option Length == 0 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 9: Options format.
The following codes can always be present in any optional field:
+----------------+----------------+----------------+----------------+
| Name | Code | Length | Description |
+----------------+----------------+----------------+----------------+
| opt_endofopt | 0 | 0 | End of |
| | | | options: it is |
| | | | used to |
| | | | delimit the |
| | | | end of the |
| | | | optional |
| | | | fields. This |
| | | | block cannot |
| | | | be repeated |
| | | | within a given |
| | | | list of |
| | | | options. |
| | | | |
| opt_comment | 1 | variable | Comment: it is |
| | | | an ascii |
| | | | string |
| | | | containing a |
| | | | comment that |
| | | | is associated |
| | | | to the current |
| | | | block. |
+----------------+----------------+----------------+----------------+
Degioanni & Risso Expires August 30, 2004 [Page 22]
Internet-Draft PCAP New Generation Dump File Format March 2004
5. Experimental Blocks (deserved to a further investigation)
5.1 Other Packet Blocks (experimental)
Can some other packet blocks (besides the two described in the
previous paragraphs) be useful?
5.2 Compression Block (experimental)
The Compression Block is optional. A file can contain an arbitrary
number of these blocks. A Compression Block, as the name says, is
used to store compressed data. Its format is shown in Figure 10.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Compr. Type | |
+-+-+-+-+-+-+-+-+ |
| |
| Compressed Data |
| |
| /* variable length, byte-aligned */ |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 10: Compression Block format.
The fields have the following meaning:
o Compression Type: specifies the compression algorithm. Possible
values for this field are 0 (uncompressed), 1 (Lempel Ziv), 2
(Gzip), other?? Probably some kind of dumb and fast compression
algorithm could be effective with some types of traffic (for
example web), but which?
o Compressed Data: data of this block. Once decompressed, it is made
of other blocks.
5.3 Encryption Block (experimental)
The Encryption Block is optional. A file can contain an arbitrary
number of these blocks. An Encryption Block is used to sotre
encrypted data. Its format is shown in Figure 11.
Degioanni & Risso Expires August 30, 2004 [Page 23]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Encr. Type | |
+-+-+-+-+-+-+-+-+ |
| |
| Compressed Data |
| |
| /* variable length, byte-aligned */ |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 11: Encryption Block format.
The fields have the following meaning:
o Compression Type: specifies the encryption algorithm. Possible
values for this field are ??? NOTE: this block should probably
contain other fields, depending on the encryption algorithm. To be
define precisely.
o Encrypted Data: data of this block. Once decripted, it consists of
other blocks.
5.4 Fixed Length Block (experimental)
The Fixed Length Block is optional. A file can contain an arbitrary
number of these blocks. A Fixed Length Block can be used to optimize
the access to the file. Its format is shown in Figure 12. A Fixed
Length Block stores records with constant size. It contains a set of
Blocks (normally Packet Blocks or Simple Packet Blocks), of wihich it
specifies the size. Knowing this size a priori helps to scan the file
and to load some portions of it without truncating a block, and is
particularly useful with cell-based networks like ATM.
Degioanni & Risso Expires August 30, 2004 [Page 24]
Internet-Draft PCAP New Generation Dump File Format March 2004
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Cell Size | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| Fixed Size Data |
| |
| /* variable length, byte-aligned */ |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 12: Fixed Length Block format.
The fields have the following meaning:
o Cell size: the size of the blocks contained in the data field.
o Fixed Size Data: data of this block.
5.5 Directory Block (experimental)
If present, this block contains the following information:
o number of indexed packets (N)
o table with position and length of any indexed packet (N entries)
A directory block must be followed by at least N packets, otherwise
it must be considered invalid. It can be used to efficiently load
portions of the file to memory and to support operations on memory
mapped files. This block can be added by tools like network analyzers
as a consequence of file processing.
5.6 Traffic Statistics and Monitoring Blocks (experimental)
One or more blocks could be defined to contain network statistics or
traffic monitoring information. They could be use to store data
collected from RMON or Netflow probes, or from other network
monitoring tools.
5.7 Event/Security Block (experimental)
This block could be used to store events. Events could contain
generic information (for example network load over 50%, server
down...) or security alerts. An event could be:
Degioanni & Risso Expires August 30, 2004 [Page 25]
Internet-Draft PCAP New Generation Dump File Format March 2004
o skipped, if the application doesn't know how to do with it
o processed independently by the packets. In other words, the
applications skips the packets and processes only the alerts
o processed in relation to packets: for example, a security tool
could load only the packets of the file that are near a security
alert; a monitorg tool could skip the packets captured while the
server was down.
Degioanni & Risso Expires August 30, 2004 [Page 26]
Internet-Draft PCAP New Generation Dump File Format March 2004
6. Conclusions
The file format proposed in this document should be very versatile
and satisfy a wide range of applications. In the simplest case, it
can contain a raw dump of the network data, made of a series of
Simple Packet Blocks. In the most complex case, it can be used as a
repository for heterogeneous information. In every case, the file
remains easy to parse and an application can always skip the data it
is not interested in; at the same time, different applications can
share the file, and each of them can benfit of the information
produced by the others. Two or more files can be concatenated
obtaining another valid file.
Degioanni & Risso Expires August 30, 2004 [Page 27]
Internet-Draft PCAP New Generation Dump File Format March 2004
7. Most important open issues
o Data, in the file, must be byte or word aligned? Currently, the
structure of this document is not consistent with respect to this
point.
Degioanni & Risso Expires August 30, 2004 [Page 28]
Internet-Draft PCAP New Generation Dump File Format March 2004
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; neither does it represent that it
has made any effort to identify any such rights. Information on the
IETF's procedures with respect to rights in standards-track and
standards-related documentation can be found in BCP-11. Copies of
claims of rights made available for publication and any assurances of
licenses to be made available, or the result of an attempt made to
obtain a general license or permission for the use of such
proprietary rights by implementors or users of this specification can
be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF Executive
Director.
Full Copyright Statement
Copyright (C) The Internet Society (2004). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assignees.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
Degioanni & Risso Expires August 30, 2004 [Page 29]
Internet-Draft PCAP New Generation Dump File Format March 2004
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Degioanni & Risso Expires August 30, 2004 [Page 30]