Main logging schema for qlog

1.1.1. Schema definition

To define events and data structures, all qlog documents use the Concise Data Definition Language [CDDL]. This document uses the basic syntax, the specific text, uint, float32, float64, bool, and any types, as well as the .default, .size, and .regexp control operators, the ~ unwrapping operator, and the $ extension point syntax from [CDDL].¶

Additionally, this document defines the following custom types for clarity:¶

; CDDL's uint is defined as being 64-bit in size
; but for many protocol fields we want to be more restrictive
; and explicit
uint8 = uint .size 1
uint16 = uint .size 2
uint32 = uint .size 4
uint64 = uint .size 8

; an even-length lowercase string of hexadecimally encoded bytes
; examples: 82dc, 027339, 4cdbfd9bf0
; this is needed because the default CDDL binary string (bytes/bstr)
; is only CBOR and not JSON compatible
hexstring = text .regexp "([0-9a-f]{2})*"

The main general CDDL syntax conventions in this document a reader should be aware of for easy reading comprehension are:¶

• ? obj : this object is optional¶
• TypeName1 / TypeName2 : a union of these two types (object can be either type 1 OR type 2)¶
• obj: TypeName : this object has this concrete type¶
• obj: [* TypeName] : this object is an array of this type with minimum size of 0 elements¶
• obj: [+ TypeName] : this object is an array of this type with minimum size of 1 element¶
• TypeName = ... : defines a new type¶
• EnumName = "entry1" / "entry2" / entry3 / ...: defines an enum¶
• StructName = { ... } : defines a new struct type¶
• ; : single-line comment¶
• * text => any : special syntax to indicate 0 or more fields that have a string key that maps to any value. Used to indicate a generic JSON object.¶

All timestamps and time-related values (e.g., offsets) in qlog are logged as float64 in the millisecond resolution.¶

Other qlog documents can define their own CDDL-compatible (struct) types (e.g., separately for each Packet type that a protocol supports).¶

1.1.2. Serialization

While the qlog schemas are format-agnostic, and can be serialized in many ways (e.g., JSON, CBOR, protobuf, ...), this document only describes how to employ [JSON], its subset [I-JSON], and its streamable derivative [JSON-Text-Sequences] as textual serialization options. As such, examples are provided in [JSON]. Other documents may describe how to utilize other concrete serialization options, though tips and requirements for these are also listed in this document (Section 6).¶

3.3.1. Configuration

We take into account that a qlog file is usually not used in isolation, but by means of various tools. Especially when aggregating various traces together or preparing traces for a demonstration, one might wish to persist certain tool-based settings inside the qlog file itself. For this, the configuration field is used.¶

The configuration field can be viewed as a generic metadata field that tools can fill with their own fields, based on per-tool logic. It is best practice for tools to prefix each added field with their tool name to prevent collisions across tools. This document only defines two optional, standard, tool-independent configuration settings: "time_offset" and "original_uris".¶

Definition:¶

Configuration = {
    ; time_offset is in milliseconds
    time_offset: float64
    original_uris:[* text]
    * text => any
}

JSON serialization example:¶

{
    "time_offset": 150,
    "original_uris": [
        "https://example.org/trace1.qlog",
        "https://example.org/trace2.qlog"
    ]
}

{
    "configuration" : {
        "qvis" : {
            "congestion_graph": {
                "startX": 1000,
                "endX": 2000,
                "focusOnEventIndex": 124
            }

            "sequence_diagram" : {
                "focusOnEventIndex": 555
            }
        }
    }
}

3.3.2. vantage_point

The vantage_point field describes the vantage point from which the trace originates, see Figure 13. Each trace can have only a single vantage_point and thus all events in a trace MUST BE from the perspective of this vantage_point. To include events from multiple vantage_points, implementers can for example include multiple traces, split by vantage_point, in a single qlog file.¶

Definitions:¶

VantagePoint = {
    ? name: text
    type: VantagePointType
    ? flow: VantagePointType
}

; client = endpoint which initiates the connection
; server = endpoint which accepts the connection
; network = observer in between client and server
VantagePointType = "client" / "server" / "network" / "unknown"

JSON serialization examples:¶

{
    "name": "aioquic client",
    "type": "client",
}

{
    "name": "wireshark trace",
    "type": "network",
    "flow": "client"
}

The flow field is only required if the type is "network" (for example, the trace is generated from a packet capture). It is used to disambiguate events like "packet sent" and "packet received". This is indicated explicitly because for multiple reasons (e.g., privacy) data from which the flow direction can be otherwise inferred (e.g., IP addresses) might not be present in the logs.¶

Meaning of the different values for the flow field: * "client" indicates that this vantage point follows client data flow semantics (a "packet sent" event goes in the direction of the server). * "server" indicates that this vantage point follow server data flow semantics (a "packet sent" event goes in the direction of the client). * "unknown" indicates that the flow's direction is unknown.¶

Depending on the context, tools confronted with "unknown" values in the vantage_point can either try to heuristically infer the semantics from protocol-level domain knowledge (e.g., in QUIC, the client always sends the first packet) or give the user the option to switch between client and server perspectives manually.¶

3.4.1. Timestamps

The "time" field indicates the timestamp at which the event occured. Its value is typically the Unix timestamp since the 1970 epoch (number of milliseconds since midnight UTC, January 1, 1970, ignoring leap seconds). However, qlog supports two more succint timestamps formats to allow reducing file size. The employed format is indicated in the "time_format" field, which allows one of three values: "absolute", "delta" or "relative".¶

Definition:¶

TimeFormat = "absolute" / "delta" / "relative"

• Absolute: Include the full absolute timestamp with each event. This approach uses the largest amount of characters. This is also the default value of the "time_format" field.¶
• Delta: Delta-encode each time value on the previously logged value. The first event in a trace typically logs the full absolute timestamp. This approach uses the least amount of characters.¶
• Relative: Specify a full "reference_time" timestamp (typically this is done up-front in "common_fields", see Section 3.4.7) and include only relatively-encoded values based on this reference_time with each event. The "reference_time" value is typically the first absolute timestamp. This approach uses a medium amount of characters.¶

The first option is good for stateless loggers, the second and third for stateful loggers. The third option is generally preferred, since it produces smaller files while being easier to reason about. An example for each option can be seen in Figure 18.¶

The absolute approach will use:
1500, 1505, 1522, 1588

The delta approach will use:
1500, 5, 17, 66

The relative approach will:
- set the reference_time to 1500 in "common_fields"
- use: 0, 5, 22, 88

One of these options is typically chosen for the entire trace (put differently: each event has the same value for the "time_format" field). Each event MUST include a timestamp in the "time" field.¶

Events in each individual trace SHOULD be logged in strictly ascending timestamp order (though not necessarily absolute value, for the "delta" format). Tools CAN sort all events on the timestamp before processing them, though are not required to (as this could impose a significant processing overhead). This can be a problem especially for multi-threaded and/or streaming loggers, who could consider using a separate postprocesser to order qlog events in time if a tool do not provide this feature.¶

Timestamps do not have to use the UNIX epoch timestamp as their reference. For example for privacy considerations, any initial reference timestamps (for example "endpoint uptime in ms" or "time since connection start in ms") can be chosen. Tools SHOULD NOT assume the ability to derive the absolute Unix timestamp from qlog traces, nor allow on them to relatively order events across two or more separate traces (in this case, clock drift should also be taken into account).¶

3.4.2. Category and Event Type

Events differ mainly in the type of metadata associated with them. To help identify a given event and how to interpret its metadata in the "data" field (see Section 3.4.3), each event has an associated "name" field. This can be considered as a concatenation of two other fields, namely event "category" and event "type".¶

Category allows a higher-level grouping of events per specific event type. For example for QUIC and HTTP/3, the different categories could be "transport", "http", "qpack", and "recovery". Within these categories, the event Type provides additional granularity. For example for QUIC and HTTP/3, within the "transport" Category, there would be "packet_sent" and "packet_received" events.¶

Logging category and type separately conceptually allows for fast and high-level filtering based on category and the re-use of event types across categories. However, it also considerably inflates the log size and this flexibility is not used extensively in practice at the time of writing.¶

As such, the default approach in qlog is to concatenate both field values using the ":" character in the "name" field, as can be seen in Figure 19. As such, qlog category and type names MUST NOT include this character.¶

JSON serialization using separate fields:
{
    "category": "transport",
    "type": "packet_sent"
}

JSON serialization using ":" concatenated field:
{
    "name": "transport:packet_sent"
}

Certain serializations CAN emit category and type as separate fields, and qlog tools SHOULD be able to deal with both the concatenated "name" field, and the separate "category" and "type" fields. Text-based serializations however are encouraged to employ the concatenated "name" field for efficiency.¶

3.4.3. Data

The data field is a generic object. It contains the per-event metadata and its form and semantics are defined per specific sort of event. For example, data field value definitons for QUIC and HTTP/3 can be found in [QLOG-QUIC] and [QLOG-H3].¶

This field is defined here as a CDDL extension point (a "socket" or "plug") named $ProtocolEventBody. Other documents MUST properly extend this extension point when defining new data field content options to enable automated validation of aggregated qlog schemas.¶

The only common field defined for the data field is the trigger field, which is discussed in Section 3.4.5.¶

Definition:¶

; The ProtocolEventBody is any key-value map (e.g., JSON object)
; only the optional trigger field is defined in this document
$ProtocolEventBody /= {
    ? trigger: text
    * text => any
}
; event documents are intended to extend this socket by using:
; NewProtocolEvents = EventType1 / EventType2 / ... / EventTypeN
; $ProtocolEventBody /= NewProtocolEvents

One purely illustrative example for a QUIC "packet_sent" event is shown in Figure 21:¶

TransportPacketSent = {
    ? packet_size: uint16
    header: PacketHeader
    ? frames:[* QuicFrame]
    ? trigger: "pto_probe" / "retransmit_timeout" / "bandwidth_probe"
}

could be serialized as

{
    packet_size: 1280,
    header: {
        packet_type: "1RTT",
        packet_number: 123
    },
    frames: [
        {
            frame_type: "stream",
            length: 1000,
            offset: 456
        },
        {
            frame_type: "padding"
        }
    ]
}

3.4.4. protocol_type

The "protocol_type" array field indicates to which protocols (or protocol "stacks") this event belongs. This allows a single qlog file to aggregate traces of different protocols (e.g., a web server offering both TCP+HTTP/2 and QUIC+HTTP/3 connections).¶

Definition:¶

ProtocolType = [+ text]

For example, QUIC and HTTP/3 events have the "QUIC" and "HTTP3" protocol_type entry values, see [QLOG-QUIC] and [QLOG-H3].¶

Typically however, all events in a single trace are of the same few protocols, and this array field is logged once in "common_fields", see Section 3.4.7.¶

3.4.5. Triggers

Sometimes, additional information is needed in the case where a single event can be caused by a variety of other events. In the normal case, the context of the surrounding log messages gives a hint as to which of these other events was the cause. However, in highly-parallel and optimized implementations, corresponding log messages might separated in time. Another option is to explicitly indicate these "triggers" in a high-level way per-event to get more fine-grained information without much additional overhead.¶

In qlog, the optional "trigger" field contains a string value describing the reason (if any) for this event instance occuring, see Section 3.4.3. While this "trigger" field could be a property of the qlog Event itself, it is instead a property of the "data" field instead. This choice was made because many event types do not include a trigger value, and having the field at the Event-level would cause overhead in some serializations. Additional information on the trigger can be added in the form of additional member fields of the "data" field value, yet this is highly implementation-specific, as are the trigger field's string values.¶

One purely illustrative example of some potential triggers for QUIC's "packet_dropped" event is shown in Figure 23:¶

TransportPacketDropped = {
    ? packet_type: PacketType
    ? raw_length: uint16

    ? trigger: "key_unavailable" / "unknown_connection_id" /
               "decrypt_error" / "unsupported_version"
}

3.4.6. group_id

As discussed in Section 3.3, a single qlog file can contain several traces taken from different vantage points. However, a single trace from one endpoint can also contain events from a variety of sources. For example, a server implementation might choose to log events for all incoming connections in a single large (streamed) qlog file. As such, we need a method for splitting up events belonging to separate logical entities.¶

The simplest way to perform this splitting is by associating a "group identifier" to each event that indicates to which conceptual "group" each event belongs. A post-processing step can then extract events per group. However, this group identifier can be highly protocol and context-specific. In the example above, we might use QUIC's "Original Destination Connection ID" to uniquely identify a connection. As such, they might add a "ODCID" field to each event. However, a middlebox logging IP or TCP traffic might rather use four-tuples to identify connections, and add a "four_tuple" field.¶

As such, to provide consistency and ease of tooling in cross-protocol and cross-context setups, qlog instead defines the common "group_id" field, which contains a string value. Implementations are free to use their preferred string serialization for this field, so long as it contains a unique value per logical group. Some examples can be seen in Figure 25.¶

Definition:¶

GroupID = text

JSON serialization example for events grouped by four tuples and QUIC connection IDs:¶

events: [
    {
        time: 1553986553579,
        protocol_type: ["TCP", "TLS", "HTTP2"],
        group_id: "ip1=2001:67c:1232:144:9498:6df6:f450:110b,
                   ip2=2001:67c:2b0:1c1::198,port1=59105,port2=80",
        name: "transport:packet_received",
        data: { ... },
    },
    {
        time: 1553986553581,
        protocol_type: ["QUIC","HTTP3"],
        group_id: "127ecc830d98f9d54a42c4f0842aa87e181a",
        name: "transport:packet_sent",
        data: { ... },
    }
]

Note that in some contexts (for example a Multipath transport protocol) it might make sense to add additional contextual per-event fields (for example "path_id"), rather than use the group_id field for that purpose.¶

Note also that, typically, a single trace only contains events belonging to a single logical group (for example, an individual QUIC connection). As such, instead of logging the "group_id" field with an identical value for each event instance, this field is typically logged once in "common_fields", see Section 3.4.7.¶

3.4.7. common_fields

As discussed in the previous sections, information for a typical qlog event varies in three main fields: "time", "name" and associated data. Additionally, there are also several more advanced fields that allow mixing events from different protocols and contexts inside of the same trace (for example "protocol_type" and "group_id"). In most "normal" use cases however, the values of these advanced fields are consistent for each event instance (for example, a single trace contains events for a single QUIC connection).¶

To reduce file size and making logging easier, qlog uses the "common_fields" list to indicate those fields and their values that are shared by all events in this component trace. This prevents these fields from being logged for each individual event. An example of this is shown in Figure 26.¶

JSON serialization with repeated field values
per-event instance:

{
    events: [{
            group_id: "127ecc830d98f9d54a42c4f0842aa87e181a",
            protocol_type: ["QUIC","HTTP3"],
            time_format: "relative",
            reference_time: 1553986553572,

            time: 2,
            name: "transport:packet_received",
            data: { ... }
        },{
            group_id: "127ecc830d98f9d54a42c4f0842aa87e181a",
            protocol_type: ["QUIC","HTTP3"],
            time_format: "relative",
            reference_time: 1553986553572,

            time: 7,
            name: "http:frame_parsed",
            data: { ... }
        }
    ]
}

JSON serialization with repeated field values instead
extracted to common_fields:

{
    common_fields: {
        group_id: "127ecc830d98f9d54a42c4f0842aa87e181a",
        protocol_type: ["QUIC","HTTP3"],
        time_format: "relative",
        reference_time: 1553986553572
    },
    events: [
        {
            time: 2,
            name: "transport:packet_received",
            data: { ... }
        },{
            7,
            name: "http:frame_parsed",
            data: { ... }
        }
    ]
}

The "common_fields" field is a generic dictionary of key-value pairs, where the key is always a string and the value can be of any type, but is typically also a string or number. As such, unknown entries in this dictionary MUST be disregarded by the user and tools (i.e., the presence of an uknown field is explicitly NOT an error).¶

The list of default qlog fields that are typically logged in common_fields (as opposed to as individual fields per event instance) are shown in the listing below:¶

Definition:¶

CommonFields = {
    ? time_format: TimeFormat
    ? reference_time: float64

    ? protocol_type: ProtocolType
    ? group_id: GroupID

    * text => any
}

Tools MUST be able to deal with these fields being defined either on each event individually or combined in common_fields. Note that if at least one event in a trace has a different value for a given field, this field MUST NOT be added to common_fields but instead defined on each event individually. Good example of such fields are "time" and "data", who are divergent by nature.¶

5.2.1. error

Importance: Core¶

Used to log details of an internal error that might not get reflected on the wire.¶

Definition:¶

GenericError = {
    ? code: uint64
    ? message: text
}

5.2.2. warning

Importance: Base¶

Used to log details of an internal warning that might not get reflected on the wire.¶

Definition:¶

GenericWarning = {
    ? code: uint64
    ? message: text
}

5.2.3. info

Importance: Extra¶

Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages.¶

Definition:¶

GenericInfo = {
    message: text
}

5.2.4. debug

Importance: Extra¶

Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages.¶

Definition:¶

GenericDebug = {
    message: text
}

5.2.5. verbose

Importance: Extra¶

Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages.¶

Definition:¶

GenericVerbose = {
    message: text
}

5.3.1. scenario

Importance: Extra¶

Used to specify which specific scenario is being tested at this particular instance. This could also be reflected in the top-level qlog's summary or configuration fields, but having a separate event allows easier aggregation of several simulations into one trace (e.g., split by group_id).¶

Definition:¶

SimulationScenario = {
    ? name: text
    ? details: {* text => any }
}

5.3.2. marker

Importance: Extra¶

Used to indicate when specific emulation conditions are triggered at set times (e.g., at 3 seconds in 2% packet loss is introduced, at 10s a NAT rebind is triggered).¶

Definition:¶

SimulationMarker = {
    ? type: text
    ? message: text
}

6.1.1. I-JSON

For some use cases, it should be taken into account that not all popular JSON parsers support the full JSON format. Especially for parsers integrated with the JavaScript programming language (e.g., Web browsers, NodeJS), users are recommended to stick to a JSON subset dubbed [I-JSON] (or Internet-JSON).¶

One of the key limitations of JavaScript and thus I-JSON is that it cannot represent full 64-bit integers in standard operating mode (i.e., without using BigInt extensions), instead being limited to the range of [-(2**53)+1, (2**53)-1]. In these circumstances, Appendix E of [CDDL] recommends defining new CDDL types for int64 and uint64 that limit their values to this range.¶

While this can be sensible and workable for most use cases, some protocols targeting qlog serialization (e.g., QUIC, HTTP/3), might require full uint64 variables in some (rare) circumstances. In these situations, it should be allowed to also use the string-based representation of uint64 values alongside the numerical representation. Concretely, the following definition of uint64 should override the original and (web-based) tools should take into account that a uint64 field can be either a number or string.¶

uint64 = text / uint .size 8

6.1.2. Truncated values

For some use cases (e.g., limiting file size, privacy), it can be necessary not to log a full raw blob (using the hexstring type) but instead a truncated value (for example, only the first 100 bytes of an HTTP response body to be able to discern which file it actually contained). In these cases, the original byte-size length cannot be obtained from the serialized value directly.¶

As such, all qlog schema definitions SHOULD include a separate, length-indicating field for all fields of type hexstring they specify, see for example Section 5.1. This not only ensures the original length can always be retrieved, but also allows the omission of any raw value bytes of the field completely (e.g., out of privacy or security considerations).¶

To reduce overhead however and in the case the full raw value is logged, the extra length-indicating field can be left out. As such, tools MUST be able to deal with this situation and derive the length of the field from the raw value if no separate length-indicating field is present. The main possible permutations are shown by example in Figure 37.¶

// both the full raw value and its length are present
// (length is redundant)
{
    "raw_length": 5,
    "raw": "051428abff"
}

// only the raw value is present, indicating it
// represents the fields full value the byte
// length is obtained by calculating raw.length / 2
{
    "raw": "051428abff"
}

// only the length field is present, meaning the
// value was omitted
{
    "raw_length": 5,
}

// both fields are present and the lengths do not match:
// the value was truncated to the first three bytes.
{
    "raw_length": 5,
    "raw": "051428"
}

6.2.1. Supporting JSON Text Sequences in tooling

Note that JSON Text Sequences are not supported in most default programming environments (unlike normal JSON). However, several custom JSON-SEQ parsing libraries exist in most programming languages that can be used and the format is easy enough to parse with existing implementations (i.e., by splitting the file into its component records and feeding them to a normal JSON parser individually, as each record by itself is a valid JSON object).¶

6.3.1. Data structure optimizations

The first general category of optimizations is to alter the representation of data within an JSON(-SEQ) qlog file to reduce file size.¶

The first option is to employ a scheme similar to the CSV (comma separated value [RFC4180]) format, which utilizes the concept of column "headers" to prevent repeating field names for each datapoint instance. Concretely for JSON qlog, several field names are repeated with each event (i.e., time, name, data). These names could be extracted into a separate list, after which qlog events could be serialized as an array of values, as opposed to a full object. This approach was a key part of the original qlog format (prior to draft-02) using the "event_fields" field. However, tests showed that this optimization only provided a mean file size reduction of 5% (100MB to 95MB) while significantly increasing the implementation complexity, and this approach was abandoned in favor of the default JSON setup. Implementations using this format should not employ a separate file extension (as it still uses JSON), but rather employ a new value of "JSON.namedheaders" (or "JSON-SEQ.namedheaders") for the "qlog_format" field (see Section 3).¶

The second option is to replace field values and/or names with indices into a (dynamic) lookup table. This is a common compression technique and can provide significant file size reductions (up to 50% in our tests, 100MB to 50MB). However, this approach is even more difficult to implement efficiently and requires either including the (dynamic) table in the resulting file (an approach taken by for example Chromium's NetLog format) or defining a (static) table up-front and sharing this between implementations. Implementations using this approach should not employ a separate file extension (as it still uses JSON), but rather employ a new value of "JSON.dictionary" (or "JSON-SEQ.dictionary") for the "qlog_format" field (see Section 3).¶

As both options either proved difficult to implement, reduced qlog file readability, and provided too little improvement compared to other more straightforward options (for example Section 6.3.2), these schemes are not inherently part of qlog.¶

6.3.2. Compression

The second general category of optimizations is to utilize a (generic) compression scheme for textual data. As qlog in the JSON(-SEQ) format typically contains a large amount of repetition, off-the-shelf (text) compression techniques typically succeed very well in bringing down file sizes (regularly with up to two orders of magnitude in our tests, even for "fast" compression levels). As such, utilizing compression is recommended before attempting other optimization options, even though this might (somewhat) increase processing costs due to the additional compression step.¶

The first option is to use GZIP compression ([RFC1952]). This generic compression scheme provides multiple compression levels (providing a trade-off between compression speed and size reduction). Utilized at level 6 (a medium setting thought to be applicable for streaming compression of a qlog stream in commodity devices), gzip compresses qlog JSON files to 7% of their initial size on average (100MB to 7MB). For this option, the file extension .(s)qlog.gz SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ").¶

The second option is to use Brotli compression ([RFC7932]). While similar to gzip, this more recent compression scheme provides a better efficiency. It also allows multiple compression levels. Utilized at level 4 (a medium setting thought to be applicable for streaming compression of a qlog stream in commodity devices), brotli compresses qlog JSON files to 7% of their initial size on average (100MB to 7MB). For this option, the file extension .(s)qlog.br SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ").¶

Other compression algorithms of course exist (for example xz, zstd, and lz4). We mainly recommend gzip and brotli because of their tweakable behaviour and wide support in web-based environments, which we envision as the main tooling ecosystem (see also Section 8).¶

6.3.3. Binary formats

The third general category of optimizations is to use a more optimized (often binary) format instead of the textual JSON format. This approach inherently produces smaller files and often has better (de)serialization performance. However, the resultant files are no longer human readable and some formats require hard tradeoffs between flexibility for performance.¶

The first option is to use the CBOR (Concise Binary Object Representation [RFC7049]) format. For our purposes, CBOR can be viewed as a straighforward binary variant of JSON. As such, existing JSON qlog files can be trivially converted to and from CBOR (though slightly more work is needed for JSON-SEQ qlogs to convert them to CBOR-SEQ, see [RFC8742]). While CBOR thus does retain the full qlog flexibility, it only provides a 25% file size reduction (100MB to 75MB) compared to textual JSON(-SEQ). As CBOR support in programming environments is not as widespread as that of textual JSON and the format lacks human readability, CBOR was not chosen as the default qlog format. For this option, the file extension .(s)qlog.cbor SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ"). The media type should indicate both whether JSON or JSON Text Sequences are used, as well as whether CBOR or CBOR Sequences are used (see the table below).¶

A second option is to use a more specialized binary format, such as Protocol Buffers (protobuf). This format is battle-tested, has support for optional fields and has libraries in most programming languages. Still, it is significantly less flexible than textual JSON or CBOR, as it relies on a separate, pre-defined schema (a .proto file). As such, it it not possible to (easily) log new event types in protobuf files without adjusting this schema as well, which has its own practical challenges. As qlog is intended to be a flexible, general purpose format, this type of format was not chosen as its basic serialization. The lower flexibility does lead to significantly reduced file sizes. Our straightforward mapping of the qlog main schema and QUIC/HTTP3 event types to protobuf created qlog files 24% as large as the raw JSON equivalents (100MB to 24MB). For this option, the file extension .(s)qlog.protobuf SHOULD BE used. The "qlog_format" field should reflect the different internal format, for example: "qlog_format": "protobuf".¶

Note that binary formats can (and should) also be used in conjunction with compression (see Section 6.3.2). For example, CBOR compresses well (to about 6% of the original textual JSON size (100MB to 6MB) for both gzip and brotli) and so does protobuf (5% (gzip) to 3% (brotli)). However, these gains are similar to the ones achieved by simply compression the textual JSON equivalents directly (7%, see Section 6.3.2). As such, since compression is still needed to achieve optimal file size reductions event with binary formats, we feel the more flexible compressed textual JSON options are a better default for the qlog format in general.¶

6.3.4. Overview and summary

In summary, textual JSON was chosen as the main qlog format due to its high flexibility and because its inefficiencies can be largely solved by the utilization of compression techniques (which are needed to achieve optimal results with other formats as well).¶

Still, qlog implementers are free to define other qlog formats depending on their needs and context of use. These formats should be described in their own documents, the discussion in this document mainly acting as inspiration and high-level guidance. Implementers are encouraged to add concrete qlog formats and definitions to the designated public repository.¶

The following table provides an overview of all the discussed qlog formatting options with examples:¶