Warp - Segmented Live Video Transport

1.1. Terms and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

Commonly used terms in this document are described below.¶

Frame:

An image to be rendered at a specific point in time.¶

I-frame:

A frame that does not depend on the contents of other frames.¶

Group of pictures (GOP):

A I-frame followed by a sequential series of dependent frames.¶

Group of samples:

A sequential series of audio samples starting at a given timestamp.¶

Segment:

A sequence of video frames and/or audio samples serialized into a container.¶

Presentation Timestamp (PTS):

A point in time when video/audio should be presented to the viewer.¶

Media producer:

An endpoint sending media over the network.¶

Media consumer:

An endpoint receiving media over the network.¶

Congestion:

Packet loss and queuing caused by degraded or overloaded networks.¶

2.1. Initialization

Initialization segments contain track metadata but no sample data.¶

Initialization segments MUST consist of a File Type Box ('ftyp') followed by a Movie Box ('moov'). This Movie Box consists of Movie Header Boxes ('mvhd'), Track Header Boxes ('tkhd'), Track Boxes ('trak'), followed by a final Movie Extends Box ('mvex'). These boxes MUST NOT contain any samples and MUST have a duration of zero.¶

Note that a Common Media Application Format Header [CMAF] meets all these requirements.¶

2.2. Media

Media segments contain media samples for a single track.¶

Media segments MUST consist of a Segment Type Box ('styp') followed by at least one media fragment. Each media fragment consists of a Movie Fragment Box ('moof') followed by a Media Data Box ('mdat'). The Media Fragment Box MUST contain a Movie Fragment Header Box ('mfhd') and Track Box ('trak') with a Track ID ('track_ID') matching a Track Box in the initialization segment.¶

Note that a Common Media Application Format Segment [CMAF] meets all these requirements.¶

3.1. Messages

Messages are used to control playback or carry metadata about upcoming segments.¶

A Warp Box ('warp') is a top-level MP4 box as defined in [ISOBMFF]. The contents of this box is a warp message. See the messages section (Section 4) for the encoding and types available.¶

3.2. Segments

Segments are transferred over streams alongside messages. Each segment MUST be preceded by an init (Section 4.1) or media (Section 4.2) message, indicating the type of segment and providing additional metadata.¶

The media producer SHOULD send each segment as a unique stream to avoid head-of-line blocking. The media producer MAY send multiple segments over a single stream, for simplicity, when head-of-line blocking is desired.¶

A segment is the smallest unit of delivery, as the tail of a segment can be safely delayed/dropped without decode errors. A future version of Warp will support layered coding (additional QUIC streams) to enable dropping or downscalling frames in the middle of a segment.¶

3.3. Prioritization

Warp utilizes precedence to deliver the most important content during congestion.¶

The media producer assigns a numeric presidence to each stream. This is a strict prioritzation scheme, such that any available bandwidth is allocated to streams in descending order. QUIC supports stream prioritization but does not standardize any mechanisms; see Section 2.3 in [QUIC]. The media producer MUST support sending priorized streams. The media producer MAY choose to delay retransmitting lower priority streams when possible within QUIC flow control limits.¶

The media consumer determines how long to wait for a given segment (buffer size) before skipping ahead. The media consumer MAY cancel a skipped segment to save bandwidth, or leave it downloading in the background (ex. to support rewind).¶

Prioritization allows a single media producer to support multiple media consumers with different latency targets. For example, one consumer could have a 1s buffer to minimize latency, while another conssumer could have a 5s buffer to improve quality, while a yet another consumer could have a 30s buffer to receive all media (ex. VOD recorder).¶

  if is_audio:
    precedence = timestamp + 3s
  else:
    precedence = timestamp

  precedence = -timestamp

3.4. Cancellation

During congestion, prioritization intentionally cause stream starvation for the lowest priority streams. Some form of starvation will last until the network fully recovers, which may be indefinite.¶

The media consumer SHOULD cancel a stream (via a QUIC STOP_SENDING frame) after it has been skipped to save bandwidth. The media producer SHOULD reset the lowest priority stream (via QUIC RESET_STREAM frame) when nearing resource limits. Both of these actions will effectively drop the tail of the segment.¶

3.5. Middleware

Media may go through multiple hops and processing steps on the path from the broadcaster to player. The full effectiveness of warp as an end-to-end protocol depends on middleware support.¶

• Middleware MUST maintain stream idependence to avoid introducing head-of-line blocking.¶
• Middleware SHOULD maintain stream prioritization when traversing networks susceptible to congestion.¶
• Middleware MUST forward the priority message (Section 4.3) for downstream servers.¶

4.1. init

The init message indicates that the remainder of the stream contains an initialization segment.¶

{
  init: {
    id: int
  }
}

id:

Incremented by 1 for each initialization segment.¶

4.2. media

The media message contains metadata about the next media segment in the stream.¶

{
  segment: {
    init: int,
    timestamp: int,
  }
}

init:

The id of the cooresponding initialization segment. A decoder MUST block until the coorespending init message to arrive.¶

timestamp:

The presentation timestamp in milliseconds for the first frame/sample in the next segment. This timestamp MUST be used when it does not match the timestamp in the media container.¶

4.3. priority

The priority message informs middleware about the intended priority of the current stream. Any middleware MAY ignore this value but SHOULD forward it.¶

{
  priority: {
    precedence: int,
  }
}

precedence:

An integer value, indicating that any available bandwidth SHOULD be allocated to streams in descending order.¶

4.4. Extensions

Custom messages MUST start with x-. Unicode LATIN SMALL LETTER X (U+0078) followed by HYPHEN-MINUS (U+002D).¶

Custom messages SHOULD use a unique prefix to reduce collisions. For example: x-twitch-load would contain identification required to start playback of a Twitch stream.¶