Media Timed Events

3.1 Audio stream with titles and images

A media content provider wants to provide visual information alongside an audio stream, such as an image of the artist and title of the current playing track, to give users live information about the content they are listening to.

Examples include HLS timed metadata [ HLS-TIMED-METADATA ], which uses in-band ID3 metadata to carry the image content, and RadioVIS in DVB ([ DVB-DASH ], section 9.1.7), which defines in-band event messages that contain image URLs and text messages to be displayed, with information about when the content should be displayed in relation to the media timeline. timeline .

3.2 MPEG DASH MPEG-DASH manifest expiry notifications

Section 5.10.4 of [ MPEGDASH ] describes a DASH an MPEG-DASH specific event that is used to notify a DASH player web application that it should refresh its copy of the manifest (MPD) document. An in-band emsg event is used an alternative to setting a cache duration in the response to the HTTP request for the manifest, so the client can refresh the MPD when it actually changes, so reducing the load on HTTP servers caused by frequent server requests.

Reference: M&E IG call 1 Feb 2018: Minutes , [ DASH-EVENTING ].

3.3 Synchronized map animations

A user records footage with metadata, including geolocation, on a mobile video device, e.g., drone or dashcam, to share on the web alongside a map, e.g., OpenStreetMap.

[ WebVMT ] is a an open format for metadata cues, synchronized with a timed media file, that can be used to drive an online map, e.g., OpenStreetMap, map rendered in a separate HTML element alongside the media element on the web page. The media playhead position controls presentation and animation of the map, e.g., pan and zoom, and allows annotations to be added and removed, e.g., markers, at specified times during media playback. Control can also be overridden by the user with the usual interactive features of the map at any time, e.g., zoom. Concrete examples are provided by the tech demos at the WebVMT website.

Reference: M&E IG TF call 17 Sept 2018: Minutes .

3.4 Media analysis visualization

A video image analysis system processes a media stream to detect and recognize objects shown in the video. This system generates metadata describing the objects, including timestamps that describe the when the objects are visible, together with position information (e.g., bounding boxes). A web application then uses this timed metadata to overlay labels and annotations on the video using HTML and CSS.

3.5 Presentation of auxiliary content in live media

During a live media presentation, dynamic and unpredictable events may occur which cause temporary suspension of the media presentation. During that suspension interval, auxiliary content such as the presentation of UI controls and media files, may be unavailable. Depending on the specific user engagement (or not) with the UI controls and the time at which any such engagement occurs, specific web resources may be rendered at defined times in a synchronized manner. For example, a multimedia A/V clip along with subtitles corresponding to an advertisement, and which were previously downloaded and cached by the UA, are played out.

4.1 MPEG-DASH

MPEG-DASH is an adaptive bitrate streaming technique in which the audio and video media is partitioned into segments. The Media Presentation Description (MPD) is an XML document that contains metadata required by a DASH client to access the media segments and to provide the streaming service to the user. The media segments can use any codec, typically within a fragmented MP4 (ISO BMFF) container or MPEG-2 transport stream.

In MPEG DASH, MPEG-DASH, media timed events may be conveyed delivered either as in-band events, e.g., as or out-of-band :

• In-band events are emsg boxes in ISO BMFF files, or out-of-band, via an EventStream fragment files. The presence of emsg events in the media container for given event schemes is signalled in the MPD (Media Presentation Description) document (i.e., by using an instance of the EventStream child of the XML element ([ MPEGDASH ], section 5.10.2).
• Out-of-band events are represented by MPD.Period Event element). In addition, the MPD document may advertise the presence of XML elements contained within an emsg EventStream events element in the ISO BMFF content for given schemas. MPD.

An emsg event contains the following information, as specified in [ MPEGDASH ], section 5.10.3.3:

• scheme_id_uri — A URI that identifies the message scheme
• value — The event value (string)
• timescale — Timescale units, in ticks per second
• presentation_time_delta — Presentation time delta (with respect to the media segment), in timescale units
• event_duration — Event duration, in timescale units
• id — Event message identifier
• message_data — Message body (may be empty)

4.2 HbbTV

HbbTV is an interactive TV application standard that supports both broadcast (DVB) media delivery, and internet streaming using MPEG-DASH. The HbbTV application environment is based on HTML and JavaScript. MPEG-DASH streaming is implemented nativey by the user agent, rather than through a JavaScript web application using Media Source Extensions.

HbbTV includes support for emsg events ([ DVB-DASH ], section 9.1) and requires this be mapped to HTML5 DataCue ([ HBBTV ], section 9.3.2). The HbbTV device test suite includes test pages and streams for this ([ revision of HTML5 referenced by [ HBBTV-TESTS HBBTV ]). ] is [ html51-20151008 ]. This feature is included in user agents shipping in connected TVs across Europe from 2017.

The HbbTV device test suite includes test pages and streams that cover emsg support ([ HBBTV-TESTS ]). HbbTV has a reference app application and content for DASH+DRM which includes emsg support. As well as HbbTV devices, this reference app and content have been tested on Microsoft Edge with MSE and EME although emsg support does not work there.

4.3 DASH Industry Forum APIs for Interactivity

The DASH-IF InterOp Working Group has an ongoing work item, DAInty , "DASH APIs for Interactivity", which aims to specify a set of APIs between the DASH client/player and interactivity-capable applications, for both web and native applications. The origin of this work is a related 3GPP work item on Service Interactivity ([ 3GPP-INTERACTIVITY-WID ], [ 3GPP-INTERACTIVITY-TR ]). The objective is to provide service enablers for user engagement with auxiliary content and UIs on mobile device during live or time-shifted viewing of streaming content delivered over 3GPP broadcast or unicast bearers, and the measurement and reporting of such interactive consumption.

Two APIs are being developed that are relevant to the scope of the present document:

• Application subscription/DASH client dispatch of DASH event stream messages containing interactivity information. Events can be delivered in-band ( emsg ) and/or as MPD events.
• Application subscription/DASH client dispatch of ISO BMFF Timed Metadata tracks providing similar functionality to DASH event streams.

Two modes for dispatching events are defined. In Mode 1, events are dispatched at the time the event arrives, and in Mode 2, events are dispatched at the given time on the media timeline. timeline . The "arrival" of events from the DASH client perspective may be either static or pre-provisioned, in the case MPD Events, or dynamic in the case of inband events carried in the emsg . The application can register with the DASH client which Mode to use.

Reference: M&E IG, Media Timed Events Task Force call 20 Aug 2018: Minutes .

4.4 BBC Subtitle Guidelines

The BBC Subtitle Guidelines ([ BBC-SUBTITLES ]) describe best practice for authoring subtitles or captions. In particular, the guidelines state: 5.2 Match subtitle to pace of speaking The subtitles should match place requirements on the pace synchronization accuracy of speaking as closely as possible. Ideally, when the speaker is caption rendering. For example, in shot, your subtitles should not anticipate speech by more than 1.5 seconds or hang up on the screen for more than 1.5 seconds after speech has stopped. 6.1 Match subtitles to shot It section 6.1, caption authors are advised "it is likely to be less tiring for the viewer if shot changes and subtitle changes occur at the same time. Many subtitles therefore start on the first frame of the shot and end on the last frame. 6.2 Maintain a minimum gap when mismatched If you have to let a subtitle hang over a shot change, do not remove it too soon after the cut. The duration of the overhang will depend on the content. 6.3 Avoid straddling shot changes Avoid creating subtitles that straddle a shot change (i.e., a subtitle that starts in the middle of shot one and ends in the middle of shot two). To do this, you may need to split a sentence at an appropriate point, or delay the start of a new sentence to coincide with the shot change. frame."

To meet these requirements, the playback system must honour the provided timings. Subtitles for video are typically authored against video at a nominal frame rate, e.g., 25 frames per second (which second, which corresponds to 40 milliseconds per frame). frame. The actual video frame rate may be adjusted dynamically according to the video encoding, but the subtitle timing must remain the same ([ EBU-TT-D ], Annex E).

Where captions are rendered by application JavaScript code, in response to TextTrackCue events, this places a requirement on user agents for timely delivery of these events, so that application code can respond and render the cues.

4.5 SCTE Media Splicing Requirements

The Society for Cable and Televison Engineers (SCTE) has produced the SCTE-35 specification "Digital Program Insertion Cueing for Cable" [ SCTE-35 SCTE35 ]. In it, the requirements for splicing (Sec. 9.1) are "In order to give advance warning of the impending splice (a pre-roll function), the splice_insert() command could be sent multiple times before the splice point. For example, the splice_insert() command could be sent at 8, 5, 4 and 2 seconds prior to the packet containing the related splice point. In order to meet other splicing deadlines in the system, any message received with less than 4 seconds of advance notice may not create the desired result."

This places an implicit requirement on the user agent in handling media-timed events related to insertion cues. The content originator may provide the cue in advance with as little as 2 seconds of the insertion time. Therefore the propagation of the event data associated with the insertion cue to the application by the user agent should be considerably less than 2 seconds.

4.6 MPEG Working Draft on Carriage of Web Resources in ISO BMFF

[ WEB-ISOBMFF ] is a draft document that specifies the use of the ISO BMFF tools container format for the storage and delivery of web data. content. The specified storage goal is designed to enable enriching audio/video content, allow web resources (HTML, JavaScript, etc.) to be parsed from the storage and processed by a user agent at specific presentation times on the media timeline , and so be synchronized with other tracks within the container, such as well audio, video, and subtitles.

The Media & Entertainment Interest Group is actively tracking this work is open to discussing specific requirements for media timed events as audio-only content, with synchronized, animated, interactive web data, including overlays. development progresses.

4.7 WebVTT

[ WEBVTT ] is a W3C specification that provides a format for web video text tracks. A VTTCue is a text track cue, and may have attributes that affect rendering of the cue text on a web page. WebVTT metadata cues are text that is time-aligned. aligned to the media timeline . Web applications can use VTTCue to schedule out-of-band metadata events by serializing the event data to a string format (JSON, for example) when creating the cue, and deserializing the data when the cue is triggered.

5.1 Synchronized event triggering

interface DataCue : TextTrackCue {
  attribute ArrayBuffer data; // Always empty
  // Proposed extensions.
  attribute any value;
  readonly attribute DOMString type;
};

value = {
  key: String
  data: String | Number | Array | ArrayBuffer | Object
  locale: String
}

5.2 Synchronized rendering of web resources

5.3 Rendering of web content embedded in media containers

There is no API for surfacing web content embedded in ISO BMFF containers into the browser (e.g., the HTMLCue proposal discussed at TPAC 2015 ).

6.1 Subscribing to event streams

The API should allow web applications to subscribe to receive specific event types. For example, to support DASH emsg and MPD events, the API should allow subscription by id and (optional) value . This is to make receiving events opt-in from the application point of view. The user agent should deliver only those events to a web application for which the application has subscribed. The API should also allow web applications to unsubscribe from specific event streams by event type.

6.2 Out-of-band events

To be able to handle out of band events, the API must allow web applications to create events to be added to the media timeline, timeline , to be triggered by the user agent. The API should allow the web application to provide all necessary parameters to define the event, including start and end times, event type, and data payload. The payload should be any data type (e.g., the set of types supported by the WebKit DataCue ). For DASH MPD events, the event type is defined by the id and (optional) value fields.

6.3 Event triggering

For those events that the application has subscribed to receive, the API must:

• Generate a JavaScript event when an in-band media timed event is parsed from the media container or media stream (DAInty Mode 1)
• Generate a JavaScript event when the current media playback position reaches the start time of a media timed event during playback (DAInty Mode 2). This applies equally to in-band events that the user agent has extracted from the media container, and out-of-band events added by the web application.

The API must provide guarantees that no events can be missed during linear playback of the media.

6.4 In-band event processing

We recommend updating [ INBANDTRACKS ] to describe handling of in-band media timed events supported on the web platform, following a registry approach with one specification per media format that describes the event details for that format.

6.5 DASH MPEG-DASH events

We recommend that browser engines support DASH MPEG-DASH emsg in-band events and MPD out-of-band events, as part of their support for the MPEG Common Media Application Format (CMAF). (CMAF) [ MPEGCMAF ].

6.6 Synchronization

The time marches on algorithm should be reviewed and updated to ensure that events are delivered to the web application within time constraints described elsewhere in this report.