Media Timed Events

3.1 Synchronised event triggering

Use cases for media timed events include:

• Displaying of images alongside an audio stream. For example, in RadioVIS in DVB, an emsg event contains an image URL, which the user agent requests. In this use case, synchronization of the image rendering to within a second or so is acceptable ([ DVB-DASH ], section 9.1.7). Such image displays may be activated and deactivated at different intervals in the duration of the associated emsg event.
• Notifying the DASH player Web application that it should refresh its copy of the MPD document ([ MPEGDASH ], section 5.10.4). This is an alternative to setting a cache duration in the HTTP response, so the client can refresh the MPD when it actually changes, and reduces the load on HTTP servers caused by frequent server requests.

Reference: M&E IG call 1 Feb 2018: Minutes , [ DASH-EVENTING ].

3.2 Synchronized rendering of web resources

[ WebVMT ] is a format for metadata cues, synchronised with the timed media file, that can drive an online map, e.g., OpenStreetMap, rendered in a separate HTML element alongside the media element on the web page. The media playhead position controls presentation and animation of the map, e.g., pan and zoom, and allows annotations to be added and removed, e.g., markers, at specified times during media playback. Control can also be overridden by the user with the usual interactive features of the map at any time, e.g., zoom. Concrete examples are provided by the tech demos at the WebVMT website.

Reference: M&E IG TF call 17 Sept 2018: Minutes .

During a live media presentation, dynamic and unpredictable events may occur which causes temporary suspension of the media presentation. During that suspension interval, auxiliary content such as the presentation of UI controls and media files, may be unavailable. Depending on the specific user engagement (or not) with the UI controls and the time at which any such engagement occurs, specific web resources may be rendered at defined times in a synchronized manner. For example, a multimedia A/V clip along with subtitles corresponding to an advertisement, and which were previously downloaded and cached by the UA, are played out.

3.3 Rendering of Web content embedded in media containers

Media-timed events can be used to trigger retrieval and/or rendering of web resources. Such resources can be used to enhance user experience in the context of media that is being rendered. Some examples include include:

• Display of social media feeds corresponding to a live broadcast such as a sporting event.
• Banner advertisements for sponsored content.
• Accessibility-related assets, such as large print rendering of captions.

4.1 MPEG-DASH

In MPEG DASH, events may be conveyed either as in-band events, e.g., as emsg boxes in ISO BMFF files, or out-of-band, via an EventStream fragment in the MPD (Media Presentation Description) document (i.e., by an instance of the EventStream child of the MPD.Period element). In addition, the MPD document may advertise the presence of emsg events in the ISO BMFF content for given schemas.

An emsg event contains the following information, as specified in [ MPEGDASH ], section 5.10.3.3:

• scheme_id_uri — A URI that identifies the message scheme
• value — The event value (string)
• timescale — Timescale units, in ticks per second
• presentation_time_delta — Presentation time delta (with respect to the media segment), in timescale units
• event_duration — Event duration, in timescale units
• id — Event message identifier
• message_data — Message body (may be empty)

The presence of emsg events in the media stream is signalled in the DASH manifest document (MPD), using an EventStream XML element ([ MPEGDASH ], section 5.10.2).

4.2 HbbTV

HbbTV includes support for emsg events ([ DVB-DASH ], section 9.1) and requires this be mapped to DataCue ([ HBBTV ], section 9.3.2). The HbbTV device test suite includes test pages and streams for this ([ HBBTV-TESTS ]). This feature is included in user agents shipping in connected TVs across Europe from 2017. HbbTV has a reference app and content for DASH+DRM which includes emsg support. As well as HbbTV devices, this reference app and content have been tested on Microsoft Edge with MSE and EME although emsg support does not work there.

4.3 DASH Industry Forum APIs for Interactivity

The DASH-IF InterOp Working Group has an ongoing work item, DAInty , , "DASH APIs for Interactivity", which aims to specify a set of APIs between the DASH client/player and interactivity-capable applications, for both Web and native applications. The origin of this work is a related 3GPP work item on Service Interactivity ([ 3GPP-INTERACTIVITY-WID ], [ 3GPP-INTERACTIVITY-TR ]). The objective is to provide service enablers for user engagement with auxiliary content and UIs on mobile device during live or time-shifted viewing of streaming content delivered over 3GPP broadcast or unicast bearers, and the measurement and reporting of such interactive consumption.

Two APIs are being developed that are relevant to the scope of the present document:

• Application subscription/DASH client dispatch of DASH event stream messages containing interactivity information. Events can be delivered in-band ( emsg ) and/or as MPD events.
• Application subscription/DASH client dispatch of ISO BMFF Timed Metadata tracks providing similar functionality to DASH event streams.

Two modes for dispatching events are defined. In Mode 1, events are dispatched at the time the event arrives, and in Mode 2, events are dispatched at the given time on the media timeline. The "arrival" of events from the DASH client perspective may be either static or pre-provisioned, in the case MPD Events, or dynamic in the case of inband events carried in the emsg . The application can register with the DASH client which Mode to use.

Reference: M&E IG, Media Timed Events Task Force call 20 Aug 2018: Minutes .

4.4 BBC Subtitle Guidelines

The BBC Subtitle Guidelines ([ BBC-SUBTITLES ]) describe best practice for authoring subtitles or captions. In particular, the guidelines state:

5.2 Match subtitle to pace of speaking

The subtitles should match the pace of speaking as closely as possible. Ideally, when the speaker is in shot, your subtitles should not anticipate speech by more than 1.5 seconds or hang up on the screen for more than 1.5 seconds after speech has stopped.

6.1 Match subtitles to shot

It is likely to be less tiring for the viewer if shot changes and subtitle changes occur at the same time. Many subtitles therefore start on the first frame of the shot and end on the last frame.

6.2 Maintain a minimum gap when mismatched

If you have to let a subtitle hang over a shot change, do not remove it too soon after the cut. The duration of the overhang will depend on the content.

6.3 Avoid straddling shot changes

Avoid creating subtitles that straddle a shot change (i.e., a subtitle that starts in the middle of shot one and ends in the middle of shot two). To do this, you may need to split a sentence at an appropriate point, or delay the start of a new sentence to coincide with the shot change.

To meet these requirements, the playback system must honour the provided timings. Subtitles for video are typically authored against video at a nominal frame rate, e.g., 25 frames per second (which corresponds to 40 milliseconds per frame). The actual video frame rate may be adjusted dynamically according to the video encoding, but the subtitle timing must remain the same ([ EBU-TT-D ], Annex E).

4.5 SCTE Media Splicing Requirements

The Society for Cable and Televison Engineers (SCTE) has produced the SCTE-35 specification "Digital Program Insertion Cueing for Cable" [ SCTE-35 ]. In it, the requirements for splicing (Sec. 9.1) are "In order to give advance warning of the impending splice (a pre-roll function), the splice_insert() command could be sent multiple times before the splice point. For example, the splice_insert() command could be sent at 8, 5, 4 and 2 seconds prior to the packet containing the related splice point. In order to meet other splicing deadlines in the system, any message received with less than 4 seconds of advance notice may not create the desired result. "

This places an implicit requirement on the user agent in handling media-timed events related to insertion cues. The content originator may provide the cue in advance with as little as 2 seconds of the insertion time. Therefore the propagation of the event data associated with the insertion cue to the application by the user agent should be considerably less than 2 seconds.

4.6 MPEG Working Draft on Carriage of Web Resources in ISO BMFF

[ WEB-ISOBMFF ] is a draft document that specifies the use of ISO BMFF tools for the storage and delivery of web data. The specified storage is designed to enable enriching audio/video content, as well as audio-only content, with synchronized, animated, interactive web data, including overlays.

4.7 WebVTT

[ WEBVTT ] is a W3C specification that provides a format for web video text tracks. A VTTCue is a text track cue, and may have attributes that affect rendering of the cue text on a web page. WebVTT metadata cues are text that is time-aligned.

5.1 Synchronized event triggering

interface DataCue : TextTrackCue {
  attribute ArrayBuffer data; // Always empty
  // Proposed extensions.
  attribute any value;
  readonly attribute DOMString type;
};

value = {
  key: String
  data: String | Number | Array | ArrayBuffer | Object
  locale: String
}

5.2 Synchronized rendering of web resources

5.3 Rendering of Web content embedded in media containers

There is no API for surfacing Web content embedded in ISO BMFF containers into the browser (e.g., the HTMLCue proposal discussed at TPAC 2015 ).

6.1 Subscribing to event streams

The API should allow Web applications to subscribe to receive specific event types. For example, to support DASH emsg and MPD events, the API should allow subscription by id and (optional) value . This is to make receiving events opt-in from the application point of view. The user agent should deliver only those events to a Web application for which the application has subscribed. The API should also allow Web applications to unsubscribe from specific event streams by event type.

6.2 Out-of-band events

To be able to handle out of band events, the API must allow Web applications to create events to be added to the media timeline, to be triggered by the user agent. The API should allow the Web application to provide all necessary parameters to define the event, including start and end times, event type, and data payload. The payload should be any data type (e.g., the set of types supported by the WebKit DataCue ). For DASH MPD events, the event type is defined by the id and (optional) value fields.

6.3 Event triggering

For those events that the application has subscribed to receive, the API must:

• Generate a JavaScript event when an in-band media timed event is parsed from the media container or media stream (DAInty Mode 1)
• Generate a JavaScript event when the current media playback position reaches the start time of a media timed event during playback (DAInty Mode 2) 2). This applies equally to in-band events that the user agent has extracted from the media container, and out-of-band events added by the Web application.

The user agent API must ensure provide guarantees that no events are can be missed during linear playback of the media.

6.4 In-band event processing

We recommend updating [ INBANDTRACKS ] to describe handling of in-band media timed events supported on the web platform, following a registry approach with one specification per media format that describes the event details for that format. In particular, we recommend that browser engines support emsg events.

6.5 Synchronization

The time marches on algorithm should be reviewed and updated to ensure that events are delivered to the Web application within time constraints described elsewhere in this report.