Media Session

1. Introduction

This section is non-normative.

Media is used extensively today, and the Web is one of the primary means of consuming media content. Many platforms can display media metadata, such as title, artist, album and album art on various UI elements such as notifications, media control center, device lockscreen, and wearable devices. This specification aims to enable web pages to specify the media metadata to be displayed in platform UI, and respond to media controls that may come from platform UI or media keys, thereby improving the user experience.

4. 5. The MediaSession interface

[Exposed=Window]
partial interface Navigator {
  [SameObject] readonly attribute MediaSession mediaSession;
};
enum MediaSessionPlaybackState {
  "none",
  "paused",
  "playing"
};
enum MediaSessionAction {
  "play",
  "pause",
  "seekbackward",
  "seekforward",
  "previoustrack",
  "nexttrack",
  "skipad",
  "stop",
  "seekto",
  "togglemicrophone",
  "togglecamera",
  "togglescreenshare",
  "hangup",
  "previousslide",
  "nextslide",
  "enterpictureinpicture",
  "voiceactivity"
};
callback MediaSessionActionHandler = undefined(MediaSessionActionDetails details);
[Exposed=Window]
interface MediaSession {
  attribute MediaMetadata? metadata;
  attribute MediaSessionPlaybackState playbackState;
  undefined setActionHandler(MediaSessionAction action, MediaSessionActionHandler? handler);
  undefined setPositionState(optional MediaPositionState state = {});
  Promise<undefined> setMicrophoneActive(boolean active);
  Promise<undefined> setCameraActive(boolean active);
  Promise<undefined> setScreenshareActive(boolean active);
};

A MediaSession object represents a media session for a given document and allows a document to communicate to the user agent some information about the playback and how to handle it.

A MediaSession has an associated metadata object represented by a MediaMetadata . It is initially null .

The mediaSession attribute MUST return the MediaSession instance associated with the Navigator object.

The metadata attribute reflects the MediaSession ’s metadata . On getting, it MUST return the MediaSession ’s metadata . On setting, it MUST run the following steps with value being the new value being set:

1. If the MediaSession ’s metadata is not null , set its media session to null .
2. Set the MediaSession ’s metadata to value .
3. If the MediaSession ’s metadata is not null , set its media session to the current MediaSession .
4. In parallel , run the update metadata algorithm .

The playbackState attribute represents the declared playback state of the media session , by which the session declares whether its browsing context is playing media or not. The initial value is none . On setting, the user agent MUST set the IDL attribute to the new value if it is a valid MediaSessionPlaybackState value. On getting, the user agent MUST return the last valid value that was set. The playbackState attribute is a hint for the user agent to determine whether the browsing context is playing or paused.

Setting playbackState may cause the actual playback state to change and run the media session actions update algorithm .

The MediaSessionPlaybackState enum is used to indicate whether a browsing context is playing media or not, the values are described as follows:

• none means the browsing context does not specify whether it’s playing or paused, it can only be used in the playbackState attribute.
• playing means the browsing context is currently playing media and it can be paused.
• paused means the browsing context has paused media and it can be resumed.

The setActionHandler(action, handler) method, when invoked, MUST run the update action handler algorithm with action and handler on the MediaSession .

The setPositionState( state ) method, when invoked MUST perform the following steps:

• If state is an empty dictionary, clear the position state and abort these steps.
• If state ’s duration is not present, throw a TypeError .
• If state ’s duration is negative or NaN , throw a TypeError .
• If state ’s position is not present, set it to zero.
• If state ’s position is negative or greater than duration , throw a TypeError .
• If state ’s playbackRate is not present, set it to 1.0.
• If state ’s playbackRate is zero, throw a TypeError .
• Update the position state and last position updated time .

The setMicrophoneActive(active) method indicates to the user agent the microphone capture state desired by the page (e.g. if the microphone is considered "inactive" by the page since it is no longer sending audio through a call, the page can invoke setMicrophoneActive(false) ). When invoked, it MUST perform the following steps:

1. Let document be this ’s relevant global object ’s associated Document .
2. Let captureKind be "microphone".
3. Return the result of running the update capture state algorithm with document , active and captureKind .

Similarly, the setCameraActive(active) method indicates to the user agent the camera capture state desired by the page. When invoked, it MUST perform the following steps:

1. Let document be this ’s relevant global object ’s associated Document .
2. Let captureKind be "camera".
3. Return the result of running the update capture state algorithm with document , active and captureKind .

Similarly, the setScreenshareActive(active) method indicates to the user agent the screenshare capture state desired by the page. When invoked, it MUST perform the following steps:

1. Let document be this ’s relevant global object ’s associated Document .
2. Let captureKind be "screenshare".
3. Return the result of running the update capture state algorithm with document , active and captureKind .

The update capture state algorithm , when invoked with document , active and captureKind , MUST perform the following steps:

1. If document is not fully active , return a promise rejected with InvalidStateError .
2. If active is true and document ’s visibility state is not "visible", the user agent MAY return a promise rejected with InvalidStateError .
3. Let p be a new promise.
4. In parallel , run the following steps:
   1. Let applyPausePolicy be true if the user agent implements a policy of pausing all input sources of type captureKind in response to UI and false otherwise.
   2. If applyPausePolicy is true , run the following substeps:
      1. Let currentlyActive be false if the user agent is currently pausing all input sources of type captureKind and true otherwise.
      2. If active is currentlyActive , resolve p with undefined and abort these steps.
      3. If active is true , the user agent MAY wait to proceed, for instance to prompt the user.
      4. If the user agent denies the request to update the capture state, reject p with a NotAllowedError and abort these steps.
   3. Update the user agent capture state UI according to captureKind and active .
   4. Queue a task using the user interaction task source to resolve p with undefined .
   5. If applyPausePolicy is true , run the following substeps:
      1. Let newMutedState be true if active is false and false otherwise.
      2. For each MediaStreamTrack whose source is of type captureKind , queue a task using the user interaction task source to set a track’s muted state to newMutedState .
5. Return p .

The setMicrophoneActive(active) , setCameraActive(active) and setScreenshareActive(active) methods can reject based on user agent specific heuristics. This might in particular happen when the web page asks to activate (unmute) the microphone, camera or screenshare. The user agent could decide to require transient activation in that case. It might also require user input through a prompt to make the actual decision.

The user agent MAY display UI which invokes handlers for media session actions .

5. 6. The MediaMetadata interface

[Exposed=Window]
interface MediaMetadata {
  constructor(optional MediaMetadataInit init = {});
  attribute DOMString title;
  attribute DOMString artist;
  attribute DOMString album;
  attribute FrozenArray<object> artwork;
  [SameObject] readonly attribute FrozenArray<ChapterInformation> chapterInfo;
};
dictionary MediaMetadataInit {
  DOMString title = "";
  DOMString artist = "";
  DOMString album = "";
  sequence<MediaImage> artwork = [];
  sequence<ChapterInformationInit> chapterInfo = [];
};

A MediaMetadata object is a representation of the metadata associated with a MediaSession that can be used by user agents to provide customized user interface.

A MediaMetadata can have an associated media session .

A MediaMetadata has an associated title , artist and album which are DOMString.

A MediaMetadata has an associated sequence of artwork images , which is a sequence of type MediaImage . A MediaMetadata also has has an associated converted artwork images which is initially undefined .

A MediaMetadata has an associated list of chapter information .

A MediaMetadata is said to be an empty metadata if it is equal to null or all the following conditions are true:

• Its title is the empty string.
• Its artist is the empty string.
• Its album is the empty string.
• Its artwork images length is 0 .
• Its chapter information length is 0 .

The MediaMetadata( init ) constructor, when invoked, MUST run the following steps:

1. Let metadata be a new MediaMetadata object.
2. Set metadata ’s title to init ’s title .
3. Set metadata ’s artist to init ’s artist .
4. Set metadata ’s album to init ’s album .
5. Run the convert artwork algorithm with init ’s artwork as input and set metadata ’s artwork images as the result if it succeeded.
6. Let chapters be an empty list of type ChapterInformation .
7. For each entry in init ’s chapterInfo , create a ChapterInformation from entry and append it to chapters .
8. Set metadata ’s chapter information to the result of creating a frozen array from chapters .
9. Return metadata .

When the convert artwork algorithm with input parameter is invoked, where the input is a sequence of type MediaImage , the user agent MUST run the following steps:

1. Let output be an empty list of type MediaImage .
2. For each entry in input (which is a MediaImage list), perform the following steps:
   1. Let image be a new MediaImage .
   2. Let baseURL be the API base URL specified by the entry settings object .
   3. Parse entry ’s src using baseURL . If it does not return failure, set image ’s src to the return value. Otherwise, throw a TypeError and abort these steps.
   4. Set image ’s sizes to entry ’s sizes .
   5. Set image ’s type to entry ’s type .
   6. Append image to the output .
3. Return output as result.

The title attribute reflects the MediaMetadata ’s title . On getting, it MUST return the MediaMetadata ’s title . On setting, it MUST set the MediaMetadata ’s title to the given value.

The artist attribute reflects the MediaMetadata ’s artist . On getting, it MUST return the MediaMetadata ’s artist . On setting, it MUST set the MediaMetadata ’s artist to the given value.

The album attribute reflects the MediaMetadata ’s album . On getting, it MUST return the MediaMetadata ’s album . On setting, it MUST set the MediaMetadata ’s album to the given value.

The artwork attribute reflects the MediaMetadata ’s artwork images . On getting, it MUST run the following steps:

1. If the MediaMetadata ’s converted artwork images is undefined , run the following steps:
   1. Let frozenArtwork be a JavaScript Array value.
   2. For each entry in the MediaMetadata ’s artwork images , perform the following steps:
      1. Let image be the result of converting to a JavaScript object entry .
      2. Perform ! SetIntegrityLevel ( image , " frozen "), to prevent accidental mutation by scripts.
      3. Push image to frozenArtwork .
   3. Perform ! SetIntegrityLevel ( frozenArtwork , " frozen ").
   4. Set the MediaMetadata ’s converted artwork images to frozenArtwork .
2. Return the MediaMetadata ’s converted artwork images .

1. Let convertedArtwork be the result of converting value to a sequence of type MediaImage .
2. Run convert artwork algorithm with convertedArtwork , and set the MediaMetadata ’s artwork images as the result if it succeeds.
3. Set the MediaMetadata ’s converted artwork images to undefined .

When MediaMetadata ’s title , artist , album or artwork images are modified, the user agent MUST run the following steps:

1. If the instance has no associated media session , abort these steps.
2. Otherwise, queue a task to run the following substeps:
   1. If the instance no longer has an associated media session , abort these steps.
   2. Otherwise, in parallel , run the update metadata algorithm .

6. 7. The ChapterInformation interface

[Exposed=Window]
interface ChapterInformation {
  readonly attribute DOMString title;
  readonly attribute double startTime;
  [SameObject] readonly attribute FrozenArray<MediaImage> artwork;
};
dictionary ChapterInformationInit {
  DOMString title = "";
  double startTime = 0;
  sequence<MediaImage> artwork = [];
};

A ChapterInformation object is a representation of metadata for an individual chapter, such as the title of the section, its timestamp, and screenshot image data of this section, that can be used by user agents to provide a customized user interface.

A ChapterInformation can have an associated media metadata .

A ChapterInformation has an associated title which is DOMString.

A ChapterInformation has an associated startTime which is double.

A ChapterInformation has an associated list of artwork images .

To create a ChapterInformation with init , run the following steps:

1. Let chapterInfo be a new ChapterInformation object.
2. Set chapterInfo ’s title to init ’s title .
3. Set chapterInfo ’s startTime to init ’s startTime . If the startTime is negative or greater than duration , throw a TypeError .
4. Let artwork be the result of running the convert artwork algorithm with init ’s artwork as input .
5. Set chapterInfo ’s artwork images to the result of creating a frozen array from artwork .
6. Return chapterInfo .

The title attribute reflects the ChapterInformation ’s title . On getting, it MUST return the ChapterInformation ’s title .

The startTime attribute reflects the ChapterInformation ’s startTime in seconds. On getting, it MUST return the ChapterInformation ’s startTime .

The artwork attribute reflects the ChapterInformation ’s artwork images . On getting, it MUST return the ChapterInformation ’s artwork images .

7. 8. The MediaImage dictionary

dictionary MediaImage {
  required USVString src;
  DOMString sizes = "";
  DOMString type = "";
};

The MediaImage dictionary members are inspired by ImageResource in [IMAGE-RESOURCE] .

The src dictionary member is used to specify the MediaImage object’s source . It is a URL from which the user agent can fetch the image’s data.

The sizes dictionary member is used to specify the MediaImage object’s sizes . It follows the spec of sizes attribute in the HTML link element, which is a string consisting of an unordered set of unique space-separated tokens which are ASCII case-insensitive that represents the dimensions of an image. Each keyword is either an ASCII case-insensitive match for the string "any", or a value that consists of two valid non-negative integers that do not have a leading U+0030 DIGIT ZERO (0) character and that are separated by a single U+0078 LATIN SMALL LETTER X or U+0058 LATIN CAPITAL LETTER X character. The keywords represent icon sizes in raw pixels (as opposed to CSS pixels). When multiple image objects are available, a user agent MAY use the value to decide which icon is most suitable for a display context (and ignore any that are inappropriate). The parsing steps for the sizes attribute MUST follow the parsing steps for HTML link element sizes attribute .

The type dictionary member is used to specify the MediaImage object’s MIME type . It is a hint as to the media type of the image. The purpose of this attribute is to allow a user agent to ignore images of media types it does not support.

8. 9. The MediaPositionState dictionary

dictionary MediaPositionState {
  unrestricted double duration;
  double playbackRate;
  double position;
};

The MediaPositionState dictionary is a representation of the current playback position associated with a MediaSession that can be used by user agents to provide a user interface that displays the current playback position and duration.

The duration dictionary member is used to specify the duration in seconds. It should always be positive and positive infinity can be used to indicate media without a defined end such as live playback.

The playbackRate dictionary member is used to specify the playback rate . It can be positive to represent forward playback or negative to represent backwards playback. It should not be zero.

The position dictionary member is used to specify the last reported playback position in seconds. It should always be positive.

9. 10. The MediaSessionActionDetails dictionary

dictionary MediaSessionActionDetails {
  required MediaSessionAction action;
  double seekOffset;
  double seekTime;
  boolean fastSeek;
  boolean isActivating;
};

The MediaSessionActionHandler MUST be run with the details parameter whose dictionary type is MediaSessionActionDetails .

The action dictionary member is used to specify the media session action that the MediaSessionActionHandler is associated with.

The seekOffset dictionary member MAY be provided when the media session action is seekbackward or seekforward . It is the time in seconds to move the playback time by. If present, it should always be positive. If it is not provided then the site should choose a sensible time (e.g. a few seconds).

When the media session action is seekto :

• The seekTime dictionary member MUST be provided and is the time in seconds to move the playback time to.
• The fastSeek dictionary member MAY be provided and will be true if the action is being called multiple times as part of a sequence and this is not the last call in that sequence.

The isActivating dictionary member will be false if the user agent is about to pause all input sources related to the capture action and true otherwise. This dictionary member MUST be present if the user agent implements a policy of pausing all input sources and the media session action is togglecamera , togglemicrophone or screenshare .

10. 11. Permissions Policy Integration

This specification defines a policy-controlled feature identified by the string "mediasession". Its default allowlist is * .

A document’s permissions policy determines whether any content in that document is allowed to use the MediaSession API. If disabled in the document, the User Agent MUST NOT select the document’s media session as the active media session .

11. 12. Examples

This section is non-normative.

navigator.mediaSession.metadata = new MediaMetadata({
  title: "Episode Title",
  artist: "Podcast Host",
  album: "Podcast Title",
  artwork: [{src: "podcast.jpg"}],
  chapterInfo: [
    {title: "Chapter 1", startTime: 0, artwork: [{src: "chapter1.jpg"}]},
    {title: "Chapter 2", startTime: 120, artwork: [{src: "chapter2.jpg"}]}
  ]
});

navigator.mediaSession.metadata = new MediaMetadata({
  title: "Episode Title",
  artist: "Podcast Host",
  album: "Podcast Title",
  artwork: [
    {src: "podcast.jpg", sizes: "128x128", type: "image/jpeg"},
    {src: "podcast_hd.jpg", sizes: "256x256"},
    {src: "podcast_xhd.jpg", sizes: "1024x1024", type: "image/jpeg"},
    {src: "podcast.png", sizes: "128x128", type: "image/png"},
    {src: "podcast_hd.png", sizes: "256x256", type: "image/png"},
    {src: "podcast.ico", sizes: "128x128 256x256", type: "image/x-icon"}
  ],
  chapterInfo: [
    {title: "Chapter 1", startTime: 0, artwork: [
       {src: "chapter1_a.jpg", sizes: "128x128", type: "image/jpeg"},
       {src: "chapter1_b.png", sizes: "256x256", type: "image/png"}
     ]},
    {title: "Chapter 2", startTime: 120, artwork: [
       {src: "chapter2_a.jpg", sizes: "128x128", type: "image/jpeg"},
       {src: "chapter2_b.png", sizes: "256x256", type: "image/png"}
     ]}
  ]
});

var audio1 = document.createElement("audio");
audio1.src = "chapter1.mp3";
var audio2 = document.createElement("audio");
audio2.src = "chapter2.mp3";
audio1.play();
audio1.addEventListener("ended", function() {
  audio2.play();
});

function updateMetadata(event) {
  navigator.mediaSession.metadata = new MediaMetadata({
    title: event.target == audio1 ? "Chapter 1" : "Chapter 2",
    artist: "An Author",
    album: "A Book",
    artwork: [{src: "cover.jpg"}]
  });
}
audio1.addEventListener("play", updateMetadata);
audio2.addEventListener("play", updateMetadata);

var tracks = ["chapter1.mp3", "chapter2.mp3", "chapter3.mp3"];
var trackId = 0;
var audio = document.createElement("audio");
audio.src = tracks[trackId];
function updatePlayingMedia() {
  audio.src = tracks[trackId];
  // Update metadata (omitted)
}
navigator.mediaSession.setActionHandler("previoustrack", function() {
  trackId = (trackId + tracks.length - 1) % tracks.length;
  updatePlayingMedia();
});
navigator.mediaSession.setActionHandler("nexttrack", function() {
  trackId = (trackId + 1) % tracks.length;
  updatePlayingMedia();
});
navigator.mediaSession.setActionHandler("seekto", function(details) {
  audio.currentTime = details.seekTime;
});

var adFrame;
var audio = document.createElement("audio");
audio.src = "foo.mp3";
function resetActionHandlers() {
  navigator.mediaSession.setActionHandler("play", _ => audio.play());
  navigator.mediaSession.setActionHandler("pause", _ => audio.pause());
}
resetActionHandlers();
// This method will be called when the page wants to play some ad.
function pauseAudioAndPlayAd() {
  audio.pause();
  navigator.mediaSession.playbackState = "playing";
  setUpAdFrame();
  adFrame.contentWindow.postMessage("play_ad");
  navigator.mediaSession.setActionHandler("pause", pauseAd);
}
function pauseAd() {
  adFrame.contentWindow.postMessage("pause_ad");
  navigator.mediaSession.playbackState = "paused";
  navigator.mediaSession.setActionHandler("play", resumeAd);
}
function resumeAd() {
  adFrame.contentWindow.postMessage("resume_ad");
  navigator.mediaSession.playbackState = "playing";
  navigator.mediaSession.setActionHandler("pause", pauseAd);
}
window.onmessage = function(e) {
  if (e.data === "ad finished") {
    removeAdFrame();
    navigator.mediaSession.playbackState = "none";
    resetActionHandlers();
  }
}
function setUpAdFrame() {
  adFrame = document.createElement("iframe");
  adFrame.src = "https://example.com/ad-iframe.html";
  document.body.appendChild(adFrame);
}
function removeAdFrame() {
  adFrame.remove();
}

// Media is loaded, set the duration.
navigator.mediaSession.setPositionState({
  duration: 60
});
// Media starts playing at the beginning.
navigator.mediaSession.playbackState = "playing";
// Media starts playing at 2x 10 seconds in.
navigator.mediaSession.setPositionState({
  duration: 60,
  playbackRate: 2,
  position: 10
});
// Media is paused.
navigator.mediaSession.playbackState = "paused";
// Media is reset.
navigator.mediaSession.setPositionState(null);

var isMicrophoneActive = false;
var isCameraActive = false;
navigator.mediaSession.setMicrophoneActive(isMicrophoneActive);
navigator.mediaSession.setCameraActive(isCameraActive);
navigator.mediaSession.setActionHandler("togglemicrophone", function() {
  if (isMicrophoneActive) {
    // Mute the microphone. Implementation omitted.
  } else {
    // Unmute the microphone. Implementation omitted.
  }
  isMicrophoneActive = !isMicrophoneActive;
  navigator.mediaSession.setMicrophoneActive(isMicrophoneActive);
});
navigator.mediaSession.setActionHandler("togglecamera", function() {
  if (isCameraActive) {
    // Disable the camera. Implementation omitted.
  } else {
    // Enable the camera. Implementation omitted.
  }
  isCameraActive = !isCameraActive;
  navigator.mediaSession.setCameraActive(isCameraActive);
});
navigator.mediaSession.setActionHandler("hangup", function() {
  // End the call. Implementation omitted.
});

var currentSlideIndex = 0;
navigator.mediaSession.setActionHandler("previousslide", function() {
  currentSlideIndex--;
  // Set current slide. Implementation omitted.
});
navigator.mediaSession.setActionHandler("nextslide", function() {
  currentSlideIndex++;
  // Set current slide. Implementation omitted.
});

navigator.mediaSession.setActionHandler("enterpictureinpicture", function() {
  remoteVideo.requestPictureInPicture();
});

// Create a MediaStream with audio enabled.
const stream = await navigator.mediaDevices.getUserMedia({audio:true});
const track = stream.getAudioTracks()[0];
navigator.mediaSession.setActionHandler("voiceactivity", function() {
  if (track.muted) {
    // Show unmute notification. If user allows to unmute, call
    // setMicrophoneActive(true) to unmute.
  }
});

Acknowledgments

The editors would like to thank Paul Adenot, Jake Archibald, Tab Atkins, Jonathan Bailey, François Beaufort, Marcos Caceres, Domenic Denicola, Ralph Giles, Anne van Kesteren, Tobie Langel, Michael Mahemoff, Jer Noble, Elliott Sprehn, Chris Wilson, and Jörn Zaefferer for their participation in technical discussions that ultimately made this specification possible.

Special thanks go to Philip Jägenstedt and David Vest for their help in designing every aspect of media sessions and for their seemingly infinite patience in working through the initial design issues; Jer Noble for his help in building a model that also works well within the iOS audio focus model; and Mounir Lamouri and Anton Vayvod for their early involvement, feedback and support in making this specification happen.