Media Capture and Streams Extensions

WebIDLpartial interface MediaDevices {
  readonly attribute GetUserMediaSemantics defaultSemantics;
};

WebIDLpartial dictionary MediaStreamConstraints {
  GetUserMediaSemantics semantics;
};

WebIDLenum GetUserMediaSemantics {
  "browser-chooses",
  "user-chooses"
};

This example shows a setup with a start button and a camera selector using the new semantics (microphone is not shown for brievity but is equivalent).

<button id="start">Start</button>
<button id="chosenCamera" disabled>Camera: none</button>
<script>
let cameraTrack = null;
start.onclick =  () => {

start.onclick = async () => {

  try {
     navigator.mediaDevices.getUserMedia({
      .cameraId}

    const stream = await navigator.mediaDevices.getUserMedia({
      video: {deviceId: localStorage.cameraId}
    });
    setCameraTrack(stream.getVideoTracks()[]);

    setCameraTrack(stream.getVideoTracks()[0]);
  } catch (err) {
    .error(err);

    console.error(err);
  }
}
chosenCamera.onclick =  () => {

chosenCamera.onclick = async () => {

  try {
     navigator.mediaDevices.getUserMedia({

    const stream = await navigator.mediaDevices.getUserMedia({
      video: true,
      semantics: "user-chooses"
    });
    setCameraTrack(stream.getVideoTracks()[]);

    setCameraTrack(stream.getVideoTracks()[0]);
  } catch (err) {
    .error(err);

    console.error(err);
  }
}
{

function setCameraTrack(track) {
  cameraTrack = track;
   {deviceId, label} = track.getSettings();
  .cameraId = deviceId;
  chosenCamera.innerText = ;
  chosenCamera.disabled = ;

  const {deviceId, label} = track.getSettings();
  localStorage.cameraId = deviceId;
  chosenCamera.innerText = `Camera: ${label}`;
  chosenCamera.disabled = false;

}


</

script

>

A MediaStreamTrack is a transferable object . This allows manipulating real-time media outside the context it was requested or created in, for instance in workers or third-party iframes.

To preserve the existing privacy and security infrastructure, in particular for capture tracks, the track source lifetime management remains tied to the context that created it. The transfer algorithm MUST ensure the following behaviors:

1. The context named originalContext that created a track named originalTrack remains in control of the originalTrack source, named trackSource , even when originalTrack is transferred into transferredTrack .

2. In particular, originalContext remains the proxy to privacy indicators of trackSource . transferredTrack or any of its clones are considered as tracks using trackSource as if they were tracks created in and controlled by originalContext .

3. When originalContext goes away, trackSource gets ended, thus transferredTrack gets ended.

4. When originalContext would have muted/unmuted originalTrack , transferredTrack gets muted/unmuted.

5. If transferredTrack is cloned in transferredTrackClone , transferredTrackClone is tied to trackSource . It is not tied to originalTrack in any way.

6. If transferredTrack is transferred into transferredAgainTrack , transferredAgainTrack is tied to trackSource . It is not tied to transferredTrack or originalTrack in any way.

The WebIDL changes to make the track transferable are the following:

WebIDL[Exposed=(Window,Worker), Transferable]
partial interface MediaStreamTrack {
};

At creation of a MediaStreamTrack object, called track , run the following steps:

1. Initialize track . [[IsDetached]] to false .

The MediaStreamTrack transfer steps , given value and dataHolder , are:

1. If value . [[IsDetached]] is true , throw a "DataCloneError" DOMException.

2. Set dataHolder . [[id]] to value . id .

3. Set dataHolder . [[kind]] to value . kind .

4. Set dataHolder . [[label]] to value . label .

5. Set dataHolder . [[readyState]] to value . readyState .

6. Set dataHolder . [[enabled]] to value . enabled .

7. Set dataHolder . [[muted]] to value . muted .

8. Set dataHolder . [[source]] to value underlying source.

9. Set dataHolder . [[constraints]] to value active constraints.

10. Set dataHolder . [[contentHint]] to value application-set content hint.

11. Set value . [[IsDetached]] to true .

12. Set value . [[ReadyState]] to " ended " (without stopping the underlying source or firing an ended event).

MediaStreamTrack transfer-receiving steps , given dataHolder and track , are:

1. Initialize track . id to dataHolder . [[id]] .

2. Initialize track . kind to dataHolder . [[kind]] .

3. Initialize track . label to dataHolder . [[label]] .

4. Initialize track . readyState to dataHolder . [[readyState]] .

5. Initialize track . enabled to dataHolder . [[enabled]] .

6. Initialize track . muted to dataHolder . [[muted]] .

7. Set track application-set content hint to dataHolder . [[contentHint]] .

8. Initialize the underlying source of track to dataHolder . [[source]] .

9. Set track 's constraints to dataHolder . [[constraints]] .

The underlying source is supposed to be kept alive between the transfer and transfer-receiving steps, or as long as the data holder is alive. In a sense, between these steps, the data holder is attached to the underlying source as if it was a track.

On microphone audio tracks, frame counters allow the application to tell the ratio of audio that is delivered as one quality indicator and the latency metrics measure the input delay from capture to application.

On camera and screenshare video tracks, frame counters allow the application to tell what the frame rate is, which may be lower than the target frameRate . For example, if the track is sourced from a camera then the production of frames could be slowed down if it's dark or frames could be dropped if the system is CPU starved. This could impact the total number of frames produced by the source and impact how many frames are delivered, discarded or dropped for other reasons.

WebIDLpartial interface MediaStreamTrack {
  [SameObject] readonly attribute
      (MediaStreamTrackAudioStats or MediaStreamTrackVideoStats)? stats;
};

Let the MediaStreamTrack have a [[Stats]] internal slot initialized it to null , unless otherwise specified below.

If the track's is of kind "audio" , run the following steps:

1. If the MediaStreamTrack is sourced from getUserMedia() , initialize [[Stats]] to a new instance of MediaStreamTrackAudioStats set up to expose audio stats for this MediaStreamTrack .

If the track's is of kind "video" , run the following steps:

1. If the MediaStreamTrack is sourced from getUserMedia() or getDisplayMedia() , initialize [[Stats]] to a new instance of MediaStreamTrackVideoStats set up to expose video stats for this MediaStreamTrack .

WebIDLpartial dictionary MediaTrackSupportedConstraints {
  boolean powerEfficient = true;
};

WebIDLpartial dictionary MediaTrackCapabilities {
  sequence<boolean> powerEfficient;
};

WebIDLpartial dictionary MediaTrackSettings {
  boolean powerEfficient;
};

WebIDLpartial dictionary MediaTrackSupportedConstraints {
  boolean powerEfficientPixelFormat = true;
};

WebIDLpartial dictionary MediaTrackCapabilities {
  sequence<boolean> powerEfficientPixelFormat;
};

WebIDLpartial dictionary MediaTrackSettings {
  boolean powerEfficientPixelFormat;
};

WebIDLpartial interface MediaStreamTrack {
  attribute EventHandler onconfigurationchange;
};

The onconfigurationchange attribute is an event handler IDL attribute for the onconfigurationchange event handler , whose event handler event type is configurationchange .

When the User Agent detects a change of configuration in a track 's underlying source, the User Agent MUST run the following steps:

1. If track . muted is true , wait for track . muted to become false or track . readyState to be "ended".

2. Queue a task on current settings object 's responsible event loop to perform the following steps:

   Note

   This task will run before any other task that may set track . muted to true .

   1. If track . readyState is "ended", abort these steps.

   2. If track 's capabilities and settings are matching source configuration, abort these steps.

   3. Update track 's capabilities and settings according track 's underlying source.

   4. Fire an event named configurationchange on track .

This example shows how to monitor external background blur changes.

const stream = await navigator.mediaDevices.getUserMedia({video: true});
const [track] = stream.getVideoTracks();
let {backgroundBlur} = track.getSettings();
applyBlurInSoftwareInstead(!backgroundBlur);
track.addEventListener( {
   (backgroundBlur != track.getSettings().backgroundBlur) {
    backgroundBlur = track.getSettings().backgroundBlur;
    applyBlurInSoftwareInstead(!backgroundBlur);

track.addEventListener("configurationchange", () => {
  if (backgroundBlur != track.getSettings().backgroundBlur) {
    backgroundBlur = track.getSettings().backgroundBlur;
    applyBlurInSoftwareInstead(!backgroundBlur);
  }
});