Media Capture and Streams Extensions

WebIDLpartial interface MediaDevices {
  readonly attribute GetUserMediaSemantics defaultSemantics;
};

WebIDLpartial dictionary MediaStreamConstraints {
  GetUserMediaSemantics semantics;
};

WebIDLenum GetUserMediaSemantics {
  "browser-chooses",
  "user-chooses"
};

This example shows a setup with a start button and a camera selector using the new semantics (microphone is not shown for brievity but is equivalent).

<button id="start">Start</button>
<button id="chosenCamera" disabled>Camera: none</button>
<script>
let cameraTrack = null;
start.onclick = async () => {
  try {
    const stream = await navigator.mediaDevices.getUserMedia({
      video: {deviceId: localStorage.cameraId}
    });
    setCameraTrack(stream.getVideoTracks()[0]);
  } catch (err) {
    console.error(err);
  }
}
chosenCamera.onclick = async () => {
  try {
    const stream = await navigator.mediaDevices.getUserMedia({
      video: true,
      semantics: "user-chooses"
    });
    setCameraTrack(stream.getVideoTracks()[0]);
  } catch (err) {
    console.error(err);
  }
}
function setCameraTrack(track) {
  cameraTrack = track;
  const {deviceId, label} = track.getSettings();
  localStorage.cameraId = deviceId;
  chosenCamera.innerText = `Camera: ${label}`;
  chosenCamera.disabled = false;
}


</

script

>

A MediaStreamTrack is a transferable object . This allows manipulating real-time media outside the context it was requested or created in, for instance in workers or third-party iframes.

To preserve the existing privacy and security infrastructure, in particular for capture tracks, the track source lifetime management remains tied to the context that created it. The transfer algorithm MUST ensure the following behaviors:

1. The context named originalContext that created a track named originalTrack remains in control of the originalTrack source, named trackSource , even when originalTrack is transferred into transferredTrack .

2. In particular, originalContext remains the proxy to privacy indicators of trackSource . transferredTrack or any of its clones are considered as tracks using trackSource as if they were tracks created in and controlled by originalContext .

3. When originalContext goes away, trackSource gets ended, thus transferredTrack gets ended.

4. When originalContext would have muted/unmuted originalTrack , transferredTrack gets muted/unmuted.

5. If transferredTrack is cloned in transferredTrackClone , transferredTrackClone is tied to trackSource . It is not tied to originalTrack in any way.

6. If transferredTrack is transferred into transferredAgainTrack , transferredAgainTrack is tied to trackSource . It is not tied to transferredTrack or originalTrack in any way.

The WebIDL changes to make the track transferable are the following:

WebIDL[Exposed=(Window,Worker), Transferable]
partial interface MediaStreamTrack {
};

At creation of a MediaStreamTrack object, called track , run the following steps:

1. Initialize track . [[IsDetached]] to false .

The MediaStreamTrack transfer steps , given value and dataHolder , are:

1. If value . [[IsDetached]] is true , throw a "DataCloneError" DOMException.

2. Set dataHolder . [[id]] to value . id .

3. Set dataHolder . [[kind]] to value . kind .

4. Set dataHolder . [[label]] to value . label .

5. Set dataHolder . [[readyState]] to value . readyState .

6. Set dataHolder . [[enabled]] to value . enabled .

7. Set dataHolder . [[muted]] to value . muted .

8. Set dataHolder . [[source]] to value underlying source.

9. Set dataHolder . [[constraints]] to value active constraints.

10. Set value . [[IsDetached]] to true .

11. Set value . [[ReadyState]] to "ended" (without stopping the underlying source or firing an ended event).

MediaStreamTrack transfer-receiving steps , given dataHolder and track , are:

1. Initialize track . id to dataHolder . [[id]] .

2. Initialize track . kind to dataHolder . [[kind]] .

3. Initialize track . label to dataHolder . [[label]] .

4. Initialize track . readyState to dataHolder . [[readyState]] .

5. Initialize track . enabled to dataHolder . [[enabled]] .

6. Initialize track . muted to dataHolder . [[muted]] .

7. Initialize the underlying source of track to dataHolder . [[source]] .

8. Set track 's constraints to dataHolder . [[constraints]] .

The underlying source is supposed to be kept alive between the transfer and transfer-receiving steps, or as long as the data holder is alive. In a sense, between these steps, the data holder is attached to the underlying source as if it was a track.

On camera and screenshare tracks, frame counters allow the application to tell what the frame rate is, which may be lower than the target frameRate . For example, if the track is sourced from a camera then the production of frames could be slowed down if it's dark or frames could be dropped if the system is CPU starved. This could impact the total number of frames produced by the source and impact how many frames are delivered, discarded or dropped for other reasons.

WebIDLpartial interface MediaStreamTrack {
  Promise<MediaTrackFrameStats> getFrameStats();
};

If a MediaStreamTrack is sourced from getUserMedia() or getDisplayMedia() , the user agent is required to count each frame from its source as follows:

• A frame is considered delivered if it either was delivered to a sink or would have been delivered to a sink, if one was connected. This is a subset of total frames and it is incremented at the same time as total frames .

• A frame is considered discarded if it was discarded in order to achieve the target frameRate . This is a subset of total frames and it is incremented at the same time as total frames .

• The total number of frames that have been processed by this source, meaning it is known whether the frame was considered delivered, discarded or dropped for any other reason. The number of dropped frames for various unknown reasons can be calculated by subtracting delivered frames and discarded frames from total frames .

  Note

  If the track is unmuted and enabled and the source is backed by a camera, total frames is incremented by frames produced by the camera. If no frames are flowing, such as if the track is muted or disabled, then total frames does not increment.

WebIDLpartial dictionary MediaTrackSupportedConstraints {
  boolean powerEfficientPixelFormat = true;
};

WebIDLpartial dictionary MediaTrackCapabilities {
  sequence<boolean> powerEfficientPixelFormat;
};

WebIDLpartial dictionary MediaTrackSettings {
  boolean powerEfficientPixelFormat;
};

WebIDLpartial interface MediaStreamTrack {
  attribute EventHandler onconfigurationchange;

};

The onconfigurationchange attribute is an event handler IDL attribute for the onconfigurationchange event handler , whose event handler event type is configurationchange .

When the User Agent detects a change of configuration in a track 's underlying source, the User Agent MUST run the following steps:

1. If track . muted is true , wait for track . muted to become false or track . readyState to be "ended".

2. Queue a task on current settings object 's responsible event loop to perform the following steps:

   Note

   This task will run before any other task that may set track . muted to true .

   1. If track . readyState is "ended", abort these steps.

   2. If track 's capabilities, constraints capabilities and settings are matching source configuration, abort these steps.

   3. Update track 's capabilities, constraints capabilities and settings according track 's underlying source.

   4. Fire an event named configurationchange on track .

This example shows how to monitor external background blur changes.

const stream = await navigator.mediaDevices.getUserMedia({video: true});const [track] = stream.getVideoTracks();let {backgroundBlur} = track.getSettings();
applyBlurInSoftwareInstead(!backgroundBlur);
track.addEventListener("configurationchange", () => {
  if (backgroundBlur != track.getSettings().backgroundBlur) {
    backgroundBlur = track.getSettings().backgroundBlur;
    applyBlurInSoftwareInstead(!backgroundBlur);
  }
});