Media Capture and Streams Extensions

A MediaStreamTrack is a transferable object. This allows manipulating real-time media outside the context it was requested or created in, for instance in workers or third-party iframes.

To preserve the existing privacy and security infrastructure, in particular for capture tracks, the track source lifetime management remains tied to the context that created it. The transfer algorithm MUST ensure the following behaviors:

1. The context named originalContext that created a track named originalTrack remains in control of the originalTrack source, named trackSource, even when originalTrack is transferred into transferredTrack.

2. In particular, originalContext remains the proxy to privacy indicators of trackSource. transferredTrack or any of its clones are considered as tracks using trackSource as if they were tracks created in and controlled by originalContext.

3. When originalContext goes away, trackSource gets ended, thus transferredTrack gets ended.

4. When originalContext would have muted/unmuted originalTrack, transferredTrack gets muted/unmuted.

5. If transferredTrack is cloned in transferredTrackClone, transferredTrackClone is tied to trackSource. It is not tied to originalTrack in any way.

6. If transferredTrack is transferred into transferredAgainTrack, transferredAgainTrack is tied to trackSource. It is not tied to transferredTrack or originalTrack in any way.

The WebIDL changes are the following:

WebIDL[Exposed=(Window,Worker), Transferable]
partial interface MediaStreamTrack {
};

At creation of a MediaStreamTrack object, called track, run the following steps:

1. Initialize track.[[IsDetached]] to false.

The MediaStreamTrack transfer steps, given value and dataHolder, are:

1. If value.[[IsDetached]] is true, throw a "DataCloneError" DOMException.

2. Set dataHolder.[[id]] to value.id.

3. Set dataHolder.[[kind]] to value.kind.

4. Set dataHolder.[[label]] to value.label.

5. Set dataHolder.[[readyState]] to value.readyState.

6. Set dataHolder.[[enabled]] to value.enabled.

7. Set dataHolder.[[muted]] to value.muted.

8. Set dataHolder.[[source]] to value underlying source.

9. Set dataHolder.[[constraints]] to value active constraints.

10. Set value.[[IsDetached]] to true.

11. Set value.[[ReadyState]] to "ended" (without stopping the underlying source or firing an ended event).

MediaStreamTrack transfer-receiving steps, given dataHolder and track, are:

1. Initialize track.id to dataHolder.[[id]].

2. Initialize track.kind to dataHolder.[[kind]].

3. Initialize track.label to dataHolder.[[label]].

4. Initialize track.readyState to dataHolder.[[readyState]].

5. Initialize track.enabled to dataHolder.[[enabled]].

6. Initialize track.muted to dataHolder.[[muted]].

7. Initialize the underlying source of track to dataHolder.[[source]].

8. Set track's constraints to dataHolder.[[constraints]].

The underlying source is supposed to be kept alive between the transfer and transfer-receiving steps, or as long as the data holder is alive. In a sense, between these steps, the data holder is attached to the underlying source as if it was a track.

Some platforms or User Agents may provide built-in support for background blurring of video frames, in particular for camera video streams. Web applications may either want to control or at least be aware that background blur is applied at the source level. This may for instance allow the web application to update its UI or to not apply background blur on its own. For that reason, we extend MediaStreamTrack with the following properties.

The WebIDL changes are the following:

WebIDLpartial dictionary MediaTrackSupportedConstraints {
  boolean backgroundBlur = true;
};

partial dictionary MediaTrackConstraintSet {
  ConstrainBoolean backgroundBlur;
};

partial dictionary MediaTrackSettings {
  boolean backgroundBlur;
};

partial dictionary MediaTrackCapabilities {
  sequence<boolean> backgroundBlur;
};

Some platforms offer functionality for voice isolation: Attempting to remove all parts of an audio track that do not correspond to a human voice. Some platforms even attempt to remove extraneous voices, leaving the "main voice" as the dominant component of the audio. The exact methods used may vary between implementations.

This constraint permits the platform to turn on that functionality, with the desired result being that the "main voice" in the audio signal is the dominant component of the audio.

This will have large effects on audio that is presented for other reasons than to transmit voice (for instance music or ambient noises), so needs to be off by default.

This constraint is a stronger version of noise cancellation, which means that if the "noiseSuppression" constraint is set to false and "voiceIsolation" is set to true, the value of "noiseCancellation" will be ignored.

This constraint has no such relationship with any other constraint; in particular it does not affect echoCancellation.

The WebIDL changes are the following:

WebIDLpartial dictionary MediaTrackSupportedConstraints {
  boolean voiceIsolation = true;
};

partial dictionary MediaTrackConstraintSet {
  ConstrainBoolean voiceIsolation;
};

partial dictionary MediaTrackSettings {
  boolean voiceIsolation;
};

partial dictionary MediaTrackCapabilities {
  sequence<boolean> voiceIsolation;
};

When the "voiceIsolation" setting is set to true by the ApplyConstraints algorithm, the UA will attempt to remove the components of the audio track that do not correspond to a human voice. If a dominant voice can be identified, the UA will attempt to enhance that voice.

When the "voiceIsolation" constraint setting is set to false by the ApplyConstraints algorithm, the UA will process the audio according to other settings in its normal fashion.

The configuration (capabilities, constraints or settings) of a MediaStreamTrack may be changed dynamically outside the control of web applications. One example is when a user decides to switch on background blur through the operating system. Web applications might want to know that the configuration of a particular MediaStreamTrack has changed. For that purpose, a new event is defined below.

WebIDLpartial interface MediaStreamTrack {
  attribute EventHandler onconfigurationchange;
};

When the User Agent detects a change of configuration in a track's underlying source, the User Agent MUST run the following steps:

1. If track.muted is true, wait for track.muted to become false or track.readyState to be "ended".

2. If track.readyState is "ended", abort these steps.

3. If track's capabilities, constraints and settings are matching source configuration, abort these steps.

4. Update track's capabilities, constraints and settings according track's underlying source.

5. Fire an event named configurationchange on track.