1. Definitions
- Codec
-
Refers generically to an instance of AudioDecoder, AudioEncoder, VideoDecoder, or VideoEncoder.
- Key Frame
-
An encoded frame that does not depend on any other frames for decoding.
- Internal Pending Output
-
Codec outputs such as
VideoFrame
s that currently reside in the internal pipeline of the underlying codec implementation. The underlying codec implementation may emit new outputs only when a new inputs are provided. The underlying codec implementation must emit all outputs in response to a flush. - Codec System Resources
-
Resources including CPU memory, GPU memory, and exclusive handles to specific decoding/encoding hardware that may be allocated by the User Agent as part of codec configuration or generation of
AudioFrameAudioDataVideoFrame
objects. Such resources may be quickly exhuasted and should be released immediately when no longer in use. - Progressive Image
-
An image that supports decoding to multiple levels of detail, with lower levels becoming available while the encoded data is not yet fully buffered.
- Progressive Image Frame Generation
-
A generational identifier for a given Progressive Image decoded output. Each successive generation adds additional detail to the decoded output. The mechanism for computing a frame’s generation is implementer defined.
- Primary Image Track
-
An image track that is marked by the given image file as being the default track. The mechanism for indiciating a primary track is format defined.
2. Codec Processing Model
2.1. Background
This section is non-normative.
The
codec
interfaces
defined
by
the
specification
are
designed
such
that
new
codec
tasks
may
be
scheduled
while
previous
tasks
are
still
pending.
For
example,
web
authors
may
call
decode()
without
waiting
for
a
previous
decode()
to
complete.
This
is
achieved
by
offloading
underlying
codec
tasks
to
a
separate
thread
for
parallel
execution.
This section describes threading behaviors as they are visible from the perspective of web authors. Implementers may choose to use more or less threads as long the exernally visible behaviors of blocking and sequencing are maintained as follows.
2.2. Control Thread and Codec Thread
All steps in this specificaiton will run on either a control thread or a codec thread .
The control thread is the thread from which authors will construct a codec and invoke its methods. Invoking a codec’s methods will typically result in the creation of control messages which are later executed on the codec thread . Each global object has a separate control thread.
The codec thread is the thread from which a codec will dequeue control messages and execute their steps. Each codec instance has a separate codec thread. The lifetime of a codec thread matches that of its associated codec instance.
The control thread uses a traditional event loop, as described in [HTML] .
The codec thread uses a specialized codec processing loop .
Communication from the control thread to the codec thread is done using control message passing. Communication in the other direction is done using regular event loop tasks.
Each codec instance has a single control message queue that is a queue of control messages .
Queuing a control message means enqueing the message to a codec ’s control message queue . Invoking codec methods will often queue a control message to schedule work.
Running a control message means performing a sequence of steps specified by the method that enqueued the message. The steps of a control message may depend on injected state , supplied by the method that enqueued the message.
Resetting the control message queue means performing these steps:
-
For each control message in the control message queue :
-
If a control message’s injected state includes a promise, reject that promise.
-
Remove the message from the queue.
-
The codec processing loop must run these steps:
-
While true:
-
If the control message queue is emtpy, continue .
-
Dequeue front message from the control message queue .
-
Run control message steps described by front message .
-
3. AudioDecoder Interface
[Exposed =(Window ,DedicatedWorker )]interface {
AudioDecoder constructor (AudioDecoderInit );
init readonly attribute CodecState state ;readonly attribute long decodeQueueSize ;undefined configure (AudioDecoderConfig );
config undefined decode (EncodedAudioChunk );
chunk Promise <undefined >flush ();undefined reset ();undefined close ();static Promise <AudioDecoderSupport >isConfigSupported (AudioDecoderConfig ); };
config dictionary {
AudioDecoderInit ;required AudioDataOutputCallback ;
output required WebCodecsErrorCallback ; };
error );callback =
AudioDataOutputCallback undefined (AudioData );
output
3.1. Internal Slots
-
[[codec implementation]]
- Underlying decoder implementation provided by the User Agent.
-
[[output callback]]
- Callback given at construction for decoded outputs.
-
[[error callback]]
- Callback given at construction for decode errors.
3.2. Constructors
AudioDecoder(init)
-
Let d be a new
AudioDecoder
object. -
Assign init.output to the
[[output callback]]
internal slot. -
Assign init.error to the
[[error callback]]
internal slot. -
Assign "unconfigured" to d.state.
-
Return d.
3.3. Attributes
-
state
, of type CodecState , readonly - Describes the current state of the codec.
-
decodeQueueSize
, of type long , readonly - The number of pending decode requests. This number will decrease as the underlying codec is ready to accept new input.
3.4. Methods
-
configure(config)
-
Enqueues
a
control
message
to
configure
the
audio
decoder
for
decoding
chunks
as
described
by
config
.
NOTE: This method will trigger a
NotSupportedError
if the user agent does not support config . Authors should first check support by callingisConfigSupported()
with config . User agents are not required to support any particular codec type or configuration.When invoked, run these steps:
-
If config is not a valid AudioDecoderConfig , throw a
TypeError
. -
If
state
is“closed”
, throw anInvalidStateError
. -
Set
state
to"configured"
. -
Queue a control message to configure the decoder with config .
Running a control message to configure the decoder means running these steps:
-
Let supported be the result of running the Check Configuration Support algorith with config .
-
If supported is
true
, assign[[codec implementation]]
with an implementation supporting config . -
Otherwise, run the Close AudioDecoder algorithm with
NotSupportedError
.
-
-
decode(chunk)
-
Enqueues
a
control
message
to
decode
the
given
chunk
.
When invoked, run these steps:
-
If
state
is not"configured"
, throw anInvalidStateError
. -
Increment
decodeQueueSize
. -
Queue a control message to decode the chunk .
Running a control message to decode the chunk means performing these steps:
-
Attempt to use
[[codec implementation]]
to decode the chunk. -
If decoding results in an error, queue a task on the control thread event loop to run the Close
VideoDecoderAudioDecoder algorithm withEncodingError
. -
Queue a task on the control thread event loop to decrement
decodeQueueSize
-
Let decoded outputs be a list of decoded video data outputs emitted by
[[codec implementation]]
. -
If decoded outputs is not empty, queue a task on the control thread event loop to run the Output
VideoFramesAudioData algorithm with decoded outputs .
-
-
flush()
-
Completes
all
control
messages
in
the
control
message
queue
and
emits
all
outputs.
When invoked, run these steps:
-
If
state
is not"configured"
, return a promise rejected withInvalidStateError
DOMException
. -
Let promise be a new Promise.
-
Queue a control message to flush the codec with promise .
-
Return promise .
Running a control message to flush the codec means performing these steps with promise .
-
Signal
[[codec implementation]]
to emit all internal pending outputs . -
Let decoded outputs be a list of decoded audio data outputs emitted by
[[codec implementation]]
. -
If decoded outputs is not empty, queue a task on the control thread event loop to run the Output
AudioFramesAudioData algorithm with decoded outputs . -
Queue a task on the control thread event loop to resolve promise .
-
-
reset()
-
Immediately
resets
all
state
including
configuration,
control
messages
in
the
control
message
queue
,
and
all
pending
callbacks.
When invoked, run the Reset AudioDecoder algorithm.
-
close()
-
Immediately
aborts
all
pending
work
and
releases
system
resources
.
Close
is
final.
When invoked, run the Close AudioDecoder algorithm.
-
isConfigSupported(config)
-
Returns
a
promise
indicating
whether
the
provided
config
is
supported
by
the
user
agent.
NOTE: The returned
AudioDecoderSupport
config
will contain only the dictionary members that user agent recognized. Unrecognized dictionary memebers will be ignored. Authors may detect unrecognized dictionary members by comparingingconfig
to their provided config .When invoked, run these steps:
-
If config is not a valid AudioDecoderConfig , return a promise rejected with
TypeError
. -
Let p be a new Promise.
-
Let checkSupportQueue be the result of starting a new parallel queue .
-
Enqueue the following steps to checkSupportQueue :
-
Let decoderSupport be a newly constructed
AudioDecoderSupport
, initialized as follows:-
Set
config
to the result of running the Clone Configuration algorithm with config . -
Set
supported
to the result of running the Check Configuration Support algorithm with config .
-
-
Resolve p with decoderSupport .
-
-
Return p .
-
3.5. Algorithms
-
Output
AudioFramesAudioData (with outputs ) -
Run
these
steps:
-
For each output in outputs :
-
Let
bufferdata be anAudioBufferAudioDataAssign
false
containingto[[detached]]
.Let resource be the
decoded audio data inmedia resource described by output .-
Let
frameresourceReference beana reference to resource . Assign resourceReference to
AudioFrame[[resource reference]]containing-
Let
buffertimestampand abe thetimestamp
of theforoutput.EncodedAudioChunk
associated with output . Assign timestamp to
[[timestamp]]
.Assign values to
[[sample format]]
,[[sample rate]]
,[[number of frames]]
, and[[number of channels]]
as determined by output .
-
Invoke
[[output callback]]
withframe.data .
-
-
- Reset AudioDecoder
-
Run
these
steps:
-
If
state
is"closed"
, throw anInvalidStateError
. -
Set
state
to"unconfigured"
. -
Signal
[[codec implementation]]
to cease producing output for the previous configuration. -
Set
decodeQueueSize
to zero.
-
- Close AudioDecoder (with error)
-
Run
these
steps:
-
Run the Reset AudioDecoder algorithm.
-
Set
state
to"closed"
. -
Clear
[[codec implementation]]
and release associated system resources . -
If error is set, queue a task on the control thread event loop to invoke the
[[error callback]]
with error .
-
4. VideoDecoder Interface
[Exposed =(Window ,DedicatedWorker )]interface {
VideoDecoder constructor (VideoDecoderInit );
init ; ;readonly attribute CodecState state ;readonly attribute long decodeQueueSize ;undefined configure (VideoDecoderConfig );
config undefined decode (EncodedVideoChunk );
chunk Promise <undefined >flush ();undefined reset ();undefined close ();static Promise <VideoDecoderSupport >isConfigSupported (VideoDecoderConfig ); };
config dictionary {
VideoDecoderInit required VideoFrameOutputCallback ;
output required WebCodecsErrorCallback ; };
error callback =
VideoFrameOutputCallback undefined (VideoFrame );
output
4.1. Internal Slots
-
[[codec implementation]]
- Underlying decoder implementation provided by the User Agent.
-
[[output callback]]
- Callback given at construction for decoded outputs.
-
[[error callback]]
- Callback given at construction for decode errors.
4.2. Constructors
VideoDecoder(init)
-
Let d be a new VideoDecoder object.
-
Assign
init.output
to the[[output callback]]
internal slot. -
Assign
init.error
to the[[error callback]]
internal slot. -
Assign "unconfigured" to
d.state
. -
Return d.
4.3. Attributes
-
state
, of type CodecState , readonly - Describes the current state of the codec.
-
decodeQueueSize
, of type long , readonly - The number of pending decode requests. This number will decrease as the underlying codec is ready to accept new input.
4.4. Methods
-
configure(config)
-
Enqueues
a
control
message
to
configure
the
video
decoder
for
decoding
chunks
as
described
by
config
.
NOTE: This method will trigger a
NotSupportedError
if the user agent does not support config . Authors should first check support by callingisConfigSupported()
with config . User agents are not required to support any particular codec type or configuration.When invoked, run these steps:
-
If config is not a valid VideoDecoderConfig , throw a
TypeError
. -
If
state
is“closed”
, throw anInvalidStateError
. -
Set
state
to"configured"
. -
Queue a control message to configure the decoder with config .
Running a control message to configure the decoder means running these steps:
-
Let supported be the result of running the Check Configuration Support algorith with config .
-
If supported is
true
, assign[[codec implementation]]
with an implementation supporting config . -
Otherwise, run the Close VideoDecoder algorithm with
NotSupportedError
.
-
-
decode(chunk)
-
Enqueues
a
control
message
to
decode
the
given
chunk
.
When invoked, run these steps:
-
If
state
is not"configured"
, throw anInvalidStateError
. -
Increment
decodeQueueSize
. -
Queue a control message to decode the chunk .
Running a control message to decode the chunk means performing these steps:
-
Attempt to use
[[codec implementation]]
to decode the chunk. -
If decoding results in an error, queue a task on the control thread event loop to run the Close VideoDecoder algorithm with
EncodingError
. -
Queue a task on the control thread event loop to decrement
decodeQueueSize
-
Let decoded outputs be a list of decoded video data outputs emitted by
[[codec implementation]]
. -
If decoded outputs is not empty, queue a task on the control thread event loop to run the Output VideoFrames algorithm with decoded outputs .
-
-
flush()
-
Completes
all
control
messages
in
the
control
message
queue
and
emits
all
outputs.
When invoked, run these steps:
-
If
state
is not"configured"
, return a promise rejected withInvalidStateError
DOMException
. -
Let promise be a new Promise.
-
Queue a control message to flush the codec with promise .
-
Return promise .
Running a control message to flush the codec means performing these steps with promise .
-
Signal
[[codec implementation]]
to emit all internal pending outputs . -
Let decoded outputs be a list of decoded video data outputs emitted by
[[codec implementation]]
. -
If decoded outputs is not empty, queue a task on the control thread event loop to run the Output VideoFrames algorithm with decoded outputs .
-
Queue a task on the control thread event loop to resolve promise .
-
-
reset()
-
Immediately
resets
all
state
including
configuration,
control
messages
in
the
control
message
queue
,
and
all
pending
callbacks.
When invoked, run the Reset VideoDecoder algorithm.
-
close()
-
Immediately
aborts
all
pending
work
and
releases
system
resources
.
Close
is
final.
When invoked, run the Close VideoDecoder algorithm.
-
isConfigSupported(config)
-
Returns
a
promise
indicating
whether
the
provided
config
is
supported
by
the
user
agent.
NOTE: The returned
VideoDecoderSupport
config
will contain only the dictionary members that user agent recognized. Unrecognized dictionary memebers will be ignored. Authors may detect unrecognized dictionary members by comparingingconfig
to their provided config .When invoked, run these steps:
-
If config is not a valid VideoDecoderConfig , return a promise rejected with
TypeError
. -
Let p be a new Promise.
-
Let checkSupportQueue be the result of starting a new parallel queue .
-
Enqueue the following steps to checkSupportQueue :
-
Let decoderSupport be a newly constructed
VideoDecoderSupport
, initialized as follows:-
Set
config
to the result of running the Clone Configuration algorithm with config . -
Set
supported
to the result of running the Check Configuration Support algorithm with config .
-
-
Resolve p with decoderSupport .
-
-
Return p .
-
4.5. Algorithms
- Output VideoFrames (with outputs )
-
Run
these
steps:
-
For each output in outputs :
-
Let timestamp and duration be the
timestamp
andduration
from theEncodedVideoChunk
associated with output . -
Let frame be the result of running the Create a VideoFrame algorithm with output , timestamp , and duration .
-
Invoke
[[output callback]]
with frame .
-
-
- Create a VideoFrame (with output , timestamp , and duration )
-
-
Let planes be a sequence of
Plane
s containing the decoded video frame data from output . -
Let pixelFormat be the
PixelFormat
of planes . -
Let frameInit be a
VideoFrameInit
with the following keys:-
Assign timestamp to
timestamp
. -
Assign duration to
duration
. -
Let
codedWidth
andcodedHeight
be the width and height of the decoded video frame output in pixels, prior to any cropping or aspect ratio adjustments. -
Let
cropLeft
,cropTop
,cropWidth
, andcropHeight
be the crop region of the decoded video frame output in pixels, prior to any aspect ratio adjustments. -
Let
displayWidth
anddisplayHeight
be the display size of the decoded video frame in pixels.
-
-
Return a new
VideoFrame
, constructed with pixelFormat , planes , and frameInit .
-
- Reset VideoDecoder
-
Run
these
steps:
-
If
state
is"closed"
, throw anInvalidStateError
. -
Set
state
to"unconfigured"
. -
Signal
[[codec implementation]]
to cease producing output for the previous configuration. -
Set
decodeQueueSize
to zero.
-
- Close VideoDecoder (with error )
-
Run
these
steps:
-
Run the Reset VideoDecoder algorithm.
-
Set
state
to"closed"
. -
Clear
[[codec implementation]]
and release associated system resources . -
If error is set, queue a task on the control thread event loop to invoke the
[[error callback]]
with error .
-
5. AudioEncoder Interface
[Exposed =(Window ,DedicatedWorker )]interface {
AudioEncoder constructor (AudioEncoderInit );
init readonly attribute CodecState state ;readonly attribute long encodeQueueSize ;undefined configure (AudioEncoderConfig );
config );undefined encode (AudioData );
data Promise <undefined >flush ();undefined reset ();undefined close ();static Promise <AudioEncoderSupport >isConfigSupported (AudioEncoderConfig ); };
config dictionary {
AudioEncoderInit required EncodedAudioChunkOutputCallback ;
output required WebCodecsErrorCallback ; };
error );callback =
EncodedAudioChunkOutputCallback undefined (EncodedAudioChunk );
output
5.1. Internal Slots
-
[[codec implementation]]
- Underlying encoder implementation provided by the User Agent.
-
[[output callback]]
- Callback given at construction for encoded outputs.
-
[[error callback]]
- Callback given at construction for encode errors.
5.2. Constructors
AudioEncoder(init)
-
Let e be a new AudioEncoder object.
-
Assign
init.output
to the[[output callback]]
internal slot. -
Assign
init.error
to the[[error callback]]
internal slot. -
Assign "unconfigured" to
e.state
. -
Return e.
5.3. Attributes
-
state
, of type CodecState , readonly - Describes the current state of the codec.
-
encodeQueueSize
, of type long , readonly - The number of pending encode requests. This number will decrease as the underlying codec is ready to accept new input.
5.4. Methods
-
configure(config)
-
Enqueues
a
control
message
to
configure
the
audio
encoder
for
decoding
chunks
as
described
by
config
.
NOTE: This method will trigger a
NotSupportedError
if the user agent does not support config . Authors should first check support by callingisConfigSupported()
with config . User agents are not required to support any particular codec type or configuration.When invoked, run these steps:
-
If config is not a valid AudioEncoderConfig , throw a
TypeError
. -
If
state
is"closed"
, throw anInvalidStateError
. -
Set
state
to"configured"
. -
Queue a control message to configure the encoder using config .
Running a control message to configure the encoder means performing these steps:
-
Let supported be the result of running the Check Configuration Support algorith with config .
-
If supported is
true
, assign[[codec implementation]]
with an implementation supporting config . -
Otherwise, run the Close AudioEncoder algorithm with
NotSupportedError
.
-
-
encode(frame)encode(data) -
Enqueues
a
control
message
to
encode
the
given
framedata .When invoked, run these steps:
-
If the value of
framedata ’s[[detached]]
internal slot istrue
, throw aTypeError
. -
If
state
is not"configured"
, throw anInvalidStateError
. -
Let
frameClonedataClone hold the result of running the CloneFrameAudioData algorithm withframedata . -
Increment
encodeQueueSize
. -
Queue a control message to encode
frameClonedataClone .
Running a control message to encode the
framedata means performing these steps.-
Attempt to use
[[codec implementation]]
to encode the media resource described byframeClonedataClone . -
If encoding results in an error, queue a task on the control thread event loop to run the Close AudioEncoder algorithm with
EncodingError
. -
Queue a task on the control thread event loop to decrement
encodeQueueSize
. -
Let encoded outputs be a list of encoded audio data outputs emitted by
[[codec implementation]]
. -
If encoded outputs is not empty, queue a task on the control thread event loop to run the Output EncodedAudioChunks algorithm with encoded outputs .
-
-
flush()
-
Completes
all
control
messages
in
the
control
message
queue
and
emits
all
outputs.
When invoked, run these steps:
-
If
state
is not"configured"
, return a promise rejected withInvalidStateError
DOMException
. -
Let promise be a new Promise.
-
Queue a control message to flush the codec with promise .
-
Return promise .
Running a control message to flush the codec means performing these steps with promise .
-
Signal
[[codec implementation]]
to emit all internal pending outputs . -
Let encoded outputs be a list of encoded audio data outputs emitted by
[[codec implementation]]
. -
If encoded outputs is not empty, queue a task on the control thread event loop to run the Output EncodedAudioChunks algorithm with encoded outputs .
-
Queue a task on the control thread event loop to resolve promise .
-
-
reset()
-
Immediately
resets
all
state
including
configuration,
control
messages
in
the
control
message
queue
,
and
all
pending
callbacks.
When invoked, run the Reset AudioEncoder algorithm.
-
close()
-
Immediately
aborts
all
pending
work
and
releases
system
resources
.
Close
is
final.
When invoked, run the Close AudioEncoder algorithm.
-
isConfigSupported(config)
-
Returns
a
promise
indicating
whether
the
provided
config
is
supported
by
the
user
agent.
NOTE: The returned
AudioEncoderSupport
config
will contain only the dictionary members that user agent recognized. Unrecognized dictionary memebers will be ignored. Authors may detect unrecognized dictionary members by comparingingconfig
to their provided config .When invoked, run these steps:
-
If config is not a valid AudioEncoderConfig , return a promise rejected with
TypeError
. -
Let p be a new Promise.
-
Let checkSupportQueue be the result of starting a new parallel queue .
-
Enqueue the following steps to checkSupportQueue :
-
Let encoderSupport be a newly constructed
AudioEncoderSupport
, initialized as follows:-
Set
config
to the result of running the Clone Configuration algorithm with config . -
Set
supported
to the result of running the Check Configuration Support algorithm with config .
-
-
Resolve p with encoderSupport .
-
-
Return p .
-
5.5. Algorithms
- Output EncodedAudioChunks (with outputs )
-
Run
these
steps:
-
For each output in outputs :
-
Let chunkInit be an
EncodedAudioChunkInit
with the following keys:-
Let
data
contain the encoded audio data from output . -
Let
type
be theEncodedAudioChunkType
of output . -
Let
timestamp
be thetimestamp
from theAudioFrameAudioData associated with output .
-
-
Let chunk be a new
EncodedAudioChunk
constructed with chunkInit . -
Invoke
[[output callback]]
with chunk .
-
-
- Reset AudioEncoder
-
Run
these
steps:
-
If
state
is"closed"
, throw anInvalidStateError
. -
Set
state
to"unconfigured"
. -
Signal
[[codec implementation]]
to cease producing output for the previous configuration. -
Set
encodeQueueSize
to zero.
-
- Close AudioEncoder (with error )
-
Run
these
steps:
-
Run the Reset AudioEncoder algorithm.
-
Set
state
to"closed"
. -
Clear
[[codec implementation]]
and release associated system resources . -
If error is set, queue a task on the control thread event loop invoke the
[[error callback]]
with error .
-
6. VideoEncoder Interface
[Exposed =(Window ,DedicatedWorker )]interface {
VideoEncoder constructor (VideoEncoderInit );
init readonly attribute CodecState state ;readonly attribute long encodeQueueSize ;undefined configure (VideoEncoderConfig );
config undefined encode (VideoFrame ,
frame optional VideoEncoderEncodeOptions = {});
options Promise <undefined >flush ();undefined reset ();undefined close ();static Promise <boolean >isConfigSupported (VideoEncoderConfig ); };
config dictionary {
VideoEncoderInit required EncodedVideoChunkOutputCallback ;
output required WebCodecsErrorCallback ; };
error callback =
EncodedVideoChunkOutputCallback undefined (EncodedVideoChunk ,
output VideoDecoderConfig ?);
output_config
6.1. Internal Slots
-
[[codec implementation]]
- Underlying encoder implementation provided by the User Agent.
-
[[output callback]]
- Callback given at construction for encoded outputs.
-
[[error callback]]
- Callback given at construction for encode errors.
-
[[active encoder config]]
-
The
VideoEncoderConfig
that is actively applied. -
[[active output config]]
-
The
VideoDecoderConfig
that describes how to decode the most recently emittedEncodedVideoChunk
.
6.2. Constructors
VideoEncoder(init)
-
Let e be a new VideoEncoder object.
-
Assign
init.output
to the[[output callback]]
internal slot. -
Assign
init.error
to the[[error callback]]
internal slot. -
Assign "unconfigured" to
e.state
. -
Return e.
6.3. Attributes
-
state
, of type CodecState , readonly - Describes the current state of the codec.
-
encodeQueueSize
, of type long , readonly - The number of pending encode requests. This number will decrease as the underlying codec is ready to accept new input.
6.4. Methods
-
configure(config)
-
Enqueues
a
control
message
to
configure
the
video
encoder
for
decoding
chunks
as
described
by
config
.
NOTE: This method will trigger a
NotSupportedError
if the user agent does not support config . Authors should first check support by callingisConfigSupported()
with config . User agents are not required to support any particular codec type or configuration.When invoked, run these steps:
-
If config is not a valid VideoEncoderConfig , throw a
TypeError
. -
If
state
is"closed"
, throw anInvalidStateError
. -
Set
state
to"configured"
. -
Queue a control message to configure the encoder using config .
Running a control message to configure the encoder means performing these steps:
-
Let supported be the result of running the Check Configuration Support algorith with config .
-
If supported is
true
, assign[[codec implementation]]
with an implementation supporting config . -
Otherwise, run the Close VideoEncoder algorithm with
NotSupportedError
and abort these steps. -
Set
[[active encoder config]]
toconfig
.
-
-
encode(frame, options)
-
Enqueues
a
control
message
to
encode
the
given
frame
.
When invoked, run these steps:
-
If the value of frame ’s
[[detached]]
internal slot istrue
, throw aTypeError
. -
If
state
is not"configured"
, throw anInvalidStateError
. -
Let frameClone hold the result of running the Clone Frame algorithm with frame .
-
Increment
encodeQueueSize
. -
Queue a control message to encode frameClone .
Running a control message to encode the frame means performing these steps.
-
Attempt to use
[[codec implementation]]
to encode frameClone according to options . -
If encoding results in an error, queue a task on the control thread event loop to run the Close VideoEncoder algorithm with
EncodingError
. -
Queue a task on the control thread event loop to decrement
encodeQueueSize
. -
Let encoded outputs be a list of encoded video data outputs emitted by
[[codec implementation]]
. -
If encoded outputs is not empty, queue a task on the control thread event loop to run the Output EncodedVideoChunks algorithm with encoded outputs .
-
-
flush()
-
Completes
all
control
messages
in
the
control
message
queue
and
emits
all
outputs.
When invoked, run these steps:
-
If
state
is not"configured"
, return a promise rejected withInvalidStateError
DOMException
. -
Let promise be a new Promise.
-
Queue a control message to flush the codec with promise .
-
Return promise .
Running a control message to flush the codec means performing these steps with promise .
-
Signal
[[codec implementation]]
to emit all internal pending outputs . -
Let encoded outputs be a list of encoded video data outputs emitted by
[[codec implementation]]
. -
If encoded outputs is not empty, queue a task on the control thread event loop to run the Output EncodedVideoChunks algorithm with encoded outputs .
-
Queue a task on the control thread event loop to resolve promise .
-
-
reset()
-
Immediately
resets
all
state
including
configuration,
control
messages
in
the
control
message
queue
,
and
all
pending
callbacks.
When invoked, run the Reset VideoEncoder algorithm.
-
close()
-
Immediately
aborts
all
pending
work
and
releases
system
resources
.
Close
is
final.
When invoked, run the Close VideoEncoder algorithm.
-
isConfigSupported(config)
-
Returns
a
promise
indicating
whether
the
provided
config
is
supported
by
the
user
agent.
NOTE: The returned
VideoEncoderSupport
config
will contain only the dictionary members that user agent recognized. Unrecognized dictionary memebers will be ignored. Authors may detect unrecognized dictionary members by comparingingconfig
to their provided config .When invoked, run these steps:
-
If config is not a valid VideoEncoderConfig , return a promise rejected with
TypeError
. -
Let p be a new Promise.
-
Let checkSupportQueue be the result of starting a new parallel queue .
-
Enqueue the following steps to checkSupportQueue :
-
Let encoderSupport be a newly constructed
VideoEncoderSupport
, initialized as follows:-
Set
config
to the result of running the Clone Configuration algorithm with config . -
Set
supported
to the result of running the Check Configuration Support algorithm with config .
-
-
Resolve p with encoderSupport .
-
-
Return p .
-
6.5. Algorithms
- Output EncodedVideoChunks (with outputs )
-
Run
these
steps:
-
For each output in outputs :
-
Let encoder_config be the
[[active encoder config]]
.The intent is for encoder_config to be the
[[active encoder config]]
that was used to encode output . But, as written, it may occur that output was encoded using a previousVideoEncoderConfig
that has since been replaced by a later call toconfigure()
. See #138 . -
Let output_config be a
VideoDecoderConfig
that describes output . Initialize output_config as follows:-
Assign
encoder_config.codec
tooutput_config.codec
. -
Assign
encoder_config.width
tooutput_config.cropWidth
. -
Assign
encoder_config.height
tooutput_config.cropHeight
. -
Assign
encoder_config.displayWidth
tooutput_config.displayWidth
. -
Assign
encoder_config.displayHeight
tooutput_config.displayHeight
. -
Assign the remaining keys of
output_config
as determined by[[codec implementation]]
. The user agent must ensure that the configuration is completely described such that output_config could be used to correctly decode output .NOTE: This includes supplying the
description
to describe codec specific "extradata", the use of which may be further described in codec registrations listed in the [WEBCODECS-CODEC-REGISTRY] .
-
-
If output_config and
[[active output config]]
are equal dictionaries , set output_config to null. Otherwise, set[[active output config]]
to output_config .NOTE: The
VideoDecoderConfig
output_config will benull
if the configuration hasn’t changed from previous outputs. The first output will always include a non-null output_config . -
Let chunkInit be an
EncodedVideoChunkInit
with the following keys:-
Let
data
contain the encoded video data from output . -
Let
type
be theEncodedVideoChunkType
of output . -
Let
timestamp
be the[[timestamp]]
from theVideoFrame
associated with output . -
Let
duration
be the[[duration]]
from theVideoFrame
associated with output .
-
-
Let chunk be a new
EncodedVideoChunk
constructed with chunkInit . -
Invoke
[[output callback]]
with chunk .
-
-
- Reset VideoEncoder
-
Run
these
steps:
-
If
state
is"closed"
, throw anInvalidStateError
. -
Set
state
to"unconfigured"
. -
Set
[[active encoder config]]
tonull
. -
Set
[[active output config]]
tonull
. -
Signal
[[codec implementation]]
to cease producing output for the previous configuration. -
Set
encodeQueueSize
to zero.
-
- Close VideoEncoder (with error )
-
Run
these
steps:
-
Run the Reset VideoEncoder algorithm.
-
Set
state
to"closed"
. -
Clear
[[codec implementation]]
and release associated system resources . -
If error is set, queue a task on the control thread event loop invoke the
[[error callback]]
with error .
-
7. Configurations
7.1. Check Configuration Support (with config )
Run these steps:-
If the user agent can provide a codec to support all entries of the config , including applicable default values for keys that are not included, return
true
.NOTE: The types
AudioDecoderConfig
,VideoDecoderConfig
,AudioEncoderConfig
, andVideoEncoderConfig
each define their respective configuration entries and defaults.NOTE: Support for a given configuration may change dynamically if the hardware is altered (e.g. external GPU unplugged) or if required hardware resources are exhausted. User agents should describe support on a best-effort basis given the resources that are available at the time of the query.
-
Otherwise, return false.
7.2. Clone Configuration (with config )
NOTE: This algorithm will copy only the dictionary members that the user agent recognizes as part of the dictionary type.
Run these steps:
-
Let dictType be the type of dictionary config .
-
Let clone be a new empty instance of dictType .
-
For each dictionary member m defined on dictType :
-
If
config[m]
is a nested dictionary, setclone[m]
to the result of recursively running the Clone Configuration algorithm withconfig[m]
. -
Otherwise, assign the value of
config[m]
toclone[m]
.
7.3. Signalling Configuration Support
7.3.1. AudioDecoderSupport
dictionary {
AudioDecoderSupport boolean supported ;AudioDecoderConfig config ; };
-
supported
, of type boolean -
A
boolean
indicating
the
whether
the
corresponding
config
is supported by the user agent. -
config
, of type AudioDecoderConfig -
An
AudioDecoderConfig
used by the user agent in determining the value ofsupported
.
7.3.2. VideoDecoderSupport
dictionary {
VideoDecoderSupport boolean supported ;VideoDecoderConfig config ; };
-
supported
, of type boolean -
A
boolean
indicating
the
whether
the
corresponding
config
is supported by the user agent. -
config
, of type VideoDecoderConfig -
A
VideoDecoderConfig
used by the user agent in determining the value ofsupported
.
7.3.3. AudioEncoderSupport
dictionary {
AudioEncoderSupport boolean supported ;AudioEncoderConfig config ; };
-
supported
, of type boolean -
A
boolean
indicating
the
whether
the
corresponding
config
is supported by the user agent. -
config
, of type AudioEncoderConfig -
An
AudioEncoderConfig
used by the user agent in determining the value ofsupported
.
7.3.4. VideoEncoderSupport
dictionary {
VideoEncoderSupport boolean supported ;VideoEncoderConfig config ; };
-
supported
, of type boolean -
A
boolean
indicating
the
whether
the
corresponding
config
is supported by the user agent. -
config
, of type VideoEncoderConfig -
A
VideoEncoderConfig
used by the user agent in determining the value ofsupported
.
7.4. Codec String
A codec string describes a given codec format to be used for encoding or decoding.A valid codec string must meet the following conditions.
-
Is valid per the relevant codec specification (see examples below).
-
It describes a single codec.
-
It is unambiguous about codec profile and level for codecs that define these concepts.
NOTE:
In
other
media
specifications,
codec
strings
historically
accompanied
a
MIME
type
as
the
"codecs="
parameter
(
isTypeSupported()
,
canPlayType()
)
[RFC6381]
.
In
this
specification,
encoded
media
is
not
containerized;
hence,
only
the
value
of
the
codecs
parameter
is
accepted.
The format and semantics for codec strings are defined by codec registrations listed in the [WEBCODECS-CODEC-REGISTRY] . A compliant implementation may support any combination of codec registrations or none at all.
7.5. AudioDecoderConfig
dictionary {
AudioDecoderConfig required DOMString codec ;required unsigned long sampleRate ;required unsigned long numberOfChannels ;BufferSource description ; };
To
check
if
an
AudioDecoderConfig
is
a
valid
AudioDecoderConfig
,
run
these
steps:
-
If codec is not a valid codec string , return
false
. -
Return
true
.
-
codec
, of type DOMString - Contains a codec string describing the codec.
-
sampleRate
, of type unsigned long - The number of frame samples per second.
-
numberOfChannels
, of type unsigned long - The number of audio channels.
-
description
, of type BufferSource -
A
sequence
of
codec
specific
bytes,
commonly
known
as
extradata.
NOTE: The registrations in the [WEBCODECS-CODEC-REGISTRY] describe whether/how to populate this sequence, corresponding to the provided
codec
.
7.6. VideoDecoderConfig
dictionary {
VideoDecoderConfig required DOMString codec ;BufferSource description ;unsigned long codedWidth ;unsigned long codedHeight ;unsigned long cropLeft ;unsigned long cropTop ;unsigned long cropWidth ;unsigned long cropHeight ;unsigned long displayWidth ;unsigned long displayHeight ;HardwareAcceleration hardwareAcceleration = "allow"; };
To
check
if
a
VideoDecoderConfig
is
a
valid
VideoDecoderConfig
,
run
these
steps:
-
If
codec
is not a valid codec string , returnfalse
. -
If
codedWidth
= 0 orcodedHeight
= 0, returnfalse
. -
If
cropWidth
= 0 orcropHeight
= 0, returnfalse
. -
If
cropTop
+cropHeight
>=codedHeight
, returnfalse
. -
If
cropLeft
+cropWidth
>=codedWidth
, returnfalse
. -
If
displayWidth
= 0 ordisplayHeight
= 0, returnfalse
. -
Return
true
.
-
codec
, of type DOMString - Contains a codec string describing the codec.
-
description
, of type BufferSource -
A
sequence
of
codec
specific
bytes,
commonly
known
as
extradata.
NOTE: The registrations in the [WEBCODECS-CODEC-REGISTRY] may describe whether/how to populate this sequence, corresponding to the provided
codec
. -
codedWidth
, of type unsigned long - Width of the VideoFrame in pixels, prior to any cropping or aspect ratio adjustments.
-
codedHeight
, of type unsigned long - Height of the VideoFrame in pixels, prior to any cropping or aspect ratio adjustments.
-
cropLeft
, of type unsigned long - The number of pixels to remove from the left of the VideoFrame, prior to aspect ratio adjustments. Defaults to zero if not present.
-
cropTop
, of type unsigned long - The number of pixels to remove from the top of the VideoFrame, prior to aspect ratio adjustments. Defaults to zero if not present.
-
cropWidth
, of type unsigned long - The width in pixels to include in the crop, starting from cropLeft. Defaults to codedWidth if not present.
-
cropHeight
, of type unsigned long - The height in pixels to include in the crop, starting from cropLeft. Defaults to codedHeight if not present.
-
displayWidth
, of type unsigned long - Width of the VideoFrame when displayed. Defaults to cropWidth if not present.
-
displayHeight
, of type unsigned long - Height of the VideoFrame when displayed. Defaults to cropHeight if not present.
-
hardwareAcceleration
, of type HardwareAcceleration , defaulting to"allow"
-
Configures
hardware
acceleration
for
this
codec.
See
HardwareAcceleration
.
7.7. AudioEncoderConfig
dictionary {
AudioEncoderConfig required DOMString codec ;unsigned long sampleRate ;unsigned long numberOfChannels ;unsigned long long bitrate ; };
NOTE:
Codec-specific
extensions
to
AudioEncoderConfig
may
be
defined
by
the
registrations
in
the
[WEBCODECS-CODEC-REGISTRY]
.
To
check
if
an
AudioEncoderConfig
is
a
valid
AudioEncoderConfig
,
run
these
steps:
-
If
codec
is not a valid codec string , returnfalse
. -
Return
true
.
-
codec
, of type DOMString - Contains a codec string describing the codec.
-
sampleRate
, of type unsigned long - The number of frame samples per second.
-
numberOfChannels
, of type unsigned long - The number of audio channels.
-
bitrate
, of type unsigned long long - The average bitrate of the encoded audio given in units of bits per second.
7.8. VideoEncoderConfig
dictionary {
VideoEncoderConfig required DOMString codec ;unsigned long long bitrate ;required unsigned long width ;required unsigned long height ;unsigned long displayWidth ;unsigned long displayHeight ;HardwareAcceleration hardwareAcceleration = "allow"; };
NOTE:
Codec-specific
extensions
to
VideoEncoderConfig
may
be
defined
by
the
registrations
in
the
[WEBCODECS-CODEC-REGISTRY]
.
To
check
if
a
VideoEncoderConfig
is
a
valid
VideoEncoderConfig
,
run
these
steps:
-
If
codec
is not a valid codec string , returnfalse
. -
If
displayWidth
= 0 ordisplayHeight
= 0, returnfalse
. -
Return
true
.
-
codec
, of type DOMString - Contains a codec string describing the codec.
-
bitrate
, of type unsigned long long - The average bitrate of the encoded video given in units of bits per second.
-
width
, of type unsigned long -
The
encoded
width
of
output
EncodedVideoChunk
s in pixels, prior to any display aspect ratio adjustments.The encoder must scale any
VideoFrame
who’s[[crop width]]
differs from this value. -
height
, of type unsigned long -
The
encoded
height
of
output
EncodedVideoChunk
s in pixels, prior to any display aspect ratio adjustments.The encoder must scale any
VideoFrame
who’s[[crop height]]
differs from this value.
-
displayWidth
, of type unsigned long -
The
intended
display
width
of
output
EncodedVideoChunk
s in pixels. Defaults towidth
if not present. -
displayHeight
, of type unsigned long -
The
intended
display
height
of
output
EncodedVideoChunk
s in pixels. Defaults towidth
if not present.
displayWidth
or
displayHeight
that
differs
from
width
and
height
signals
that
chunks
should
be
scaled
after
decoding
to
arrive
at
the
final
display
aspect
ratio.
For many codecs this is merely pass-through information, but some codecs may optionally include display sizing in the bitstream.
-
hardwareAcceleration
, of type HardwareAcceleration , defaulting to"allow"
-
Configures
hardware
acceleration
for
this
codec.
See
HardwareAcceleration
.
7.9. Hardware Acceleration
enum {
HardwareAcceleration "allow" ,"deny" ,"require" , };
When supported, hardware acceleration offloads encoding or decoding to specialized hardware.
allow
.
This
gives
the
user
agent
flexibility
to
optimize
based
on
its
knowledge
of
the
system
and
configuration.
A
common
strategy
will
be
to
prioritize
hardware
acceleration
at
higher
resolutions
with
a
fallback
to
software
codecs
if
hardware
acceleration
fails.
Authors should carefully weigh the tradeoffs setting a hardware acceleration preference. The precise trade-offs will be device-specific, but authors should generally expect the following:
-
Setting a value of
require
may significantly restrict what configurations are supported. It may occur that the user’s device does not offer acceleration for any codec, or only for the most common profiles of older codecs. -
Hardware acceleration does not simply imply faster encoding / decoding. Hardware acceleration often has higher startup latency but more consistent throughput performance. Acceleration will generally reduce CPU load.
-
For decoding, hardware acceleration is often less robust to inputs that are mislabeled or violate the relevant codec specification.
-
Hardware acceleration will often be more power efficient than purely software based codecs.
-
For lower resolution content, the overhead added by hardware acceleration may yield decreased performance and power efficiency compared to purely software based codecs.
Given these tradeoffs, a good example of using "require" would be if an author intends to provide their own software based fallback via WebAssembly.
Alternatively, a good example of using "disallow" would be if an author is especially sensitive to the higher startup latency or decreased robustness generally associated with hardware acceleration.
-
allow
- Indicates that the user agent may use hardware acceleration if it is available and compatible with other aspects of the codec configuration.
-
deny
-
Indicates
that
the
user
agent
must
not
use
hardware
acceleration.
NOTE: This will cause the configuration to be unsupported on platforms where an unaccelerated codec is unavailable or is incompatible with other aspects of the codec configuration.
-
require
-
Indicates
that
the
user
agent
must
use
hardware
acceleration.
NOTE: This will cause the configuration to be unsupported on platforms where an accelerated codec is unavailable or is incompatible with other aspects of the codec configuration.
7.10. Configuration Equivalence
Two dictionaries are equal dictionaries if they contain the same keys and values. For nested dictionaries, apply this definition recursively.7.11. VideoEncoderEncodeOptions
dictionary {
VideoEncoderEncodeOptions boolean keyFrame =false ; };
-
keyFrame
, of type boolean , defaulting tofalse
-
A
value
of
true
indicates that the given frame MUST be encoded as a key frame. A value offalse
indicates that the user agent has flexibility to decide whether the frame will be encoded as a key frame.
7.12. CodecState
enum {
CodecState "unconfigured" ,"configured" ,"closed" };
-
unconfigured
- The codec is not configured for encoding or decoding.
-
configured
- A valid configuration has been provided. The codec is ready for encoding or decoding.
-
closed
- The codec is no longer usable and underlying system resources have been released.
7.13. WebCodecsErrorCallback
callback =
WebCodecsErrorCallback undefined (DOMException );
error
8. Encoded Media Interfaces (Chunks)
These interfaces represent chunks of encoded media.8.1. EncodedAudioChunk Interface
[Exposed =(Window ,DedicatedWorker )]interface {
EncodedAudioChunk constructor (EncodedAudioChunkInit );
init readonly attribute EncodedAudioChunkType type ;; // microsecondsreadonly attribute unsigned long long timestamp ; // microsecondsreadonly attribute ArrayBuffer data ; };dictionary {
EncodedAudioChunkInit required EncodedAudioChunkType ;
type required unsigned long long ;
timestamp required BufferSource ; };
data enum {
EncodedAudioChunkType ,
"key" , };
"delta"
8.1.1. Constructors
EncodedAudioChunk(init)
-
Let chunk be a new
EncodedAudioChunk
object, initialized as follows-
Assign
init.type
tochunk.type
. -
Assign
init.timestamp
tochunk.timestamp
. -
Assign a copy of
init.data
tochunk.data
.
-
-
Return chunk .
8.1.2. Attributes
-
type
, of type EncodedAudioChunkType , readonly - Describes whether the chunk is a key frame.
-
timestamp
, of type unsigned long long , readonly - The presentation timestamp, given in microseconds.
-
data
, of type ArrayBuffer , readonly - A sequence of bytes containing encoded audio data.
8.2. EncodedVideoChunk Interface
[Exposed =(Window ,DedicatedWorker )]interface {
EncodedVideoChunk constructor (EncodedVideoChunkInit );
init readonly attribute EncodedVideoChunkType type ;readonly attribute unsigned long long timestamp ; // microsecondsreadonly attribute unsigned long long ?duration ; // microsecondsreadonly attribute ArrayBuffer data ; };dictionary {
EncodedVideoChunkInit required EncodedVideoChunkType ;
type required unsigned long long ;
timestamp unsigned long long ;
duration required BufferSource ; };
data enum {
EncodedVideoChunkType ,
"key" , };
"delta"
8.2.1. Constructors
EncodedVideoChunk(init)
-
Let chunk be a new
EncodedVideoChunk
object, initialized as follows-
Assign
init.type
tochunk.type
. -
Assign
init.timestamp
tochunk.timestamp
. -
If duration is present in init, assign
init.duration
tochunk.duration
. Otherwise, assign null tochunk.duration
.
-
-
Assign a copy of
init.data
tochunk.data
. -
Return chunk .
8.2.2. Attributes
-
type
, of type EncodedVideoChunkType , readonly - Describes whether the chunk is a key frame or not.
-
timestamp
, of type unsigned long long , readonly - The presentation timestamp, given in microseconds.
-
duration
, of type unsigned long long , readonly, nullable - The presentation duration, given in microseconds.
-
data
, of type ArrayBuffer , readonly - A sequence of bytes containing encoded video data.
9.
Raw
Media
Interfaces
(Frames)
These
interfaces
represent
unencoded
(raw)
media.
9.1.
AudioFrame
Memory
Model
9.1.1. Background
This section is non-normative.
Decoded
media
data
may
occupy
a
large
amount
of
system
memory.
To
minimize
the
need
for
expensive
copies,
this
specification
defines
a
scheme
for
reference
counting
(
clone()
and
close()
).
9.1.2. Reference Counting
A
media
resource
is
storage
for
the
actual
pixel
data
or
the
audio
sample
data
described
by
a
VideoFrame
or
AudioData
.
The
AudioData
[[resource
reference]]
and
VideoFrame
[[resource
reference]]
internal
slots
hold
a
reference
to
a
media
resource
.
VideoFrame
.
clone()
and
AudioData
.
clone()
return
new
frame
objects
who’s
[[resource
reference]]
points
to
the
same
media
resource
as
the
original
frame.
VideoFrame
.
close()
and
AudioData
.
close()
will
clear
their
[[resource
reference]]
slot,
releasing
the
reference
their
media
resource
A
media
resource
must
remain
alive
at
least
as
long
as
it
continues
to
be
referenced
by
a
[[resource
reference]]
.
NOTE:
When
a
media
resource
is
no
longer
referenced
by
a
[[resource
reference]]
,
the
resource
may
be
destroyed.
User
agents
are
encouraged
to
destroy
such
resources
quickly
to
reduce
memory
pressure
and
facilitate
resouce
reuse.
9.2. AudioData Interface
[Exposed =(Window ,DedicatedWorker )]{ ); ; ; ();interface {
AudioData constructor (AudioDataInit );
init readonly attribute AudioSampleFormat sampleFormat ;readonly attribute float sampleRate ;readonly attribute unsigned long numberOfFrames ;readonly attribute unsigned long numberOfChannels ;readonly attribute unsigned long allocationSize ;readonly attribute unsigned long long duration ;readonly attribute unsigned long long timestamp ;undefined copyTo ([AllowShared ]BufferSource ,
destination unsigned long );
planeNumber AudioData clone ();undefined close (); };{ ; ;dictionary {
AudioDataInit required AudioSampleFormat ;
sampleFormat required float ;
sampleRate required unsigned long ;
numberOfFrames required unsigned long ;
numberOfChannels required unsigned long long ;
timestamp required BufferSource ; };
data
9.1.1.
9.2.1.
Internal
Slots
-
[[detached]]
-
Boolean indicating whether
close()
was invokedand underlying resources have been released.on thisAudioData
. [[resource reference]]
A reference to a media resource that stores the audio sample data for this
AudioData
.[[sample format]]
The
AudioSampleFormat
used by thisAudioData
.[[sample rate]]
The sample-rate, in Hz, for this
AudioData
.[[number of frames]]
The number of frames (samples per channel) for this
AudioData
.[[number of channels]]
The number of audio channels for this
AudioData
.[[timestamp]]
The presentation timestamp, in microseconds, for this
AudioData
.
9.1.2.
9.2.2.
Constructors
AudioFrame(init)
AudioData(init)
-
Let frame be a new
AudioFrameAudioDataobject.object, initialized as follows:-
Assign
init.timestampfalseframe.timestamp[[detached]]
. -
Assign
init.bufferinit .sampleFormat
toframe.buffer[[sample format]]
. -
Assign
falseinit .sampleRate
tothe[[detached]][[sample rate]] Assign init .
numberOfFrames
internal slot.to[[number of frames]]
.Assign init .
numberOfChannels
to[[number of channels]]
.Assign init .
timestamp
to[[timestamp]]
.Let resource be a media resource containing a copy of init .
data
.Let resourceReference be a reference to resource .
Assign resourceReference to
[[resource reference]]
.
-
-
Return frame .
9.1.3.
9.2.3.
Attributes
-
sampleFormat
, of type AudioSampleFormat , readonly The
AudioSampleFormat
used by thisAudioData
.The
sampleFormat
getter steps are to return[[sample format]]
.sampleRate
, of type float , readonlyThe sample-rate, in Hz, for this
AudioData
.The
sampleRate
getter steps are to return[[sample rate]]
.numberOfFrames
, of type unsigned long , readonlyThe number of frames (samples per channel) for this
AudioData
.The
numberOfFrames
getter steps are to return[[number of frames]]
.numberOfChannels
, of type unsigned long , readonlyThe number of audio channels for this
AudioData
.The
numberOfChannels
getter steps are to return[[number of channels]]
.allocationSize
, of type unsigned long , readonlyThe the number of bytes allocated to hold all of the samples in this
AudioData
.The
allocationSize
getter steps are to:Let sampleSize be the number of bytes per sample, as defined by the
[[sample format]]
.Return the product of multiplying sampleSize by
[[number of channels]]
and[[number of frames]]
.
-
timestamp
, of type unsigned long long , readonly -
The presentation timestamp,
giveninmicroseconds.microseconds, for thisAudioData
.The
numberOfChannels
getter steps are to return[[timestamp]]
. -
bufferdurationAudioBufferunsigned long long ,readonly, nullablereadonly -
The
buffer containing decoded audio data.duration, in microseconds, for thisAudioData
.The
duration
getter steps are to:Let microsecondsPerSecond be
1,000,000
.Let durationInSeconds be the result of dividing
[[number of frames]]
by[[sample rate]]
.Return the product of durationInSeconds and microsecondsPerSecond .
9.1.4.
9.2.4.
Methods
-
close()copyTo( destination , planeNumber )Immediately frees system resources -
Copies the samples from the specified plane of the
AudioData
to the destination buffer.When invoked, run these steps:
If the value of frame ’s
[[detached]]
internal slot istrue
, throw anInvalidStateError
DOMException
.Let allocationSize be the number of bytes allocated to hold this
AudioData
's media resource , as described by the getter steps ofallocationSize
.If allocationSize is greater than
destination.byteLength
, throw aTypeError
.Let resource be the media resource referenced by
[[resource reference]]
.Let planeBytes be the region of resource corresponding to planeNumber .
Copy the planeBytes into destination .
clone()
Creates a new AudioData with a reference to the same media resource .
When invoked, run these steps:
-
Release system resourcesIf the value of frame ’s[[detached]]
internal slot isfor buffertrue
, throw anInvalidStateError
DOMException
. Return the result of running the Clone AudioData algorithm with this .
-
close()
Clears all state and
set its valuereleases the reference tonull.the media resource . Close is final.When invoked, run these steps:
-
Assign
true
to the[[detached]]
internal slot. -
Assign
null
to[[resource reference]]
.
-
NOTE:
9.2.5.
This
section
needs
work.
We
should
use
Algorithms
- Clone AudioData (with data )
Run these steps:
Let clone be a new
AudioData
initialized as follows:Let resource be the
name and semanticsmedia resource refrenced by data ’s[[resource reference]]
.Let reference be a new reference to resource .
Assign reference to
[[resource reference]]
.Assign the values of
VideoFrame destroy(). Similarly, we should add clone()data ’s[[detached]]
,[[sample format]]
,[[sample rate]]
,[[number of frames]]
,[[number of channels]]
, and[[timestamp]]
slots tomakethe corresponding slots in clone .
Return clone .
9.3. Audio Sample Format
Audo sample formats describe the numeric type used to represent aenum {
AudioSampleFormat "U8" ,"S16" ,"S32" ,"FLT" ,"S16P" ,"S32P" ,"FLTP" , };
U8
8-bit unsigned integer samples with interleaved channel arrangement.
S16
16-bit signed integer samples with interleaved channel arrangement.
S32
32-bit signed integer samples with interleaved channel arrangement.
FLT
32-bit float samples with interleaved channel arrangement.
S16P
16-bit signed integer samples with planar channel arrangement.
S32P
32-bit signed integer samples with planar channel arrangement.
FLTP
32-bit float samples with planar channel arrangement.
9.2.
9.4.
VideoFrame
Interface
[Exposed =(Window ,DedicatedWorker )]interface {
VideoFrame constructor (ImageBitmap ,
imageBitmap optional VideoFrameInit = {});
frameInit constructor (PixelFormat ,
pixelFormat sequence <(Plane or PlaneInit )>,
planes optional VideoFrameInit = {});
frameInit readonly attribute PixelFormat format ;readonly attribute FrozenArray <Plane >planes ;; ; ; ; ; ; ; ; ; ;readonly attribute unsigned long codedWidth ;readonly attribute unsigned long codedHeight ;readonly attribute unsigned long cropLeft ;readonly attribute unsigned long cropTop ;readonly attribute unsigned long cropWidth ;readonly attribute unsigned long cropHeight ;readonly attribute unsigned long displayWidth ;readonly attribute unsigned long displayHeight ;readonly attribute unsigned long long ?duration ;readonly attribute unsigned long long ?timestamp ;(); ();undefined destroy ();VideoFrame clone ();Promise <ImageBitmap >createImageBitmap (optional ImageBitmapOptions = {}); };
options dictionary {
VideoFrameInit ; ; ; ; ; ; ; ; ; ;unsigned long ;
codedWidth unsigned long ;
codedHeight unsigned long ;
cropLeft unsigned long ;
cropTop unsigned long ;
cropWidth unsigned long ;
cropHeight unsigned long ;
displayWidth unsigned long ;
displayHeight unsigned long long ;
duration unsigned long long ; };
timestamp
9.2.1.
9.4.1.
Internal
Slots
-
[[detached]]
-
A boolean indicating whether
destroy()
was invoked and underlying resources have been released. -
[[format]]
-
A
PixelFormat
describing the pixel format of theVideoFrame
. -
[[planes]]
-
A list of
Plane
s describing the memory layout of the pixel data inVideoFrame
. The number ofPlane
s and their semantics are determined by[[format]]
. -
[[coded width]]
-
Width of the
VideoFrame
in pixels, prior to any cropping or aspect ratio adjustments. -
[[coded height]]
-
Height of the
VideoFrame
in pixels, prior to any cropping or aspect ratio adjustments. -
[[crop left]]
-
The number of pixels to remove from the left of the
VideoFrame
, prior to aspect ratio adjustments. -
[[crop top]]
-
The number of pixels to remove from the top of the
VideoFrame
, prior to aspect ratio adjustments. -
[[crop width]]
-
The width of pixels to include in the crop, starting from cropLeft.
-
[[crop height]]
-
The height of pixels to include in the crop, starting from cropLeft.
-
[[display width]]
-
Width of the
VideoFrame
when displayed after applying aspect ratio adjustments. -
[[display height]]
-
Height of the
VideoFrame
when displayed after applying aspect ratio adjustments. -
[[duration]]
-
The presentation duration, given in microseconds. The duration is copied from the
EncodedVideoChunk
corresponding to thisVideoFrame
. -
[[timestamp]]
-
The presentation timestamp, given in microseconds. The timestamp is copied from the
EncodedVideoChunk
corresponding to thisVideoFrame
.
9.2.2.
9.4.2.
Constructors
NOTE:
this
section
needs
work.
Current
wording
assumes
a
VideoFrame
can
always
be
easily
represented
using
one
of
the
known
pixel
formats.
In
practice,
the
underlying
UA
resources
may
be
GPU
backed
or
formatted
in
such
a
way
that
conversion
to
an
allowed
pixel
format
requires
expensive
copies
and
translation.
When
this
occurs,
we
should
allow
planes
to
be
null
and
format
to
be
"opaque"
to
avoid
early
optimization.
We
should
make
conversion
explicit
and
user
controlled
by
offering
a
videoFrame.convertTo(format)
that
returns
a
Promise
containing
a
new
VideoFrame
for
which
the
copies/translations
are
performed.
VideoFrame(imageBitmap,
frameInit)
-
If frameInit is not a valid VideoFrameInit , throw a
TypeError
. -
If the value of imageBitmap ’s'
[[Detached]]
internal slot is set totrue
, then throw anInvalidStateError
DOMException. -
Let frame be a new
VideoFrame
. -
Assign
false
to frame ’s[[detached]]
internal slot. -
Use a copy of the pixel data in imageBitmap to initialize to following frame internal slots:
-
Initialize
[[format]]
be the underlying format of imageBitmap. -
Initialize
[[planes]]
to describe the arrangement of memory of the copied pixel data. -
Assign regions of the copied pixel data to the
[[plane buffer]]
internal slot of each plane as appropriate for the pixel format. -
Initialize
[[coded width]]
and[[coded height]]
to describe the width and height of the imageBitamp prior to any cropping or aspect ratio adjustments.
-
-
Use frameInit to initialize the remaining frame internal slots:
-
If
frameInit.cropLeft
is present, assign it to[[crop left]]
. Otherwise, assign0
to[[crop left]]
. -
If
frameInit.cropTop
is present, assign it to[[crop top]]
. Otherwise, assign0
to[[crop top]]
-
If
frameInit.cropWidth
is present, assign it to[[crop width]]
. Otherwise, assign[[coded width]]
to[[crop width]]
. -
If
frameInit.cropHeight
is present, assign it to[[crop height]]
. Otherwise, assign[[coded height]]
to[[crop height]]
. -
If
frameInit.displayWidth
is present, assign it to[[display width]]
. Otherwise, assign[[crop width]]
to[[display width]]
. -
If
frameInit.displayHeight
is present, assign it to[[display height]]
. Otherwise, assign[[crop height]]
to[[display height]]
. -
If
frameInit.duration
is present, assign it to[[duration]]
. Otherwise, assignnull
to[[duration]]
. -
If
frameInit.timestamp
is present, assign it to[[timestamp]]
. Otherwise, assignnull
to[[timestamp]]
.
-
-
Return frame .
VideoFrame(pixelFormat,
planes,
frameInit)
-
If either
codedWidth
orcodedHeight
is not present in frameInit , throw aTypeError
. -
If frameInit is not a valid VideoFrameInit , throw a
TypeError
. -
If the length of planes is incompatible with the given pixelFormat, throw a
TypeError
. -
Let frame be a new
VideoFrame
object. -
Assign
false
to frame ’s[[detached]]
internal slot. -
Assign
init.format
to frame ’s[[format]]
. -
For each element p in planes :
-
If p is a
Plane
, append a copy of p to frame ’s[[planes]]
and continue . -
If p is a
PlaneInit
, append a newPlane
q to frame ’s[[planes]]
, initialized as follows:-
Assign a copy of
p.src
to q’s[[plane buffer]]
internal slot.NOTE: the samples should be copied exactly, but the user agent may add row padding as needed to improve memory alignment.
-
Assign the width of each row in [[plane buffer]], including any padding, to
q.stride
. -
Assign
p.rows
toq.rows
. -
Assign the product of (
q.rows
*q.stride)
toq.length
-
-
-
Assign
frameInit.codedWidth
to frame ’s[[coded width]]
. -
Assign
frameInit.codedHeight
to frame ’s[[coded height]]
. -
If
frameInit.cropLeft
is present, assign it frame ’s[[crop left]]
. Otherwise, assign0
to[[crop left]]
. -
If
frameInit.cropTop
is present, assign it to frame ’s[[crop top]]
. Otherwise, assign0
to[[crop top]]
. -
If
frameInit.cropWidth
is present, assign it to frame ’s[[crop width]]
. Otherwise, assign[[coded width]]
to[[crop width]]
. -
If
frameInit.cropHeight
is present, assign it to frame ’s[[crop height]]
. Otherwise, assign[[coded height]]
to[[crop height]]
. -
If
frameInit.displayWidth
is present, assign it to frame ’s[[display width]]
. Otherwise, assign[[crop width]]
to[[display width]]
. -
If
frameInit.displayHeight
is present, assign it to frame ’s[[display height]]
. Otherwise, assign[[crop height]]
to[[display height]]
. -
If
frameInit.duration
is present, assign it to[[duration]]
. Otherwise, assignnull
to[[duration]]
. -
If
frameInit.timestamp
is present, assign it to[[timestamp]]
. Otherwise, assignnull
to[[timestamp]]
. -
Return frame.
9.2.3.
9.4.3.
Attributes
-
format
, of type PixelFormat , readonly -
Describes the arrangement of bytes in each plane as well as the number and order of the planes.
The
format
getter steps are to return[[format]]
. -
planes
, of type FrozenArray< Plane >, readonly -
Holds pixel data data, laid out as described by format and Plane attributes.
The
planes
getter steps are to return[[planes]]
. -
codedWidth
, of type unsigned long , readonly -
Width of the
VideoFrame
in pixels, prior to any cropping or aspect ratio adjustments.The
codedWidth
getter steps are to return[[coded width]]
. -
codedHeight
, of type unsigned long , readonly -
Height of the VideoFrame in pixels, prior to any cropping or aspect ratio adjustments.
The
codedHeight
getter steps are to return[[coded height]]
. -
cropLeft
, of type unsigned long , readonly -
The number of pixels to remove from the left of the VideoFrame, prior to aspect ratio adjustments.
The
cropLeft
getter steps are to return[[crop left]]
. -
cropTop
, of type unsigned long , readonly -
The number of pixels to remove from the top of the VideoFrame, prior to aspect ratio adjustments.
The
cropTop
getter steps are to return[[crop top]]
. -
cropWidth
, of type unsigned long , readonly -
The width of pixels to include in the crop, starting from cropLeft.
The
cropWidth
getter steps are to return[[crop width]]
. -
cropHeight
, of type unsigned long , readonly -
The height of pixels to include in the crop, starting from cropLeft.
The
cropHeight
getter steps are to return[[crop height]]
. -
displayWidth
, of type unsigned long , readonly -
Width of the VideoFrame when displayed after applying aspect ratio adjustments.
The
displayWidth
getter steps are to return[[display width]]
. -
displayHeight
, of type unsigned long , readonly -
Height of the VideoFrame when displayed after applying aspect ratio adjustments.
The
displayHeight
getter steps are to return[[display height]]
. -
timestamp
, of type unsigned long long , readonly, nullable -
The presentation timestamp, given in microseconds. The timestamp is copied from the
EncodedVideoChunk
corresponding to this VideoFrame.The
timestamp
getter steps are to return[[timestamp]]
. -
duration
, of type unsigned long long , readonly, nullable -
The presentation duration, given in microseconds. The duration is copied from the
EncodedVideoChunk
corresponding to this VideoFrame.The
duration
getter steps are to return[[duration]]
.
9.2.4.
9.4.4.
Methods
destroy()
Immediately
frees
system
resources
.
Destruction
applies
to
all
references,
including
references
that
are
serialized
and
passed
across
Realms.
NOTE:
Authors
should
take
care
to
manage
frame
lifetimes
by
calling
destroy()
immediately
when
frames
are
no
longer
needed.
NOTE: Use clone() to create a deep copy. Cloned frames have their own lifetime and will not be affected by destroying the original frame.
When invoked, run these steps:
-
If
[[detached]]
istrue
, throw anInvalidStateError
. -
Remove all
Plane
s from[[planes]]
and release associated memory. -
Assign
true
to the[[detached]]
internal slot.
clone()
Creates
a
new
VideoFrame
with
a
separate
lifetime
containing
a
deep
copy
of
this
frame’s
resources.
NOTE:
VideoFrames
may
require
a
large
amount
of
memory.
Use
clone()
sparingly.
When invoked, run the following steps:
-
If the value of the
[[detached]]
slot istrue
, return a promise rejected withInvalidStateError
DOMException
. -
Let p be a new Promise.
-
In parallel, resolve p with the result of running the Clone Frame algorithm with this .
-
Return p .
createImageBitmap(options)
Creates
an
ImageBitmap
from
this
VideoFrame
.
When invoked, run these steps:
-
Let p be a new Promise.
-
If either options ’s
resizeWidth
orresizeHeight
is present and is 0, then return p rejected with anInvalidStateError
DOMException
. -
If the this'
[[detached]]
internal slot is set totrue
, then return p rejected with anInvalidStateError
DOMException
. -
Let imageBitmap be a new
ImageBitmap
object. -
Set imageBitmap ’s bitmap data to a copy of the
VideoFrame
pixel data, at the frame’s intrinsic width and intrinsic height (i.e
., after any aspect-ratio correction has been applied), cropped to the source rectangle with formatting. -
If the origin of imageBitmap ’s image is not same origin with entry settings object’s origin, then set the origin-clean flag of imageBitmap ’s bitmap to
false
. -
Run this step in parallel:
-
Resolve p with imageBitmap.
9.2.5.
9.4.5.
Algorithms
To
check
if
a
VideoFrameInit
is
a
valid
VideoFrameInit
,
run
these
steps:
-
If
codedWidth
= 0 orcodedHeight
= 0, returnfalse
. -
If
cropWidth
= 0 orcropHeight
= 0, returnfalse
. -
If
cropTop
+cropHeight
>=codedHeight
, returnfalse
. -
If
cropLeft
+cropWidth
>=codedWidth
, returnfalse
. -
If
displayWidth
= 0 ordisplayHeight
= 0, returnfalse
. -
Return
true
.
9.3.
9.5.
Plane
Interface
A
Plane
acts
like
a
thin
wrapper
around
an
ArrayBuffer
,
but
may
actually
be
backed
by
a
texture.
Plane
s
hide
any
padding
before
the
first
sample
or
after
the
last
row.
A
Plane
is
solely
constructed
by
its
VideoFrame
.
During
construction,
the
User
Agent
may
use
knowledge
of
the
frame’s
PixelFormat
to
add
padding
to
the
Plane
to
improve
memory
alignment.
A
Plane
cannot
be
used
after
the
VideoFrame
is
destroyed.
A
new
VideoFrame
can
be
assembled
from
existing
Plane
s,
and
the
new
VideoFrame
will
remain
valid
when
the
original
is
destroyed.
This
makes
it
possible
to
efficiently
add
an
alpha
plane
to
an
existing
VideoFrame
.
[Exposed =(Window ,DedicatedWorker )]interface {
Plane ; ; ;readonly attribute unsigned long stride ;readonly attribute unsigned long rows ;readonly attribute unsigned long length ;);undefined readInto (ArrayBufferView ); };
dst dictionary {
PlaneInit ; ; ;required BufferSource ;
src required unsigned long ;
stride required unsigned long ; };
rows
9.3.1.
9.5.1.
Internal
Slots
-
[[parent frame]]
-
Refers
to
the
VideoFrame
that constructed and owns this plane. -
[[plane buffer]]
- Internal storage for the plane’s pixel data.
9.3.2.
9.5.2.
Attributes
-
stride
, of type unsigned long , readonly - The width of each row including any padding.
-
rows
, of type unsigned long , readonly - The number of rows.
-
length
, of type unsigned long , readonly - The total byte length of the plane (stride * rows).
9.3.3.
9.5.3.
Methods
readInto(dst)
Copies the plane data into dst.
When invoked, run these steps:
-
If
[[parent frame]]
has been destroyed, throw anInvalidStateError
. -
If
length
is greater than |dst.byteLength
|, throw aTypeError
. -
Copy the
[[plane buffer]]
into dst .
9.4.
9.6.
Pixel
Format
Pixel
formats
describe
the
arrangement
of
bytes
in
each
plane
as
well
as
the
number
and
order
of
the
planes.
NOTE: This section needs work. We expect to add more pixel formats and offer much more verbose definitions. For now, please see http://www.fourcc.org/pixel-format/yuv-i420/ for a more complete description.
enum {
PixelFormat "I420" };
-
I420
- Planar 4:2:0 YUV.
9.5.
9.7.
Algorithms
- Clone Frame (with frame )
-
-
Let cloneFrame be a new object of the same type as frame (either
AudioFrameAudioDataVideoFrame
). -
Initialize each attribute and internal slot of clone with a copy of the value from the corresponding attribute of this frame.
NOTE: User Agents are encouraged to avoid expensive copies of large objects (for instance,
VideoFrame
pixel data). Frame types are immutable, so the above step may be implemented using memory sharing techniques such as reference counting. -
Return cloneFrame .
-
10. Image Decoding
10.1. Background
This section is non-normative.
Image
codec
definitions
are
typically
accompanied
by
a
definition
for
a
corresponding
file
format.
Hence
image
decoders
often
perform
both
duties
of
unpacking
(demuxing)
as
well
as
decoding
the
encoded
image
data.
The
WebCodecs
ImageDecoder
follows
this
pattern,
which
motivates
an
interface
design
that
is
notably
different
from
that
of
VideoDecoder
and
AudioDecoder
.
In
spite
of
these
differences,
ImageDecoder
uses
the
same
codec
processing
model
as
the
other
codec
interfaces.
Additionally,
ImageDecoder
uses
the
VideoFrame
interface
to
describe
decoded
outputs.
10.2. ImageDecoder Interface
[Exposed =(Window ,DedicatedWorker )]interface {
ImageDecoder constructor (ImageDecoderInit );
init readonly attribute boolean complete ;;readonly attribute Promise <undefined >completed ;readonly attribute ImageTrackList tracks ;Promise <ImageDecodeResult >decode (optional ImageDecodeOptions );
options (); ();undefined reset ();undefined close ();static Promise <boolean >isTypeSupported (DOMString ); };
type
10.2.1. Internal Slots
-
[[ImageTrackList]]
-
An
ImageTrackList
describing the tracks found in[[encoded data]]
-
[[complete]]
-
A boolean indicating whether
[[encoded data]]
is completely buffered. -
[[completed promise]]
-
The promise used to signal when
[[complete]]
becomestrue
. -
[[codec implementation]]
-
An underlying image decoder implementation provided by the User Agent.
-
[[encoded data]]
-
A byte sequence containing the encoded image data to be decoded.
-
[[prefer animation]]
-
A boolean reflecting the value of
preferAnimation
given at construction. -
[[pending decode promises]]
-
A list of unresolved promises returned by calls to decode().
-
[[internal selected track index]]
-
Identifies the image track within
[[encoded data]]
that is used by decoding algorithms on the codec thread . -
[[tracks established]]
-
A boolean indicating whether the track list has been established in
[[ImageTrackList]]
. -
[[closed]]
-
A boolean indicating that the ImageDecoder is in a permanent closed state and can no longer be used.
-
[[progressive frame generations]]
-
A mapping of frame indices to Progressive Image Frame Generations . The values represent the Progressive Image Frame Generation for the
VideoFrame
which was most recently output by a call todecode()
with the given frame index.
10.2.2. Constructor
-
ImageDecoder(init)
-
NOTE: Calling
decode()
on the constructedImageDecoder
will trigger aNotSupportedError
if the user agent does not support type . Authors should first check support by callingisTypeSupported()
with type . User agents are not required to support any particular type.When invoked, run these steps:
-
If init is not valid ImageDecoderInit , throw a
TypeError
. -
Let d be a new
ImageDecoder
object. In the steps below, all mentions ofImageDecoder
members apply to d unless stated otherwise. -
Assign
[[ImageTrackList]]
a newImageTrackList
initialized as follows:-
Assign a new list to
[[track list]]
. -
Assign
-1
to[[selected index]]
.
-
-
Assign
null
to[[codec implementation]]
. -
If
init.preferAnimation
exists , assigninit.preferAnimation
to the[[prefer animation]]
internal slot. Otherwise, assign 'null' to[[prefer animation]]
internal slot. -
Assign a new list to
[[pending decode promises]]
. -
Assign
-1
to[[internal selected track index]]
. -
Assign
false
to[[tracks established]]
. -
Assign
false
to[[closed]]
. -
Assign a new map to
[[progressive frame generations]]
. -
If init ’s
data
member is of typeReadableStream
:-
Assign a new list to
[[encoded data]]
. -
Assign
false
tocomplete
-
Queue a control message to configure the image decoder with init .
-
Let reader be the result of getting a reader for
data
. -
In parallel, perform the Fetch Stream Data Loop on d with reader .
-
-
Otherwise:
-
Assert that
init.data
is of typeBufferSource
. -
Assign a copy of
init.data
to[[encoded data]]
. -
Assign
true
tocomplete
. -
Reslove
[[completed promise]]
. -
Queue a control message to configure the image decoder with init .
-
Queue a control message to decode track metadata .
-
-
return d .
Running a control message to configure the image decoder means running these steps:
-
Let supported be the result of running the Check Type Support algorithm with
init.type
. -
If supported is
false
, queue a task on the control thread event loop to run the Close ImageDecoder algorithm with aNotSupportedError
DOMException
and abort these steps. -
If supported is
true
, assign the[[codec implementation]]
internal slot with an implementation supportinginit.type
-
Configure
[[codec implementation]]
in accordance with the values given forpremultiplyAlpha
,colorSpaceConversion
,desiredWidth
, anddesiredHeight
.
Running a control message to decode track metadata means running these steps:
-
Run the Establish Tracks algorithm.
-
10.2.3. Attributes
-
complete
, of type boolean , readonly -
Indicates whether
[[encoded data]]
is completely buffered.The
complete
getter steps are to return[[complete]]
. -
completed
, of type Promise< undefined >, readonly -
The promise used to signal when
complete
becomestrue
.The
completed
getter steps are to return[[completed promise]]
. -
tracks
, of type ImageTrackList , readonly -
Returns a live
ImageTrackList
, which provides metadata for the available tracks and a mechanism for selecting a track to decode.The
tracks
getter steps are to return[[ImageTrackList]]
.
10.2.4. Methods
-
decode(options)
-
Enqueues a control message to decode the frame according to options .
When invoked, run these steps:
-
If
[[closed]]
istrue
, return aPromise
rejected with anInvalidStateError
DOMException
. -
If
[[ImageTrackList]]
's[[selected index]]
is '-1', return aPromise
rejected with anInvalidStateError
DOMException
. -
If options is
undefined
, assign a newImageDecodeOptions
to options . -
Let promise be a new
Promise
. -
Queue a control message to decode the the image with options , and promise .
-
Append promise to
[[pending decode promises]]
. -
Return promise .
Running a control message to decode the image means running these steps:
-
Wait for
[[tracks established]]
to becometrue
. -
If options .
completeFramesOnly
isfalse
and the image is a Progressive Image for which the user agent supports progressive decoding, run the Decode Progressive Frame algorithm with options .frameIndex
and promise . -
Otherwise, run the Decode Complete Frame algorithm with options .
frameIndex
and promise .
-
-
reset()
-
Immediately aborts all pending work.
When invoked, run the Reset ImageDecoder algorithm with and
AbortError
DOMException
. -
close()
-
Immediately aborts all pending work and releases system resources. Close is final.
When invoked, run the Close ImageDecoder algorithm with and
AbortError
DOMException
. -
isTypeSupported(type)
-
Returns a promise indicating whether the provided config is supported by the user agent.
When invoked, run these steps:
-
If type is not a valid image MIME type , return a
Promise
rejected withTypeError
. -
Let p be a new
Promise
. -
In parallel, resolve p with the result of running the Check Type Support algorithm with type .
-
Return p .
-
10.2.5. Algorithms
- Fetch Stream Data Loop (with reader )
-
Run these steps:
-
Let readRequest be the following read request .
- chunk steps , given chunk
-
-
If
[[closed]]
istrue
, abort these steps. -
If chunk is not a Uint8Array object, queue a task on the control thread event loop to run the Close ImageDecoder algorithm with a
DataError
DOMException
and abort these steps. -
Let bytes be the byte sequence represented by the Uint8Array object.
-
Append bytes to the
[[encoded data]]
internal slot. -
If
[[tracks established]]
isfalse
, run the Establish Tracks algorithm. -
Otherwise, run the Update Tracks algorithm.
-
Run the Fetch Stream Data Loop algorithm with reader .
-
- close steps
-
-
Assign
true
tocomplete
-
Resolve
[[completed promise]]
.
-
- error steps
-
-
Queue a task on the control thread event loop to run the Close ImageDecoder algorithm with a
NotReadableError
DOMException
-
-
Read a chunk from reader given readRequest .
-
- Establish Tracks
-
Run these steps:
-
Assert
[[tracks established]]
isfalse
. -
If
[[encoded data]]
does not contain enough data to determine the number of tracks:-
If
complete
istrue
, queue a task on the control thread event loop to run the Close ImageDecoder algorithm. -
Abort these steps.
-
-
If the number of tracks is found to be
0
, queue a task on the control thread event loop to run the Close ImageDecoder algorithm and abort these steps. -
Let newTrackList be a new list .
-
For each image track found in
[[encoded data]]
:-
Let newTrack be a new
ImageTrack
, initialized as follows:-
Assign this to
[[ImageDecoder]]
. -
Assign
tracks
to[[ImageTrackList]]
. -
If image track is found to be animated, assign
true
to newTrack ’s[[animated]]
internal slot. Otherwise, assignfalse
. -
If image track is found to describe a frame count, assign that count to newTrack ’s
[[frame count]]
internal slot. Otherwise, assign0
.NOTE: If this was constructed with
data
as aReadableStream
, theframeCount
may change as additional bytes are appended to[[encoded data]]
. See the Update Tracks algorithm. -
If image track is found to describe a repetition count, assign that count to
[[repetition count]]
internal slot. Otherwise, assign0
.NOTE: A value of
Infinity
indicates infinite repetitions. -
Assign
false
to newTrack ’s[[selected]]
internal slot.
-
-
Append newTrack to newTrackList .
-
-
Let selectedTrackIndex be the result of running the Get Default Selected Track Index algorithm with newTrackList .
-
Let selectedTrack be the track at position selectedTrackIndex within newTrackList .
-
Assign
true
to selectedTrack ’s[[selected]]
internal slot. -
Assign selectedTrackIndex to
[[internal selected track index]]
. -
Assign
true
to[[tracks established]]
. -
Queue a task on the control thread event loop to perform the following steps:
-
Assign newTrackList to the
tracks
[[track list]]
internal slot. -
Assign selectedTrackIndex to
tracks
[[selected index]]
. -
Resolve
[[ready promise]]
.
-
-
- Get Default Selected Track Index (with trackList )
-
Run these steps:
-
If
[[encoded data]]
identifies a Primary Image Track :-
Let primaryTrack be the
ImageTrack
from trackList that describes the Primary Image Track . -
Let primaryTrackIndex be position of primaryTrack within trackList .
-
If
[[prefer animation]]
isnull
, return primaryTrackIndex . -
If primaryTrack .
animated
equals[[prefer animation]]
, return primaryTrackIndex .
-
-
If any
ImageTrack
s in trackList haveanimated
equal to[[prefer animation]]
, return the position of the earliest such track in trackList . -
Return
0
.
-
- Update Tracks
-
A track update struct is a struct that consists of a track index (
unsigned long
) and a frame count (unsigned long
).Run these steps:
-
Assert
[[tracks established]]
istrue
. -
Let trackChanges be a new list .
-
Let trackList be a copy of
tracks
'[[track list]]
. -
For each track in trackList :
-
Let trackIndex be the position of track in trackList .
-
Let latestFrameCount be the frame count as indicated by
[[encoded data]]
for the track corresponding to track . -
Assert that latestFrameCount is greater than or equal to
track.frameCount
. -
If latestFrameCount is greater than
track.frameCount
:-
Let change be a track update struct whose track index is trackIndex and frame count is latestFrameCount .
-
Append change to tracksChanges .
-
-
-
If tracksChanges is empty , abort these steps.
-
Queue a task on the control thread event loop to perform the following steps:
-
For each update in trackChanges :
-
Let updateTrack be the
ImageTrack
at positionupdate.trackIndex
withintracks
'[[track list]]
. -
Assign
update.frameCount
to updateTrack ’s[[frame count]]
.
-
-
-
- Decode Complete Frame (with frameIndex and promise )
-
-
Assert that
[[tracks established]]
istrue
. -
Assert that
[[internal selected track index]]
is not-1
. -
Let encodedFrame be the encoded frame identified by frameIndex and
[[internal selected track index]]
. -
Wait for any of the following conditions to be true (whichever happens first):
-
[[encoded data]]
contains enough bytes to completely decode encodedFrame . -
[[encoded data]]
is found to be malformed. -
complete
istrue
. -
[[closed]]
istrue
.
-
-
If
[[encoded data]]
is found to be malformed, run the Fatally Reject Bad Data algorithm and abort these steps. -
If
[[encoded data]]
does not contain enough bytes to completely decode encodedFrame , run the Reject Infeasible Decode algorithm with promise and abort these steps. -
Attempt to use
[[codec implementation]]
to decode encodedFrame . -
If decoding produces an error, run the Fatally Reject Bad Data algorithm and abort these steps.
-
If
[[progressive frame generations]]
contains an entry keyed by frameIndex , remove the entry from the map. -
Let output be the decoded image data emitted by
[[codec implementation]]
corresponding to encodedFrame . -
Let decodeResult be a new
ImageDecodeResult
initialized as follows:-
Assign 'true' to
complete
. -
Let timestamp and duration be the presentation timestamp and duration for output as described by encodedFrame . If encodedFrame does not describe a timestamp or duration, assign
null
to the corresponding variable. -
Assign
image
with the result of running the Create a VideoFrame algorithm with output , timestamp , and duration .
-
-
Run the Resolve Decode algorithm with promise and decodeResult .
-
- Decode Progressive Frame (with frameIndex and promise )
-
-
Assert that
[[tracks established]]
istrue
. -
Assert that
[[internal selected track index]]
is not-1
. -
Let encodedFrame be the encoded frame identified by frameIndex and
[[internal selected track index]]
. -
Let lastFrameGeneration be
null
. -
If
[[progressive frame generations]]
contains a map entry with the key frameIndex , assign the value of the map entry to lastFrameGeneration . -
Wait for any of the following conditions to be true (whichever happens first):
-
[[encoded data]]
contains enough bytes to decode encodedFrame to produce an output who’s Progressive Image Frame Generation exceeds lastFrameGeneration . -
[[encoded data]]
is found to be malformed. -
complete
istrue
. -
[[closed]]
istrue
.
-
-
If
[[encoded data]]
is found to be malformed, run the Fatally Reject Bad Data algorithm and abort these steps. -
Otherwise, if
[[encoded data]]
does not contain enough bytes to decode encodedFrame to produce an output who’s Progressive Image Frame Generation exceeds lastFrameGeneration , run the Reject Infeasible Decode algorithm with promise and abort these steps. -
Attempt to use
[[codec implementation]]
to decode encodedFrame . -
If decoding produces an error, run the Fatally Reject Bad Data algorithm and abort these steps.
-
Let output be the decoded image data emitted by
[[codec implementation]]
corresponding to encodedFrame . -
Let decodeResult be a new
ImageDecodeResult
. -
If output is the final full-detail progressive output corresponding to encodedFrame :
-
Assign
true
to decodeResult ’scomplete
. -
If
[[progressive frame generations]]
contains an entry keyed by frameIndex , remove the entry from the map.
-
-
Otherwise:
-
Assign
false
to decodeResult ’scomplete
. -
Let frameGeneration be the Progressive Image Frame Generation for output .
-
Add a new entry to
[[progressive frame generations]]
with key frameIndex and value frameGeneration .
-
-
Let timestamp and duration be the presentation timestamp and duration for output as described by encodedFrame . If encodedFrame does not describe a timestamp or duration, assign
null
to the corresponding variable. -
Assign
image
with the result of running the Create a VideoFrame algorithm with output , timestamp , and duration . -
Remove promise from
[[pending decode promises]]
. -
Resolve promise with decodeResult .
-
- Resolve Decode (with promise and result )
-
-
Queue a task on the control thread event loop to run these steps:
-
If
[[closed]]
, abort these steps. -
Assert that promise is an element of
[[pending decode promises]]
. -
Remove promise from
[[pending decode promises]]
. -
Resolve promise with result .
-
-
- Reject Infeasible Decode (with promise )
-
-
Assert that
complete
istrue
or[[closed]]
istrue
. -
If
complete
istrue
, let exception be aRangeError
. Otherwise, let exception be anInvalidStateError
DOMException
. -
Queue a task on the control thread event loop to run these steps:
-
If
[[closed]]
, abort these steps. -
Assert that promise is an element of
[[pending decode promises]]
. -
Remove promise from
[[pending decode promises]]
. -
Reject promise with exception .
-
-
- Fatally Reject Bad Data
-
-
Queue a task on the control thread event loop to run these steps:
-
If
[[closed]]
, abort these steps. -
Run the Close ImageDecoder algorithm with an
EncodingError
DOMException
.
-
-
- Check Type Support (with type )
-
-
If the user agent can provide a codec to support decoding type , return
true
. -
Otherwise, return
false
.
-
- Reset ImageDecoder (with exception )
-
-
Signal
[[codec implementation]]
to abort any active decoding operation. -
For each decodePromise in
[[pending decode promises]]
:-
Reject decodePromise with exception .
-
Remove decodePromise from
[[pending decode promises]]
.
-
-
- Close ImageDecoder (with exception )
-
-
Run the Reset ImageDecoder algorithm with exception .
-
Assign
true
to[[closed]]
. -
Clear
[[codec implementation]]
and release associated system resources . -
Remove all entries from
[[ImageTrackList]]
. -
Assign
-1
to[[ImageTrackList]]
's[[selected index]]
.
-
10.3. ImageDecoderInit Interface
typedef (BufferSource or ReadableStream );
ImageBufferSource dictionary {
ImageDecoderInit required DOMString type ;required ImageBufferSource data ;PremultiplyAlpha premultiplyAlpha = "default";ColorSpaceConversion colorSpaceConversion = "default";; ;unsigned long desiredWidth ;unsigned long desiredHeight ;boolean preferAnimation ; };
To
determine
if
an
ImageDecoderInit
is
a
valid
ImageDecoderInit
,
run
these
steps:
-
If type is not a valid image MIME type , return
false
. -
If data is of type
ReadableStream
and the ReadableStream is disturbed or locked , returnfalse
. -
If data is of type
BufferSource
:-
If the result of running IsDetachedBuffer (described in [ECMASCRIPT] ) on data is
false
, returnfalse
. -
If data is empty , return
false
.
-
-
If
desiredWidth
exists anddesiredHeight
does not exist, returnfalse
. -
If
desiredHeight
exists anddesiredWidth
does not exist, returnfalse
. -
Return
true
.
A
valid
image
MIME
type
is
a
string
that
is
a
valid
MIME
type
string
and
for
which
the
type
,
per
Section
3.1.1.1
of
[RFC7231]
,
is
image
.
-
type
, of type DOMString -
String containing the MIME type of the image file to be decoded.
-
data
, of type ImageBufferSource -
BufferSource
orReadableStream
of bytes representing an encoded image file as described bytype
. -
premultiplyAlpha
, of type PremultiplyAlpha , defaulting to"default"
-
Controls whether decoded outputs' color channels are to be premultiplied by their alpha channel, as defined by
premultiplyAlpha
inImageBitmapOptions
. -
colorSpaceConversion
, of type ColorSpaceConversion , defaulting to"default"
-
Controls whether decoded outputs' color space is converted or ignored, as defined by
colorSpaceConversion
inImageBitmapOptions
. -
desiredWidth
, of type unsigned long -
Indicates a desired width for decoded outputs. Implementation is best effort; decoding to a desired width may not be supported by all formats/ decoders.
-
desiredHeight
, of type unsigned long -
Indicates a desired height for decoded outputs. Implementation is best effort; decoding to a desired height may not be supported by all formats/decoders.
-
preferAnimation
, of type boolean -
For images with multiple tracks, this indicates whether the initial track selection should prefer an animated track.
NOTE: See the Get Default Selected Track Index algorithm.
10.4. ImageDecodeOptions Interface
dictionary {
ImageDecodeOptions = 0;unsigned long frameIndex = 0;boolean completeFramesOnly =true ; };
-
frameIndex
, of type unsigned long , defaulting to0
-
The index of the frame to decode.
-
completeFramesOnly
, of type boolean , defaulting totrue
-
For Progressive Images , a value of
false
indicates that the decoder may output animage
with reduced detail. Each subsequent call todecode()
for the sameframeIndex
will resolve to produce an image with a higher Progressive Image Frame Generation (more image detail) than the previous call, until finally the full-detail image is produced.If
completeFramesOnly
is assignedtrue
, or if the image is not a Progressive Image , or if the user agent does not support progressive decoding for the given image type, calls todecode()
will only resolve once the full detail image is decoded.NOTE: For Progressive Images , settingcompleteFramesOnly
tofalse
may be used to offer users a preview an image that is still being buffered from the network (via thedata
ReadableStream
).Upon decoding the full detail image, the
ImageDecodeResult
'scomplete
will be set to true.
10.5. ImageDecodeResult Interface
dictionary {
ImageDecodeResult ;required VideoFrame image ;required boolean complete ; };
-
image
, of type VideoFrame -
The decoded image.
-
complete
, of type boolean -
Indicates whether
image
contains the final full-detail output.NOTE:
complete
is alwaystrue
whendecode()
is invoked withcompleteFramesOnly
set totrue
.
10.6. ImageTrackList Interface
interface {
ImageTrackList );getter ImageTrack (unsigned long );
index ; ;readonly attribute Promise <undefined >ready ;readonly attribute unsigned long length ;readonly attribute long selectedIndex ;readonly attribute ImageTrack ?selectedTrack ; };
10.6.1. Internal Slots
-
[[ready promise]]
-
The promise used to signal when the
ImageTrackList
has been populated withImageTrack
s.NOTE:
ImageTrack
frameCount
may receive subsequent updates untilcomplete
istrue
. -
[[track list]]
-
The list of
ImageTrack
s describe by thisImageTrackList
. -
[[selected index]]
-
The index of the selected track in
[[track list]]
. A value of-1
indeicates that no track is selected.
10.6.2. Attributes
-
ready
, of type Promise< undefined >, readonly -
The
ready
getter steps are to return the[[ready promise]]
. -
length
, of type unsigned long , readonly -
The
length
getter steps are to return the length of[[track list]]
. -
selectedIndex
, of type long , readonly -
The
selectedIndex
getter steps are to return[[selected index]]
; -
selectedTrack
, of type ImageTrack , readonly, nullable -
The
selectedTrack
getter steps are:-
If
[[selected index]]
is-1
, returnnull
. -
Otherwise, return the ImageTrack from
[[track list]]
at the position indicated by[[selected index]]
.
-
10.7. ImageTrack Interface
interface :
ImageTrack EventTarget {readonly attribute boolean animated ;;readonly attribute unsigned long frameCount ;readonly attribute unrestricted float repetitionCount ;attribute EventHandler onchange ;attribute boolean selected ; };
10.7.1. Internal Slots
-
[[ImageDecoder]]
-
The
ImageDecoder
instance that constructed thisImageTrack
. -
[[ImageTrackList]]
-
The
ImageTrackList
instance that lists thisImageTrack
. -
[[animated]]
-
Indicates whether this track contains an animated image with multiple frames.
-
[[frame count]]
-
The number of frames in this track.
-
[[repetition count]]
-
The number of times the animation is intended to repeat.
-
[[selected]]
-
Indicates whether this track is selected for decoding.
10.7.2. Attributes
-
animated
, of type boolean , readonly -
The
animated
getter steps are to return the value of[[animated]]
.NOTE: This attribute provides an early indication that
frameCount
will ultimately exceed 0 for images where theframeCount
starts at0
and later increments as new chunks of theReadableStream
data
arrive. -
frameCount
, of type unsigned long , readonly -
The
frameCount
getter steps are to return the value of[[frame count]]
. -
repetitionCount
, of type unrestricted float , readonly -
The
repetitionCount
getter steps are to return the value of[[repetition count]]
. -
onchange
, of type EventHandler -
An event handler IDL attribute whose event handler event type is
change
. -
selected
, of type boolean -
The
selected
getter steps are to return the value of[[selected]]
.The
selected
setter steps are:-
If
[[ImageDecoder]]
's[[closed]]
slot istrue
, abort these steps. -
Let newValue be the given value .
-
If newValue equals
[[selected]]
, abort these steps. -
Assign newValue to
[[selected]]
. -
Let parentTrackList be
[[ImageTrackList]]
-
Let oldSelectedIndex be the value of parentTrackList
[[selected index]]
. -
If oldSelectedIndex is not
-1
:-
Let oldSelectedTrack be the
ImageTrack
in parentTrackList[[track list]]
at the position of oldSelectedIndex . -
Assign
false
to oldSelectedTrack[[selected]]
-
-
If newValue is
true
, let selectedIndex be the index of thisImageTrack
within parentTrackList ’s[[track list]]
. Otherwise, let selectedIndex be-1
. -
Assign selectedIndex to parentTrackList
[[selected index]]
. -
Run the Reset ImageDecoder algorithm on
[[ImageDecoder]]
. -
Queue a control message to
[[ImageDecoder]]
's control message queue to update the internal selected track index with selectedIndex .
Running a control message to update the internal selected track index means running these steps:
-
Assign selectedIndex to
[[internal selected track index]]
. -
Remove all entries from
[[progressive frame generations]]
.
-
10.7.3. Event Summary
-
change
-
Fired at the
ImageTrack
when theframeCount
is altered.
11. Security Considerations
The primary security impact is that features of this API make it easier for an attacker to exploit vulnerabilities in the underlying platform codecs. Additionally, new abilities to configure and control the codecs may allow for new exploits that rely on a specific configuration and/or sequence of control operations.
Platform
codecs
are
historically
an
internal
detail
of
APIs
like
HTMLMediaElement
,
[WebAudio]
[WEBAUDIO]
,
and
[WebRTC]
.
In
this
way,
it
has
always
been
possible
to
attack
the
underlying
codecs
by
using
malformed
media
files/streams
and
invoking
the
various
API
control
methods.
For
example,
you
can
send
any
stream
to
a
decoder
by
first
wrapping
that
stream
in
a
media
container
(e.g.
mp4)
and
setting
that
as
the
src
of
an
HTMLMediaElement
.
You
can
then
cause
the
underlying
video
decoder
to
be
reset()
by
setting
a
new
value
for
<video>.currentTime
.
WebCodecs makes such attacks easier by exposing low level control when inputs are provided and direct access to invoke the codec control methods. This also affords attackers the ability to invoke sequences of control methods that were not previously possible via the higher level APIs.
User agents should mitigate this risk by extensively fuzzing their implementation with random inputs and control method invocations. Additionally, user agents are encouraged to isolate their underlying codecs in processes with restricted privileges (sandbox) as a barrier against successful exploits being able to read user data.
An additional concern is exposing the underlying codecs to input mutation race conditions. Specifically, it should not be possible for a site to mutate a codec input or output while the underlying codec may still be operating on that data. This concern is mitigated by ensuring that input and output interfaces are immutable.
EncodedVideoChunk and EncodedAudioChunk currently expose a mutable data. See #80 .
12. Privacy Considerations
The primary privacy impact is an increased ability to fingerprint users by querying for different codec capabilities to establish a codec feature profile. Much of this profile is already exposed by existing APIs. Such profiles are very unlikely to be uniquely identifying, but may be used with other metrics to create a fingerprint.
An
attacker
may
accumulate
a
codec
feature
profile
by
calling
IsConfigSupported()
methods
with
a
number
of
different
configuration
dictionaries.
Similarly,
an
attacker
may
attempt
to
configure()
a
codec
with
different
configuration
dictionaries
and
observe
which
configurations
are
accepted.
Attackers
may
also
use
existing
APIs
to
establish
much
of
the
codec
feature
profile.
For
example,
the
[media-capabilities]
decodingInfo()
API
describes
what
types
of
decoders
are
supported
and
its
powerEfficient
attribute
may
signal
when
a
decoder
uses
hardware
acceleration.
Similarly,
the
[WebRTC]
getCapabilities()
API
may
be
used
to
determine
what
types
of
encoders
are
supported
and
the
getStats()
API
may
be
used
to
determine
when
an
encoder
uses
hardware
acceleration.
WebCodecs
will
expose
some
additional
information
in
the
form
of
low
level
codec
features.
A codec feature profile alone is unlikely to be uniquely identifying. Underlying codecs are often implemented entirely in software (be it part of the user agent binary or part of the operating system), such that all users who run that software will have a common set capabilities. Additionally, underlying codecs are often implemented with hardware acceleration, but such hardware is mass produced and devices of a particular class and manufacture date (e.g. flagship phones manufactured in 2020) will often have common capabilities. There will be outliers (some users may run outdated versions of software codecs or use a rare mix of custom assembled hardware), but most of the time a given codec feature profile is shared by a large group of users.
Segmenting groups of users by codec feature profile still amounts to a bit of entropy that can be combined with other metrics to uniquely identify a user. User agents may partially mitigate this by returning an error whenever a site attempts to exhaustively probe for codec capabilities. Additionally, user agents may implement a "privacy budget", which depletes as authors use WebCodecs and other identifying APIs. Upon exhaustion of the privacy budget, codec capabilities could be reduced to a common baseline or prompt for user approval.