Immersive Audio Model and Formats

1. Introduction

This specification defines an immersive audio model and formats (IAMF) to provide immersive audio experience to end-users.

• The term Immersive Audio (IA) means the combination of 3D audio signal s recreating a sound experience close to that of a natural environment.

• The term 3D audio signal means a representation of sound that incorporates additional information beyond traditional stereo or surround sound formats such as Ambisonics (Scene-based), Object-based audio and Channel-based audio (e.g. 3.1.2ch or 7.1.4ch).

This specification defines a model for representing Immersive Audio contents based on Audio Substream s contributing to Audio Element s meant to be rendered and mixed to form one or more presentations as depicted in the figure below.

The model comprises a number of coded Audio Substream s and the metadata that describes how to decode, render and mix the Audio Substream s for playback. The model itself is codec-agnostic; any supported audio codec may be used to code the Audio Substream s.

The model includes one or more Audio Element s, each of which consists of one or more Audio Substream s. The Audio Substream s, which composing of an Audio Element , are grouped into one or more ChannelGroup s. The model further includes Mix Presentation s and Parameter Substream s.

• The term Audio Substream means a sequence of audio samples, which may be encoded with any compatible audio codec.

• The term Audio Element means a 3D audio signal , and is constructed from one or more Audio Substream s and the metadata describing them. The Audio Substream s associated with one Audio Element use the same audio codec.

• The term ChannelGroup means a set of Audio Substream (s) which is(are) able to provide a spatial resolution of audio contents by itself or which is(are) able to provide an enhanced spatial resolution of audio contents by combining with the preceding ChannelGroup s.

• The term Parameter Substream means a sequence of parameter values that is associated with the algorithms used for decoding, reconstructing, rendering and mixing. It is applied to its associated Audio Element .

  • Parameter Substream may change their values over time and may further be animated; for example, any changes in values may be smoothed over some time duration. As such, they may be viewed as a 1D signal that have different metadata specified for different time durations.

• The term Mix Presentation means a series of processes to present Immersive Audio contents to end-users by using Audio Element (s). It contains metadata, that describe how the Audio Element s are rendered and mixed together for playback through physical loudspeakers or headsets, and loudness information.

IAMF can be used to provide Immersive Audio contents for presentation on a wide range of devices in both streaming and offline applications. These applications include internet audio streaming, multicasting/broadcasting services, file download, gaming, communication, virtual and augmented reality, and others. In these applications, audio may be played back on a wide range of devices, e.g. headsets, mobile phones, tablets, TVs, sound bars, home theater systems and big screens.

Here are some typical IAMF use cases and examples of how to instantiate the model for the use cases.

• UC1: One Audio Element (e.g. 3.1.2ch or First Order Ambisonics (FOA)) is delivered to a big-screen TV (in Home) or a Mobile device through unicast network. It is rendered to a loudspeaker layout (e.g. 3.1.2ch) or Binaural (or Stereo) with loudness normalization and is played on loudspeakers built in the big-screen TV or headset through the Mobile device, respectively.

• UC2: Two Audio Element s (e.g. 5.1.2ch and Stereo) are delivered to a big-screen TV through unicast network. Both are rendered to the same loudspeaker layout builted in the big-screen TV and are mixed. After loudness normalization appropriate to Home environment, the mixed Audio Element is playbacked on the loudspeakers.

• UC3: Two Audio Element s (e.g. FOA and Stereo) are delivered to a Mobile device through unicast network. Both are rendered to the same loudspeaker layout to be mixed. After mixing of them, it is rendered to Binaural (or Stereo) with loudness normalization and is playbacked on headset through the Mobile device.

Example 1: 3D audio signal = 3.1.2ch of UC1,

• Audio Substream: L and R channels are coded as one audio stream, Ltf and Rtf channels as one audio stream, Center as one audio stream and LFE as one audio stream.

• Audio Element (3.1.2ch): Consists of 4 Audio Substreams which are grouped into one ChannelGroup .

• Mix Presentation: Provide rendering algorithm of Audio Element to popular loudspeaker layouts and loudness information of the 3D audio signal .

Example 2: Two 3D audio signal s = 5.1.2ch and Stereo of UC2,

• Audio Substream: L and R channels are coded as one audio steam, Ls and Rs channels as one audio stream, Ltf and Rtf channels as one audio stream, Center as one audio stream and LFE as one audio stream.

• Audio Element 1 (5.1.2ch): Consists of 5 Audio Substreams which are grouped into one ChannelGroup .

• Audio Element 2 (Stereo): Consists of 1 Audio Substream which is grouped into one ChannelGroup .

• Parameter Substream 1-1: Mixing parameter values applied to Audio Element 1 by considering Home environment

• Parameter Substream 1-2: Mixing parameter values applied to Audio Element 2 by considering Home environment

• Mix Presentation: Provide rendering algorithm of Audio Element 1 & 2 to popular loudspeaker layouts, mixing based on Parameter Substream 1-1 & 1-2 and loudness information of the mixed Audio Element .

Example 3: Two 3D audio signal s = FOA and Stereo of UC3,

• Audio Substream: L and R channels are coded as one audio steam and each channel of FOA as one audio stream.

• Audio Element 1 (FOA) : Consists of 4 Audio Substreams which are grouped into one ChannelGroup .

• Audio Element 2 (Stereo) : Consists of 1 Audio Substream which is grouped into one ChannelGroup .

• Parameter Substream 1-1: Mixing parameter values applied to Audio Element 1 by considering Mobile environment

• Parameter Substream 1-2: Mixing parameter values applied to Audio Element 2 by considering Mobile environment

• Mix Presentation: Provide rendering algorithm of Audio Element 1 & 2 to popular loudspeaker layouts, mixing based on Parameter Substream 1-1 & 1-2 and loudness information of the mixed Audio Element .

NOTE: Big-screen TVs select Mix Presentation 1 and Mobile devices select Mix Presentation 2.

Based on the model, this specification defines a hypothetical immersive audio model and format ( IAMF ) architecture as depicted in the figure below.

For a given input 3D audio,

• A Pre-Processor generates ChannelGroup (s), Descriptors and Parameter Substream (s).

• A Codec Enc generates coded Audio Substream (s).

• An OBU Packetizer generates an IA Sequence from Audio Substream (s) and Descriptors and Parameter Substream (s).

• A File Packager (ISOBMFF Encapsulation) generates an IAMF File by encapsulating the IA Sequence into [ISOBMFF] track(s).

• A File Parser (ISOBMFF Parser) reconstructs the IA Sequence by decapsulating the IAMF File.

• An OBU Parser outputs the Audio Substream (s) and the Parameter Substream (s).

• A Codec Dec outputs decoded ChannelGroup (s) after decoding of the Audio Substream (s).

• A Post-Processor outputs an Immersive Audio by using the ChannelGroup (s), the Descriptors and the Parameter Substream (s).

• Pre-Processor, ChannelGroup (s), Codec Enc and OBU Packetizer are defined in § 7 IAMF Generation Process (Informative) .

• IA Sequence is defined in § 1.1 IA Sequence

• ISOBMFF Encapsulation, IAMF file (ISOBMFF file) and ISOBMFF Parser are deifned in § 5 ISOBMFF IAMF Encapsulation .

• OBU Parser, Codec Dec and Post-Processor are defined in § 6 IAMF processing .

1.1. IA Sequence

An IA Sequence is a bitstream to represent Immersive Audio contents and consists of Descriptors and IA Data .

Each of Descriptors and each of IA Data are packetized by Open Bitstream Unit (OBU), respectively. The term Open Bitstream Unit (OBU) is the concrete, physical unit used to represent the components in the model.

1.2. Use of OBU Syntax

1.2.1. Descriptors

Descriptors contain all the information that is required to setup and configure the decoders, reconstruction algorithm, renderers and mixers. Descriptors do not contain audio signal.

• IA Sequence Header OBU indicates the start of a full IA Sequence description and contains information related to profiles.

• Codec Config OBU describes information to set up a decoder for an coded Audio Substream .

• Audio Element OBU describes information to combine one or more Audio Substream s to reconstruct an Audio Element .

• Mix Presentation OBU describes information to render and mix one or more Audio Element s to generate the final 3D audio output.

  • Multiple Mix Presentation s can be defined as alternatives to each other within the same IA Sequence . Furthermore, the choice of which Mix Presentation to use at playback is left to the user. For example, multi-language support is implemented by defining different Mix Presentation s, where the first mix describes the use of the Audio Element with English dialogue, and the second mix describes the use of the Audio Element with French dialogue.

1.2.2. IA Data

IA Data contains the actual time-varying data that is required in the generation of the final 3D audio output.

• Audio Frame OBU provides the coded audio frame for an Audio Substream . It has the start timestamp and duration. So, a coded Audio Substream is represented as a sequence of Audio Frame OBU s with the same identifier, in time order.

• Parameter Block OBU provides the parameter values in a block for an time-varying Parameter Substream . It has the start timestampe and duration. So, a time-varying Parameter Substream is represented as a sequence of parameter values in Parameter Block OBU s with the same identifier, in time order.

• Temporal Delimiter OBU identifies the Temporal Unit s.

1.3. Timing Model

An Audio Substream is made of consecutive Audio Frame OBU s. Each Audio Frame OBU is made of audio samples at a given sample rate. The decode duration of an Audio Frame OBU is the number of audio samples divided by the sample rate. The presentation duration of an Audio Frame OBU is the number of audio samples remaining after trimming divided by the sample rate. The decode start time (respectively presentation start time) of an Audio Frame OBU is the sum of the decode durations (respectively presentation durations) of previous Audio Frame OBU s in the IA Sequence, or 0 otherwise. The decode duration (respectively presentation duration) of an Audio Substream is the sum of the decode durations (respectively presentation durations) all its Audio Frame OBU s. The start time of an Audio Substream is the presentation start time of its first Audio Frame OBU .

A Parameter Substream is made of consective Parameter Block OBU s. Each Parameter Block OBU is made of parameter values at a given sample rate. The decode duration of a Parameter Block OBU is the number of parameter values divided by the sample rate. The decode start time of a Parameter Block OBU s is the sum of the decode duration of previous Parameter Block OBU s, if any, 0 otherwise. The decode duration of a Parameter Substream is the sum of the decode duration of all its Parameter Block OBU s. The start time of an Parameter Substream is the decode start time of its first Audio Frame OBU . When all parameter values of Parameter Substream are constant, no Parameter Block OBU s may present in IA sequence.

Within an Audio Element , the presentation start times of all Audio Substream s coincide and is the presentation start time of the Audio Element . All Audio Substream s have the same presentation duration which is the presentation duration of the Audio Element .

• The decode start times of all Audio Substream s and all Parameter Substream s coincide and is the decode start time of the Audio Element .

• All Audio Substream s and all Parameter Substream s have the same decode duration which is the decode duration of the Audio Element .

Within an Mix Presentation , the presentation start time of all Audio Element s coincide and all Audio Element s have the same duration defining the duration of the Mix Presentation .

Within an IA Sequence , all Mix Presentation s have the same duration, defining the duration of the IA Sequence and have the same presentation start time defining the presentation start time of the IA Sequence .

The term Temporal Unit means a set of all Audio Frame OBU s with the same decode start time and the same duration from all Audio Substream s and all non-redundant Parameter Block OBU s with the decode start time within the duration.

The below figure shows an example of Timing Model in terms of the decode start times and durations of Audio Substream and Parameter Substream .

2. Open Bitstream Unit (OBU) Syntax and Semantics

The IA Sequence uses the OBU syntax.

This section specifies the OBU syntax elements and their semantics.

2.1. Immersive Audio OBU Syntax and Semantics

Syntax

class ia_open_bitstream_unit() {
  obu_header();
  if (obu_type == OBU_IA_Sequence_Header)
    ia_sequence_header_obu();
  else if (obu_type == OBU_IA_Codec_Config)
    codec_config_obu();
  else if (obu_type == OBU_IA_Audio_Element)
    audio_element_obu();
  else if (obu_type == OBU_IA_Mix_Presentation)
    mix_presentation_obu();
  else if (obu_type == OBU_IA_Parameter_Block)
    parameter_block_obu();
  else if (obu_type == OBU_IA_Temporal_Delimiter)
    temporal_delimiter_obu();
  else if (obu_type == OBU_IA_Audio_Frame)
    audio_frame_obu(true);
  else if (obu_type >= 9 and <= 30)
    audio_frame_obu(false);
  else if (obu_type == 5 or 6 or 7)
    reserved_obu();
  byte_alignment():
}

Semantics

If the syntax element obu_type is equal to OBU_IA_Sequence_Header, an ordered series of OBUs is presented to the decoding process as a string of bytes.

2.2. OBU Header Syntax and Semantics

Syntax

class obu_header() {
  unsigned int (5) obu_type;
  unsigned int (1) obu_redundant_copy;
  unsigned int (1) obu_trimming_status_flag;
  unsigned int (1) obu_extension_flag;
  leb128() obu_size;
  if (obu_trimming_status_flag) {
    leb128() num_samples_to_trim_at_end;
    leb128() num_samples_to_trim_at_start;
  }
  if (obu_extension_flag == 1)
    leb128() extension_header_size;
}

Semantics

OBUs are structured with a header and a payload.

obu_type specifies the type of data structure contained in the OBU payload.

obu_type: Name of obu_type
   0    : OBU_IA_Codec_Config
   1    : OBU_IA_Audio_Element
   2    : OBU_IA_Mix_Presentation
   3    : OBU_IA_Parameter_Block
   4    : OBU_IA_Temporal_Delimiter
  5~7   : Reserved
   8    : OBU_IA_Audio_Frame
  9~30  : OBU_IA_Audio_Frame_ID0 to OBU_IA_Audio_Frame_ID21
   31   : OBU_IA_Sequence_Header

obu_redundant_copy indicates whether this OBU is a redundant copy of the previous OBU in the IA sequence with the same obu_type. A value of 1 indicates that it is a redundant copy, while a value of 0 indicates that it is not.

It shall always be set to 0 for the following obu_type values:

• OBU_IA_Temporal_Delimiter

• OBU_IA_Audio_Frame

• OBU_IA_Audio_Frame_ID0 to OBU_IA_Audio_Frame_ID21

obu_trimming_status_flag indicates whether this OBU has audio samples to be trimmed or not. It SHALL be set only when obu_type is set to OBU_IA_Audio_Frame or OBU_IA_Audio_Frame_ID0 to OBU_IA_Audio_Frame_ID21.

For a given substream,

• If an Audio Frame OBU has its num_samples_to_trim_at_start field set to a non-zero value N, the decoder SHALL discard the first N audio samples.

• If an Audio Frame OBU has its num_samples_to_trim_at_end field set to a non-zero value N, the decoder SHALL discard the last N audio samples.

NOTE: Because of coding dependency, discarding a sample can sometimes mean decoding the entire audio frame.

• For a given Audio Frame OBU, the sum of num_samples_to_trim_at_start and num_samples_to_trim_at_end shall be less or equal to the number of samples (i.e. num_samples_per_frame ) in the Audio Frame OBU.

NOTE: This means that if one of the value is set to the the number of samples (i.e. num_samples_per_frame ) in the Audio Frame OBU, the other value is set to 0.

• When num_samples_to_trim_at_start is non-zero, all Audio Frame OBUs with the same substream id, and preceding this OBU back until the Codec OBU defining this substream, SHALL have their num_samples_to_trim_at_start field equal to the number of samples (i.e. num_samples_per_frame ) in the corresponding Audio Frame OBU.

• When num_samples_to_trim_at_end is non-zero in an Audio Frame OBU, there shall be no subsequent Audio Frame OBU with the same substream id until a non-redundant Codec OBU defining a substream with the same substream id.

obu_extension_flag indicates whether the extension_header_size field presents or not. If it set to 0, the extension_header_size field shall not be present. Otherwise, the extension_header_size field shall be present.

This flag shall be set to 0 for this version of the specification (i.e. version = 0). An OBU parser which is conformant with the current version of the specification shall be able to parse this flag and extension_header_size .

NOTE: A future version of specification may use this flag to specify an extension header field by setting obu_extension_flag = 1 and setting the size of extended header to extension_header_size .

obu_size indicates the size in bytes of the OBU immediately following the obu_size field of the OBU.

num_samples_to_trim_at_start indicates the number of samples that needs to be trimmed from the start of the samples in this Audio Frame OBU.

num_samples_to_trim_at_end indicates the number of samples that needs to be trimmed from the end of the samples in this Audio Frame OBU.

extension_header_size indicates the size in bytes of the extension header including this field.

2.3. Byte Alignment Syntax and Semantics

Syntax

class byte_alignment() {
  while (get_position() & 7)
    unsigned int (1) zero_bit;
}

Semantics

zero_bit shall be equal to 0 and is inserted into the bitstream to align the bit position to a multiple of 8 bits.

2.4. Reserved OBU Syntax and Semantics

Reserved OBUs shall be ignored by parsers compliant to this version of the specification. Future versions of the specification may define semantics for these reserved OBUs that would only be supported by parsers compliant to these future versions.

2.5. IA Sequence Header OBU Syntax and Semantics

This section specifies obu payload of OBU_IA_Sequence_Header.

This OBU is used to indicate the start of IA Sequence. So, the first OBU of IA Sequence shall be OBU_IA_Sequence_Header.

NOTE: When an IA Sequence is stored in a file, the IA Sequence Header OBU can be used to detect that the file contains an IA Sequence.

This OBU may be placed frequently within one single IA Sequence for an application such as broadcasting or multicasting scenario. In that case, the other IA Sequence Header OBUs except the first one shall be marked as redundant (i.e. obu_redundant_copy = 1).

Syntax

class ia_sequence_header_obu() {
  unsigned int (32) ia_code;
  unsigned int (8) profile_name;
  unsigned int (8) profile_compatible;
}

Semantics

ia_code indicates a ‘four-character code’ (4CC), ‘iamf’.

NOTE: When IA OBUs are delivered over a protocol that does not provide explicit IA Sequence boundaries, a parser may locate the IA Sequence start by searching for the code iamf preceded by specific OBU header values, e.g. assuming obu_extension_flag is set to 0 and because obu_trimming_status_flag is set to 0 for a IA Sequence Header OBU, the OBU header can be 0xF806 or 0xFC06

profile_name indicates that this IA sequence is able to be processed by IA decoders that support the profile indicated in this field. The IA decoders shall be able to parse all OBUs explicitly listed in that profile and can still encounter reserved OBUs that it should skip and it is acceptable to do so. This allows future versions of the specification to define new profiles that can be backwards compatible to old profiles.

• 0: Simple Profile

• 1: Base Profile

• 2~255: Reserved

profile_compatible indicates an additional profile of this specification to which this IA sequence complies. If a sequence only complies with the profile_name profile, this field shall be set to same profile_name value.

• 0: Simple Profile

• 1: Base Profile

• 2~255: Reserved

NOTE: For example, an IA sequence with profile_name = 0 and profile_compatible = 1 means that the IA sequence complies to Base profile of specification and IA decoder to support Simple profile can playback the IA sequence. When a future profile is defined with profile_compatible = 2 as called Enhanced profile, an IA sequence with profile_name = 1 and profile_compatible = 2 will mean that the IA sequence will comply with Enhanced profile and IA decoder to support Base profile of specification will be able to playback it by ignoring unknown OBUs and/or unknown syntaxes if present.

2.6. Codec Config OBU Syntax and Semantics

This section specifies the OBU payload of OBU_IA_Codec_Config.

Syntax

class codec_config_obu() {
  leb128() codec_config_id;  
  codec_config();
}
class codec_config() {
  unsigned int (32) codec_id;
  leb128() num_samples_per_frame;
  signed int (16) audio_roll_distance;
  decoder_config(codec_id);
}

Semantics

codec_config_id defines an identifier for a codec configuration. Within an IA Sequence, there shall be exactly one non-redundant Codec Config OBU with a given identifier. Audio Elements that need a decoder configuration based on this codec configuration refer to this identifier.

codec_id indicates a ‘four-character code’ (4CC) to identify the codec used to generate the audio substreams. For version 0 of this specification, it shall be set to one of four codec_id values defined below:

• Opus : The payload of all substreams of all Audio Elements referring to this codec configuration shall comply to the specification [RFC6716] and the decoder_config() structure shall comply with the constraints given in § 2.12.1 OPUS Specific .

• mp4a : The payload of all substreams of all Audio Elements referring to this codec configuration shall comply to the specification [AAC] and the decoder_config() structure shall comply with the constraints given in § 2.12.2 AAC-LC Specific .

• fLaC : The payload of all substreams of all Audio Elements referring to this codec configuration shall comply to the specification [FLAC] and the decoder_config() structure shall comply with the constraints given in § 2.12.3 FLAC Specific .

• ipcm : The payload of all substreams of all Audio Elements referring to this codec configuration shall be audio samples for linear PCM (LPCM) and the decoder_config() structure shall comply with the constraints given in § 2.12.4 LPCM Specific .

NOTE: ipcm should not be confused with lpcm which is another 4CC to identify codecs in other container formats (e.g. QuickTime).

num_samples_per_frame indicates the frame length, in samples, of the audio_frame() provided in by audio_frame_obu(). It shall not be set to zero. If the decoder_config() structure for a given codec specifies a value for the frame length, the two values shall be equal.

audio_roll_distance is a signed integer that gives the number of frames that need to be decoded in order for a frame to be decoded correctly. A negative value indicates the number of frames before the frame to be decoded correctly.

• It shall be set to -1 for IAMF-AAC-LC and -R (R = 4 when the frame size = 960) for IAMF-OPUS. IAMF-FLAC may ignore this field. Where, R is the smallest integer greater than or equal to 3840 divided by the frame size.

decoder_config() specifies the set of codec parameters required to decode the payload an substream for the given codec_id. It is byte aligned.

2.7. Audio Element OBU Syntax and Semantics

This section specifies the OBU payload of OBU_IA_Audio_Element.

Syntax

class audio_element_obu() {
  leb128() audio_element_id;
  unsigned int (3) audio_element_type;
  unsigned int (5) reserved;
  
  leb128() codec_config_id;  
  leb128() num_substreams;
  for (i = 0; i < num_substreams; i++) {
    leb128() audio_substream_id;
  }
  
  leb128() num_parameters;
  for (i = 0; i < num_parameters; i++) {
    leb128() param_definition_type;
    if (param_definition_type == PARAMETER_DEFINITION_DEMIXING) {
        DemixingParamDefinition demixing_info;
    }
    if (param_definition_type == PARAMETER_DEFINITION_RECON_GAIN) {
        ReconGainParamDefinition recon_gain_info;
    }
  }
  if (audio_element_type == CHANNEL_BASED) {
    scalable_channel_layout_config();
  } else if (audio_element_type == SCENE_BASED) {
    ambisonics_config();
  }  
}

class DemixingParamDefinition() extends ParamDefinition() {
  default_demixing_info_parameter_data();
  unsigned int (4) default_w;
  unsigned int (4) reserved;
}

class ReconGainParamDefinition() extends ParamDefinition() {
}

Semantics

audio_element_id defines an identifier for an Audio Element. Within an IA Sequence, there shall be exactly one non-redundant Audio Element OBU with a given identifier. Mix Presentations that use an Audio Element refer to this identifier.

audio_element_type specifies the audio representation of this audio element which is constructed from one or more audio substreams.

audio_element_type: The type of audio representation.
   0    : CHANNEL_BASED
   1    : SCENE_BASED
  2~7   : Reserved

codec_config_id indicates the identifier for the codec configuration which this Audio Element refers to.

num_substreams specifies the number of audio substreams that are used to reconstruct this audio element.

audio_substream_id indicates the identifier for a Substream which this Audio Element refers to.

Let a particular ChannelGroup’s substream be indexed as [ c , n_c ], where

• c = [1, ..., C] is the ChannelGroup index and C is the number of ChannelGroups.

• n_c = [1, ..., N_c] is the substream index in the c-th ChannelGroup and N_c is the number of substreams in the c-th ChannelGroup.

• The i-th audio_substream_id maps to a ChannelGroup’s substream as follows, where i is the index of the array:

[[1, 1], [1, 2], ..., [1, N_1], [2, 1], [2, 2], ..., [2, N_2], ..., [C, 1], [C, 2], ..., [C, N_c]]

A ChannelGroup is defined in § 7 IAMF Generation Process (Informative) . The order of the substreams in each ChannelGroup., i.e. the semantics of n_c, is specified in § 2.7.2 Scalable Channel Layout Config Syntax and Semantics .

num_parameters specifies the number of parameters that are used by the algorithms specified in this audio element.

• When audio_element_type = 0, this field shall be set to 0, 1 or 2.

• When audio_element_type = 1, this field shall be set to 0.

param_definition_type specifies the type of the parameter definition. All parameter definition types described in this version of the specification are listed in the table below, along with their associated parameter definitions.

• The type PARAMETER_DEFINITION_MIX_GAIN shall not be present in Audio Element OBU.

• The type shall not be duplicated in one Audio Element OBU.

• When codec_id = fLaC or ipcm , the type PARAMETER_DEFINITION_RECON_GAIN shall not be present.

• When num_layers > 1, the type PARAMETER_DEFINITION_RECON_GAIN shall be present.

• When the highest loudspeaker_layout of (non-)scalable channel audio is less than or equal to 3.1.2ch, the type PARAMETER_DEFINITION_DEMIXING shall not be present.

• When the highest loudspeaker_layout of scalable channel audio ( num_layers > 1) is greater than 3.1.2ch, the types of both PARAMETER_DEFINITION_DEMIXING and both PARAMETER_DEFINITION_RECON_GAIN shall be present.

• When num_layers = 1 and loudspeaker_layout is greater than 3.1.2ch, the type PARAMETER_DEFINITION_DEMIXING may be present.

demixing_info provides the parameter definition for the demixing information to reconstruct channel audios according to loudspeaker_layout from scalable channel audio. The parameter definition is provided by DemixingParamDefinition() and the corresponding parameter data to be provided in parameter blocks is specified in demixing_info_parameter_data().

• In this parameter definition, parameter_rate shall be set to the sample rate of this Audio Element and param_definition_mode shall be set to 0.

  • duration shall be same as num_samples_per_frame of this Audio Element.

  • num_subblocks shall be set to 1.

  • constant_subblock_duration shall be same as duration

recon_gain_info provides the parameter definition for the gain value to reconstruct channel audios according to loudspeaker_layout from scalable channel audio. The parameter definition is provided by ReconGainParamDefinition() and the corresponding parameter data to be provided in parameter blocks is specified in recon_gain_info_parameter_data().

• In this parameter definition, parameter_rate shall be set to the sample rate of this Audio Element and param_definition_mode shall be set to 0.

  • duration shall be same as num_samples_per_frame of this Audio Element.

  • num_subblocks shall be set to 1.

  • constant_subblock_duration shall be same as duration

scalable_channel_layout_config() is a class that provides the metadata required for combining the substreams identified here in order to reconstruct a scalable channel layout.

ambisonics_config() is a class that provides the metadata required for combining the substreams identified here in order to reconstruct an Ambisonics layout.

default_demixing_info_parameter_data() and default_w specify the default parameter data for demixing to apply to all audio samples when there are no Parameter Block OBUs (with parameter_id defined in this DemixingParamDefinition()) provided.

• default_demixing_info_parameter_data() conforms to demixing_info_parameter_data() except that w_idx_offset shall be ignored.

• Instead of that, default_w directly indicates the weight value w(k) for TF2toT2 de-mixer specified in § 6.2.2 De-mixer .

Mapping of default_w to w(k) should be as follows:

default_w :   w(k)
   0      :    0
   1      :  0.0179
   2      :  0.0391
   3      :  0.0658
   4      :  0.1038
   5      :  0.25
   6      :  0.3962
   7      :  0.4342
   8      :  0.4609
   9      :  0.4821
   10     :  0.5
   11     :  reserved

A default recon gain value of 0db is implied when there are no Parameter Block OBUs (with parameter_id defined in this ReconGainParamDefinition()) provided.

2.7.1. Parameter Definition Syntax and Semantics

Parameter definition classes inherits from the abstract ParamDefinition() class.

For a given parameter, its timeline is fully aligned with that of the Audio Element which the given parameter is applied to. Where, the timeline of the Audio Element is on post coding domain (i.e. before trimming data). So, when we assume the same sample rate between the given Parameter and the Audio Element In other words, the start timestamp and the duration of the given Parameter are same as those of the Audio Elememnt, respectively.

Syntax

abstract class ParamDefinition() {
  leb128() parameter_id;
  leb128() parameter_rate;
  unsigned int (1) param_definition_mode;
  unsigned int (7) reserved;
  if (param_definition_mode == 0) {
    leb128() duration;
    leb128() num_subblocks;
    leb128() constant_subblock_duration;
    if (constant_subblock_duration == 0) {
      for (i=0; i< num_subblocks; i++) {
        leb128() subblock_duration;
      }
    }
    
  }
}

Semantics

parameter_id indicates the identifier for the Parameter which this parameter definition refers to.

parameter_rate specifies the rate used by this parameter, expressed as ticks per second. Time-related fields associated with this parameter, such as durations, shall be expressed in the number of ticks.

• The rate shall be such a value that (the rate x num_samples_per_frame ) / (the sample rate of Audio Element) is a non-zero integer.

param_definition_mode indicates if this parameter definition specifies duration, num_subblocks, constant_subblock_duration and subblock_duration fields for the parameter blocks associated to the parameter_id .

• When this field is set to 0, all of duration, num_subblocks, constant_subblock_duration and subblock_duration fields shall be specified in this parameter definition mapped to the parameter_id . In that case, none of parameter blocks associated to this parameter definition shall specify duration, num_subblocks, constant_subblock_duration and subblock_duration fields.

• When this field is set to 1, none of duration, num_subblocks, constant_subblock_duration and subblock_duration fields shall be specificed in this parameter definition. In that case, each of parameter blocks associated to this parameter definition shall specify its own duration, num_subblocks, constant_subblock_duration and subblock_duration fields.

duration specifies the duration for which all of parameter blocks associated to this parameter definition are valid and applicable.

num_subblocks specifies the number of different sets of parameter values specified in all of parameter blocks associated to this parameter definition, where each set describes a different subblock of the timeline, contiguously.

constant_subblock_duration specifies the duration of each subblock, in the case where all subblocks except the last subblock have equal durations. If all subblocks except the last subblock do not have equal durations, the value of constant_subblock_duration shall be set to 0.

When it defines D = the value of duration , NS = the value of num_subblocks , CSI = the value of constant_subblock_duration and SI = the value of subblock_duration .

• When CSI != 0, NS x CSI shall be equal to or greater than D .

  • If NS x CSI > D , the actual duration of the last subblock shall be D - ( NS - 1) x CSI .

• When CSI = 0, the summation of all SI s in this parameter block shall be equal to D .

subblock_duration specifies the duration for the given subblock.

Each value of duration , constant_subblock_duration and subblock_duration shall be expressed as the number of ticks at the parameter_rate specified in the corresponding parameter definition.

2.7.2. Scalable Channel Layout Config Syntax and Semantics

scalable_channel_layout_config() contains information regarding the configuration of scalable channel audio.

Syntax

class scalable_channel_layout_config() {
  unsigned int (3) num_layers;
  unsigned int (5) reserved;
  for (i = 1; i <= num_layers; i++) {
    channel_audio_layer_config(i);
  }
}
class channel_audio_layer_config(i) {
  unsigned int (4) loudspeaker_layout(i);
  unsigned int (1) output_gain_is_present_flag(i);
  unsigned int (1) recon_gain_is_present_flag(i);
  unsigned int (2) reserved;
  unsigned int (8) substream_count(i);
  unsigned int (8) coupled_substream_count(i);
  if (output_gain_is_present_flag(i) == 1) {
    unsigned int (6) output_gain_flag(i);
    unsigned int (2) reserved;
    signed int (16) output_gain(i);
  }
}

When an audio element is composed of G(r) number of substreams, scalable channel audio for the audio element is layered into num_layers = r number of ChannelGroups.

• The order of ChannelGroups in each temporal unit shall be same as the order of channel_audio_layer_config()s in scalable_channel_layout_config().

• ChannelGroup #q consists of G(q)-G(q-1) number of substreams. Where, q = 1, 2, ..., r and G(0) = 0.

• IA frame is a set of audio_frame_obus with the same start timestamp of the single audio element for scalable channel audio. Each of them comes from each substream.

• Every IA frame shall have the same number of audio_frame_obus.

• When r > 1, parameter_block_obu() may present with IA frame.

The IA decoder shall select one of one or more channel audios provided by scalable channel audio. The IA decoder should select the appropriate channel audio according to the following rules, in order:

• The IA decoder should first attempt to select the channel audio whose loudspeaker layout matches the physical playback layout.

• If there is no match, the IA decoder should select the channel audio with the closest specified loudspeaker layout to the physical layout and then apply up or down-mixing appropriately, after decoding and reconstruction of the channel audio. § 9.2.2 Annex B-2: Down-mix Mechanism and § 6.6 Down-mix Matrix provide examples of dynamic and static down-mixing matrices for some common layouts that may be used.

Semantics

num_layers indicates the number of ChannelGroups for scalable channel audio. It shall not be set to zero and its maximum number shall be limited to 6.

• For Binaural, this field shall be set to 1.

channel_audio_layer_config() is a class that provides the information regarding the configuration of ChannelGroup for scalable channel audio. channel_audio_layer_config(i) provides information regarding the configuration of ChannelGroup #i.

loudspeaker_layout indicates the channel layout for the channels to be reconstructed from the precedent ChannelGroups and the current ChannelGroup among ChannelGroups for scalable channel audio.

In the current version of the specification, loudspeaker_layout indicates one of 10 channel layouts including Mono, Stereo, 5.1ch, 5.1.2ch, 5.1.4ch, 7.1ch, 7.1.2ch, 7.1.4ch, 3.1.2ch and Binaural. Where,

• Stereo is the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System A (0+2+0) of [ITU2051-3] .

• 5.1ch is the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System B (0+5+0) of [ITU2051-3] .

• 5.1.2ch is the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System C (2+5+0) of [ITU2051-3] .

• 5.1.4ch is the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System D (4+5+0) of [ITU2051-3] .

• 7.1ch is the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System I (0+7+0) of [ITU2051-3] .

• 7.1.2ch is the combination of the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System I (0+7+0) of [ITU2051-3] and the left and right top front pair of the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System J (4+7+0) of [ITU2051-3] .

• 7.1.4ch is the loudspeaker configuration as depicted in Loudspeaker configuration for Sound System J (4+7+0) of [ITU2051-3] .

• 3.1.2ch is the front subset (L/C/R/Ltf/Rtf/LFE) of 7.1.4ch .

Loudspeaker Layout (4 bits) :  Channel Layout  : Loudspeaker Location Ordering
             0000           :       Mono       : C
             0001           :      Stereo      : L/R
             0010           :      5.1ch       : L/C/R/Ls/Rs/LFE
             0011           :     5.1.2ch      : L/C/R/Ls/Rs/Ltf/Rtf/LFE
             0100           :     5.1.4ch      : L/C/R/Ls/Rs/Ltf/Rtf/Ltr/Rtr/LFE
             0101           :      7.1ch       : L/C/R/Lss/Rss/Lrs/Rrs/LFE
             0110           :     7.1.2ch      : L/C/R/Lss/Rss/Lrs/Rrs/Ltf/Rtf/LFE
             0111           :     7.1.4ch      : L/C/R/Lss/Rss/Lrs/Rrs/Ltf/Rtf/Ltb/Rtb/LFE
             1000           :     3.1.2ch      : L/C/R//Ltf/Rtf/LFE
             1001           :     Binaural     : L/R
            others          :     reserved     :

Where, C: Center, L: Left, R: Right, Ls: Left Surround, Lss: Left Side Surround, 
Rs: Right Surround, Rss: Right Side Surround, Lrs: Left Rear Surroud, Rrs: Right Rear Surround
Ltf: Left Top Front, Rtf: Right Top Front, Ltr: Left Top Rear, Rtr: Right Top Rear, 
Ltb: Left Top Back, Rtb: Right Top Back, LFE: Low-Frequency Effects

For a given input audio with audio_element_type = CHANNEL_BASED, if the input audio has height channels (e.g 7.1.4ch or 5.1.2ch), each of the list of channel layouts for scalable channel audio is recommended to have height channels (i.e it is recommended to be higher than or equal to 3.1.2ch).

• Examples for recommended list of channel layouts: 3.1.2ch/5.1.2ch, 3.1.2ch/5.1.2ch/7.1.4ch, 5.1.2ch/7.1.4ch etc..

• Examples for not-recommended list of channel layouts: 2ch/3.1.2ch/5.1.2ch, 2ch/3.1.2ch/5.1.2ch/7.1.4ch, 2ch/5.1.2ch/7.1.4ch, 2ch/7.1.4ch etc..

NOTE: Contents providers may be satisfied with the down-mixed audio having no height channels even though the down-mix mechanism, specified in § 9.2.2 Annex B-2: Down-mix Mechanism , drops height channels when it does down-mix from input channel audio with height channels to surround channels for example from 7.14ch to Mono, Stereo, 5.1ch or 7.1ch. In that case, encoder may generate a scalable audio with the down-mixed audio without having height channels from the input channel audio with height channels. In other words, this specification does not disallow for scalable audios to have a down-mixed audio without having height channels from input channel audio having height channels.

NOTE: The Ltr and Rtr of 5.1.4ch down-mixed from 7.1.4ch is within the range of Ltb and Rtb of 7.1.4ch.

output_gain_is_present_flag indicates if output_gain information fields for the ChannelGroup presents .

• 0: No output_gain information fields for the ChannelGroup shall be present.

• 1: output_gain information fields for the ChannelGroup present. In this case, output_gain_flags and output_gain fields shall be present.

recon_gain_is_present_flag indicates if recon_gain information fields for the ChannelGroup presents in recon_gain_info_parameter_data().

• 0: No recon_gain information fields for the ChannelGroup shall be present in recon_gain_info_parameter_data().

• 1: recon_gain information fields for the ChannelGroup present in recon_gain_info_parameter_data(). In this case, recon_gain_flags and recon_gain fields shall be present.

substream_count specifies the number of audio substreams. It must be the same as num_substreams in its corresponding audio_element().

coupled_substream_count specifies the number of referenced substreams that are coded as coupled stereo channels.

Mono or stereo coding shall be only allowed for the version of this specification.

• Each pair of coupled stereo channels in the same ChannelGroup shall be coded as stereo mode to generate one single substream and each of non-coupled channels in the same ChannelGroup shall be coded as mono mode to generate one single substream.

  • Coupled stereo channels : L/R, Ls/Rs, Lss/Rss, Lrs/Rrs, Ltf/Rtf, Ltb/Rtb

  • Non-coupled channels: C, LFE, L

The order of substreams in each ChannelGroup shall be as follows:

• Coupled Substreams comes first and followed by non-coupled Substreams.

• Coupled Substreams for surround channels comes first and followed by one(s) for top channels.

• Coupled Substreams for front channels comes first and followed by one(s) for side, rear and back channels.

• Coupled Substreams for side channels comes first and followed by one(s) for rear channels.

• Center channel comes first and followed by LFE and followed by the other one.

• Where, non-coupled substream is a coded substream from one of non-coupled channels.

output_gain_flags indicates the channels which output_gain is applied to. If a bit set to 1, output_gain shall be applied to the channel. Otherwise, output_gain shall not be applied to the channel.

Bit position : Channel Name
    b5(MSB)  : Left channel (L1, L2, L3)
      b4     : Right channel (R2, R3)
      b3     : Left Surround channel (Ls5)
      b2     : Right Surround channel (Rs5)
      b1     : Left Top Front channel (Ltf)
      b0     : Right Top Front channel (Rtf)

output_gain indicates the gain value to be applied to the mixed channels which are indicated by output_gain_flags. It is 20*log10 of the factor by which to scale the mixed channels. It is stored in a 16-bit, signed, two’s complement fixed-point value with 8 fractional bits (i.e. Q7.8 in [Q-Format] ). Where, each mixed channel is generated by downmixing two or more input channels.

2.7.3. Ambisonics Config Syntax and Semantics

ambisonics_config() contains information regarding the configuration of Ambisonics. In this specification, the [AmbiX] format is adopted, which uses Ambisonics Channel Number (ACN) channel ordering and normalizes the channels with Schmidt Semi-Normalization (SN3D).

Syntax

class ambisonics_config() {
  leb128() ambisonics_mode;
  if (ambisonics_mode == MONO) {
    ambisonics_mono_config();
  } else if (ambisonics_mode == PROJECTION) {
    ambisonics_projection_config();
  }
}
class ambisonics_mono_config() {
  unsigned int (8) output_channel_count (C);
  unsigned int (8) substream_count (N);
  unsigned int (8 * C) channel_mapping;
}
class ambisonics_projection_config() {
  unsigned int (8) output_channel_count (C);
  unsigned int (8) substream_count (N);
  unsigned int (8) coupled_substream_count (M);
  signed int (16 * (N + M) * C) demixing_matrix;
}

Semantics

ambisonics_mode specifies the method of coding Ambisonics.

ambisonics_mode: Method of coding Ambisonics.
   0    : MONO
   1    : PROJECTION

If ambisonics_mode is equal to MONO, this indicates that the Ambisonics channels are coded as individual mono substreams. For LPCM, ambisonics_mode shall be equal to MONO.

If ambisonics_mode is equal to PROJECTION, this indicates that the Ambisonics channels are first linearly projected onto another subspace before coding as a mix of coupled stereo and mono substreams.

output_channel_count complies with channel count in [RFC8486] with following restrictions:

• The allowed numbers of output_channel_count are (1+n)^2, where n = 0, 1, 2, ..., 14.

• In other words, the scene-based audio element shall not include non-diegetic channels.

substream_count specifies the number of audio substreams. It must be the same as num_substreams in its corresponding audio_element().

channel_mapping complies with the one for ChannelMappingFamily = 2 in [RFC8486] .

coupled_substream_count specifies the number of referenced substreams that are coded as coupled stereo channels, where M <= N.

demixing_matrix complies with the one for ChannelMappingFamily = 3 in [RFC8486] except the byte order of each of matrix coefficients is converted to big endian.

The order of substreams in ChannelGroup shall conform to [RFC8486] .

2.8. Mix Presentation OBU Syntax and Semantics

This section specifies the OBU payload of OBU_IA_Mix_Presentation.

The metadata in mix_presentation() specifies how to render, process and mix one or more audio elements, with details provided in § 6.3 Mix Presentation .

An IA sequence may have one or more mix presentations specified. The IA parser shall select the appropriate mix presentation to process according to the rules specified in § 6.3.1 Selecting a Mix Presentation .

A mix presentation may contain one or more sub-mixes. Common use-cases may specify only one sub-mix, which includes all rendered and processed audio elements used in the mix presentation. The use-case for specifying more than one sub-mix arises if an IA multiplexer is merging two or more IA sequences. In this case, it may choose to capture the loudness information from the original IA sequences in multiple sub-mixes, instead of recomputing the loudness information for the final mix.

Syntax

class mix_presentation_obu() {
  leb128() mix_presentation_id;
  leb128() count_label;
  for (i = 0; i < count_label; i++) {
    string language_label;
  }
  for (i = 0; i < count_label; i++) {
    mix_presentation_annotations();
  }
  leb128() num_sub_mixes;
  for (i = 0; i < num_sub_mixes; i++) {	  
    leb128() num_audio_elements;
    for (j = 0; j < num_audio_elements; j++) {
      leb128() audio_element_id;
      for (i = 0; i < count_label; i++) {
        mix_presentation_element_annotations();
      }
      rendering_config();
      element_mix_config();
    }
    output_mix_config();
    
    leb128() num_layouts;
    for (j = 0; j < num_layouts; j++) {
      layout loudness_layout;
      loudness_info loudness; 
    }
  }
}  

Semantics

mix_presentation_id defines an identifier for a Mix Presentation. Within an IA Sequence, there shall be exactly one non-redundant Mix Presentation OBU with a given identifier. This identifier may be used by the application to select which Mix Presentation(s) to offer.

count_label indicates the number of labels in different languages.

language_label specifies the language which both mix_presentation_friendly_label and audio_element_friendly_label are written in. It shall comform to [BCP47] . The same language shall not be duplicated in this loop.

• The ith label of both mix_presentation_annotation() and mix_presentation_element_annotation() shall be written in the language indicated by the ith language_label . Where, i = 0, 1, ..., count_label -1.

mix_presentation_annotations() is a class that provides informational metadata that an IA parser should refer to when selecting the mix presentation to use. The metadata may also be used by the playback system to display information to the user, but is not used in the rendering or mixing process to generate the final output audio signal.

num_sub_mixes specifies the number of sub-mixes.

num_audio_elements specifies the number of audio elements that are used in this mix presentation to generate the final output audio signal for playback.

audio_element_id indicates the identifier for an Audio Element which this Mix Presentation refers to.

mix_presentation_element_annotations() is a class that provides informational metadata that an IA parser should refer to when selecting the referenced audio element to use. The metadata may also be used by the playback system to display information to the user, but is not used in the rendering or mixing process to generate the final output audio signal.

rendering_config() is a class that provides the metadata required for rendering the referenced audio element.

element_mix_config() is a class that provides the metadata required for applying any processing to the referenced and rendered audio element before being summed with other processed audio elements.

output_mix_config() is a class that provides the metadata required for post-processing the mixed audio signal to generate the audio signal for playback.

num_layouts specifies the number of layouts for this sub-mix which the loudness informations were measured on.

loudness_layout identifies the layout that was used to measure the loudness information provided in this sub-mix.

loudness provides the loudness information which was measured on loudness_layout for the mixed audio element by this sub-mix.

The layout specified in loudness_layout should not be higher than the highest layout among layouts provided by the audio elements except zero-order Ambisonics or Mono. In other words, rendering from an audio element with the highest layout (except zero-order Ambisonics or Mono) to the loudness_layout should not require an upmix.

• loudness_layout for zero-order Ambisonics or Mono should not be higher than Stereo. Zero-order Ambisonics or Mono may be rendered to Stereo.

If one sub-mix of Mix Presentation OBU includes only one single scalable channel audio, then it complies with as follows:

• num_layouts shall be greater than or equal to num_layers specified in scalable_channel_layout_config() of Audio Element OBU for the audio_element_id except the following cases:

The highest loudness_layout specified in one sub-mix except zero-order Ambisonics or Mono is the layout which was used for authoring the sub-mix.

• The highest loudness_layout for zero-order Ambisonics or Mono is Stereo.

Each sub-mix shall include loudness_layout to identify Loudspeaker configuration for Sound System A (0+2+0) (i.e. Stereo). In other words, each sub-mix shall include loudness_info() for Stereo.

• If the mixed audio element by each sub-mix is Mono, then its loudness for loudness_layout = Stereo should be measured on the Stereo generated from the Mono by the equations, L = 0.707 x Mono and R = 0.707 x Mono.

2.8.1. Mix Presentation Annotations Syntax and Semantics

Syntax

class mix_presentation_annotations() {
  string mix_presentation_friendly_label;
}

Semantics

mix_presentation_friendly_label specifies a human-friendly label to describe this mix presentation.

2.8.2. Mix Presentation Element Annotations Syntax and Semantics

Syntax

class mix_presentation_element_annotations() {
  string audio_element_friendly_label;
}

Semantics

audio_element_friendly_label specifies a human-friendly label to describe the referenced audio element.

2.8.3. Rendering Config Syntax and Semantics

During playback, an audio element should be rendered using a pre-defined renderer according to § 6.3.2 Rendering an Audio Element . In this version of the specification, no additional metadata is required to configure the renderers, and as such, rendering_config() has an empty payload.

Syntax

class rendering_config() {
}

Semantics

2.8.4. Element Mix Config Syntax and Semantics

element_mix_config() provides a gain value to be applied to the rendered audio element signal.

Syntax

class element_mix_config() {
  MixGainParamDefinition mix_gain;
}

class MixGainParamDefinition() extends ParamDefinition() {
  signed int (16) default_mix_gain;
}

Semantics

mix_gain provides the parameter definition for the gain value that is applied to all channels of the rendered audio element signal. The parameter definition is provided by MixGainParamDefinition() and the corresponding parameter data to be provided in parameter blocks is specified in mix_gain_parameter_data().

default_mix_gain specifies the default mix gain value to apply when there are no mix gain parameter blocks provided. This value is expressed in dB and shall be applied to all channels in the rendered audio element. It is stored as a 16-bit, signed, two’s complement fixed-point value with 8 fractional bits (i.e. Q7.8 in [Q-Format] ).

2.8.5. Output Mix Config Syntax and Semantics

output_mix_config() provides a gain value to be applied to the mixed audio signal.

Syntax

class output_mix_config() {
  MixGainParamDefinition output_mix_gain;
}

Semantics

output_mix_gain provides the parameter definition for the gain value that is applied to all channels of the mixed audio signal. The parameter definition is provided by MixGainParamDefinition() and the corresponding parameter data to be provided in parameter blocks is specified in mix_gain_parameter_data().

2.8.6. Layout Syntax and Semantics

The layout class specifies either a binaural system or the list of physical loudspeaker positions according to [ITU2051-3] .

Syntax

class layout() {
  unsigned int (2) layout_type;
  
  if (layout_type == LOUDSPEAKERS_SP_LABEL) {
    unsigned int (6) num_loudspeakers;
    for (i = 0; i < num_loudspeakers; i++) {
      unsigned int (8) sp_label;
    }
  } 
  else if (layout_type == LOUDSPEAKERS_SS_CONVENTION) {
    unsigned int (4) sound_system;
    unsigned int (2) reserved;
  }
  else if (layout_type == BINAURAL or NOT_DEFINED) {
    unsigned int (6) reserved;
  }
}

Semantics

layout_type specifies the layout type.

layout_type : Layout type
     0      : NOT_DEFINED
     1      : LOUDSPEAKERS_SP_LABEL
     2      : LOUDSPEAKERS_SS_CONVENTION
     3      : BINAURAL

• A value of 0 indicates no specific layout.

• A value of 1 indicates that the layout is defined using the SP Label of [ITU2051-3] .

• A value of 2 indicates that the layout is defined using the sound system convention of [ITU2051-3] .

• A value of 3 indicates that the layout is binaural.

num_loudspeakers specifies the number of loudspeakers.

sp_label defines the SP Label as specified in [ITU2051-3] .

sound_system specifies the sound system A to J as specified in [ITU2051-3] , 7.1.2ch and 3.1.2ch of loudspeaker_layout as follows:

• 0: It indicates Loudspeaker configuration for Sound System A (0+2+0)

• 1: It indicates Loudspeaker configuration for Sound System B (0+5+0)

• 2: It indicates Loudspeaker configuration for Sound System C (2+5+0)

• 3: It indicates Loudspeaker configuration for Sound System D (4+5+0)

• 4: It indicates Loudspeaker configuration for Sound System E (4+5+1)

• 5: It indicates Loudspeaker configuration for Sound System F (3+7+0)

• 6: It indicates Loudspeaker configuration for Sound System G (4+9+0)

• 7: It indicates Loudspeaker configuration for Sound System H (9+10+3)

• 8: It indicates Loudspeaker configuration for Sound System I (0+7+0)

• 9: It indicates Loudspeaker configuration for Sound System J (4+7+0)

• 10: It indicates the same loudspeaker configuration as loudspeaker_layout = 0110 (i.e. 7.1.2ch)

• 11: It indicates the same loudspeaker configuration as loudspeaker_layout = 1000 (i.e. 3.1.2ch)

• 12: It indicates Mono

• 13 ~ 15: Reserved

2.8.7. Loudness Info Syntax and Semantics

loudness_info() provides loudness information for a given audio signal.

All signed values are stored as signed Q7.8 fixed-point values (in [Q-Format] ).

Syntax

class loudness_info() {
  unsigned int (8) info_type;
  signed int (16) integrated_loudness;
  signed int (16) digital_peak;
  if (info_type & 1) {
    signed int (16) true_peak;
  }
  if (info_type & 2) {
    unsigned int (8) num_anchored_loudness;
    for (i = 0; i < num_anchored_loudness; i++) {
      unsigned int (8) anchor_element;
      signed int (16) anchored_loudness;
    }
  }
}

Semantics

info_type is a bitmask that specifies the type of optional loudness information provided. The bits are set as follows, where the first bit is the LSB:

Bit : Type of information provided
 0  : True peak
 1  : Anchored Loudness (one or more)
2~7 : Reserved

integrated_loudness provides the program integrated loudness information, specified in LKFS as defined in [ITU1770-4] , and measured according to [ITU1770-4] .

digital_peak specifies the digital (sampled) peak value of the audio signal, specified in dBFS.

true_peak specifies the true peak of the audio signal, specified in dBFS and measured according to [ITU1770-4] .

anchor_element specifies the anchor element used in computation of the anchored_loudness which follows, as defined in [ISOIEC-23091-3-2018] , as follows:

  0   : Unknown
  1   : Dialogue
  2   : Album
3~255 : Reserved

There shall be no duplicate values of anchor_element within one loudness_info().

anchored_loudness specifies the loudness information according to the anchor element, specified in LKFS as defined in [ITU1770-4] .

NOTE: [ITU1770-4] adopts the convention of using the dBov unit for dBFS, where the RMS value of a full-scale square wave is 0 dBov. The same convention is adopted here.

2.9. Parameter Block OBU Syntax and Semantics

This section specifies the OBU payload of OBU_IA_Parameter_Block.

The metadata specified in this OBU defines the parameter values for an algorithm for an indicated duration, including any animation of the parameter values over this duration. The metadata are used in conjunction with a corresponding parameter definition and parameter data specification. The parameter definition is specified based on ParamDefinition() . The parameter data provides the values to apply in each parameter block. These are specified using the AnimatedParameterData() function template if parameter animation is supported.

Syntax

class parameter_block_obu() {
  leb128() parameter_id;
  
  (param_definition_type, param_definition_mode, duration, num_subblocks, constant_subblock_duration, subblock_duration) = get_param_definition(parameter_id);
  
  if (param_definition_mode) {
    leb128() duration;
    leb128() num_subblocks;
    leb128() constant_subblock_duration;
  }
  for (i = 0; i < num_subblocks; i++) {
    if (param_definition_mode) {
      if (constant_subblock_duration == 0) {
        leb128() subblock_duration;
      }
    }
    if (param_definition_type == PARAMETER_DEFINITION_MIX_GAIN) {
      mix_gain_parameter_data();
    }
    if (param_definition_type == PARAMETER_DEFINITION_DEMIXING) {
      demixing_info_parameter_data();
    }
    if (param_definition_type == PARAMETER_DEFINITION_RECON_GAIN) {
      recon_gain_info_parameter_data();
    }
  }
}

Semantics

parameter_id defines an identifier for a Parameter. Within an IA Sequence, there shall be exactly one non-redundant Audio Element OBU with this identifier. Parameter Block OBU refer to the Parameter through this identifier.

get_param_definition() is a run-time function to get the parameter definition type, the parameter definition mode, duration, num_subblocks, constant_subblock_duration and subblock_duration mapped to the parameter_id.

duration specifies the duration for which this parameter block is valid and applicable.

num_subblocks specifies the number of different sets of parameter values specified in this parameter block, where each set describes a different subblock of the timeline, contiguously.

constant_subblock_duration specifies the duration of each subblock, in the case where all subblocks except the last subblock have equal durations. If all subblocks except the last subblock do not have equal durations, the value of constant_subblock_duration shall be set to 0.

Audio Element OBU and/or Mix Presentation OBU is mapping a parameter_id to the parameter definition type. So, IA decoders can know the definition type mapped to the parameter_id.

subblock_duration specifies the duration for the given subblock.

Each value of duration, constant_subblock_duration and subblock_duration shall be expressed as the number of ticks at the parameter_rate specified in the corresponding parameter definition.

2.9.1. Mix Gain Parameter Data Syntax and Semantics

Syntax

class mix_gain_parameter_data() {
  leb128() animation_type;
  AnimatedParameterData<signed int (16)> param_data;
}

Semantics

animation_type specifies the type of animation applied to the parameter values.

param_data uses the AnimatedParameterData function template. Each of the values defined within this instance (start_point_value, end_point_value and control_point_value) is expressed in dB and shall be applied to all channels in the rendered audio element. They are stored as 16-bit, signed, two’s complement fixed-point values with 8 fractional bits (i.e. Q7.8 in [Q-Format] ).

animation_type : Animation Type
       0       : STEP
       1       : LINEAR
       2       : BEZIER

Classes that take animation_type as an input argument use the AnimatedParameterData() function template. The method of applying the animation is described in § 6.4 Animated Parameters .

template <class T>
class AnimatedParameterData(animation_type) {
  if (animation_type == STEP) {
    T start_point_value;
  }
  if (animation_type == LINEAR) {
    T start_point_value;
    T end_point_value;
  }
  if (animation_type == BEZIER) {
    T start_point_value;
    T end_point_value;
    T control_point_value;
    unsigned int (8) control_point_relative_time;
  }
}

start_point_value specifies the parameter value that is applied at the start of the subblock.

end_point_value specifies the parameter value that is applied at the end of the subblock.

control_point_value specifies the parameter value of the middle control point of a quadratic Bezier curve, i.e. its y-axis value.

control_point_relative_time specifies the time of the middle control point of a quadratic Bezier curve, i.e. its x-axis value. This value is expressed as a fraction of the parameter subblock duration with valid values in the range of 0 and 1, inclusively. A value equal to 0 or 1 indicates that this animation implements a linear Bezier curve, in which case control_point_value shall be ignored by the IA parser. It is stored as an 8-bit, unsigned, fixed-point value with 8 fractional bits (i.e. Q0.8 in [Q-Format] ).

2.9.2. Demixing Info Parameter Data Syntax and Semantics

demixing_info_parameter_data() specifies demixing parameter mode to be used to reconstruct output channel audio according to its loudspeaker_layout .

Syntax

class demixing_info_parameter_data() {
  unsigned int (3) dmixp_mode;
  unsigned int (5) reserved;
}

Semantics

dmixp_mode indicates a mode of pre-defined combinations of five demix parameters.

• 0: mode1, (alpha, beta, gamma, delta, w_idx_offset) = (1, 1, 0.707, 0.707, -1)

• 1: mode2, (alpha, beta, gamma, delta, w_idx_offset) = (0.707, 0.707, 0.707, 0.707, -1)

• 2: mode3, (alpha, beta, gamma, delta, w_idx_offset) = (1, 0.866, 0.866, 0.866, -1)

• 3: reserved

• 4: mode1, (alpha, beta, gamma, delta, w_idx_offset) = (1, 1, 0.707, 0.707, 1)

• 5: mode2, (alpha, beta, gamma, delta, w_idx_offset) = (0.707, 0.707, 0.707, 0.707, 1)

• 6: mode3, (alpha, beta, gamma, delta, w_idx_offset) = (1, 0.866, 0.866, 0.866, 1)

• 7: reserved

alpha and beta are gain values used for S7to5 down-mixer, gamma for T4to2 down-mixer, delta for S5to3 down-mixer and w_idx_offset is the offset to generate a gain value w used for T2toTF2 down-mixer.

2.9.3. Recon Gain Info Parameter Data Syntax and Semantics

recon_gain_info_parameter_data() contains recon gain values for demixed channels.

NOTE: recon_gain_info_parameter_data() is required to compensate the propagated errors by De-mixer and Gain modules specified in § 6.2.2 De-mixer and § 6.2.1 Gain due to the error caused by lossy codecs such as OPUS and AAC-LC. However, it is not required for lossless codecs such as FLAC and LPCM because the propagated errors are negligible.

Syntax

class recon_gain_info_parameter_data() {
  for (i=0; i< num_layers; i++) {
    if (recon_gain_is_present_flag(i) == 1) {
      leb128() recon_gain_flags(i);
      for (j=0; j< n(i); j++) {
        if (recon_gain_flag(i)(j) == 1)
          unsigned int (8) recon_gain;
      }
    }
  }
}

Semantics

recon_gain_flags indicates the channels which recon_gain is applied to.

The each bit of recon_gain_flags indicates the presence of recon_gain applied to the channel as depicted in the above figure.

• 0: It indicates that no recon_gain presents for the channel.

• 1: It indicates that recon_gain presents for the channel.

n(i) indicates the number of bits for recon_gain_flag(i). It shall be 7 or 12 as depicted in the above figure. Where, i = 0, 1, ..., num_layers - 1.

recon_gain indicates the gain value to be applied to the channel, which is indicated by recon_gain_flags , after decoding of the associated frames and demixing operation. Where, the channel is indicated by recon_gain_flags. Detailed operation by using this value is specified in § 6.2.3 Recon Gain .

2.10. Audio Frame OBU Syntax and Semantics

This section specifies OBU payloads of OBU_IA_Audio_Frame and OBU_IA_Audio_Frame_ID0 to OBU_IA_Audio_Frame_ID21.

audio_substream_id is an identifier for the substream associated with this audio frame. Within an IA Sequence, there shall be exactly one non-redundant Audio Element OBU with a audio_substream_id .

Syntax

class audio_frame_obu(audio_substream_id_in_bitstream) {
  if(audio_substream_id_in_bitstream) {
     leb128() explicit_audio_substream_id;
  }
  unsigned int (8*coded_frame_size) audio_frame();
}

Semantics

explicit_audio_substream_id defines the audio_substream_id of this frame. The value shall be greater than 21. When this field is not present audio_substream_id is implicit and is defined as a value from 0 to 21 for OBU_IA_Audio_Frame_ID0 to OBU_IA_Audio_Frame_ID21 respectively.

NOTE: The first 22 audio substreams in an IA sequence can use the OBU types OBU_IA_Audio_Frame_ID0 to OBU_IA_Audio_Frame_ID21, which have predefined audio substream identifiers associated with them. This reduces bitrate by avoiding the extra explicit_audio_substream_id field in the bitstream.

coded_frame_size is the size of audio_frame() in bytes.

audio_frame() is the raw coded audio data for the frame. It shall be opus packet of [RFC6716] for OPUS, raw_data_block() of [AAC] for AAC-LC and FRAME of [FLAC] for FLAC.

For LPCM, audio_frame() shall be LPCM samples. When more than one byte is used to represent a LPCM sample, the byte order is indicated in sample_format_flags .

2.11. Temporal Delimiter OBU Syntax and Semantics

This section specifies the OBU payload of OBU_IA_Temporal_Delimiter.

Syntax

class temporal_delimiter_obu() {
}

NOTE: The Temporal Delimiter OBU has an empty payload.

2.12. Codec Specific

This section defines codec specific information for Codec_Specific_Info and Substream.

• Codec_Specific_Info is composed of codec_id and decoder_config() .

For legacy codecs, decoder_config() shall be exactly the same information as the conventional file parser feeds to the codec decoders for decoding of the Substream. For future codecs, decoder_config() shall include all of decoding parameters which are required to decode the Substream.

• The payload of Substream is an coded stream for one or more channels. The payload format of Substream is exactly the same as the sample format (before packing OBU) for the audio file which consists of only one single coded stream by the codec_id .

2.12.1. OPUS Specific

codec_id shall be Opus .

decoder_config() for OPUS conforms to ID Header with ChannelMappingFamily = 0 of [RFC7845] with following constraints:

• Magic Signature shall not be present.

• Output Channel Count must be set to 2. Output Channel Count can be ignored because the real value can be determined from the Audio Element OBU and from the opus packet header.

• Pre-skip shall be same as the number of audio samples to be trimmed at the start of substreams.

• Output Gain shall not be used. In other words, it shall be set to 0dB.

• The byte order of each field in ID Header is converted to big endian.

The payload format of Substream is opus packet of [RFC6716] which contains only one single frame of mono or stereo channels and which has non-delimiting frame structure.

The sample rate used for computing offsets shall be 48 kHz.

2.12.2. AAC-LC Specific

codec_id shall be mp4a .

decoder_config() for AAC-LC is DecoderConfigDescriptor() of [MP4-Systems] , which is a subset of ESDBox for [MP4-Audio] , with following constraints:

• objectTypeIndication = 0x40

• streamType = 0x05 (Audio Stream)

• upstream = 0

• decSpecificInfo() : The syntax and values conforms to AudioSpecificConfig() of [MP4-Audio] with following constraints:

  • audioObjectType = 2

  • channelConfiguration must be set to 2. The real value can be implied from the Audio Element OBU.

  • GASpecificConfig() : The syntax and values conform to GASpecificConfig() of [MP4-Audio] with following constraints:

    • frameLengthFlag = 0 (1024 lines IMDCT)

    • dependsOnCoreCoder = 0

    • extensionFlag = 0

The payload format of Substream is one single raw_data_block() of [AAC] which contains only one single frame of mono or stereo channels.

The sample rate used for computing offsets shall be the rate indicated by the samplingFrequencyIndex in GASpecificConfig()

2.12.3. FLAC Specific

codec_id shall be fLaC , the FLAC stream marker in ASCII, meaning byte 0 of the stream is 0x66, followed by 0x4C 0x61 0x43.

decoder_config() for FLAC is METADATA_BLOCK of [FLAC] .

The payload format of Substream is FRAME of [FLAC] , which is composed of FRAME_HEADER , followed by SUBFRAME (s) (one SUBFRAME per channel) and followed by FRAME_FOOTER .

The sample rate used for computing offsets shall be the sampling rate indicated in the METADATA_BLOCK .

2.12.4. LPCM Specific

codec_id shall be ipcm .

decoder_config() for LPCM is as follows:

class decoder_config(ipcm) {
  unsigned int (8) sample_format_flags;
  unsigned int (8) sample_size;
  unsigned int (32) sample_rate;
}

sample_size complies with PCM_sample_size specified in [MP4-PCM] . In other words, it shall take a value from the set 16, 24 and 32.

sample_rate indicates the sample rate of the input audio in Hz. It shall take a value from the set 44.1k, 16k, 32k, 48k and 96k.

The payload format of Substream is one audio_frame() of Audio_Frame_OBU which contains only one single PCM audio frame of mono or stereo channels.

• In case of that audio_frame() contains one single PCM audio frame of stereo channels, the ith audio sample of the left channel is followed by the ith audio sample of the right channel, and then the (i+1)th audio sample of the left channel is followed by the (i+1)th audio sample of the right channel, where i = 1, 2, ..., num_samples_per_frame .

The sample rate used for computing offsets shall be sample_rate .

3. Profiles

The IA Profiles define a set of capabilities that are required to parse, decode and process the corresponding IA sequence.

NOTE: In this section and subsections, the meaning of an unique OBU is that it is still unique if it only varies by reduntant flag.

Common restrictions on the IA sequence for all profiles specified in this version of specification:

• There shall be only one unique set of Descriptor OBUs. So, if the Descriptor OBUs present in the middle of the IA sequence, all OBUs of the Decriptor OBUs shall be marked as redundant (i.e. obu_redundant_copy = 1)

  • The set of Descriptor OBUs shall not be placed in the middle of a Temporal Unit. In other words, the set of Descriptor OBUs shall be present after the last OBU of a Temporal Unit and before the first OBU of the next Temporal Unit.

• There shall be only one unique Codec Config OBU.

• Every Substream in the IA sequence shall have the same start timestamp, shall consist of the same number of Audio Frame OBUs and shall have the same trimming information.

• Every Parameter stream in the IA sequence shall have the same start timestamp as the audio substream, shall consist of the same number of Parameter Block OBUs.

  • Every Parameter Block OBU has the same duration as that of Audio Frame OBU under the same sample rate.

    • For example, when Audio Frame OBU has 960 audio samples at 48000Hz, the duration of every Parameter Block OBU shall be 960 units at 48000Hz or 480 units at 24000Hz.

• In every Temporal Unit, the start timestamp of every Audio Frame OBU shall be same as that of every Parameter Block OBU if present.

  • There shall be no redundant Parameter Block OBU.

  • Parameter Block OBUs shall come first and followed by Audio Frame OBUs.

• num_sub_mixes shall be set to 1 for this profile.

• When num_layers = 1, DemixingParameDefinion() for demixng may be present in the Audio Element OBU and IA decoders may use the (default) parameter data for demixing for (dynamic) down-mixing.

• All parameter definitons except DemixingParamDefinition() and ReconGainParamDefinition() shall have the same param_definition_mode .

  • When param_definition_mode = 0, duration, num_subblocks, constant_subblock_duration and subblock_duration shall be same in all parameter definitions, respectively.

3.1. IA Simple Profile

This section specifies the conformance points of the simple profile.

Restrictions on the IA sequence:

• There shall be only one unique Audio Element OBU.

• There shall not be any Temporal Delimiter OBUs present.

• profile_compatible shall be set to 0 to indicate that the IA sequence complies to this profile.

• profile_name shall be set to 0.

Capabilities of the IA parser, decoder and processor:

• They shall be able to parse an IA sequence with profile_name = 0.

• They shall be able to decode and process up to 16 channels.

• They shall be able to reconstruct one audio element.

• They may use (default_)demixing_info_parameter_data() to do down-mixing.

3.2. IA Base Profile

This section specifies the conformance points of the base profile.

Restrictions on IA sequence:

• There shall be at most two unique Audio Element OBUs at any one time.

  • There shall be at most one Channel-based Audio Element having num_layers > 1 at any one time.

  • There shall be at most one Scene-based Audio Element at any one time.

  • In other words, following combinations of two Audio Elements are only allowed.

    • Channel-based Audio Element having num_layers = 1 + Channel-based Audio Element having num_layers = 1.

    • Channel-based Audio Element having num_layers = 1 + Channel-based Audio Element having num_layers > 1.

    • Scene-based Audio Element + Channel-based Audio Element having num_layers = 1.

    • Scene-based Audio Element + Channel-based Audio Element having num_layers > 1.

• There may be Temporal Delimiter OBUs present.

• profile_compatible shall be set to 1 to indicate that the IA sequence complies to this profile.

• profile_name may be set to 0 or 1.

Capabilities of the IA parser, decoder and processor:

• They shall be able to parse an IA sequence with profile_name = 0 or 1.

• They shall be able to support the capabilities of the Simple Profile.

• They shall be able to decode and process up to 18 channels.

  • Where, 18 channels means not the number of channels after mixing of two Audio Elements but the total sum of channels before mixing of two Audio Elements .

  • One specific example of 18 channels is 3rd-order Ambisonics (16 channels) + non-diegetic stereo (2 channels).

• They shall be able to reconstruct two audio elements.

• They shall be able to mix two audio elements.

4. Standalone IAMF Representation

This section details the order in which the OBUs are sequenced in a standalone IAMF representation.

4.1. OBU Sequence Order

An IA sequence is composed of a series of OBUs in the sequence of a set of Descriptor OBUs followed by their associated Data OBUs.

NOTE: In typically case, the first Descriptor OBUs of the IA sequence are all non-redundant (i.e. obu_redundant_copy = 0).

The descriptor OBUs may additionally be repeated redundantly and as frequently as necessary. In this case, the "obu_redundant_copy" field in the OBU header of each of the descriptor OBUs shall be set to 1.

The below figure shows an example of IA sequence.

4.1.1. Descriptor OBUs

1. One IA Sequence Header OBU

2. All Codec Config OBUs

3. All Audio Element OBUs

4. All Mix Presentation OBUs

4.1.2. Data OBUs

Data OBUs consists of a sequence of Audio Frame OBUs, Parameter Block OBUs and Temporal Delimiter OBUs (if present), according to the rules below:

• Audio Frame OBUs and Parameter Block OBUs shall be ordered by their implied timestamp in the timeline, and may be interleaved.

• If there are multiple Audio Frame OBUs that have the same implied start timestamp, they shall be grouped by audio elements.

• A Temporal Delimiter OBU may be inserted at the beginning of a Temporal Unit.

• If Temporal Delimiter OBUs are present, they shall be inserted at the beginning of every Temporal Unit.

Additionally, the following constraints apply to the Audio Frame and Parameter Block OBUs:

• Audio Frame OBUs shall be provided non-redundantly, such that for each substream, there shall not be two audio frames that are overlapping in time.

• Non-redundant Parameter Block OBUs must not provide data for overlapping time regions.

4.1.3. Refreshing Descriptor OBUs

The above describes the full sequence of OBUs for a given set of Descriptor OBUs and their associated Data OBUs.

If the IAMF configuration changes, a new set of Descriptor OBUs is required. In that case, a new IA sequence of the complete set of Descriptor OBUs and their corresponding Data OBUs shall follow, in the same order as described above.

• Each of the Descriptor OBUs of the new IA sequence shall be marked as non-redundant (i.e. obu_redundant_copy = 0).

5. ISOBMFF IAMF Encapsulation

5.1. General Requirements & Brands

A file conformant to this specification satisfies the following:

• It shall conform to the normative requirements of [ISOBMFF]

• It shall have the iamf brand among the compatible brands array of the FileTypeBox

• It shall contain at least one track using an IASampleEntry

• It SHOULD indicate a structural ISOBMFF brand among the compatible brands array of the FileTypeBox, such as iso6

• It MAY indicate other brands not specified in this specification provided that the associated requirements do not conflict with those given in this specification

Parsers shall support the structures required by the iso6 brand and MAY support structures required by further ISOBMFF structural brands.

5.2. ISOBMFF IAMF Encapsulation

This section describes the basic data structures used to signal encapsulation of IA sequence in [ISOBMFF] containers.

5.2.1. Requirement of IA sequence

Even though an IA sequence can theoretically group audio data coded with different codecs, potentially with different timing properties, which would require multiple tracks, this version of the specification only supports storing an IA Sequence as a single track thanks to the restrictions of the selected profiles.

5.2.2. Encapsulation Scheme

The result of encapsulating an IA Sequence into an [ISOBMFF] file is as follows:

• If there are audio samples to be trimmed at the start or at the end, then edts and elst boxes shall be present to reflect the trimming status.

• Sample Entry

  • For an IA Sample, the required Descriptor OBUs required for processing the samples shall be in the configOBUs of the associated sample entry.

NOTE: Multiple sample entries may be used in a track, for example when the track is the concatenation of multiple tracks or multiple IA Sequences and some IA samples have different configOBUs values.

• Decoding Time to IA Sample

  • stts or trun box shall indicate the number of audio samples which IA Sample includes (i.e. the duration of IA Sample).

  • The duration of IA Sample is duration including audio samples trimmed at the beginning but excluding audio samples trimmed at the end.

• Sample Group

  • When codec_id is set to Opus or mp4a , in an IA Track, every sample shall be associated with a sample group of type roll . The roll_distance value SHALL equal to the value of the audio_roll_distance field in the Codec Config OBU stored in the configOBUs array in the sample entry.

5.2.3. IA Sample Entry

Sample Entry Type: iamf
Container:         Sample Description Box ('stsd')
Mandatory:         Yes
Quantity:          One or more.

IASampleEntry identifies that the track contains IA Samples , and contains configOBUs.

Syntax

class IASampleEntry extends AudioSampleEntry('iamf') {
  unsigned int (8) configOBUs[];
}

The channelcount and samplerate fields of AudioSampleEntry are unused.

None of AudioSampleEntry’s optional boxes shall be present.

Semantics

configOBUs shall contain the following OBUs in order.

• IA Sequence Header OBU

• Codec Config OBU

• One or more Audio Element OBUs

• One or more Mix Presentation OBUs

5.2.4. IA Sample Format

For tracks using the IASampleEntry , an IA Sample has the following constraints:

• One IA Sample data shall be one Temporal Unit and shall not contain Temporal Delimiter OBU.

• The decode duration of an IA Sample SHALL equal the duration of the underlying temporal unit, i.e. the decode durations of the Audio Frame OBU.

NOTE: Per the restriction of the profiles carried in an IA track, all Audio Frame OBUs in an IA Sample have the same duration and have the same trimming information. If Audio Frame OBUs in the IA sample contain trimming information, the corresponding audio samples SHALL be removed from presentation using edit list information.

NOTE: In typical case, when a track contains a single IA Sequence, trimming can only happen at the beginning or end of the IA sequence and therefore at the beginning or end of the track and the edit list can describe the start and end trimming with a single edit entry. Track storing consecutive IA Sequences may need multiple edits in the edit list.

5.3. Codecs Parameter String

• For OPUS

iamf.IAMF-specific-needs.Opus

• For AAC-LC

iamf.IAMF-specific-needs.mp4a.40.2

• For FLAC

iamf.IAMF-specific-needs.fLaC

• For LPCM

iamf.IAMF-specific-needs.ipcm

IAMF-specific-needs shall be PC as follows:

• PC is three digits within the range 0 to 255 and represents that IA decoder supports the profile of specification.

For example, for this version of the specification

• The codecs parameter string of OPUS for the simple profile:

iamf.000.Opus

• The codecs parameter string of AAC-LC for the base profile:

iamf.001.mp4a.40.2

5.4. ISOBMFF IAMF Decapsulation (Informative)

5.4.1. ISOBMFF IAMF Decapsulation with single track

This section provides a guideline for IAMF parser to reconstruct IA sequences from IAMF file.

When IAMF parser feeds the reconstructed IA sequences to OBU parser, descriptor OBUs shall be placed at the first and followed by Temporal Units.

During decapsulation process, IAMF file is decapsulated into IA sequences which conform to § 2 Open Bitstream Unit (OBU) Syntax and Semantics as follows:

• The ith IA sequence is reconstructed as follows:

  • Step1: Take the ith descriptor OBUs from its associated IASampleEntry.

  • Step2: Take jth sample as it is and add Temporal Delimiter OBU in front of the jth sample.

    • Every Temporal Unit shall have Temporal Delimiter OBU or no Temporal Unit shall have Temporal Delimiter OBU.

  • Step3: Place the ith descriptor OBUs, and followed by Temporal Units in order (j = i1, i2, …, im) without gap, to reconstruct the ith IA sequence.

• Place IA sequences in order (i = 1, 2, 3, ...) to reconstruct the IA sequences.

5.4.2. Recommended handling of Trimming Information

This section recommends how to handle trimming information of ISOBMFF file.

As depicted in the above figure,

• ISOBMFF parser passes descriptor OBUs, PTS1 and Samples (or Temporal Units) to IAMF decoder.

• ISOBMFF parser passes PTS1 and trimming information to ISOBMFF player.

• IAMF decoder passes PTS and audio samples after decoding to ISOBMFF player.

  • If IAMF decoder trims the audio samples to be trimmed based on the trimming information within Audio Frame OBUs, then IAMF decoder passes PTS2 and audio samples after trimming.

  • If IAMF decoder does not trim, then IAMF decoder passes PTS1 and audio samples before trimming.

• ISOBMFF player playbacks audio samples starting at PTS2 to Loudspeakers.

Where, PTS1 is the presentation time stamp of the first audio sample before trimming and PTS2 is the presentation time stamp of the first audio sample after trimming.

6. IAMF processing

This section provides processes for IA decoding for a given IA sequence .

IA decoding can be done by using the combination of following decoding processing.

• Decoding of a scene-based audio element (Ambisonics decoding)

• Decoding of a channel-based audio element (Scalable Channel Audio decoding)

• Rendering and mixing of each audio element before mixing of multiple audio elements.

  • It may include re-sampling of each audio element.

• Mixing of multiple audio elements with synchronization

• Post processing such as Loudness and Limiter.

Ambisonics decoding , it shall conform to [RFC8486] except codec specific processing and shall output Ambisonics channels in ACN (Ambisonics Channel Number) order.

Scalable Channel Audio decoding , it shall output the channel audio (e.g. 3.1.2ch or 7.1.4ch) for the target channel layout.

IA decoder is composed of OBU parser, Codec decoder, Audio Element Renderer and Post-processor as depicted in below figure.

• OBU parser shall depacketize IA sequence to output one or more substreams with one or more Decoder_Config() but one decoder_config() per audio element, descriptors and parameters.

• Codec decoder for each substream shall output decoded channels.

• Audio Element Renderer reconstructs audio channels from decoded channels of Codec decoders according to the type of audio element which is specified audio element OBU, and renders the audio channels to the target loudspeaker layout.

  • For scene-based audio element, it shall output audio channels for the target loudspeaker layout from the reconstructed ambisonics channels.

  • For channel-based audio element, it shall output audio channels for the target loudspeaker layout from the reconstructed audio channels.

• Post-processor outputs audio channels according to the target loudspeaker layout after processing mixing and post processing such as Loudness and Limiter.

6.1. Ambisonics decoding

This section describes the decoding of Ambisonics.

Below figure shows the decoding flowchart of Ambisonics decoding.

• OBU parser shall output the substreams for the scene-based audio element in IA sequence.

  • OBU parser shall output channel_mapping or demixing_matrix according to ambisonics_mode to Channel_Mapping/Demixing_Matrix module

• Codec decoder shall output decoded channels (PCM) in the transmission order as many as output_channel_count after decoding of each Substream.

• Channel_Mapping/Demixing_Matrix module shall apply channel_mapping or demixing_matrix according to Ambisonics_Mode to the channels (PCM) and outputs channels as many as output_channel_count in ACN order.

• Ambisonics to Channel Format module may convert the output channels to channel audio according to the target loudspeaker layout.

6.2. Scalable Channel Audio decoding

This section describes the decoding of Scalable Channel Audio.

Below figure shows the decoding flowchart of the decoding for Scalable Channel Audio.

For a given loudspeaker layout (i.e. CL #i) among the list of loudspeaker_layout in scalable channel layout config,

• OBU Parser shall get substreams for ChannelGroup #1 ~ ChannelGroup #i and pass them to Codec decoder with Decoder_Config() .

• Codec decoder shall output decoded channels (PCM) in the transmission order.

  • For non-scalable audio (i.e i = 1), its order shall be converted to the loudspeaker location order for CL #1.

• Following are further processed for scalable audio (i.e. i > 1)

  • When Output_Gain_Is_Present_Flag(j) for ChannelGroup #j (j = 1, 2, …, i-1) is on, Gain module shall apply Output_Gain(j) to all audio samples of the mixed channels in the ChannelGroup #j indicated by Output_Gain_Flag(j).

  • De-Mixer shall output de-mixed channels (PCM) for CL #i generated through de-mixing of the mixed channels from Gain module by using non-mixed channels and demixing parameters for each frame.

  • Recon_Gain module shall output smoothed channels (PCM) by applying Recon_Gain to each frame of the de-mixed channels.

  • The order for Non-mixed channels and Smoothed channels shall be converted to the loudspeaker location order for CL #i after going through necessary modules such as Gain, De-Mixer, Recon_Gain etc..

• Following may be further processed

  • Loudness normalization module may output loudness normalized channels at -24 LKFS from non-mixed channels and smoothed channels (if present) by using loudness value for CL #i.

  • Limiter module may limit the true peak of input channels at -1dB.

Following sections, § 6.2.1 Gain , § 6.2.2 De-mixer and § 6.2.3 Recon Gain are only needed for decoding of scalable audio with num_layers > 1.

6.2.1. Gain

Gain module is the mirror process of Attenuation module. It recovers the reduced sample values using Output_Gain when its flag for ChannelGroup #j is on. When its flag is off, then this module shall be bypassed for ChannelGroup #j. Output_Gain(j) for ChannelGroup #j shall be applied to all samples of the mixed channels in the ChannelGroup #j. Where, mixed channels means the mixed channels from an input channel audio (i.e. a channel audio for CL #n).

To apply the gain, an implementation MUST use the following:

Sample *= pow(10, Output_Gain(j) / (20.0*256))

Where, Output_Gain(j) is the raw 16-bit value for jth layer which is specified in channel_audio_layer_config() .

6.2.2. De-mixer

For scalable channel audio with num_layers > 1, some channels of down-mixed audio for CL #i are delivered as is but the rest are mixed with other channels for CL #i-1.

De-mixer module reconstructs the rest of the down-mixed audio for CL #i from the mixed channels, which is passed by Gain module, and its relevant non-mixed channels using its relevant demixing parameters.

De-mixing for down-mixed audio for CL #i shall comply with the result by the combination of following surround and top de-mixers:

• Surround de-mixers

  • S1to2 de-mixer : R2 = 2 x Mono – L2

  • S2to3 de-mixer : L3 = L2 – 0.707 x C and R3 = R2 – 0.707 x C

  • S3to5 de-mixer : Ls = 1/δ(k) x (L3 – L5) and Rs = 1/δ(k) x (R3 – R5)

  • S5to7 de-mixer : Lrs = 1/β(k) x (Ls – α(k) x Lss) and Rrs = 1/β(k) x (Rs – α(k) x Rss)

• Top de-mixers

  • TF2toT2 de-mixer : Ltf2 = Ltf3 – w(k) x (L3 – L5) and Rtf2 = Rtf3 – w(k) x (R3 – R5)

  • T2to4 de-mixer : Ltb = 1/γ(k) x (Ltf2 – Ltf4) and Rtb = 1/γ(k) x (Rtf2 – Rtf4)

• Where, Ltf2 / Rtf2 is top channel of x.1.2ch, Ltf3 / Rtf3 is top channel of 3.1.2ch, and Ltf4 / Rtf4 is to channel of x.1.4ch (x = 5 or 7) and w(k) is determined from the value of wIdx(k).

Initially, wIdx(0) = 0 and the value of wIdx(k) shall be derived as follows:

• wIdx(k) = Clip3(0, 10, wIdx(k-1) + w_idx_offset(k))

Mapping of wIdx(k) to w(k) should be as follows:

wIdx(k) :   w(k)
   0    :    0
   1    :  0.0179
   2    :  0.0391
   3    :  0.0658
   4    :  0.1038
   5    :  0.25
   6    :  0.3962
   7    :  0.4342
   8    :  0.4609
   9    :  0.4821
   10    : 0.5

When D_set = { x | S1 < x ≤ Si and x is an integer},

• If 2 is an element of D_set, the combination shall include S1to2 de-mixer .

• If 3 is an element of D_set, the combination shall include S2to3 de-mixer .

• If 5 is an element of D_set, the combination shall include S3to5 de-mixer .

• If 7 is an element of D_set, the combination shall include S5to7 de-mixer .

When Ti = 2,

• If Sj = 3 (j=1,2,…, i-1), the combination shall include TF2toT2 de-mixer .

When Ti = 4,

• If Sj = 3 (j=1,2,…, i-1), the combination shall include TF2toT2 de-mixer and T2to4 de-mixer .

• Elseif Tj = 2 (j=1,2,…, i-1), the combination shall include T2to4 de-mixer .

For example, when CL #1 = 2ch, CL #2 = 3.1.2ch, CL #3 = 5.1.2ch and CL #4 = 7.1.4ch. To reconstruct the rest (i.e. Ls5/Rs5/Ltf/Rtf) of the down-mixed 5.1.2ch,

• The combination includes S2to3 de-mixer , S3to5 de-mixer and [=TF2toF2 de-mixer].

• Ls5 and Rs5 are recovered by S2to3 de-mixer and S3to5 de-mixer.

• Ltf and Rtf are recovered by S2to3 de-mixer and TF2toT2 de-mixer.

Ls5 = 1/δ(k) × (L2 - 0.707 × C - L5) and Rs5 = 1/δ(k) × (R2 - 0.707 × C - R5).
Ltf = Ltf3 - w(k) x (L2 - 0.707 x C - L5) and Rtf = Rtf3 - w(k) x (R2 - 0.707 x C - R5).

6.2.3. Recon Gain

recon_gain shall be only applied to all of audio samples of the de-mixed channels from De-mixer module.

• recon_gain_info_parameter_data() indicates each channel of CL #i which Recon_Gain needs to be applied to and provides Recon_Gain value for each frame of the channel.

  • Sample (k,i) *= Smoothed_Recon_Gain (k,i), where k is the frame index and i is the sample index of the frame.

  • Smoothed_Recon_Gain (k) = MA_gain (k-1) x e_window + MA_gain (k) x s_window

  • MA_gain (k) = 2 / (N+1) x Recon_Gain (k) / 255 + (1 – 2/(N+1)) x MA_gain (k-1), where MA_gain (0) = 1.

  • e_window[0: olen] = hanning[olen:], e_window[olen:flen] = 0.

  • s_window[0: olen] = hanning[:olen], s_window[olen:flen] = 1.

  • Where, hanning = np.hanning (2*olen), flen is the frame size and olen is the overlap size.

  • Recommend values: N = 7

Below figure shows the smoothing scheme of recon_gain .

Recommend values for specific codecs are as follows

• OPUS: olen = 60.

• AAC-LC: olen = 64.

6.3. Mix Presentation

An IA sequence may contain more than one mix presentation. § 6.3.1 Selecting a Mix Presentation details how a mix presentation should be selected from multiple of them.

A mix presentation specifies how to render, process and mix one or more audio elements. Each audio element should first be individually rendered and processed before mixing. Then, any additional processing specified by output_mix_config() should be applied to the mixed audio signal in order to generate the final output audio for playback. § 6.3.2 Rendering an Audio Element details how each audio element should be rendered, while § 6.3.3 Mixing Audio Elements details how the audio elements should be processed and mixed.

6.3.1. Selecting a Mix Presentation

An IA sequence may contain more than one mix presentations. The IA parser should select the appropriate mix presentation in the following order.

1. If there are any user-selectable mixes, the IA parser should select the mix, or mixes, that match the user’s preferences. An example might be a mix with a specific language. Mix presentations may use mix_presentation_friendly_label to describe such mixes.

2. If there are more than one valid mixes remaining, the IA parser should select an appropriate mix for rendering, in the following order.

   1. If the playback layout is binaural, i.e. headphones:

      1. Select the mix with audio_element_id whose loudspeaker_layout is BINAURAL.

      2. If there is no such mix, select the mix with the highest available loudness_layout .

   2. If the playback layout is loudspeakers:

      1. If there is a mix with an loudness_layout that matches the playback loudspeaker layout, it should be selected. If there are more than one matching mixes, the first one should be selected.

      2. If there is no such mix, select the mix presentation with the highest available loudness_layout .

6.3.2. Rendering an Audio Element

This specification supports the rendering of either a multichannel or ambisonics audio element to either a target loudspeaker layout or a binaural output.

In this section, for a given x.y.z layout, the next highest layout x'.y'.z' means that x', y' and z' are greater than or equal to x, y and z, respectively.

6.3.2.1. Rendering a channel-based audio element to loudspeakers

This section defines the renderer to use, given a channel-based audio element and a loudspeaker playback layout.

• The input layout of the IA renderer is set as follows:

  • If num_layers = 1, use the loudspeaker_layout of the audio element.

  • Else, if there is a loudness_layout that matches the playback layout, use it.

  • Else, use the next highest available layout from all available loudspeaker_layout and loudness_layout .

• The output layout of the IA renderer is set to the playback layout.

• The IA renderer used is selected according to the following rules:

  • If the playback layout matches a loudspeaker_layout which can be generated from the highest loudspeaker layout of the audio element according to § 9.2.3 Annex B-3: Channel Layout Generation Rule , use demixing_info_parameter_data().

    • If demixing_info_parameter_data() is not delivered,

      • If the playback layout complies with loudspeaker layouts supported by [ITU2051-3] , use EAR Direct Speakers renderer ( [ITU2127-0] ).

      • Else if the playback layout is 3.1.2ch, use the static down-mix matrices specified in § 6.6.2 Static Down-mix Matrix .

      • Else, use implementation-specific renderer

  • Else if the playback layout complies with loudspeaker layouts supported by [ITU2051-3] , use EAR Direct Speakers renderer ( [ITU2127-0] ).

  • Else if the playback layout is 3.1.2ch, use the static down-mix matrices specified in § 6.6.2 Static Down-mix Matrix .

  • Else, use implementation-specific renderer.

If the EAR Direct Speakers renderer is used, the following should be provided for each audio channel of the audio element:

• speaker label: the label of the speaker position, using the same convention as "SP Label" in [ITU2051-3] . This is defined for each audio channel of the audio element based on the information from loudspeaker_layouts .

In [ITU2051-3] , an LFE audio channel may be identified either by an explicit label or its frequency content. In this specification, the LFE channel is identified based on the explicit label only, given by loudspeaker_layout .

6.3.2.2. Rendering a scene-based audio element to loudspeakers

This section defines the renderer to use, given a scene-based audio element and a loudspeaker playback layout.

• The input layout of the IA renderer is set to Ambisonics.

• The output layout of the IA renderer is set to the playback layout.

• The IA renderer used is selected according to the following rules:

  • If the playback layout complies with loudspeaker layouts supported by [ITU2051-3] , use EAR HOA renderer ( [ITU2127-0] ).

  • Else, use implementation-specific renderer.

    • If there is no implementation-specific Ambisonics renderer, use the EAR HOA renderer to render to the next highest [ITU2051-3] layout compared to the playback layout, and then downmix using implementation-specific renderer.

If the EAR HOA renderer is used, the following metadata should be provided to the renderer for each audio channel:

1. Ambisonics order

2. Ambisonics degree

3. Ambisonics normalization method

In this specification, the [AmbiX] format is adopted, which uses SN3D normalization and ACN channel ordering. Accordingly, the Ambisonics order and degree can be computed from the channel index k as follows:

order   n = floor(sqrt(k)),
degree  m = k - n * (n + 1).

6.3.2.3. Rendering a channel-based audio element to a binaural output

Given a channel-based audio element and a binaural playback layout, the Binaural EBU ADM Direct Speaker renderer [BEAR] should be used. The highest layout provided in scalable_channel_layout_config() should be used as the input to the renderer.

6.3.2.4. Rendering a scene-based audio element to a binaural output

Given a scene-based audio element and a binaural playback system, the Resonance Audio renderer [Resonance-Audio] should be used.

6.3.3. Mixing Audio Elements

Each audio element is processed individually before mixing as follows:

1. Render to the playback layout.

2. If all audio elements do not have a common sample rate, re-sample to 48 kHz.

3. If all audio elements do not have a common bit-depth, convert to a common bit-depth. This specification recommends using 16 bits.

4. If loudness_layout matches with the playback layout, apply any per-element processing according to element_mix_config() .

The rendered and processed audio elements are then summed, and then apply output_mix_config() to generate one sub-mixed audio signal. If there are more than one sub-mixes, the output of each sub-mix is further summed to generate the final mixed audio signal.

6.4. Animated Parameters

This section describes how a set of parameters is animated over a subblock in a parameter block and applied to the corresponding audio samples, using the information provided in AnimatedParameterData() .

If animation_type is equal to STEP, the parameter value provided by start_point_value should be applied to all time steps in the subblock.

If animation_type is equal to LINEAR or BEZIER, the information provided in AnimatedParameterData() describes how the parameter is animated as a Bezier curve. Let T be the subblock duration defined in the parameter_block_obu() and P0 , P1 and P2 be 2D coordinates defined as

P0 = (t0, start_point_value),
P1 = (t1, control_point_value),
P2 = (t2, end_point_value),

where t0 = 0 is the subblock start time, t2 = D is the subblock end time and t1 is the control point time given by

t1 = round(D * control_point_relative_time).

The values of t0 , t1 and t2 are expressed as ticks at the parameter_rate given in the associated parameter definition.

If animation_type is equal to LINEAR, the parameter value is linearly interpolated between start_point_value and end_point_value at a given point in time as:

B_linear(a) = (1 - a) * P0 + a * P2,
0 <= a <= 1,

where B_linear(a) = (t, y) is a 2D coordinate with the parameter value y at time t .

If animation_type is equal to BEZIER, the parameter value is interpolated following a quadratic Bezier curve between start_point_value and end_point_value at a given point in time as:

B_quad(a) = (1 - a)^2 * P0 + 2 * (1 - a) * a * P1 + a^2 * P2,
0 <= a <= 1.

where B_quad(a) = (t, y) is a 2D coordinate with parameter value y at time t .

To apply the parameter values to the audio samples in the subblock without interpolation, the parameter_rate is first resampled to the audio sample rate to give:

n0 = t0 * audio_sample_rate / parameter_rate,
n1 = t1 * audio_sample_rate / parameter_rate,
n2 = t2 * audio_sample_rate / parameter_rate.

Then, P0 , P1 , P2 can be rewritten as:

P0 = (n0, start_point_value),
P1 = (n1, control_point_value),
P2 = (n2, end_point_value).

Next, the parameter value y is computed for each time t that corresponds to an integer audio sample index, t = n = [0, 1, 2, ..., n2] . This is done by computing the equivalent value of a for every n , and then applying the Bezier equations B_linear(a) and B_quad(a) to find the parameter value y .

In the case of B_linear(a) , the mapping between n and a is given by:

a = n ÷ n2.

In the case of B_quad(a) , the mapping between n and a is given by:

a = (-beta + sqrt(beta^2 - 4 * alpha * gamma)) ÷ (2 * alpha),

where

alpha = n0 - 2 * n1 + n2,
beta = 2 * (n1 - n0),
gamma = n0 - n.

6.5. Post Processing

6.5.1. Loudness Normalization

Loudness normalization should be done by adjusting the loudness level to a target output level using information provided in § 2.8.7 Loudness Info Syntax and Semantics . A control may be provided to set unique target output levels for each anchored loudness and/or the integrated loudness. If loudness normalization increases the output level, a peak limiter to prevent saturation and/or clipping may be necessary; true_peak or digital_peak may be used to determine if peak limiting is needed. Alternately, the total amount of normalization may be limited.

The rendered layouts that were used to measure the loudness information of a sub-mix are provided by loudness_layout s.

If one of them matches the playback layout, the loudness information should be used directly for normalization. If there is a mismatch between loudness_layout and the playback layout, the implementation may choose to use the provided loudness information of the highest loudness_layout as-is.

If there is more than one selected loudness_info() specified in the mix presentation (i.e. in case of multiple sub-mixes), the implementation should normalize the loudness of each sub-mix independently before summing them.

6.5.2. Limiter

The limiter should limit the true peak of audio signal at -1 dBTP, where true peak is defined in [ITU1770-4] . The limiter should apply to multichannel signals in a linked manner and further support auto-release.

6.6. Down-mix Matrix

6.6.1. Dynamic Down-mix Matrix

This section recommends dynamic down-mixing matrices.

The dynamic down-mixing matrics complies with the down-mixing mechanism which is specified in § 9.2.2 Annex B-2: Down-mix Mechanism .

6.6.2. Static Down-mix Matrix

This section specifies static down-mix matrices to render to 3.1.2ch from each of 5.1.2ch, 5.1.4ch, 7.1.2ch and 7.1.4ch.

The figures below show static down-mix matrices to 3.1.2ch.

Where, p1 = 0.707. Implementations may use limiter defined in § 6.5.2 Limiter to preserve energy of audio signals instead of normalization factors.

7. IAMF Generation Process (Informative)

This section provides a guideline for IA encoding for a given input audio format.

Recommended input audio format for IA encoding is as follows:

• Ambisonics format: It conforms to ChannelMappingFamily = 2 or 3 of [RFC8486] .

• Channel Audio format: It conforms to loudspeaker_layout specified in channel_audio_layer_config().

• Input Sampling Rate: 48000Hz

• Bitdepth: 16 bits or 24 bits

  • 16 bits are recommended for OPUS.

• Input file format: .wav file (Linear PCM, simply called as PCM)

For a given input audio and user inputs, IA encoder outputs IA sequence which conforms to § 2 Open Bitstream Unit (OBU) Syntax and Semantics .

Input audio is as follows:

• Ambisonics format

• Channel Audio format

User inputs are:

• Ambisonics mode to indicate if ChannelMappingFamily = 2 or 3 of [RFC8486] .

• List of channel layouts to be supported for scalable channel audio: it conforms to loudspeaker_layout .

IA encoding can be done by using the combination of following generation processing.

• Encoding of an audio element (Ambisonics encoding or Scalable Channel Audio encoding)

• Encoding of mix presentation

The below figure shows IA encoder configuration for one single audio element.

The IA encoder is composed of Pre-processor, Codec encoder and OBU packetizer.

• Pre-processor outputs one or more ChannelGroups, descriptors and optional parameter blocks based on the input audios and user inputs.

  • It outputs one single ChannelGroup for scene-based audio element.

  • It outputs one or more ChannelGroups for channel-based audio element.

  • It outputs descriptors which are composed of one IA Sequence Header, one Codec Config, one Audio Element config, one or more Mix Presentation config.

  • It may output parameter blocks

    • For channel-based audio element with num_layers = 1, it may output parameter blocks for demixing info.

    • For channel-based audio element with num_layers > 1, it outputs parameter blocks for demixing_info_parameter_data and recon_gain_info_parameter_data.

    • It may further output parameter blocks for mixing gain.

• Codec encoder generates one or more substreams from each ChannelGroup based on Codec Config.

• OBU packetizer packetize descriptors, parameter blocks and audio frames by OBU, and outputs IA sequence.

  • Temporal unit generator generates temporal unit for each frame from audio frame OBUs and parameter block OBUs (if present).

7.1. Ambisonics Encoding

For Ambisonics encoding:

• Pre-processor outputs one ChannelGroup and descriptors and it is only composed of Meta Generator.

  • Meta generator generates descriptors based on Ambisonics mode and the number of channels for Ambisonics.

    • ambisonics_mode is set to 0 for ChannelMappingFamily = 2 of [RFC8486] or 1 for ChannelMappingFamily = 3 of [RFC8486] .

    • ambisonics_config is set to as follows:

      • output_channel_count is set to the number of channels for Ambisonics. For example 1, 4, 9 or 16.

      • channel_mapping for ambisonics_mode = 0 is assigned to according to the order of substreams in ChannelGroup.

      • demixing_matrix for ambisonics_mode = 1 is assigned to according to the order of substreams in ChannelGroup.

• Codec Enc. outputs substreams as many as the number of channels which is indicated in substream_count .

• Temporal unit is composed of audio frame OBUs for substreams.

  • It may have the immediately preceding temporal delimiter OBU.

  • The order of substreams in ChannelGroup is aligned with channel_mapping for Ambisonics_Mode = 0 or demixing_matrix for Ambisonics_Mode = 1.

7.2. Scalable Channel Audio Encoding

For Scalable Channel Audio encoding:

• Pre-processor outputs one or more ChannelGroups, descriptors and parameter blocks. It is composed of Down-mix parameter generator, Down-mixer, Loudness, ChannelGroup generator, Attenuation and Meta generator.

  • For non-scalable channel audio (i.e. num_layers = 1):

    • Parameter blocks for recon_gain_info_parameter_data is not be generated.

    • Parameter blocks for demixing_info_parameter_data may be generated by implementers who assume it to be recommended for dynamic downmixing in a decoder side.

    • Down-mixer, ChannelGroup generator and Attenuation modules do not needed.

  • Down-mix parameter generator generates 5 down-mix parameters (α(k), β(k), γ(k), δ(k) and w(k)) by analyzing input channel audio.

  • Down-mixer generates down-mixed audios according to the list of channel layouts and the down-mix parameters.

  • Loudness module should output the loudness level ( LKFS ) of each down-mixed audio based on [ITU1770-4] .

  • ChannelGroup generator transforms the input channel audio to ChannelGroups for scalable channel audio with num_layers > 1 scalability by using the down-mix parameters and the list of channel layouts.

  • Attenuation module applies a gain to the transformed ChannelGroups to prevent clipping.

  • Meta generator generates descriptors and parameter block(s) for each frame.

    • descriptors are set to as follows:

      • num_layers is set to the number of channel layouts.

      • channel_audio_layer_config() is set to as follows:

        • loudspeaker_layout is set to the ith list of channel layouts for the ith ChannelGroup.

        • output_gain_is_present_flag is set to 1 for the ith ChannelGroup if attenuation is applied to the mixed channels of the ith ChannelGroup. Otherwise it is set to 0 for the ith ChannelGroup.

        • recon_gain_is_present_flag is set to 1 for the ith ChannelGroup if the preceding ChannelGroups has one or more mixed channels from the down-mixed audio for the ith channel layout. Otherwise, it shall be set to 0 for the ith ChannelGroup. Especially, when num_layers = 1, this flag shall be set to 0.

          • This flag is set to 0 for lossless codecs including LPCM.

        • substream_count is set to the number of substreams composing of the ith ChannelGroup.

        • coupled_substream_count is set to the number of coupled substreams among the substreams composing of the ith ChannelGroup.

        • loudness is set to the loudness ( LKFS ) of the down-mixed audio for the ith channel layout for the ith ChannelGroup.

        • Each bit of output_gain_flags is set to 1 for the ith ChannelGroup if attenuation is applied to the relevant channel of the ith ChannelGroup. Otherwise it is set to 0 for the ith ChannelGroup.

        • output_gain is set to the inverse number of the gain which is applied to the channels which are indicated by output_gain_flags.

    • Parameter blocks can be composed of demixing_info_parameter_data() and recon_gain_info_parameter_data() . When recon_gain_is_present_flag = 0 for all ChannelGroup, recon_gain_info does not present in IA sequence.

      • dmixp_mode of demixing_info_parameter_data for the kth frame is set to indicate (α(k), β(k), γ(k), δ(k)) and w_idx_offset(k). Where w_idx_offset(k) = 1 or -1.

      • recon_gain_flags of recon_gain_info_parameter_data is set to indicate the de-mixed channels, which need to apply recon_gain among the output channels after demixing for ith channel layout.

      • recon_gain is set to the gain value to be applied to the channel which is indicated by recon_gain_flags for the ith ChannelGroup.

• Temporal unit for kth frame is composed of audio frame OBUs for the kth frames of the substreams and followed by OBUs for zero or more parameter block OBUs.

  • It may have the immediately preceding temporal delimiter OBU,

  • ChannelGroups in temporal unit is placed in order. In other words, ChannelGroup for the first channel layout comes first, followed by ChannelGroup for the second channel layout, followed by ChannelGroup for the third channel layout and so on.

Below figure shows IA encoding flowchart for Scalable Channel Audio.

• For a given Channel Audio and a given list of channel layouts for scalability, PCMs for Channel Audio are passed to CG Generation module.

• CG Generation module generates the transformed audio according to CG generation rule based on the list of CLs and the down-mix parameters.

  • The transformed audio is structured as ChannelGroups.

• Non-mixed channels of the transformed audio (i.e., the original channels of the input channel audio) are directly input to Codec encoder, but the mixed channels may be input first to Attenuation module and then to Codec encoder.

• The Attenuation module reduces all sample values of the mixed channels in the same CG at a uniform rate (Output_Gain).

  • A range of 0dB to -6dB is recommended for the attenuation. (i.e. a range of 0dB to 6dB for Output_Gain)

• Codec Enc. generates the coded substreams from PCMs and passes substreams and one single decoder_config to OBU Packetizer.

• OBU packetizer generates descriptor OBUs which consists of one IA Sequence Header OBU, one Codec Config OBU, one Audio Element OBU and zero or more Mix Presentation OBU.

  • Codec Config OBU is generated based on decoder_config() .

• OBU packetizer generates zero or more parameter block OBUs for each frame which contains demixing_info_parameter_data and recon_gain_info_parameter_data.

• OBU packetizer generates audio frame OBUs for each frame of the substreams.

• OBU packetizer generates temporal unit for each frame.

  • Temporal unit consists of audio frame OBUs and followed by zero or more parameter block OBUs.

    • It may have the immediately preceding temporal delimiter OBU,

• OBU Packetizer outputs IA sequence which is composed of OBUs for descriptor OBUs and followed by OBUs for temporal units.

7.3. Mix Presentation Encoding

For Mix Presentation OBU for one single channel-based audio element, Mix Presentation OBU follows following restrictions:

• num_sub_mixes : set to 1

• num_audio_elements : set to 1

• element_mix_config() : No parameter block for element_mix and default_mix_gain = 0dB

• output_mix_config() : No parameter block for output_mix and default_mix_gain = 0dB

• num_layouts : set to num_layers

• loudness_layout : set to L(1), L(2), ..., L( num_layers ).

• loudness_info() on L(1), loudness_info on L(2), ..., loudness_info on L( num_layers ): loudness information of the rendered audio to the measured layout L(i).

• Where L(i) is the measured layout for the ith layer and i = 1, 2, ..., num_layers

For Mix Presentation for one single scene-based audio element, Mix Presentation OBU follows following restrictions:

• num_sub_mixes : set to 1

• num_audio_elements : set to 1

• element_mix_config() : set to mix_gain

• output_mix_config() : set to output_mix_gain

• num_layouts : set to M1, the number of loudness informations which are provided.

• loudness_layout : set to L(1), L(2), ..., L(M1).

• loudness_info() on L(1), loudness_info on L(2), ..., loudness_info on L(M1): loudness information of the rendered audio to the measured layout L(i).

• Where L(i) is the measured layout for the ith loudness information and i = 1, 2, ..., M1.

• This Mix Presentation is authored by using the highest loudness_layout .

For Mix Presentation for N (>1) audio elements (when num_sub-mixes = 1), Mix Presentation OBU follows following restrictions:

• num_sub_mixes : set to 1

• num_audio_elements : set to N

• element_mix_config() for each audio element: set to mix_gain

• output_mix_config() : set to output_mix_gain

• num_layouts : set to M2, the number of loudness informations which are provided.

• loudness_layout : set to L(1), L(2), ..., L(M2).

• loudness_info() on L(1), loudness_info on L(2), ..., loudness_info on L(M2): loudness information of the rendered audio to the measured layout L(i).

• Where L(i) is the measured layout for the ith loudness information and i = 1, 2, ..., M2.

• This Mix Presentation is authored by using the highest loudness_layout .

7.3.1. Element Mix Config

This section provide a guideline to generate element_mix_config().

An IA multiplexer may merge two or more IA sequences. In this case, it should adjust the gain values for element_mix_config() s as necessary to describe the desired relative gains between the IA sequences when they are summed to generate the final mix. It should also ensure that the gains selected do not result in clipping when the final mix is generated.

7.4. Multiple Audio Elements Encoding

This section provide a way to generate IA sequence having multiple audio elements from two IA simple or base profiles.

7.4.1. Multiple Audio Elements with One Codec Config

This section provides a way how to generate IA sequence having multiple audio elements from two IA simple profiles with the same codec config OBU. However, the result complies with the base profile of IA sequence.

Step1: Descriptor OBUs are generated as follows:

• IA Sequence Header OBU: get the larger profile_name field and the larger profile_compatible field, respectively.

• Codec Config OBU

  • take just one codec_id and codec_config()

• Audio Element OBUs: just take all of them except followings:

  • codec_config_id in each Audio Element OBU is updated to indicate codec_config_id specified in the taken Codec Config OBU.

  • audio_element_id s are updated to be unique in all of Audio Element OBUs.

  • audio_substream_id s are updated to be unique in all of Audio Element OBUs.

  • parameter_ids in ParamDefinition() s carried in each Audio Element OBU are updated to refer its associated parameters correctly.

• Mix Presentation OBUs: just take all of them and generate new ones which are used for mixing of multiple audio elements if needed except following:

  • audio_element_ids in each Mix Presentation OBU are updated to indicate the audio_element_id s specified in Audio Element OBUs.

  • parameter_ids in ParamDefinition() s carried in each Mix Presentation OBU are updated to refer its associated parameters correctly.

Step2: ith temporal unit is generated as follows:

• Just take all of temporal units for ith frames from each audio element and keep the order of temporal units as the order of audio element OBUs in descriptor OBUs except following:

  • obu_type s are updated to be aligned according to audio_substream_id s specified in Audio Element OBUs.

  • parameter_id s in Parameter Block OBUs are updated to uniquely indentify the parameter in IA sequence and based on the parameter_ids in ParamDefinition() s carried in Descriptor OBUs.

• It may have the immediately preceding temporal delimiter OBU for each temporal unit.

Step3: Generate IA sequence which starts descriptor OBUs and followed by temporal units in order.

7.5. Post Processing

This section provides a way to generate algorithms for post processing.

7.5.1. Loudness Information

This section provides a way to generate loudness_info().

For a given Mix Presentation OBU and a given loudness_layout of a sub-mix of the given Mix Presentation OBU, the followings are processed in order to produce loudness_info().

• Each of the Audio Elements specified in the sub-mix of the given Mix Presentation OBU is rendered to the given loudness_layout according to rendering_config() for the Audio Element.

• Each of the Audio Elements specified in the sub-mix of the given Mix Presentation OBU applies mix_gain according to element_mix_config() for the Audio Element.

• All of the Audio Elements specified in the sub-mix of the given Mix Presentation OBU are summed, and then applies mix_gain according to output_mix_config() for the sub-mix.

• Generate loudness_info() of the mixed audio according to § 2.8.7 Loudness Info Syntax and Semantics .

8. Convention

8.1. Syntax Description

All of syntax elements conform to Syntactic Description Language specified in [MP4-Systems] unless it is explicitly described in the specification.

8.1.1. Data types

leb128() syntaxName

leb128() indicates the type of an unsigned integer. It indicates the following unsigned integer syntaxName is encoded by leb128() specified in [AV1-Convention] .

syntaxName is an unsigned integer which is encoded by leb128() specified in [AV1-Convention] .

sleb128() syntaxName

sleb128() indicates the type of an signed integer. It indicates the following signed integer syntaxName is encoded by leb128() specified in [AV1-Convention] .

syntaxName is an signed integer which is encoded by leb128() specified in [AV1-Convention] .

string syntaxName

string indicates a null-terminated (i.e. ending at the first byte set to 0x00), UTF-8 encoded as defined in [RFC3629] and whose length shall be limited to 128 bytes.

syntaxName is a human readable label. The label shall not include the number 0.

8.2. Arithmetic Operators

8.3. Function

8.3.1. Function templates

When the template keyword is used to decorate the class declaration, it indicates that the code is a template with a placeholder type that can be reused by other classes. Only classes that use the template present in the bitstream; the template itself does not present in the bitstream. Classes that use a function template pass a data type that is specified in either [MP4-Systems] or § 8.1.1 Data types .

Example

template <class T>
class Foo {
  T t;
}
class Bar {
  Foo<int> f;
}

8.3.2. Mathematical functions

Clip3(x, y, z)

It conforms to Clip3 specified in [AV1-Convention] .

round(x)

The round() function returns the integer value closest to x and may be implemented as

round(x) = floor(x + 0.5).

MOD(Number, Divisor)

The MOD() function returns the remainder after Number is divided by Divisor .

9. Annex

9.1. Annex A: ID Linking Scheme (Informative)

The below figure shows the linking scheme among IDs in obu_header or obu payload.

In the above figure,

• Codec Config OBU with codec_config_id = 0 is providing codec_id and its decoder_config(). decoder_config() .

• Mix Presentation OBU with mix_presentation_id = 21 is saying:

  • There are two Audio Elements(audio_element_id Element s( audio_element_id = 11 and 12) which need to be mixed. The audio_element_id = 11 and the audio_element_id = 12 are linked to the Audio Element OBUs OBU s with audio_element_id = 11 and audio_element_id = 12, respectively.

    • There is one or more are Parameter Block OBUs OBU s with parameter_id = 32 to be used for mixing of the Audio Element with audio_element_id = 11.

    • There is one or more are Parameter Block OBUs OBU s with parameter_id = 33 to be used for mixing of the Audio Element with audio_element_id = 12.

  • There is one or more are Parameter Block OBUs OBU s with parameter_id = 34 to be used for mixing of the two Audio Elements. Element s.

• Audio Element OBU with audio_element_id = 11 is saying:

  • This audio element Audio Element has been coded using Codec Config OBU with codec_config_id = 0.

  • There are two Substreams (audio_substream_id Audio Substream s ( audio_substream_id = 0 and 1) in this audio element. Audio Element . The audio_substream_id = 0 and the audio_substream_id = 1 are linked to the Audio Frame OBUs OBU s with audio_substream_id = 0 and audio_substream_id = 1(i.e. obu_type = OBU_IA_Audio_Frame_ID0 and obu_type = OBU_IA_Audio_Frame_ID1), respectively.

  • There is one or more are Parameter Block OBUs OBU s with parameter_id = 31 to be used for demixing of this audio element. Audio Element .

• Audio Element OBU with audio_element_id = 12 is saying:

  • This audio element Audio Element has been coded by using Codec Config OBU with codec_config_id = 0.

  • There is one Audio Substream (audio_substream_id ( audio_substream_id = 2) in this audio element. Audio Element . The audio_substream_id = 2 is linked to the Audio Frame OBUs OBU s with audio_substream_id = 2 (i.e. obu_type = OBU_IA_Audio_Frame_ID2).

• Audio Frame OBU with ID audio_substream_id = 0 (i.e. obu_type = OBU_IA_Audio_Frame_ID0) is providing the coded data which has been coded by using Codec Config OBU with codec_config_id = 0 of Audio Substream with audio_substream_id = 0.

• Audio Frame OBU with ID audio_substream_id = 1 (i.e. obu_type = OBU_IA_Audio_Frame_ID1) is providing the coded data which has been coded by using Codec Config OBU with codec_config_id = 0 of Audio Substream with audio_substream_id = 1.

• Audio Frame OBU with ID audio_substream_id = 2 (i.e. obu_type = OBU_IA_Audio_Frame_ID2) is providing the coded data which has been coded by using Codec Config OBU with codec_config_id = 0 of Audio Substream with audio_substream_id = 2.

• Parameter Block OBU with parameter_id = 31 is providing demixing_info_parameter_data() to be applied for demixing of the Audio Element with audio_element_id = 11.

• Parameter Block OBU with parameter_id = 32 is providing mix_gain_parameter_data() to be applied to the rendered audio element Audio Element after rendering according to render_config() of the Audio Element with audio_element_id = 11.

• Parameter Block OBU with parameter_id = 33 is providing mix_gain_parameter_data() to be applied to the rendered audio element Audio Element after rendering according to render_config() of the Audio Element with audio_element_id = 12.

• Parameter Block OBU with parameter_id = 34 is providing mix_gain_parameter_data() to be applied to the mixed audio Mixed Audio Element of the two rendered audios. Audio Element s.

9.2. Annex B: Rules for Scalable Channel Audio (Normative)

This Annex specifies normative rules for scalable channel audio with mum_layers num_layers > 1.

9.2.1. Annex B-1: Down-mix parameter and Loudness

This section describes how to generate down-mix parameters and loudness level for a given channel audio and a given list of channel layouts for scalability. scalability (i.e. num_layers > 1).

Below figure shows a block diagram for down-mix parameter and loudness module including down-mixer.

For a given Channel Audio channel-based input audio (e.g. 7.1.4ch) and a given list of channel layouts based on the Channel Audio, input audio,

• Down-mix parameter generator shall SHALL generate 5 down-mix parameters (α(k), β(k), γ(k), δ(k) and w(k)) by analyzing the input Channel Audio, audio, by referring [AI-CAD-Mixing] . Where, k is a frame index.

  • It is composed of Audio Scene Classification module and Height Energy Quantification module as depicted in Figure 11-2.

  • Audio Scene Classification module generates 4 parameters (α(k), β(k), γ(k), δ(k)) by classifying audio scenes of the input channel audio in three modes.

    • Default scene: Neither Dialog nor Effect

    • Dialog scene: Center-channel oriented and clear dialog/voice sounds

    • Effect scene: Directional and spatially moving sounds.

  • Height Energy Quantification module generates a surround to height mixing parameter (w(k)) which is decided according to the relative energy difference between the top and surround channels of the input channel audio.

    • If the energy of top channels is bigger than that of surround ones, then w_idx_offset(k) is set to 1. Otherwise, it is set to -1. And, w(k) is calculated based on w_idx_offset(k) and conforms to § 6.2 Scalable Channel Audio decoding .

• Down-mixer generates down-mixed audios from the input Channel Audio audio according to the list of channel layouts and the down-mix parameters, and outputs down-mixed audio for each channel layout to Loudness module.

  • It is not depicted in the figure but Down-mixer further generates Dmixp_Mode dmixp_mode and Recon_Gains recon_gain for each frame to be passed to OBU packetizer.

• Loudness module measures the loudness level ( LKFS ) of each down-mixed audio based on [ITU1770-4] , and passes them to OBU packetizer.

9.2.2. Annex B-2: Down-mix Mechanism

This section specifies the down-mixing mechanism to generate down-mixed audio for scalable channel audio.

For a given Channel Audio channel-based input audio which conforms to loudspeaker_layout , the surround and top channels (if any) are separately down-mixed and especially step by step until to get a target channels.

Implementers may MAY use another method to get the down-mixed audio from the given channel input audio, but the down-mixed audio shall SHALL comply with that by this section.

Therefore, a down-mixer based on the down-mix mechanism is a combination of following surround down-mixer(s) and top down-mixer(s) as depicted in below figure.

• Surround Down-mixers: down-mixers

  • S7to5 enc., enc. : Ls5 = α(k) x Lss7 + β(k) x Lrs7 and Rs5 = α(k) x Rss7 + β(k) x Rrs7.

  • S5to3 enc., enc. : L3 = L5 + δ(k) x Ls5 and R3 = R5 + δ(k) x Rs5

  • S3to2 enc., enc. : L2 = L3 + 0.707 x C and R2 = R3 + 0.707 x C

  • S2to1 enc. : Mono = 0.5 x (L2 + R2)

  S7to5 enc.: Ls5 = α(k) x Lss7 + β(k) x Lrs7 and Rs5 = α(k) x Rss7 + β(k) x Rrs7. S5to3 enc.: L3 = L5 + δ(k) x Ls5 and R3 = R5 + δ(k) x Rs5 S3to2 enc.: L2 = L3 + 0.707 x C and R2 = R3 + 0.707 x C S2to1 enc.: Mono = 0.5 x (L2 + R2)

• Top Down-mixers: Down-mixers

  • T4to2 enc., enc. : Ltf2 = Ltf4 + γ(k) x Ltb4 and Rtf2 = Rtf4 + γ(k) x Rtb4.

  • T2toTF2 enc. : Ltf3 = Ltf2 + w(k) x δ(k) x Ls5 and Rtf3 = Rtf2 + w(k) x δ(k) x Rs5.

  T4to2 enc.: Ltf2 = Ltf4 + γ(k) x Ltb4 and Rtf2 = Rtf4 + γ(k) x Rtb4. T2toTF2 enc.: Ltf3 = Ltf2 + w(k) x δ(k) x Ls5 and Rtf3 = Rtf2 + w(k) x δ(k) x Rs5.

For example, to get down-mixed audio 3.1.2ch from 7.1.4ch:

• S3 of 3.1.2ch is generated by using S7to5 enc. and S5to3 enc. .

• TF2 of 3.1.2ch is generated by using T4to2 enc. and T2toTF2 enc. .

9.2.3. Annex B-3: Channel Layout Generation Rule

This section describes the generation rule for channel layouts for scalable channel audio.

For a given channel layout (CL #n) of an channel-based input Channel Audio, audio, any list of CLs ({CL #i: i = 1, 2, ..., n}) for a scalable channel audio shall SHALL conform with following rules:

• Si ≤ Si+1 and Wi ≤ Wi+1 and Ti ≤ Ti+1 except Si = Si+1 and Si+1, Wi = Wi+1 and Ti = Ti+1 for i = n-1, n-2, …, 1. Where the ith Channel Layout channel layout CL #i = Si.Wi.Ti.

• CL #i is one of loudspeaker_layouts loudspeaker_layout s supported in this version of the specification.

Down-mix paths, which conform to the above rule, shall SHALL be only allowed for scalable channel audio with num_layers > 1 as depicted in below figure.

9.2.4. Annex B-4: Recon Gain Generation

This section describes how to generate recon_gain .

NOTE: Recon gain generation is not required when the codec is lossless such as codec_id is set to ipcm or fLaC .

Recon_Gain Recon gain needs to be applied to de-mixed channels. For this, IA encoder needs to deliver it to IA decoders.

Let’s define followings:

• Level Ok is the signal power for the frame #k of a channel of the down-mixed audio for CL #i.

• Level Mk is the signal power for the frame #k of the relevant mixed channel of the down-mixed audio for CL #i-1.

• Level Dk is the signal power for the frame #k of the de-mixed channel for CL #i (after demixing). demixing in decode side).

If 10*log10(level Ok / maxL^2) is less than the first threshold value (e.g. -80dB), Recon_Gain (k, i) = 0. Where, maxL = 32767 for 16bits.

If 10*log10(level Ok / level Mk ) is less than the second threshold value (e.g. -6dB), Recon_Gain (k, i) is set to the value which makes level Ok = Recon_Gain (k, i)^2 x level Dk. Otherwise, Recon_Gain (k, i) = 1. Actual value (i.e. recon_gain ) to be delivered is floor(255*Recon_Gain).

For example, if we assume CL #i = 7.1.4ch and CL #i-1 = 5.1.2ch, then de-mixed channels are D_Lrs7, D_Rrs7, D_Ltb4 and D_Rtb4.

• D_Lrs7 and D_Rrs7 are de-mixed from Ls5 and Rs5 in the (i-1)th ChannelGroup by using Lss7 and Rss7 in the ith ChannelGroup and its relevant demixing parameters (i.e., α(k) and β(k)) , respectively.

• D_Ltb4 and D_Rtb4 are de-mixed from Ltf2 and Rtf2 in the (i-1)th ChannelGroup by using Ltf4 and Rtf4 in the ith ChannelGroup and its relevant demixing parameter (i.e., γ(k)), respectively.

Recon_Gain for D_Lrs7:

• Level Ok is the signal power for the frame #k of Lrs7 in the ith ChannelGroup .

• Level Mk is the signal power for the frame #k of Ls5 in the (i-1)th ChannelGroup .

• Level Dk is the signal power for the frame #k of D_Lrs7.

Recon_Gain for D_Rrs7:

• Level Ok is the signal power for the frame #k of Rrs7 in the ith ChannelGroup .

• Level Mk is the signal power for the frame #k of Rs5 in the (i-1)th ChannelGroup .

• Level Dk is the signal power for the frame #k of D_Rrs7.

Recon_Gain for D_Ltb4:

• Level Ok is the signal power for the frame #k of Ltf4 in the ith ChannelGroup .

• Level Mk is the signal power for the frame #k of Ltf2 in the (i-1)th ChannelGroup .

• Level Dk is the signal power for the frame #k of D_Ltb4.

Recon_Gain for D_Rtb4:

• Level Ok is the signal power for the frame #k of Rtf4 in the ith ChannelGroup .

• Level Mk is the signal power for the frame #k of Rtf2 in the (i-1)th ChannelGroup .

• Level Dk is the signal power for the frame #k of D_Rtb4.

9.2.5. Annex B-5: ChannelGroup Generation Rule

This section describes the generation rule for ChannelGroup. ChannelGroup .

For a given Channel Audio channel-based input audio and the list of CLs ({CL #i: i = 1, 2, ..., n}), CG Generation module outputs the transformed audio (i.e. ChannelGroups) which shall SHALL conform to following rules:

• It consists of C number of channels and is structured to n number of CGs, ChannelGroup s, where C is the number of channels for the Channel Audio. input audio.

• CG ChannelGroup #1 (as called BCG): This CG ChannelGroup is the down-mixed audio itself for CL #1 generated from the Channel Audio. input audio. It contains C1 number of channels.

• CG ChannelGroup #i (as called DCG, i = 2, 3, …, n): This CG ChannelGroup contains (Ci – Ci-1) number of channels. (Ci – Ci-1) channel(s) consists of as follows:

  • (Si – Si-1) surround channel(s) if Si > Si-1 . When S_set = { x | Si-1 < x ≤ Si and x is an integer},

    • If 2 is an element of S_set, the L2 channel is contained in this CG #i.

    • If 3 is an element of S_set, the Center channel is contained in this CG #i.

    • If 5 is an element of S_set, the L5 and R5 channels are contained in this CG #i.

    • If 7 is an element of S_set, the Lss7 and Rss7 channels are contained in this CG #i.

  • The LFE channel if Wi > Wi-1 .

  • (Ti – Ti-1) top channels if Ti > Ti-1 .

    • If Ti-1 = 0, the top channels of the down-mixed audio for CL #i are contained in this CG ChannelGroup #i.

    • If Ti-1 = 2, the Ltf and Rtf channels of the down-mixed audio for CL #i are contained in this CG ChannelGroup #i.

Below figure shows one example of transformation matrix with 4 CGs (2ch/3.1.2ch/5.1.2ch/7.1.4ch).

9.3. Annex C: Consumption of IAMF bitstream (informative)

TODO. Fill in example workflows.