Copyright © 2019 W3C ® ( MIT , ERCIM , Keio , Beihang ). W3C liability , trademark and permissive document license rules apply.
Distributed tracing is a set of tools and practices to monitor the health and reliability of a distributed application. A distributed application is an application that consists of multiple components that are deployed and operated separately. It is also known as micro-service.
The main concept behind distributed tracing is event correlation. Event correlation is a way to correlate events from one component to the events from another. It allows to find the cause-and-effect relationship between these events. For instance – find which user action in a browser caused a failure in the business logic layer.
To correlate events between components, these components need to exchange and store a piece of information called context. Typically context consists of an originating event identifier, an originating component identity and other event properties. Context has two parts. The first part is a trace context. Trace context consists of properties crucial for event correlation. The second part is correlation context. Correlation context carries user-defined properties. These properties may be helpful for correlation scenarios. But they are not required and components may choose to not carry or store them.
Unifying the format of distributed tracing context as well as aligning on semantic meaning of the values is the main objective of this working group. The goal is to share this with the community so that various tracing and diagnostics products can operate together.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at https://www.w3.org/TR/.
This is the First Public Working Draft ( FPWD ). There are a few implementations of this protocol available. Experimental interoperability scenarios were run and have demonstrated promising results. The specification will be progressed into Candidate Recommendation stage after that, drafts for binary, AMQP and MQTT protocols will be written to make sure the concepts and structure defined in this specification can be ported to other protocols.
This document was published by the Distributed Tracing Working Group as an Editor's Draft.
GitHub
Issues
are
preferred
for
discussion
of
this
specification.
Alternatively,
you
can
send
comments
to
our
mailing
list.
Please
send
them
to
public-trace-context@w3.org
(
archives
)
with
trace-context
at
the
start
of
your
email's
subject
.
Publication as an Editor's Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document was produced by a group operating under the W3C Patent Policy . W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy .
This document is governed by the 1 February 2018 W3C Process Document .
Trace
context
consists
of
a
set
of
properties
crucial
for
event
correlation
in
distributed
applications.
Trace
context
positions
requests
in
a
distributed
trace.
If
you
look
at
a
distributed
trace
as
a
graph
of
correlated
requests
-
a
single
request
can
be
part
of
multiple
graphs.
For
example,
let's
say
component
A
calls
component
B
which
in
turn
calls
C
.
One
distributed
tracing
system
may
see
component
A
calling
component
B
.
Another
may
see
all
three
components
communicating.
So
the
call
from
B
to
C
will
carry
trace
context
about
service
A
as
well
as
information
about
its
direct
communication
upstream
to
B
.
The situation when a single request is a part of multiple graphs is becoming more common. One distributed application may have components monitored by different distributed tracing systems. Those distributed tracing systems may not be easily replaceable as they might be provided by cloud vendors or distributed as pre-built components.
The trace context is designed to allow extensibility for all distributed tracing systems. It requires them to respect context set by other systems.
Trace context is represented by a set of name/value pairs describing the identity of every http request. Two propagation fields carry the common and vendor-specific properties that make up the trace context.
traceparent
describes
the
position
of
the
incoming
request
in
its
trace
graph
in
a
portable,
fixed-length
format.
Its
design
focuses
on
fast
parsing.
tracestate
maps
all
graphs
the
incoming
parent
is
a
part
of
in
potentially
vendor-specific
formats.
For
example,
if
a
request
crosses
tracing
systems,
there
will
be
one
entry
in
tracestate
for
each
system.
Notably,
the
tracestate
field
is
unreliant
on
data
in
the
traceparent
.
Libraries
and
platforms
MUST
propagate
traceparent
and
tracestate
headers
to
guarantee
that
trace
will
not
be
broken.
This
section
describes
the
binding
of
the
distributed
trace
context
to
The
For
example,
a
client
traced
in
the
congo
system
adds
the
following
headers
to
an outbound http request.
If
the
receiving
server
is
traced
in
the
You'll
notice
that
the
If
the
receiving
server
of
the
above
is
Notice
when
Finally,
you'll
see
Field
In
order
to
increase
interoperability
across
multiple
protocols
and
encourage
successful
integration
by
default
it
is
recommended
to
keep
the
header
name
lower
case.
Header
name
is
a
single
word
without
any
delimiters
like
hyphen
(
Header
name:
Platforms
and
libraries
MUST
expect
header
name
in
any
casing
and
SHOULD
send
header
name
in
lower
case.
This
section
uses
the
Augmented
Backus-Naur
Form
(ABNF)
notation
of
RFC5234
,
including
the
DIGIT
rule
from
that document.
The
value
is
US-ASCII
encoded
(which
is
UTF-8
compliant).
Character
Version
(
The following
Is
the
ID
of
the
whole
trace
forest.
It
is
represented
as
a
16-bytes
array,
for
example,
When
a
system
operates
with
a
shorter
Implementations
HAVE
TO
ignore
the
Is
the
ID
of
this
call
as
known
by
the
caller.
It
is
also
known
as
Implementation
HAVE
TO
ignore
the
An
8-bit
field
that
controls
tracing
flags
such
as
sampling,
trace
level
etc.
These
flags
are
recommendations
given
by
the
caller
rather
than
strict
rules
to
follow
for
three
reasons:
You
can
find
more
in
security
section
of
this
specification.
Like
other
fields,
As
this
is
a
bit
field,
you
cannot
interpret
flags
by
decoding
the
hex
value
and
looking
at
the
resulting
number.
For
example,
a
flag
Here is an example of properly handing trace flags:
Current
version
of
specification
only
supports
a
single
flag
called
When
set,
the
least
significant
bit
documents
that
the
caller
may
have
recorded
trace
data.
A
caller
who
does
not
record
trace
data
out-of-band
leaves
this
flag
unset.
Many
distributed
tracing
scenarios
may
be
broken
when
only
a
subset
of
calls
participated
in
a
distributed
trace
were
recorded.
At
certain
load
recording
information
about
every
incoming
and
outgoing
request
becomes
prohibitively
expensive.
Making
a
random
or
component-specific
decision
for
data
collection
will
lead
to
fragmented
data
in
every
distributed
trace.
Thus
it
is
typical
for
tracing
vendors
and
platforms
to
pass
recording
decision
for
given
distributed
trace
or
information
needed
to
make
this
decision.
There
is
no
consensus
on
what
is
the
best
algorithm
to
make
a
recording
decision.
Various
techniques
include:
probability
sampling
(sample
1
out
of
100
distributed
traced
by
flipping
a
coin),
delayed
decision
(make
collection
decision
based
on
duration
or
a
result
of
a
call),
deferred
sampling
(let
callee
decide
whether
information
about
this
request
need
to
be
collected).
There
are
variations
and
customizations
of
every
technique
which
can
be
tracing
vendor
specific
or
application
defined.
Field
For
example,
when
SaaS
services
participate
in
distributed
trace
-
this
service
has
no
knowledge
of
tracing
system
used
by
its
caller.
But
this
service
may
produce
records
of
incoming
requests
for
monitoring
or
troubleshooting
purposes.
Flag
Flag
The
behavior
of
other
flags,
such
as
(
Valid traceparent when caller recorded this request:
Valid
traceparent
when
caller
haven't
recorded
this
request:
Implementation
is
opinionated
about
future
version
of
the
specification.
Current
version
of
this
specification
assumes
that
the
future
versions
of
Implementation
should
follow
the
following
rules
when
parsing
headers
with
an
unexpected
format:
The
In
order
to
increase
interoperability
across
multiple
protocols
and
encourage
successful
integration
by
default
it
is
recommended
to
keep
the
header
name
lower
case.
Header
name
is
a
single
word
without
any
delimiters
like
hyphen
(
Header
name:
Platforms
and
libraries
MUST
expect
header
name
in
any
casing
and
SHOULD
send
header
name
in
lower
case.
Multiple
This
section
uses
the
Augmented
Backus-Naur
Form
(ABNF)
notation
of
RFC5234
,
including
the
DIGIT
rule
in
appendix
B.1
for
RFC5234
.
It
also
includes
the
The
The
Empty
and
whitespace-only
list
members
are
allowed.
Libraries
and
platforms
MUST
accept
empty
A
simple
example
of
a
Identifiers
are
short
(up
to
256
characters)
textual
identifiers.
Note
that
identifiers
MUST
begin
with
a
lowercase
letter,
and
can
only
contain
lowercase
letters
Value
is
opaque
string
up
to
256
characters
printable
ASCII
RFC0020
characters
(i.e.,
the
range
0x20
to
0x7E)
except
comma
The
length
of
a
combined
header
MUST
be
less
than
or
equal
to
512
bytes.
If
the
length
of
a
combined
header
is
more
than
512
bytes
it
SHOULD
be
ignored.
Example:
The
value
of
a
concatenation
of
trace
graph
key-value
pairs.
Only
one
entry
per
key
is
allowed
because
the
entry
represents
that
last
position
in
the
trace.
Hence
implementors
must
overwrite
their
entry
upon
reentry
to
their
tracing
system.
For
example,
if
tracing
system
name
is
Rather,
the
entry
would
be
rewritten
to
only
include
the
most
recent
position:
Limits:
There
might
be
multiple
If
the
Single
tracing
system
(generic
format):
Multiple
tracing
systems
(with
different
formatting):
Version
of
Library
or
platform
receiving
If
the
value
of
the
Here
is
the
list
of
allowed
mutations:
Libraries
and
platforms
MUST
NOT
make
any
other
mutations
to
the
Library
or
platform
receiving
Here
is
the
list
of
allowed
mutations:
traceparent
and
tracestate
http
headers.
2.1
Relationship
between
the
headers
traceparent
header
represents
the
incoming
request
in
a
tracing
system
in
a
common
format.
The
tracestate
header
includes
the
parent
in
a
potentially
vendor-specific
format.
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate
:
congo=BleGNlZWRzIHRohbCBwbGVhc3VyZS4
rojo
tracing
system,
it
carries
over
the state it received and adds a new entry with the position in its trace.traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01
tracestate
:
rojo=00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01,congo=BleGNlZWRzIHRohbCBwbGVhc3VyZS4
rojo
system
reuses
the
value
of
traceparent
in
its
entry
in
tracestate
.
This
means
it
is
a
generic
tracing
system.
Otherwise,
tracestate
entries
are
opaque.
congo
again,
it
continues
from
its
last position, overwriting its entry with one representing the new parent.traceparent: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
tracestate
:
congo=lZWRzIHRoNhcm5hbCBwbGVhc3VyZS4,rojo=00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01
congo
wrote
its
traceparent
entry,
it
reuses
the
last
trace
ID
which
helps
in
consistency
for
those
doing
correlation.
However,
the
value
of
its
entry
tracestate
is
opaque
and
different.
This
is
ok.
tracestate
retains
an
entry
for
rojo
exactly
as
it
was,
except
pushed
to
the
right.
The
left-most
position
lets
the
next
server
know
which
tracing
system
corresponds
with
traceparent
.
In
this
case,
since
congo
wrote
traceparent
,
its
tracestate
entry
should
be
left-most.
2.2
Traceparent
field
traceparent
identifies
the
request
in
a
tracing
system.
2.2.1
Header
name
-
).
traceparent
2.2.2
Field
value
DIGIT rule defines a single number character 0-9.HEXDIGLC = DIGIT / "a" / "b" / "c" / "d" / "e" / "f" ; lower case hex character
value = version "-" version-format
version
=
2
HEXDIGLC
;
this
document
assumes
version
00.
Version
255
is
forbidden
-
is
used
as
a
delimiter
between
fields.
version
)
is
a
1
byte
representing
an
8-bit
unsigned
integer.
Version
255 is invalid. Current specification assumes the version is set to 00.version-format definition is used for version 00.version-format = trace-id "-" parent-id "-" trace-flags
trace-id = 32HEXDIGLC ; 16 bytes array identifier. All zeroes forbidden
parent-id = 16HEXDIGLC ; 8 bytes array identifier. All zeroes forbidden
trace-flags
=
2
HEXDIGLC
;
8
bit
flags.
Currently
only
one
bit
is
used.
See
below
for
details
2.2.2.1
Trace-id
4bf92f3577b34da6a3ce929d0e0e4736
.
All
bytes
0
are
considered
invalid.
Trace-id
is
used
to
uniquely
identify
a
distributed
trace.
So
implementation
should
generate
globally
unique
values.
Many
algorithms
of
unique
identification
generation
are
based
on
some
constant
part
-
time
or
host
based
and
a
random
value.
There
are
systems
that
make
random
sampling
decisions
based
on
the
value
of
trace-id
.
So
to
increase
interoperability
it
is
recommended
to
keep
the
random
part
on
the
right
side
of
trace-id
value.
trace-id
-
it
is
recommended
to
fill-in
the
extra
bytes
with
random
values
rather
than
zeroes.
Let's
say
the
system
works
with
a
8-byte
trace-id
like
3ce929d0e0e4736
.
Instead
of
setting
trace-id
value
to
0000000000000003ce929d0e0e4736
it
is
recommended
to
generate
a
value
like
4bf92f3577b34da6a3ce929d0e0e4736
where
4bf92f3577b34da6a
is
a
random
value
or
a
function
of
time
&
host
value.
Note,
even
though
a
system
may
operate
with
a
shorter
trace-id
for
distributed
trace
reporting
-
full
trace-id
should
be
propagated
to
conform
to
the
specification.
traceparent
when
the
trace-id
is
invalid.
For
instance,
if
it
contains
non-allowed
characters.
2.2.2.2
Parent-id
span-id
as
a
few
telemetry
systems
call
the
execution
of
a
client
call
a
span.
It
is
represented
as
an
8-byte
array,
for
example,
00f067aa0ba902b7
.
All
bytes
0
is
considered
invalid.
traceparent
when
the
parent-id
is
invalid.
For
instance,
if
it
contains
non-allowed
characters.
2.2.3
Trace-flags
trace-flags
is
hex-encoded.
For
example,
all
8
flags
set
would
be
ff
and
no
flags
set
would
be
00
.
00000001
could
be
encoded
as
01
in
hex,
or
09
in
hex
if
present
with
the
flag
00001000
.
A
common
mistake in bit fields is forgetting to mask when interpreting flags.static final byte FLAG_RECORDED = 1; // 00000001
...
boolean
recorded
=
(traceFlags
&
FLAG_RECORDED)
==
FLAG_RECORDED
recorded
.
2.2.3.1
Recorded
Flag
(00000001)
tracestate
is
designed
to
handle
the
variety
of
techniques
for
making
recording
decision
specific
(along
any
other
specific
information)
for
a
given
tracing
system
or
a
platform.
Flag
recorded
is
introduced
for
better
interoperability
between
vendors.
It
allows
to
communicate
recording
decision
and
enable
better
experience
for
the
customer.
recorded
can
be
used
to
ensure
that
information
about
requests
that
were
marked
for
recording
by
caller
will
also
be
recorded
by
SaaS
service.
So
caller
can
troubleshoot
the
behavior
of
every
recorded
request.
recorded
has
no
restriction
on
its
mutations
except
that
it
can
only
be
mutated
when
parent-id
was
updated.
See
section
"Mutating
the
traceparent
field".
However
there
are
set
of
suggestions
that
will
increase
vendors
interoperability.
recorded
flag.
recorded
flag
value.
Security
considerations
should
be
applied
to
protect
from
abusive
or
malicious
use
of
this
flag
-
see
security
section.
recorded
should
be
propagated
unchanged.
And
set
to
0
as
a
default
option
when
trace
is
initiated
by
this
component.
There
are
two
additional
options:
recorded
flag
to
1
for
a
subset
of
requests.
recorded
to
1
for
the
subset
of
requests.
2.2.3.2
Other
Flags
00000100
)
is
not
defined
and
reserved
for future use. Implementation MUST set those to zero.2.2.4 Examples of HTTP headers
Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
base16(version) = 00
base16(trace-id) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(parent-id) = 00f067aa0ba902b7
base16(trace-flags)
=
01
//
recorded
Value
=
00
-4
bf92f3577b34da6a3ce929d0e0e4736
-00
f067aa0ba902b7
-00
base16(version)
=
00
base16(trace-id)
=
4
bf92f3577b34da6a3ce929d0e0e4736
base16(parent-id)
=
00
f067aa0ba902b7
base16(trace-flags)
=
00
//
not
recorded
2.2.5
Versioning
of
traceparent
traceparent
header
will
be
additive
to
the
current
one.
-
)),
implementation
should
restart
the
trace.
trace-id
:
from
the
first
dash
-
next
32
characters.
Implementation
MUST
check
32
characters
to
be
hex.
Make
sure
they
are
followed
by
dash.
parent-id
:
from
the
second
dash
at
35th
position
-
16
characters.
Implementation
MUST
check
16
characters
to
be
hex.
Make
sure
this
is
followed
by
a
dash.
flags
:
2
characters
from
third
dash.
Following
with
either
end
of
string
or
a
dash.
If
all
three
values
were
parsed
successfully
-
implementation
should
use
them.
Implementation
MUST
NOT
parse
or
assume
anything
about
any
fields
unknown
for
this
version.
Implementation
MUST
use
these
fields
to
construct
the
new
traceparent
field
according
to
the
highest
version
of
the
specification
known
to
the
implementation
(in
this
specification
it
is
00
).
2.3
Tracestate
field
tracestate
HTTP
header
field
conveys
information
about
request
position
in
multiple
distributed
tracing
graphs.
This
header
is
a
companion
header
for
the
traceparent
.
If
library
or
platform
failed
to
parse
traceparent
-
it
MUST
NOT
attempt
to
parse
the
tracestate
.
Note,
that
opposite
it
not
true
-
failure
to
parse
tracestate
MUST
NOT
affect
the
parsing
of
traceparent
.
2.3.1
Header
name
-
).
tracestate
2.3.2
Header
value
tracestate
headers
are
allowed.
Values
from
multiple
headers
in
incoming
requests
SHOULD
be
combined
in
a
single
header
according
to
the
RFC7230
and
send
as
a
single
header
in
outgoing
request.
OWS
rule
from
RFC7230
section
3.2.3
.
DIGIT
rule
defines
number
0
-
9
.
OWS
rule
defines
an
optional
whitespace.
It
is
used
where
zero
or
more
whitespace
characters
might
appear.
When
it
is
preferred
to
improve
readability
-
a
sender
SHOULD
generate
the
optional
whitespace
as
a
single
space;
otherwise,
a
sender
SHOULD
NOT
generate
optional
whitespace.
See
details
in
corresponding
RFC.
tracestate
field
value
is
a
list
as
defined
below.
The
list
is
a
series
of
list-members
separated
by
commas
,
,
and
a
list-member
is
a
key/value
pair
separated
by
an
equals
sign
=
.
Spaces
and
horizontal
tabs
surrounding
list-member
s
are
ignored.
There
can
be
a
maximum
of
32
list-member
s
in
a
list
.
tracestate
headers,
but
SHOULD
avoid
sending
them.
The
reason
for
allowing
of
empty
list
members
in
tracestate
is
a
difficulty
for
implementor
to
recognize
the
empty
value
when
multiple
tracestate
headers
were
sent.
Whitespace
characters
are
allowed
for
a
similar
reason
as
some
frameworks
will
inject
whitespace
after
,
separator
automatically
even
in
case
of
an
empty
header.
list
with
two
list-member
s
might
look
like:
vendorname1=opaqueValue1,vendorname2=opaqueValue2.list = list-member 0*31( OWS "," OWS list-member )
list-member = key "=" value
list-member
=
OWS
key
=
lcalpha
0
*
255
(
lcalpha
/
DIGIT
/
"_"
/
"-"
/
"*"
/
"/"
)
key
=
lcalpha
0
*
240
(
lcalpha
/
DIGIT
/
"_"
/
"-"
/
"*"
/
"/"
)
"@"
lcalpha
0
*
13
(
lcalpha
/
DIGIT
/
"_"
/
"-"
/
"*"
/
"/"
)
lcalpha
=
%x61
-7
A
;
a-z
a
-
z
,
digits
0
-
9
,
underscores
_
,
dashes
-
,
asterisks
*
,
and
forward
slashes
/
.
For
multi-tenant
vendors
scenarios
@
sign
can
be
used
to
prefix
vendor
name.
Suggested
use
is
to
allow
set
tenant
id
in
the
beginning
of
key
like
fw529a3039@dt
-
fw529a3039
is
a
tenant
id
and
@dt
is
a
vendor
name.
Searching
for
@dt=
would
be
more
robust
for
parsing
(searching
for
all
vendor's
keys).
,
and
=
.
Note
that
this
also
excludes
tabs,
newlines, carriage returns, etc.value = 0*255(chr) nblk-chr
nblk-chr = %x21-2B / %x2D-3C / %x3E-7E
chr
=
%x20
/
nblk-chr
vendorname1=opaqueValue1,vendorname2=opaqueValue2
congo
,
and
a
trace
started
in
their
system,
went
through
a
system
named
rojo
and
later
returned
to
congo
,
the
tracestate
value
would
not
be:
congo=congosFirstPosition,rojo=rojosFirstPosition,congo=congosSecondPosition
congo=congosSecondPosition,rojo=rojosFirstPosition
tracestate
headers
in
a
single
request
according
to
RFC7230
section
3.2.2
.
Maximum
length
of
a
combined
header
MUST
be
less
than
512
characters.
This
length
includes
commas
required
to
separate
list
items.
But
SHOULD
NOT
include
optional
white
space
(OWA)
characters.
tracestate
field
contains
essential
information
for
request
correlation.
Platforms
and
tracing
systems
MUST
propagate
this
header.
Compliance
with
the
specification
will
require
storing
of
tracestate
as
part
of
the
request
payload
or
associated
metadata.
Allowing
the
long
field
values
can
make
compliance
to
the
specification
impossible.
Thus,
the
aggressive
limit
of
512
characters
was
chosen.
tracestate
value
has
more
than
512
characters,
the
tracer
CAN
decide
to
forward
the
tracestate
.
When
propagating
tracestate
with
the
excessive
length
-
the
assumption
SHOULD
be
that
the
receiver
will
drop
this
header.
2.3.3
Examples
of
HTTP
headers
tracestate
:
rojo=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate
:
rojo=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01,congo=lZWRzIHRoNhcm5hbCBwbGVhc3VyZS4
2.3.4
Versioning
of
tracestate
tracestate
is
defined
by
the
version
prefix
of
traceparent
header.
Implementation
needs
to
attempt
parsing
of
tracestate
if
a
higher
version
is
detected
to
the
best
of
its
ability.
It
is
the
implementor's
decision
whether
to
use
partially-parsed
tracestate
key-value
pairs
or
not.
2.4
Mutating
the
traceparent
field
2.4.1
Base
mutations
traceparent
request
header
MUST
send
it
to
outgoing
requests.
It
MAY
mutate
the
value
of
this
header
before
passing
to
outgoing
requests.
traceparent
field
wasn't
changed
before
propagation
-
tracestate
MUST
NOT
be
modified
as
well.
Unmodified
headers
propagation
is
typically
implemented
in
pass-thru
services
like
proxies.
This
behavior
may
also
be
implemented
in
a
service
which
currently
does
not
collect
distributed
tracing
information.
parent-id
.
The
value
of
property
parent-id
can
be
set
to
the
new
value
representing
the
ID
of
the
current
operation.
This
is
the
most
typical
mutation
and
should
be
considered
a
default.
recorded
flag
of
trace-flags
may
be
set
to
1
if
it
had
value
0
before
or
vice
versa.
parent-id
MUST
be
set
to
the
new
value
with
the
recorded
flag
update.
See
details
of
recorded
flag
for
more
information
on
how
this
flag
is
recommended
to
be
used.
recorded
.
The
value
of
recorded
reflects
the
caller's
recording
behavior:
either
the
trace
data
were
dropped
or
may
have
been
recorded
out-of-band.
This
mutation
gives
the
downstream
tracer
information
about
the
likelihood
its
parent's
information
was
recorded.
trace-id
,
parent-id
,
trace-flags
are
regenerated.
This
mutation
is
used
in
the
services
defined
as
a
front
gate
into
secure
networks
and
eliminates
a
potential
denial
of
service
attack
surface.
traceparent
header.
2.5
Mutating
the
tracestate
field
tracestate
request
header
MUST
send
it
to
outgoing
requests.
It
MAY
mutate
the
value
of
this
header
before
passing
to
outgoing
requests.
The
main
concept
of
tracestate
mutations
is
that
the
order
of
unmodified
key-value
pairs
MUST
be
preserved.
Modified
keys
MUST
be
moved
to
the
beginning
of
the
list.
tracestate
keys
for
privacy
and
security
concerns.
The
second
scenario
is
a
truncation
of
long
tracestate
's.
This section is non-normative.
Trace
Context
propagation
is
crucial
to
enable
distributed
tracing
scenarios
for
applications
spans
multiple
components.
Http
is
one
of
the
communication
protocols
used
for
cross
components
correlation.
Extensions
of
this
specification
as
well
as
specifications
defined
by
external
organizations
define
the
format
of
trace
context
serialization
and
deserialization
for
other
protocols.
Note,
that
those
extensions
may
be
at
a
different
maturity
level
than
this
specification.
AMQP
.
Binary
.
Please refer to the protocols registry for the details of trace context implementation for other protocols.
Requirements
to
propagate
headers
to
downstream
services
as
well
as
storing
values
of
these
headers
opens
up
potential
privacy
concerns.
Trace
vendors
MUST
NOT
use
traceparent
and
tracestate
fields
for
any
personally
identifiable
or
otherwise
sensitive
information.
The
only
purpose
of
these
fields
is
to
enable
telemetry
correlation.
Trace
vendors
MUST
assess
the
risk
of
header
abuse.
This
section
provides
some
considerations
and
initial
assessment
of
the
risk
associated
with
storing
and
propagating
these
headers.
Trace
vendors
may
choose
to
inspect
and
remove
sensitive
information
from
the
fields
before
allowing
the
platform
or
tracing
system
to
execute
code
that
potentially
can
propagate
or
store
these
fields.
All
mutations
should,
however,
conform
to
the
list
of
mutations
defined
in
this
specification.
Another
privacy
risk
of
the
Note,
both
privacy
concerns
of
The
field
Platforms
and
tracing
systems
MUST
NOT
include
any
personally
identifiable
information
in
the
Platforms
and
tracing
systems
extremely
sensitive
to
personal
information
exposure
MAY
implement
selective
removal
of
values
corresponding
to
the
unknown
keys.
This
mutation
of
the
4.1
Privacy
of
traceparent
field
traceparent
field
has
a
predefined
set
of
values.
These
values
are
randomly-generated
numbers.
If
a
random
number
generator
has
any
logic
of
using
user-identifiable
information
like
IP
address
-
this
information
may
be
exposed.
Random
number
generators
MUST
NOT
rely
on
any
information
that
can
potentially
be
user-identifiable.
traceparent
field
is
an
ability
to
correlate
calls
made
as
part
of
a
single
transaction.
A
downstream
service
may
track
and
correlate
two
or
more
calls
made
in
a
single
transaction
and
make
assumptions
about
the
identity
of
the
caller
of
one
call
base
on
information
in
another
call.
Service
initiating
calls
MAY
choose
to
restart
trace
while
making
calls
that
might
identify
caller
in
the
downstream
service.
traceparent
field
are
theoretical
rather
than
practical.
4.2
Privacy
of
tracestate
field
tracestate
may
contain
any
opaque
value
in
any
of
the
keys.
The
main
purpose
of
this
header
is
to
provide
additional
information
about
the
position
of
request
in
the
multiple
distributed
tracing
graphs.
tracestate
header.
tracestate
field
is
not
forbidden,
but
highly
discouraged.
As
it
defeats
the
purpose
of
this
field
for
allowing
multiple
tracing
systems
to
collaborate.
There are two types of potential security risks associated with this specification: information exposure and denial of service attacks against the tracing system.
Services
and
platforms
relying
on
As
mentioned
in
the
privacy
section,
information
in
Application
owners
should
either
ensure
that
no
proprietary
or
confidential
information
is
stored
in
the
When
distributed
tracing
is
enabled
on
a
service
with
a
public
API
and
naively
continues
any
trace
with
the
Tracing
vendors
and
platforms
should
account
for
these
situations
and
make
sure
that
checks
and
balances
are
in
place
to
protect
denial
of
monitoring
by
malicious
or
badly
authored
callers.
One
examples
of
such
protection
may
be
different
tracing
behavior
for
authenticated
and
unauthenticated
requests.
Various
rate
limiters
for
data
recording
can
also
be
implemented.
Application
owners
need
to
make
sure
to
test
all
code
paths
leading
to
the
sending
of
traceparent
and
tracestate
headers
should
also
follow
all
the
best
practices
of
parsing
potentially
malicious
headers.
Including
checking
for
header
length
and
content
of
header
values.
These
practices
help
to
avoid
buffer
overflow
and
html
injection
attacks.
5.1
Information
exposure
traceparent
and
tracestate
headers
may
carry
information
that
can
be
considered
sensitive.
For
example,
traceparent
may
allow
one
call
to
be
correlated
to
the
data
sent
with
another
call.
tracestate
may
imply
the
version
of
monitoring
software
used
by
the
caller.
This
information
could
potentially
be
used
to
create
a
larger
attack.
tracestate
,
or
they
should
ensure
that
tracestate
isn't
present
in
requests
to
external
systems.
5.2
Denial
of
service
recorded
flag
set,
a
malicious
attacker
could
overwhelm
an
application
with
tracing
overhead,
forge
trace-id
collisions
that
make
monitoring
data
unusable,
or
run
up
your
tracing
bill
with
your
SaaS
tracing
vendor.
5.3
Other
risks
traceparent
and
tracestate
headers.
For
example,
in
single
page
browser
applications
it
is
typical
to
make
cross-origin
calls.
If
one
of
these
code
path
leads
to
the
sending
of
traceparent
and
tracestate
headers
-
cross-origin
calls
restricted
via
Access-Control-Allow-Headers
header
,
it
may
fail.