Abstract
The communities behind Decentralized Identifiers (DIDs) bring together a
diverse group of contributors who have decidedly different notions of exactly
what "decentralization" means.
Rather than attempting to resolve this potentially unresolvable question, we
propose a rubric — a scoring guide used to evaluate performance, a
product, or a project — that teaches how to evaluate a given DID Method
according to one's own requirements.
This rubric presents a set of criteria which an Evaluator can apply to
any DID Method based on the use cases most relevant to them. We avoid reducing
the Evaluation to a single number because the criteria tend to be multidimensional
and many of the possible responses are not necessarily good or bad. It is up
to the Evaluator to understand how each response in each criteria might
illuminate favorable or unfavorable consequences for their needs.
This rubric is a collection of criteria for creating Evaluation
Reports that assist people in choosing a DID Method.
While this rubric assists in the evaluation of many aspects of a
DID Method, it is not exhaustive.
This section is non-normative.
A rubric is a tool used in academia to communicate expectations and evaluate
performance. It consists of a set of criteria to be evaluated, possible
responses for each criteria, and a scoring guide explaining both how to
choose and interpret each response. The act of evaluating a rubric, which we
call an Evaluation, provides basis for self-evaluation, procurement decisions,
or even marketing points. Written records of
an evaluation, which we'll call an Evaluation Report, document how a
particular subject is measured against the criteria. For students, a rubric
helps to clarify how their work will be evaluated by others. For
Evaluators, a rubric provides a consistent framework for investigating
and documenting the performance of a subject against a particular set of
criteria.
We were inspired to develop a rubric for
decentralization when discussions about the requirements for decentralized
identifiers, aka DIDs, led to intractable disagreement. It became clear that
no single definition of "decentralized" would suffice for all of the
motivations that inspired developers of DID Methods to work on a new
decentralized approach to identity. Despite this challenge, two facts remained
clear:
-
the people invested in this work shared a common goal of reversing the
problems with centralized identity systems and
-
they also had numerous, distinct reasons for doing so.
Rather than attempt to force a definition of "decentralized" that might work
for many but would alienate others, the group set out to capture the
measurable factors that could enable Evaluators to judge the
decentralization of DID Methods based on their own independent requirements,
with minimal embedded bias.
Pick the most important criteria for your use, ask each question, and select the
most appropriate response. Do this for all of the DID
Methods under consideration.
Each Evaluation should start with an explicit framing of the use under
consideration. Are you evaluating the Method for use in Internet-of-Things
(IoT)? For school childrens' extra-curricular activities? For international
travel? The use, or set of uses, will directly affect how some of the
questions are answered.
Where a given Method offers variability, such as multiple networks for the
same Method, then evaluate each variant. For example, did:ethr supports
Ethereum mainnet, multiple testnets and permissioned EVM-compliant networks
such as Quorum. To apply a criteria to did:ethr, you will evaluate it against
all the variations that matter to you. Each variation should get its
own Evaluation. This applies to Level 2 Networks that can operate on
multiple Level 1 Networks as well as DID Methods that directly offer support
for multiple underlying DID registries.
When creating an Evaluation Report, we recommend noting both the
Evaluator and the date of the Evaluation. Many of the criteria
are subjective and all of them may evolve. Tracking who
made the Evaluation and when they made it will help readers better
understand any biases or timeliness issues that may affect the applicability
of the Evaluation.
Be selective and acknowledge the subjective. Evaluations do not need to
be exhaustive. There is no requirement to answer all the questions. Simply
answer the ones most relevant to the use contemplated. Similarly, understand
that any recorded Evaluation is going to represent the biases of the
Evaluator in the context of their target use. Even the same
Evaluator, evaluating the same Method for a different use, may come up
with slightly different answers—for example, that which is economically
accessible for small businesses might not be cost-effective for refugees, and
that could affect how well-suited a given Method is for a specific use.
Finally, note that this particular rubric is about decentralization. It
doesn't cover all of the other criteria that might be relevant to evaluating a
given DID Method. There are security, privacy, and economic concerns that
should be considered. We look forward to working with the community to develop
additional rubrics for these other areas and encourage Evaluators to
use this rubric as a starting point for their own work rather than the final
say in the merit of any given Method.
In short, use this rubric to help understand if a given DID Method is
decentralized enough for your needs.
To record and report an Evaluation, we recommend two possible formats,
either comprehensive or comparative.
A comprehensive Evaluation applies a single set of criteria to just one
Method. This set is chosen by the Evaluator; it need not be all
possible criteria, but it is all relevant criteria as judged by the
Evaluator.
A comparative Evaluation includes multiple Methods in the same table to
easily compare and contrast two or more different Methods. This may include
any number of criteria. These are the type of reports we use as examples
throughout the criteria list.
In addition to the selected criteria, we recommend each report specify
-
The Method(s) being evaluated
-
A link to the Method specification
-
The Evaluator(s)
-
The date of the Evaluation
-
A description of the use case(s) for which the Method is being evaluated
-
The rubric used for the Evaluation, along with reference to the
specific version.
-
Optionally, a URL for retrieving the report.
We have grouped our criteria into several categories:
-
Rulemaking
-
Design
-
Operations
-
Enforcement
-
Alternatives
-
Adoption & Diversity
-
Security
-
Privacy
Evaluators should consider criteria from all groups, as best fits
your use cases.
Three categories cover how a given Method is governed:
Rulemaking, Operations, and Enforcement.
Our approach parallels the same separation of
authority embodied in the United States Constitution.
Rulemaking addresses who makes the rules and how. (This is
the legislative branch in the US.)
Operations addresses how those rules are executed and how
everyone knows that they are carried out. (This is the executive branch in the
US.)
Enforcement addresses how we find and respond to rule
breaking. (This is the judicial branch in the US.)
This mental model is key to understanding the criteria of each section as well
as why we included some criteria and not others.
The remaining categories each covers different areas worth considering
when evaluating DID Methods.
Design addresses the method as designed. In other words, the output of the rulemaking:
what rules apply
to this DID method?
Alternatives address the availability and quality
of different implementation choices.
Adoption & Diversity covers questions related to how widely a
DID Method is used.
Security influences overall trust in the
ecosystem. Different DID methods offer different security guarantees, or guarantees of different strengths.
Privacy addresses the ability of a DID method to
ensure various privacy mechanisms. When DIDs are used as
identifiers for people, it becomes important to consider what
tools a DID method offers to operate at different levels
of privacy.
When evaluating the governance of DID Methods, three potentially independent
layers should be considered: the specification, the network, and the registry.
-
The specification is the governing document for the Method
that outlines how that particular Method implements the required and any
optional components of the DID Core specification [DID-CORE].
-
The network is the underlying communications layer, i.e., how
users of the Method communicate with others to invoke the operations of the
Method.
-
The registry is a given instance of recorded state changes,
managed according to the specification, using the communications channel.
For Rulemaking, the criteria should be evaluated against all three of the
above layers.
For Operations, the criteria should be evaluated against the network and the
registry. The specification is taken as a given (it is the core output of
Rulemaking).
For Adoption, the criteria should be evaluated for each major software
component: wallet, resolver, and registry.
For Alternatives, the criteria should be evaluated against the particular DID
Method.
For the examples in the rest of this document we refer to a set of Methods
that are familiar to the authors and exhibit interesting characteristics for
Evaluation. See the tables below.
The following sources provided example evaluations for this rubric. These are
not presented as objective fact, but rather as attempts by contributors to illustrate
noteworthy differences using their subjective judgement. Additional contributions
that flow into the registry should include a section that documents their source
with an additional row in the table.
| Relative ID | Citation | Note |
|---|
| usecase-1 |
Long term verifiable credentials |
The use of DIDs as subject
identifiers for long term (life-long) verifiable credentials such as earned
academic degrees. |
| usecase-2 |
User authentication |
The use of DIDs for authenticating users
for access to system services. |
| usecase-3 |
Verifiable software development |
The signing of commits by
developers and their verification to ensure that source code in a particular
git repository is authentic. |
The term "criteria" is often treated as both a
singular and a plural noun. In the singular, we say "The most
important criteria is the buyer's age". This singular use of
criteria has been in use for over half a century https://www.merriam-webster.com/dictionary/criterion#usage-1
In the plural we say "That proposal doesn't meet all of the
criteria." Sometimes people use both in the same sentence: "Select
one criteria from the list of criteria."
However, for formal use, "criteria" is more broadly accepted as
plural while "criterion" is singular. By this style rule, the last
sentence in the previous paragraph should be "Select one
criterion from the list of criteria."
As editors of a Note published by the World Wide Web Consortium,
we are torn. We would prefer to use
rigorous grammar and be consistent in doing so. At the same time,
we find attempts to enforce the formal rule sometimes leads people
to using the inverse, such as "Select a criteria from the list of
criterion." This is the exact opposite of our desired outcome.
In our own work with implementers and developers of the DID
Core specification, we have found that the singular "criteria" is
readily accepted and understood and leads to no confusion when
used, even alongside plural usage. Since the DID Method Rubric
is, at its core, a set of criteria with ample reason to refer to,
for example, "Criteria #23", we find the combined singular and
plural use is cleaner (just stick with "criteria"), less
confusing, and more aligned with common usage among our audience.
As such, throughout the DID Method Rubric, we use the term
"criteria" to refer to both singular instances of
criteria and plural sets of criteria.
This registry — the DID Method Rubric Registry — provides a public vehicle for publishing updated DID Method Rubric
criteria. In order to add or update a criteria, a submitter MUST submit a modification request for this registry, as a
pull request on the repository where this registry is hosted, where the modification request adheres to the following
policies:
- If there are copyright, trademark, or intellectual property
rights concerns, the addition and use MUST be authorized
in writing by the intellectual property rights holder under a F/RAND
license. For example, criteria that use trademarked brand names, use the titles or excerpts from copyrighted works, or require patented technology to perform the evaluation.
- Additions MUST NOT create unreasonable legal, security,
moral, or privacy issues that may result in direct or
indirect harm to others, including violations of the W3C Code Of
Ethics And Professional Conduct
https://www.w3.org/Consortium/cepc/
Examples of unacceptable additions include
any containing hate speech, unprofessional or
unethical attacks, proprietary information, and any personal data or
personally identifiable information (excepting
identification details of the submitter).
- All criteria MUST be uniquely identified and versioned to ensure
permanent linkability with version trackability. Those
identifiers MUST be present as an
href anchor in the HTML for the
criteria itself. See the sections below on identifiers
and versioning. Subcomponents, such as questions, responses,
relevance, and examples, are not separately versioned; any
change to a subcomponent triggers an appropriate version change to
its primary component.
- Use cases, methods, and evaluations cited within a criteria MUST
include a local link to a full citation in
the relevant citation section: Use Cases Referenced,
Methods Evaluated, and Evaluations Cited, respectively.
- Rubric criteria and associated metadata in the DID Method
Rubric Registry MAY be updated or removed at the editors’
discretion, including the addition or removal of categories of
metadata such as Use Cases Referenced or Implementations
of Note.
The primary components managed by this registry are criteria for
evaluating DID Methods, with as many as eight
subcomponents: name, id, version, question, responses, relevance,
examples, and, optionally, a source. In addition, the
DID Method Rubric maintains a list of cited references: DID methods,
use cases, and evaluations. The criteria are
independently identified and versioned (see those sections for
details) while references cited in the criteria
themselves link to their full citation in the appropriate list.
- Updates to criteria MUST follow the following versioning
guidelines.
- Proposed criteria without at least three independent example
evaluations MUST have the word “PROVISIONAL” as the last
term in its name. Once at least three independent example
evaluations exist, the word “PROVISIONAL” SHOULD be removed.
- Accepted (non-provisional) criteria MUST have at least three example evaluations, and generally SHOULD have no more example
evaluations than there are possible responses to its question. Each example should distinctly illustrate one possible response to the
criteria's question.
- To be considered, criteria MUST have the following subcomponents:
-
Name
-
Identifier
-
Version
-
Question
-
Possible Responses
-
Relevance
-
Example Evaluations
-
Additionally, criteria MAY have the following optional subcomponents:
-
Source
Each criteria needs a human-friendly, short, descriptive name
that captures the essence of the criteria as concisely as
possible.
The criteria id must be explicit, unique, and persistent. See the section on Identifiers for details.
The criteria version must increment, as appropriate, any time
the criteria, or any of its subcomponents, is updated. See
the section on Versioning for details.
This subcomponent presents the key question for evaluating the
criteria. The question MUST be a single inquiry that gets to
the heart of the criteria. It should be reasonably clear to a
competent professional skilled in the art how to resolve
the question into one of the possible responses.
This subcomponent lists the expected responses to the question.
-
Responses are either labeled or open ended.
-
Labeled responses MUST include a label and a meaning for that
label.
-
Labels for a response MUST start with “A” in each criteria and progress sequentially through the English alphabet.
-
Open ended responses specify how an evaluator should fill in the response.
This subcomponent explains why this criteria is useful. Readers
should be able to understand the extent to which this
particular criteria is applicable to their situation.
This section lists example evaluations of different DID methods
using this criteria, presented as a table for easy
correlation between evaluations of different methods using the
same criteria.
- Each example must have the following entries, each as its own column(s) in the examples table.
- Method
- Responses (on column for each element)
- Notes
-
Some criteria MAY require an additional "Referent" entry, in its
own column in the examples table.
- Example Evaluations for a given criteria SHOULD highlight
distinct responses. That is, each evaluation should have a
different set of responses from the other example evaluations.
The point of the examples is to illustrate how different
DID methods score differently.
- The editors will curate the list of proposed examples to
provide a best effort illustration of each criteria, with
consideration to highlighting a broad corpus of methods.
- Criteria with fewer than 3 example evaluations MUST be
marked “PROVISIONAL”
- Criteria SHOULD have no more example evaluations than there
are possible responses.
- Example evaluations MUST specify the evaluated method in
the first column, using a relative link to the method’s entry
in the Methods Evaluated section. The entry in the Methods
Evaluated table MUST conform to the requirements in that
section.
- The title of this column MUST be “Method”.
- The Method entry MUST uniquely identify a specific
DID Method specification suitable for evaluation,
including any specified variant such as testnet, mainnet,
etc.
- Evaluations MAY refer to open ended responses from other
evaluations. If present, this referent MUST be listed as a
possible response in another criteria’s evaluation. In this
manner, a first criteria can elicit structural elements to
be separately evaluated in subsequent criteria. This
“structure-variable” pattern allows the Rubric to support
criteria that matches the structural complexity of the
Method in question.
- If present, the Referent MUST be in the second column
- The column title MUST be appropriate for the referred
structure, e.g., “Layer” or “Governing body”, as understood
by the criteria which establishes the structural elements.
- Example evaluations MUST include at least one response,
and may include multiple columns when appropriate. Each
column allows for responses to particular system elements
such as “Specification”, “Network”, and “Registry”.
-
Column names such as "Specification", "Network", and
"Registry" may be abbreviated as "Spec.", "Net.", and "Reg."
-
The number and labels for response columns MUST be proposed
by the initial PR and will be finalized when the criteria
moves from PROVISIONAL to accepted.
- Each column MUST contain an entry that communicates the
evaluator’s judgment of the best response to the question for
this method for that element (Spec, Reg, Net).
- For improved comparability across different methods,
responses SHOULD be based on the possible responses listed
in the criteria. However, evaluators MAY provide any
appropriate response, based on their own judgment
-
Responses MAY be a variant of the Possible Responses, to
capture a nuance beyond those listed. For example, one might
append a “+” or a “-”, e.g., A+, or even provide two
different responses when different situations merit a
distinction, e.g., A/D. The reasoning behind these variants
&nbps;
SHOULD be explained in the Notes column.
- A response of “n/a” (short for Not Applicable) SHOULD be
used if the criteria doesn’t apply to the evaluated method.
- This entry SHOULD include any details that help explain
why that particular response was chosen.It MAY be blank.
However, the column MUST be included.
- The Notes entry MUST specify a distinct use case listed
explicitly in this registry in the Use Cases Referenced
section, using the label from that section.
-
The use case label MUST be delimited by square brackets.
-
The label text itself must be a hyperlink to the entry
in the Use Cases Referenced section, for example [UC.1]
where “UC.1” links to “#useCase-1” in the current
document.
-
The Notes entry MUST identify a published evaluation by
reference to an entry in the Evaluations Cited section of
the Rubric, using the label from that section.
-
The evaluation label MUST be delimited by square
brackets.
-
The label text itself must be a hyperlink to the entry
in the Evaluations Cited section, for example [E.1]
where “E.1” links to “#eval-1” in the current document.
This subcomponent, if present, MUST list the specific
external source of the criteria, if any, by referring to its
entry in the Rubric’s list of Sources Cited. Source entries
should include both the name (or title) of the source as well as
a URL. When possible, sources should specify the precise page,
section, and/or anchor that points to the specific criteria
in the source document. Each source listed in the criteria MUST
also be listed in the Sources Cited section.
Each criteria must be explicitly, uniquely, and persistently
identified using incremental numbers. New criteria should
use the next available increment based on the highest numbered
identifier in the current publication.
Previous numbered identifiers MUST NOT be re-used, even if the
criteria so identified are no longer published; this
maintains the persistent linkability of previously published
versions. Such “retired” criteria MAY be listed in a Prior
Criteria section linking directly to the latest published Rubric
containing the retired criteria; this Prior Criteria
section MUST also contain an anchor with the canonical id.
The registry shall maintain an entry with the “next available”
number. Additions should use the “next available”
number to construct an identifier for the new criteria, and update
the “next available” entry at the same time. Editors
will manage any sequencing errors when accepting PRs.
Identifiers must be constructed by appending that numerical value
to the string “criteria-” and incorporated in the ID
of the heading element for that criteria, which when placed in the
Rubric appears as something like (note the version in
its own span after the permalink):
<h4 id="criteria-1">Open contribution (participation)</h4>
<a href="#criteria-1">https://www.w3.org/TR/did-rubric#criteria-1</a> <span>1.0.0</span>
This allows permanent links to citations based on the publication
date of a given Rubric:
https://www.w3.org/TR/2021/NOTE-did-rubric-20210826/#criteria-32 as well as version-free links that resolve to the
latest version of the criteria, and if retired, to an entry in the
Prior Criteria list which itself links to the last
valid presentation of the criteria, so that, for example, https://www.w3.org/TR/did-rubric/#criteria-32 points to the
current criteria-32 in the latest published document. When that
criteria is retired, that same URL points to the entry
in the Prior Criteria list, which links to its last versioned
publication, e.g.,
https://www.w3.org/TR/2021/NOTE-did-rubric-20210826/#criteria-32
Criteria versions allow for incremental improvements while
retaining long-term referenceability.
- Versions contain a Major, Minor, and Patch number, based on
SemVer 2.0
- Every new criteria has a version of 1.0.0, even if PROVISIONAL.
- Updates to criteria must increment the appropriate version
number based on the scope of change:
- Major number MUST increment if, and only if,
the criteria is fundamentally changed in a way that invalidates
existing
evaluations. This could be a material change to any subcomponent
of the criteria, including removing or changing
Possible Responses.
- Minor number MUST increment if, and only
if, new responses are added, without invalidating existing
responses. It’s understood that an evaluator of an existing
evaluation may desire to choose the new response if they were
evaluating the criteria anew; however, as long as prior evaluations
remain valid, a Minor increment SHOULD be applied.
- Patch number MUST increment if, and only
if, the change to the question is “editorial”, defined by
clarifying the original intent of the criteria WITHOUT
invalidating existing evaluations.
-
As long as a given criteria is an evolution of essentially the same
consideration, it retains the original identifier,
even as its version is updated throughout its lifetime.
-
Criteria that address fundamentally different considerations MUST be assigned a new criteria number rather than
iterating an existing version number.
The Editors of the DID Method Rubric Registry MUST consider all of the
policies above when reviewing additions to the registry and MUST reject
registry entries if they violate any of the policies in this section.
Entities registering additions can challenge rejections first with the W3C
DID Working Group and then, if they are not satisfied with the outcome,
with the W3C Staff. W3C Staff need not be consulted on changes to the DID
Method Rubric Registry, but do have the final authority on registry
contents. This is to ensure that W3C can adequately respond to time
sensitive legal, privacy, security, moral, or other pressing concerns
without putting an undue operational burden on W3C Staff.
Submissions to the registry that meet all the criteria listed above
will be considered for inclusion, however the editors retain the
responsibility of curating the content to ensure the broadest
applicability of the DID Method Rubric with the goal of enabling effective
evaluations of any DID Method for any legitimate use case.