Accessibility Conformance Testing Rules Format 1.0

1. Introduction

There are currently many test procedures and tools available which aid their users in testing web content for conformance to accessibility standards such as the Web Content Accessibility Guidelines [WCAG]. As the web develops in both size and complexity, these procedures and tools are essential for managing the accessibility of resources available on the web.

This format is intended to enable a consistent interpretation of how to test conformance to WCAG and other accessibility requirements documents and promote consistent results in accessibility testing. The rules format is designed to describe both manual accessibility tests, as well as automated tests as performed by accessibility testing tools.

Documenting how to test accessibility requirements will result in accessibility tests that are transparent, with test results that are reproducible. The Accessibility Conformance Testing (ACT) Rules Format defines the requirements for these test descriptions, known as Accessibility Conformance Testing Rules (ACT Rules).

An ACT Rule is a plain language description of how to test a specific type of content for a specific aspect of an accessibility requirement. An ACT Rule describes what kind of content it applies to and which conditions must be true about the applicable elements for them to meet all expectations of the rule.

The ACT Rules Format defines the requirements and rule structure for the types of information each rule needs to include to be called an ACT Rule. The structure of the ACT rule is defined in the ACT Rule Structure section. Each ACT Rule also contains a plain language description of the type of content under test, the test to perform, and the expected result. Where the test result affects conformance, the rule documents the particular requirement being tested. The resulting outcomes from the test can be used to help determine conformance or non-conformance to the requirement. Test cases are also written as part of the ACT rule to provide a way to verify that implementations of the rule can successfully determine the expected outcomes.

2. Scope

The ACT Rules Format defined in this specification is designed for rules that can be used in testing content created using web technologies, such as Hypertext Markup Language [HTML], Cascading Style Sheets [css-2018], Accessible Rich Internet Applications [WAI-ARIA], Scaleable Vector Graphics [SVG2], EPUB 3.1, and more. The ACT Rules Format is designed to be technology agnostic, meaning that it can conceivably be used to describe test rules for other technologies.

The ACT Rules Format can be used to describe ACT Rules dedicated to testing the accessibility requirements defined in Web Content Accessibility Guidelines [WCAG], which are specifically designed for web content. Other accessibility requirements applicable to web technologies can also be testable with ACT Rules. For example, ACT Rules could be developed to test the conformance of web-based user agents to the User Agent Accessibility Guidelines [UAAG20]. The ACT Rules Format might not always be suitable to describe tests for other types of accessibility requirements.

3. ACT Rule Types

In accessibility, there are often different technical solutions to make the same type of content accessible. For example, there are multiple methods for giving an img element in HTML an accessible name. Multiple solutions could be tested in a single rule; however, such a rule tends to be quite complex, making it difficult to understand and maintain. The ACT Rules Format solves this by providing two types of rules:

• Atomic rules describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other "parts" of a test subject are to be tested, and when the test subject is considered to fail the rule. These rules are to be kept small and atomic. This means that atomic rules test a single "failure condition", and do so without using the findings from other rules.

• Composite rules describe how the findings of multiple atomic rules or composite rules can be combined into a single outcome. A composite rule can have multiple "satisfying conditions", each of these tested in separate rules. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them, is used to determine a single outcome.

Not all atomic rules have to be part of a composite rule. Composite rules are used when the findings of multiple rules need to be combined in order to determine if a test subject does not satisfy an accessibility requirement.

The separation between atomic rules and composite rules creates a division of responsibilities. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules can test if a combination of findings from other rules satisfy the accessibility requirement, in part or as a whole.

4. ACT Rule Structure

An ACT Rule MUST consist of at least the following items:

• Descriptive Title

• Rule Identifier

• Rule Description

• Rule type

• Accessibility Requirements Mapping

• Rule Input, which is one of the following:

  • Input Aspects (for atomic rules) OR

  • Input Rules (for composite rules)

• Applicability

• Expectations

• Assumptions

• Accessibility Support

• Test Cases

• Change Log

• Glossary

• Issues List (optional)

• Background (optional)

• Acknowledgements (optional)

The ACT Rules format does not prescribe what format ACT Rules can be written in (e.g. HTML, DOCX, PDF, etc.). However, ACT Rules MUST be written in a document that conforms to the Web Content Accessibility Guidelines [WCAG] or a comparable accessibility standard. This ensures that ACT Rules are accessible to people with disabilities. ACT Rule test cases are allowed to contain inaccessible content. If any test case contains accessibility issues listed in WCAG 2.1 Section 5.2.5 Non-Interference, users MUST be warned of this in advance. In addition to supporting people with disabilities, using an accessible format also makes internationalization of ACT Rules easier.

4.1. Rule Identifier

An ACT Rule MUST have a unique identifier that can be any unique text, such as plain text, URL or a database identifier.

4.2. Rule Description

An ACT Rule MUST have a description that is in plain language, and provides a brief explanation of what the rule does.

4.3. Accessibility Requirements Mapping

When an ACT Rule is designed to test one or more Accessibility requirements documents, the rule MUST list all accessibility requirements from those documents that are not satisfied when the outcome of the rule is failed. For example, when designing a rule for WCAG 2.1 that tests if image buttons have an alternative text, the rule maps to success criteria 1.1.1 Non-text content, and 4.1.2 Name, Role, Value. That ACT Rule will list both success criteria in its accessibility requirements mapping.

Each accessibility requirement in the mapping MUST include the following:

1. the name, title or summary of the accessibility requirement, and

2. the name of the accessibility requirements document, and

3. a link to the accessibility requirements document if one exists, and

4. the conformance level associated with the accessibility requirement, if one exists.

4.3.1. Outcome Mapping

For each accessibility requirement in the mapping, an ACT Rule MUST indicate what the outcome of the rule means for satisfying that accessibility requirement. When the outcome is failed, the accessibility requirements are *not satisfied*. When the outcome is passed or inapplicable, the accessibility requirements could be *satisfied, or further testing could be necessary. Rules that can be used to determine if an accessibility requirement is satisfied* are called satisfying tests.

4.3.2. Mapping Outside WCAG

ACT Rules can be used to test accessibility requirements that are not part of a W3C accessibility standard, such as accessibility requirements in Hypertext Markup Language [HTML], or tests in a methodology like RGAA 3 2016. An ACT Rule MUST indictate whether or not the accessibility requirement it maps to is required for conformance in its accessibility requirements document. Examples of accessibility requirements that are not required for conformance are WCAG sufficient techniques, or a company style guide that includes both requirements and optional "best practices". The distinction between what is required and what is optional has to be clear.

4.3.3. Rules Without Accessibility Requirements

If the rule does not map to any accessibility requirement, the accessibility requirement mapping will only contain the explainer that it is not required for conformance to the accessibility requirements document. This is common with atomic rules used in composite rules.

If the failed outcome can not be mapped to not satisfied for an accessibility requirement, that requirement MUST NOT be included in the accessibility requirements mapping. The optional Background section could be used to list accessibility requirements and standards when they are thematically related, but for which the rule is not a failure test.

4.3.4. External Accessibility Requirements Mapping

This section is non-normative.

While rules are often designed for one, or possibly a small collection of accessibility requirements documents, it is likely that other accessibility requirements documents also map to those ACT Rules. For example, rules can be written for WCAG 2.1, but many of those could map to a company’s internal accessibility policy. In such a scenario, an external accessibility requirements mapping could be created. An external accessibility requirements mapping ammend the accessibility requirements mapping of an ACT rule by adding mapping to a different accessibility requirements document. An external accessibility requirements mapping avoids duplication of a rule for the sole purpose of changing the mapping.

4.4. Rule Input

To evaluate content using an ACT Rule, the applicability and expectations of that rule require some information from the test subject. This is the input for the rule. What input is required is made explicit, to help testers understand what capabilities are required to use a rule. Atomic rules and composite rules have different input.

• Atomic rules have an Input Aspects

• Composite rules have an Input Rules

4.4.1. Input Aspects (Atomic rules only)

An aspect is a distinct part of the test subject. Rendering a particular piece of content to an end user commonly involves different technologies, some or all of which are required as input for atomic rule. For example, some rules need to operate directly on the Hypertext Transfer Protocol [http11] messages exchanged between a server and a client, while others need to operate on the Document Object Model [DOM] tree exposed by a web browser.

Atomic rules MUST list the aspects used as input for the applicability and expectations of the atomic rule. Rules can operate on several aspects simultaneously, such as both the HTTP messages and the DOM tree. The method through which an input aspect is served is not relevant. For example a DOM tree can be served through HTTP as HTML, it can be bundled as several pages in an EPUB document, or it can be inferred from a JSX source file. All rules that have only DOM tree as an input aspect can be applied to those technologies.

Some input aspects are well defined in a formal specification, such as HTTP messages, DOM tree, and CSS styling [css-2018]. For these, a reference to the corresponding section in the Common Input Aspects note is sufficient as a description of the aspect. For input aspects that are not well defined, an ACT Rule MUST include either a detailed description of the aspect in question, or a reference to a well defined description.

4.4.2. Input Rules (Composite rules only)

A composite rule uses findings from atomic rules or other composite rules and applies a logic to them so that a single outcome can be determined for the test subject. The identifier and descriptive title of all rules used in the expectations MUST be listed in the composite rule. The input rules describes the input for composite rules, similar to how input aspects describe the input for atomic rules.

4.5. Applicability

The applicability describes what parts of the test subject are tested.

4.5.1. Applicability for Atomic Rules

The applicability section is a required part of an atomic rule. It MUST contain a precise description of the parts of the test subject to which the rule applies. For example, specific nodes in the DOM [DOM] tree, or tags that are incorrectly closed in an HTML [HTML] document. These are known as the test targets. The applicability MUST only use information made available through the listed input aspects in the rule. No other information can be used in the applicability.

Applicability MUST be described objectively, unambiguously and in plain language.

An objective description is one that can be resolved without uncertainty, in a given technology. Examples of objective properties in HTML are tag names, their computed role, the distance between two elements, etc. Subjective properties on the other hand, are concepts like decorative, navigation mechanism and pre-recorded. Even concepts like headings and images can be misunderstood. These terms could refer to the tag name, the semantic role, or the element’s purpose on the web page. The latter of which is almost impossible to define objectively. When used in applicability, potentially ambiguous concepts MUST be defined objectively. Definitions can be put in the rule glossary, or they can be defined in the section they are used.

4.5.2. Applicability for Composite Rules

A composite rule defines how the findings from rules in its input rules are used to determine a single outcome.

The applicability of a composite rule is defined as the union of all the applicability sections of rules in the input rules. This includes atomic rules, and composite rules which themselves have a union applicability. Rule authors MAY ommit a description of the applicability for composite rules. This can be useful if it is difficult to express the combined applicability in plain language. If the composite rule includes applicability, it MUST be the union of all the applicability in the input rules.

Note that input rules in a composite rule MAY have different applicability. Because of this, not every test target applicable within the composite rule is tested by every input rule.

4.6. Expectations

An ACT Rule MUST contain one or more expectations. The expectations describe what the requirements are for test targets derived from the applicability. An expectation is an assertion about all test targets that is true if the test subject satisfies the accessibility requirement. When all expectations are true for a test subject, the test subject passed the rule. If one or more expectations are false, the test subject failed the rule. If there are no test targets, the outcome for the test subject is inapplicable.

Each expectation MUST be distinct, unambiguous, and be written in plain language. Unlike in applicability, a certain level of subjectivity is allowed in expectations. Meaning that the expectation has only one possible meaning, but that meaning MAY not fully quantifiable.

Commonly, an expectation comes in the form of an "Each test target has ..." statement. In such an expectation, all test targets need to meet all expectations for the rule to pass. It is also possible to have an expectation where not all test targets meet all expectations. For example: "85% of all test targets have an accessible name" is an expectation that is true even when 15% of the test targets do not meet the expectations.

4.6.1. Expectations for Atomic Rules

An atomic rule expectation MUST only use information available in the input aspects, from the applicability, and other expectations of the same rule. No other information can be used in the expectation. So for instance, an expectation could be "Expectation 1 is true and ...", but it can’t be "Rule XYZ passed and ...". This ensures that atomic rules are encapsulated.

4.6.2. Expectations for Composite Rules

All expectations of a composite rule MUST describe the logic that is used to determine a single passed or failed outcome for a test subject, based on the findings of rules in its input rules. The outcome for a test target is inapplicable when all input rules are inapplicable.

The expectations of a composite rule can produce a passed outcome, even if the outcome of one of its input rules is failed. A composite rule expectation MUST NOT use information from input aspects.

4.7. Assumptions

An ACT Rule MUST list any known assumptions, limitations or any exceptions for the evaluation, the test environment, technologies being used or the subject being tested. For example, a rule that would partially test WCAG 2.1 Success Criterion 1.4.3 Contrast (Minimum) based on the inspection of CSS properties could state that it is only applicable to HTML text content stylable with CSS, and that the rule does not support images of text.

Sometimes there are multiple plausible ways that an accessibility requirement can be interpreted. For instance, it is not immediately obvious if emoji characters are "text" or "non-text content" under WCAG 2.1. Whatever the interpretation is, this MUST be documented in the rule.

While this item MUST be included in the ACT Rule, it MAY be empty when there are no known assumptions, limitations or exceptions.

4.8. Accessibility Support

Content can be designed to rely on the support for particular accessibility features by different assistive technologies and user agents. For example, content using a particular WAI-ARIA [WAI-ARIA] feature relies on that feature to be supported by assistive technologies and user agents. This content would not work for assistive technologies and user agents that do not support WAI-ARIA. WCAG [WCAG] provides a definition for accessibility supported use of a Web technology.

An ACT Rule MUST include known limitations on accessibility support.

While an accessibility support section MUST be included in the ACT Rule, it MAY be empty when there are no known accessibility support issues.

4.9. Test Cases

Test cases are (snippets of) content that can be used to validate the implementation of ACT Rules. Each rule MUST have one or more test cases for passed, failed, and inapplicable outcome. A test case consists of two pieces of data, a snippet of each input aspect for a rule, and the expected outcome for that rule. Test cases serve two functions, firstly as example scenarios for readers to understand when a rule passes, when it fails, and when it is inapplicable. It also serves developers and users of accessibility testing tools and methodologies to validate that a rule is correctly implemented.

4.10. Change Log

It is important to keep track of changes to the ACT Rules so that users of the rules can understand if changes in test results are due to changes in the rules used when performing the tests, or from changes in the content itself. All changes to an ACT Rule that can change the outcome of a evaluation MUST be recorded in a change log. The change log can either be part of the rule document itself or be referenced from it.

Each new release of an ACT Rule MUST be identifiable with either a date or a version number. Additionally, a reference to the previous version of that rule MUST be available. For extensive changes, a new rule SHOULD be created and the old rule SHOULD be deprecated.

4.11. Glossary

ACT Rules MUST have a glossary section. The glossary MUST contain the outcome definition, as well as any definitions used in applicability and expectations sections in the rule. Since changes to the definition change the rule, those definitions can not be maintained independently of the rule. If a shared glossary is used for rules, any definition changes MUST be included in the change log of all rules that use that definition.

4.12. Issues List (optional)

An ACT Rule MAY include a list or a reference to a list of any known issues. The issues list would be used to record cases where the outcome of an ACT Rule was failed where it ought to have been passed or inapplicable, or vice versa. There are several reasons why this might occur. See rule accuracy for more information.

The issues list serves two purposes. For users of ACT Rules, the issues list gives insight into why an inaccurate result occurred, as well as provide confidence in the result of that rule. For the designer of the rule, the issues list is also useful to plan future updates to the rule. In a new version of the rule, resolved issues would be moved to the change log.

4.13. Background (optional)

An ACT Rule MAY contain information about the background for the development of the rule, or references to relevant reading. The relationship to the relevant reading should be specified. Examples of relevant background references for a rule for a WCAG [WCAG] success criterion could be Understanding documents, WCAG Techniques, or WAI-ARIA [WAI-ARIA], CSS [css-2018], or HTML [HTML] specifications.

4.14. Acknowledgements (optional)

An ACT Rule MAY contain acknowledgements. This can include, but is not limited to:

• List of rule authors

• List of rule reviewers/contributors

• Funding or other support

5. Rule Accuracy

This section is non-normative.

While test cases can be used to determine if an ACT Rule was correctly implemented, they do not guarantee that implementations will never produce incorrect results. When writing ACT Rules, it is almost inevitable that edge cases will be overlooked. Technologies are always evolving, and content authors are constantly coming up with new and unexpected ways to use those. Some examples of causes for inaccuracy are:

• Assumptions were made about the test subject that turned out to be untrue

• Technologies were used in an unusual and difficult to predict manner

• Technologies have changes, or aspects of the technologies were overlooked

• The accessibility requirement was not correctly interpreted

There are two types of inaccuracies that can produce incorrect results. Inaccuracies in the implementation can be addressed with test cases, but inaccuracies in the ACT Rule itself can not. After all, rule inaccuracies come from the rule author being unaware of a particular edge case.

When a test result incorrectly indicates non-conformance to an accessibility requirement, this is known as a false positive. Opposite, when a rule incorrectly indicates conformance, this is a false negative. A percentage of false positives and false negatives can be calculated by comparing it to results from an accessibility audit:

• False positives: This is the percentage of test subjects, that were failed by the rule, but conform to the accessibility requirements.

• False negatives: This is the percentage of test subjects, that were passed by the rule, but were non-conformant to the accessibility requirements.

The ever present possibility of false positives and false negatives with ACT Rules mean they will likely require ongoing maintainence. Designing a process for maintaining ACT Rules is outside the scope of the ACT Rules Format, which is limited to the rules themselves. Neverthless, it is suggested that rule authors work out a process for maintaining their rules.

6. Harmonization

This section is non-normative.

While the ACT Rules Format is designed to stimulate harmonization, there are no direct requirement in the ACT Rules Format that prevent a rule author from writing rules inconsistent with already established ACT Rules. Neither are there requirements for ACT Rules to have a certain number of implementations, or to have a certain level of accuracy. This allows quality requirements to be different for different user groups, and allows them to develop over time.

Harmonization occurs when a group of rule implementors collectively accept the validity of an ACT Rule. For example, a community group of accessibility testing tool vendors could declare they’ve harmonized on a particular set of ACT Rules. Such a group can set acceptance criteria for rules, and have quality requirements that go beyond the ACT Rules Format.

An example of such a process is the WCAG ACT Review Process.

7. Definitions

Accessibility requirement

A requirement is a condition that has to be satisfied in order to conform to a standard, or to comply with a contract, policy or regulation. An accessibility requirement is a requirement aimed at improving access for people with disabilities to an ICT product.

A common example of accessibility requirements are the WCAG 2.1 success criteria. There are other standards, including W3C standards that have recommendations for accessibility, such as WAI-ARIA and HTML. Accessibility requirements are also often found in company policies, regional standards or in legislation.

Accessibility requirements document

A document, such as a standard, contract, policy or regulation, that includes accessibility requirements. For example, WCAG 2.1, WAI-ARIA 1.1, HTML 5.2, BBC HTML Accessibility Standards v2.0

Finding

When evaluating a test subject using a rule, the findings of that rule are:

• All test targets in the test subject
• Which test targets met which expectations in the test subject
• The outcome (passed, failed or inapplicble) for the test subject

meet expectations

A test target meets an expectation, when that expectation is true for the test target individually, or when the test target increases a count that is required for the expectation to be true.

Example: An HTML img element an alt attribute meet the following expectation:

Each test target has an alt attribute

Example: An HTML element with a duplicate ID does not meet the following expectation:

No test target has the same ID

Example: An HTML video element without captions does not meet the following expectation:

80% of all test targets has a caption

Expectations are true or false assertions about a test subject. Individual test targets can meet all expectations, even if some expectations are false about the test subject as a whole

Outcome

One of three types of conclusions that come from evaluating an ACT Rule on a test subject. An outcome can be:

• Inapplicable: No part of the test subject matches the applicability
• Passed: All expectations are true for the test subject
• Failed: Some expectations are false for the test subject

Note: While inapplicable is a valid outcome for ACT Rules, it might not be a valid result for all accessibility requirements. Notably the success criteria of WCAG 2.0 and WCAG 2.1 can only be evaluated to satisfied (passed and inapplicable) or not satisfied (failed).

Note: Implementers using the [EARL10-Schema] can express the outcome with the outcome property. In addition to passed failed and inapplicable, EARL 1.0 also defined an incomplete outcome. While this can not be the outcome of an ACT Rule when applied in its entirety, it often happens that rules are only partially evaluated. For example, when applicability was automated, but the expectations have to be evaluated manually. Such "interim" results can be expressed with the incomplete outcome.

Test Subject

A resource or collection of resources that can be evaluated by an ACT Rule.

Example:

• An HTML page, including all embedded scripts, style and images
• An EPUB document
• A Vue component file

Note:>: Implementers using the [EARL10-Schema] can express the test subject with the subject property

Test Target

A distinct part of the test subject.

Example:

• Nodes within an HTML page
• A period of time within a video
• A range of characters within a text document

Note: Implementers using the [EARL10-Schema] can express the test target with the pointer property

Appendix 1: Expressing ACT Rule results with JSON-LD and EARL

This section is non-normative.

Example: Output data using EARL and JSON-LD. (See Evaluation and Report Language [EARL10-Schema] and Java Script Object Notation (JSON).)

{
  "@context": "https://raw.githubusercontent.com/w3c/wcag-act/master/earl-act.json",
  "@type": "Assertion",
  "subject": "https://example.org/",
  "test": "auto-wcag:rules/SC1-1-1-css-image.html",
  "result": {
    "outcome": "earl:failed",
    "pointer": "html > body > h1:first-child"
  }
}

Example:

{
  "@context": "https://raw.githubusercontent.com/w3c/wcag-act/master/earl-act.json",
  "@type": "Assertion",
  "subject": "https://example.org/",
  "test": "auto-wcag:SC1-1-1-css-image.html",
  "result": {
    "outcome": "earl:failed",
    "source": [{
      "test": "auto-wcag:SC1-1-1-css-image.html",
      "result": {
        "outcome": "earl:failed",
        "pointer": "html > body > h1:first-child"
      }
    }, {
      "test": "auto-wcag:SC1-1-1-css-image.html",
      "result": {
        "outcome": "earl:passed",
        "pointer": "html > body > h1:nth-child(2)"
      }
    }]
  }
}

Example: Aggregate outcomes to a WCAG success criterion

{
  "@context": "https://raw.githubusercontent.com/w3c/wcag-act/master/earl-act.json",
  "@type": "Assertion",
  "subject": "https://example.org/",
  "test": {
    "@id": "wcag20:#text-equiv-all",
    "title": "1.1.1 Non-text Content"
  },
  "result": {
    "outcome": "Failed",
    "source": [{
      "test": "auto-wcag:SC1-1-1-css-image.html",
      "result": {
        "outcome": "Failed",
        "pointer": "html > body > h1:first-child"
      }
    }, {
      "test": "auto-wcag:SC1-1-1-longdesc.html",
      "result": {
        "outcome": "Passed",
        "pointer": "html > body > img:nth-child(2)"
      }
    }]
  }
}

Example: Aggregate a list of results to a result for the website

{
  "@context": "https://raw.githubusercontent.com/w3c/wcag-act/master/earl-act.json",
  "@type": "Assertion",
  "subject": {
    "@type": ["WebSite", "TestSubject"],
    "@value": "https://example.org/"
  }
  "test": "http://www.w3.org/WAI/WCAG2A-Conformance",
  "result": {
    "outcome": "Failed",
    "source": [{
      "test": "wcag20:text-equiv-all",
      "result": {
        "outcome": "Failed",
        "source": [ … ]
      }
    }, {
      "test": "wcag20:media-equiv-av-only-alt",
      "result": {
        "outcome": "Passed",
        "source": [ … ]
      }
    }, {
      "test": "wcag20:media-equiv-captions",
      "result" : {
        "outcome": "Inapplicable",
        "source": [ … ]
      }
    }, … ]
  }
}

Appendix 2: Acknowledgments

This section is non-normative.

Participants of the AG WG active in the development of this document

Shadi Abou-Zahra, Trevor Bostic, Romain Deltour, Kathy Eng, Wilco Fiers, Alistair Garrison, Kasper Isager, Maureen Kraft, Mary Jo Mueller, Jey Nandakumar, Charu Pandhi, Stein Erik Skotkjerra, Anne Thyme Nørregaard, Kathleen Wahlbin

Enabling funders

This publication has been developed with support from the WAI-Tools Project, co-funded by the European Commission (EC) under the Horizon 2020 Program (Grant Agreement 780057). The content of this publication does not necessarily reflect the views or policies of the European Commission (EC) or any of the European Union (EU) Member States.