Reporting API

4. Report Delivery

Over time, various features will queue up a list of reports in the user agent’s reporting cache. The user agent will periodically grab the list of currently pending reports, and deliver them to the associated endpoints. This document does not define a schedule for the user agent to follow, and assumes that the user agent will have enough contextual information to deliver reports in a timely manner, balanced against impacting a user’s experience.

That said, a user agent SHOULD make a effort to deliver reports as soon as possible after queuing, as a report’s data might be significantly more useful in the period directly after its generation than it would be a day or a week later.

4.1. Queue data as type for endpoint group on settings

Given a serializable object (data), a string (type), another string (endpoint group), and an environment settings object (settings), the following algorithm will create a report, and add it to reporting cache’s queue for future delivery.

1. Let report be a new report object with its values initialized as follows:

   body

   data

   origin

   settings’s origin

   group

   endpoint group

   type

   type

   timestamp

   The current timestamp.

   attempts

   0

2. Let url be settings’s creation URL.

3. Set url’s username to the empty string, and its password to null.

4. Set report’s url to the result of executing the URL serializer on url with the exclude fragment flag set.

5. Append report to the reporting cache.

6. Let environment be settings’s realm execution context’s realm’s ECMAScript global environment.

7. Execute § 5.2 Notify reporting observers on environment with report with environment and report.

Note: reporting observers can only observe reports from the same environment settings object.

Note: We strip the username, password, and fragment from the serialized URL in the report. See § 10.1 Capability URLs.

Note: The user agent MAY reject reports for any reason. This API does not guarantee delivery of arbitrary amounts of data, for instance.

Note: Non user agent clients (with no JavaScript engine) should not interact with reporting observers, and thus should return in step 6.

4.2. Choose an endpoint from a group

Given an endpoint group (group), this algorithm chooses an arbitrary eligible endpoint from the group, if there is one, taking into account the priority and weight of the endpoints.

1. If group is expired, return null.

   Note: In this case, the user agent MAY remove group from its client, or it may wait and collect garbage en masse at some point in the future as described in § 7.2 Garbage Collection.

2. Let endpoints be a copy of group’s endpoints list.

3. Remove every endpoint from endpoints that is pending.

4. If endpoints is empty, return null.

5. Let priority be the minimum priority value of each endpoint in endpoints.

6. Remove every endpoint from endpoints whose priority value is not equal to priority.

7. If endpoints is empty, return null.

8. Let total weight be the sum of the weight value of each endpoint in endpoints.

9. Let weight be a random number ≥ 0 and < total weight.

10. For each endpoint in endpoints:

    1. If weight is less than endpoint’s weight, return endpoint.

    2. Subtract endpoint’s weight from weight.

11. It should not be possible to fall through to here, since the random number chosen earlier will be less than total weight.

4.3. Send reports

A user agent sends reports by executing the following steps:

1. Let reports be a copy of the list of queued report objects in reporting cache.

2. Let endpoint map be an empty map of endpoint objects to lists of report objects.

3. For each report in reports:

   1. Let origin be report’s origin.

   2. Let client be the entry in the reporting cache for origin.

   3. If there exists an endpoint group (group) in client’s endpoint-groups list whose name is report’s group:

      1. Let endpoint be the result of executing § 4.2 Choose an endpoint from a group on group.

      2. If endpoint is a not null:

         1. Append report to endpoint map’s list of reports for endpoint.

         2. Skip to the next report.

   4. For each parent origin that is a superdomain match for origin [RFC6797]:

      1. Let client be the entry in the reporting cache for parent origin.

      2. If there exists an endpoint group (group) in client’s endpoint-groups list whose name is report’s group and whose subdomains flag is "include":

         1. Let endpoint be the result of executing § 4.2 Choose an endpoint from a group on group.

         2. If endpoint is an endpoint:

            1. Append report to endpoint map’s list of reports for endpoint.

            2. Skip to the next report.

   5. If we reach this step, the report did not match any endpoint and the user agent MAY remove report from the reporting cache directly. Depending on load, the user agent MAY instead wait for § 7.2 Garbage Collection at some point in the future.

4. For each (endpoint, reports) pair in endpoint map, execute the following steps asynchronously:

   1. Let result be the result of executing § 4.4 Attempt to deliver reports to endpoint on endpoint and reports.

   2. If result is "Success":

      1. Set endpoint’s failures to 0, and its retry_after to null.

      2. Remove each report in reports from the reporting cache.

      Otherwise, if result is "Remove Endpoint":

      1. Remove endpoint from the reporting cache.

         Note: reports remain in the reporting cache for potential delivery to other endpoints.

      Otherwise (if result is "Failure"):

      1. Increment endpoint’s failures.

      2. Set endpoint’s retry_after to a point in the future which the user agent chooses.

         Note: We don’t specify a particular algorithm here, but user agents are encouraged to employ some sort of exponential backoff algorithm which increases the retry period with the number of failures, with the addition of some random jitter to ensure that temporary failures don’t lead to a crush of reports all being retried on the same schedule.

         Add in a reasonable reference describing a good algorithm. Wikipedia, if nothing else.

Note: User agents MAY decide to attempt delivery for only a subset of the collected reports or endpoints (because, for example, sending all the reports at once would consume an unreasonable amount of bandwidth, etc). As reports are only removed from the cache when they’re successfully delivered, skipped reports will simply be delivered later.

4.4. Attempt to deliver reports to endpoint

Given a list of reports (reports) and an endpoint (endpoint), this algorithm will construct a request, and attempt to deliver it to endpoint. It returns "Success" if that delivery succeeds, "Remove Endpoint" if the endpoint explicitly removes itself as a reporting endpoint by sending a 410 response, and "Failure" otherwise.

1. Let collection be a new ECMAScript Array object [ECMA-262].

2. For each report in reports:

   1. Let data be a new ECMAScript Object with the following properties [ECMA-262]:

      age

      The number of milliseconds between report’s timestamp and the current time.

      type

      report’s type

      url

      report’s url

      body

      report’s body

      Note: Client clocks are unreliable and subject to skew. We therefore deliver an age attribute rather than an absolute timestamp. See also § 11.2 Clock Skew

   2. Increment report’s attempts.

   3. Append data to collection.

3. Let request be a new request with the following properties [FETCH]:

   url

   endpoint’s url

   header list

   A new header list containing a header named "Content-Type" whose value is "application/report"

   client

   null

   window

   "no-window"

   skip-service-worker flag

   Set.

   initiator

   ""

   type

   "report"

   destination

   ""

   mode

   "cors"

   credentials

   "include"

   body

   The string resulting from executing the JSON.stringify() algorithm on collection [ECMA-262]

   The "report" type does not exist in Fetch. Talk to Anne.

4. Queue a task to fetch request.

5. Wait for a response (response).

6. If response’s status is an OK status (200-299), return "Success".

7. If response’s status is 410 Gone [RFC7231], return "Remove Endpoint".

8. Return "Failure".

5. Reporting Observers

A reporting observer observes some types of reports from JavaScript, and is represented in JavaScript by the ReportingObserver object.

Each ECMAScript global environment has a registered reporting observer list, which is an ordered set of reporting observers.

Any reporting observer that is in a registered reporting observer list is considered registered.

Each ECMAScript global environment has a report buffer, which is a list of reports that have been generated in that ECMAScript global environment. This list is initially empty, and the reports are stored in the same order in which they are generated.

Note: The purpose of the report buffer is to allow reporting observers to observe reports that were generated earlier than that observer could be created (via the buffered option). For example, some reports might be generated during an earlier stage of page loading than when an observer could first be created, or before a JavaScript library is loaded that wishes to observe these reports.

Note: Reporting observers are only relevant for user agents with JavaScript engines.

5.1. Interface ReportingObserver

interface ReportBody {
};

interface Report {
  readonly attribute DOMString type;
  readonly attribute DOMString url;
  readonly attribute ReportBody? body;
};

[Constructor(ReportingObserverCallback callback, optional ReportingObserverOptions options)]
interface ReportingObserver {
  void observe();
  void disconnect();
  ReportList takeRecords();
};

callback ReportingObserverCallback = void (sequence<Report> reports, ReportingObserver observer);

dictionary ReportingObserverOptions {
  sequence<DOMString> types;
  boolean buffered = false;
};

typedef sequence<Report> ReportList;

A Report is the application exposed representation of a report. type returns type, url returns url, and body returns body.

Each ReportingObserver object has these associated concepts:

• A callback function set on creation.

• A ReportingObserverOptions dictionary called options.

• A list of Report objects called the report list, which is initially empty.

A ReportList represents a sequence of Reports, providing developers with all the convenience methods found on JavaScript arrays.

The ReportingObserver(callback, options) constructor, when invoked, must run these steps:

1. Create a new ReportingObserver object observer.

2. Set observer’s callback to callback.

3. Set observer’s options to options.

4. Return observer.

The observe() method, when invoked, must run these steps:

1. Let environment be the ECMAScript global environment associated with the context object.

2. Append the context object to the registered reporting observer list of its associated ECMAScript global environment.

3. If the context object’s buffered option is false, return.

4. Set context object’s buffered option to false.

5. For each report in the report buffer associated with environment, execute § 5.3 Add report to observer with report and the context object.

The disconnect() method, when invoked, must run these steps:

1. If the context object is not registered, return.

2. Remove the context object from the registered reporting observer list of its associated ECMAScript global environment.

The takeRecords() method, when invoked, must run these steps:

1. Let reports be a copy of the context object’s report list.

2. Empty the context object’s report list.

3. Return reports.

5.2. Notify reporting observers on environment with report

This algorithm makes report’s contents available to any registered reporting observers on the provided ECMAScript global environment environment.

1. For each ReportingObserver observer registered with environment, execute § 5.3 Add report to observer on report and observer.

2. Append report to the report buffer associated with environment.

3. If the report buffer now contains more than 100 reports, remove the item at the beginning of the report buffer.

5.3. Add report to observer

Given a report report and a ReportingObserver observer, this algorithm adds report to observer’s report list, so long as report’s type is observable by observer.

1. If report’s type is not observable from JavaScript, return.

2. If observer’s options has a non-empty types member which does not contain report’s type, return.

3. Create a new Report r with type initialized to report’s type, url initialized to report’s url, and body initialized to report’s body.

how to polymorphically initialize body?

3. Append r to observer’s report list.

4. If the size of observer’s report list is 1, Queue a task to § 5.4 Invoke reporting observers with notify list with a copy of the registered reporting observer list of the ECMAScript global environment associated with observer.

5.4. Invoke reporting observers with notify list

This algorithm invokes observer callback functions for reports of previously observed behavior.

1. For each ReportingObserver observer in notify list:

   1. If observer’s report list is empty, then continue.

   2. Let reports be a copy of observer’s report list

   3. Empty observer’s report list

   4. Invoke observer’s callback with a list of arguments consisting of reports and observer, and observer as the callback this value. If this throws an exception, report the exception.

6. Report Types

6.1. Deprecation

Deprecation reports indicate that a browser API or feature has been used which is expected to stop working in a future update to the browser.

Deprecation reports have the report type "deprecation".

A deprecation report’s body contains the following fields:

• id: an implementation-defined string identifying the feature or API that will be removed. This string can be used for grouping and counting related reports.

• anticipated_removal: A date indicating roughly when the browser version without the specified API will be generally available (excluding "beta" or other pre-release channels). This value should be used to sort or prioritize warnings. If unknown, this field should be set to null, and the deprecation should be considered low priority (removal may not actually occur).

• message: A human-readable string with details typically matching what would be displayed on the developer console. The message is not guaranteed to be unique for a given id (eg. it may contain additional context on how the API was used).

• source_file: If known, the file which first used the indicated API, or null otherwise.

• line_number: If known, the line number in source_file where the indicated API was first used, or null otherwise.

• column_number: If known, the column number in source_file where the indicated API was first used, or null otherwise.

Deprecation reports are observable from JavaScript.

7. Implementation Considerations

7.1. Delivery

The user agent SHOULD attempt to deliver reports as soon as possible to provide feedback to developers as quickly as possible. However, when this desire is balanced against the impact on the user, the user wins. With that in mind, the user agent MAY delay delivery of reports based on its knowledge of the user’s activities and context.

For instance, the user agent SHOULD prioritize the transmission of reporting data lower than other network traffic. The user’s explicit activities on a website should preempt reporting traffic.

The user agent MAY choose to withhold report delivery entirely until the user is on a fast, cheap network in order to prevent unnecessary data cost.

The user agent MAY choose to prioritize reports from particular origins over others (perhaps those that the user visits most often?)

7.2. Garbage Collection

Periodically, the user agent SHOULD walk through the cached reports and endpoints, and discard those that are no longer relevant. These include:

• endpoint groups which are expired

• endpoint groups which have not been used in some arbitrary period of time (perhaps a ~week?)

• endpoints whose failures exceed some user-agent-defined threshold (~5 seems reasonable)

• reports whose attempts exceed some user-agent-defined threshold (~5 seems reasonable)

• reports which have not been delivered in some arbitrary period of time (perhaps ~2 days?)

For any reports that are discarded, these reports should also be removed from the report buffer of any reporting observer.

8. Deployment Considerations

8.1. Custom Metadata

A server might want to include additional metadata in reports that are generated for their origin. This can be accomplished by encoding the extra metadata in the url of any endpoints in the Report-To response headers for the origin — for example, in the URL path or query parameters.

Report-To: { "group": "csp",
             "max-age": 10886400,
             "endpoints": [
               { "url": "https://example.com/reports?nonce=e897932f" }
             ] }

Since the instructions in a Report-To header will be used for future requests to the same origin, the server SHOULD NOT use this mechanism to encode metadata that is only valid for the current request. The metadata MUST be valid for all requests to the same origin from the same user.

8.2. Spam Mitigation

One potential use of § 8.1 Custom Metadata is to help prevent spam — report uploads that don’t correspond to a real request made by a real user. For instance, when constructing the Report-To for a response, the server could create a nonce whose value depends on the origin of the request, and the public IP address of the client. The server would then embed this nonce into the url values of the header.

When the collector receives a report, it will have access to the nonce (since that will be part of the URL in the POST request to the collector). It can construct a nonce for each report in the upload, using the origin of the report’s url and the IP address of the uploading client. If any of the per-report nonces don’t match the nonce in the upload URL, the corresponding reports can be considered fraudulent, and dropped.

9. Sample Reports

POST / HTTP/1.1
Host: example.com
...
Content-Type: application/report

[{
  "type": "csp",
  "age": 10,
  "url": "https://example.com/vulnerable-page/",
  "body": {
    "blocked": "https://evil.com/evil.js",
    "directive": "script-src",
    "policy": "script-src 'self'; object-src 'none'",
    "status": 200,
    "referrer": "https://evil.com/"
  }
}, {
  "type": "hpkp",
  "age": 32,
  "url": "https://www.example.com/",
  "body": {
    "date-time": "2014-04-06T13:00:50Z",
    "hostname": "www.example.com",
    "port": 443,
    "effective-expiration-date": "2014-05-01T12:40:50Z"
    "include-subdomains": false,
    "served-certificate-chain": [
      "-----BEGIN CERTIFICATE-----\n
      MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT\n
      ...
      HFa9llF7b1cq26KqltyMdMKVvvBulRP/F/A8rLIQjcxz++iPAsbw+zOzlTvjwsto\n
      WHPbqCRiOwY1nQ2pM714A5AuTHhdUDqB1O6gyHA43LL5Z/qHQF1hwFGPa4NrzQU6\n
      yuGnBXj8ytqU0CwIPX4WecigUCAkVDNx\n
      -----END CERTIFICATE-----",
      ...
    ]
  }
}, {
  "type": "nel",
  "age": 29,
  "url": "https://example.com/thing.js",
  "body": {
    "referrer": "https://www.example.com/",
    "server-ip": "234.233.232.231",
    "protocol": "",
    "status-code": 0,
    "elapsed-time": 143,
    "age": 0,
    "type": "http.dns.name_not_resolved"
  }
}]

10. Security Considerations

10.1. Capability URLs

Some URLs are valuable in and of themselves. To mitigate the possibility that such URLs will be leaked via this reporting mechanism, we strip out credential information and fragment data from the URL we store as a report’s originator. It is still possible, however, for a feature to unintentionally leak such data via a report’s body. Implementers SHOULD ensure that URLs contained in a report’s body are similarly stripped.

11. Privacy Considerations

11.1. Network Leakage

Because this reporting mechanism is out-of-band, and doesn’t rely on a page being open, it’s entirely possible for a report generated while a user is on one network to be sent while the user is on another network, even if they don’t explicitly open the page from which the report was sent.

Consider mitigations. For example, we could drop reports if we change from one network to another. <https://github.com/WICG/BackgroundSync/issues/107>

11.2. Clock Skew

Each report is delivered along with an age property, rather than the timestamp at which it was generated. We do this because each user’s local clock will be skewed from the clock on the server by an arbitrary amount. The difference between the time the report was generated and the time it was sent will be stable, regardless of clock skew, and we can avoid the fingerprinting risk of exposing the clock skew via this API.

11.3. Cross-origin correlation

If multiple origins all use the same reporting endpoint, that endpoint may learn that a particular user has interacted with a certain set of websites, as it will receive origin-tagged reports from each. This doesn’t seem worse than the status quo ability to track the same information from cooperative origins, and doesn’t grant any new tracking ability above and beyond what’s possible with <img> today.

11.4. Subdomains

This specification allows any resource on a host to declare a set of reporting endpoints for that host and each of its subdomains. This doesn’t have privacy implications in and of itself (beyond those noted in § 11.5 Clearing the reporting cache), as the reporting endpoints themselves don’t take any real action, as features will need to opt-into using these reporting endpoints explicitly. Those features certainly will have privacy implications, and should carefully consider whether they should be enabled across origin boundaries.

11.5. Clearing the reporting cache

A user agent’s reporting cache contains data about a user’s activity on the web, and user agents ought to handle this data carefully. In particular, if a user agent gives users the ability to clear their site data, browsing history, browsing cache, or similar, the user agent MUST also clear the reporting cache. Note that this includes both the pending reports themselves, as well as the endpoints to which they would be sent. Both MUST be cleared.

11.6. Disabling Reporting

Reporting is, to some extent, a question of commons. In the aggregate, it seems useful for everyone for reports to be delivered. There is direct benefit to developers, as they can fix bugs, which means there’s indirect benefit to users, as the sites they enjoy will be more stable and enjoyable. As a concrete example, Content Security Policy grants something like herd immunity to cross-site scripting attacks by alerting developers about potential holes in their sites' defenses. Fixing those bugs helps every user, even those whose user agents don’t support Content Security Policy.

The calculus, of course, depends on the nature of data that’s being delivered, and the relative maliciousness of the reporting endpoints, but that’s the value proposition in broad strokes.

That said, it can’t be the case that this general benefit be allowed to take priority over the ability of a user to individually opt-out of such a system. Sending reports costs bandwidth, and potentially could reveal some small amount of additional information above and beyond what a website can obtain in-band ([NETWORK-ERROR-LOGGING], for instance). User agents MUST allow users to disable reporting with some reasonable amount of granularity in order to maintain the priority of constituencies espoused in [HTML-DESIGN-PRINCIPLES].

12. IANA Considerations

The permanent message header field registry should be updated with the following registration: [RFC3864]

12.1. Report-To

Header field name

Report-To

Applicable protocol

http

Status

standard

Author/Change controller

W3C

Specification document

This specification (see § 3.1 The Report-To HTTP Response Header Field)