Fetch

Living Standard — Last Updated 9 May 2013

This Version:
http://fetch.spec.whatwg.org/
Participate:
Send feedback to whatwg@whatwg.org or file a bug (open bugs)
IRC: #whatwg on Freenode
Version History:
https://github.com/whatwg/fetch/commits
Editor:
Anne van Kesteren (Mozilla) <annevk@annevk.nl>

CC0 To the extent possible under law, the editor has waived all copyright and related or neighboring rights to this work. In addition, as of 9 May 2013, the editor has made this specification available under the Open Web Foundation Agreement Version 1.0, which is available at http://www.openwebfoundation.org/legal/the-owf-1-0-agreements/owfa-1-0.

Abstract

The Fetch standard defines the overall architecture for obtaining resources given a URL and a set of parameters. This concept is named fetching.

Table of Contents

  1. Goals
  2. Legacy
  3. 1 Conformance
  4. 2 Terminology
    1. 2.1 HTTP
    2. 2.2 Requests
    3. 2.3 Responses
  5. 3 HTTP extensions
    1. 3.1 HTTP Origin header
    2. 3.2 HTTP CORS protocol
      1. 3.2.1 General
      2. 3.2.2 HTTP requests
      3. 3.2.3 HTTP responses
      4. 3.2.4 HTTP new header syntax
  6. 4 Fetching
    1. 4.1 Basic fetch
    2. 4.2 CORS fetch with preflight
    3. 4.3 CORS preflight fetch
    4. 4.4 CORS preflight cache
    5. 4.5 CORS check
  7. References
  8. Acknowledgments

Goals

To unify fetching across the web platform this specification supplants a number of algorithms and specifications:

Unifying fetching provides consistent handling of:

In due course, it should also help clarify the semantics of CSP and make provide a model to hook offline networking into.

Legacy

HTML defines two algorithms. This maps these "legacy" algorithms to the new fetch.

When a user agent is to fetch a resource or URL, optionally from an origin origin, optionally using a specific referrer source as an override referrer source, and optionally with any of a synchronous flag, a manual redirect flag, a force same-origin flag, and a block cookies flag, the following steps must be run:

The block cookies flag is obsolete now.

  1. If there is a specific override referrer source, and it is a URL, then let referrer be the override referrer source.

  2. Otherwise, run these substeps:

    1. Let document be the appropriate Document as given by the following list:

      If there is a specific override referrer source
      The override referrer source.
      When navigating
      The active document of the source browsing context.
      When fetching resources for an element
      The element's node document.

    2. While document is an iframe srcdoc document, set document to document's browsing context's browsing context container's Document instead.

    3. If document's origin is not a scheme/host/port tuple, then let referrer be null.

    4. Otherwise, let referrer be the document's address of document.

    5. If referrer's scheme is about/data/javascript, set referrer to null.

  3. If the Document with which any tasks queued by this algorithm would be associated doesn't have an associated browsing context, then terminate these steps.

  4. Let req be a new request.

  5. If this algorithm was invoked with "from an origin", set req's origin to origin.

  6. Set req's referrer to referrer.

  7. Set req's mode to same-origin if the force same-origin flag is set, and to tainted cross-origin otherwise.

  8. Set req's manual redirect flag if manual redirect flag is set.

  9. Set req's synchronous flag if synchronous flag is set.

  10. fetch req.

When the user agent is required to perform a potentially CORS-enabled fetch of an URL with a mode mode that is either "No CORS", "Anonymous", or "Use Credentials", optionally using a referrer source referrer source, with an origin origin, and with a default origin behaviour default which is either "taint" or "fail", it must run these steps:

  1. Let req be a new request.

  2. Set req's origin to origin.

  3. Set req's mode to the value corresponding to the first matching statement:

    mode is "No CORS" and default is "fail"
    same-origin
    mode is "No CORS" and default is "taint"
    tainted cross-origin
    Otherwise
    CORS

  4. If mode is "Anonymous", set req's omit credentials mode to always.

  5. Determine req's referrer as per fetch.

  6. fetch req.

1 Conformance

All diagrams, examples, and notes in this specification are non-normative, as are all sections explicitly marked non-normative. Everything else in this specification is normative.

The key words "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative parts of this specification are to be interpreted as described in RFC2119. For readability, these words do not appear in all uppercase letters in this specification. [RFC2119]

2 Terminology

This specification uses terminology from the Encoding and URL Standards. [ENCODING] [URL]

A byte string is a byte sequence written down as a string. To avoid confusion with an actual string quotation marks are omitted.

"true" is a string, while true is a byte string.

A case-insensitive byte string is a byte string that when compared considers bytes in the range 0x41 to 0x5A to be a match for their corresponding byte in the range 0x61 to 0x7A.

The case-insensitive byte strings Content-Type and content-TYPE are equal.

2.1 HTTP

While fetching encompasses more than just HTTP, it borrows a number of concepts from HTTP and applies these to resources obtained via other means (e.g. via data URL).

A method is a byte string.

A header consists of a name and value. A name is a case-insensitive byte string and a value is a byte string.

A simple method is a method that is GET, HEAD, or POST.

A simple header is a header whose name is Accept, Accept-Language, or Content-Language, or whose name is Content-Type and value, once parsed, is application/x-www-form-urlencoded, multipart/form-data, or text/plain.

What we never resolved in the old specification was whether only certain parameters should be allowed and how exactly the MIME type parsing would work. Hopefully parsing MIME types will solve this.

To parse a header value parse it per the ABNF and return the values obtained for the given name, and return failure if the value did not match the ABNF.

2.2 Requests

The input to fetch is a request.

A request has an associated method. Unless stated otherwise a request's method is GET.

A request has an associated url (a parsed URL).

A request has associated author headers and headers. Unless stated otherwise request's author headers and headers are empty lists of headers.

A request has an associated origin, and force Origin header flag. Unless stated otherwise a request's origin is null and its force Origin header flag is unset.

A request has an associated referrer. Unless stated otherwise it is null.

A request has an associated body. Unless stated otherwise a request's body is null.

A request has an associated synchronous flag. Unless stated otherwise a request's synchronous flag is unset.

A request has an associated mode, which is one of same-origin, tainted cross-origin, and CORS. Unless stated otherwise, request's mode is tainted cross-origin.

A request has an associated force preflight flag. Unless stated otherwise, it is unset and only relevant when request's mode is CORS.

A request has an associated omit credentials mode, which is one of always, CORS, and never.

I think we want the default to become always, but currently it is never.

A arequest has an associated use URL credentials flag. Unless stated otherwise, it is unset.

A request has an associated manual redirect flag. Unless stated otherwise, it is unset.

A request has an associated redirect count. Unless stated otherwise a request's redirect count is zero.

2.3 Responses

The result of fetch is a response. A response is build incrementally over time. All its fields are not necessarily available straight away.

A response has an associated type which is one of default, redirect, and error. Unless stated otherwise, it is default.

A response has an associated status, status message, location, headers, and a body.

A response is either CORS-same-origin or CORS-cross-origin. Unless otherwise indicated a response is CORS-same-origin.

When introducing new APIs, unless there is a very good reason, only use responses that are CORS-same-origin and treat others as network errors. This distinction is used today e.g. to allow responses from a different origin not fetched using request's mode CORS to still be displayed using the HTML img element, but not have their body extracted using the HTML canvas element.

Unless stated otherwise, a response's status is 200, status message is OK, location is null, headers is an empty list, and body is null,

A response whose type is error is known as a network error.

A network error's status is 0, status message is the empty byte string, location is null, headers is an empty list, and body is null,

A response whose type is redirect is known as a redirect.

3 HTTP extensions

3.1 HTTP Origin header

The Origin request header indicates where a fetch originates from.

The Origin header is a stripped-down version of the Referer [sic] header as to not reveal confidential path information. It is used for all basic fetches whose CORS flag is set as well as fetches where request's method is POST. Due to compatibility constraints it is not included in all fetches.

Its value ABNF: 

Origin                           = origin-or-null

origin-or-null                   = origin / %x6E.75.6C.6C ; "null", case-sensitive
origin                           = scheme "://" host [ ":" port ]

This replaces the syntax defined for the Origin header by The Web Origin Concept. Unfortunately that document cannot be updated to match reality without involving layers and layers of bureaucracy. [ORIGIN]

3.2 HTTP CORS protocol

To allow sharing resources cross-origin and allow for more versetile HTTP requests than possible with HTML's form element, the platform has a CORS protocol layered on top of HTTP. It allows resources to declare they can be shared with resources residing on a different origin.

It needs to be an opt-in mechanism to prevent leaking data from resources behind a firewall (intranets). Additionally, for credentialed HTTP requests it needs to be opt-in to prevent leaking potentially-sensitive data.

This section explains the CORS protocol as it pertains to servers. Requirements for user agents are part of the fetch algorithm.

3.2.1 General

The CORS protocol consists of a set of headers that indicates whether a particular resource can be shared cross-origin.

For HTTP requests that are more involved than what is possible with HTML's form element, a CORS preflight fetch is performed, to ensure the resource understands the CORS protocol.

3.2.2 HTTP requests

A HTTP request can be identified as pertaining in the CORS protocol if it includes an Origin header.

A HTTP request can be identified as being a CORS preflight (a check to see if the CORS protocol is understood) if it pertains in the CORS protocol (includes an Origin header), uses OPTIONS as method, and includes these headers:

Access-Control-Request-Method

Indicates which method a future HTTP request to the same resource might use.

Access-Control-Request-Headers

Indicates which headers a future HTTP request to the same resource might use.

3.2.3 HTTP responses

A HTTP response to a HTTP request pertaining in the CORS protocol can include the following headers:

Access-Control-Allow-Origin

Indicates whether a resource can be shared, via returning the value (literally) of the Origin request header, *, or null in a response.

Access-Control-Allow-Credentials

Indicates whether a resource can be shared when request's omit credentials mode is never.

For a CORS preflight request's omit credentials mode is always never, but for any subsequent HTTP requests it might not be. Support therefore needs to be indicated for the CORS preflight as well.

A HTTP response to a CORS preflight can include the following headers:

Access-Control-Allow-Methods

Indicates which methods are supported by the resource for the purposes of the CORS protocol.

Access-Control-Allow-Headers

Indicates which headers are supported by the resource for the purposes of the CORS protocol.

Access-Control-Max-Age

Indicates how long the information provided by the Access-Control-Allow-Methods and Access-Control-Allow-Headers headers can be cached.

A HTTP response to a HTTP request pertaining in the CORS protocol that is not a CORS preflight can also include the following header:

Access-Control-Expose-Headers

Indicates which headers can be exposed as part of the HTTP response, via listing their names.

3.2.4 HTTP new header syntax

ABNF for the values of the headers used by the CORS protocol: 

Access-Control-Request-Method    = Method
Access-Control-Request-Headers   = #field-name

Access-Control-Allow-Origin      = origin-or-null / "*"
Access-Control-Allow-Credentials = %x74.72.75.65 ; "true", case-sensitive
Access-Control-Expose-Headers    = #field-name
Access-Control-Max-Age           = delta-seconds
Access-Control-Allow-Methods     = #Method
Access-Control-Allow-Headers     = #field-name

4 Fetching

To perform a fetch using request, optionally with a CORS flag, run these steps:

The CORS flag within the fetch algorithm is used for handling redirects. When using fetch externally only the request parameter should be passed.

  1. Let url be request's url.

  2. Let origin be request's origin.

  3. If request's synchronous flag is unset and fetch is not invoked recursively, run the remaining steps asynchronously.

  4. Let response be the value corresponding to the first matching statement:

    url's origin is origin and the CORS flag is unset
    url's scheme is one of "about", "blob", and "data", and request's redirect count is zero

    The result of performing a basic fetch using request.

    request's mode is "same-origin"

    A network error.

    request's mode is "tainted cross-origin"

    The result of performing a basic fetch using request, marked CORS-cross-origin.

    url's scheme is not one of "http" and "https"

    A network error.

    request's force preflight flag is set
    request's method is not a simple method
    A header in request's author headers is not a simple header

    The result of performing a CORS fetch with preflight using request.

    Otherwise

    The result of performing a basic fetch using request with the CORS flag set.

  5. If fetch is invoked recursively, return response.

  6. If request's synchronous flag is set, wait for response to have been fully transmitted and then return it.

  7. Keep queueing tasks on the networking task source until the response has been fully transmitted. If no transmission is taking place, queue at least one task.

4.1 Basic fetch

To perform a basic fetch using request, with an optional CORS flag and HTTP authentication flag, switch on request's url's scheme and run the associated steps:

"about"

If request's url's scheme data is "blank", return a response whose headers consist of a single header whose name is Content-Type and value is "text/html;charset=utf-8", and body is the empty string.

Otherwise, return a network error.

"blob"

It has been argued this should be handled outside of fetching.

"data"

If request's method is GET and obtaining a resource from request's url does not return failure, return a response whose headers consist of a single header whose name is Content-Type and value is the MIME type and parameters returned from obtaining a resource, and body is the data returned from obtaining a resource.

Otherwise, return a network error.

"file"

For now, unfortunate as it is, file URLs are left as an exercise for the reader.

When in doubt, return a network error.

"ftp"

Follow the requirements from FTP to retrieve a resource.

Map the result to response.

"http"
"https"

Set up an HTTP request following the requirements from HTTP and TLS as appropriate, with these additional constraints:

  1. If request's referrer is not null, include a Referer header whose value is request's referrer.

    User agents are encouraged to provide the end user with options to always exclude Referer headers or have it expose less sensitive information.

  2. If request's force Origin header flag is set, include an Origin header whose value is request's origin, serialized to bytes.

  3. Include all of request's headers and let them replace any headers with the same name already present in the HTTP request.

  4. Include all of request's author headers as long as their name is not present in the HTTP request.

  5. If request's omit credentials mode is always or request's omit credentials mode is CORS and the CORS flag is unset, run these substeps:

    1. Include Cookie headers associated with request's url. [COOKIES]

    2. If there's an authentication entry for request's url and either request's use URL credentials flag is unset or request's url does not include credentials, include the Authorization header from the authentication entry.

    3. Otherwise, if request's url includes credentials, and the HTTP authentication flag is set, include an Authorization header using those credentials.

  6. If there's a proxy authentication entry, use it as appropriate. [HTTP]

    This intentionally does not depend on request's omit credentials mode.

If the HTTP request is terminated before all headers from the resource are obtained, return a network error.

Otherwise, when the headers from the resource are obtained, let response be a new response and set its fields appropriately.

If there are cookies to be set and either the CORS flag is set and request's omit credentials mode is CORS or request's omit credentials mode is always, run these steps:

  1. Wait until ownership of the storage mutex can be taken by this instance of the fetch algorithm.

  2. Take ownership of the storage mutex.

  3. Update the cookies. [COOKIES]

    This is a fingerprinting vector.

  4. Release the storage mutex so that it is once again free.

If the CORS flag is set and a CORS check for request and response returns failure, return a network error.

If response's status is 301, 302, 303, 307, or 308, run these steps:

  1. If response's headers do not contain a header whose name is Location, return response.

  2. If response's headers contain more than one header whose name is Location, return a network error.

  3. Let location be the value of the header whose name is Location within response's headers.

  4. Let parsed location be the result of parsing location with request's url.

  5. If parsed location is failure, return a network error.

  6. If request's redirect count is twenty, return a network error.

  7. Increase request's redirect count by one.

  8. Set response's type to redirect.

  9. Set response's location to parsed location.

  10. If request's manual redirect flag is unset, run these substeps:

    1. If the CORS flag is set and response's location's origin is not request's url's origin, set request's origin to a globally unique identifier.

    2. If the CORS flag is set and response's location's username or password is non-null, return a network error.

    3. Set request's url to response's location.

    4. Return the result of performing a fetch using request, with the CORS flag set if set.

Otherwise, if response's status is 401, run these steps:

  1. If the CORS flag is set, return response.

  2. Needs testing: multiple WWW-Authenticate headers, missing, parsing issues.

  3. If request's use URL credentials flag is unset, or the HTTP authentication flag is set, run these substeps:

    1. Prompt the end user for a username and password.

    2. Set request's url's username and password to the given values.

  4. Return the result of performing a basic fetch using request, with the HTTP authentication flag set.

Otherwise, if response's status is 407, run these steps:

  1. Needs testing: multiple Proxy-Authenticate headers, missing, parsing issues.

  2. Prompt the end user as appropriate and store the result as appropriate as a proxy authentication entry. [HTTP]

    Remaining details surrounding proxy authentication are defined by HTTP.

  3. Return the result of performing basic fetch using request.

If the HTTP authentication flag is set, create an authentication entry for request's url and the given realm. [HTTPAUTH]

Return response.

Otherwise

Return a network error.

4.2 CORS fetch with preflight

To perform a CORS fetch with preflight using request, run these steps:

  1. If

    then run these substeps:

    1. Let response be a CORS preflight fetch using request.

    2. If response is a network error, return response.

  2. Set request's manual redirect flag.

  3. Let response be a basic fetch using request with the CORS flag set.

  4. If response is a redirect, set its type to error.

  5. If response is a network error, clear using request.

  6. Return response.

4.3 CORS preflight fetch

To perform a CORS preflight fetch using request, run these steps:

  1. Let preflight be a new request.

  2. Set preflight's url to request's url.

  3. Set preflight's origin to request's origin.

  4. Set preflight's referrer to request's referrer.

  5. Set preflight's method to OPTIONS.

  6. Append a header named Access-Control-Request-Method with value request's method to preflight's headers.

  7. Let headers be the names of request's author headers, sorted lexicographically and byte lowercased.

  8. Let header value be the items in headers separated from each other by 0x2C 0x20.

  9. Append a header named Access-Control-Request-Headers with value header value to preflight's headers.

  10. Set preflight's manual redirect flag.

  11. Let response be the result of performing a basic fetch using preflight with the CORS flag set.

  12. If response's status is in the range 200 to 299, run these substeps:

    1. Let methods be the result of parsing all response's headers' named Access-Control-Allow-Methods.

    2. Let header names be the result of parsing all response's headers' named Access-Control-Allow-Headers.

    3. If either methods or header names is failure, return a network error.

    4. If methods is empty request's force preflight flag is set, append request's method to methods.

      This ensures that a CORS preflight fetch that happened solely because of request's force preflight flag is too.

    5. If request's method is not in methods and is not a simple method, return failure.

    6. If one of request's headers' names is not in header names or its corresponding header is not a simple header, return a network error.

    7. Let max-age be the result of parsing all response's headers' named Access-Control-Max-Age.

    8. If max-age is failure, set max-age to zero.

    9. If max-age is greater than an imposed limit on max-age, set max-age to the imposed limit.

    10. If the user agent does not provide for a cache, return response.

    11. For each method in methods for which there is a method cache match using request, set matching entry's max-age to max-age.

    12. For each method in methods for which there is no method cache match using request, create a new entry in the CORS preflight cache as follows:

      origin
      request's origin
      url
      request's url
      max-age
      max-age
      credentials
      False if request's omit credentials mode is not never, and true otherwise
      method
      method

    13. For each header name in header names for which there is a header name cache match using request, set matching entry's max-age to max-age.

    14. For each header name in header names for which there is no header name cache match using request, create a new entry in the CORS preflight cache as follows:

      origin
      request's origin
      url
      request's url
      max-age
      max-age
      credentials
      False if request's omit credentials mode is not never, and true otherwise
      header name
      header name

    15. Return response.

  13. Otherwise, return a network error.

4.4 CORS preflight cache

A CORS preflight cache consists of a collection of entries where each entry has these fields: origin, url, max-age, credentials, method, and header name.

Entries must be removed after the seconds specified in the max-age field have passed since storing the entry. Entries may be removed before that moment arrives.

To clear entries using a request, remove any entries in the CORS preflight cache whose origin is request's origin and whose url is request's url.

There is a cache match for a request if origin is request's origin, url is request's url, and either credentials is false and request's omit credentials mode is not never or credentials is true and request's omit credentials mode is never.

There is a method cache match for a given method using request when there is an entry in CORS preflight cache for which there is a cache match for request and its method is the given method.

There is a header name cache match for a given header name using request when there is an entry in CORS preflight cache for which there is a cache match for request and its header name is the given header name.

4.5 CORS check

To perform a CORS check for a request and response, run these steps:

  1. If response's headers contains zero or more than one header whose name is Access-Control-Allow-Origin, return failure.

  2. Let result be the value of the header whose name is Access-Control-Allow-Origin within response's headers.

  3. If request's omit credentials mode is not never and result is *, return success.

  4. If request's origin serialized to bytes is not result, return failure.

  5. If request's omit credentials mode is not never, return success.

  6. If response's headers contains zero or more than one header whose name is Access-Control-Allow-Credentials, return failure.

  7. If the value of the header whose name is Access-Control-Allow-Origin within response's headers is true, return success.

  8. Return failure.

References

[COOKIES]
HTTP State Management Mechanism, Adam Barth. IETF.
[ENCODING]
Encoding Standard, Anne van Kesteren. WHATWG.
[HTTP]
Hypertext Transfer Protocol -- HTTP/1.1, Roy Fielding, James Gettys, Jeffrey Mogul et al.. IETF.
[HTTPAUTH]
HTTP Authentication: Basic and Digest Access Authentication, J. Franks, Phillip Hallam-Baker, J. Hostetler et al.. IETF.
[ORIGIN]
The Web Origin Concept, Adam Barth. IETF.
[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels, Scott Bradner. IETF.
[URL]
URL Standard, Anne van Kesteren. WHATWG.

Acknowledgments

The editor would like to thank Adam Barth, Alexey Proskuryakov, Arne Johannessen, Arthur Barstow, Benjamin Hawkes-Lewis, Bert Bos, Björn Höhrmann, Boris Zbarsky, Brad Hill, Brad Porter, Cameron McCormack, Collin Jackson, David Håsäther, David Orchard, Dean Jackson, Eric Lawrence, Frank Ellerman, Frederick Hirsch, Gavin Carothers, Glenn Maynard, Graham Klyne, Hal Lockhart, Henri Sivonen, Ian Hickson, Jesse M. Heines, Jonas Sicking, Julian Reschke, Lachlan Hunt, 呂康豪 (Kang-Hao Lu), Maciej Stachowiak, Marc Silbey, Marcos Caceres, Mark Nottingham, Mark S. Miller, Martin Dürst, Matt Oshry, Matt Womer, Mhano Harkness, Michael Smith, Mohamed Zergaoui, Nikunj Mehta, Odin Hørthe Omdal, Philip Jägenstedt, R. Auburn, Sharath Udupa, Simon Pieters, Sunava Dutta, Surya Ismail, Thomas Roessler, Tyler Close, Vladimir Dzhuvinov, Wayne Carr, and Zhenbin Xu for being awesome.