Chunked transfer encoding

Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which a web server serves content in a series of chunks. It uses the Transfer-Encoding HTTP response header in place of the Content-Length header, which the protocol would otherwise require. Because the Content-Length header is not used, the server does not need to know the length of the content before it starts transmitting a response to the client (usually a web browser). Web servers can begin transmitting responses with dynamically-generated content before knowing the total size of that content.

The size of each chunk is sent right before the chunk itself so that a client can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.

Rationale

The introduction of chunked encoding into HTTP 1.1 provided a number of benefits:

• Chunked transfer encoding allows a server to maintain an HTTP persistent connection for dynamically generated content. Normally, persistent connections require the server to send a Content-Length field in the header before starting to send the entity body, but for dynamically generated content this is usually not known before the content is created.^[1]
• Chunked encoding allows the sender to send additional header fields after the message body. This is important in cases where values of a field cannot be known until the content has been produced such as when the content of the message must be digitally signed. Without chunked encoding, the sender would have to buffer the content until it was complete in order to calculate a field value and send it before the content.
• HTTP servers sometimes use compression (gzip) or deflate methods to optimize transmission. Chunked transfer encoding can be used to delimit parts of the compressed object. In this case the chunks are not individually compressed. Instead, the complete payload is compressed and the output of the compression process is chunk encoded. In the case of compression, chunked encoding has the benefit that the compression can be performed on the fly while the data is delivered, as opposed to completing the compression process beforehand to determine the final size.

Applicability

For version 1.1 of the HTTP protocol, the chunked transfer mechanism is considered to be always acceptable, even if not listed in the TE request header field, and when used with other transfer mechanisms, should always be applied last to the transferred data and never more than one time. This transfer coding method also allows additional entity header fields to be sent after the last chunk if the client specified the "trailers" parameter as an argument of the TE field. The origin server of the response can also decide to send additional entity trailers even if the client did not specify the "trailers" option in the TE request field, but only if the metadata is optional (i.e. the client can use the received entity without them). Whenever the trailers are used, the server should list their names in the Trailer header field; 3 header field types are specifically prohibited from appearing as a trailer field: Transfer-Encoding, Content-Length and Trailer.

Format

If a Transfer-Encoding field with a value of chunked is specified in an HTTP message (either a request sent by a client or the response from the server), the body of the message consists of an unspecified number of chunks, a terminating last-chunk, an optional trailer of entity header fields, and a final CRLF sequence.

Each chunk starts with the number of octets of the data it embeds expressed in hexadecimal followed by optional parameters (chunk extension) and a terminating CRLF (carriage return and line feed) sequence, followed by the chunk data. The chunk is terminated by CRLF. If chunk extensions are provided, the chunk size is terminated by a semicolon followed with the extension name and an optional equal sign and value.

The last chunk is a zero-length chunk, with the chunk size coded as 0, but without any chunk data section.

The final chunk may be followed by an optional trailer of additional entity header fields that are normally delivered in the HTTP header to allow the delivery of data that can only be computed after all chunk data has been generated. The sender may indicate in a Trailer header field which additional fields it will send in the trailer after the chunks.

Example

Encoded response

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

25
This is the data in the first chunk

1C
and this is the second one

3
con
8
sequence
0

Note: Chunk size only indicates size of chunk data not the trailer. does not include CRLF("\r\n")^[2]

Anatomy of encoded response

The first two chunks in the above sample contain explicit \r\n characters in the chunk data.

"This is the data in the first chunk\r\n"      (37 chars => hex: 0x25)
"and this is the second one\r\n"               (28 chars => hex: 0x1C)
"con"                                          (3  chars => hex: 0x03)
"sequence"                                     (8  chars => hex: 0x08)

The response ends with a zero-length last chunk: "0\r\n" and the final "\r\n".

Decoded data

This is the data in the first chunk
and this is the second one
consequence

See also

• List of HTTP header fields

References

1. ^ Roy T. Fielding (10 Oct 1995). "Keep-Alive Notes". HTTP Working Group (HTTP-WG) mailing list. http://ftp.ics.uci.edu/pub/ietf/http/hypermail/1995q4/0063.html. 
2. ^ http://skrb.org/ietf/http_errata.html

• See RFC 2616 for further details; section 3.6.1 in particular.