Sharovatov’s Weblog

HTTP Chunked Encoding

Posted in browsers, http by sharovatov on 30 April 2009

Did you notice that some pages on the internet start rendering in your browser incrementally, block after block, and on some sites you have to sit and look at the white screen and then get the full page in one second?

There’re two main problems that can make your browser wait for the whole page to load before it starts parsing it:

  1. if you’re using IE6 and the page you’re viewing has table-based layout without COLGROUPs/COLs specifying width or table-layout: fixed CSS rule.
  2. if the page’s not being served from server using chunked encoding.

The first issue is really simple – IE6 has to know the exact width of the columns before it starts displaying the table, and if you have table-layout: fixed rule for the table or COLs with specified width – it will wait for the whole content to load, calculate the width and only then display the table. Other browsers (such as Opera, Firefox, Google Chrome) and newer versions of IE don’t have this issue and start displaying content right after they get at least a piece of it.

So while the first issue is really simple, the second is definitely more interesting!

Normally when HTTP client receives a response, it parses HTTP headers and then tries to read from the input the exact amount of bytes as specified in Content-Length header. So if it takes 3 seconds for the server-side script to prepare the page, HTTP client (and the user!) will be just waiting with opened connection for these seconds.

What was OK at the time when HTTP1.0 was invented and the web had almost only static content, authors of HTTP1.1 thought was inacceptable for era of web applications.

And HTTP 1.1 Spec introduces a concept of “Chunked” transfer encoding:

The chunked encoding modifies the body of a message in order to transfer it as a series of chunks, each with its own size indicator, followed by an OPTIONAL trailer containing entity-header fields. This allows dynamically produced content to be transferred along with the information necessary for the recipient to verify that it has received the full message

The main goal of HTTP Chunked Encoding is to allow clients to parse and display data immediately after the first chunk is read!

Here’s a sample of HTTP response with chunked encoding:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/html

c
<h1>go!</h1>

1b
<h1>first chunk loaded</h1>

2a
<h1>second chunk loaded and displayed</h1>

29
<h1>third chunk loaded and displayed</h1>

0
 

As you may see, there’s no Content-Length field in the beginning of the message, but there’s a hexadecimal chunk-size before every chunk. And 0 with two CRLFs specifies the end of the payload.

So the server doesn’t need to calculate Content-Length before it starts serving data to client. This is an amazing functionality! It means that the server can start sending the first part of the response while still processing the other parts of it.

Say, you have a dynamic page with two elements, both of which are queried from the database. So you can either wait for both queries to finish, populate your template with results and send the whole page to client, or you can get first query result, send it in one chunk to the client, then do the next query and send its results in another chunk. You may not notice the difference between chunked and normal serving mode in most of the cases – but if the page is created from different sources or it takes significant time to prepare the data – user experience may be seriously improved.

Before the widespread popularization of AJAX (another Microsoft-invented technology) – Chunked Encoding was used as a core for so-called “Server Push” approach for building web-chats and other streaming purposes. The idea was simple – server didn’t close the HTTP connection and kept on sending chunk after chunk with new messages or other data. This approach had serious drawbacks – e.g. for each new client server had to instantiate a new connection (which eats resources), browsers had a limit on waiting time, so the page had to be reloaded once in a while and so on. But anyway, Chunked Encoding was widely used.

In my company we use Chunked Encoding to show loading progressbar in our online train tickets ordering system – we serve the first chunk with nicely styled <div id=”loading”></div> and when the data for the main table is ready, we serve it in the second chunk. And after the document is fully loaded, javascript routine hides <div id=”loading”></div> :) Simple and nice.

Share :

Tagged with:

9 Responses

Subscribe to comments with RSS.

  1. Mackle said, on 22 December 2009 at 9:00 am

    Quote:

    c
    go!
    1b
    first chunk loaded
    —————————————————————————————–
    chunk-sizes in your example is really small: 0c,1b bytes .

    When I use Apache and PHP to try HTTP Chunked Transfer Encoding,
    I found all chunk-sizes are about 8k bytes.

    I guess the size has somthing to do with Apache’ configuration, but I am not sure yet.

    Could you tell me how you set the small chunk-size? Thanks!

  2. sharovatov said, on 30 December 2009 at 3:09 pm

    Hi Mackle

    You may be experiencing this bug if you’re using php4 under apache.

    If that’s the case, the only way to fix this is to upgrade to php5.

    Thanks,
    Vitaly Sharovatov

  3. Mackle said, on 31 December 2009 at 9:13 am

    Hi sharovatov,
    I am using PHP 5.2.11 under Apache.
    Could you have other comments or suggestions?

    Thanks!

  4. sharovatov said, on 31 December 2009 at 10:17 am

    Hi Mackle

    Do you run php with mod_fastcgi? What is your server OS?

    If you run mod_fastcgi, then you might want to enable -flush in mod_fastcgi config (as per http://www.fastcgi.com/mod_fastcgi/docs/mod_fastcgi.html#FastCgiConfig) so that mod_fastcgi library won’t be buffering the response from php before passing on to apache.

    WBR,
    Vitaly

  5. Amit Tomar said, on 19 October 2010 at 12:23 pm

    Hii sharovatov,

    Am Running Apache frontend to mongrel and in my rails application upload some large file say 5GB ,am doing it successfully from google chrome but When i
    tried it from IE am getting

    Internet Explorer cannot display the webpage
    Expert says that since chorme does the chunked transfer encoding but IE doesn’t so am getting this error . is there any to provide chunked support in IE or should i use latest version of IE

  6. BS-Harou said, on 5 December 2010 at 10:14 pm

    Hi,
    I’ve decided to read and learn HTTP1.1 protocol and I found Chunked Encoding very interesting. Mainly the part that it’s possible to send headers after sending some part of entity-body. Does this mean that can for example set cookie when already sended half of the web page?

    + I tried to create simple website with “header(“Transfer-Encoding: chunked”);”. The problem is that the page keep loading and never stops (and all I can see is only white screen). I use PHP 3 and Apache 2.2.

    There is also the part with “, followed by an OPTIONAL trailer containing entity-header fields” but I can’t find out how to write the header with this “fields”.

    I hope you can help me – this whole thing is completely new for me and unfortunately there isn’t many resources on the internet.

  7. Nikhil Kumar Saraf said, on 27 April 2011 at 9:02 am

    Hi,
    I have a query regarding chunked request.I have a web server running on a device.
    Now I want to send a POST request using java script in chunked request mode.How do I that.
    Please help

  8. Ville said, on 10 May 2011 at 10:31 pm

    Hi Nikhil,
    Check out the Node.js (http://nodejs.org/), which is a very nice framework to build fast servers in JavaScript and it also uses chunked encoding.

  9. Ranc said, on 18 October 2011 at 12:43 pm

    He asked for a client side JavaScript code to run on the browser. node.js is a standalone environment. basically a JavaScript execution environment nothing more..


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: