Dev Learning Lab

The labs on this site need JavaScript to run. The same content is written out below.

Learn › Networking › HTTP

Content negotiation and compression

Why do different people get different responses from the same address?

Even for the same address, a server can give several representations of a response that differ in format, language, and compression method. When a client says what it wants with the Accept, Accept-Language, and Accept-Encoding headers, the server chooses among them. This is called content negotiation.

Checked against real execution · Ran 88 combinations against a test server (Node http + negotiator 1.0 + zlib) and a caching proxy written for this purpose (http-cache-semantics 4.2), both started on 127.0.0.1, and checked them against the model. The compressed sizes are the real results of zlib and brotli in Node 24

Concepts in this lab

Content negotiation (Accept and Content-Type)
This is the process in which the client and the server agree, through headers, on the format in which the response body is sent. The client states the response formats it wants with the Accept header, and the server picks one of them that it can produce and writes it in the Content-Type header of the response. When a request has a body, the client states the format of that body in the Content-Type header of the request, but this is not negotiation. The sending side simply tells the other side. The values are written as media types such as application/json or text/html. When the server cannot read the format of the request body, it responds with 415 (Unsupported Media Type), and when it cannot produce a response in the format the client wants, it responds with 406 (Not Acceptable). Spring does not guess the format from the content of the body and follows the Content-Type header, so if you send JSON but write a different Content-Type, Spring cannot read it.
Quality value (qvalue)
A qvalue (quality value) is a number from 0 to 1 that shows which of several candidates the client wants more in a request header such as Accept, Accept-Language, or Accept-Encoding. It is written after a candidate as ;q=, as in "en;q=0.5, ko;q=0.9", and a candidate without it counts as 1. The server chooses the one with the highest q among those it can give, so a candidate is not chosen first just because it is written first in the header. q=0 means "I do not accept this". Which candidate to choose among those with equal preference is up to the server.
Compression in HTTP and Content-Encoding
This is the way a server sends the body of a response in a smaller form, using a method such as gzip or br (Brotli). When the client lists the methods it can decompress in the Accept-Encoding header of the request, the server compresses with one of them and writes the method it used in the Content-Encoding header of the response. The receiver decompresses the body by looking at that header and gets the original body, and Content-Type stays the original format whatever the compression. A body made of text, such as HTML or JSON, shrinks a lot, but a very small body or an image or video that is already compressed does not shrink and even grows a little. So a server is usually set to compress only text formats above a certain size.
Vary header
The Vary header is a response header with which the server tells which request headers this response depends on. For example, Vary: Accept-Encoding means that the response differs, even for the same address, depending on the value of Accept-Encoding in the request. A cache that stores a response and gives it out again uses the stored response only when the values of the headers listed in Vary are the same as in the first request, even if the address is the same. Without Vary, a shared cache can give a response made for one user (a compressed one, or one in another language) to another user as it is. On the other hand, if you put in Vary a header whose value is different for every user, the cache can hardly be used.
Cache
Storing frequently used data or computed results ahead of time in a place that can be accessed quickly, and reusing them. Responses get faster and the load on the source drops, because the source (usually a database) is not queried every time. In return, you must manage the problem of stale values that remain when the source changes.

Try these

Running with the default settings

  1. Client A sends the request GET /guide. It uses request headers to say what it wants. Accept is the format it wants to receive, Accept-Language is the language, and Accept-Encoding is the compression methods it can decompress. The shared cache in the middle has nothing stored yet, so the request goes all the way to the server.
  2. The server looks at the Accept header (text/html) and chooses the format. This server can give /guide in three formats: HTML, JSON, and PNG. Of these, it picks the long HTML document, and writes the chosen format (text/html) in the Content-Type header of the response.
  3. The server looks at the Accept-Language header (ko) and chooses the language. This means that the client wants the Korean version. The server has a Korean version, so it chooses that. It writes the chosen language in the Content-Language header of the response.
  4. Accept-Encoding: gzip, br means that the client can decompress both. No q is written, so both have the same qvalue, 1. When the qvalues are equal, the server decides. This server is set to choose br (Brotli) first.
  5. The 12,542-byte body became 425 bytes when compressed with br. The server adds the header Content-Encoding: br to the response to tell the receiver which method to use to decompress it. Content-Type stays text/html. Compression does not change the content; it changes only the packaging used for transfer.
  6. The server responds with 200 OK. It did not add a Vary header to the response. The shared cache stores this response as the response for /guide and passes it to A.
  7. Next, another client, B, requests the same address, /guide. B wants English (Accept-Language: en) and sends no Accept-Encoding. It is a simple program that cannot decompress anything. This request reaches the shared cache first.
  8. The cache has a response for /guide and there is no Vary header. The cache assumes that "the response for this address is the same for everyone" and gives B the response made for A as it is. B cannot decompress br, so it cannot read the body. B wanted English but received the Korean version.

Result of this run

Code

// Server (Node.js). Negotiator is a library that interprets the Accept family of headers
http.createServer((req, res) => {
    const want = new Negotiator(req);
    const type = want.mediaType(['text/html', 'application/json', 'image/png']);        // Accept
    const lang = want.language(['ko', 'en']) || 'ko';                                    // Accept-Language, ko if nothing matches
    const enc  = want.encoding(['br', 'gzip', 'identity'], { preferred: ['br', 'gzip'] });  // Accept-Encoding

    let body = load(type, lang);                       // the body in the chosen format and language
    if (enc === 'gzip') body = zlib.gzipSync(body);
    if (enc === 'br')   body = zlib.brotliCompressSync(body);

    res.writeHead(200, {
        'Content-Type': type,
        'Content-Language': lang,                      // not added for the image
        'Content-Encoding': enc,                       // not added when the body was not compressed
        'Cache-Control': 'public, max-age=60',         // a shared cache may store it for 60 seconds
    });
    res.end(body);
});

// Request of client A
// curl -H 'Accept: text/html' -H 'Accept-Language: ko' -H 'Accept-Encoding: gzip, br' http://shop.example/guide

Assumptions of this model

The three bodies (a long HTML document, a short JSON document, and a PNG image) are examples made for testing. The sizes are the real values obtained by compressing those bodies with the default settings of zlib 1.3.1 (gzip) and brotli 1.2.0 in Node.js 24 on the PC used to write this lab. With a different compression level or library version, they can differ by a few bytes. The HTML in the example repeats a table of the same shape, so it shrinks much more than an ordinary document does. 88 different combinations (the language setting is not used for an image) were run against a test server started on 127.0.0.1, and the format, language, compression, and size sent by the server, Vary, and the response received by B were confirmed to match the model. The test server was built from the http module of Node with the header-parsing library negotiator 1.0 (the one Express uses) and zlib. The shared cache was a small caching proxy, written for this purpose, that makes its decisions with http-cache-semantics 4.2, an implementation of the HTTP caching rules. It was not verified with a real CDN or proxy product. In addition, these were confirmed by execution: fetch (which sends Accept-Encoding: gzip, deflate by default) and curl --compressed decompress a compressed response by themselves and return it; curl without options does not send Accept-Encoding; and, behind a cache without Vary, curl without options receives the Korean response compressed with br as it is. The meaning of qvalues and the fact that a qvalue is 1 when omitted, and the fact that when no language matches the server can either ignore the header or respond with 406, follow RFC 9110 sections 12.4.2 and 12.5.4. The meaning of Vary follows section 12.5.5, and the fact that a cache compares the request headers listed in Vary follows RFC 9111 section 4.1. Which compression method to choose among those with equal qvalues is not set by the standard, so it differs from server to server (this test server prefers br). According to the standard (RFC 9110 section 12.5.3), a server may use any compression for a request without Accept-Encoding, but most servers, including this test server, send the body without compression. That B "cannot decompress" assumes that B is a program that does not support compression. A client that decompresses by itself by looking at Content-Encoding, such as a browser, can read the content, but it still has the problem of receiving a language it did not want.

Previous lab: Redirection · Next lab: CORS

Concepts in this lab

What this lab shows

Settings

Code


      
    


    

      
Diagram

      

      

    


    

      

      

      

    


  


  

    

      
Run comparison

      

      

    

    

      
Try these