In what cases HTTP referer will be truncated

Question

I'm trying to understand the behavior of HTTP referer header. I noticed that sometimes the referer is full (full URL, including path and query string) but mostly it includes the domain only.

For example 'https://www.google.com/' instead of 'https://www.google.com/search?q=http+referer+truncated&oq=http+referer+truncated&aqs=chrome..69i57.6485j0j1&sourceid=chrome&ie=UTF-8#q=http+referer+is+not+full'

Are there any rules as to when the refere and is full and when it's truncated?

Tuberose

Never seen a truncated one, unless the request came from /. In what scenarios do you observe this? – Disserve

I just edited the question with example – Tuberose

The only case that i know HTTP referrer would be truncated is if you are using a packet analyzer or something similar. Internal display limits might be smaller than the actual size. – Suter

@UlugToprak, give it a try. Search for whatever you want on google, click on one of the results and look at the HTTP request referer. – Tuberose

BTW, with Bing I do see the full URL – Tuberose

it works with me with this <meta name="Referrer" content="no-referrer-when-downgrade">. – Varietal

Answer 1

HTTP referrer headers are created by browsers according to desired criteria using Referrer Policy even though there is a general standard used by majority of the browsers there are some differences about how the browsers handles the servers instructions, mainly mobile web browsers are the ones which does not cooperate nicely with WWWC recommendations on this matter.

So why is there need for different HTTP referrer headers? To understand this we need to look at what are these headers are used for first. Main purpose in its simplest form is "carrying information from the originating page to the new page".

Everywhere we see the word "information" in the web there is a information security concept attached to it and HTTP header is no different. Depending on what kind of information headers carry, server can specify the type of referrer policy needs to be used. Here is the list of referrer policies from W3

enum ReferrerPolicy { "", "no-referrer", "no-referrer-when-downgrade", "same-origin", "origin", "strict-origin", "origin-when-cross-origin", "strict-origin-when-cross-origin", "unsafe-url" };

Detailed information about each of these are available in the Referrer policy link i included above.

To give an example; Using google searching for "Yellow Pages". in this case

origin:https://www.google.ie

referer:https://www.google.ie/

Referrer Policy:origin

generated URL:https://www.google.ie/gen_204?atyp=i&ct=&cad=udla=3&ei=x65kGDkdyKGHDkF0KeoBg&e=12&zx=1494785478502

link to the first result is

https://www.google.ie/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiA26TfiHSGDFHKFAKHQXoCWUQFggoMBB&url=https%3A%2F%2Fwww.goldenpages.ie%2F&usg=AFQjCNGTG-tsBSFHgMkXw_GuvOcLEOD2hg

While the actual URL is https://www.goldenpages.ie/

When we actually click the link referrer changes to

Referer:https://www.goldenpages.ie/ and the referrer policy is

Referrer Policy:no-referrer-when-downgrade

This means if we click another link from the current page we won't see all the additional parameters similar to the ones we saw in the URL from google search results page.

To prove this is the case; click any link from the current page and watch the referrer header changing according to the the policy type (Which can be found in the associated js file if you use developer tools and inspect the network activity)

When i click the "List your business" link referrer stay as

https://www.goldenpages.ie/list-your-business/

and no other parameters are passed

So just to tidy up this messy explanation; What URL gets generated is dependent on what rules are set regarding to Referrer policy may that be a simple base rule with no parameters or a very long URL with loads of information relating to the user and origin of the navigation.

Note: URLs wont work i have jumbled some letters.

Suter

Answer 2

Updated details as of Nov 2020...

Many browsers have started to default to a stricter referrer policy (strict-origin-when-cross-origin) when making a cross-domain request instead of the old default (no-referrer-when-downgrade). This will most often result in truncated urls, but occasionally means that the referrer will not be set at all (no-referrer).

Here is an excerpt from a good article about this: https://plausible.io/blog/referrer-policy

Chrome is using strict-origin-when-cross-origin from version 85. Strict-origin-when-cross-origin is where the full path is sent if on the same domain but only sends the domain itself if going to another domain. Previously it used no-referrer-when-downgrade.

Firefox is using no-referrer-when-downgrade by default. It always passes the full path unless the request is sent from HTTPS to HTTP. Firefox is using strict-origin-when-cross-origin in the Private Browsing tabs and for known trackers.

Edge is using no-referrer-when-downgrade. Same as Firefox.

Safari is using strict-origin-when-cross-origin. Same as Chrome.

Brave is using no-referrer where the referrer header is completely removed. It never shares the full URL even for same-origin requests and you cannot even see the domain name for cross-origin requests.

Fireboard

Answer 3

There is both the Referrer-Policy header AND the referrer meta tag.

<meta name="referrer" content="none">

They seem to do exactly the same job (as described in @Ulug's answer). If both are present I don't know how the browser decides which to choose, I just deleted the HTML one to solve my problem.

Cloaca

Answer 4

HTTP Referer sent by the client can be influenced by the Referrer-Policy implemented on the source site. Many websites are using 'strict-origin-when-cross-origin' policy now. While loading a 3rd party JS file, this could modify/truncate the original 'referer'.

Fanniefannin