How big can a user agent string get?

Question

If you were going to store a user agent in a database, how large would you accomdate for?

I found this technet article which recommends keeping UA under 200. It doesn't look like this is defined in the HTTP specification at least not that I found. My UA is already 149 characters, and it seems like each version of .NET will be adding to it.

I know I can parse the string out and break it down but I'd rather not.

---

EDIT
Based on this Blog IE9 will be changing to send the short UA string. This is a good change.

---

Austria

I posted this question: #17732199 – Milord

What is your UA string? I found only some strings with 137 characters in my database (which is not too big). – Batchelor

When I asked this question five years ago or so. UA strings were getting long they included lots of extra stuff... – Austria

I must ask, are the answers on this thread still relevant? Most of these answers are 8 years old. – Secrest

Answer 1

HTTP specification does not limit length of headers at all. However web-servers do limit header size they accept, throwing 413 Entity Too Large if it exceeds.

Depending on web-server and their settings these limits vary from 4KB to 64KB (total for all headers).

Striptease

Apache limits the maximum field length to 8k (httpd.apache.org/docs/2.2/mod/core.html#limitrequestfieldsize). – Reproachless

I'm less concerned with server limits since I am on IIS, I know it won't ever be bigger then their limit which is still preety large if memory serves.... – Austria

@Josh -- memory serves you well, on IIS it's 16K by default. ;-) – Striptease

For Tomcat server the max header length allowed is 8KB. Check here tomcat.apache.org/tomcat-8.0-doc/config/http.html – Raimundo

Answer 2

My take on this:

• Use a dedicated table to store only UserAgents (normalize it)
• In your related tables, store an Foreign Key value to point back to the UserAgent auto-increment primary key field
• Store the actual UserAgent string in a TEXT field and care not about the length
• Have another UNIQUE BINARY(32) (or 64, or 128 depending on your hash length) and hash the UserAgent

Some UA strings can get obscenely long. This should spare you the worries. Also enforce a maximum length in your INSERTer to keep UA strings it under 4KB. Unless someone is emailing you in the user-agent, it should not go over that length.

Analogous

TEXT field should not be used anymore as stated in MSDN : msdn.microsoft.com/en-us/library/ms187993(v=sql.90).aspx Instead, use NVARCHAR(MAX). Source: #565255 – Bruni

My database has 10,235 distinct user agent strings. I wanted to find the fastest hash algorithm that didn't produce any collisions. For my PHP environment I found md5 performed quickly at 2.3 seconds with no collisions. Interestingly I tried crc32 and crc32b and they also performed at 2.3 seconds with no collisions. But, because md5 has more combinations than crc32 and crc32b, md5 would likely have fewer possible collisions. Anyway, md5 is my choice and I expect it will work fine. – Wirra

Why hash the User Agent? Is this for quick lookup or something? – Bereniceberenson

@Bereniceberenson Lookups and uniqueness as DB unique keys can only be so long. – Analogous

@Wirra crc32 is not a hash, it does not attempt to be collision resistant. – Demission

@Bereniceberenson For uniqueness since a varchar(max) can't be unique'ed. And I hold all UAs in a single table and reference IDs wherever I need them. Hence the hash table. – Analogous

I do not understand why normalization? Is there any reason behind it? I guess raw data can be useful and more reliable - AFAIK it is very hard to keep an up to date normalizer for almost pretty anything, regardless what is it. But things like email worth it but IMO normalizing user agent is a little overwhelming. Any Idea @Analogous – Subtend

@KasirBarati I always normalize strings in a DB. Especially those that can run long. And since I only use them in reports... I don't care they are in another table when I need to query against them. But Imagine a 1KB UserAgent repeating 10K times in your Table where, let's say, you log traffic. I'd rather have a bigint repeat 10K times than 1KB. – Analogous

Answer 3

Noticed something like this in our apache logs. It looks abnormal to me but I regularly see such things in logs mostly from Windows systems.

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; (R1 1.6); SLCC1; .NET CLR 2.0.50727; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; 66760635803; runtime 11.00294; 876906799603; 97880703; 669602703; 9778063903; 877905603; 89670803; 96690803; 8878091903; 7879040603; 999608065603; 799808803; 6666059903; 669602102803; 888809342903; 696901603; 788907703; 887806555703; 97690214703; 66760903; 968909903; 796802422703; 8868026703; 889803611803; 898706903; 977806408603; 976900799903; 9897086903; 88780803; 798802301603; 9966008603; 66760703; 97890452603; 9789064803; 96990759803; 99960107703; 8868087903; 889801155603; 78890703; 8898070603; 89970603; 89970539603; 89970488703; 8789007603; 87890903; 877904603; 9887077703; 798804903; 97890264603; 967901703; 87890703; 97690420803; 79980706603; 9867086703; 996602846703; 87690803; 6989010903; 977809603; 666601903; 876905337803; 89670603; 89970200903; 786903603; 696901911703; 788905703; 896709803; 96890703; 998601903; 88980703; 666604769703; 978806603; 7988020803; 996608803; 788903297903; 98770043603; 899708803; 66960371603; 9669088903; 69990703; 99660519903; 97780603; 888801803; 9867071703; 79780803; 9779087603; 899708603; 66960456803; 898706824603; 78890299903; 99660703; 9768079803; 977901591603; 89670605603; 787903608603; 998607934903; 799808573903; 878909603; 979808146703; 9996088603; 797803154903; 69790603; 99660565603; 7869028603; 896707703; 97980965603; 976907191703; 88680703; 888809803; 69690903; 889805523703; 899707703; 997605035603; 89970029803; 9699094903; 877906803; 899707002703; 786905857603; 69890803; 97980051903; 997603978803; 9897097903; 66960141703; 7968077603; 977804603; 88980603; 989700803; 999607887803; 78690772803; 96990560903; 98970961603; 9996032903; 9699098703; 69890655603; 978903803; 698905066803; 977806903; 9789061703; 967903747703; 976900550903; 88980934703; 8878075803; 8977028703; 97980903; 9769006603; 786900803; 98770682703; 78790903; 878906967903; 87690399603; 99860976703; 796805703; 87990603; 968906803; 967904724603; 999606603; 988705903; 989702842603; 96790603; 99760703; 88980166703; 9799038903; 98670903; 697905248603; 7968043603; 66860703; 66860127903; 9779048903; 89670123903; 78890397703; 97890603; 87890803; 8789030603; 69990603; 88880763703; 9769000603; 96990203903; 978900405903; 7869022803; 699905422903; 97890703; 87990903; 878908703; 7998093903; 898702507603; 97780637603; 966907903; 896702603; 9769004803; 7869007903; 99660158803; 7899099603; 8977055803; 99660603; 7889080903; 66660981603; 997604603; 6969089803; 899701903; 9769072703; 666603903; 99860803; 997608803; 69790903; 88680756703; 979805677903; 9986047703; 89970803; 66660603; 96690903; 8997051603; 789901209803; 8977098903; 968900326803; 87790703; 98770024803; 697901794603; 69990803; 887805925803; 968908903; 97880603; 897709148703; 877909476903; 66760197703; 977908603; 698902703; 988706504803; 977802026603; 88680964703; 8878068703; 987705107903; 978902878703; 8898069803; 9768031703; 79680803; 79980803; 669609328703; 89870238703; 99960593903; 969904218703; 78890603; 9788000703; 69690630903; 889800982903; 988709748803; 7968052803; 99960007803; 969900800803; 668604817603; 66960903; 78790734603; 8868007703; 79780034903; 8878085903; 976907603; 89670830803; 877900903; 969904889703; 7978033903; 8987043903; 99860703; 979805903; 667603803; 976805348603; 999604127603; 97790701603; 78990342903; 98770672903; 87990253903; 9877027703; 97790803; 877901895603; 8789076903; 896708595603; 997601903; 799806903; 97690603; 87790371703; 667605603; 99760303703; 97680283803; 788902750803; 787909803; 79780603; 79880866903; 9986050903; 87890543903; 979800803; 97690179703; 876901603; 699909903; 96990192603; 878904903; 877904734903; 796801446903; 977904803; 9887044803; 797805565603; 98870789703; 7869093903; 87790727703; 797801232803; 666604803; 9778071903; 9799086703; 6969000903; 89670903; 8799075903; 897708903; 88680903; 97980362603; 97980503903; 889803256703; 88980388703; 789909376803; 69690703; 6969025903; 89970309903; 96690703; 877901847803; 968901903; 96690603; 88680607603; 7889001703; 789904761803; 976807703; 976902903; 878907889703; 9897014903; 896707046603; 696909903; 666603998903; 969902703; 79680421803; 9769075603; 798800192703; 97990903; 9689024903; 668604803; 969908671903; 9996094703; 69990642703; 97890895903; 977805619903; 79980859903; 88980443803; 98970649603; 997602703; 888802169903; 699907803; 667602028803; 786903283903; 997607703; 969909803; 798809925903; 9976045603; 97790903; 9789001903; 966903603; 9789069603; 968906603; 6989091803; 896701603; 6979059803; 978803903; 997606362603; 88980803; 98970803; 88880921703; 8997065703; 899700703; 698908703; 797801027903; 7889050903; 87890603; 78690703; 99660069703; 97980309903; 976800603; 666606803; 898707703; 79880019803; 66960250803; 7978049803; 88780602603; 79680903; 88880792703; 96990903; 667608603; 87790730903; 98970903; 9699032903; 8987004803; 88880703; 89770046603; 978800803; 969908903; 9798022603; 696901903; 799803703; 989703703; 668605903; 79780903; 998601371703; 796803339703; 87890922603; 898708903; 9966061903; 66960891903; 96790903; 8779050803; 98870858803; 976909298603; 9887029903; 669608703; 979806903; 878903803; 99960703; 9789086703; 979801803; 66960008703; 979806830803; 99760212703; 786906603; 797807603; 789907297703; 96990703; 786901603; 796807766603; 896702651603; 789902585603; 66660925903; 9986085703; 66960302703; 69890703; 789900703; 89970903; 9679060703; 9789002903; 979908821603; 986708140803; 976809828703; 7988082803; 79680997903; 99960803; 9788081903; 979805703; 787908603; 66960602803; 9887098703; 978803237703; 888806804603; 999604703; 977904703; 966904635703; 97680291703; 977809345603; 8878046703; 988709803; 976900773603; 989703903; 88780198603; 87790603; 986708703; 78890604703; 87790544803; 976809850903; 887806703; 987707527603; 79880803; 9897059603; 897709820603; 97880804803; 66960026703; 9789062803; 9867090803; 669600603; 8967087703; 78890903; 89770903; 97980703; 976802687603; 66860400803; 979901288603; 96990160903; 99860228903; 966900703; 66760603; 9689035703; 9779064703; 7968023603; 87890791903; 98770870603; 9798005803; 6969087903; 9779097903; 6979065703; 699903252603; 79780989703; 87690901803; 978905763903; 977809703; 97790369703; 899703269603; 8878012703; 78790803; 87690395603; 8888042803; 667607689903; 8977041803; 6666085603; 6999080703; 69990797803; 88680721603; 99660519803; 889807603; 87890146703; 699906325903; 89770603; 669608615903; 9779028803; 88880603; 97790703; 79780703; 97680355603; 6696024803; 78790784703; 97880329903; 9699077703; 89870803; 79680227903; 976905852703; 8997098903; 896704796703; 66860598803; 9897036703; 66960703; 9699094703; 9699008703; 97780485903; 999603179903; 89770834803; 96790445603; 79680460903; 9867009603; 89870328703; 799801035803; 989702903; 66960758903; 66860150803; 6686088603; 9877092803; 96990603; 99860603; 987703663603; 98870903; 699903325603; 87790803; 97680703; 8868030703; 9799030803; 89870703; 97680803; 9669054803; 6979097603; 987708046603; 999608603; 878904803; 998607408903; 968903903; 696900703; 977907491703; 6686033803; 669601803; 99960290603; 887809169903; 979803703; 69890903; 699901447903; 8987064903; 799800603; 98770903; 8997068703; 967903603; 66760146803; 978805087903; 697908138603; 799801603; 88780964903; 989708339903; 8967048603; 88880981603; 789909703; 796806603; 977905977603; 989700603; 97780703; 9669062603; 88980714603; 897709545903; 988701916703; 667604694903; 786905664603; 877900803; 886805490903; 89970559903; 99960531803; 7998033903; 98770803; 78890418703; 669600872803; 996605216603; 78690962703; 667604903; 996600903; 999608903; 9699083803; 787901803; 97780707603; 787905312703; 977805803; 8977033703; 97890708703; 989705521903; 978800703; 698905703; 78890376903; 878907703; 999602903; 986705903; 668602719603; 979901803; 997606903; 66760393903; 987703603; 78790338903; 96890803; 97680596803; 666601603; 977902178803; 877902803; 78790038603; 8868075703; 99960060603)

Extinction

Is there anyone that would like to comment on what on earth is going on with this user agent? lol I must add, I am curious how such a beast can form. – Milord

If anyone is curious; this one clocks in at 8010 chars. How could anyone on the browser team have thought that this was a good idea? It's as mad as a bag of cats! – Templin

Does truncating this user agent string at 256 or 512 get rid of any data that is useful at all? – Estrellaestrellita

I've made some observations, but not yet worked it out. There are 642 numbers. The first four numbers are always 6, 7, 8, or 9. The fifth number is always 0. The last three are always 603, 703, 803, or 903. Perhaps someone might recognise that pattern? (Half-life 3 confirmed?) – Hootenanny

Interesting. I have now added code to truncate the UA string to 255 chars for my db logs. – Bassesalpes

Answer 4

Since it's for database purposes and there is no practical limit i'd go for a UserAgents Table with UserAgentId as Int and UserAgentString as NVarChar(MAX) and use a foreign key on the original table.

Tuberous

You would probably end up with user agents on a 1-to-a-handful relationship with your users. Most user agents get so tweaked by the items a user has installed, and in a particular order, that they are almost personally identifiable (one other answer has a good example of this happening). In fact, the EFF did a study (pdf) about it. – Balky

@Balky +1 for link, very good study. It's a bit off topic because they look at several fingerprints and not only the user agent strings. In a real world scenario, for a site that gets several million page views per month you would end up with a few thousand user agent string, so normalizing makes sense IMHO. With that said, I'm not very positive on storing user agent strings in the database :P – Tuberous

@Balky The link to the study is now broken: updated link – Magdalen

@Balky I agree that your idea sounds plausible, but my data disagrees with us both. I am working with exactly this kind of system right now, and I have around 70k unique UAs for 1.2m users. The reason I am on this page is that I chose 256 as a limit on my database field and have found that 50k out of the 70k were truncated so I have lost some information. I'm going to increase it to 4k now. Will be interesting to know how many would have been unique if they were not truncated – Yuzik

Answer 5

How's this for big?:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; YPC 3.2.0; SearchSystem6829992239; SearchSystem9616306563; SearchSystem6017393645; SearchSystem5219240075; SearchSystem2768350104; SearchSystem6919669052; SearchSystem1986739074; SearchSystem1555480186; SearchSystem3376893470; SearchSystem9530642569; SearchSystem4877790286; SearchSystem8104932799; SearchSystem2313134663; SearchSystem1545325372; SearchSystem7742471461; SearchSystem9092363703; SearchSystem6992236221; SearchSystem3507700306; SearchSystem1129983453; SearchSystem1077927937; SearchSystem2297142691; SearchSystem7813572891; SearchSystem5668754497; SearchSystem6220295595; SearchSystem4157940963; SearchSystem7656671655; SearchSystem2865656762; SearchSystem6520604676; SearchSystem4960161466; .NET CLR 1.1.4322; .NET CLR 2.0.50727; Hotbar 10.2.232.0; SearchSystem9616306563; SearchSystem6017393645; SearchSystem5219240075; SearchSystem2768350104; SearchSystem6919669052; SearchSystem1986739074; SearchSystem1555480186; SearchSystem3376893470; SearchSystem9530642569; SearchSystem4877790286; SearchSystem8104932799; SearchSystem2313134663; SearchSystem1545325372; SearchSystem7742471461; SearchSystem9092363703; SearchSystem6992236221; SearchSystem3507700306; SearchSystem1129983453; SearchSystem1077927937; SearchSystem2297142691; SearchSystem7813572891; SearchSystem5668754497; SearchSystem6220295595; SearchSystem4157940963; SearchSystem7656671655; SearchSystem2865656762; SearchSystem6520604676; SearchSystem4960161466; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)

Castra

For those keeping score, that's 1546 characters, including the leading and trailing quotes. – Bim

Answer 6

There is no stated limit, only the limit of most HTTP servers. Keeping that in mind however, I would implement a column with a reasonable fixed length (use Google to find a list of known user agents, find the largest and add 50%), and just crop any user agent that is too long - any exceptionally long user agent is probably unique enough even when cropped, or is the result of some kind of bug or "hack" attempt.

Renatorenaud

Looks like the standard length is 120-150, based on networkinghowtos.com/howto/common-user-agent-list . Therefore I would crop the Max Length at 200. – Mutule

Answer 7

I got this user agent today, overflowing our vendor's storage field:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; MDDR; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)

Ridiculous! 229 chars?

So take that size, double it, double it again, and you should be set until Microsoft's next blunder (maybe this time next year).

Go bigger than 1000!

Ruderal

I wonder what do .Net CLR and Trident have to do with Mozilla – Implosive

Answer 8

Assume the user agent string has no limit on its length and prepare to store such a value. As you've seen, length is unpredictable.

In Postgres, there's a text type that accepts strings of unlimited length. Use that.

Most likely though, you'll have to start truncating at some point. Call it good at a reasonably useful increment (200, 1k, 4k) and throw away the rest.

Aeolipile

Answer 9

Not an indication of how big a user agent can get, as there's plenty of answers showing the edge cases they've came across, but the longest that could find on http://www.useragentstring.com/pages/useragentstring.php?name=All was 250 bytes.

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Media Center PC 5.0; SLCC1; OfficeLiveConnector.1.5; OfficeLivePatch.1.3; .NET4.0C; Lunascape 6.3.

Beiderbecke

Answer 10

I'll give you the standard answer:

Take the largest possible value you can possibly imagine it being, double it, and that's your answer.

Fragile

heh so how large do you think it will be? – Austria

Twice whatever I think it is, of course. Though 256 seems like a nice round number to double. – Fragile

I find it funny whenever we don't know what a good length would be we always end up with 256 or another multiple of 2. – Austria

Well 512 sounds good that gives me at least 10 years of .net releases and other junk to accumulate and by then I hope to be retired. Thanks again – Austria

@Josh: "by then I hope to be retired"... where have I heard that before?! ;-) – Ceiba

Answer 11

Here is one that is 257

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; InfoPath.2; .NET CLR 3.0.04506.648; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)

Ventage

I've seen up to 255 characters so far on a very very low traffic site. So not surprising. .Net 4.0 will probally add another 20 chars as well. – Austria

Answer 12

There is no strict limit on the size of a User-Agent string defined by any official standards or specifications. The HTTP/1.1 RFC 2616, which defines the HTTP protocol, does not specify a maximum length for the User-Agent header field.

However, practical limitations may apply depending on the specific web server, browser, or application handling the User-Agent string. Some servers or proxies may impose their own limits on header field sizes to prevent abuse or denial-of-service attacks. Additionally, excessively long User-Agent strings could potentially cause issues with parsing or handling on the server side.

In practice, User-Agent strings can vary in length, but they are typically not excessively long. Browsers and user-agent strings aim to provide useful information about the client application without being overly verbose. Most User-Agent strings are well within a few hundred characters.

It's essential to design web applications and services to handle reasonably long User-Agent strings while also considering potential security and performance implications.

Brig