#1071 - Specified key was too long; max key length is 1000 bytes

Question

I know questions with this title have been answered before, but please do read on. I've read thoroughly all the other questions/answers on this error before posting.

I am getting the above error for the following query:

CREATE TABLE IF NOT EXISTS `pds_core_menu_items` (
  `menu_id` varchar(32) NOT NULL,
  `parent_menu_id` int(32) unsigned DEFAULT NULL,
  `menu_name` varchar(255) DEFAULT NULL,
  `menu_link` varchar(255) DEFAULT NULL,
  `plugin` varchar(255) DEFAULT NULL,
  `menu_type` int(1) DEFAULT NULL,
  `extend` varchar(255) DEFAULT NULL,
  `new_window` int(1) DEFAULT NULL,
  `rank` int(100) DEFAULT NULL,
  `hide` int(1) DEFAULT NULL,
  `template_id` int(32) unsigned DEFAULT NULL,
  `alias` varchar(255) DEFAULT NULL,
  `layout` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`menu_id`),
  KEY `index` (`parent_menu_id`,`menu_link`,`plugin`,`alias`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Does anyone have idea why and how to fix it? The catch is - this same query works perfectly on my local machine, and worked as well on my previous host. Btw.it's from a mature project - phpdevshell - so I'd guess these guys know what they are doing, although you never know.

Any clue appreciated.

I'm using phpMyAdmin.

Cayman

If with Java Hibernate refer to answer https://mcmap.net/q/151985/-caused-by-java-sql-sqlsyntaxerrorexception-specified-key-was-too-long-max-key-length-is-1000-bytes – Preventer

see solution here youtu.be/EWZfHyNs08c – Antinode

Answer 1

As @Devart says, the total length of your index is too long.

The short answer is that you shouldn't be indexing such long VARCHAR columns anyway, because the index will be very bulky and inefficient.

The best practice is to use prefix indexes so you're only indexing a left substring of the data. Most of your data will be a lot shorter than 255 characters anyway.

You can declare a prefix length per column as you define the index. For example:

...
KEY `index` (`parent_menu_id`,`menu_link`(50),`plugin`(50),`alias`(50))
...

But what's the best prefix length for a given column? Here's a method to find out:

SELECT
 ROUND(SUM(LENGTH(`menu_link`)<10)*100/COUNT(`menu_link`),2) AS pct_length_10,
 ROUND(SUM(LENGTH(`menu_link`)<20)*100/COUNT(`menu_link`),2) AS pct_length_20,
 ROUND(SUM(LENGTH(`menu_link`)<50)*100/COUNT(`menu_link`),2) AS pct_length_50,
 ROUND(SUM(LENGTH(`menu_link`)<100)*100/COUNT(`menu_link`),2) AS pct_length_100
FROM `pds_core_menu_items`;

It tells you the proportion of rows that have no more than a given string length in the menu_link column. You might see output like this:

+---------------+---------------+---------------+----------------+
| pct_length_10 | pct_length_20 | pct_length_50 | pct_length_100 |
+---------------+---------------+---------------+----------------+
|         21.78 |         80.20 |        100.00 |         100.00 |
+---------------+---------------+---------------+----------------+

This tells you that 80% of your strings are less than 20 characters, and all of your strings are less than 50 characters. So there's no need to index more than a prefix length of 50, and certainly no need to index the full length of 255 characters.

PS: The INT(1) and INT(32) data types indicates another misunderstanding about MySQL. The numeric argument has no effect related to storage or the range of values allowed for the column. INT is always 4 bytes, and it always allows values from -2147483648 to 2147483647. The numeric argument is about padding values during display, which has no effect unless you use the ZEROFILL option.

Granthem

There's a subtle bug in your handy query to measure how long the strings really are: you're assuming that the strings are all present; that is, that they are all there. If some are null, it will throw off your calculations and underreport short strings. You want to use count([field_name]) instead of count(*). – Ciliary

It won't use more than the first "prefix" index. – Barbell

This is a life saver, when you find a mysql database which has all its varchar fields set to length 255 (even on columns that have max 10 characters in real life) and you try to add a composite index. Thank you! – Voice

What if your key is a unique key, and this isn't about performance, but about ensuring correct values? – Ophthalmia

@Ophthalmia A unique key on a prefix of a string means the part in the prefix must be unique. If that doesn't work for the strings in your column, then define a longer prefix, or no prefix. – Granthem

@BillKarwin What I meant was -- what if we need a unique key with all these full prefixes (or no prefix), and we've exceeded the built-in 3072 limit? – Ophthalmia

@Ophthalmia I don't know what you want me to say. MySQL cannot make an index, unique or non-unique, on such a long string. You could add another column to store a hash of your string and put a unique index on that. – Granthem

I think we won't get the real max column length because of ROUND(). When I use your method, the length of 2000 will cover 100 percent of my records. While using SELECT MAX(LENGTH(source)) FROM tb;, the result is more than 6000. – Biometry

@Bill Karwin Remove ROUND() from your solution, or use SELECT MAX(LENGTH(source)) FROM tb;, if the purpose is to find the max column length;. Yours is still OK for deciding on a proper length for indexing, but not accurate enough for getting the max column length, because ROUND() will round 99.9% to 100% apparently. – Biometry

I find it confusing to use SUM() this way. To me, using COUNT() instead of SUM() makes more sense, i.e. SELECT ROUND(COUNT(LENGTH(menu_link)<10)*100/COUNT(menu_link),2) AS pct_length_10 FROM pds_core_menu_items;, but this would give the wrong result of 100.00. – Biometry

@wbzy00, Correct, COUNT() counts values of 0 the same as it counts values of 1. But SUM() counts only values of 1, since adding a 0 doesn't change the sum. Thus SUM() of a boolean expression is equal to the COUNT() of rows where the boolean expression is true. This works in MySQL because MySQL treats false as the integer value 0 and true as the integer value 1. – Granthem

In what way is this too long? My create table statement worked perfectly in MySQL, but it is now failing after trying to run it in MariaDB. I see no reason why 255 characters is to long for a key in this day and age. Anything requiring smaller than 255 is antiquated. – Heavyset

@JohnFoll You're aware that MySQL and MariaDB are different products? – Granthem

@JohnFoll It also depends which version of MariaDB you use (check with SELECT VERSION();). MariaDB 10.3.1 and later always default to large index prefixes (cf. mariadb.com/kb/en/innodb-limitations). If you use an older version, see https://mcmap.net/q/151986/-how-to-enable-large-index-in-mariadb-10 for instructions on what you can do. – Granthem

Answer 2

This error means that length of index index is more then 1000 bytes. MySQL and storage engines may have this restriction. I have got similar error on MySQL 5.5 - 'Specified key was too long; max key length is 3072 bytes' when ran this script:

CREATE TABLE IF NOT EXISTS test_table1 (
  column1 varchar(500) NOT NULL,
  column2 varchar(500) NOT NULL,
  column3 varchar(500) NOT NULL,
  column4 varchar(500) NOT NULL,
  column5 varchar(500) NOT NULL,
  column6 varchar(500) NOT NULL,
  KEY `index` (column1, column2, column3, column4, column5, column6)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

UTF8 is multi-bytes, and key length is calculated in this way - 500 * 3 * 6 = 9000 bytes.

But note, next query works!

CREATE TABLE IF NOT EXISTS test_table1 (
  column1 varchar(500) NOT NULL,
  column2 varchar(500) NOT NULL,
  column3 varchar(500) NOT NULL,
  column4 varchar(500) NOT NULL,
  column5 varchar(500) NOT NULL,
  column6 varchar(500) NOT NULL,
  KEY `index` (column1, column2, column3, column4, column5, column6)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

...because I used CHARSET=latin1, in this case key length is 500 * 6 = 3000 bytes.

Gerhardine

Thanks for reply, this works, but at cost of giving up utf8 charset. Can this restriction somehow be overcome (I have full server access), is it there for a good reason? – Cayman

This is terrible advice, please learn what charsets are and how this will cause major issues in the future. A mixed charset database not only causes issues when using joins or sub-selects, it puts your data into a non-normalised format and can be near impossible to correct later. – Shopwindow

I moved the charset to latin1, got it working and than changed it again to utf-8 – Heinrick

Answer 3

I had this issue, and solved by following:

Cause

There is a known bug with MySQL related to MyISAM, the UTF8 character set and indexes that you can check here.

Resolution

• Make sure MySQL is configured with the InnoDB storage engine.

• Change the storage engine used by default so that new tables will always be created appropriately:

  set GLOBAL storage_engine='InnoDb';

• For MySQL 5.6 and later, use the following:

  SET GLOBAL default_storage_engine = 'InnoDB';

• And finally make sure that you're following the instructions provided in Migrating to MySQL.

Reference

Sapless

In most cases, we forget to configure the storage engine as 'InnoDB'. The following answer is the most simple one and probably will solve the problem for most of the users here. Thanks. – Vociferation

Answer 4

run this query before creating or altering table.

SET @@global.innodb_large_prefix = 1;

this will set max key length to 3072 bytes

Typhoon

FWIW, MySQL 8.0 removed this variable because it is always enabled in that version. There is no more need for it as a variable. – Granthem

Answer 5

if you are using Laravel 7 or Laravel 8, goto to config/database.php

'engine' => 'innoDb',

that should work especially using Wamp or Xampp.

Tissue

In my scenario, this did the trick. I have utf8mb4 database with utf8mb4_general_ci collation using InnoDB engine and my config was null. Changing engine to innodb I was able to run migrations. – Schopenhauerism

this worked for me, i was running laravel 8 – Andro

Answer 6

I was facing the same issue; used the below query to resolve it.

While creating the DB you can use utf-8 encoding.

eg. create database my_db character set utf8 collate utf8mb4;

EDIT:

(Considering suggestions from comments)

Changed utf8_bin to utf8mb4

Unison

This pointed me in the right direction. For me I had to change my collation to: utf8_general_ci – Disadvantageous

This is NOT the right direction, do not do this! MySQL's utf8 is NOT utf8, it is a bugged proprietary format that should never have come to fruition. utf8mb4 is true utf8 and is the recommended default for proper utf8 support. – Shopwindow

utf8mb4 is the correct encoding to use on mysql and not utf8 – Spunk

Answer 7

This index size limit seems to be larger on 64 bit builds of MySQL.

I was hitting this limitation trying to dump our dev database and load it on a local VMWare virt. Finally I realized that the remote dev server was 64 bit and I had created a 32 bit virt. I just created a 64 bit virt and I was able to load the database locally.

Pseudohemophilia

Answer 8

Well , I just changed from MyISAM to InnoDB like this

Before changing

ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

After changing

ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Euphoria

Answer 9

I have just made bypass this error by just changing the values of the "length" in the original database to the total of around "1000" by changing its structure, and then exporting the same, to the server. :)

Bina

Answer 10

See three kinds of solutions of this error here or follow this as well

Getting the same exception while running back-end application in this case we can set out engine "InnoDb" set GLOBAL storage_engine='InnoDb';

if the above configuration is not working for you then change your key length for example if it is by default selected 255 char you can change it below of 100 Example - create table role ( role_name varchar(99) not null, role_description varchar(255), primary key (role_name)); it is works for me

Schappe

Answer 11

I had this error and I changed my tables column length smaller for the indexed foreign key columns so I changed it like this:

VARCHAR(1024)

To:

VARCHAR(512)

And run the query again.

Aplanatic

Answer 12

I have gone through so many tricks, but none worked for me. Then I found the best and simplest one:

If you are working with mysql-workbench, while altering index select (Engine:)-innoDb.

Quadrinomial

Answer 13

If the sql is not created yourself, maybe just check and set the length of column to usual length.

Tini

Answer 14

I had this error and I made another indexing command for the userEmail in a separate command;

CREATE INDEX users_index_v1 ON users (admin,testingAccount,deleted,userEmail,fname,lname, country);
ERROR 1071 (42000): Specified key was too long; max key length is 1000 bytes

i just remove userEmail from first statement and retype it separately :

CREATE INDEX users_index_email ON users (userEmail);

Yoshikoyoshio