Does PostgreSQL support "accent insensitive" collations?
Asked Answered
P

3

156

In Microsoft SQL Server, it's possible to specify an "accent insensitive" collation (for a database, table or column), which means that it's possible for a query like

SELECT * FROM users WHERE name LIKE 'João'

to find a row with a Joao name.

I know that it's possible to strip accents from strings in PostgreSQL using the unaccent_string contrib function, but I'm wondering if PostgreSQL supports these "accent insensitive" collations so the SELECT above would work.

Pickmeup answered 12/6, 2012 at 21:19 Comment(2)
See this answer for creating an FTS dictionary with unaccent: https://mcmap.net/q/28685/-does-postgresql-support-quot-accent-insensitive-quot-collationsRumble
Do you want case-sensitivity or case insensitive searches?Rumble
C
311

Update for Postgres 12 or later

Postgres 12 adds nondeterministic ICU collations, enabling case-insensitive and accent-insensitive grouping and ordering. The manual:

ICU locales can only be used if support for ICU was configured when PostgreSQL was built.

If so, this works for you:

CREATE COLLATION ignore_accent (provider = icu, locale = 'und-u-ks-level1-kc-true', deterministic = false);

CREATE INDEX users_name_ignore_accent_idx ON users(name COLLATE ignore_accent);

SELECT * FROM users WHERE name = 'João' COLLATE ignore_accent;

fiddle

Read the manual for details. This blog post by Laurenz Albe may help to understand.

But ICU collations also have drawbacks. The manual:

[...] they also have some drawbacks. Foremost, their use leads to a performance penalty. Note, in particular, that B-tree cannot use deduplication with indexes that use a nondeterministic collation. Also, certain operations are not possible with nondeterministic collations, such as pattern matching operations. Therefore, they should be used only in cases where they are specifically wanted.

My "legacy" solution is typically still superior:

For all versions

Use the unaccent module for that - which is completely different from what you are linking to.

unaccent is a text search dictionary that removes accents (diacritic signs) from lexemes.

Install once per database with:

CREATE EXTENSION unaccent;

If you get an error like:

ERROR: could not open extension control file
"/usr/share/postgresql/<version>/extension/unaccent.control": No such file or directory

Install the contrib package on your database server like instructed in this related answer:

Among other things, it provides the function unaccent() you can use with your example (where LIKE seems not needed).

SELECT *
FROM   users
WHERE  unaccent(name) = unaccent('João');

Index

To use an index for that kind of query, create an index on the expression. However, Postgres only accepts IMMUTABLE functions for indexes. If a function can return a different result for the same input, the index could silently break.

unaccent() only STABLE not IMMUTABLE

Unfortunately, unaccent() is only STABLE, not IMMUTABLE. According to this thread on pgsql-bugs, this is due to three reasons:

  1. It depends on the behavior of a dictionary.
  2. There is no hard-wired connection to this dictionary.
  3. It therefore also depends on the current search_path, which can change easily.

Some tutorials on the web instruct to just alter the function volatility to IMMUTABLE. This brute-force method can break under certain conditions.

Others suggest a simple IMMUTABLE wrapper function (like I did myself in the past).

There is an ongoing debate whether to make the variant with two parameters IMMUTABLE which declares the used dictionary explicitly. Read here or here.

Another alternative would be this module with an IMMUTABLE unaccent() function by Musicbrainz, provided on Github. Haven't tested it myself. I think I have come up with a better idea:

Best for now

This approach is more efficient than other solutions floating around, and safer.
Create an IMMUTABLE SQL wrapper function executing the two-parameter form with hard-wired, schema-qualified function and dictionary.

Since nesting a non-immutable function would disable function inlining, base it on a copy of the C-function, (fake) declared IMMUTABLE as well. Its only purpose is to be used in the SQL function wrapper. Not meant to be used on its own.

The sophistication is needed as there is no way to hard-wire the dictionary in the declaration of the C function. (Would require to hack the C code itself.) The SQL wrapper function does that and allows both function inlining and expression indexes.

CREATE OR REPLACE FUNCTION public.immutable_unaccent(regdictionary, text)
  RETURNS text
  LANGUAGE c IMMUTABLE PARALLEL SAFE STRICT AS
'$libdir/unaccent', 'unaccent_dict';

Then:

CREATE OR REPLACE FUNCTION public.f_unaccent(text)
  RETURNS text
  LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT AS
$func$
SELECT public.immutable_unaccent(regdictionary 'public.unaccent', $1)
$func$;

In Postgres 14 or later, an SQL-standard function is slightly cheaper, yet. Using the short form for a single statement:

CREATE OR REPLACE FUNCTION public.f_unaccent(text)
  RETURNS text
  LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT
RETURN public.immutable_unaccent(regdictionary 'public.unaccent', $1);

See:

Drop PARALLEL SAFE from both functions for Postgres 9.5 or older.

public being the schema where you installed the extension (public is the default).

The explicit type declaration (regdictionary) defends against hypothetical attacks with overloaded variants of the function by malicious users.

Previously, I advocated a wrapper function based on the STABLE function unaccent() shipped with the unaccent module. That disabled function inlining. This version executes ten times faster than the simple wrapper function I had here earlier.
And that was already twice as fast as the first version which added SET search_path = public, pg_temp to the function - until I discovered that the dictionary can be schema-qualified, too. Still (Postgres 12) not too obvious from documentation.

If you lack the necessary privileges to create C functions, you are back to the second best implementation: An IMMUTABLE function wrapper around the STABLE unaccent() function provided by the module:

CREATE OR REPLACE FUNCTION public.f_unaccent(text)
  RETURNS text
  LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT AS
$func$
SELECT public.unaccent('public.unaccent', $1)  -- schema-qualify function and dictionary
$func$;

Finally, the expression index to make queries fast:

CREATE INDEX users_unaccent_name_idx ON users(public.f_unaccent(name));

Remember to recreate indexes involving this function after any change to function or dictionary, like an in-place major release upgrade that would not recreate indexes. Recent major releases all had updates for the unaccent module.

Adapt queries to match the index (so the query planner will use it):

SELECT * FROM users
WHERE  f_unaccent(name) = f_unaccent('João');

We don't need the function in the expression to the right of the operator. There we can also supply unaccented strings like 'Joao' directly.

The faster function does not translate to much faster queries using the expression index. Index look-ups operate on pre-computed values and are very fast either way. But index maintenance and queries not using the index benefit. And access methods like bitmap index scans may have to recheck values in the heap (the main relation), which involves executing the underlying function. See:

Security for client programs has been tightened with Postgres 10.3 / 9.6.8 etc. You need to schema-qualify function and dictionary name as demonstrated when used in any indexes. See:

Ligatures

In Postgres 9.5 or older ligatures like 'Œ' or 'ß' have to be expanded manually (if you need that), since unaccent() always substitutes a single letter:

SELECT unaccent('Œ Æ œ æ ß');

unaccent
----------
E A e a S

You will love this update to unaccent in Postgres 9.6:

Extend contrib/unaccent's standard unaccent.rules file to handle all diacritics known to Unicode, and expand ligatures correctly (Thomas Munro, Léonard Benedetti)

Bold emphasis mine. Now we get:

SELECT unaccent('Œ Æ œ æ ß');

unaccent
----------
OE AE oe ae ss

Pattern matching

For LIKE or ILIKE with arbitrary patterns, combine this with the module pg_trgm in PostgreSQL 9.1 or later. Create a trigram GIN (typically preferable) or GIST expression index. Example for GIN:

CREATE INDEX users_unaccent_name_trgm_idx ON users
USING gin (f_unaccent(name) gin_trgm_ops);

Can be used for queries like:

SELECT * FROM users
WHERE  f_unaccent(name) LIKE ('%' || f_unaccent('João') || '%');

GIN and GIST indexes are more expensive (to maintain) than plain B-tree:

There are simpler solutions for just left-anchored patterns. More about pattern matching and performance:

pg_trgm also provides useful operators for "similarity" (%) and "distance" (<->).

Trigram indexes also support simple regular expressions with ~ et al. and case insensitive pattern matching with ILIKE:

Carsoncarstensz answered 13/6, 2012 at 1:51 Comment(15)
In your solution, are indexes used, or would I need to create an index on unaccent(name)?Pickmeup
@ErwinBrandstetter In psql 9.1.4, I get "functions in index expression must be marked IMMUTABLE", because of the unaccent function is STABLE, instead of INMUTABLE. What do you recommend?Goulden
@e3matheus: Feeling guilty for not having tested the previous solution I provided, I investigated and updated my answer with a new and better (IMHO) solution for the problem than what is floating around so far.Carsoncarstensz
Isn't the collation utf8_general_ci the answer for this kind of issues?Hypochlorite
@Med: This would apply to MySQL and the better choice would be utf8_unicode_ci there..Carsoncarstensz
Thank you for such a quality answer, you really covered every possibility. Even after reading the thread, I still don't understand why they didn't make it IMMUTABLE. Unaccenting doesn't depend on a dictionary, all rules are stored in my Postgres directory under share/postgresql/tsearch_data/unaccent.rules, so I'm positive it's the same for every dictionary. And why would it change anyway, maybe I'll have some foreign words in English.Dulosis
Can we use varchar_pattern_ops instead of gin_trgm_ops?Fulani
@Teejay: It depends. Please ask your question in a new question, comments are not the place. You can always link to this one for context.Carsoncarstensz
@DanielSerodio: Consider the update. Should make it much faster, yet.Carsoncarstensz
I wonder if accent-insensitive is now possible with ICU collations.Tekla
@a_horse_with_no_name: I didn't have time to test it out, yet, but that's an intended use case.Carsoncarstensz
@ErwinBrandstetter using pg_dump and pg_restore I get this error: pg_restore: error: could not execute query: ERROR: text search dictionary "public.unaccent" does not exist LINE 2: SELECT public.immutable_unaccent(regdictionary 'public.una... Any idea what I do wrong?Douce
@phlegx: You probably did not install the additional module unaccent (as instructed at the top) in the target database - or installed it to a different schema.Carsoncarstensz
@ErwinBrandstetter my file created with pg_dump (pg_dump -Fc -b -f /tmp/dump.sql db_dev) includes CREATE EXTENSION IF NOT EXISTS unaccent WITH SCHEMA public; DROP EXTENSION unaccent; Why pg_dump adds the drop?Douce
@phlegx: Not sure why. But if the drop comes after the create, that would explain the problem.Carsoncarstensz
R
15

No, PostgreSQL does not support collations in that sense

PostgreSQL does not support collations like that (accent insensitive or not) because no comparison can return equal unless things are binary-equal. This is because internally it would introduce a lot of complexities for things like a hash index. For this reason collations in their strictest sense only affect ordering and not equality.

Workarounds

Full-Text-Search Dictionary that Unaccents lexemes.

For FTS, you can define your own dictionary using unaccent,

CREATE EXTENSION unaccent;

CREATE TEXT SEARCH CONFIGURATION mydict ( COPY = simple );
ALTER TEXT SEARCH CONFIGURATION mydict
  ALTER MAPPING FOR hword, hword_part, word
  WITH unaccent, simple;

Which you can then index with a functional index,

-- Just some sample data...
CREATE TABLE myTable ( myCol )
  AS VALUES ('fóó bar baz'),('qux quz');

-- No index required, but feel free to create one
CREATE INDEX ON myTable
  USING GIST (to_tsvector('mydict', myCol));

You can now query it very simply

SELECT *
FROM myTable
WHERE to_tsvector('mydict', myCol) @@ 'foo & bar'

    mycol    
-------------
 fóó bar baz
(1 row)

See also

Unaccent by itself.

The unaccent module can also be used by itself without FTS-integration, for that check out Erwin's answer

Rumble answered 30/5, 2018 at 1:33 Comment(1)
Note that the opening paragraph here is no longer strictly true as of Postgres 12 which introduced nondeterministic collations. However they're still not supported by pattern matching operators.Fortepiano
B
3

I'm pretty sure PostgreSQL relies on the underlying operating system for collation. It does support creating new collations, and customizing collations. I'm not sure how much work that might be for you, though. (Could be quite a lot.)

Bloodstock answered 12/6, 2012 at 21:54 Comment(1)
New collation support is currently basically limited to wrappers and aliases for operating system locales. It's very basic. There's no support for filter functions, custom comparators, or any of what you'd need for true custom collations.Cholesterol

© 2022 - 2024 — McMap. All rights reserved.