Python/psycopg2 WHERE IN statement

Question

What is the correct method to have the list (countryList) be available via %s in the SQL statement?

# using psycopg2
countryList=['UK','France']

sql='SELECT * from countries WHERE country IN (%s)'
data=[countryList]
cur.execute(sql,data)

As it is now, it errors out after trying to run "WHERE country in (ARRAY[...])". Is there a way to do this other than through string manipulation?

Thanks

Rimskykorsakov

Answer 1

For the IN operator, you want a tuple instead of list, and remove parentheses from the SQL string.

# using psycopg2
data=('UK','France')

sql='SELECT * from countries WHERE country IN %s'
cur.execute(sql,(data,))

During debugging you can check that the SQL is built correctly with

cur.mogrify(sql, (data,))

Submissive

If you're having trouble even after reading this answer re-read it again very slowly. It's a tuple of tuples and you must remove the parans around the %s if you have them there. This tripped me up because a simpler test of mine used only a single value and everything worked. Just follow this exactly as Bryan wrote it out. – Larsen

cur.mogrify is a great tip. I keep forgetting about that one... 😅 – Begot

I got error 'psycopg.errors.SyntaxError: syntax error at or near "$1" LINE 1: SELECT * from countries WHERE country IN $1' – Slowpoke

Use ANY instead. See that doc: psycopg.org/psycopg3/docs/basic/… – Slowpoke

Please edit this answer as it is not valid for psycopg >= 3.0 - see dzav's comments or mjuopperi's answer – Miele

Answer 2

To expland on the answer a little and to address named parameters, and converting lists to tuples:

countryList = ['UK', 'France']

sql = 'SELECT * from countries WHERE country IN %(countryList)s'

cur.execute(sql, { # You can pass a dict for named parameters rather than a tuple. Makes debugging hella easier.
    'countryList': tuple(countryList), # Converts the list to a tuple.
})

Hanako

Thank you for the tip on passing in a dict. That is much much better. – Downes

Can this be implemented with multiple WHERE x IN clauses and multiple lists in the dictionary? – Saucy

Correction: @Saucy Yes, with the almighty OR. Eg: cur.execute("SELECT * FROM table WHERE col IN %(list1)s OR col IN %(list2)s", {'list1': tuple(1,2,3), 'list2' = tuple(4,5,6)}) – Hanako

Warning, this will throw a postgres syntax error if the list / tuple is empty. Alternatively, you may use a WHERE ANY statement as shown in other answers. – Luxembourg

Answer 3

You could use a python list directly as below. It acts like the IN operator in SQL and also handles a blank list without throwing any error.

data=['UK','France']
sql='SELECT * from countries WHERE country = ANY (%s)'
cur.execute(sql,(data,))

source: http://initd.org/psycopg/docs/usage.html#lists-adaptation

Limewater

Can also use accepted answer but with cur.execute(sql, (tuple(data),)) – Stjohn

Answer 4

Since the psycopg3 question was marked as a duplicate, I'll add the answer to that here too.

In psycopg3, you can not use in %s with a tuple, like you could in psycopg2. Instead you have to use ANY() and wrap your list inside another list:

conn.execute("SELECT * FROM foo WHERE id = ANY(%s)", [[10,20,30]])

Docs: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html#you-cannot-use-in-s-with-a-tuple

Planarian

More on this topic can be found here: psycopg.org/psycopg3/docs/basic/from_pg2.html – Whitsun

Answer 5

I'm using 'row_factory': dict_row and the following works for me:

SQL: ... where col_name = ANY(%(col_name)s);

Python: connection.execute(sql, {'col_name': ['xx',...,'yy']})

Unfreeze

This solution is already provided by existing answers (using a dict for params is a trivial difference) – Soulsearching

@Soulsearching No, my answer differs – Unfreeze

How does it differ? This and this both seem identical save for the style of parameter substitution. – Soulsearching

@Soulsearching Both solution uses (%s) placeholder and raw array with values, while my solution uses named placeholder column_name and dict with a column name key and array values – Unfreeze