Add a value to a Redis list only if it doesn't already exist in the list?

Question

I'm trying to add a value to a list but only if it hasn't been added yet.

Is there a command to do this or is there a way to test for the existence of a value within a list?

Thanks!

Thallus

Any reason you cannot use Sets for this? (redis.io/commands#set) – Cancellation

@Cancellation blocking pop – Mainstay

Answer 1

I need to do the same. I think about to remove the element from the list and then add it again. If the element is not in the list, redis will return 0, so there is no error

lrem mylist 0 myitem
rpush mylist myitem 

Myrticemyrtie

This must be the selected answer. – Gwynethgwynne

Nice solution. But beware of using it for the FIFO queues as it moves the element to the end of the list – Mainstay

Answer 2

As Tommaso Barbugli mentioned you should use a set instead a list if you need only unique values. see REDIS documentation SADD

redis>  SADD myset "Hello"
(integer) 1
redis>  SADD myset "World"
(integer) 1
redis>  SADD myset "World"
(integer) 0
redis>  SMEMBERS myset
1) "World"
2) "Hello"

If you want to check the presence of a value in the set you may use SISMEMBER

redis>  SADD myset "one"
(integer) 1
redis>  SISMEMBER myset "one"
(integer) 1
redis>  SISMEMBER myset "two"
(integer) 0

Hangnail

is not the same – Mandrel

Answer 3

It looks like you need a set or a sorted set.

Sets have O(1) membership test and enforced uniqueness.

Wey

useless for a fifo list – Mandrel

Answer 4

If you can't use the SETs (in case you want to achieve some blocking POP/PUSH list features) you can use a simple script:

script load 'local exists = false; for idx=1, redis.call("LLEN",KEYS[1]) do if (redis.call("LINDEX", KEYS[1], idx) == ARGV[1]) then exists = true; break; end end; if (not exists) then redis.call("RPUSH", KEYS[1], ARGV[1]) end; return not exists or 0'

This will return the SHA code of the script you've added.

Just call then:

evalsha 3e31bb17571f819bea95ca5eb5747a373c575ad9 1 test-list myval

where

• 3e31bb17571f819bea95ca5eb5747a373c575ad9 (the SHA code of the script you added)
• 1 — is number of parameters (1 is constant for this function)
• test-list — the name of your list
• myval - the value you need to add

it returns 1 if the new item was added or 0 if it was already in the list.

Mainstay

Answer 5

Such feature is available in set using hexistshexists command in redis.

Pyrotechnic

This is unfortunately the best option although it is vulnerable to race conditions. – Agility

@Salgat could wrap it in a Lua script – Impunity

You're right @Impunity a Lua script would be one way around. A bit hacky but provides an atomic guarantee of whatever your action is. – Agility

Answer 6

Checking a list to see if a member exists within it is O(n), which can get quite expensive for big lists and is definitely not ideal. That said, everyone else seems to be giving you alternatives. I'll just tell you how to do what you're asking to do, and assume you have good reasons for doing it the way you're doing it. I'll do it in Python, assuming you have a connection to Redis called r, some list called some_list and some new item to add called new_item:

lst = r.lrange(list_name, -float('Inf'), float('Inf'))
if new_item not in lst:
     r.rpush(list_name, new_item)

Newsy

Wouldn't LRANGE 0 -1 be better for getting the entire list? – Greening

Answer 7

I encountered this problem while adding to a task worker queue, because I wanted to avoid adding many duplicate tasks. Using a Redis set (as many people are suggesting) would be nice, but Redis sets don't have a "blocking pop" like BRPOPLPUSH, so they're not good for task queues.

So, here's my slightly non-ideal solution (in Python):

def pushOnlyNewItemsToList(redis, list_name, items):
    """ Adds only the items that aren't already in the list.
    Though if run simultaneously in multiple threads, there's still a tiny chance of adding duplicate items.
    O(n) on the size of the list."""
    existing_items = set(redis.lrange(list_name,0,-1))
    new_items = set(items).difference(existing_items)
    if new_items:
        redis.lpush(list_name, *new_items)

Note the caveats in the docstring.

If you need to truly guarantee no duplicates, the alternative is to run LREM, LPUSH inside a Redis pipeline, as in 0xAffe's answer. That approach causes less network traffic, but has the downside of reordering the list. It's probably the best general answer if you don't care about list order.

Geyer

Answer 8

As @Eli states, checking for existence in a list is a O(N) operation, which is costly especially if the list is large. I'm faced with a similar problem. For me using a SET is not an option because I need the insert ordered to be guaranteed when removing items from the list. Redis SETS remove/retrieve items in random order, which is a deal breaker for me. What I'm thinking of doing is maintaining a separate HASH in Redis just to check if the item exists already exists before adding it to the list. This means that whenever I add an item in the list, I have to add it to this secondary data structure (the HASH), to support a O(1) lookup to check for existence. I know this is duplicating data, but given the other options the additional space consumption might not seem so bad.

Gomer