Limit list length in redis

Question

I'm using redis lists and pushing to new items to a list. The problem is I really only need the most recent 10 items in a list.

I'm using lpush to add items to a list and lrange to get the most recent 10.

Is there anyway to drop items after a certain number? I'll end up with lists that may have 1,000's of items and can cause performance issues with latency.

Thank you!

Rang

How are you interfacing with Redis? CLI? Python? C#? – Jacindajacinta

It should be something like LTRIM list -10 -1 just like I answered detailed below. – Aphrodisia

Answer 1

After every lpush, call ltrim to trim the list to 10 elements

See http://redis.io/commands/ltrim

Ducan

Wouldn't retrieving the current list length and only inserting if the length is below 10 be a better performing solution for large entities? – Conjuncture

@bornSwift, wouldn't your solution keep the 10 earliest entries instead of the 10 most recent? – Clepsydra

@TerryLewis, yes good point. Didn't see the "most recent" in the question.. oops – Conjuncture

Answer 2

You can use LTRIM intermittently after any LPUSH, no need to call LTRIM after every LPUSH as that would add to overall latency in your app ( though redis is really fast, but you can save lots of LPUSH operations )

Here is a pseudo code to achieve an LTRIM on approximately every 5th LPUSH:

LPUSH mylist 1
random_int = some random number between 1-5
if random_int == 1:  # trim my list with 1/5 chance
   LTRIM mylist 0 10

Though your list may grow to be a few elements more than 10 elements at times, but it will surely get truncated at regular intervals. This approach is good for most practical purposes and saves a lot of LTRIM operations, keeping your pushes fast.

Ordonnance

You can almost smell when a piece of advice comes straight from someone's experiences in production ;) – Atonement

How can you be sure that getting a random number and checking if it is something isn't more expensive than an LTRIM call? – Ferdelance

@Ferdelance Mostly because of LTRIM being an over the network operation, however if both operations are part of a single Lua script, we might have to benchmark. However, I do strongly feel, that the random number operation is going to be simpler than LTRIM in both cases. – Ordonnance

@Ordonnance If one is using a language/library that supports pipelining, then it is not necessary to write a script to save the round trip time. – Charest

Answer 3

The following code,

• pushes the item to the list,
• keep the size fixed to 10,
• and returns the most recent 10 elements

in a transaction.

MULTI
LPUSH list "item1"
LTRIM list 0 9
LRANGE list 0 9
EXEC

Anecdotage

@PaulT.Rawkeen why? it can be achieved with Reids builtin commands (wrapped in transaction), I don't see a reason to make it a lua script. – Farandole

Answer 4

No one has ever mentioned the real solution about storing only most 10 recent items.

Let's create a sample list with 15 items (here just numbers):

RPUSH list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Now indicate offset from the end of the list:

LTRIM list -10 -1

Show list

LRANGE list 0 -1

 1) "6"
 2) "7"
 3) "8"
 4) "9"
 5) "10"
 6) "11"
 7) "12"
 8) "13"
 9) "14"
10) "15"

Now you can add new items and run trim:

RPUSH list 16
LTRIM list -10 -1

 1) "7"
 2) "8"
 3) "9"
 4) "10"
 5) "11"
 6) "12"
 7) "13"
 8) "14"
 9) "15"
10) "16"

Aphrodisia

Answer 5

Just an alternative. According to official doc of LPUSH, it returns the length of the list after the push operations. You can set a threshold length like k (in your case k > 10) and call LTRIM when returned length is bigger than k. Sample pseudo code as follows:

len = LPUSH mylist xxx 
if len > k:   
  LTRIM mylist 0 9
LRANGE mylist 0 9

It's more controllable than random method. Greater k triggers less LTRIM but with more memory cost. You can adjust k according to how often you want to call LTRIM since calling extra command is more expensive.

Thanatopsis

This is a very smart implementing, great suggestion! – Tertia

Answer 6

Calling LTRIM <list-name> -1 -10 after LPUSH <list-name> <item> is the simplest answer. Many had covered it.

You must do this two operations in a transaction or must use Lua script to ensure the operation is atomic.

Claraclarabella

Why a Lua script specifically? – Tavia

Lua ensures the operation is atomic. On code level, if you are using Lua for some other work, using Lua instead of transaction is simpler – Claraclarabella