Is it possible to use a cassandra table as a basic queue
Asked Answered
A

3

6

Is it possible to use a table in cassandra as a queue, I don't think the strategy I use in mysql works, ie given this table:

create table message_queue(id integer, message varchar(4000), retries int, sending boolean);

We have a transaction that marks the row as "sending", tries to send, and then either deletes the row, or increments the retries count. The transaction ensures that only one server will be attempting to process an item from the message_queue at any one time.

There is an article on datastax that describes the pitfalls and how to get around it, however Im not sure what the impact of having lots of tombstones lying around is, how long do they stay around for?

Aflcio answered 30/7, 2013 at 11:33 Comment(1)
It is possible, but not trivial. To do this you need to leverage partitioning and bucketizing. There is an implementation available as a docker container github.com/paradoxical-io/cassieq (disclaimer I am a contributor on the project)Mudslinger
R
21

Don't do this. Cassandra is a terrible choice as a queue backend unless you are very, very careful. You can read more of the reasons in Jonathan Ellis blog post "Cassandra anti-patterns: Queues and queue-like datasets" (which might be the post you're alluding to). MySQL is also not a great choice for backing a queue, us a real queue product like RabbitMQ, it's great and very easy to use.

The problem with using Cassandra as the storage for a queue is this: every time you delete a message you write a tombstone for that message. Every time you query for the next message Cassandra will have to trawl through those tombstones and deleted messages and try to determine the few that have not been deleted. With any kind of throughput the number of read values versus the number of actual live messages will be hundreds of thousands to one.

Tuning GC grace and other parameters will not help, because that only applies to how long tombstones will hang around after a compaction, and even if you dedicated the CPUs to only run compactions you would still have dead to live rations of tens of thousands or more. And even with a GC grace of zero tombstones will hang around after compactions in some cases.

There are ways to mitigate these effects, and they are outlined in Jonathan's post, but here's a summary (and I don't write this to encourage you to use Cassandra as a queue backend, but because it explains a bit more about Cassandra works, and should help you understand why it's a bad fit for the problem):

To avoid the tombstone problem you cannot keep using the same queue, because it will fill upp with tombstones quicker than compactions can get rid of them and your performance will run straight into a brick wall. If you add a column to the primary key that is deterministic and depends on time you can avoid some of the performance problems, since fewer tombstones have time to build up and Cassandra will be able to completely remove old rows and all their tombstones.

Using a single row per queue also creates a hotspot. A single node will have to handle that queue, and the rest of the nodes will be idle. You might have lots of queues, but chances are that one of them will see much more traffic than the others and that means you get a hotspot. Shard the queues over multiple nodes by adding a second column to the primary key. It can be a hash of the message (for example crc32(message) % 60 would create 60 shards, don't use a too small number). When you want to find the next message you read from all of the shards and pick one of the results, ignoring the others. Ideally you find a way to combine this with something that depends on time, so that you fix that problem too while you're at it.

If you sort your messages after time of arrival (for example with TIMEUUID clustering key) and can somehow keep track of the newest messages that has been delivered, you can do a query to find all messages after that message. That would mean less thrawling through tombstones for Cassandra, but it is no panacea.

Then there's the issue of acknowledgements. I'm not sure if they matter to you, but it looks like you have some kind of locking mechanism in your schema (I'm thinking of the retries and sending columns). This will not work. Until Cassandra 2.0 and it's compare-and-swap features there is no way to make that work correctly. To implement a lock you need to read the value of the column, check if it's not locked, then write that it should now be locked. Even with consistency level ALL another application node can do the same operations at the same time, and both end up thinking that they locked the message. With CAS in Cassandra 2.0 it will be possible to do atomically, but at the cost of performance.

There are a couple of more answers here on StackOverflow about Cassandra and queues, read them (start with this: Table with heavy writes and some reads in Cassandra. Primary key searches taking 30 seconds.

Remise answered 30/7, 2013 at 15:24 Comment(1)
Thanks for your useful/detailed reply. I figured this would be the case, but I wanted to make sure before I end up having to go ahead and come up with an alternative solution.Aflcio
I
2

The grace period can be defined. Per default it is 10 days:

gc_grace_seconds¶

(Default: 864000 [10 days]) Specifies the time to wait before garbage collecting tombstones (deletion markers). The default value allows a great deal of time for consistency to be achieved prior to deletion. In many deployments this interval can be reduced, and in a single-node cluster it can be safely set to zero. When using CLI, use gc_grace instead of gc_grace_seconds.

Taken from the documentation

On a different note, I do not think that implementing a queue pattern in Cassandra is very useful. To prevent your worker to process one entry twice, you need to enforce "ALL" read consistency, which defeats the purpose of distributed database systems. I highly recommend looking at specialized systems like messaging systems which support the queue pattern natively. Take a look at RabbitMQ for instance. You will be up and running in no time.

Inspection answered 30/7, 2013 at 12:14 Comment(3)
It's true that using a product designed for queues would be better, however in this case, the existing product (on MySQL) has one single low throughput queue, its overkill to have to install and maintain another software product for just this one queue.Aflcio
I understand your reservations. If you need to rethink your MySQL queue, I recommend this read: blog.engineyard.com/2011/… RabbitMQ is very easy to install, maintain and understand. It may open up new opportunities for your project.Inspection
ALL consistency is not enough to implement a lock. You can still end up with two application nodes locking the same value (all that ALL gives you is that all nodes agree on the most recent value, but it doesn't guarantee that that value hasn't changed before you write your new value).Remise
F
0

Theo's answer about not using Cassandra for queues is spot on.

Just wanted to add that we have been using Redis sorted sets for our queues and it has been working pretty well. Some of our queues have tens of millions of elements and are accessed hundreds of times per second.

Futile answered 6/12, 2013 at 13:30 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.