redis for logging

Question

I am thinking of using Redis for web application logging purposes. I googled that there are people using this approach by dumping the logs into a Redis queue/list, and then a scheduled worker to write to disk.

http://nosql.mypopescu.com/post/8652869828/another-redis-use-case-centralized-logging

I wish to seek understanding that why not directly use Redis to persist to disk? If I have allocated a small server which Redis will write to, separated from the database, app server, is it feasible to use Redis to persist the logs directly?

I also need help in querying Redis by datetime, user, etc. For example, each log is as follow.

datetime=>2012-03-24 17:45:12
userid=>123
message=>test message
category=>my category

How can I query for results within a datetime range, by a specific user, of a particular category?

Thanks!

Palazzo

Answer 1

You need to keep in mind that Redis is an in-memory database (even if it can persist the data to disk). The data you put in Redis have to fit in memory.

The proposal in the article you mention is about using Redis as a distributed queuing system. Worker processes dequeue the items and write them to disk, so there are not that many items in Redis memory. This design has a flaw: if the worker processes cannot write the data fast enough to disk, Redis memory consumption will explode - so it has to be limited by configuration (Redis maxmemory parameter) or software (trim the queue at insert time, or empty the queue when it is full).

Now your proposal does not really work since all the data you write in Redis will be kept in memory (even if they are persisted to disk by Redis itself).

Another point is you cannot query Redis. Redis is not a relational database, it supports no ad-hoc query mechanism, only commands involving previously defined access paths. If you want to search data with different parameters, you have to anticipate all the possible searches and build the relevant data structures (set, sorted sets, etc ...) at insert time.

Another store (MongoDB, or a relational database) will probably be a much better fit for your use case.

Studnia

I just watched a Redis demo. Peter Cooper said that Redis was designed/implemented for logging(he's close enough to the source to to be authoritative). Therefore, while your comments are true enough some additional reading is required. – Karolinekaroly

Redis was designed to support lloogg (lloogg.com). But the lloogg service is not about keeping everything in memory and then process queries. It is about storing/aggregating data on-the-fly within the right data structures allowing easy access to the data. This is not the same thing. – Studnia

You're completely wrong. Log messages can be compressed to save memory. Even specialized time-series databases have memory and number-of-items limits. Worker can harvest data in time cycles so "everything" is not keep in memory all the time. If fact queues as proxy for databases are a common scenarios. – Ummersen

Answer 2

You can store logs with following structure:

"logs:{category}:{userid}:{datetime}" = message

And then request it as following:

"logs:*:{userid}:{datetime}"

Or

"logs:{category}:*:{datetime}"

Hunsaker

Answer 3

Redis is in memory datastore. Direct persistence of data to disk is possible with Save or BGSAVE command. Persistence (RDB/AOF) is a feature in addition to storage in-memory.

Requirement mentioned is to store logs to disk. Using any of message queues (like RabbitMQ) instead of in-memory datastore should make things simple. (logs will not eat-up memory)

Applications generating logs can publish them on queues and with separate consumers consuming log messages and writing them to disk.

How can I query for results within a datetime range, by a specific user, of a particular category?

Every block of log should be saved as a structure (example for C/C++) something like this:

   struct log{
     long datatime;
     string userId;
     string message;
     string category;
   };

Serialize this structure to string and store it in Redis as value. Keys for such values would be like: key = userId + DELIMITER + category + DELIMITER + datatime

You can have function which gets all the keys back and split them to get list of data for your specific keyword.

Autobus

Is it possible to issue range queries (on datetime) using this approach? – Flybynight

Answer 4

It works quite well if you use a sorted set with timestamp as the score. Drawbacks are the memory issue (as mentioned in other answers) and the manual querying you'll be doing.

I've been playing around with it, in case someone is interested: https://github.com/hugollm/redis-logs-example

Inebriety