how to import a ".csv " data-file into the Redis database

Question

How do I import a .csv data-file into the Redis database? The .csv file has "id, time, latitude, longitude" columns in it. Can you suggest to me the best possible way to import the CSV file and be able to perform spatial queries?

Abscise

Answer 1

This is a very broad question, because we don't know what data structure you want to have. What queries do you expect, etc. In order to solve your question you need:

1. Write down expected queries. Write down expected partitions. Is this file your complete dataset?

2. Write down your data structure. It will heavily depend on answers from p1.

3. Pick any (scripting) language you are most comfortable with. Load your file, process it in CSV library, map to your data structure from p2, push to Redis. You can do the latter with client library or with redis-cli.

The if for example, you want to lay your data in sorted sets where your id is zset's key, timestamp is score and lat,lon is the payload, you can do this:

$ cat data.csv
id1,1528961481,45.0,45.0
id1,1528961482,45.1,45.1
id2,1528961483,50.0,50.0
id2,1528961484,50.1,50.0

cat data.csv | awk -F "," '{print $1" "$2" "$3" "$4}' | xargs -n4 sh -c 'redis-cli -p 6370 zadd $1 $2 "$3,$4"' sh

127.0.0.1:6370> zrange id2 0 -1
1) "50.0,50.0"
2) "50.1,50.0"

Lichfield