Generating heatmap layer for milions of points

Question

I'm using heatmap layer from Google Maps to display a heatmap, however, I now have too many points and it stopped working, because browser cannot handle it anymore. I've found that they provide Fusion Tables, but they're also limited: to 100k rows, which is way too low. I need to render heatmap of milions or maybe even more points. I'd be perfect if my server can have some PHP script to render a heatmap, for example, once a day. And then a client from js will just download a preloaded heatmap (on the map like google maps, but may be different map too). Is this possible with some existing technology (can be commercial)?

Nano

Do you really need to show ALL points at ALL times? Could you not load only the points that fall within the viewport, maybe limit the minimum zoom level, the map container dimensions, etc.? – Us

Yes and no. For now, all points are around one city and I want to see the whole city (about 1.5 mil points as for today). Later on, it'll be available for other cities so then I can think of it, but as I said it's still too much for one city. I need prerendered heatmap :) – Nano

Mapbox pretends you can load millions of points on a map. Have a look at their blog. I found this post and this post. Might be worth trying. – Us

You can add an image overlay. Here is a link to the google maps developer documentation: developers.google.com/maps/documentation/javascript/… Pre-generate heat maps and add to the javascript as an overlay. – Groundless

I've tried MapBox: mapbox.com/mapbox-gl-js/example/heatmap-layer but it loads points from json file. Well it could barely work with 100 000 points after really long loading. – Nano

Geo heat map(google) with large data this discussion should help! – Pensionary

For fast heatmap generation of million points use something like WebGL Heatmap which accomplishes high-performance by harness power of video-card through GL/pixel shaders – Taynatayra

Consider looking at caching tiles of data on something like geoserver that way you can have a long running task update the layer on the server and still maintain an interactive map which uses that layer. – Lubet

Answer 1

All you need is to pre-clusterize your points into smaller number of points and pass these groups to Google Maps as if it were your original points. So, if you had 3 nearby points with values 3, 5 and 10, you create one point with value 18 at weighted average of their coordinates. Do it for all your dataset and you will reduce it 3 times. Change 3 to any appropriate value to further reduce your dataset.

The easy way to cluster your data is to use geohashing[1]. Here is nice geohashing library for PHP[2]. You may calculate a set of geohashes of different precision for each of your points only once when adding, and then use desired precision to reduce your dataset using simple GROUP BY. Example (meta):

use Lvht\GeoHash;

class DataPoint extends ActiveRecord {
    public geo_hash_precision4, geo_hash_precision5, geo_hash_precision6;
    public function save() {
        $this->geo_hash_precision4 = GeoHash::encode($this->lat,$this->lon, 0.0001);
        $this->geo_hash_precision5 = GeoHash::encode($this->lat,$this->lon, 0.00001);
        $this->geo_hash_precision6 = GeoHash::encode($this->lat,$this->lon, 0.000001);
        parent::save();
    }
}

class DataSet extends ActiveQuery {
    /**
     * @param int $p Desired precision
     */
    public function getValues($p = 6) {
        $data = $this->rawQuery("SELECT geo_hash_precision{$p}, SUM(value) FROM {$this->table} GROUP BY geo_hash_precision{$p}");
        // Add bounding box WHERE to reduce set for map size
        foreach ($data as $row) {
            list($minLon, $maxLon, $minLat, $maxLat) = GeoHash::decode($row["geo_hash_precision{$p}"]);
            $row['lat'] = ($maxLat - $minLat) / 2;
            $row['lon'] = ($maxLon - $minLon) / 2;
            unset($row["geo_hash_precision{$p}"]);
        }
    }
}

1. https://en.wikipedia.org/wiki/Geohash
2. https://github.com/lvht/geohash

Hanseatic

I don't have a possibility to test this answer right now, but it looks like it's the best solution for my problem :) – Nano

Please be advised that this is kinda pseudocode to get you an idea, real implementation will vary according to your db, ORM and framework. Feel free to ask additional questions. – Hanseatic

Answer 2

I use heatmap.js for visualization. It's really fast and can handle alot of points. Not sure though if 1.5 mil points will work.

What I can imagine is to use one of the Javascript solutions with node.js to prerender. A quick google search about 'heatmap js nodejs prerender' found me this https://github.com/substack/node-heatmap and this https://mango-is.com/blog/engineering/pre-render-d3-js-charts-at-server-side/

Diametral

It can be tens of milions of points in the future. Can it be run on php server on cron task? – Nano

Well nodejs can be installed on a linux server and surely run via cron task. The only PHP heatmap server-side rendering I found was fusion-tables, which you already mentioned does not meet your requirement. So setting up a small nodejs script and try generating a heatmap with that might be your best option. – Diametral