generate server-side google charts as images

Question

I've just started using the (new) Google Charts api and its pretty cool. I'm successfully creating charts using this api.

I have a throughput challenge however. In my application I generate three charts from real-time data I'm pulling down from NOAA. The time it takes to get the data, massage it into chart form and then draw the chart client-side is an intolerably slow user experience.

So my thought was to generate the chart periodically (every 15-30 min) on the (hosted) server and then just serve up an image of the most recent to visitors.

I looked at phantomjs (as recommended in this post), but it looks like its an .exe file is used and I can't upload that to my shared host.

There's also this thread for a proprietary solution (Highcharts), but I want to explore open source alternatives first before going down the Highcharts path.

Other solutions focus on allowing the user to save a rendered chart as an image, but my goal is to never render the chart in the browser or have any server load other than including an image at the time of page request.

I just haven't seen anything that handles dynamically generated charts that are "automatically" converted into an image that is "automatically" served when the page is rendered.

In summary, here are the three pieces I am trying to cobble together:

1) pulling data from a third party (NOAA in this case) and rendering data as a Google Chart (done, no issues here) 2) converting each rendered chart into an image automatically, server side and creating image urls 3) sticking the image URL of the chart (which will be refreshed frequently) into the html of the web-page before rendering (via php)

P.S. its ok to have a static url for each chart image...I'm not creating an archive of images...

Any recommendations? Am I missing something?

Elviselvish

I would use caching. Each time the page is loaded, check if there is a cache or if the cached version is too old; if so, generate a new one. This is the only option for shared. But are you more interested in actually how to convert this chart to an image? If so what format is the chart? SVG? – Hie

I've not used caching so forgive my naive questions. Are you thinking server-side? So the first user to request the page would still have the performance hit, but subsequent users (theoretically) would not? As for the second part of your comment, the Google Chart api natively creates an SVG chart. I want the page that's rendered to the user to include an image that was created asynchronously to the users page request. – Elviselvish

The Visualization API is entirely client-side, so you would have to render the chart in a browser and then send the image data to the server to save. Most charts render quite quickly, however, so I would try caching your data query to NOAA and sending the data to the client to render the chart before looking into static image solutions. – Heraclid

@Heraclid - thanks for the suggestion! your comment prompted me to investigate the relative elapsed times for each process. Indeed, downloading and massaging data from NOAA was taking about 3100 msecs while the actual downloading of the google chart js and rendering the chart was about 400 msec! So I'll turn my attention to data acquisition vs. transformation! If you'll write your comment up as an answer, I'll accept it! – Elviselvish

Update: I've implemented asgallant's suggestion and its cut the page load time from 6+ secs to around 3 secs. The next 50% improvement (cutting the page load time to 1.5 secs) would come if I could eliminate the real-time rendering of the charts... – Elviselvish

What is the scale of data that are you dealing with (1000 records, 10000 records, 100000 records, or more)? – Heraclid

The raw data is less than 500 text rows as the data is recorded by NOAA in 6 minute intervals and keeps 48 hours of data live. Each of the ~500 rows has 13 values. Each line chart (and there are 3) has three or four of the data elements (an x-axis set of labels and 2 or 3 lines). Why do you ask? – Elviselvish

Answer 1

Use Watir to load the web page and save the images. Watir is a Ruby-based test platform aimed at testing web pages. I would do the following.

1. Create a web page that renders the charts. It would not be for public use, only for generating the chart images.
2. Add Javascript that calls getImageURI() for each chart to get the PNG data.
3. Use Ajax to call a PHP script on the server. Pass in the PNG data and save it to a file.
4. Write a script in Watir that simply opens a browser and loads your web page.
5. Schedule the Watir script to run as often as you desire.

The Watir script can run on any local computer (PC or Mac) or even on a server.

Here's Javascript to send the PNG data to a PHP script via Ajax. It is just the snippet that gets the image data and sends it, not a complete solution.

// Instantiate and draw our chart, passing in some options.
var chart = new google.visualization.PieChart(chartDiv);
// Wait for the chart to finish drawing before calling the getImageURI() method.
google.visualization.events.addListener(chart, 'ready', function () {
    pieDone=1;
    piePNG=chart.getImageURI();
    saveChartPNGs(piePNG); 
});
chart.draw(data, options);

function saveChartPNGs(png)
{
  var jsonSavePNG = $.ajax({
    type: "POST",
    url: pngSaveUrl,
    dataType:"json",
    data: 'png=' + png
  }).done();
}

This PHP code processes the Ajax call and saves the file.

if (isset($_REQUEST['png']))
{
    // Remove header string
    $data=str_replace('data:image/png;base64,','',$_REQUEST['png']);
    // Restore data to original format; convert space to '+'
    $data=str_replace(' ','+',$data);
    // Save PNG file
    $outFile=DOC_ROOT.'/atest.png';
    // Decode base 64 before saving
    $fres=file_put_contents($outFile, base64_decode($data));
}

On your local computer, the Watir script (actually written in Ruby) is simple.

# Load watir
require 'watir-webdriver'
require "watir-webdriver/extensions/alerts"

# Launch the browser; common choices: chrome, firefox, ie.
# Headless browsers are available.
browser = Watir::Browser.new :chrome # chrome browser window

# Load the web page
browser.goto 'http://domain.net/page/'

To fully automate the Watir side I would write the chart rendering web page to load a new page when it has finished saving the files. You can have Watir check for that page in the browser and then exit until it executes again.

If you can arrange to install Ruby, Watir and a browser on your server, then you can automate this entire scenario with a cron job.

Fertilization

You say watir can run on a server, but that seems to make some assumptions, for instance that it has a GUI that can run Chrome or other web browsers? For many headless servers this isn't true. – Clarino

@Michael, I don't know what a headless server is, but watir can launch a headless browser, which means no window displayed. So having a GUI on the server (like X Windows) is not necessary. – Fertilization

Answer 2

I've just released a relevant open-source project Google Charts Node that renders chart images in Puppeteer, a headless Chromium browser.

It can be used either as an NPM library or as a web service. Here's an example of using the NPM library:

const GoogleChartsNode = require('google-charts-node');

function drawChart() {
  // Set up your chart here, just like in the browser
  // ...

  const chart = new google.visualization.BarChart(container);
  chart.draw(data, options);
}

// Render the chart to image
const image = await GoogleChartsNode.render(drawChart);

Your question was tagged PHP, so I will note that it's also available as a hosted web service that can be used in any language. You can read more here.

For example:

// Generate the image
$url = 'https://quickchart.io/google-charts/render';
$data = array(
  'width' => 600,
  'height' => 300,
  'packages' => array('gannt'),
  'code' => 'function drawChart() {...}'
);

$postdata = json_encode($data);

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
$result = curl_exec($ch);
curl_close($ch);

// Write it to file
$fp = fopen('/tmp/image.png','x');
fwrite($fp, $raw);
fclose($fp);

Now you can save this image buffer as a file or return it as an HTTP response, etc.

The main caveats are:

1. Not all charts support getImageURI, so the library makes puppeteer take a screenshot when this happens.
2. It's slow! But if you must use Google Charts as a requirement, you don't really have an alternative. If you're able to switch chart libraries, I recommend an open format like Chart.js via QuickChart.

You can view the full source for generating server-side Google Charts at the Github project.

Aaberg