How to do interactive 2D scatter plot zoom / point selection in Vaex?

Question

I saw that it is possible to do it during the demo: https://youtu.be/2Tt0i823-ec?t=769

There, the presenter has a huge dataset, and can quickly zoom in by selecting a rectangle with the mouse.

I also saw the "Interactive Widgets" section of the tutorial: https://docs.vaex.io/en/latest/tutorial.html#Interactive-widgets

However, I was not able to easily replicate that setup. What are the minimal steps to achieve it?

On Ubuntu 19.04 vaex 2.0.2, I have tried:

python3 -m pip install --user vaex scipy pandas vaex-jupyter
jupyter nbextension enable --py widgetsnbextension
jupyter nbextension enable --py bqplot
jupyter nbextension enable --py ipyvolume
jupyter nbextension enable --py ipympl
jupyter nbextension enable --py ipyleaflet
jupyter nbextension enable --py ipyvuetify
jupyter notebook

Then I created a notebook and paste in the notebook:

import vaex
import vaex.jupyter
import numpy as np
import pylab as plt
%matplotlib inline
df = vaex.example()
df.plot_widget(df.x, df.y, f='log1p', backend='bqplot')

but all I get is no graph and the message:

Plot2dDefault(w=None, what='count(*)', x='x', y='y', z=None)

If instead I do:

df.plot(df.x, df.y, f='log1p')

then I do get an plot, but it is just a non-interactive image.

I also tried to git clone the notebook that is the source of the read the docs page: https://github.com/vaexio/vaex/blob/0247f0673c5c0473001b0b66adcbc716560536aa/docs/source/tutorial.ipynb but the result was the same.

My motivation is to find a plotting program that can handle large number of points as mentioned at: Large plot: ~20 million samples, gigabytes of data

Cysticercus

Are you running this from jupyter lab or the classical notebook? – Statampere

@MaartenBreddels I'm not sure, I just typed jupyter notebook as shown on question :-) OK, I just searched and jupyter lab is started with jupyter lab, so I guess classical notebook. – Cysticercus

Is it possible to do a 3D instead of 2D? – Amphitrite

@TiagoMartinsPeres no idea! Give it a try, and if you don't find it, do open a new question and link to it. – Cysticercus

Gonna use Mayavi first because know it's possible (docs.enthought.com/mayavi/mayavi/mlab.html). If there's enough time I'm gonna try it. – Amphitrite

Answer 1

Use virtualenv

Not sure why, but this fixed it. I think it is because the Jupyter executable was on Python 2 and could not find Python 3 extensions.

virtualenv --python=python3 .venv
. .venv/bin/activate
python3 -m pip install vaex scipy pandas vaex-jupyter
jupyter nbextension enable --py widgetsnbextension
jupyter nbextension enable --py bqplot
jupyter nbextension enable --py ipyvolume
jupyter nbextension enable --py ipympl
jupyter nbextension enable --py ipyleaflet
jupyter nbextension enable --py ipyvuetify
jupyter notebook

and inside a new notebook:

import vaex
df = vaex.example()
df.plot_widget(df.x, df.y, f='log1p', backend='bqplot')

and now I see the interactive widget with zoom!

Versions:

pandas==0.25.0
scipy==1.3.0
vaex==2.0.2
vaex-jupyter==0.3.0

Cysticercus

Answer 2

Jupyter widgets exist of front-end code and kernel code, the kernel code (Python in this case) is usually not the problem, if you can import for instance the ipywidgets module, you should be fine.

The biggest problem is making sure the front-end (classical notebook or Jupyter lab) has the matching javascript libraries loaded. Assuming you are working from the classical notebook, the way to debug is as following:

Since the javascript code of ipywidgets libraries are added to the frontend using the nbextension mechanism, you can check if all libraries are enabled and validated using

$ jupyter nbextension list

Which should show all green 'OK' and 'enabled' (unless you explicitly disabled it).

If not properly installed, you may want to do an installation of the front-end code, e.g.:

$ jupyter nbextension install --py --symlink --sys-prefix ipywidgets
$ jupyter nbextension install --py --symlink --sys-prefix ipyvuetify

If for some reason the extension is not automatically enabled (older versions of the notebook), run:

$ jupyter nbextension enable bqplot --py --sys-prefix
$ jupyter nbextension enable ipyvuetify --py --sys-prefix

If things aren't working, you should test which library isn't working, for instance, start with ipywidgets:

import ipywidgets as widgets
widget.FloatSlider()

If this doesn't show a slider, you are in trouble. Continue with the other libraries, and see when it fails. When it fails, check the javascript console (google that for your browser) for error messages, and see if you get hints from that.

Lastly, make sure you use a modern browser, Internet Explorer won't cut it anymore, it's too old. I recommend Chrome/Chromium or Firefox.

Statampere

Oh, I got it to work by using virtualenv. Hard to say the cause, but I think my jupyter command was running in Python 2 which prevented it from finding python3 extensions. Attempting to enable ipywidgets failed with: The Python module ipywidgets is not a valid nbextension, it is missing the _jupyter_nbextension_paths() method. BTW, I think widgetsnbextension is the correct new way. – Cysticercus