How to make multi-file upload widget in Ipython Notebook?

Question

I would like to make a widget in Ipython Notebook 3.x or 4.x (Jupyter, Python 3) for remote file upload which allows a user to select multiple files in the browser's file picker when uploading. Unfortunatelly, I have no clue about the javascript side.

I have found blueimp's widgets, however, I have no idea how to use them inside a notebook.

This is how a single file upload widget is made:

import base64
from __future__ import print_function # py 2.7 compat.
from IPython.html import widgets # Widget definitions.
from IPython.utils.traitlets import Unicode # Traitlet needed to add synced attributes to the widget.

class FileWidget(widgets.DOMWidget):
    _view_name = Unicode('FilePickerView', sync=True)
    value = Unicode(sync=True)
    filename = Unicode(sync=True)

    def __init__(self, **kwargs):
        """Constructor"""
        widgets.DOMWidget.__init__(self, **kwargs) # Call the base.

        # Allow the user to register error callbacks with the following signatures:
        #    callback()
        #    callback(sender)
        self.errors = widgets.CallbackDispatcher(accepted_nargs=[0, 1])

        # Listen for custom msgs
        self.on_msg(self._handle_custom_msg)

    def _handle_custom_msg(self, content):
        """Handle a msg from the front-end.

        Parameters
        ----------
        content: dict
            Content of the msg."""
        if 'event' in content and content['event'] == 'error':
            self.errors()
            self.errors(self)

%%javascript

require(["widgets/js/widget", "widgets/js/manager"], function(widget, manager){

    var FilePickerView = widget.DOMWidgetView.extend({
        render: function(){
            // Render the view.
            this.setElement($('<input />')
                .attr('type', 'file'));
        },

        events: {
            // List of events and their handlers.
            'change': 'handle_file_change',
        },

        handle_file_change: function(evt) { 
            // Handle when the user has changed the file.

            // Retrieve the first (and only!) File from the FileList object
            var file = evt.target.files[0];
            if (file) {

                // Read the file's textual content and set value to those contents.
                var that = this;
                var file_reader = new FileReader();
                file_reader.onload = function(e) {
                    that.model.set('value', e.target.result);
                    that.touch();
                }
                file_reader.readAsText(file);
            } else {

                // The file couldn't be opened.  Send an error msg to the
                // back-end.
                this.send({ 'event': 'error' });
            }

            // Set the filename of the file.
            this.model.set('filename', file.name);
            this.touch();
        },
    });

    // Register the FilePickerView with the widget manager.
    manager.WidgetManager.register_widget_view('FilePickerView', FilePickerView);
});

file_widget = FileWidget()

# Register an event to echo the filename when it has been changed.
def file_loading():
    print("Loading %s" % file_widget.filename)
file_widget.on_trait_change(file_loading, 'filename')

# Register an event to echo the filename and contents when a file
# has been uploaded.
def file_loaded():
    print("Loaded, file contents: %s" % file_widget.value)
file_widget.on_trait_change(file_loaded, 'value')

# Register an event to print an error message when a file could not
# be opened.  Since the error messages are not handled through
# traitlets but instead handled through custom msgs, the registration
# of the handler is different than the two examples above.  Instead
# the API provided by the CallbackDispatcher must be used.
def file_failed():
    print("Could not load file contents of %s" % file_widget.filename)
file_widget.errors.register_callback(file_failed)

file_widget

Ballplayer

I know there is an upload button on the Dashboard. I would like to have it in my own notebook. – Ballplayer

Answer 1

Comments, suggestions and fixes are welcome.

I got inspiration from Jupyter Notebook (4.x) itself from the NotebookList.prototype.handleFilesUpload function of the notebooklist.js file. After reading up on some javascript syntax, I came up with the following:

(Please note that files are uploaded in text mode without checking.)

import base64 # You need it if you define binary uploads
from __future__ import print_function # py 2.7 compat.
import ipywidgets as widgets # Widget definitions.
from traitlets import List, Unicode  # Traitlets needed to add synced attributes to the widget.

class FileWidget(widgets.DOMWidget):
    _view_name = Unicode('FilePickerView').tag(sync=True)
    _view_module = Unicode('filepicker').tag(sync=True)
    filenames = List([], sync=True)
    # values = List(trait=Unicode, sync=True)

    def __init__(self, **kwargs):
        """Constructor"""
        super().__init__(**kwargs)

        # Allow the user to register error callbacks with the following signatures:
        #    callback()
        #    callback(sender)
        self.errors = widgets.CallbackDispatcher(accepted_nargs=[0, 1])

        # Listen for custom msgs
        self.on_msg(self._handle_custom_msg)

    def _handle_custom_msg(self, content):
        """Handle a msg from the front-end.

        Parameters
        ----------
        content: dict
            Content of the msg."""
        if 'event' in content and content['event'] == 'error':
            self.errors()
            self.errors(self)

%%javascript
requirejs.undef('filepicker');

define('filepicker', ["jupyter-js-widgets"], function(widgets) {

    var FilePickerView = widgets.DOMWidgetView.extend({
        render: function(){
            // Render the view using HTML5 multiple file input support.
            this.setElement($('<input class="fileinput" multiple="multiple" name="datafile"  />')
                .attr('type', 'file'));
        },

        events: {
            // List of events and their handlers.
            'change': 'handle_file_change',
        },

        handle_file_change: function(evt) { 
            // Handle when the user has changed the file.

            // Save context (or namespace or whatever this is)
            var that = this;

            // Retrieve the FileList object
            var files = evt.originalEvent.target.files;
            var filenames = [];
            var file_readers = [];
            console.log("Reading files:");

            for (var i = 0; i < files.length; i++) {
                var file = files[i];
                console.log("Filename: " + file.name);
                console.log("Type: " + file.type);
                console.log("Size: " + file.size + " bytes");
                filenames.push(file.name);

                // Read the file's textual content and set value_i to those contents.
                file_readers.push(new FileReader());
                file_readers[i].onload = (function(file, i) {
                    return function(e) {
                        that.model.set('value_' + i, e.target.result);
                        that.touch();
                        console.log("file_" + i + " loaded: " + file.name);
                    };
                })(file, i);

                file_readers[i].readAsText(file);
            }

            // Set the filenames of the files.
            this.model.set('filenames', filenames);
            this.touch();
        },
    });

    // Register the FilePickerView with the widget manager.
    return {
        FilePickerView: FilePickerView
    };
});

file_widget = FileWidget()

def file_loaded(change):
    '''Register an event to save contents when a file has been uploaded.'''
    print(change['new'])
    i = int(change['name'].split('_')[1])
    fname = file_widget.filenames[i]
    print('file_loaded: {}'.format(fname))

def file_loading(change):
    '''Update self.model when user requests a list of files to be uploaded'''
    print(change['new'])
    num = len(change['new'])
    traits = [('value_{}'.format(i), Unicode(sync=True)) for i in range(num)]
    file_widget.add_traits(**dict(traits))
    for i in range(num):
        file_widget.observe(file_loaded, 'value_{}'.format(i))
file_widget.observe(file_loading, names='filenames')

def file_failed():
    print("Could not load some file contents.")
file_widget.errors.register_callback(file_failed)


file_widget

A button with the text Browse... should appear stating how many files are selected. Since print statements are included in the file_loading and file_loaded functions you should see filenames and file contents in the output. Filenames and file types are shown in the console log, as well.

Ballplayer

Answer 2

This issue https://github.com/ipython/ipython/issues/8383 answers your question partially. There is already upload button available in jupyter 4.0 on Dashboard screen. This upload button supports selecting multiple files.

Note that up to date links are located here for upload widget:

https://github.com/ipython/ipywidgets/blob/master/docs/source/examples/File%20Upload%20Widget.ipynb

Also there is a extension available for download and quick installation in your notebooks:

https://github.com/peteut/ipython-file-upload

pip install fileupload

or

pip install git+https://github.com/peteut/ipython-file-upload

Note that the extension is confirmed to work on linux only according to the author.

Nightshirt

Thanks, I am aware of that. I would like to have the widget in my notebook. – Ballplayer

Thanks for the update. The provided links show single file upload mechanisms. I am looking for a multi-file upload widget or hints towards a multi-file upload widget. The question already shows a solution for a single file upload widget. – Ballplayer

Answer 3

There is an even newer approach than fileupload, which I used in the past and works pretty well (posted by @denfromufa) with a natively supported file-upload widget.

import io
import ipywidgets as widgets

widgets.FileUpload(
        accept='.txt',  # Accepted file extension e.g. '.txt', '.pdf', 'image/*', 'image/*,.pdf'
        multiple=True  # True to accept multiple files upload else False
    )

A couple tips:

1. If your notebook has code 'under' the uploader, it will proceed. I use ipython blocking to 'pause' the execution of the rest of the notebook until the file is successfully uploaded... I then have a second button that basically restarts code execution after the user has uploaded the right file(s) and is ready to proceed.
2. The ipywidget uploader differs from [fileupload] (pip install fileupload) in that it pulls the file into python so you can work with it. If you want to drop it into a directory, you have to use some other method to write the value data to a file:

For an fileupload widget 'myupload' you could write an event-driven function that includes something like the following when a file is uploaded:

    # Get the original filename which is the first key in the widget's value dict:
    uploaded_filename = next(iter(myupload.value))
    content = myupload.value[uploaded_filename]['content']
    with open('myfile', 'wb') as f: f.write(content)

Ramsgate