Lazy loaded list view in GTK#

Question

I'm looking to display a large dataset via a list view in GTK# and performance is an issue here. I'm currently using a TreeView backed with a ListStore, but adding all my data to the ListStore takes forever. Is there a list view widget of some sort in GTK that supports lazy loading of data? In Winforms, you can use the VirtualMode property of DataGridView to handle this, but I don't see anything of the sort for GTK.

Looming

Also, you don't need a different widget, rather a different implementation of list store. I don't know anything about GTK#, but if you feel like writing code yourself, you could probably adapt Py-gtktree (Python). – Perfuse

The data going into the ListStore is sorted (by address -- it's a hex editor view), but I'm not doing any sorting in the list itself. Looking into Py-gtktree now. Thanks. – Looming

Answer 1

There, as far as I am aware, no widget to do what you want in Gtk, however, you can do something similar, in the end result, to the VirtualMode property, in the TreeView.

The problem with The TreeView control is that it will fetch all data from it's model in advance. If it were not for this then I would suggest a model only approach to this problem, but unfortunately the TreeView is greedy when it comes to fetching the data, so controlling when data is loaded from the view is needed, else how else is it going to be able to tell when a row is visible and thus inform the model, or a proxy to fetch the data for the row as it becomes visible.

You need 3 things to get this to work

1) a model to use in the treeview, which initially has all the rows but no data in any of the fields
2) a way of fetching the data from whatever database you use
3) a means of determining which rows to fetch data for

The first two items can be done on the model level. Determining what rows to fetch needs the Treeview widget and a way of determining what rows are being displayed. The method I use below is not optimal, but it does work, and can be tidied up and/or adapted for whatever use you have in mind.

I am using a proxy class to be stored in the model and is used to fetch data specific to that row. In my example it is called ProxyClass. It fetches and holds the data for a row, which is initially null. In this case the Fetch method just creates and returns a string "some data " + id

This would be held inside a instance of MyNode, which inherits from TreeNode, representing a row of data. the first column returns the data held in the proxy and the second column, which is never shown, holds the proxy class instance.

You then create your NodeStore, your model, populating it with instances of MyNode(id) as you can see below in the example.

The control over when data is loaded is controlled from CellDataFunc. This method is the key to getting this to work. The CellDataFunc is responsible for setting the text in the CellRendererText for a specific column in a row identified by the iterator passed to it. It is called every time the treeview reveals a row and for the newly revealed row only. Thus will only fetch data for cells that are rendered in the display. This gives you a means of controlling when the data is fetched, thus fetching it only when you need it.

You can make the TreeView use the CellDataFunc to load your data as needed by applying it to one of the columns with TreeViewColumn.SetCellDataFunc. You only need to do this on one column, as it can fetch the data for the entire row.

To stop all but the visible rows from having it's data fetched, can be done by checking to see if the current cell is in the visible range or not. To do this you call TreeView.GetVisibleRange(out start,out end) then see if the current iterator passed to this function is within the start and end range , which are TreePath objects, so they need to be changed into TreeIters first. Model.GetIter(out iter_start, start). Then check to see if iter.UserData.ToInt32() >= iter_start.UserData.ToInt32() and less than iter_end. If the current iter falls in the range from iter_start to iter_end then fetch the data, else leave it be.

Here is my example.

The ProxyClass

namespace LazyTree
{

    public class ProxyClass 
    {
      int id;
      string data;

      public ProxyClass (int id)
      {
        this.id = id;
        data = null;
      }


      public void Fetch()
      {
        data = "some data " + id;
      }


      public int Id
      {
        get { return id; }
      }

      public string Data
      {
        get {return data;}
      }
  }
}

The custom TreeNode instance

namespace LazyTree
{
    [Gtk.TreeNode (ListOnly=true)]
    public class MyNode : Gtk.TreeNode
    {
        protected ProxyClass proxy;

        public MyNode (int id)
        {
            proxy = new ProxyClass(id);
        }

        [Gtk.TreeNodeValue (Column=1)]
        public ProxyClass Proxy
        {
            get {return proxy;}
        }

        [Gtk.TreeNodeValue (Column=0)]
        public string Data
        {
            get { return proxy.Data; }
        }
    }
}

The window which includes the scrolled window, and treeview. This is also where the CellDataFunc is defined, although that could be put anywhere.

namespace LazyTree
{

    public class MyWindow : Gtk.Window
    {
        int NUMBER_COLUMNS = 10000;
        Gtk.NodeStore store;
        Gtk.NodeStore Store {
            get {
                if (store == null) {
                    store = new Gtk.NodeStore (typeof (MyNode));
                    for(int i = 0; i < NUMBER_COLUMNS; i++)
                    {
                        store.AddNode (new MyNode (i));
                    }
                }
                return store;
            }
        }


        protected void CellDataFunc(Gtk.TreeViewColumn column,
                                    Gtk.CellRenderer cell,
                                    Gtk.TreeModel model,
                                    Gtk.TreeIter iter)
        {
            try {
                string data = (string)model.GetValue(iter, 0);
                ProxyClass proxy = (ProxyClass)model.GetValue(iter, 1);
                Gtk.TreeView view = (Gtk.TreeView)column.TreeView;
                Gtk.TreePath start, end;
                bool go = view.GetVisibleRange(out start,out end);
                Gtk.TreeIter iter_start, iter_end;
                if(go)
                {
                    model.GetIter(out iter_start, start);
                    model.GetIter(out iter_end, end);
                }
                if (go &&
                    data == null && 
                    iter.UserData.ToInt32() >= iter_start.UserData.ToInt32() &&
                    iter.UserData.ToInt32() <= iter_end.UserData.ToInt32())
                {
                    Console.WriteLine("Lazy Loading " + proxy.Id + ", Visible: " + cell.Visible);
                    proxy.Fetch();
                }

                ((Gtk.CellRendererText)cell).Text = data;
            } catch(Exception e) {
                Console.WriteLine("error: " + e);
            }
        }


        public MyWindow () : base("Lazy Tree")
        {
            Gtk.NodeView view = new Gtk.NodeView(Store);

            Gtk.ScrolledWindow scroll = new Gtk.ScrolledWindow();
            scroll.Add(view);
            Add(scroll);
            Gtk.CellRendererText cell = new Gtk.CellRendererText ();
            view.AppendColumn ("Lazy Data", cell, "text", 0);

            Gtk.TreeViewColumn column = view.GetColumn(0);

            column.SetCellDataFunc(cell, CellDataFunc);
        }


        protected override bool OnDeleteEvent (Gdk.Event ev)
        {
            Gtk.Application.Quit ();
            return true;
        }

        public static void Main()
        {
            Gtk.Application.Init ();
                MyWindow win = new  MyWindow();
            win.SetDefaultSize(200, 200);
                    win.ShowAll ();
            Gtk.Application.Run ();
        }
    }


}

Hope that is what your after.

See the c documentation for a better explanation of what each of the methods, and their parameters do. The Mono docs leave a lot to be desired.

SetCellDataFunc (C docs) http://developer.gnome.org/gtk/stable/GtkTreeViewColumn.html#gtk-tree-view-column-set-cell-data-func

(CeCellDataFunc) http://developer.gnome.org/gtk/stable/GtkTreeViewColumn.html#GtkTreeCellDataFunc

(DestroyFunc) http://developer.gnome.org/glib/unstable/glib-Datasets.html#GDestroyNotify

Cyb

Answer 2

It makes a huge difference if you insert the data into the model while it is hooked up to the view, or you insert all data while it is "offline", and only connect it to the view when you are done. The later is a lot faster, because otherwise the treeview has to regenerate it's internal structures all the time.

If inserting is not the main problem, but fetching the data from your data source is the actual slow part, then what helps a lot is if you can retrieve the amount of rows for the model quickly. If that's the case, then I'd suggest to first create a liststore with all rows allocated with empty contents which you can hook to the view and then fill the actual contents from a idle-callback or thread. Unfortunately there is no batch-api to update the model, so that several rows could be updated at once.

Cholula

Answer 3

Maybe you can add the data using a different thread, so that the current application doesn't "freeze", but just keeps running. It will probably still take the same amount of time, but at least the user can work with the rest of the application in mean time.

Macmahon

Answer 4

Alternately, you can look at implementing your own Gtk.TreeModelImplementor as described at Implementing GInterfaces on the Mono Project website. You can see one example here.

It should be fairly trivial to make such an implementation "lazy".

Vander

Answer 5

In GTK 4 you can use Gtk.ListViews.

You can find an example in python here.

Crystlecs