ReactJS: Modeling Bi-Directional Infinite Scrolling

Question

Our application uses infinite scrolling to navigate large lists of heterogenous items. There are a few wrinkles:

• It's common for our users to have a list of 10,000 items and need to scroll through 3k+.
• These are rich items, so we can only have a few hundred in the DOM before browser performance becomes unacceptable.
• The items are of varying heights.
• The items may contain images and we allow the user to jump to a specific date. This is tricky because the user can jump to a point in the list where we need to load images above the viewport, which would push the content down when they load. Failing to handle that means that the user may jump to a date, but then be shifted to an earlier date.

Known, incomplete solutions:

• (react-infinite-scroll) - This is just a simple "load more when we hit the bottom" component. It does not cull any of the DOM, so it will die on thousands of items.

• (Scroll Position with React) - Shows how to store and restore the scroll position when inserting at the top or inserting at the bottom, but not both together.

I'm not looking for the code for a complete solution (although that would be great.) Instead, I'm looking for the "React way" to model this situation. Is scroll position state or not? What state should I be tracking to retain my position in the list? What state do I need to keep so that I trigger a new render when I scroll near the bottom or top of what is rendered?

Diagnostician

Answer 1

This is a mix of an infinite table and an infinite scroll scenario. The best abstraction I found for this is the following:

Overview

Make a <List> component that takes an array of all children. Since we do not render them, it's really cheap to just allocate them and discard them. If 10k allocations is too big, you can instead pass a function that takes a range and return the elements.

<List>
  {thousandelements.map(function() { return <Element /> })}
</List>

Your List component is keeping track of what the scroll position is and only renders the children that are in view. It adds a large empty div at the beginning to fake the previous items that are not rendered.

Now, the interesting part is that once an Element component is rendered, you measure its height and store it in your List. This lets you compute the height of the spacer and know how many elements should be displayed in view.

Image

You are saying that when the image are loading they make everything "jump" down. The solution for this is to set the image dimensions in your img tag: <img src="..." width="100" height="58" />. This way the browser doesn't have to wait to download it before knowing what size it is going to be displayed. This requires some infrastructure but it's really worth it.

If you can't know the size in advance, then add onload listeners to your image and when it is loaded then measure its displayed dimension and update the stored row height and compensate the scroll position.

Jumping at a random element

If you need to jump at a random element in the list that's going to require some trickery with scroll position because you don't know the size of the elements in between. What I suggest you to do is to average the element heights you already have computed and jump to the scroll position of last known height + (number of elements * average).

Since this is not exact it's going to cause issues when you reach back to the last known good position. When a conflict happens, simply change the scroll position to fix it. This is going to move the scroll bar a bit but shouldn't affect him/her too much.

React Specifics

You want to provide a key to all the rendered elements so that they are maintained across renders. There are two strategies: (1) have only n keys (0, 1, 2, ... n) where n is the maximum number of elements you can display and use their position modulo n. (2) have a different key per element. If all the elements share a similar structure it's good to use (1) to reuse their DOM nodes. If they don't then use (2).

I would only have two pieces of React state: the index of the first element and the number of elements being displayed. The current scroll position and the height of all the elements would be directly attached to this. When using setState you are actually doing a rerender which should only happen when the range changes.

Here is an example of infinite list using some of the techniques I describe in this answer. It's going to be some work but React is definitively a good way to implement an infinite list :)

Wellbred

This is an awesome technique. Thanks! I got it working on one of my components. However, I have another component that I'd like to apply this to, but the rows don't have a consistent height. I'm working on augmenting your example to calculate the displayEnd/visibleEnd to account for the varying heights... unless you have a better idea? – Oxymoron

I have implemented this with a twist, and ran into an issue: For me, the records I'm rendering are somewhat complex DOM, and because of the # of them it is not prudent to load them all into the browser, so I am doing async fetches from time to time. For some reason, on occasion when I scroll and position jumps very far (say I go off the screen and back), the ListBody doesn't re-render, even though the state changes. Any ideas why this might be? Great example otherwise! – Requiescat

Your JSFiddle currently throws an error: Uncaught ReferenceError: generate is not defined – Batey

I have made an updated fiddle, I think it should work the same. Anyone care to verify? @Batey – Signore

can i code like github link? for latest version 0.14 ? – Throckmorton

the updated fiddle jsfiddle.net/aknuds1/cchujfb1 seems to be working for me, thanks – Leadbelly

@ThomasModeneis hi, can you clarify the calculations done on line 151 and 152, the displayStart and displayEnd – Ohm

Hi , what are displayStart and visibleStart? – Trish

So I think visibleStart and visibleEnd are self explanatory. They represent the index's of the records that are actually visible within the scrollable container. displayStart and displayEnd appear to be the range of records to mount in the dom. displayStart will mostly be less than visibleStart were possible in case the user scrolls up. displayEnd will be visibleStart + recordsPerBody * some arbitrary number, in case the user scrolls down. So essentially what they allow for is a bit of a buffer. – Constitutionalism

@Signore it not working now. Can you update that. Thanks so much – Toro

Answer 2

have a look at http://adazzle.github.io/react-data-grid/index.html# This looks like a powerful and performant datagrid with Excel-like features and lazy loading/optimized rendering (for millions of rows) with rich editing features (MIT licenced). Not yet tried in our project but will do so pretty soon.

A great resource to search for things like these is also http://react.rocks/ In this case, a tag search is helpful: http://react.rocks/tag/InfiniteScroll

Rickety

Answer 3

I was facing a similar challenge for modeling single-direction infinite scrolling with heterogeneous item heights and so made an npm package out of my solution:

https://www.npmjs.com/package/react-variable-height-infinite-scroller

and a demo: http://tnrich.github.io/react-variable-height-infinite-scroller/

You can check out the source code for the logic, but I basically followed the recipe @Vjeux outlined in the above answer. I haven't yet tackled jumping to a particular item, but I'm hoping to implement that soon.

Here's the nitty-gritty of what the code currently looks like:

var React = require('react');
var areNonNegativeIntegers = require('validate.io-nonnegative-integer-array');

var InfiniteScoller = React.createClass({
  propTypes: {
    averageElementHeight: React.PropTypes.number.isRequired,
    containerHeight: React.PropTypes.number.isRequired,
    preloadRowStart: React.PropTypes.number.isRequired,
    renderRow: React.PropTypes.func.isRequired,
    rowData: React.PropTypes.array.isRequired,
  },

  onEditorScroll: function(event) {
    var infiniteContainer = event.currentTarget;
    var visibleRowsContainer = React.findDOMNode(this.refs.visibleRowsContainer);
    var currentAverageElementHeight = (visibleRowsContainer.getBoundingClientRect().height / this.state.visibleRows.length);
    this.oldRowStart = this.rowStart;
    var newRowStart;
    var distanceFromTopOfVisibleRows = infiniteContainer.getBoundingClientRect().top - visibleRowsContainer.getBoundingClientRect().top;
    var distanceFromBottomOfVisibleRows = visibleRowsContainer.getBoundingClientRect().bottom - infiniteContainer.getBoundingClientRect().bottom;
    var rowsToAdd;
    if (distanceFromTopOfVisibleRows < 0) {
      if (this.rowStart > 0) {
        rowsToAdd = Math.ceil(-1 * distanceFromTopOfVisibleRows / currentAverageElementHeight);
        newRowStart = this.rowStart - rowsToAdd;

        if (newRowStart < 0) {
          newRowStart = 0;
        } 

        this.prepareVisibleRows(newRowStart, this.state.visibleRows.length);
      }
    } else if (distanceFromBottomOfVisibleRows < 0) {
      //scrolling down, so add a row below
      var rowsToGiveOnBottom = this.props.rowData.length - 1 - this.rowEnd;
      if (rowsToGiveOnBottom > 0) {
        rowsToAdd = Math.ceil(-1 * distanceFromBottomOfVisibleRows / currentAverageElementHeight);
        newRowStart = this.rowStart + rowsToAdd;

        if (newRowStart + this.state.visibleRows.length >= this.props.rowData.length) {
          //the new row start is too high, so we instead just append the max rowsToGiveOnBottom to our current preloadRowStart
          newRowStart = this.rowStart + rowsToGiveOnBottom;
        }
        this.prepareVisibleRows(newRowStart, this.state.visibleRows.length);
      }
    } else {
      //we haven't scrolled enough, so do nothing
    }
    this.updateTriggeredByScroll = true;
    //set the averageElementHeight to the currentAverageElementHeight
    // setAverageRowHeight(currentAverageElementHeight);
  },

  componentWillReceiveProps: function(nextProps) {
    var rowStart = this.rowStart;
    var newNumberOfRowsToDisplay = this.state.visibleRows.length;
    this.props.rowData = nextProps.rowData;
    this.prepareVisibleRows(rowStart, newNumberOfRowsToDisplay);
  },

  componentWillUpdate: function() {
    var visibleRowsContainer = React.findDOMNode(this.refs.visibleRowsContainer);
    this.soonToBeRemovedRowElementHeights = 0;
    this.numberOfRowsAddedToTop = 0;
    if (this.updateTriggeredByScroll === true) {
      this.updateTriggeredByScroll = false;
      var rowStartDifference = this.oldRowStart - this.rowStart;
      if (rowStartDifference < 0) {
        // scrolling down
        for (var i = 0; i < -rowStartDifference; i++) {
          var soonToBeRemovedRowElement = visibleRowsContainer.children[i];
          if (soonToBeRemovedRowElement) {
            var height = soonToBeRemovedRowElement.getBoundingClientRect().height;
            this.soonToBeRemovedRowElementHeights += this.props.averageElementHeight - height;
            // this.soonToBeRemovedRowElementHeights.push(soonToBeRemovedRowElement.getBoundingClientRect().height);
          }
        }
      } else if (rowStartDifference > 0) {
        this.numberOfRowsAddedToTop = rowStartDifference;
      }
    }
  },

  componentDidUpdate: function() {
    //strategy: as we scroll, we're losing or gaining rows from the top and replacing them with rows of the "averageRowHeight"
    //thus we need to adjust the scrollTop positioning of the infinite container so that the UI doesn't jump as we 
    //make the replacements
    var infiniteContainer = React.findDOMNode(this.refs.infiniteContainer);
    var visibleRowsContainer = React.findDOMNode(this.refs.visibleRowsContainer);
    var self = this;
    if (this.soonToBeRemovedRowElementHeights) {
      infiniteContainer.scrollTop = infiniteContainer.scrollTop + this.soonToBeRemovedRowElementHeights;
    }
    if (this.numberOfRowsAddedToTop) {
      //we're adding rows to the top, so we're going from 100's to random heights, so we'll calculate the differenece
      //and adjust the infiniteContainer.scrollTop by it
      var adjustmentScroll = 0;

      for (var i = 0; i < this.numberOfRowsAddedToTop; i++) {
        var justAddedElement = visibleRowsContainer.children[i];
        if (justAddedElement) {
          adjustmentScroll += this.props.averageElementHeight - justAddedElement.getBoundingClientRect().height;
          var height = justAddedElement.getBoundingClientRect().height;
        }
      }
      infiniteContainer.scrollTop = infiniteContainer.scrollTop - adjustmentScroll;
    }

    var visibleRowsContainer = React.findDOMNode(this.refs.visibleRowsContainer);
    if (!visibleRowsContainer.childNodes[0]) {
      if (this.props.rowData.length) {
        //we've probably made it here because a bunch of rows have been removed all at once
        //and the visible rows isn't mapping to the row data, so we need to shift the visible rows
        var numberOfRowsToDisplay = this.numberOfRowsToDisplay || 4;
        var newRowStart = this.props.rowData.length - numberOfRowsToDisplay;
        if (!areNonNegativeIntegers([newRowStart])) {
          newRowStart = 0;
        }
        this.prepareVisibleRows(newRowStart , numberOfRowsToDisplay);
        return; //return early because we need to recompute the visible rows
      } else {
        throw new Error('no visible rows!!');
      }
    }
    var adjustInfiniteContainerByThisAmount;

    //check if the visible rows fill up the viewport
    //tnrtodo: maybe put logic in here to reshrink the number of rows to display... maybe...
    if (visibleRowsContainer.getBoundingClientRect().height / 2 <= this.props.containerHeight) {
      //visible rows don't yet fill up the viewport, so we need to add rows
      if (this.rowStart + this.state.visibleRows.length < this.props.rowData.length) {
        //load another row to the bottom
        this.prepareVisibleRows(this.rowStart, this.state.visibleRows.length + 1);
      } else {
        //there aren't more rows that we can load at the bottom so we load more at the top
        if (this.rowStart - 1 > 0) {
          this.prepareVisibleRows(this.rowStart - 1, this.state.visibleRows.length + 1); //don't want to just shift view
        } else if (this.state.visibleRows.length < this.props.rowData.length) {
          this.prepareVisibleRows(0, this.state.visibleRows.length + 1);
        }
      }
    } else if (visibleRowsContainer.getBoundingClientRect().top > infiniteContainer.getBoundingClientRect().top) {
      //scroll to align the tops of the boxes
      adjustInfiniteContainerByThisAmount = visibleRowsContainer.getBoundingClientRect().top - infiniteContainer.getBoundingClientRect().top;
      //   this.adjustmentScroll = true;
      infiniteContainer.scrollTop = infiniteContainer.scrollTop + adjustInfiniteContainerByThisAmount;
    } else if (visibleRowsContainer.getBoundingClientRect().bottom < infiniteContainer.getBoundingClientRect().bottom) {
      //scroll to align the bottoms of the boxes
      adjustInfiniteContainerByThisAmount = visibleRowsContainer.getBoundingClientRect().bottom - infiniteContainer.getBoundingClientRect().bottom;
      //   this.adjustmentScroll = true;
      infiniteContainer.scrollTop = infiniteContainer.scrollTop + adjustInfiniteContainerByThisAmount;
    }
  },

  componentWillMount: function(argument) {
    //this is the only place where we use preloadRowStart
    var newRowStart = 0;
    if (this.props.preloadRowStart < this.props.rowData.length) {
      newRowStart = this.props.preloadRowStart;
    }
    this.prepareVisibleRows(newRowStart, 4);
  },

  componentDidMount: function(argument) {
    //call componentDidUpdate so that the scroll position will be adjusted properly
    //(we may load a random row in the middle of the sequence and not have the infinte container scrolled properly initially, so we scroll to the show the rowContainer)
    this.componentDidUpdate();
  },

  prepareVisibleRows: function(rowStart, newNumberOfRowsToDisplay) { //note, rowEnd is optional
    //setting this property here, but we should try not to use it if possible, it is better to use
    //this.state.visibleRowData.length
    this.numberOfRowsToDisplay = newNumberOfRowsToDisplay;
    var rowData = this.props.rowData;
    if (rowStart + newNumberOfRowsToDisplay > this.props.rowData.length) {
      this.rowEnd = rowData.length - 1;
    } else {
      this.rowEnd = rowStart + newNumberOfRowsToDisplay - 1;
    }
    // var visibleRows = this.state.visibleRowsDataData.slice(rowStart, this.rowEnd + 1);
    // rowData.slice(rowStart, this.rowEnd + 1);
    // setPreloadRowStart(rowStart);
    this.rowStart = rowStart;
    if (!areNonNegativeIntegers([this.rowStart, this.rowEnd])) {
      var e = new Error('Error: row start or end invalid!');
      console.warn('e.trace', e.trace);
      throw e;
    }
    var newVisibleRows = rowData.slice(this.rowStart, this.rowEnd + 1);
    this.setState({
      visibleRows: newVisibleRows
    });
  },
  getVisibleRowsContainerDomNode: function() {
    return this.refs.visibleRowsContainer.getDOMNode();
  },


  render: function() {
    var self = this;
    var rowItems = this.state.visibleRows.map(function(row) {
      return self.props.renderRow(row);
    });

    var rowHeight = this.currentAverageElementHeight ? this.currentAverageElementHeight : this.props.averageElementHeight;
    this.topSpacerHeight = this.rowStart * rowHeight;
    this.bottomSpacerHeight = (this.props.rowData.length - 1 - this.rowEnd) * rowHeight;

    var infiniteContainerStyle = {
      height: this.props.containerHeight,
      overflowY: "scroll",
    };
    return (
      <div
        ref="infiniteContainer"
        className="infiniteContainer"
        style={infiniteContainerStyle}
        onScroll={this.onEditorScroll}
        >
          <div ref="topSpacer" className="topSpacer" style={{height: this.topSpacerHeight}}/>
          <div ref="visibleRowsContainer" className="visibleRowsContainer">
            {rowItems}
          </div>
          <div ref="bottomSpacer" className="bottomSpacer" style={{height: this.bottomSpacerHeight}}/>
      </div>
    );
  }
});

module.exports = InfiniteScoller;

Suntan