AngularFire2 infinite scrolling

Question

I'm trying to implement an infinite scrolling with Ionic2 and Firebase.

I use AngularFire2. What I'd like to do is to add new items to the fetched list and not to reload the whole list.

let query$:Observable<any> = this.af.database.list(`quests/`, {
    query: {
        orderByChild: 'date_published',
        limitToFirst: this.recentChunkLimit$ //Subject where I push new limit length
    }
}).publishReplay(1).refCount();

However, when I query list like that, the whole list is reloaded each time via websockets making each next update slower and slower. Here is a screenshot of Network websockets tab: And also I noticed that requests are made 2 times for each next chunk (though I put publishReplay). And it is happening in all apps where I used AngularFire2. I might misunderstand something though. I definitely need some clarification.

//==========Edit============

Now, I somehow managed to implement what I want without reloading the whole list each time. Not the best implementation but it works. Basically, I created an observable array and load new values into it by subscribing to the next chunk observable (where I also get the last element to start with). However the later problem still remains - in the sockets display I get data requested 2 times.

Unsaddle

what do you mean by the whole list? You requested 5 items. Then you requested for 10. The first 5 will be part of the 10 you requested. Isn't that your expectation? – Bowles

I meant that the whole list is requested from the firebase, thus loading already loaded items once again. That's an expected behaviour but definitely not a desired one as for each higher limit, the loading time increases. – Unsaddle

Answer 1

Using observables for query options just does not work like that. There is no facility in the underlying SDK to dynamically modify a query's limitToFirst and there's no way of doing it in AngularFire2.

Each time an observable query option emits a new value, a new Firebase ref is created. You can see it in the source here.

However, it would be possible to create an observable that represents an infinite list by doing something like this:

import { Observable } from "rxjs/Observable";
import { Subject } from "rxjs/Subject";
import rxjs/add/observable/defer";
import rxjs/add/observable/zip";
import rxjs/add/operator/concatMap";
import rxjs/add/operator/filter";
import rxjs/add/operator/first";
import rxjs/add/operator/map";
import rxjs/add/operator/scan";
import rxjs/add/operator/share";
import rxjs/add/operator/startWith";

const pageSize = 100;
let notifier = new Subject<any>();
let last: Observable<any>;

let infiniteList = Observable

  // Use zip to combine the notifier's emissions with the last
  // child value:

  .zip(notifier, Observable.defer(() => last))

  // Use concatMap to emit a page of children into the
  // composed observable (note that first is used to complete
  // the inner list):

  .concatMap(([unused, last]) => this.af.database.list("quests", {
      query: {

        // If there is a last value, start at that value but ask
        // for one more:

        limitToFirst: last ? (pageSize + 1) : pageSize,
        orderByChild: "date_published",
        startAt: last
      }
    })
    .first()
  )

  // Use scan to accumulate the page into the infinite list:

  .scan((acc, list) => {

    // If this isn't the initial page, the page was started
    // at the last value, so remove it from the beginning of
    // the list:

    if (acc.length > 0) {
      list.shift();
    }
    return acc.concat(list);
  }, [])

  // Use share so that the last observable (see below) doesn't
  // result in a second subscription:

  .share();

// Each time a page is emitted, map to its last child value so
// that it can be fed back into the composed infinite list:

last = infiniteList
  .filter((list) => list.length > 0)
  .map((list) => list[list.length - 1].date_published)
  .startWith(null);

infiniteList.subscribe((list) => console.log(list));

// Each time the notifier emits, another page will be retrieved
// and added to the infinite list:

notifier.next();
notifier.next();
notifier.next();

That will work, but if the child upon which you are ordering has duplicate values, AngularFire2 won't be able to page through the results reliably until this issue is re-opened and resolved.

The resultant list is static. That is, children already paged in to the list won't be updated if the database changes. Implementing a dynamic list is more challenging, as duplicated and missing children can easily be effected by the limit-based paging mechanism.

---

Since writing this answer, I've made available tested implementations of forward and reverse, non-realtime and realtime infinite list observables in a library of Firebase observables that I have open sourced. See this GitHub repo.

Pungy

Thanks for the answer. I solved it in a bit different fashion. Your solution is more pretty. And it's a good RxJS lesson. Could you also have a look at the second part of the question. I don't know if it is supposed to be this way; – Unsaddle

By second part, do you mean the information appearing to be sent twice? I will have a quick look at it, tomorrow. – Pungy

Regarding the WebSocket traffic showing the data being retrieved twice, you've found what I consider to be a bug. I'm talking with the AngularFire2 team about it. – Pungy

FYI: the related issue and PR. – Pungy

One of the most-descriptive answers I've ever seen! Solved my problem and works great, thank you. This is actually an RxJS lesson! – Zabaglione

There is one problem - It starts from the first firebase item, and I need to start it from the end. I changed startAt to endAt but the next page is not working correctly and shows thes same messages – Zabaglione

Answer 2

To add on to cartant's answer, if you wish to start at the end and retrieve list items in reverse, here is how you do it (I've added comments where code was changed).

import { Observable } from "rxjs/Observable";
import { Subject } from "rxjs/Subject";
import rxjs/add/observable/defer";
import rxjs/add/observable/zip";
import rxjs/add/operator/concatMap";
import rxjs/add/operator/filter";
import rxjs/add/operator/first";
import rxjs/add/operator/map";
import rxjs/add/operator/scan";
import rxjs/add/operator/share";
import rxjs/add/operator/startWith";

const pageSize = 100;
let notifier = new Subject<any>();
let last: Observable<any>;

let infiniteList = Observable

  .zip(notifier, Observable.defer(() => last))

  .concatMap(([unused, last]) => this.af.database.list("quests", {
      query: {

        // Use limitToLast to move upward the list instead of downward

        limitToLast: last ? (pageSize + 1) : pageSize,
        orderByChild: "date_published",

        // Use endAt to start at the end of the list

        endAt: last
      }
    })
    .first()
  )

  .scan((acc, list) => {

    // Swap the roles of acc and list, as we want to 
    // concatenate from the beginning

    if (list.length > 0) {
      acc.shift();
    }
    return list.concat(acc);
  }, [])

  .share();



last = infiniteList
  .filter((list) => list.length > 0)

  // Use the first child in this list as the next endAt value

  .map((list) => list[0].date_published)

  // Use undefined instead of null, as endAt: null in angularfire2
  // will search for the last child that is null

  .startWith(undefined);

infiniteList.subscribe((list) => console.log(list));

notifier.next();
notifier.next();
notifier.next();

Hertz