How do I join data from two Firestore collections in Flutter?

Question

I have a chat app in Flutter using Firestore, and I have two main collections:

• chats, which is keyed on auto-ids, and has message, timestamp, and uid fields.
• users, which is keyed on uid, and has a name field

In my app I show a list of messages (from the messages collection), with this widget:

class ChatList extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    var messagesSnapshot = Firestore.instance.collection("chat").orderBy("timestamp", descending: true).snapshots();
    var streamBuilder = StreamBuilder<QuerySnapshot>(
          stream: messagesSnapshot,
          builder: (BuildContext context, AsyncSnapshot<QuerySnapshot> querySnapshot) {
            if (querySnapshot.hasError)
              return new Text('Error: ${querySnapshot.error}');
            switch (querySnapshot.connectionState) {
              case ConnectionState.waiting: return new Text("Loading...");
              default:
                return new ListView(
                  children: querySnapshot.data.documents.map((DocumentSnapshot doc) {
                    return new ListTile(
                      title: new Text(doc['message']),
                      subtitle: new Text(DateTime.fromMillisecondsSinceEpoch(doc['timestamp']).toString()),
                    );
                  }).toList()
                );
            }
          }
        );
        return streamBuilder;
  }
}

But now I want to show the user's name (from the users collection) for each message.

I normally call that a client-side join, although I'm not sure if Flutter has a specific name for it.

I've found one way to do this (which I've posted below), but wonder if there is another/better/more idiomatic way to do this type of operation in Flutter.

So: what is the idiomatic way in Flutter to look up the user name for each message in the above structure?

Gambrill

I think the only solution I have researched a lot of rxdart – Hallucinatory

Answer 1

You can do it with RxDart like that.. https://pub.dev/packages/rxdart

import 'package:rxdart/rxdart.dart';

class Messages {
  final String messages;
  final DateTime timestamp;
  final String uid;
  final DocumentReference reference;

  Messages.fromMap(Map<String, dynamic> map, {this.reference})
      : messages = map['messages'],
        timestamp = (map['timestamp'] as Timestamp)?.toDate(),
        uid = map['uid'];

  Messages.fromSnapshot(DocumentSnapshot snapshot)
      : this.fromMap(snapshot.data, reference: snapshot.reference);

  @override
  String toString() {
    return 'Messages{messages: $messages, timestamp: $timestamp, uid: $uid, reference: $reference}';
  }
}

class Users {
  final String name;
  final DocumentReference reference;

  Users.fromMap(Map<String, dynamic> map, {this.reference})
      : name = map['name'];

  Users.fromSnapshot(DocumentSnapshot snapshot)
      : this.fromMap(snapshot.data, reference: snapshot.reference);

  @override
  String toString() {
    return 'Users{name: $name, reference: $reference}';
  }
}

class CombineStream {
  final Messages messages;
  final Users users;

  CombineStream(this.messages, this.users);
}

Stream<List<CombineStream>> _combineStream;

@override
  void initState() {
    super.initState();
    _combineStream = Observable(Firestore.instance
        .collection('chat')
        .orderBy("timestamp", descending: true)
        .snapshots())
        .map((convert) {
      return convert.documents.map((f) {

        Stream<Messages> messages = Observable.just(f)
            .map<Messages>((document) => Messages.fromSnapshot(document));

        Stream<Users> user = Firestore.instance
            .collection("users")
            .document(f.data['uid'])
            .snapshots()
            .map<Users>((document) => Users.fromSnapshot(document));

        return Observable.combineLatest2(
            messages, user, (messages, user) => CombineStream(messages, user));
      });
    }).switchMap((observables) {
      return observables.length > 0
          ? Observable.combineLatestList(observables)
          : Observable.just([]);
    })
}

for rxdart 0.23.x

@override
      void initState() {
        super.initState();
        _combineStream = Firestore.instance
            .collection('chat')
            .orderBy("timestamp", descending: true)
            .snapshots()
            .map((convert) {
          return convert.documents.map((f) {

            Stream<Messages> messages = Stream.value(f)
                .map<Messages>((document) => Messages.fromSnapshot(document));

            Stream<Users> user = Firestore.instance
                .collection("users")
                .document(f.data['uid'])
                .snapshots()
                .map<Users>((document) => Users.fromSnapshot(document));

            return Rx.combineLatest2(
                messages, user, (messages, user) => CombineStream(messages, user));
          });
        }).switchMap((observables) {
          return observables.length > 0
              ? Rx.combineLatestList(observables)
              : Stream.value([]);
        })
    }

Hallucinatory

Very cool! Is there a way to not need f.reference.snapshots(), as that's essentially reloading the snapshot and I would prefer not to rely on the Firestore client being smart enough to deduplicate those (even though I'm almost certain that it does dedupe). – Gambrill

Found it. Instead of Stream<Messages> messages = f.reference.snapshots()..., you can do Stream<Messages> messages = Observable.just(f).... What I like about this answer is that it observes the user documents, so if a user name is updated in the database, the output reflects that right away. – Gambrill

@CenkYAGMUR can you please answer this its same kind of question but I want to retrieve whole 'X' collections field from another collection in Flutter https://mcmap.net/q/504475/-how-to-query-documents-of-39-x-39-collection-from-document-id-of-39-y-39-collection-in-flutter-and-firestore/13984728 – Humo

Answer 2

I got another version working which seems slightly better than my answer with the two nested builders.

Here I isolated on the data loading in a custom method, using a dedicated Message class to hold the information from a message Document and the optional associated user Document.

class Message {
  final message;
  final timestamp;
  final uid;
  final user;
  const Message(this.message, this.timestamp, this.uid, this.user);
}
class ChatList extends StatelessWidget {
  Stream<List<Message>> getData() async* {
    var messagesStream = Firestore.instance.collection("chat").orderBy("timestamp", descending: true).snapshots();
    var messages = List<Message>();
    await for (var messagesSnapshot in messagesStream) {
      for (var messageDoc in messagesSnapshot.documents) {
        var message;
        if (messageDoc["uid"] != null) {
          var userSnapshot = await Firestore.instance.collection("users").document(messageDoc["uid"]).get();
          message = Message(messageDoc["message"], messageDoc["timestamp"], messageDoc["uid"], userSnapshot["name"]);
        }
        else {
          message = Message(messageDoc["message"], messageDoc["timestamp"], "", "");
        }
        messages.add(message);
      }
      yield messages;
    }
  }
  @override
  Widget build(BuildContext context) {
    var streamBuilder = StreamBuilder<List<Message>>(
          stream: getData(),
          builder: (BuildContext context, AsyncSnapshot<List<Message>> messagesSnapshot) {
            if (messagesSnapshot.hasError)
              return new Text('Error: ${messagesSnapshot.error}');
            switch (messagesSnapshot.connectionState) {
              case ConnectionState.waiting: return new Text("Loading...");
              default:
                return new ListView(
                  children: messagesSnapshot.data.map((Message msg) {
                    return new ListTile(
                      title: new Text(msg.message),
                      subtitle: new Text(DateTime.fromMillisecondsSinceEpoch(msg.timestamp).toString()
                                         +"\n"+(msg.user ?? msg.uid)),
                    );
                  }).toList()
                );
            }
          }
        );
        return streamBuilder;
  }
}

Compared to the solution with nested builders this code is more readable, mostly because the data handling and the UI builder are better separated. It also only loads the user documents for users that have posted messages. Unfortunately, if the user has posted multiple messages, it will load the document for each message. I could add a cache, but think this code is already a bit long for what it accomplishes.

Gambrill

If you don't take "storing user info inside message" as answer, I think this is the best you can do. If you store user info inside message, there is this obvious drawback that user info might change in users collection, but not inside message. Using a scheduled firebase function, you may also resolve this. From time to time, you can go thru messages collection and update user info according to latest data in users collection. – Radioactive

Personally I prefer simpler solution like this compared to combining streams unless really necessary. Even better, we could refactor this data loading method into something like a service class or follow the BLoC pattern. As you've already mentioned, we could save the user information in a Map<String, UserModel> and only load the user document once. – Uterine

Agreed Joshua. I'd love to see a write-up of how this'd look in a BLoC pattern. – Gambrill

Answer 3

If I'm reading this correctly, the problem abstracts to: how do you transform a stream of data which requires making an asynchronous call to modify data in the stream?

In the context of the problem, the stream of data is a list of messages, and the async call is to fetch the user data and update the messages with this data in the stream.

It's possible to do this directly in a Dart stream object using the asyncMap() function. Here's some pure Dart code that demonstrates how to do it:

import 'dart:async';
import 'dart:math' show Random;

final random = Random();

const messageList = [
  {
    'message': 'Message 1',
    'timestamp': 1,
    'uid': 1,
  },
  {
    'message': 'Message 2',
    'timestamp': 2,
    'uid': 2,
  },
  {
    'message': 'Message 3',
    'timestamp': 3,
    'uid': 2,
  },
];

const userList = {
  1: 'User 1',
  2: 'User 2',
  3: 'User 3',
};

class Message {
  final String message;
  final int timestamp;
  final int uid;
  final String user;
  const Message(this.message, this.timestamp, this.uid, this.user);

  @override
  String toString() => '$user => $message';
}

// Mimic a stream of a list of messages
Stream<List<Map<String, dynamic>>> getServerMessagesMock() async* {
  yield messageList;
  while (true) {
    await Future.delayed(Duration(seconds: random.nextInt(3) + 1));
    yield messageList;
  }
}

// Mimic asynchronously fetching a user
Future<String> userMock(int uid) => userList.containsKey(uid)
    ? Future.delayed(
        Duration(milliseconds: 100 + random.nextInt(100)),
        () => userList[uid],
      )
    : Future.value(null);

// Transform the contents of a stream asynchronously
Stream<List<Message>> getMessagesStream() => getServerMessagesMock()
    .asyncMap<List<Message>>((messageList) => Future.wait(
          messageList.map<Future<Message>>(
            (m) async => Message(
              m['message'],
              m['timestamp'],
              m['uid'],
              await userMock(m['uid']),
            ),
          ),
        ));

void main() async {
  print('Streams with async transforms test');
  await for (var messages in getMessagesStream()) {
    messages.forEach(print);
  }
}

Most of the code is mimicking the data coming from Firebase as a stream of a map of messages, and an async function to fetch user data. The important function here is getMessagesStream().

The code is complicated slightly by the fact that it's a list of messages coming in the stream. To prevent calls to fetch user data from occurring synchronously, the code uses a Future.wait() to gather a List<Future<Message>> and create a List<Message> when all the Futures have completed.

In the context of Flutter, you can use the stream coming from getMessagesStream() in a FutureBuilder to display the Message objects.

Devour

Answer 4

Allow me to put forth my version of an RxDart solution. I use combineLatest2 with a ListView.builder to build each message Widget. During the construction of each message Widget I lookup the name of the user with the corresponding uid.

In this snippet I use a linear lookup for the user's name but that can be improved by creating a uid -> user name map

import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:flutter/widgets.dart';
import 'package:rxdart/rxdart.dart';

class MessageWidget extends StatelessWidget {
  // final chatStream = Firestore.instance.collection('chat').snapshots();
  // final userStream = Firestore.instance.collection('users').snapshots();
  Stream<QuerySnapshot> chatStream;
  Stream<QuerySnapshot> userStream;

  MessageWidget(this.chatStream, this.userStream);

  @override
  Widget build(BuildContext context) {
    Observable<List<QuerySnapshot>> combinedStream = Observable.combineLatest2(
        chatStream, userStream, (messages, users) => [messages, users]);

    return StreamBuilder(
        stream: combinedStream,
        builder: (_, AsyncSnapshot<List<QuerySnapshot>> snapshots) {
          if (snapshots.hasData) {
            List<DocumentSnapshot> chats = snapshots.data[0].documents;

            // It would be more efficient to convert this list of user documents
            // to a map keyed on the uid which will allow quicker user lookup.
            List<DocumentSnapshot> users = snapshots.data[1].documents;

            return ListView.builder(itemBuilder: (_, index) {
              return Center(
                child: Column(
                  children: <Widget>[
                    Text(chats[index]['message']),
                    Text(getUserName(users, chats[index]['uid'])),
                  ],
                ),
              );
            });
          } else {
            return Text('loading...');
          }
        });
  }

  // This does a linear search through the list of users. However a map
  // could be used to make the finding of the user's name more efficient.
  String getUserName(List<DocumentSnapshot> users, String uid) {
    for (final user in users) {
      if (user['uid'] == uid) {
        return user['name'];
      }
    }
    return 'unknown';
  }
}

Antonia

Very cool to see Arthur. This is like a much cleaner version of my initial answer with nested builders. Definitely one of the more simpler solutions to read. – Gambrill

Answer 5

Ideally you want to exclude any business logic such as data loading into a separate service or following the BloC pattern, e.g.:

class ChatBloc {
  final Firestore firestore = Firestore.instance;
  final Map<String, String> userMap = HashMap<String, String>();

  Stream<List<Message>> get messages async* {
    final messagesStream = Firestore.instance.collection('chat').orderBy('timestamp', descending: true).snapshots();
    var messages = List<Message>();
    await for (var messagesSnapshot in messagesStream) {
      for (var messageDoc in messagesSnapshot.documents) {
        final userUid = messageDoc['uid'];
        var message;

        if (userUid != null) {
          // get user data if not in map
          if (userMap.containsKey(userUid)) {
            message = Message(messageDoc['message'], messageDoc['timestamp'], userUid, userMap[userUid]);
          } else {
            final userSnapshot = await Firestore.instance.collection('users').document(userUid).get();
            message = Message(messageDoc['message'], messageDoc['timestamp'], userUid, userSnapshot['name']);
            // add entry to map
            userMap[userUid] = userSnapshot['name'];
          }
        } else {
          message =
              Message(messageDoc['message'], messageDoc['timestamp'], '', '');
        }
        messages.add(message);
      }
      yield messages;
    }
  }
}

Then you can just use the Bloc in your component and listen to the chatBloc.messages stream.

class ChatList extends StatelessWidget {
  final ChatBloc chatBloc = ChatBloc();

  @override
  Widget build(BuildContext context) {
    return StreamBuilder<List<Message>>(
        stream: chatBloc.messages,
        builder: (BuildContext context, AsyncSnapshot<List<Message>> messagesSnapshot) {
          if (messagesSnapshot.hasError)
            return new Text('Error: ${messagesSnapshot.error}');
          switch (messagesSnapshot.connectionState) {
            case ConnectionState.waiting:
              return new Text('Loading...');
            default:
              return new ListView(children: messagesSnapshot.data.map((Message msg) {
                return new ListTile(
                  title: new Text(msg.message),
                  subtitle: new Text('${msg.timestamp}\n${(msg.user ?? msg.uid)}'),
                );
              }).toList());
          }
        });
  }
}

Uterine

Answer 6

The first solution I got working is to nest two StreamBuilder instances, one for each collection/query.

class ChatList extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    var messagesSnapshot = Firestore.instance.collection("chat").orderBy("timestamp", descending: true).snapshots();
    var usersSnapshot = Firestore.instance.collection("users").snapshots();
    var streamBuilder = StreamBuilder<QuerySnapshot>(
      stream: messagesSnapshot,
      builder: (BuildContext context, AsyncSnapshot<QuerySnapshot> messagesSnapshot) {
        return StreamBuilder(
          stream: usersSnapshot,
          builder: (context, usersSnapshot) {
            if (messagesSnapshot.hasError || usersSnapshot.hasError || !usersSnapshot.hasData)
              return new Text('Error: ${messagesSnapshot.error}, ${usersSnapshot.error}');
            switch (messagesSnapshot.connectionState) {
              case ConnectionState.waiting: return new Text("Loading...");
              default:
                return new ListView(
                  children: messagesSnapshot.data.documents.map((DocumentSnapshot doc) {
                    var user = "";
                    if (doc['uid'] != null && usersSnapshot.data != null) {
                      user = doc['uid'];
                      print('Looking for user $user');
                      user = usersSnapshot.data.documents.firstWhere((userDoc) => userDoc.documentID == user).data["name"];
                    }
                    return new ListTile(
                      title: new Text(doc['message']),
                      subtitle: new Text(DateTime.fromMillisecondsSinceEpoch(doc['timestamp']).toString()
                                          +"\n"+user),
                    );
                  }).toList()
                );
            }
        });
      }
    );
    return streamBuilder;
  }
}

As stated in my question, I know this solution is not great, but at least it works.

Some problems I see with this:

• It loads all users, instead of just the users who posted messages. In small data sets that won't be a problem, but as I get more messages/users (and use a query to show a subset of them) I'll be loading more and more users who didn't post any messages.
• The code is not really very readable with the nesting of two builders. I doubt this is idiomatic Flutter.

If you know a better solution, please post as an answer.

Gambrill

There is no need to add to StreamBuilder. The can be composed in another layer and only expose an only one Future or Stream. – Saunderson

Cool. If you can write it up in an answer, showing the code of how to do example that, others can upvote that. – Gambrill