Getting IEnumerable<T> semantics with a NetTcpBinding WCF service?

Question

First, this is not a duplicate of IEnumerable<T> as return type for WCF methods, I think I understand that the WCF architecture only allows concrete types to be transferred that can be stuffed into a message.

Second, our setup however is not a general service but connecting up a bundle of proprietary apps via C# + WCF + NetTcpBinding + Protobuf (only) so we may have more room for some tricks that something that needs to be more binding neutral.

Third, it is neither my place nor this question's to propose a different RPC or messaging framework.

---

"IEnumerable semantics", for the purpose of this question, are:

• The returned sequence can be arbitrarily large -- it is therefore not possible to convert the sequence to a List or similar.
• It is not known in advance how many items will be returned
• Caller can just use foreach and be done with it.

---

In a local assembly, a C# interface my look like this:

interface IStuffProvider {
  IEnumerable<Stuff> GetItems(); // may open large file or access database
}

You can't map that directly to a WCF service. Something that might achieve the same could look like:

[ServiceContract(SessionMode = SessionMode.Required)]
interface IStuffService {
  [OperationContract]
  void Reset(); // may open large file or access database
  [OperationContract]
  List<Stuff> GetNext(); // return next batch of items (empty list if no more available)
}

Of course, using IStuffService will be more error prone than a IStuffProvider and add in to the mix than many usage scenarios would involve using both service and client on the same machine, so for "user code" it wouldn't be super important to be aware that "the network" is involved, the user code is just interested in a simple interface.

One option would be of course to have a client side interface wrapper implementation, that exposes IStuffProvider and internally forwards to and uses IStuffService. However, it seems it would really be desirable to not have to maintain two interfaces, one for user code, one solely for the WCF communication, especially as these applications are all tightly coupled anyway, so the additional abstraction just seems overhead.

What are the options we have here with WCF?

---

Note that after reading up on it, the Streamed Binding seems a poor solution, as I would still need a wrapper on the client side and the service interface would get more complex for no real gain in my case: I don't need maximum binary transfer efficiency, I would like good implementation + maintenance efficiency.

Cookery

Maybe you can use the Streaming functionality that is provided by WCF? (I haven't used WCF for a while, though, so I'm not sure it's suitable...) – Delano

@AasmundEldhuset - certainly a worthwhile read. There seem to be a lot of strings attached though. – Cookery

WCF tends to be that way, doesn't it? ;-) – Delano

arbitrarily large => streamed binding all the way. no-brainer. – Maniacal

@Lucas - what I haven't grasped so far is whether the streamed binding can be applied to arbitrary types or whether it only works for bytes ... might be already an answer to this here ... – Cookery

It only works for... Stream ;) (or message contracts including a stream in the body and some buffered headers) but you should be able to use protobuf on top of that for the serialization. – Maniacal

@LucasTrzesniewski - ah, ok, thanks. Rather not what I'd like to use, because Stream "Provides a generic view of a sequence of bytes" and I would like strongly typed data. – Cookery

You could write a wrapper around the WCF client/channel that would provide strong-typing. But your underlying message contract would still have to send just byte streams. – Maniacal

I suggest that you do not use a stateful WCF service for this. That's associated with many problems. – Marlenamarlene

Answer 1

What I did in the end is have:

a) The OO interface IStuffProvideras above with the GetLines() member as above.

b) The WCF service interface (and it's implementation) implements an access pattern like this:

    [OperationContract]
    ReadToken StartReadingLines(...);

    [OperationContract]
    // return next batch of items (empty list if no more available)
    List<Stuff> ReadNextLines(ReadToken readToken);

    [OperationContract]
    void FinishReadingLines(ReadToken readToken);

c) The client accesses the service through a proxy class that implements IStuffProvider and maps a call to the GetLines() function to the above three functions:

    // implementation in the proxy class:
    public IEnumerable<Stuff> GetLines()
    {
        var readToken = _dataService.StartReadingLines(...);
        try {
            for (List<Stuff> lines = _dataService.ReadNextLines(readToken); lines.Count > 0; lines = _dataService.ReadNextLines(readToken)) {
                foreach (var line in lines) {
                    yield return line;
                }
            }
        } finally {
            _dataService.FinishReadingLines(readToken);
        }
    }

Cookery

Answer 2

Some time ago we faced the same WCF "restriction" in our project. To be short, we ended up with

interface IStuffProvider {
  List<Stuff> GetItems(int page, int pageSize); // may open large file or access database
}

Yes, it is not the same as IEnumerable<Stuff> GetItems(); Yes, we can get in troubles when some item added/removed on already received page. Yes, it is required some server-side tweaks, if server works with items in terms of IEnumerable<Stuff>. But it is still strictly typed and does not bring much additional logic in the client or the server.

Williamsen

"Please note, it is not an answer to the question, but some common thoughts." - Should this not be posted as a comment then? – Pancreatin

This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. – Indicative

@Williamsen - while I don't think it is a good match for my problem, with some elaboration on how you handled the initial loading of the data (what if someone requests page 5 first? What if the same data is requested multiple times?) etc. it could make a good "alternative" answer. At least it looks low-tech and easy to maintain (if not to use). – Cookery

@TJ well, as I wrote, we faced the same restriction in our project, and (no matter how sad it is) it was an answer in our case. Therefore (and only therefore) I wrote this as an answer. Anyway, I'll be glad to see the "real" answer. And yes, this was neither critique nor request, for those two I always use comments ;) – Williamsen

@MartinBa right, the approach far from perfect, but it is how we solved problem of "sequential" access with ability to break when we want without getting the rest. – Williamsen

@Williamsen (note that I didn't downvote) - Did you have any validation in place for the usage of the page/pageSize stuff? Where did your data come from? Did you run a new query on every call, did you do some buffering? (Just curious.) – Cookery

@MartinBa (it is ok, I also downvote, if I do not like an answer, so why others can't? ;) Our data comes from the DB with a bunch of BL calculations. We run query on the first call and then trying to prefetch and cache data for the next several pages in background. On the next calls we check, if the cached data still valid, and re-new it if needed. – Williamsen