TPL Dataflow: Flatten incoming collection to sequential items

Question

I'm building an application using TPL dataflow. Actually I've the following problem. I've one transformblock var tfb1 = new TranformBlock<InMsg, IReadOnlyCollection<OutMsg>>. So tfb1 receives on in message and creates a list of out-messages. This list of out-messages shall be linked to a router datablock, which receives OutMsg as input (and not IReadOnlyCollection<OutMsg>).

How can I flatten the IReadOnlyCollection so that the containing message can be used as input for e.g. a transform block in the form of TransformBlock<OutMsg, SomeOtherType>. Is it possible via LinkTo()?

Thx

Espinosa

Have you tried TransformManyBlock? – Blowpipe

I thought TransformMany is a 1 to n relationsship. So 1 message creates a collection of messages. – Espinosa

Isn't that want you do here? Produce multiple OutMsg objects and pass them to the router block? – Blowpipe

Even if you only want to extract individual elements from a collection, you'd still use TransformManyBlock – Blowpipe

Answer 1

You can use the TransformManyBlock instead of TransformMany to flatten any IEnumerable<T> result, eg:

var flattenBlock = new TransformManyBlock<InMsg,OutMsg>(msg=>{
    List<OutMsg> myResultList;
    //Calculate some results
    return myResultList;
});

This will pass individual OutMsg instances to the next block.

You can use an iterator so individual messages are propagated immediatelly :

  var flattenBlock = new TransformManyBlock<InMsg,OutMsg>(msg=>{
    //Calculate some results
    foreach(var item in someDbResults)
    {
        yield return item;
    }
});

Or you can just flatten the input :

var flattenBlock = new TransformManyBlock<IEnumerable<OutMsg>,OutMsg>(items=>items);

You can link this block to any block that accepts OutMsg :

var block = new ActionBlock<OutMsg>(...);

flattenBlock.LinkTo(block);

You can route messages by passing a predicate to LinkTo. For example, if you want to route failure messages to a logging block, you can type:

flattenBlock.LinkTo(logBlock,msg=>msg.HasError);
flattenBlock.LinkTo(happyBlock);

Messages that don't match any predicate will get stuck in the output buffer and prevent the block from completing. By having one link without predicate you ensure that all messages will be processed

Blowpipe

Hmm... but above code generates a list of output messages from a input message. I would need the opposite operation. – Espinosa

You posted that you want to flatten the list and send individual messages to a router block. BTW you can specify a predicate on LinkTo(), you don't need a block to route messages – Blowpipe

Correct. Thx a lot for your updated answer, I'll try your proposal. – Espinosa