Task Dataflow, can a data block be changed from completion state?

Question

I would like to know whether it is possible to change the completion state of data blocks?

For example, I marked a var block = new BufferBlock<int>(); data block complete with block.Complete(). The block is linked to other data blocks. I would like to know whether I can make block run again by changing its completion state back to its original !complete state.

If that is not possible how could I perform multiple runs, including completion, without having to a) de-link all blocks, b) re-instantiate all blocks,and c) re-link all blocks again?

Any ideas which may ease the task of running data flow cycles including completing each block down the chain without having to re-create the whole framework?

Thanks

Quirinal

Why do you even need separate runs? Also, why don't you want to recreate the dataflow network? I doubt it's going to affect performance (unless you do that many times per second) and it also shouldn't make your code more complicated by much. – Menace

The separate runs are part of the requirement. Re-creating the whole network would completely defeat the efficiency gained through choice of tpl dataflow. The main headache is the unlinking and relinking not so much the re-creation of each data flow block. If changing the completion state is impossible how could I otherwise signal completion without affecting the completion state of each data flow block? I cannot just send an atomic flag because the object types down the chain are not identical (there are transformblocks involved). – Quirinal

Answer 1

No, you can't “uncomplete” a completed dataflow block. I think what you should do is to add a flag to each message that says whether it's last message in a run. To make it simpler to work with it, you could create set of helper methods like:

public static TransformBlock<Tuple<TInput, bool>, Tuple<TOutput, bool>>
    CreateEnhancedTransformBlock<TInput, TOutput>(Func<TInput, TOutput> transform)
{
    return new TransformBlock<Tuple<TInput, bool>, Tuple<TOutput, bool>>(
        tuple => Tuple.Create(transform(tuple.Item1), tuple.Item2));
}

This way, you enter a transform delegate that deals just with TInput and TOuput and the flag is transfered along with each message.

Menace

Interesting idea, would you care to comment on computational overhead during execution by going this route? So far I only see an additional check on the "completion flag" of the receiver block for each incoming message. Anything else I am omitting? Would it be more expensive to instead ship an enum rather than bool? In that way I could communicate other states than just completion. – Quirinal

@Freddy There could be some tiny overhead since there are two delegate invocations instead of one, but that should be really negligible. And I think replacing the bool with an enum shouldn't affect the performance either. – Menace

definitely a good idea to solve this issue, but in the end I went with your first advice: I peruse the neat completion feature of data blocks and re-create the whole pipeline. I provide callbacks to upstream data block containing modules and upon the completion of each data block it re-creates the same and hands it through the callback downstream where the two blocks are re-linked. – Quirinal

I also find adding an indexing system to be helpful for identifying runs or batches of data. This makes it so that the runs do not need to be completely isolated. – Ephesus