Data throttling C#

Question

I am using thirdparty component which accepts maximum 25KB data at a time. I am passing array of the objects to this third party component from my application.

However the amount of data that my application writes is much more than 25KB. I am retrieving data from database and calling the component directly.

I have added reference of the component in the application. The data I am passing to the component as array of object which contains primitive and non primitive types.

How can I implement the data throttling here?

Carpentaria

Please add some more info, like how is the data being sent to the component, i.e. what sort of communication is it. – Rafflesia

What is the architecture and method of communication? – Antifreeze

What kind of objects do you store in the array, primitives? – Nones

Also, how are you determining if an 'object' is more than 25KB ? – Gomuti

@DominicZukiewicz: I assume that this is exactly the question OP wants to have answered. I would simply measure(f.e. with GC.GetTotalMemory) how many objects you can store in the array until the 25KB exceeds, then i would query and send only that array. – Nones

@TimSchmelter: 'GC.GetTotalMemory()' will not be a good representation because if you create a stream, that will create internal buffers, which may take you over the 25KB limit. There may also be some internal allocations you may not be aware of. I'll add an answer to this effect. – Gomuti

@DominicZukiewicz: That was just my first thought and my intention was not to use it during streaming but once to measure the number of objects he can store before it exceeds the limit. (obsolete anyway since he mentioned that it's no an array of primitives) – Nones

Answer 1

You can calculate the size of the one row in your database. After that every time you pass something you add that size to a variable. At the same time you are using a Stopwatch that runs. Just check if the Stopwatch.EllapsedSeconds is bigger than 1 second. If yes reset the Stopwatch and reset your variable with the size you ve already passed. If not check if the size you ve already passed (the amount of variable) is bigger than 25KB. If that is true call System.Thread.Thread.Sleep(Math.Max(1000 - StopWatch.EllapsedMilliseconds, 0)). But remember you have to do that in an extra thread because sleep blocks the whole thread.

Dannielledannon

Answer 2

You must implement a data buffer between your app and the component. The best way to do that is to:

• make a class that has it's own internal thread,
• on the public part of the class' interface implement Write method which accepts a byte array and stores it to a queue
• internal thread writes up to 25kb chunk of data from queue the and sleeps for 1 second, minus the time it took to write the chunk.

You also must take care, if the data stream is continuous and is produced at a rate > 25kbps, then you produce more then the component can consume and your queue will overflow.

Rafflesia

Answer 3

Have a look at the BufferedStream class example in MSDN. You should be able to adapt it so that you write 25KB maximum per second.

When writing to stream, you specify a data size to read/write, so if you say write 25KB and it returns 24KB, you know you can still write 1K.

If it comes back with 25KB and there is still data outstanding in the buffer, you can do a Thread.Sleep(1000) to ensure it does not exceed the quota.

Gomuti