Much memory fragmentation with System.Threading.OverlappedData (async await)

Question

I'm constantly running slowly out of memory. I've used WinDbg to have a look at the memory dump and this it what it looks like:

000007f975ee0630   557377     41027736 System.Object[]
000007f975f004d0    18781     47746604 System.Byte[]
000007f975efc358   561433     54013658 System.String
000000000137b170     7616   1322123700      Free

Total 7627101 objects
Fragmented blocks larger than 0.5 MB:
        Addr     Size      Followed by        
000000004a9f62b0   18.2MB 000000004bc28050 System.Threading.OverlappedData
000000004dc2ce68   16.1MB 000000004ec522e8 System.Threading.OverlappedData
0000000050adaec0   10.3MB 0000000051525620 System.Threading.OverlappedData
000000005d47fd98   10.2MB 000000005deab618 System.Threading.OverlappedData
0000000071718ab8   23.0MB 0000000072e13a80 System.Threading.OverlappedData
0000000072e13e50   11.8MB 00000000739e4898 System.ServiceModel.Channels.SocketConnection
00000000801a7830   29.7MB 0000000081f5dd60 System.Threading.OverlappedData
000000008264ab58   14.0MB 000000008344bac0 System.Threading.OverlappedData
000000008344bb30   11.6MB 0000000083fecf80 System.Threading.OverlappedData
0000000083fecff0   13.6MB 0000000084d8dae8 System.Threading.OverlappedData
0000000084d8db58  148.3MB 000000008e1d65f8 System.Threading.OverlappedData
0000000093db04e0   19.4MB 00000000951158b8 System.Threading.OverlappedData
0000000095115928   33.9MB 00000000972f2620 System.Threading.OverlappedData
00000000abaa6738   71.9MB 00000000b0285cb0 System.Byte[]

Note that there are many more items in fragmented heap, I've only pasted the ones over 10MB. You can tell that the main problem is System.Threading.OverlappedData.

Recently I've refactored all the code to use async & await to something like this:

await Task.Factory.FromAsync<string, MessageSender>(
                                  this.MessageFactory.BeginCreateMessageSender, 
                                  this.MessageFactory.EndCreateMessageSender, 
                                  this.Topic.Path,
                                  null)
      .ConfigureAwait(false);

There is a lot of socket inbound (socket) and outbound (azure service bus) communication. How to avoid such memory fragmentation?

Tetreault

Are you sure you're disposing everything that needs to be disposed? – Hardi

Yes, this is .net 'handling' of async network requests – Tetreault

Answer 1

I think that you should check those places:

• KB947862

• this SO question and discussion

I'm working on a similar problem now: my server receives numerous client requests on async sockets, but for numerous reasons clients do not end their communication properly (most probably due to frequent IP changes - they're mobile on GPRS/3G). That seems to make the callback wait forever and EndReceive on sockets never gets called, which makes those Overlapped objects (which are AsyncPinned) dangle forever in memory, causing both high memory usage and fragmentation. Hence the idea about limiting the number of concurrent connections and assuring by yourself that overdue connections are manually closed.

Elbrus