Append two or more byte arrays in C#

Question

Is there a best (see below) way to append two byte arrays in C#?

Pretending I have complete control, I can make the first byte array sufficiently large to hold the second byte array at the end and use the Array.CopyTo function. Or I can loop over individual bytes and make an assignment.

Are there better ways? I can't imagine doing something like converting the byte arrays to string and joining them and converting them back would be better than either method above.

In terms of best/better (in order):

1. Fastest
2. Least RAM consumption

A constraint is that I must work in the .NET 2.0 framework.

The two choices recommended are MemoryStream and BlockCopy. I have run a simple speed test of 10,000,000 loops 3 times and got the following results:

Average of 3 runs of 10,000,000 loops in milliseconds:

• BlockCopy Time: 1154, with a range of 13 milliseconds
• MemoryStream GetBuffer Time: 1470, with a range of 14 milliseconds
• MemoryStream ToArray Time: 1895, with a range of 3 milliseconds
• CopyTo Time: 2079, with a range of 19 milliseconds
• Byte-by-byte Time: 2203, with a range of 10 milliseconds

Results of List<byte> AddRange over 10 million loops: List<byte> Time: 16694

Relative RAM Consumption (1 is baseline, higher is worse):

• Byte-by-byte: 1
• BlockCopy: 1
• Copy To: 1
• MemoryStream GetBuffer: 2.3
• MemoryStream ToArray: 3.3
• List<byte>: 4.2

The test shows that in general, unless you are doing a lot of byte copies [which I am], looking at byte copies is not worth a focus [e.g. 10 million runs yielding a difference of as much as 1.1 seconds].

Highjack

Re: Memory usage The best way to find that is to use a memory profiler. – Coussoule

Thanks, I used the CLR Profiler to get the memory data. Forgot to have it run as administrator, so getting results for that was delayed. – Highjack

possible duplicate of Best way to combine two or more byte arrays in C# – Uvarovite

#415791 – Iqbal

Answer 1

You want BlockCopy

According to this blog post it is faster than Array.CopyTo.

Coussoule

yeah - Array.CopyTo is O(n). It's unfortunate that MS didn't optimize this at all for byte arrays... – Harpp

BlockCopy beat MemoryStream in speed (don't have a reliable test for RAM), so selecting as answer. – Highjack

BlockCopy can also copy a 2D array into a 1D array and vice versa. Array.Copy cannot. – Nebraska

Answer 2

You could also use an approach with a MemoryStream. Suppose b1 and b2 are two byte arrays, you can get a new one, b3, by using the MemoryStream in the following fashion:

var s = new MemoryStream();
s.Write(b1, 0, b1.Length);
s.Write(b2, 0, b2.Length);
var b3 = s.ToArray();

This should work without LINQ and is in fact quite a bit faster.

Equipotential

I like that this is quite a bit smaller in code. Easier to understand maybe. Clean :) – Passable

Answer 3

Create a new MemoryStream passing into the constructor a buffer that's exactly the size of the merged one. Write the individual arrays, and then finally use the buffer:

byte[] deadBeef = new byte[] { 0xDE, 0xAD, 0xBE, 0xEF};
byte[] baadF00d = new byte[] { 0xBA, 0xAD, 0xF0, 0x0D};
int newSize = deadBeef.Length + baadF00d.Length;
var ms = new MemoryStream(new byte[newSize], 0, newSize, true, true);
ms.Write(deadBeef, 0, deadBeef.Length);
ms.Write(baadF00d, 0, baadF00d.Length);
byte[] merged = ms.GetBuffer();

A lot of the low-level I/O functions in .NET take byte arrays and offsets. This was done to prevent needless copies. Be sure you really need the merged array if this is performance sensitive, otherwise just use buffers and offsets.

Sunshinesunspot

If you go this route, use ms.ToArray() rather than ms.GetBuffer() - even though you're passing the explicit length, it's still possible that the internal buffer returned via GetBuffer() will have more bytes than specified. That is to say: don't rely on the internal implementation of MemoryStream. – Nguyetni

ToArray will create a copy which will hurt performance. If you go this route with an explicit buffer size, it'll be exactly the size you specified. If you try to go larger, you'll get a NotSupportedException. – Sunshinesunspot

Answer 4

Another option, although I haven't tested it to see how it fares in terms of speed and memory consumption, would the LINQ approach:

byte[] combined = bytesOne.Concat(bytesTwo).Concat(bytesThree).ToArray();

...where bytesOne, bytesTwo, and bytesThree are byte arrays. Since Concat uses deferred execution, this shouldn't create any intermediate arrays, and it shouldn't duplicate the original arrays until it constructs the final merged array at the end.

Edit: LINQBridge will allow you to use LINQ-to-Objects (which this is an example of) in the 2.0 framework. I understand if you don't want to depend on this, but it's an option.

Mar

Thanks, I am unfortunately limited to 2.0. I have updated my question to state that. – Highjack

Answer 5

If you have arrays where the size will change from time to time, you're probably better off using a List<T> in the first place. Then you can just call the AddRange() method of the list.

Otherwise, Array.Copy() or Array.CopyTo() are as good as anything else you're likely to see.

Burdock

Answer 6

Have you taught about using List or ArrayList instead of an Array? With these types they can grow or shrink and append via InsertRange

Brubeck

Answer 7

Do you need the output to actually be a byte array?

If not, you could create yourself a "smart cursor" (which is similar to what LINQ does): Create a custom IEnumerator<byte> that will first iterate the first array, and just continue on the second one without interuption.

This would work in the 2.0 framework be fast (in that the joining of arrays has virtually no cost), and use no more RAM than the arrays already consume.

Crupper

Answer 8

Your first option of making the first array large enough to contain the second array and using Array.CopyTo ends up being roughly the same as manually iterating over each item and making the assignment. Array.CopyTo() just makes it more concise.

Converting to string and back to array will be horribly slow in contrast to the above. And would likely use more memory.

Mainz