Using Rijndael encryption for large files

Question

I'm in a situation where I need to encrypt / decrypt a file of n length securely, ideally using Rijndael, but definitely at 256bit encryption.

I've played around with encryption before and have encrypted/decrypted strings and byte arrays quite happily. However, because I don't know the size of the file (and it's very feasible that the files in question could be quite large (~2.5gb) I can't just load them up into a byte array and enc/decrypt them in a single bound as I have before.

So, after a bit of reading around on Google, I knew the answer was to encrypt and decrypt the file in chunks, and so I produced the following code:

private static void Enc(string decryptedFileName, string encryptedFileName)
{            
   FileStream fsOutput = File.OpenWrite(encryptedFileName);
   FileStream fsInput = File.OpenRead(decryptedFileName);

   byte[] IVBytes = Encoding.ASCII.GetBytes("1234567890123456");

   fsOutput.Write(BitConverter.GetBytes(fsInput.Length), 0, 8);
   fsOutput.Write(IVBytes, 0, 16);

   RijndaelManaged symmetricKey = new RijndaelManaged() { Mode = CipherMode.CBC};
   ICryptoTransform encryptor = symmetricKey.CreateEncryptor(passwordDB.GetBytes(256 / 8), IVBytes);
   CryptoStream cryptoStream = new CryptoStream(fsOutput, encryptor, CryptoStreamMode.Write);

   for (long i = 0; i < fsInput.Length; i += chunkSize)
   {
      byte[] chunkData = new byte[chunkSize];
      fsInput.Read(chunkData, 0, chunkSize);
      cryptoStream.Write(chunkData, 0, chunkData.Length);
   }
   cryptoStream.Close();
   fsInput.Close();
   fsInput.Dispose();
   cryptoStream.Dispose();
}

private static void Dec(string encryptedFileName, string decryptedFileName)
{
    FileStream fsInput = File.OpenRead(encryptedFileName);
    FileStream fsOutput = File.OpenWrite(decryptedFileName);

    byte[] buffer = new byte[8];
    fsInput.Read(buffer, 0, 8);

    long fileLength = BitConverter.ToInt64(buffer, 0);

    byte[] IVBytes = new byte[16];
    fsInput.Read(IVBytes, 0, 16);

    RijndaelManaged symmetricKey = new RijndaelManaged() { Mode = CipherMode.CBC };
    ICryptoTransform decryptor = symmetricKey.CreateDecryptor(passwordDB.GetBytes(256 / 8), IVBytes);
    CryptoStream cryptoStream = new CryptoStream(fsOutput,decryptor,CryptoStreamMode.Write);

    for (long i = 0; i < fsInput.Length; i += chunkSize)
    {
        byte[] chunkData = new byte[chunkSize];
        fsInput.Read(chunkData, 0, chunkSize);
        cryptoStream.Write(chunkData, 0, chunkData.Length);
    }
    cryptoStream.Close();
    cryptoStream.Dispose();
    fsInput.Close();
    fsInput.Dispose();                      
} 

It all "looks" good to me, but sadly looks appear to be deceiving!

Encryption works without error, but during decryption, the "cryptoStream.Close()" method throws the following exception:

System.Security.Cryptography.CryptographicException was unhandled Message="Padding is invalid and cannot be removed."
Source="mscorlib" StackTrace: at System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast) at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) at System.Security.Cryptography.CryptoStream.FlushFinalBlock() at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing) at System.IO.Stream.Close()

It also appears that the unencrypted file size isn't matching the file size expected (ranging from around 8 bytes, to around 60)

I "fixed" the exception by altering the RijndaelManaged object creation lines to include a padding type, as below:

RijndaelManaged symmetricKey = new RijndaelManaged() { Mode = CipherMode.CBC,Padding=PaddingMode.None };

But the file sizes still don't match up and, predictably, the freshly unencrypted file is baloney!

I will admit that I'm now outside of my comfort zone with encryption/decryption, and it's probably a rookie mistake - but I can't spot it!

Any help on resolving this would be greatly appreciated!

Orcutt

Answer 1

The problem is that I was using:

passwordDB.GetBytes(256 / 8)

within the constructor for the RijndaelManaged object in both the Encryption and Decryption methods, and I wasn't re-initialising the passwordDB object before attempting to decrypt.

The resolution was to simply including the construction of the passwordDB object within the first lines of both the Enc and Dec methods, as follows:

        private static void Enc(string decryptedFileName, string encryptedFileName)
        {
            PasswordDeriveBytes passwordDB = new PasswordDeriveBytes("ThisIsMyPassword", Encoding.ASCII.GetBytes("thisIsMysalt!"), "MD5", 2);
            byte[] passwordBytes = passwordDB.GetBytes(128 / 8);

            using (FileStream fsOutput = File.OpenWrite(encryptedFileName))
            {
                using(FileStream fsInput = File.OpenRead(decryptedFileName))
                {
                    byte[] IVBytes = Encoding.ASCII.GetBytes("1234567890123456");

                    fsOutput.Write(BitConverter.GetBytes(fsInput.Length), 0, 8);
                    fsOutput.Write(IVBytes, 0, 16);

                    RijndaelManaged symmetricKey = new RijndaelManaged() { Mode = CipherMode.CBC,Padding=PaddingMode.ANSIX923};
                    ICryptoTransform encryptor = symmetricKey.CreateEncryptor(passwordBytes, IVBytes);                   

                    using (CryptoStream cryptoStream = new CryptoStream(fsOutput, encryptor, CryptoStreamMode.Write))
                    {
                        for (long i = 0; i < fsInput.Length; i += chunkSize)
                        {
                            byte[] chunkData = new byte[chunkSize];
                            int bytesRead = 0;
                            while ((bytesRead = fsInput.Read(chunkData, 0, chunkSize)) > 0)
                            {
                                if (bytesRead != 16)
                                {
                                    for (int x = bytesRead - 1; x < chunkSize; x++)
                                    {
                                        chunkData[x] = 0;
                                    }
                                }
                                cryptoStream.Write(chunkData, 0, chunkSize);
                            }
                        }
                        cryptoStream.FlushFinalBlock();
                    }
                }
            }            
        }

        private static void Dec(string encryptedFileName, string decryptedFileName)
        {
            PasswordDeriveBytes passwordDB = new PasswordDeriveBytes("ThisIsMyPassword", Encoding.ASCII.GetBytes("thisIsMysalt!"), "MD5", 2);
            byte[] passwordBytes = passwordDB.GetBytes(128 / 8);

            using (FileStream fsInput = File.OpenRead(encryptedFileName))
            {
                using (FileStream fsOutput = File.OpenWrite(decryptedFileName))
                {
                    byte[] buffer = new byte[8];
                    fsInput.Read(buffer, 0, 8);

                    long fileLength = BitConverter.ToInt64(buffer, 0);

                    byte[] IVBytes = new byte[16];
                    fsInput.Read(IVBytes, 0, 16);


                    RijndaelManaged symmetricKey = new RijndaelManaged() { Mode = CipherMode.CBC,Padding=PaddingMode.ANSIX923};
                    ICryptoTransform decryptor = symmetricKey.CreateDecryptor(passwordBytes, IVBytes);

                    using (CryptoStream cryptoStream = new CryptoStream(fsOutput, decryptor, CryptoStreamMode.Write))
                    {
                        for (long i = 0; i < fsInput.Length; i += chunkSize)
                        {
                            byte[] chunkData = new byte[chunkSize];
                            int bytesRead = 0;
                            while ((bytesRead = fsInput.Read(chunkData, 0, chunkSize)) > 0)
                            {
                                cryptoStream.Write(chunkData, 0, bytesRead);
                            }
                        }
                    }
                }
            }
        }

Knew it had to be a schoolboy error :P

Orcutt

if (bytesRead != 16) should be if (bytesRead != chunkSize). Correct? – Ironlike

With your first loop, aren't you creating an ever-increasing buffer size with each iteration instead of a static buffer? – Scullion

Actually you're reading through the file once and then going to throw a whole lot of exceptions since the stream pointer will be at EOF. – Scullion

@Scullion not at all. fsInput.Read() will return zero once it hits the end of the file, continuously. The while clause takes care of trying to write zero data, and the whole thing will graciously come out of the loops with zero exceptions. This code is currently running, without exceptions, on a server as a background service. Try it yourself ;) – Orcutt

I did... and modified the code to just use the while loop to read and encrypt the entire contents, without error. when I use the outer for loop, it throws an incredible number of exceptions. – Scullion

Answer 2

The Stream.Read method, returns the number of bytes actually being read from the stream.

You should use this return value as the last parameter in the Write method on the next line.

My code would look like this:

byte[] chunkData = new byte[chunkSize];   
var bytesRead = 0;
while ((bytesRead = fsInput.Read(chunkData, 0, chunkSize)) > 0)
{
    cryptoStream.Write(chunkData, 0, bytesRead);
}

Admonitory

OK - that's a fair comment, but I don't think this will work, as I have read you need to write blocks of bytes divisible by 16 for the encryption to work. If you use the code you mentioned, it's possible that you'll write, say, a chunk of just 4 bytes? – Orcutt

As thought, once this is implemented, encryption fails unless you "pad" the end of the file with zero'd bytes to ensure you get a total file length that's divisible by 16. However, doing that still didn't resolve the issue, as I either get the same exception regarding invalid padding, or if I remove padding, end up with a broken unencrypted file. :( – Orcutt

In my production code, I actually use a do...while loop, and I use using everywhere, maybe that's the trick. Use using instead of Close and Dispose yourself. – Admonitory

funnily enough, that is exactly what I'm trying out this very moment. I'll let you know the outcome momentarily - thanks. – Orcutt

sadly still no joy. I think it's something more obvious than that... I'm working on another idea at present, so we'll see how that goes :) – Orcutt

yep - that didn't work either - with all the suggested code changes in place, I receive the same error. – Orcutt

Got it in the end :D. See my answer for the solution if you're interested, but thanks for the help anyways :) – Orcutt

Answer 3

There is a CryptoStream class to encrypt/decrypt Streams

Sefton

Hi, welcome to Stackoverflow. I see, you're new here. Please read the guidelines for writing a good, standalone and reproducable answer here: stackoverflow.com/help/how-to-answer – Insomnia